Google made some changes to their site, so the Crawler bash script doesn't
work anymore.
However the example links the author provides which allow for crawling a
Google Group and getting all the messages in text form DO still work (just
the logic of the script needs to change - the text-only link used to return
all topics in one giant list and now it's broken up into 20 topics/page).
Since I am not a super-linux-bash-script user, I will probably use the
example links provided by the author to create a new, python-based scraper
script that will first build a list of all of the topics, then get the
messages from each topic. One folder per topic. (The current script just
dumps all the messages in one folder, I think one-folder-per-topic makes
more sense).
Then we can use something like http://www.mhonarc.org/ to create an
HTML-ized archive of the old groups for posterity.
Andrew B.
On Mon, Jul 27, 2015 at 10:00 AM, Andrew Bingham <***@gmail.com> wrote:
> Lets spread the work around a bit - I have a small private Google Group
> from a project I did a while back that is now defunct. It probably has
> 15-20 topics on it.
>
> I'll take point on trying different methods of getting the data out of a
> Google Group using that group as a test, then moving on to the main N8VEM
> group.
>
> I'll get back to everyone by the end of the week.
>
> Andrew B.
>
> On Mon, Jul 27, 2015 at 8:27 AM, Alan Hightower <***@alanlee.org> wrote:
>
>>
>>
>>
>>
>>
>> Also a couple California yahoo's named John M. and Gary K. sent me a huge
>> stack of S100 boards this week too. They suckered me with a fun
>> distraction. Rev.P2 of the 4GB S100 RAM boards came in this week from the
>> board house. So I will need to spend some time getting them up and running
>> with the 386 jig board. It's just a hobby for me too. I have a pretty
>> demanding day job during the week.
>>
>>
>>
>> -Alan
>>
>>
>>
>> On 2015-07-27 11:17, Alan Hightower wrote:
>>
>> I need to do an OS upgrade on the server this week. I'm waiting on
>> that. Tentatively it should be done by tomorrow. Then I can start moving
>> data over during this week.
>>
>>
>>
>> It would be greatly beneficial to get a dump of site data in bulk - if
>> only to use as a check-list for content migration. If someone with admin
>> rights could export via the settings panel or otherwise get a bulk tar, it
>> would save some time. If not, I believe I can inject my authentication
>> cookie into a wget crawl and download most of the content direct.
>>
>>
>>
>> -Alan
>>
>>
>>
>>
>>
>>
>> On 2015-07-27 10:54, Andrew Lynch wrote:
>>
>>
>> There are reasons but at a bare minimum to disassociate N8VEM from the
>> project name. Consolidation under a new project website using common
>> tools. Reduce maintenance overhead. Reduce dependence on big data and the
>> obvious privacy concerns. I am stuck with an ongoing time consuming
>> responsibility with no effective way to delegate. It is well past time to
>> transition to a separate project that can support the community properly
>> preferably one consolidated site that supports both the wiki and the
>> mailing lists.
>>
>> PBWorks and Google Groups are poor choices I made at the outset when I
>> thought the group was small and manageable. It turns out the project grew
>> uncontrollably and led to multiple scaling problems. It is time to scrap
>> this kludge iteration and redesign the project. It happens and its normal.
>>
>>
>>
>> ------------------------------
>> *From:* yoda <***@r2d2.org>
>> *To:* N8VEM <***@googlegroups.com>
>> *Cc:* ***@yahoo.com; ***@gmail.com
>> *Sent:* Monday, July 27, 2015 10:42 AM
>> *Subject:* Re: [N8VEM: 19869] n8vem homepage requiring login?
>>
>> Thought the discussion was only to get off of PBworks - not abandon
>> Google Groups - it is working so why not leave it alone?
>>
>>
>>
>>
>> On Sunday, July 26, 2015 at 7:27:22 PM UTC-5, Andrew Bingham wrote:
>>
>> As far as I can tell there is no way to get data out of a Google Groups
>> group, so if these get shut down we will loose all of the posts and
>> information from the past.
>>
>> Andrew B
>>
>> On Friday, July 24, 2015 at 2:07:55 PM UTC-7, lynchaj wrote:
>>
>> Hi Alan
>>
>> How is the wiki transition coming along?
>>
>> Do you have an estimated date for a fully operational replacement site so
>> I can shut down the pbworks wiki?
>>
>> After the wiki is transferred we need a plan to move the mailing lists as
>> well.
>>
>> Once the new site is working it should be straight forward.
>>
>> Andrew Lynch
>>
>> *From:* ***@googlegroups.com [mailto:***@googlegroups.com] *On
>> Behalf Of *Alan Hightower
>> *Sent:* Wednesday, July 15, 2015 12:11 PM
>> *To:* ***@googlegroups.com
>> *Subject:* Re: [N8VEM: 19869] n8vem homepage requiring login?
>>
>>
>> See inline.
>>
>> On 2015-07-15 11:09, Nikolay Dimitrov wrote:
>>
>> Hi Alan,
>>
>> Do you have some hosting constraints in terms of monthly bandwidth
>> and/or disk storage? Also, do you have your preferences for the CMS
>> which will be installed?
>>
>>
>>
>> There are constraints. Years ago I found it less costly just to lease
>> virtual servers (LKVM, etc) rather than hauling my own 1U racks downtown in
>> my hatchback. I figured it out after the second or third time rebuilding
>> raid sets at 3am downtown. I host a few sites on the same machine (Linux)
>> and plan on a few others near term. So the resources will be shared.
>> However I don't see the bandwidth or disk-storage of this group causing
>> undue load - even if also hosting git/svn, mailman, and other services. My
>> preference is Wordpress. The on-going discussion has pressed the pause
>> button in recent weeks. It ultimately comes down to the page content
>> producers and maintainers.
>>
>>
>>
>>
>> I guess that it's better to start sooner or later to copy/migrate some
>> content to the new server, as otherwise we'll never complete it :D.
>>
>> Regarding the hostname: it looks like this is still an on-going
>> discussion, but I think it shouldn't stop the migration of the
>> projects' design documents.
>>
>>
>> Fundamentally correct but I am well versed in the scientific properties
>> of technical inertia. If we're mostly done with the open discussion part
>> of this, the best way forward is just to take a lead and do it. Unless
>> there are any objections (prefer to reply direct and privately):
>>
>> I need to do a planned maintenance to the current web-host which I can
>> complete this weekend.
>>
>> Next week: 1) I will register retrosbc.org. 2) Migrate the current
>> site. 3) Re-org a few basic things. 4) Lift the underconstruction page.
>> 5) And open new account creation.
>>
>> Follow-on week: 6) I will put together the start of a site hand-book
>> page(s) for different level(s) of users. 7) Start helping in the migration
>> and updating of legacy project content.
>>
>> That same week is a bit of a sabbatical from by 9-5 to play catch-up on
>> other personal projects. I can devote much more time during that period.
>>
>> It would be nice to designate a page owner for every page. Even if the
>> only duty is taking content off-line so it doesn't confuse other. I've
>> discussed a few pages with people involved in the creation of the original
>> boards and the summary was the page had not been updated in
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "N8VEM" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to n8vem+***@googlegroups.com.
>> To post to this group, send email to ***@googlegroups.com.
>> Visit this group at http://groups.google.com/group/n8vem.
>> For more options, visit https://groups.google.com/d/optout.
>>
>>
>>
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "N8VEM" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to n8vem+***@googlegroups.com.
>> To post to this group, send email to ***@googlegroups.com.
>> Visit this group at http://groups.google.com/group/n8vem.
>> For more options, visit https://groups.google.com/d/optout.
>>
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "N8VEM" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to n8vem+***@googlegroups.com.
>> To post to this group, send email to ***@googlegroups.com.
>> Visit this group at http://groups.google.com/group/n8vem.
>> For more options, visit https://groups.google.com/d/optout.
>>
>>
>
--
You received this message because you are subscribed to the Google Groups "N8VEM" group.
To unsubscribe from this group and stop receiving emails from it, send an email to n8vem+***@googlegroups.com.
To post to this group, send email to ***@googlegroups.com.
Visit this group at http://groups.google.com/group/n8vem.
For more options, visit https://groups.google.com/d/optout.