Recovering from planetary disaster

So, as Devrim has already posted, there was a major disaster with Planet PostgreSQL a while ago. The result was that both the aggregator (www.planetpostgresql.org) and the blog-hosting-for-many-PostgreSQL-community-people-including-me (people.planetpostgresql.org) went down. This was not so good, but it happens. Also, there were no backups. This is a lot worse. This is a resource with a lot of high-value information, and it's now been offline for a long time. We still do not know exactly what happened, but Devrim has now indicated that we may be able to recover the data somehow at some point, but we don't know when - hopefully soon.

There were two parts to this:

The aggregator

The aggregator, Planet PostgreSQL, contained no actual data (that's in it's nature) other than the list of blogs it was pulling from. And since we had already been experimenting with some new software running on a community server to do this, we could rapidly bring this server and software into production when we realized this issue wouldn't be resolved quickly. Moving the planet over to a community managed server was discussed and agreed on a long time ago, but I was too lazy to finish off the last pieces of the software. This was now done in a hurry, during pgday.eu, to get something up. Since we could not reach Devrim (his email was also on the server that was down), we set up http://planet.postgresql.org in the official postgresql.org namespace to point to this server. When we got hold of Devrim, he also changed www.planetpostgresql.org to point to this new, community managed, planet.

The day after this, when Devrim had a few more things under control, he came back to us saying that he was not comfortable having Planet PostgreSQL under community control, co-managed by him and the rest of the team that manages our infrastructure. At this point we pushed for the point that had been made a long time ago - the web team is not comfortable having such an important service with such a prominent location on www.postgresql.org not managed by the community team (with Devrim still being the head maintainer, just with the rest of the team as backup in case something happened - and of course with the standard community requirements on backups etc). Devrim's choice in this case was to repoint the planetpostgresql.org domain to his server (even though it at the time had nothing on it - though he did get the aggregator back up not too long after that), and ask us to remove it from the front page of the website if we would not accept that. This is when the decision was made to keep http://planet.postgresql.org running as a community managed service and as the official PostgreSQL blog aggregator service that is linked from the main website.

The conclusion of this, was a fork of the aggregator service. There is now the PostgreSQL community official aggregator, at http://planet.postgresql.org, and there is Devrim's aggregator at http://www.planetpostgresql.org. They both provide similar service to the end user, through different software and different policies. Only the first one feeds to www.postgresql.org.

This has exactly nothing to do with the blog hosting, this only deals with the aggregator.

The blog hosting

The blog-hosting service at people.planetpostgresql.org is the one that contained all the data. This is the part that we are still hoping we will be able to recover some data from. This is a second service provided by Devrim, that is unrelated to the aggregator - other than that they were running on the same, crashed box.

There are no plans by the PostgreSQL web/infrastructure team to provide this service. There are a lot of services out there on the net that provide blog hosting, Devrim's included (once he gets the system back online). Both commercial and free. The aggregation service will be equally happy to work with both. So if you are looking to set up a PostgreSQL blog, either talk to Devrim or look at one of the external offerings.

I've personally decided to move my blog to my own hosting. It's now available at http://blog.hagander.net. I will try to recover the old data as soon as Devrim makes it available either into this blog, or into the old location, depending on what's possible. I know others, for example Robert have done the same. AFAIK, we were both considering this beforehand as well but found the existing service convenient. The feeds have been updated on the main planet site, but if you were using the direct feeds, you need to update the link (see sidebar for feed links). And a big thanks to Devrim for hosting my blog there as long as he did.

I give no recommendations to other people who had their blogs on people.planetpostgresql.org about what to do with their blogs, and there will be no statement from the web or infrastructure team about it. It's an unrelated service, that everybody needs to decide on their own about.

The conclusion of this part is that my blog now lives at a new URL. Update your links. Sorry for the inconvenience.


Comments

you might want to check http://warrick.cs.odu.edu/.

from their website:

"Warrick is a free utility for reconstructing (or recovering) a website when a back-up is not available. Warrick will search the following web repositories for missing resources: Internet Archive, Google, Live Search, and Yahoo. All of the resources are gathered together and provided to you as a single collection of files. Please read our disclaimer."

Posted on Oct 28, 2008 at 10:43 by depesz.

I'll check. I have scraped Google and Internet Archive manually already, and recovered some but far from all posts. Though I think the most popular ones would've been there.

But as Devrim is saying the data will be recoverable, I'll just wait for him.

Posted on Oct 28, 2008 at 12:28 by Magnus.

Conferences

I speak at and organize conferences around Open Source in general and PostgreSQL in particular.

Upcoming

Stockholm PUG 2016/4
Aug 31, 2016
Stockholm, Sweden
Postgres Open
Sep 13-16, 2016
Dallas, USA
Postgres Vision 2016
Oct 11-Nov 13, 2016
San Francisco, USA
PGConf.EU 2016
Nov 1-Jan 4, 2016
Tallinn, Estonia
PGConf.Asia 2016
Dec 2-3, 2016
Tokyo, Japan

Past

PG Day'16 Russia
Jul 6-8, 2016
St Petersburg, Russia
PGDay UK 2016
Jul 05, 2016
London, UK
Stockholm PUG 2016/3
Jun 16, 2016
Stockholm, Sweden
PGCon
May 17-21, 2016
Ottawa, Canada
Pycon Sweden
May 9-10, 2016
Stockholm, Sweden
More past conferences