At PG-east^H^H^H^H^H^H^HJDCon here in Philly yesterday, Dave announced the infrastructure changes that we decided on late last year, and have been working on and off to get the groundwork done for over the past months (mostly off, or we'd be done already). Since I've had a number of questions around it, I'm going to post a summary here.
The bottom line is, we're changing - over time - how we're going to run and manage the infrastructure behind the postgresql.org services. Things like website, source code repositories, etc. The reason is simple - the way we are doing it now has become too hard to maintain, and just requires too much manual work.
There are several parts to these changes. For one, we are building automated methods for a lot of things that previously required a little manual work. But even a little manual work rapidly becomes very time-consuming when you have a lot of machines - and the number of machines we have have been increasing fast over the past couple of years.
The biggest change is that we will be moving our services off FreeBSD, that we use now, onto Debian GNU/Linux. At the same time, we will switch virtualization solution from FreeBSD Jails to KVM. FreeBSD jails have served us very well over time, but now that we have access to a well working KVM product that uses hardware virtualization, the gains of having full virtualization are much easier to get at.
One of the main drivers for the change is that maintaining ports-based installs are just taking way too much time. The new system will be based heavily around using Debian packages and the apt system, to the point that everything being installed will always be done using "meta-packages" that will ensure that all our machines look the same way. This also plugs into our monitoring systems very well, making applying (security) updates across the many machines much easier. In particular, it should get rid of what's sometimes a multi-day operation to get everything security patched, due to dependencies and the slowness of updating ports.
It is quite possible - in fact, I'd say probable - that there are perfectly fine ways of doing this on FreeBSD. But this outlines another big reason for this change - it's simply a lot easier to find people who can do these things on a Linux based system today. Our efforts are entirely volunteer based, and in the end we just don't have people who know enough FreeBSD volunteering to do this work. In fact, we have had several people retire from the sysadmin team over the years specifically because they refuse to work with FreeBSD. We don't expect this change to magically create more volunteers, but it will make it easier to recruit in the future.
The system will also automate a lot of the configuration work that needs to be done manually today, as well as automatically configure our monitoring systems (primarily Nagios and Munin) and backups. And finally, the system will automatically provision (and un-provision) users and access permissions across all machines from a central repository.
The code is not entirely done yet, but we do have enough in place to automatically provision and configure basic virtual machines in just minutes, as well as much of the configuration. We have prototype deployments online, but we haven't started moving production services yet. We'll hope to get started on that shortly. It will obviously take a long time before we have migrated all services - in fact, we may never migrate all services. The focus will obviously be on the most manpower-intensive ones first.
As usual the code that's used to build these services will be published under the PostgreSQL licence once it's finished. Some security sensitive parts may be removed, but the bulk of it is very generic and should be re-usable as a whole or in parts.
Another issue that's going to delay some changes is the fact that a few of our infrastructure servers are currently not VT-capable, and thus cannot run KVM. In the short term these machines will continue to run FreeBSD Jails, but eventually we want to replace these machines with more modern ones. So if you are sitting on one or more VT-capable machines with enough RAM to run a number of virtual machines, hosted in a datacenter somewhere that can provide us with multiple IP addresses and a decent availability of network and services, that you would like donate to the PostgreSQL project, please get in touch with our sysadmins team to see if this is something we can work with!
I wonder if there was an ask for volunteers on any of the FreeBSD lists.
As someone who has worked with FreeBSD for many years and has had to work with Linux for 2 years I would tend to disagree that port/package management is easier in Linux.
I wasn't even consulted about this, and I'm not sure why anyone would think I'd actually agree to such a move ... but, then again,that's probably why I wasn't consulted ...
Marc,
I sent you an email about this. If the true reason for this is the need of volunteers you can count me in to help with the FreeBSD admin.
Francisco,
I like FreeBSD. (I also like Linux and Solaris) However, it's simply true that we can pick up 10X as many WWW volunteers for Debian as we can for FreeBSD. And when we're asking for volunteer help, we can't tell people what tools to use. We've stuck with FreeBSD for 15 years, and for the last 8 have limped along with never quite enough people.
Given the statement that there is a lack of FreeBSD admin volunteers I simply offered to help.
Please reconsider using your own tools vs. something like Puppet. We too started to use our own set of tools for auto deployment, even though I was aware of Puppet at its early stages, simply due to lack of time to learn it, but eventually we switched and I don't regret it (or the need to learn Ruby and introducing yet another language into our environment). Puppet's power lies in the size of its community and support. Granted, it's a learning curve to climb and I'm still not 100%25 comfortable with Puppet syntax but at least I'm not alone when I have a question to tuckle with it. I suspect you are digging yourself out'of one hole (FreeBSD) only to get yourself into another (home grown tools reinventing a wheel many others have already invented before you).
Interesting. Not a direct match for what we want, but certainly an interesting project. With a great name!
We (obviously) don't ask for "general volunteers" to give root access to our most important servers to. We are only interested in people who have been active in the PostgreSQL community for years, and have contributed enough to other parts to be trusted with something like that.
Thanks for offering, though :) We'll get in touch if/when we find the need!
It was decided after a number of discussion sessions both online and in person between the people who maintain the vast majority of the PostgreSQL infrastructure. I for one have tried to initiate conversations with you about it, with no response.
And either way, we don't always get 100%25 buyin to any decision in the PostgreSQL project - code-wise, web-wise, advocacy-wise, infrastructure-wise, whichever. But with a large majority in favor of something, we usually proceed, as we did here.
This is not about web volunteers, this is about the infrastructure services. Web is just a very small part of that.
Other than that, your arguments are correct, of course.
This is something we considered very carefully before making a decision. The fact is that just the customizations to Puppet end up being a lot more code than what we have in total now. There are a number of existing things we have that it can't easily be made to make use of
It also brings in a large set of dependencies on all the machines (with the client being in Ruby, it pulls in everything from that), which is certainly not a dealbreaker, but it comes in on the minus side.
The stuff we have now does exactly what we want and nothing more. That keeps it really really really really simple. Which is the whole point. It doesn't automate 100%25 - it automates the stuff that takes unnecessary time.
I am 100%25 for this move.
New comments can no longer be posted on this entry.
debian? KVM? Have a look at http://www.niftyname.org/