My webserver update checklist

Ok, so I updated from Ubuntu 9.10 to 10.04 on this blog’s webserver yesterday. Just about everything went well with only a minor downtime while ‘apt’ processed the MySQL and Apache packages.

A short introduction to the webserver might be in place:

So, not very impressive performance-wise or storage wise for that matter. And only the standard free & Free software components. Rather ordinary and pretty much out of the box.

What might be a bit out of the ordinary is mod_perl, which enables me to run a MySQL backend for the VHost configurations. What it does is that it dynamically adds configuration snippets when Apache loads its configuration. Using my knowledge in Perl, it’s quite easy to make advanced, unique configurations without very much administration and file handling. Only problem is that I have to rely on the MySQL database being up and running… But then again, most sites require databases anyhow. And a fail-safe system simply seems too overkill.

Anyhow, I mentioned mpm-worker too. Which leads me to the topic for this post – my webserver update checklist. I’m a rather strong opponent of dumbification and prefer that tasks seldom should be automated. Certain, irrelevant or unnecessary, tasks may very well be automated (such as with my mod_perl configuration). You see, assuming and guessing has never been a machine’s strong side. Brute-force and tediousness for all it’s worth, but – for the love of Adanot guessing!

Assumption, I guess (…hah…), led apt to reinstall libapache2-mod-php5 and in turn mpm-prefork. Without asking. This of course overrided the suEXEC + php-cgi setting, causing all my previously suEXECd websites running as user ‘www-data’, effectively blocking write capabilites for vhosts (chown/chmod stuff) in their respective folder and opening up all vhosts to be readable by any PHP script running on any vhosts. Pöh.

After disabling libapache2-mod-php5 and regaining some sense of security, I was content for a while. With the minor amount of visitors me and other hosted sites get, I didn’t notice any other problems until the next morning. Soon I noticed that sites were inaccessible, slow and the load was pacing at about 50. Fortunately this was something I learned to configure after the noticable downgrade I had to do when my office was raided. At that point I could, at least, take the same machine but only half the amount of RAM. At the same point I switched back to Apache2 from lighttpd. This led to many hours of configuring and tweaking and the following conclusion:

Apache2 has the default setting to use libapache2-mod-php5. Suck my balls. Sure it’s fast, easy to configure and works for your average Joe-user, but it’s a severe security hazard for anyone hosting for a third-party. Also, libapache2-mod-php5 seems (based on apt package rules) incompatible with mpm-worker, causing apt to uninstall it and replace it with mpm-prefork. And for those of you who haven’t tried the two under a heavy load (or even just as a hobby), I can tell you the performance increase of mpm-worker is huge. Heck, it can even make you accept Apache2 over lighttpd without having to choose between configurability and speed too harshly.

So my personal confusion lies in why Ubuntu’s (and Debian’s?) repository forces back a lousy, insecure configuration when doing apt dist-upgrade. I agree that it’s partly my fault for not inspecting the package lists better or for that matter double-checking.

In any case, now my webserver update checklist has “reinstall mpm-worker if removed” (which happily removes both mpm-prefork and libapache2-mod-php5) right next to “ignore any php.ini updates” and “MySQL’s best the way with my own my.cnf”.

PS. Unfortunately mod_fcgid appears to have a new bug since v2.2 that Karmic used (Lucid uses v2.3.4 as of writing). Any file that’s larger than the value of FcgidMaxRequestInMem may (has in all of my cases) be corrupted on upload. It was a long time ago since I had this much problems with an upgrade, besides other peoples’ Windows related stuff of course. A workaround is to set the FcgidMaxRequestInMem value higher – though the problem has been fixed in mod_fcgid v2.3.5. However, a fix has been released already in the package libapache2-mod-fcgid – 1:2.3.4-2ubuntu0.1.

2 thoughts on “My webserver update checklist”

  1. “on a single 20GB disk for all websites”, i’ll happily send you another 20gb disk so you can atleast run raid1 ;P

    my single drive server started acting up this morning, “The device, \Device\Harddisk0, has a bad block.”
    i’m running 2003r2 and a virtual freebsd inside that, apparently the bad blocks at the moment are right where my http-access.log file resides, as soon as anything attempts to read/write there i get lockups. not really sure how to access this tomorrow, but just as a warning, never use single-drive systems ;)

Leave a Reply

Your email address will not be published. Required fields are marked *