Tag Archives: system administration

My webserver update checklist

Ok, so I updated from Ubuntu 9.10 to 10.04 on this blog’s webserver yesterday. Just about everything went well with only a minor downtime while ‘apt’ processed the MySQL and Apache packages.

A short introduction to the webserver might be in place:

So, not very impressive performance-wise or storage wise for that matter. And only the standard free & Free software components. Rather ordinary and pretty much out of the box.

What might be a bit out of the ordinary is mod_perl, which enables me to run a MySQL backend for the VHost configurations. What it does is that it dynamically adds configuration snippets when Apache loads its configuration. Using my knowledge in Perl, it’s quite easy to make advanced, unique configurations without very much administration and file handling. Only problem is that I have to rely on the MySQL database being up and running… But then again, most sites require databases anyhow. And a fail-safe system simply seems too overkill.

Anyhow, I mentioned mpm-worker too. Which leads me to the topic for this post – my webserver update checklist. I’m a rather strong opponent of dumbification and prefer that tasks seldom should be automated. Certain, irrelevant or unnecessary, tasks may very well be automated (such as with my mod_perl configuration). You see, assuming and guessing has never been a machine’s strong side. Brute-force and tediousness for all it’s worth, but – for the love of Adanot guessing!

Assumption, I guess (…hah…), led apt to reinstall libapache2-mod-php5 and in turn mpm-prefork. Without asking. This of course overrided the suEXEC + php-cgi setting, causing all my previously suEXECd websites running as user ‘www-data’, effectively blocking write capabilites for vhosts (chown/chmod stuff) in their respective folder and opening up all vhosts to be readable by any PHP script running on any vhosts. Pöh.

After disabling libapache2-mod-php5 and regaining some sense of security, I was content for a while. With the minor amount of visitors me and other hosted sites get, I didn’t notice any other problems until the next morning. Soon I noticed that sites were inaccessible, slow and the load was pacing at about 50. Fortunately this was something I learned to configure after the noticable downgrade I had to do when my office was raided. At that point I could, at least, take the same machine but only half the amount of RAM. At the same point I switched back to Apache2 from lighttpd. This led to many hours of configuring and tweaking and the following conclusion:

Apache2 has the default setting to use libapache2-mod-php5. Suck my balls. Sure it’s fast, easy to configure and works for your average Joe-user, but it’s a severe security hazard for anyone hosting for a third-party. Also, libapache2-mod-php5 seems (based on apt package rules) incompatible with mpm-worker, causing apt to uninstall it and replace it with mpm-prefork. And for those of you who haven’t tried the two under a heavy load (or even just as a hobby), I can tell you the performance increase of mpm-worker is huge. Heck, it can even make you accept Apache2 over lighttpd without having to choose between configurability and speed too harshly.

So my personal confusion lies in why Ubuntu’s (and Debian’s?) repository forces back a lousy, insecure configuration when doing apt dist-upgrade. I agree that it’s partly my fault for not inspecting the package lists better or for that matter double-checking.

In any case, now my webserver update checklist has “reinstall mpm-worker if removed” (which happily removes both mpm-prefork and libapache2-mod-php5) right next to “ignore any php.ini updates” and “MySQL’s best the way with my own my.cnf”.

PS. Unfortunately mod_fcgid appears to have a new bug since v2.2 that Karmic used (Lucid uses v2.3.4 as of writing). Any file that’s larger than the value of FcgidMaxRequestInMem may (has in all of my cases) be corrupted on upload. It was a long time ago since I had this much problems with an upgrade, besides other peoples’ Windows related stuff of course. A workaround is to set the FcgidMaxRequestInMem value higher – though the problem has been fixed in mod_fcgid v2.3.5. However, a fix has been released already in the package libapache2-mod-fcgid – 1:2.3.4-2ubuntu0.1.

System administration hardcore

There’s music for every occasion. Monotonous, hierarchical thinking (monotonous, hierarchical programming) demands nice, retro chip music or similar. (at least it makes you feel more hackerish.)

Reggae suits the softer, more laid back moments. Or just general throw-your-hands-around dancing of joy.

System administration, when retarded users run random trojans every once in a while, demands the hardest, most destructive of hardcore.

So meet Lesra, from Umeå. Their latest album available at The Pirate Bay of course. Fortunately they’re playing at Verket tomorrow. I might use that for the very effective therapy-through-moshpit method.

Yes. I do ponder banning my subordinated lusers (BOFH speak!) from the internet. If they can’t handle it, they shan’t use it!

MySQL replication

Uhm. I’m about to fiddle about with MySQL database replication to load balance database access. It’ll be interesting to see how well it actually works.

If I manage fine, it will finally be a good time to centralise my database management. This includes getting a single mail server db for accounts/password, instead of having two slightly different mail server setups which can’t work together – instead of backup MX servers etc.

So by centralising I don’t mean a monolithic setup. More like implementing identical systems working as redundant fallbacks, while maintaining a single entry point. Though the entry point might still be rather redundant if I work some black IP magic.

Also it will increase backup simplicity, which I have noticed is beginning to feel necessary. For backup I’m currently using sloppy rsyncing between systems. This could easily be outsourced to two or more separate systems just filled to the brim with harddrives. Tape is so expensive. :)

Silently malfunctioning harddrive

There are some hardware difficulties which are easy enough to manage. Either your computer stops/reboots every once in a while – check the PSU or faulty memory. Or maybe your harddrive is making the click of death. It might even be as obvious as a bulgy, leaking capacitor on the motherboard.

But when you start noticing new ways for a computer to silently fail, you’re up for an interesting – though frustrating – night without sleep.

The other night the server for Asian DVD Club stopped responding. Sure enough I hopped on my bike and got to the server just to see that it still reacted on Num Lock switching, but there was no VGA output.

Bah! Humbug.

Having gone through a memory check, PSU change, Live USB boot, and even a motherboard switch etc… I had broken it down to the harddrives. Granted, I had my suspicions to start with because the disk activity LED was kept lit whenever the computer crashed or stopped responding.

But there were no DMA errors. S.M.A.R.T. didn’t notice anything peculiar. It was as fast as usual and wouldn’t immediatly crash on a high load. The server _seemed_ to be running ok (for my tests) when I unplugged the disks from my secondary IDE controller – but lo, I was fooled. I just didn’t try it hard enough

A couple of hours later, I dd if=/dev/sda of=/dev/sdb to a different drive (fortunately only 20GB system drive) and popped it in. Smeg me sideways, it “didn’t work”. I booted from the new disk and all, even fscked all the filesystems and stuff.

So I was confused. I took my mind off things and came back after a while. What I didn’t think could have been a problem apparently was. The old system drive was connected to the secondary IDE controller. The new system disk to the primary. Then why on Earth would Linux freeze up when accessing the old drive?

Sigh. By this time I had already reinstalled Ubuntu server 9.04 (jaunty) to make sure it wasn’t ReiserFS spooking up. Now everything is ext4 and cleanly installed – aka barely configured.

And it seems to all be working fine. The only problem I actually had was a harddrive that would lock up the IDE controller (both of them even!). Even though it seemed perfectly healthy.

Mail backup with rsync and cron

Quick and dirty crontab, but ’tis good enough I suppose?

0 1-23 * * * rsync -avP /srv/mail/ /backup/auto/mail/hourly/
0 0 * * * rsync -avP --del /srv/mail/ /backup/auto/mail/hourly/
@daily rsync -avP --del /srv/mail/ /backup/auto/mail/daily/
@weekly rsync -avP --del /srv/mail/ /backup/auto/mail/weekly/
@monthly rsync -avP --del /srv/mail/ /backup/auto/mail/monthly/

Further on, /backup will be nfs-mounted over OpenVPN or similar. Currently offline-backup is done once a week over the internets to two separate locations.

Database is on a separate system and handled there in a similar fashion. Reports are automatically mailed through crontab’s output catcher.

However I’m still pondering how to do /etc neatly. I’m guessing filelist argument to rsync with the stuff that just have to be replaced. Configurations are rather identical between Debian/Ubuntu machines at least.