I can't seem to find this on Beatport or iTunes to buy :(

Sometimes, I feel like people give Java a bad rap. It's a language that's in demand out on the field and after using it for my school classes, I have found that it is easy to program in, it performs very well and your code is portable/cross-platform. Sounds great, right? At first glance Java magically turns everything that's hard to do into something easy.

But then I'll use Java-based software and somehow it manages to consistently be extremely complex to setup/configure (ie, tomcat+webapps), to have horrible looking UIs (ie, LimeWire, FreeNode, OpenOffice) and often it consumes lots of resources needlessly (ie, OpenOffice, LimeWire). I mean seriously, you can get a PHP or Python-enabled webserver up and running in under 2 commands on RHEL and then you just need to create a single script file to start serving pages. The Java community really needs to spend some time working on developer/user experience in my opinion.

I've always cared a lot about user experience because I find that no matter how great a single piece or a collection of software is, it is the user experience that shapes your impression of that software. That said, I was cleaning up my (very, very messy) desktop and came across an old backup of a Fedora installation. There was a file on my desktop from July 2008 that I had completely forgotten about... I had collected my thoughts at the time on how Linux-based distributions could be improved to make the user experience better. It's really neat to see how many of these have been implemented in only 3 short years:

my apologies for the messy read, I tend to write my notes in Wiki format and I don't feel like copy/pasteing

This how-to will show you how to configure:

• An Apache 2 web server using virtual hosts
• The ITK MPM allows each virtual host to serve requests as its own user/group
• mod_ssl to serve pages over the secure HTTP (HTTPS) protocol
• mod_security to help prevents everything from SQL injections to data leaks
• mod_php for PHP scripts along with mod_suhosin to help protect mitigate risks from known and unknown flaws in PHP scripts

Rebuilding httpd for ITK

About privilege separation

By default, the Apache web server runs as the 'apache' user. This is good because a successful attack on the web server will only cause limited damage to the system, as they do not have root access. However on a shared servers which hosts multiple websites, an attack can still be very dangerous because the 'apache' user is used by all of the websites. Thus, an attack on one website can potentially grant access to any other websites that are being hosted on the system!

Privilege separation is a technique that can be used to mitigate the risk of an attack against a shared hosting server. By allowing webpages to be served by different users, each host can be assigned its own user and so the damage from a site hack will be limited to just that account and not all of the accounts hosted on the system. We will be using the ITK Apache MPM to achieve this.

Rebuild process

Unfortunately, the ITK MPM is not included in the stock httpd distribution. Fortunately, it is relatively easy to add it. Run this as your regular user to install the httpd source RPM and download the ITK patch sources:

With that done, let's make a few quick modifications to the RPM spec file located at ~/rpmbuild/SPECS/httpd.spec.

We will first need to add the ITK patches. Find the last patch line in the RPM spec file, for example, Patch202: httpd-2.2.3-deflate2215.patch, then add after it:

Now in the %pre section - after the Red Hat patches are applied - add the following lines:

This will copy the (fully Red Hat-patched) prefork MPM code into a new folder and then apply the ITK patches.

In the %build section of the spec file, you will see these lines:

Following the same format, add a line for the ITK MPM:

Similarly, we find in the %install section:

Add a line for the ITK MPM:

Last of all, in the %check section there is another section were we need to add in the ITK MPM:

Change the middle line to read:

We are now ready to rebuild our ITK-enabled httpd package.

Install the dependencies listed and then re-run the rpmbuild again if necessary. After the build has finished, install your new RPMs:

Remember to replace [arch] in the second to last command with the appropriate value (most probably i686 for 32-bit machines or x86_64 for 64-bit machines). The last step is to set the httpd worker to the ITK binary by editing /etc/sysconfig/httpd and adding after the line #HTTPD=/usr/sbin/httpd.worker:

Installing add-on modules and additional software

Let's install PHP, AWStats and a few other useful add-ons:

Keep in mind that by default, mod_security's rules are very restrictive and you will almost certainly need to tweak them. Be sure to test thoroughly before enabling mod_security on your live server. You can modify the mod_security settings by editing the rule files in /etc/httpd/modsecurity.d.

Configuring the web server

You will find in a stock installation, no indexes are permitted at all and that visiting localhost displays the standard CentOS test page. This can be changed by editing /etc/httpd/conf.d/welcome.conf and comment out the entire LocationMatch section:

Next, let's add a small configuration file with some custom settings. Below we will try and prevent information leaks by restricting access to backup files (files ending with a "~"), .sql and .inc plaintext files which could potentially reveal critical information about how a site functions. We will also add a one-line include directive so that all files in the virtual hosts configuration directory get included as well.

The following will initialize a template VirtualHost configuration that can be used by a script to automate the installation of new VirtualHosts:

Similarly, we will setup a awstats template that can create configuration via a script:

(For those wondering, this template was derived by removing all comments from the stock awstats model configuration at /etc/awstats/awstats.model.conf and then substituting important values such as SiteDomain with variables so that can be easily changed via sed)

Because the web server will be using the ITK MPM, each site's requests will be run under its respective owner and group. This causes problems with the standard PHP session setup, which uses one directory owned by "apache" for all session information. To solve this, a session directory will need to be created for each user. Let's create the default one for the apache user now:

You are now ready to add VirtualHosts for your domains. See Adding a new web hosting account section of Administering the server below for more information on how to do so.

The last step is to add the firewall port exceptions for http/https and start Apache:

Suhosin PHP extension

We will now be configuring /etc/php.d/suhosin.ini. Because suhosin can interfere with some site's functionality, it is best to enable it in simulation mode so that errors are logged and you can configure suhosin accordingly:

Limit the maximum PHP memory_limit that scripts can set:

Lastly, disable session encryption (breaks roundcubemail if enabled):

Optional add-on: roundcubemail

Let's rebuild roundcubemail 0.3 for CentOS 5:

Edit the spec file roundcubemail.specand comment the line Requires: php-pear-Mail-mimeDecode so that it reads #Requires: php-pear-Mail-mimeDecode. This is the only change required for CentOS 5, so let's build the package now:

Now it is time to configure roundcubemail. In the file /etc/httpd/conf.d/roundcubemail.conf, add after the line :

As well, by default roundcubemail denies access to any non-localhost clients. To change this, comment out Deny from all and add Allow from all:

We will also need to create the roundcubemail database. If you have not setup your database server, follow the database tutorial now.

Then configure /etc/roundcubemail accordingly. You should now be able to access the webmail portal at http://www.yourdomain.com/roundcubemail.

But I want PHP 5.2!

PHP 5.2 is available in the CentOS 5 Testing repository. Certain PHP add-ons, such as php-suhosin, have not been rebuilt so you will need to do that manually.

Now rebuild and reinstall php-suhosin:

Now install the resulting php-suhosin binary RPM and then move the configuration file (/etc/php.d/suhosin.ini.rpmsave) back into place.

But I want PHP 5.3!

PHP 5.3 is available as of CentOS 5.6 the php53 package. Certain PHP add-ons, such as php-suhosin and php-mcrypt have not been rebuilt so you will need to do that manually. This is a bit tricky as you will need to rebuild the php-extras SRPM among others; I'll leave you to figure out the details. See the instructions above for PHP 5.2 for a rough workflow using php-suhosin as an example.

Some final words

Two 3rd party MPM modules satisfied my need privilege separation in this setup, the ITK MPM and the Peruser MPM. Each has its own advantages and disadvantages.

ITK's approach is to accept a request, determine which virtual host the request belongs to, fork, set the UID and GID of the fork, serve the request and then to terminate the fork after the request has been completed allowing it to fork again later. Because ITK is very simple in nature and functions similarly to the traditional prefork MPM, it is more compatible with modules like mod_ssl, mod_php and mod_python. That said, it is also slower than because of the fork/kill overhead associated with each request.

Peruser, on the other hand, starts a pool of processes that fork only on startup. There are multiplexers than accept requests, determine which virtual host the request belongs to, and then forwards the request to the appropriate process from the pool. This approach incurs practically no overhead as the process from the pool has already been forked and had its UID/GID changed. Peruser is therefore much faster than ITK; its performance is nearly almost on par with that of the traditional prefork MPM! The downfall of Peruser is that because of this connection passing from the multiplexer to a preforked process, it is more complex and has had a long history of bugs and incompatibilities. Experimental support for mod_ssl was only recently added, and upstream development still seems to have a number of bugs to work out. In addition, because Peruser requires at least one process per virtual host for the pool and then requires multiplexers to handle incoming requests, it is much more resource intensive that the ITK MPM, where a process can dynamically handle a request from any virtual host. For these reasons, I favoured ITK over Peruser.

Keep in mind that ITK does have one security flaw. The request headers must be processed in order to determine which virtual host that request belongs to, and therefore which UID/GID ITK would need to switch to. Because of this, all of the header processing is performed as root (albeit with restricted POSIX capabilities). Should someone be able to take advantage of a flaw in Apache or another module (e.g. mod_ssl), the server could theoretically be rooted.

There is an emphasis on theoretically because an attacker would need to be extremely clever (or lucky) to successfully root the server via a flaw in the request processing given the restricted POSIX capabilities. In most cases, that specific apache process would just crash and life goes on. All in all, I still consider the ITK MPM to be more secure than the standard prefork MPM because a much more likely attack on the server is a hacker breaking in through a flaw in one of the hosted website's scripts (e.g. an outdated CMS installation). In this scenario the ITK MPM protects you completely; the potential damage from the attacker is essentially limited to that user's home directory, since each user's scripts run as their own UID/GID. They do not get access to any other user's scripts, nor to any other user's emails.

Administering the server

Setting up the scripts

The following code will setup the "web_domain_add" script which can be used to create the initial website configurations for a hosting users on your server:

Adding a new web hosting account

In order to create the new website configurations, you must also create a new system user that the VirtualHost will be mapped to via ITK. See the SSH+SFTP tutorial for details on how to add a new restricted system user. Once you have done so, you can run the web_domain_add script:

This will initialize a new configuration for the domain primarydomain.tld mapped to system_username. You may optionally specify as many domain aliases as needed, for example:

www.example.com will be used as the primary domain, and the script will automatically alias example.com as well as www.example.net, example.net, www.example.org and example.org as domain aliases for www.example.com.
Note: The script does not create any DNS entries for these domains! That task must be performed separately with your DNS provider.

Resources and further reading

• The Apache Wiki page on privilege separation
• The peruser MPM homepage
• The ITK MPM homepage
• Stuart Herber's post on securing an Apache shared hosting server using the ITK MPM
• Roman Barczyński's itk vs peruser performance comparison
  
• My own quick performance comparison of prefork/suphp/ITK
• PHP manual on (the now deprecated) Safe Mode

This how-to will show you how to configure:

• A MySQL database to store information about email accounts, aliases (per-address or per-domain) and autoresponders
• Postfix as your mail transfer agent (MTA) for SMTP
  • amavisd-new with clamav & spamassassin for automatic virus and spam filtering
  • Ability to define virtual users (via MySQL database) mapped to a real UID/GID or system users
  • Aliases generated automatically from the MySQL database, allowing one address to forward to another or all addresses on a domain to forward to another
  • Response handling email autoresponders
  • Dovecot as the local delivery agent (LDA) delivering mail to the corresponding user's mailbox
  • Dovecot as SASL authenticator
• Dovecot for POP3 & IMAP
  • Dovecot LDA delivering mail as any system user (better security)
  • SASL authentication based on virtual user information stored in the MySQL database

As this may seem like a bit complex at first, below is a simple overview of how the system works once put together:

• When a user connects to the server to retrieve mail (POP3 or IMAP), Dovecot performs the SASL authentication and then connects the user to their message inbox.
• When a user sends mail to the server destined for another domain (outgoing SMTP), Postfix calls on Dovecot to perform SASL authentication. If it succeeds, relay access is granted and the message is sent to the external mail server.
• When an external user sends mail to the server destined for a domain handled by Postfix (incoming SMTP), Postfix checks the system and virtual user tables to ensure that the recipient is valid. If so, then it passes through a few verifications first (valid sender, virus filter, spam filter) and if all pass then the message is delivered to the corresponding user's inbox via Dovecot LDA.

Before starting

Please ensure that you have followed the instructions in the getting started guide here. This tutorial may require you to rebuild RPMs as well as generate SSL certificates and assumes you have the proper environment described in that guide already setup.

If you have not setup the database server yet, please follow the database how-to first.

Setting up the database

Before installing dovecot or postfix, the MySQL database which will be used to authenticate all email clients needs to be initialized. Execute this SQL snippet:

Replace random-password by a long squence of random characters. The mailconfig user will only be used by scripts, so you will not need to remember this password.

Incoming mail: Dovecot

Grab dovecot from the repositories:

Dovecot is now ready to be configured. Execute the following to configure the Dovecot to allow imap/pop3 access securely with compatibility for some older clients with broken protocol support:

Once again, substitute yourdomain.com and the SSL certification/key file locations accordingly. You'll notice we reference the non-existent file /etc/dovecot-mysql.conf in this configuration. Let's create that now so Dovecot has access to the MySQL database:

This creates dovecot's MySQL configuration, /etc/dovecot-mysql.conf and sets the correct permissions accordingly so that users can't go snooping for the MySQL credentials. Again, replace random-password with the correct password.

Because the 'deliver' (Dovecot local delivery agent) executable run by Postfix has to deliver mail to different system users, it will need a setuid bit set so that Postfix can run deliver with root privileges, have it change to the appropriate user and then deliver mail to that user's inbox. We can do this easily with two commands:

Notice that because the last permission bit is "0", nobody except root or users in the mail group will be able to execute deliver, reducing the security risk introduced by using enabling setuid bit.

Lastly, Dovecot needs to be started and exceptions need to be added to the firewall:

Outgoing mail: Postfix

Normally we could just install Postfix directly from the repository, however the stock CentOS postfix is provided without MySQL support. We can easily enable this support by rebuilding the RPM:

Then install the generated Postfix RPM (rpm -Uhv ~/rpmbuild/RPMS/[arch]/postfix-[version].[arch].rpm)

Next, we will configure Postfix to accept/relay mail:

Be sure to substitute yourdomain.com for your actual domain name as well as set the smtpd_tls_cert_file and smtpd_tls_key_file configuration keys appropriately. You can use the cert.pem and private/localhost.key files that get autogenerated for the system, however it is recommended that you generate your own self-signed certificate so that the domain name matches (or even better, purchase validated ones). You may create a self-signed certificate and private key pair by running genkey --days 365 yourdomain.com and then following the on-screen instructions.

The configuration above references several configuration files that do not exist yet, so we need to create them now. A description of what each file does is available in the comments.

Postfix now needs to be configured to add support for additional ports (26, 465, 587) as well as to add support for dovecot's local delivery agent (LDA):

Finally, add the firewall port exceptions and start postfix:

Spam and virus filtering: amavisd-new

First, install amavisd-new and some dependencies:

In the /etc/amavisd/amavisd.conf configuration file, the following items need to be changed:

• Set $max_servers to 15
• Set $mydomain to 'mail.yourdomain.com'
• Set $virus_admin, $mailfrom_notify_admin, $mailfrom_notify_recip, and $mailfrom_notify_spamadmin each to "postmaster\@$mydomain"
• Set $final_banned_destiny to D_DISCARD
• Set $final_bad_header_destiny to D_DISCARD

If you wish to send spam to a certain user (ie spam@yourdomain.com) instead of just discarding it, you may also set the $spam_quarantine_to and $bad_header_quarantine_to options.

Next, Postfix needs to be configured so that it makes use of amavisd's spam/virus filtering:

Unfortunately, the default clamd configuration for amavisd contains some errors, preventing the amavisd.clamd service from starting correctly. We need to overwrite it with this configuration:

Now it is time to enable and start the services.

Response mail autoresponder

Unfortunately, no CentOS 5 RPM package is currently available for the response daemon. I have attached the SRPM to this page which you can download and rebuild:

Note that we are also rebuilding MySQL-python26, a dependency of response. Simply install the resulting two RPMs after the builds finish.

Before configuring Postfix, we need to create the MySQL tables that will be used to keep information about the autoresponders as well as which users have already been sent an autoresponse within a 24-hour period:

Similar to before, replace random-password2 with a long and complicated password. You'll never have to remember it, it is simply used internally by the response daemon.

In the next file, you'll notice we use port 10026 because ports 10024 and 10025 have already been taken by amavisd-new.

Now we need to inform postfix about the transport map and add the bcc mapping that will forward appropriate messages to the response-lmtpd daemon:

Of course, you'll need to replace random-password2 in /etc/postfix/mysql-virtual-autoresponses.cf with the random password created earlier for the mailresponse user.

That's it! You can now add a row in the mailconfig.autoresponse_config table to have response-lmtpd automatically reply to any message sent to that user.

Resources and further reading

• Dovecot documentation wiki, especially the Postfix and Dovecot SASL and Dovecot LDA with Sendmail pages
• The entire set of Postfix documentation
• The response installation and configuration pages
• The CentOS 5 wiki amavisd howto