• 15 min read
  • This how-to will show you how to configure:

    • An Apache 2 web server using virtual hosts
    • The ITK MPM allows each virtual host to serve requests as its own user/group
    • mod_ssl to serve pages over the secure HTTP (HTTPS) protocol
    • mod_security to help prevents everything from SQL injections to data leaks
    • mod_php for PHP scripts along with mod_suhosin to help protect mitigate risks from known and unknown flaws in PHP scripts

    Rebuilding httpd for ITK

    About privilege separation

    By default, the Apache web server runs as the 'apache' user. This is good because a successful attack on the web server will only cause limited damage to the system, as they do not have root access. However on a shared servers which hosts multiple websites, an attack can still be very dangerous because the 'apache' user is used by all of the websites. Thus, an attack on one website can potentially grant access to any other websites that are being hosted on the system!

    Privilege separation is a technique that can be used to mitigate the risk of an attack against a shared hosting server. By allowing webpages to be served by different users, each host can be assigned its own user and so the damage from a site hack will be limited to just that account and not all of the accounts hosted on the system. We will be using the ITK Apache MPM to achieve this.

    Rebuild process

    Unfortunately, the ITK MPM is not included in the stock httpd distribution. Fortunately, it is relatively easy to add it. Run this as your regular user to install the httpd source RPM and download the ITK patch sources:

    yumdownloader --source httpd
    rpm -i httpd*.src.rpm
    rm httpd*.src.rpm
    cd ~/rpmbuild/SOURCES
    wget http://mpm-itk.sesse.net/apache2.2-mpm-itk-2.2.17-01/{02-rename-prefork-to-itk.patch,03-add-mpm-to-build-system.patch,04-correct-output-makefile-location.patch,05-add-copyright.patch,06-hook-just-after-merging-perdir-config.patch,07-base-functionality.patch,08-max-clients-per-vhost.patch,09-capabilities.patch,10-nice.patch,11-fix-htaccess-reads-for-persistent-connections.patch}

    With that done, let's make a few quick modifications to the RPM spec file located at ~/rpmbuild/SPECS/httpd.spec.

    We will first need to add the ITK patches. Find the last patch line in the RPM spec file, for example, Patch202: httpd-2.2.3-deflate2215.patch, then add after it:

    # ITK MPM
    Patch802: 02-rename-prefork-to-itk.patch
    Patch803: 03-add-mpm-to-build-system.patch
    Patch804: 04-correct-output-makefile-location.patch
    Patch805: 05-add-copyright.patch
    Patch806: 06-hook-just-after-merging-perdir-config.patch
    Patch807: 07-base-functionality.patch
    Patch808: 08-max-clients-per-vhost.patch
    Patch809: 09-capabilities.patch
    Patch810: 10-nice.patch

    Now in the %pre section - after the Red Hat patches are applied - add the following lines:

    # ITK MPM
    mkdir server/mpm/experimental/itk/
    cp -d --preserve=all server/mpm/prefork/* server/mpm/experimental/itk/
    mv server/mpm/experimental/itk/prefork.c server/mpm/experimental/itk/itk.c
    %patch802 -p1 -b .mpm02
    %patch803 -p1 -b .mpm03
    %patch804 -p1 -b .mpm04
    %patch805 -p1 -b .mpm05
    %patch806 -p1 -b .mpm06
    %patch807 -p1 -b .mpm07
    %patch808 -p1 -b .mpm08
    %patch809 -p1 -b .mpm09
    %patch810 -p1 -b .mpm10

    This will copy the (fully Red Hat-patched) prefork MPM code into a new folder and then apply the ITK patches.

    In the %build section of the spec file, you will see these lines:

    # For the other MPMs, just build httpd and no optional modules
    mpmbuild worker --enable-modules=none
    mpmbuild event --enable-modules=none

    Following the same format, add a line for the ITK MPM:

    mpmbuild itk --enable-modules=none

    Similarly, we find in the %install section:

    install -m 755 worker/httpd $RPM_BUILD_ROOT%{_sbindir}/httpd.worker
    install -m 755 event/httpd $RPM_BUILD_ROOT%{_sbindir}/httpd.event

    Add a line for the ITK MPM:

    install -m 755 itk/httpd $RPM_BUILD_ROOT%{_sbindir}/httpd.itk

    Last of all, in the %check section there is another section were we need to add in the ITK MPM:

    # Verify that the same modules were built into the httpd binaries
    ./prefork/httpd -l | grep -v prefork > prefork.mods
    for mpm in worker; do

    Change the middle line to read:

    for mpm in worker event itk

    We are now ready to rebuild our ITK-enabled httpd package.

    cd ~/rpmbuild/SPECS
    rpmbuild -ba httpd.spec

    Install the dependencies listed and then re-run the rpmbuild again if necessary. After the build has finished, install your new RPMs:

    rpm -Uhv ../RPMS/[arch]/{httpd,mod_ssl}-2*.rpm

    Remember to replace [arch] in the second to last command with the appropriate value (most probably i686 for 32-bit machines or x86_64 for 64-bit machines). The last step is to set the httpd worker to the ITK binary by editing /etc/sysconfig/httpd and adding after the line #HTTPD=/usr/sbin/httpd.worker:

    HTTPD=/usr/sbin/httpd.itk

    Installing add-on modules and additional software

    Let's install PHP, AWStats and a few other useful add-ons:

    yum install mod_security php php-{suhosin,mysql,mcrypt,mhash,gd} awstats

    Keep in mind that by default, mod_security's rules are very restrictive and you will almost certainly need to tweak them. Be sure to test thoroughly before enabling mod_security on your live server. You can modify the mod_security settings by editing the rule files in /etc/httpd/modsecurity.d.

    Configuring the web server

    You will find in a stock installation, no indexes are permitted at all and that visiting localhost displays the standard CentOS test page. This can be changed by editing /etc/httpd/conf.d/welcome.conf and comment out the entire LocationMatch section:

    #
    #    Options -Indexes
    #    ErrorDocument 403 /error/noindex.html
    #

    Next, let's add a small configuration file with some custom settings. Below we will try and prevent information leaks by restricting access to backup files (files ending with a "~"), .sql and .inc plaintext files which could potentially reveal critical information about how a site functions. We will also add a one-line include directive so that all files in the virtual hosts configuration directory get included as well.

    mkdir /etc/httpd/vhosts.d
    cat << EOF > /etc/httpd/conf.d/z_custom.conf
    # named with a z_ prefix to ensure this is parsed last.

    # Restrict access to sensitive file extensions that shouldn't be read via a
    # browser: *~ for temp/backup files, *.sql, *.inc as it isn't registered as
    # a php file.

      Order allow,deny
      Deny from all

    # Include our VirtualHost configurations
    Include vhosts.d/*.conf
    EOF

    The following will initialize a template VirtualHost configuration that can be used by a script to automate the installation of new VirtualHosts:

    cat << EOF > /etc/httpd/vhosts.d/sample.conf
    #
    #    ServerName \$WEBDOMAIN
    #    ServerAdmin webmaster@\$DOMAIN
    #    ServerAlias \$DOMAIN_ALIASES
    #    DocumentRoot /home/\$USER/web/public_html
    #    ErrorLog /home/\$USER/web/logs/error_log
    #    CustomLog /home/\$USER/web/logs/access_log combined
    #    php_admin_flag log_errors on
    #    php_admin_value error_log /home/\$USER/web/php_error_log
    #   
    #        php_admin_value session.save_path "/var/lib/php/session/\$USER"
    #        AssignUserId \$USER \$USER
    #   

    #   
    #        Options FollowSymLinks
    #        AllowOverride FileInfo AuthConfig Limit Indexes Options
    #   

    #

    EOF

    Similarly, we will setup a awstats template that can create configuration via a script:

    cat << EOF > /etc/awstats/awstats.model_custom.conf
    LogFile="/home/\$USER/web/logs/access_log"
    LogType=W
    LogFormat=1
    LogSeparator=" "

    SiteDomain="\$WEBDOMAIN"
    HostAliases="\$DOMAIN_ALIASES"
    DNSLookup=2
    DirData="/var/lib/awstats"
    DirCgi="/awstats"
    DirIcons="/awstatsicons"
    AllowToUpdateStatsFromBrowser=0
    AllowFullYearView=3
    EnableLockForUpdate=1
    DNSStaticCacheFile="dnscache.txt"
    DNSLastUpdateCacheFile="dnscachelastupdate.txt"
    SkipDNSLookupFor=""

    AllowAccessFromWebToAuthenticatedUsersOnly=1
    AllowAccessFromWebToFollowingAuthenticatedUsers="\$USER"
    AllowAccessFromWebToFollowingIPAddresses=""

    CreateDirDataIfNotExists=0
    BuildHistoryFormat=text
    BuildReportFormat=html
    SaveDatabaseFilesWithPermissionsForEveryone=0
    PurgeLogFile=0
    ArchiveLogRecords=0
    KeepBackupOfHistoricFiles=0
    DefaultFile="index.html"
    SkipHosts="127.0.0.1"
    SkipUserAgents=""
    SkipFiles=""
    SkipReferrersBlackList=""
    OnlyHosts=""
    OnlyUserAgents=""
    OnlyUsers=""
    OnlyFiles=""
    NotPageList="css js class gif jpg jpeg png bmp ico rss xml swf"
    ValidHTTPCodes="200 304"
    ValidSMTPCodes="1 250"
    AuthenticatedUsersNotCaseSensitive=0
    URLNotCaseSensitive=0
    URLWithAnchor=0
    URLQuerySeparators="?;"
    URLWithQuery=0
    URLWithQueryWithOnlyFollowingParameters=""
    URLWithQueryWithoutFollowingParameters=""
    URLReferrerWithQuery=0
    WarningMessages=1
    ErrorMessages=""
    DebugMessages=0
    NbOfLinesForCorruptedLog=50
    WrapperScript="/awstats"
    DecodeUA=0
    MiscTrackerUrl="/js/awstats_misc_tracker.js"
    LevelForBrowsersDetection=2
    LevelForOSDetection=2
    LevelForRefererAnalyze=2
    LevelForRobotsDetection=2
    LevelForSearchEnginesDetection=2
    LevelForKeywordsDetection=2
    LevelForFileTypesDetection=2
    LevelForWormsDetection=0
    UseFramesWhenCGI=1
    DetailedReportsOnNewWindows=1
    Expires=3600
    MaxRowsInHTMLOutput=1000
    Lang="auto"
    DirLang="./lang"
    ShowMenu=1
    ShowSummary=UVPHB
    ShowMonthStats=UVPHB
    ShowDaysOfMonthStats=VPHB
    ShowDaysOfWeekStats=PHB
    ShowHoursStats=PHB
    ShowDomainsStats=PHB
    ShowHostsStats=PHBL
    ShowAuthenticatedUsers=0
    ShowRobotsStats=HBL
    ShowWormsStats=0
    ShowEMailSenders=0
    ShowEMailReceivers=0
    ShowSessionsStats=1
    ShowPagesStats=PBEX
    ShowFileTypesStats=HB
    ShowFileSizesStats=0
    ShowOSStats=1
    ShowBrowsersStats=1
    ShowScreenSizeStats=0
    ShowOriginStats=PH
    ShowKeyphrasesStats=1
    ShowKeywordsStats=1
    ShowMiscStats=a
    ShowHTTPErrorsStats=1
    ShowSMTPErrorsStats=0
    ShowClusterStats=0
    AddDataArrayMonthStats=1
    AddDataArrayShowDaysOfMonthStats=1
    AddDataArrayShowDaysOfWeekStats=1
    AddDataArrayShowHoursStats=1
    IncludeInternalLinksInOriginSection=0
    MaxNbOfDomain = 10
    MinHitDomain  = 1
    MaxNbOfHostsShown = 10
    MinHitHost    = 1
    MaxNbOfLoginShown = 10
    MinHitLogin   = 1
    MaxNbOfRobotShown = 10
    MinHitRobot   = 1
    MaxNbOfPageShown = 10
    MinHitFile    = 1
    MaxNbOfOsShown = 10
    MinHitOs      = 1
    MaxNbOfBrowsersShown = 10
    MinHitBrowser = 1
    MaxNbOfScreenSizesShown = 5
    MinHitScreenSize = 1
    MaxNbOfWindowSizesShown = 5
    MinHitWindowSize = 1
    MaxNbOfRefererShown = 10
    MinHitRefer   = 1
    MaxNbOfKeyphrasesShown = 10
    MinHitKeyphrase = 1
    MaxNbOfKeywordsShown = 10
    MinHitKeyword = 1
    MaxNbOfEMailsShown = 20
    MinHitEMail   = 1
    FirstDayOfWeek=1
    ShowFlagLinks=""
    ShowLinksOnUrl=1
    UseHTTPSLinkForUrl=""
    MaxLengthOfShownURL=64
    HTMLHeadSection=""
    HTMLEndSection=""
    Logo="awstats_logo6.png"
    LogoLink="http://awstats.sourceforge.net"
    BarWidth   = 260
    BarHeight  = 90
    StyleSheet=""
    color_Background="FFFFFF" # Background color for main page (Default = "FFFFFF")
    color_TableBGTitle="CCCCDD" # Background color for table title (Default = "CCCCDD")
    color_TableTitle="000000" # Table title font color (Default = "000000")
    color_TableBG="CCCCDD" # Background color for table (Default = "CCCCDD")
    color_TableRowTitle="FFFFFF" # Table row title font color (Default = "FFFFFF")
    color_TableBGRowTitle="ECECEC" # Background color for row title (Default = "ECECEC")
    color_TableBorder="ECECEC" # Table border color (Default = "ECECEC")
    color_text="000000" # Color of text (Default = "000000")
    color_textpercent="606060" # Color of text for percent values (Default = "606060")
    color_titletext="000000" # Color of text title within colored Title Rows (Default = "000000")
    color_weekend="EAEAEA" # Color for week-end days (Default = "EAEAEA")
    color_link="0011BB" # Color of HTML links (Default = "0011BB")
    color_hover="605040" # Color of HTML on-mouseover links (Default = "605040")
    color_u="FFAA66" # Background color for number of unique visitors (Default = "FFAA66")
    color_v="F4F090" # Background color for number of visites (Default = "F4F090")
    color_p="4477DD" # Background color for number of pages (Default = "4477DD")
    color_h="66DDEE" # Background color for number of hits (Default = "66DDEE")
    color_k="2EA495" # Background color for number of bytes (Default = "2EA495")
    color_s="8888DD" # Background color for number of search (Default = "8888DD")
    color_e="CEC2E8" # Background color for number of entry pages (Default = "CEC2E8")
    color_x="C1B2E2" # Background color for number of exit pages (Default = "C1B2E2")
    LoadPlugin="geoip GEOIP_STANDARD /var/lib/GeoIP/GeoIP.dat"
    ExtraTrackedRowsLimit=500
    EOF

    (For those wondering, this template was derived by removing all comments from the stock awstats model configuration at /etc/awstats/awstats.model.conf and then substituting important values such as SiteDomain with variables so that can be easily changed via sed)

    Because the web server will be using the ITK MPM, each site's requests will be run under its respective owner and group. This causes problems with the standard PHP session setup, which uses one directory owned by "apache" for all session information. To solve this, a session directory will need to be created for each user. Let's create the default one for the apache user now:

    chown root.apache /var/lib/php/session
    chmod 0771 /var/lib/php/session
    mkdir -m 0770 /var/lib/php/session/apache
    chown root.apache /var/lib/php/session/apache

    You are now ready to add VirtualHosts for your domains. See Adding a new web hosting account section of Administering the server below for more information on how to do so.

    The last step is to add the firewall port exceptions for http/https and start Apache:

    chkconfig httpd on
    service httpd start
    iptables -I RH-Firewall-1-INPUT 4 -m state --state NEW -m tcp -p tcp --dport 80 -j ACCEPT
    iptables -I RH-Firewall-1-INPUT 4 -m state --state NEW -m tcp -p tcp --dport 443 -j ACCEPT
    service iptables save

    Suhosin PHP extension

    We will now be configuring /etc/php.d/suhosin.ini. Because suhosin can interfere with some site's functionality, it is best to enable it in simulation mode so that errors are logged and you can configure suhosin accordingly:

    suhosin.simulation = On

    Limit the maximum PHP memory_limit that scripts can set:

    suhosin.memory_limit = 512M

    Lastly, disable session encryption (breaks roundcubemail if enabled):

    suhosin.session.encrypt = Off

    Optional add-on: roundcubemail

    Let's rebuild roundcubemail 0.3 for CentOS 5:

    su - myusername
    cd ~/rebuilds
    fedpkg clone -a roundcubemail
    cd roundcubemail
    fedpkg switch-branch el6

    Edit the spec file roundcubemail.specand comment the line Requires: php-pear-Mail-mimeDecode so that it reads #Requires: php-pear-Mail-mimeDecode. This is the only change required for CentOS 5, so let's build the package now:

    fedpkg local
    exit
    yum install --nogpgcheck /home/myusername/rebuilds/roundcubemail/noarch/roundcubemail*.rpm

    Now it is time to configure roundcubemail. In the file /etc/httpd/conf.d/roundcubemail.conf, add after the line :

           
                AssignUserId apache apache
           

    As well, by default roundcubemail denies access to any non-localhost clients. To change this, comment out Deny from all and add Allow from all:

            Order Deny,Allow
            #Deny from all
            Allow from 127.0.0.1
            Allow from all

    We will also need to create the roundcubemail database. If you have not setup your database server, follow the database tutorial now.

    CREATE DATABASE roundcubemail;
    USE roundcubemail;
    source /usr/share/doc/roundcubemail-0.3.1/SQL/mysql.initial.sql
    GRANT ALL PRIVILEGES ON roundcubemail.* TO 'roundcubemail'@'localhost' IDENTIFIED BY 'random-password';
    FLUSH PRIVILEGES;

    Then configure /etc/roundcubemail accordingly. You should now be able to access the webmail portal at http://www.yourdomain.com/roundcubemail.

    But I want PHP 5.2!

    PHP 5.2 is available in the CentOS 5 Testing repository. Certain PHP add-ons, such as php-suhosin, have not been rebuilt so you will need to do that manually.

    rpm -e php-suhosin
    yum --enablerepo=c5-testing update php\*

    Now rebuild and reinstall php-suhosin:

    yumdownloader --source php-suhosin
    rpmbuild --rebuild php-suhosin-*.src.rpm

    Now install the resulting php-suhosin binary RPM and then move the configuration file (/etc/php.d/suhosin.ini.rpmsave) back into place.

    But I want PHP 5.3!

    PHP 5.3 is available as of CentOS 5.6 the php53 package. Certain PHP add-ons, such as php-suhosin and php-mcrypt have not been rebuilt so you will need to do that manually. This is a bit tricky as you will need to rebuild the php-extras SRPM among others; I'll leave you to figure out the details. See the instructions above for PHP 5.2 for a rough workflow using php-suhosin as an example.

    Some final words

    Two 3rd party MPM modules satisfied my need privilege separation in this setup, the ITK MPM and the Peruser MPM. Each has its own advantages and disadvantages.

    ITK's approach is to accept a request, determine which virtual host the request belongs to, fork, set the UID and GID of the fork, serve the request and then to terminate the fork after the request has been completed allowing it to fork again later. Because ITK is very simple in nature and functions similarly to the traditional prefork MPM, it is more compatible with modules like mod_ssl, mod_php and mod_python. That said, it is also slower than because of the fork/kill overhead associated with each request.

    Peruser, on the other hand, starts a pool of processes that fork only on startup. There are multiplexers than accept requests, determine which virtual host the request belongs to, and then forwards the request to the appropriate process from the pool. This approach incurs practically no overhead as the process from the pool has already been forked and had its UID/GID changed. Peruser is therefore much faster than ITK; its performance is nearly almost on par with that of the traditional prefork MPM! The downfall of Peruser is that because of this connection passing from the multiplexer to a preforked process, it is more complex and has had a long history of bugs and incompatibilities. Experimental support for mod_ssl was only recently added, and upstream development still seems to have a number of bugs to work out. In addition, because Peruser requires at least one process per virtual host for the pool and then requires multiplexers to handle incoming requests, it is much more resource intensive that the ITK MPM, where a process can dynamically handle a request from any virtual host. For these reasons, I favoured ITK over Peruser.

    Keep in mind that ITK does have one security flaw. The request headers must be processed in order to determine which virtual host that request belongs to, and therefore which UID/GID ITK would need to switch to. Because of this, all of the header processing is performed as root (albeit with restricted POSIX capabilities). Should someone be able to take advantage of a flaw in Apache or another module (e.g. mod_ssl), the server could theoretically be rooted.

    There is an emphasis on theoretically because an attacker would need to be extremely clever (or lucky) to successfully root the server via a flaw in the request processing given the restricted POSIX capabilities. In most cases, that specific apache process would just crash and life goes on. All in all, I still consider the ITK MPM to be more secure than the standard prefork MPM because a much more likely attack on the server is a hacker breaking in through a flaw in one of the hosted website's scripts (e.g. an outdated CMS installation). In this scenario the ITK MPM protects you completely; the potential damage from the attacker is essentially limited to that user's home directory, since each user's scripts run as their own UID/GID. They do not get access to any other user's scripts, nor to any other user's emails.

    Administering the server

    Setting up the scripts

    The following code will setup the "web_domain_add" script which can be used to create the initial website configurations for a hosting users on your server:

    cat << EOF > /root/bin/web_domain_add
    #!/bin/sh
    username=\$1
    shift
    domain=\$1
    shift

    if [ -z \$domain ] || [ "\$username" == "-h" ];then
      echo "Usage: \$1 user domain [alias1] [alias2] [...]"
      exit 1
    fi

    aliases="\$domain"
    for alias in "\$@";do
      aliases="\$aliases \$alias www.\$alias "
    done

    cat /etc/httpd/vhosts.d/sample.conf | \
      sed "s/\\\$WEBDOMAIN/www.\${domain}/g" | \
      sed "s/\\\$DOMAIN_ALIASES/\${aliases}/g" | \
      sed "s/\\\$USER/\${username}/g" | \
      sed "s/\\\$DOMAIN/\${domain}/g" | \
      sed "s/^#//g" \
      > "/etc/httpd/vhosts.d/\${username}.conf"
    echo "*** Writing VHost configuration /etc/httpd/vhosts.d/\${username}.conf"

    cat /etc/awstats/awstats.model_custom.conf | \
      sed "s/\\\$WEBDOMAIN/www.\${domain}/g" | \
      sed "s/\\\$DOMAIN_ALIASES/\${aliases}/g" | \
      sed "s/\\\$USER/\${username}/g" | \
      sed "s/\\\$DOMAIN/\${domain}/g" \
      > /etc/awstats/awstats.www.\${domain}.conf
    echo "*** Writing AWStats configuration /etc/awstats/awstats.www.\${domain}.conf"

    echo "*** Creating password for AWStats"
    htpasswd /home/awstats-htpasswd "\$username"

    echo "*** Done"
    EOF
    chmod +x /root/bin/web_domain_add

    Adding a new web hosting account

    In order to create the new website configurations, you must also create a new system user that the VirtualHost will be mapped to via ITK. See the SSH+SFTP tutorial for details on how to add a new restricted system user. Once you have done so, you can run the web_domain_add script:

    /root/bin/web_domain_add system_username primarydomain.tld

    This will initialize a new configuration for the domain primarydomain.tld mapped to system_username. You may optionally specify as many domain aliases as needed, for example:

    /root/bin/web_domain_add exampleuser example.com example.net example.org

    www.example.com will be used as the primary domain, and the script will automatically alias example.com as well as www.example.net, example.net, www.example.org and example.org as domain aliases for www.example.com.
    Note: The script does not create any DNS entries for these domains! That task must be performed separately with your DNS provider.

    Resources and further reading