• 2 min read
  • This how-to will show you how to configure:

    • MySQL as a secure and performant database server
    • Create new databases
    • Additional MySQL users with restricted privileges

    Installing MySQL

    yum install mysql mysql-server
    chkconfig mysqld on
    service mysqld start
    iptables -I RH-Firewall-1-INPUT 4 -m state --state NEW -m tcp -p tcp --dport 3306 -j ACCEPT
    service iptables save

    Adding a new database+user

    Execute in MySQL:

    CREATE DATABASE `username_purpose`
    GRANT ALL ON `username_purpose` TO 'username'@'localhost' IDENTIFIED BY 'new_password'

    The new user will only have privileges on the newly created database.

    You can choose anything you'd like for username_purpose, username and new_password but to make maintenance easier, name your database users after the corresponding system user for that site. Database names have a relatively small maximum length, so often you can use a shorted version of the system username followed by the purpose of that database. For example, with a system user of foobar_baz then you could create the database foobar_websites to host the tables of user foobar_baz's CMS. At a glance, it's clear who's database it is and what it is used for.

    As well, remember that most of your users will simply need to enter the password once at installation time of the CMS or other software. It's in your best interest to set a long, random alphanumerical password.

  • 9 min read
  • This how-to will show you how to configure:

    • Remote access over SSH via OpenSSH
      • Secure, password-less authentication
      • Optional: OpenSSH 5.4p1 to allow restrict shell access and jail users by group
    • Secure file transfers over SFTP

    Configuring OpenSSH

    openssh-server is already installed by default, it just needs to be configured. We will disable root logins as well as all password-based logins in favour of the more secure public key authentication. If you do not already have a SSH key, you should take the time to create one now by running ssh-keygen on the computer you will be using to access the server remotely.

    The following will configure SSH as described above:

    cat << EOF >> /etc/ssh/sshd_config
    ## Customizations ##
    # Some of the settings here duplicate defaults, however this is to ensure that
    # if for some reason the defaults change in the future, your server's
    # configuration will not be affected.

    # Do not allow root to login over SSH. If you need to become root, login as your
    # regular use and use su - instead.
    PermitRootLogin no

    # Disable password authentication and enable key authentication. This will
    # force users to use key-based authentication which is more secure and will
    # protect against some automated brute force attacks on the server. As well,
    # this section disables some unneeded authentication types. If you wish to use
    # them, modify this section accordingly.
    PasswordAuthentication no
    PubkeyAuthentication yes
    ChallengeResponseAuthentication no
    KerberosAuthentication no

    # Do not allow TCP or X11 forwarding by default.
    AllowTcpForwarding no
    X11Forwarding no

    # Why give such a large window? If the user has not provided credentials in 30
    # seconds, disconnect the user.
    LoginGraceTime 30s

    Let's make sure SSH starts on boot, restart the service immediately and finally add the firewall exception for port 22:

    chkconfig sshd on
    service sshd restart
    iptables -I RH-Firewall-1-INPUT 4 -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT
    service iptables save

    Because we have disabled root access over SSH, it is time to create a regular user that you can used to login over ssh and then gain root access:

    useradd myusername
    passwd myusername
    su - myusername
    mkdir -m 0700 .ssh
    touch .ssh/authorized_keys
    chmod 600 .ssh/authorized_keys
    restorecon -v -r /home/myusername

    Now add the contents of your ~/.ssh/id_rsa.pub file to .ssh/authorized_keys on the server.

    Optional (but recommended): Rebuilding OpenSSH 5.x

    Although SSH will function perfectly fine with this bare configuration, it is not the most secure possible. CentOS 5 comes with OpenSSH version 4.3p2 which is rather outdated. Instead of using 4.3p2, OpenSSH version 5.4p1 (from Fedora 13) can be rebuilt which offers a slew of new features such as access control via user/group matching and SFTP jailrooting.

    yum install fedora-packager
    su - myusername
    cd ~/rebuilds
    fedpkg clone -a openssh
    cd openssh
    fedpkg switch-branch f13

    Before the package can be rebuilt, a few changes need to be made to make it work on CentOS 5. Edit openssh/F-13/openssl.spec and find the line BuildRequires: tcp_wrappers-devel at approximately line number 142. Simply remove the -devel so that the line now reads BuildRequires: tcp_wrappers. Just below, you will also notice a statement BuildRequires: openssl-devel >= 0.9.8j.
    Remove the version requirement so that the line reads BuildRequires: openssl-devel. Lastly, near line 178 find the line Requires: pam >= 1.0.1-3 and once again, remove the version requirement so that the line reads Requires: pam.

    Now that the RPM spec file has been modified, we also need to change the PAM configuration file as the one from Fedora 13 uses some modules that are not present in CentOS 5:

    cat << EOF > sshd.pam
    auth       include      system-auth
    account    required     pam_nologin.so
    account    include      system-auth
    password   include      system-auth
    # pam_selinux.so close should be the first session rule
    session    required     pam_selinux.so close
    session    required     pam_loginuid.so
    # pam_selinux.so open should only be followed by sessions to be executed in the user context
    session    required     pam_selinux.so open env_params
    session    optional     pam_keyinit.so force revoke
    session    include      system-auth

    The package is ready to be rebuild for CentOS 5. Execute the following to rebuild and install OpenSSH 5.4p1:

    yum install gtk2-devel libX11-devel autoconf automake zlib-devel audit-libs-devel pam-devel fipscheck-devel openssl-devel krb5-devel libedit-devel ncurses-devel libselinux-devel xauth
    fedpkg local
    rpm -Uhv /home/myusername/rebuilds/openssh/[arch]/openssh-{5,server,clients}*.rpm
    rm -f /etc/ssh/sshd_config.rpmnew

    Remember to replace [arch] in the second to last command with the appropriate value (most probably i686 for 32-bit machines or x86_64 for 64-bit machines). We can take now advantage of the new features to harden SSH! The configuration segment below will restrict access for members of the serv_sftponly group such that only SFTP access is permitted and those users are jailed to the "web" folder in their home directory (so that they can only upload/download files from their website). Members of the serv_sshall group have full SSH and SFTP access, as well as X11 and TCP forwarding.

    mkdir /srv/sftp
    groupadd serv_sftponly
    groupadd serv_sshall
    usermod -a -G serv_sshall myusername
    sed -i'' 's|Subsystem\tsftp\t/usr/libexec/openssh/sftp-server|#Subsystem\tsftp\t/usr/libexec/openssh/sftp-server|' /etc/ssh/sshd_config
    cat << EOF >> /etc/ssh/sshd_config
    ## Access control ##
    # We need to use the internal sftp subsystem
    Subsystem       sftp    internal-sftp
    # Allow access if user is in these groups
    AllowGroups serv_sftponly serv_sshall

    # We can't use a path relative to ~ (or %h) because we make the user homes
    # /public_html in order to get the chroot above working properly. As a result,
    # we need to set an absolute path that will make SSH look in the usual place
    # for authorized keys.
    AuthorizedKeysFile /home/%u/.ssh/authorized_keys

    # Give tunnelling + X11 access to users who are members of group "serv_sshall"
    Match group serv_sshall
        X11Forwarding yes
        AllowTcpForwarding yes

    # Restrict users who are members of group "serv_sftponly"
    Match group serv_sftponly
        # Some settings here may duplicate the global settings, just to be safe.
        #PasswordAuthentication yes
        X11Forwarding no
        AllowTcpForwarding no
        # Force the internal SFTP subsystem and jailroot the user in their home.
        # %u gets substituted with the user name, %h with home
        ForceCommand internal-sftp
        ChrootDirectory /srv/sftp/%u
    service sshd restart

    The /srv/sftp/username folder is used instead of the user's entire home because it prevents the user from making any potentially unwanted configuration changes (such as authorizing additional ssh public keys) as well as accidentally deleting files, such as the mail folder which holds all of that domain's emails. One now simply needs to link /srv/sftp/username to the appropriate web folder to jail the user there. For example:

    ln -s ../../home/username/web /srv/sftp/username

    You do not need to do this manually, as the user setup script will run this for you.

    As well, note that the configuration includes the commented line #PasswordAuthentication Yes in the serv_sftponly MatchGroup section. If you so wish, you can uncomment this line to have password authentication enabled ONLY for users of the serv_sftponly group. While password authentication is less secure than public key authentication, it is much more convenient for your clients if you are building a shared hosting machine and if a hacker does gain access because a user had an easy to guess password, they only gain access to a single jailed SFTP client.


    You may be wondering why I haven't included any information about software that can block repeated SSH intrusions such as denyhosts... I have placed this information, along with other server security tips, in the security tutorial (coming soon). Please see it for more information.

    Administering the server

    Setting up the scripts

    The following code will setup the "hosting_user_add" script which can be used to add new hosting users on your server:

    mkdir -p /root/bin
    cat << EOF > /root/bin/hosting_user_add
    # "chown root.root"s are implied, but kept to be safe

    if [ -z \$1 ];then
      echo "Usage: \$1 user1 [user2]"
      exit 1

    for username in "\$@";do
      read -p "Restrict \$username (make member of serv_sftponly)? [Y/n] " -t 60 -n 1 response
      if [ "\$response" == "n" ] || [ "\$response" == "N" ];then
        echo "*** Creating normal user \$username"
        useradd -G serv_sshall \$username
        echo "*** Creating restricted user \$username"
        useradd -G serv_sftponly -s /sbin/nologin \$username
      chown \$username.apache /home/\$username
      chmod 710 /home/\$username
      # Set password
      passwd \$username
      # Initialize mail storage folder
      mkdir -m 0700 /home/\$username/mail
      chown \$username.\$username /home/\$username/mail
      # Initialize web folders
      mkdir -p -m 0755 /home/\$username/web
      chown root.root /home/\$username/web
      # Web: logs
      mkdir -p -m 0750 /home/\$username/web/logs
      chown root.\$username /home/\$username/web/logs
      # Web: offline/private storage
      mkdir -p -m 0755 /home/\$username/web/storage
      chown \$username.\$username /home/\$username/web/storage
      # Web: docroot
      mkdir -m 0755 /home/\$username/web/public_html
      ln -s public_html /home/\$username/web/www
      # make it so they can't remove the symlink
      chown -h root.root /home/\$username/web/www
      chown \$username.\$username /home/\$username/web/public_html
      # Web: PHP error log
      touch /home/\$username/web/php_error_log
      chown \$username.\$username /home/\$username/web/php_error_log
      chattr +u /home/\$username/web/php_error_log

      # Initialize session folder
      mkdir -m 0770 /var/lib/php/session/\$username
      chown root.\$username /var/lib/php/session/\$username

      # SSH: SFTP login
      ln -s ../../home/\$username/web /srv/sftp/\$username

      # SSH: Authorized keys dir
      mkdir -m 0700 /home/\$username/.ssh
      chown \$username.\$username /home/\$username/.ssh
      # Key description here
      echo "your_key_here" >> /home/\$username/.ssh/authorized_keys

      chmod 600 /home/\$username/.ssh/authorized_keys
      chown \$username.\$username /home/\$username/.ssh/authorized_keys
      restorecon -v -r /home/\$username
    chmod +x /root/bin/hosting_user_add

    You will need to edit /root/bin/hosting_user_add later and replace your_key_here with your own SSH key so that you can login to the account should you ever need to test or do administration work.

    Adding a new system account

    /root/bin/hosting_user_add new_username

    If you are adding many accounts, you can optionally specify more than one username to have each account be created at once. For each user specified, you will be prompted if for both their password and restricted status. Passwords can and should be set randomly because with key-based authentication, they should never have to enter it anyways.

    Resources and further reading

  • 2 min read
  • After much research, experimentation, testing and tweaking I'm happy to announce that I have completed the first part of my CentOS 5 server setup howto series!

    As of today, you'll notice a new CentOS 5 Howtos link on the where I have listed the first two parts of the howto series, the getting started howto which will help you setup a basic system environment and more importantly, the mail server howto which documents how to setup a secure mail server offering POP3/IMAP/SMTP with virtual users stored in a MySQL database.

    I'm very happy with this setup because it uses virtual users that cam be mapped to system users and also keeps the software set relatively small; Dovecot is used for SASL authentication (both for POP3/IMAP and SMTP) and for postfix's local delivery agent, so with only 2 servers we've got it all covered (of course technically it's 3 servers with an extra transport if you take amavisd and response-lmtpd into account).

    The virtual user database is currently only used in this tutorial for the mail server, but I have plans to introduce (with an upgrade path) a new database structure that will unify several authentication data pools and make managing clients for a shared hosting server easier... But I'll talk more about that later once I've finished posting my other guides. I plan on adding ones for other services such as DNS & Web, although I cannot promise when those will be finished. The mail server tutorial alone is 16 printed pages (!) so it does take me quite some time to ensure that the tutorial is well documented and that the configurations listed work properly.

    I still have to add some notes here and there about the implementation, but the core material is there. Enjoy!

  • 10 min read
  • Preface

    As of writing, the most recent version of CentOS available is 5.6 so I will be using it as the basis for this howto. If a newer version is available, I recommend you use that version instead. Much of these instructions should still apply, especially if it is only a newer 5.x release.

    This series of tutorials will help you set up a shared hosting server using hardened CentOS 5. The server will:

    • Run hardened CentOS 5
    • Use the Apache HTTPD daemon to serve webpages (optionally with SSL) using the ITK MPM
      • Have PHP 5 installed with the gd, mhash, mcrypt and suhosin extensions
      • Use mod_evasive installed to avoid DDoS attacks and mod_security to prevent hacks
    • Use dovecot to serve email to clients over IMAP and POP (both with SSL)
      • Dovecot looks up virtual users in the MySQL database and performs the SASL authentication
      • Dovecot is responsible for delivering all mail to each user's inbox
      • Each virtual user can be mapped to a system user and group
      • Roundcubemail configured for easy web access on all email accounts
    • Use Postfix MTA to accepts mail on ports 25 and 26 (for ISPs who block port 25) as well as 465 and 587
      • Relay access is denied without prior SASL authentication (which is handled by Dovecot)
      • User administration is easy; a change in the MySQL database will affect Dovecot and Postfix simultaneously
      • All incoming mail is scanned for spam (SpamAssassin) and viruses (clamd) with amavisd-new
      • Mail is rejected with an error instead of mail notices to prevent backscatter spam
    • Allow you to host multiple websites easily
      • Email addresses & passwords are stored in a MySQL database
      • Email aliases and domain-wide aliases (forward each user and alias in one domain to another) are supported
      • Each website gets its own system user making permission control easier and limiting damage in the event of a break-in or site hack
      • Flexible SSH configuration: each system user can independently be granted full SSH addess or be jailed to their ~/web folder

    Before starting

    When writing this tutorial, I assumed that you:

    • Have installed a Red Hat-based Linux distribution (RHEL, Fedora, CentOS, etc) before
    • Have a basic knowledge of how to use the command line (how to create/edit files, executing commands)
    • Have an extensive knowledge of the various security problems you will need to deal with while adminstering the server, including:
      • Rootkits and malware
      • Intrusion detection & prevention
      • Denial of service (DoS/DDoS) attacks
      • Spam prevention and backscatter spam
    • Are familiar with editing configuration files
    • Know how to add/remove software on the system using Yum
    • Are familiar with the basics of RPM rebuilding

    Legal Disclaimer

    Please note that this guide is provided on an informal, as-is basis. Although I have done my best to research, test and secure this setup, I am not responsible for any damanges you suffer as a result of using this configuration. Remember that you need to thoroughly test any setup before implementing it on a live server!

    Installing CentOS 5

    Visit the CentOS project website and download the latest 5.x netinstall ISO image from a nearby mirror. You will be installing the bare minimum set of packages, so there is no need to spend hours downloading all 7 CDs or the large DVD - you will only need the netinstall image. However, because we will be installing from the network, please be sure to mark down a mirror URL that you wish to use for the installation from the list of CentOS mirrors. You will need to enter this mirror's hostname and the path to the CentOS RPMs later on in the installation process.

    Once you have started the installation process, choose the HTTP install method and enter the URL and path that you noted earlier. In the example below, I am using mirror.iweb.ca.
    HTTP installation method
    For more information on this process, see section 12.11 of the Red Hat Enterprise Linux installation guide, Installing via HTTP.

    When promted for choosing a disk partition layout, choose the option that best suits your needs from the drop-down menu. However, ensure that Review and modify partitioning layout is selected before clicking Next.
    Partitioning setup
    Although not strictly nescessary, it is also wise to create a separate /home partition for your user's data. Keep in mind when allocating space to partitions that by the time you have finished configuring the server, all of the user data except MySQL databases will be stored on the /home partition (MySQL databases are stored under /var/lib/mysql instead).

    After selecting the root password, you will find yourself at the package group selection step. Unselect all groups, click Customize now at the bottom of the screen and then click Next.
    Customizing the initial software to be installed
    Browse in every section and unselect all groups, including the Base group. A server is only as secure as its weakest component, and so there is no sense in running or even installing any unnessesary software that can provide hackers with potential attack vectors. By installing a smaller software set, your server will be more secure.
    Deselecting all package groups
    Now that you've unselected all package groups, hit Next again to start the installation process. After a reboot, you will be running a bare-minimum CentOS 5 system!

    Repository setup

    The first step after installation is to install a few extra repositories. These will be used to install additional packages that are not included by default in the official CentOS repositories.


    EPEL (Enterprise Packages for Enterprise Linux) is a community-run repository as a part of the Fedora Project that contains many additional packages that we will find useful during the server configuration.

    rpm -Uhv http://download.fedoraproject.org/pub/epel/5/i386/epel-release-5-4.noarch.rpm

    Note: this command works at the time of writing. However, in the future the release of the epel-release RPM may change. If this command fails, please see the FedoraProject EPEL page for the most up-to-date instructions.

    CentOS test repositories

    The CentOS test repository contains packages that are on their way to CentOS Plus and/or CentOS Extras. Currently, it holds PHP 5.2 which we will use in the web server tutorial and it may contain other package upgrades that you find useful.

    wget http://dev.centos.org/centos/5/CentOS-Testing.repo -O /etc/yum.repos.d/CentOS-Testing.repo

    CentOS source repositories

    Over the course of the server setup howto series, you may need to (re)build a few RPMs. Because of this, we will need to configure the CentOS source repositories so that the original CentOS SRPM packages can be downloaded and rebuilt.

    cat << EOF > /etc/yum.repos.d/CentOS-Source.repo
    # See http://bugs.centos.org/view.php?id=1646

    name=CentOS-\$releasever - Base SRPMS

    #released updates
    name=CentOS-\$releasever - Updates SRPMS

    #additional packages that may be useful
    name=CentOS-\$releasever - Extras SRPMS

    #additional packages that extend functionality of existing packages
    name=CentOS-\$releasever - Plus SRPMS

    64-bit users only

    If you are running a x86_64 system, yum will install the x86 (32-bit) alongside the x86_64 version of any package you install. This is rather pointless, so we can start by blacklisting all 32-bit packages. To do so, add the following line to the [main] section of /etc/yum.conf:


    Then we can proceed to remove the x86 packages that are preinstalled by the CentOS installation process.

    yum remove $(rpm -qa --qf '%{NAME}.%{ARCH}\n' | grep i[3,5,6]86 | awk '{printf $1" "}')

    WARNING: DO NOT run this if you are running a 32-bit CentOS 5 system (for example i686). Accepting the removal confirmation on a x86 machine will essentially remove your entire system.

    Additional packages

    The following list of packages are either used by the tutorials or are good general-purpose/troubleshooting utilities to have:

    yum install wget yum-utils {un,}zip curl bzip2 which vim-enhanced \
        vixie-cron rng-utils quota patch nano man logwatch jwhois acpid \
        crypto-utils fedora-packager bind-utils jwhois nmap telnet traceroute \

    Also, let's replace the rather old syslog daemon with the much more modern syslog-ng:

    yum install syslog-ng
    /sbin/chkconfig syslog off
    /sbin/service syslog stop
    /sbin/chkconfig syslog-ng on
    /sbin/service syslog-ng start

    Basic firewall & SELinux configuration

    Because all groups were deselected during installation, iptables has not been installed yet. Let's install it now:

    yum install iptables iptables-ipv6 system-config-securitylevel-tui

    Next, the system security configuration utility can establish some basic firewall rules and put SELinux into permissive mode so that you may test your server setup and then configure SELinux based on the denials logged (more on that in the security & reliability guide).


    Press the Tab key twice and select Permissive from the menu, then Tab twice again to select Customize.

    Because we only want to establish the base firewall rules, do not select any exceptions (they will be added later) and just press Tab until you can choose OK. Finally, select OK again to close the configuration utility and save the system security settings. Your /etc/sysconfig/iptables file should now look something like this:

    :INPUT ACCEPT [0:0]
    :OUTPUT ACCEPT [0:0]
    :RH-Firewall-1-INPUT - [0:0]
    -A INPUT -j RH-Firewall-1-INPUT
    -A FORWARD -j RH-Firewall-1-INPUT
    -A RH-Firewall-1-INPUT -i lo -j ACCEPT
    -A RH-Firewall-1-INPUT -p icmp --icmp-type any -j ACCEPT
    -A RH-Firewall-1-INPUT -p 50 -j ACCEPT
    -A RH-Firewall-1-INPUT -p 51 -j ACCEPT
    -A RH-Firewall-1-INPUT -p udp --dport 5353 -d -j ACCEPT
    -A RH-Firewall-1-INPUT -p udp -m udp --dport 631 -j ACCEPT
    -A RH-Firewall-1-INPUT -p tcp -m tcp --dport 631 -j ACCEPT
    -A RH-Firewall-1-INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
    -A RH-Firewall-1-INPUT -j REJECT --reject-with icmp-host-prohibited

    If you wish, you may remove the rules for corresponding to ICMP (ping among others), ports 50, 51 (VPN), 631 (shared printing via CUPS) and 5353 (Multicast DNS).

    RPM building environment

    Note that you may wish to rebuild the package in another virtual machine/test CentOS environment, as the build process will require the installation of many additional developer packages that are not required on a server machine.

    Building RPMs as root is highly discouraged, as a simple typo in the spec file could damage your system. Thus if you do not already have a normal user account created on the server you should take the time to create one now:

    useradd youruser
    passwd youruser

    Next, switch to your regular user and setup the RPM building environment:

    su - youruser
    mkdir rebuilds

    System Updates

    Before configuring any services, make sure the system is up to date:

    yum update

    Now, let's move on to the more interesting part, installing and configuring the services!

  • 2 min read
  • I have been working with a client to setup an Ubercart store customized to their needs and one of the things we came across as we launched the store is that because of the nature of the items being sold, it was very difficult to give a accurate shipping estimate. Thus, I set off to find a way to enable customer payments after checkout and to enter the shipping quotes manually.

    These two modules, uc_manual_shipping and uc_payafter, are the fruit of these efforts. I hope you find them useful!

    Note that this is my first release of these modules and the code should be considered a beta and work in progress. They are untested so far and not recommended for use on a production site just yet. All information about installation, configuration and further development is available in the README.txt file, DEVELOPERS.txt file and/or source code comments. If you have made some changes or improvements, please let me know in the comments! I would love to hear about your changes and I would be more than happy to apply any patches or bugfixes.

    uc_manual_shipping enables the store administrators to manually enter shipping quotes on orders after a user has passed through checkout. It can be used in combination with uc_payafter to have users create go through regular checkout without paying, and then pay later once a shipping quote has been submitted by a store administrator.
    Download uc_manual_shipping-6.x-1.0.tar.gz

    uc_payafter duplicates the checkout process and allows users to perform payments on their orders after checkout at the URL cart/checkout/pay/$ORDER_ID. Store administrators can email users different invoice templates after payment.
    Download uc_payafter-6.x-1.1.tar.gz

    Update 2011-06-08: A user in the comments, Moises, has pointed out that in the 6.x-1.0 release of uc_payafter there was a typo that would prevent users from being able to select the request shipping invoices in the conditional actions configuration. I have updated uc_payafter below to fix this bug.

    Update 2011-11-08: uc_payafter has a new home! I have created a Drupal Sandbox project for the code here. Once I have the chance to work on the code a bit more, I will promote it to full project status and update the download links in this post.