• 13 min read
  • This how-to will not configure any one service in particular, but rather focus on the operating system as a whole in order to improve security and reliability. This how-to
    will show you how to:

    • Configure the GRUB bootloader to gracefully handle kernel panics during boots
    • Create, edit & manage custom SELinux policy modules
    • Deny access to remote users with too many failed authentication attempts over SSH (Denyhosts), POP3 or IMAP (Fail2ban)
    • Improve the password hash strength
    • Enable shell timeouts
    • ...and much more!

    Gracefully recovering from kernel panics: boot robustness & more

    GRUB has two very important features that you can use to make your system more robust in the event of a kernel panic or other boot error: saved default boot entries and fallbacks.

    The fallback command is extremely handy when installing and testing new kernels. Without fallbacks, testing a new kernel can be a big hassle. The new kernel may panic upon boot, so one would need to attach a KVM over IP to the server so that if the new kernel panics, it is possible to force the older [functioning] kernel to boot.

    GRUB's fallback feature works around this problem by allowing you to specify a series of boot entries that will be attempted sequentially. Fallbacks become even more useful when combined with the saved default feature, which allows GRUB to store the default boot entry. Upon booting any kernel, the stored default boot target can be changed to the next fallback kernel. The workflow is like so:

    1. GRUB reads the stored default boot target
    2. GRUB sets the default boot target to the next designated fallback entry, then boots the kernel
    3. If the kernel panics, reboot the machine and go to step 1.

    Notice that in each iteration the stored boot target changes and we are certain that at least one kernel version is functional (since the machine is already running at this point) so we can have GRUB attempt to boot the most recent kernel, and should it fail, work its way towards the known working version until a successful boot completes.

    So, how do we get this setup? First, let's create the grub-set-default script that will be used to update the default boot target:

    cat << EOF > /usr/local/sbin/grub-set-default
    echo "savedefault --default=\$1" | grub --batch
    chmod +x /usr/local/sbin/grub-set-default

    Next, edit /etc/grub.conf and replace the default=N parameter with default saved. This will cause GRUB to boot the saved default entry which can be specified by running the grub-set-default command.

    Next, GRUB needs to know which entries to boot in case of a failure. In most cases you will want this to be the second entry (n=1), so add on a new line after default saved:


    Should you wish for GRUB to attempt a third entry in case the second also fails, you can specify instead:

    fallback=1 2

    In this case, just be sure that your system actually has 3 boot entries, ie 3 kernels installed simultaneously. Be warned though, GRUB will probably not like it if you reference a boot entry that has not yet been defined.

    The key step to this setup is to have GRUB save the next fallback kernel as the default when booting an entry. This can be done by finding the line that looks like

    kernel /path/to/vmlinuz various_boot_options

    and adding directly underneath it on a new line:

    savedefault fallback

    On the final fallback boot entry, do not add fallback at the end and simply use savedefault instead.

    The last thing is to inform GRUB of the default boot target:

    grub-set-default 0

    That's it! For your reference, here is a sample configuration using the saved default + fallback method with 3 kernels:

    # grub.conf generated by anaconda
    # Note that you do not have to rerun grub after making changes to this file
    # NOTICE:  You have a /boot partition.  This means that
    #          all kernel and initrd paths are relative to /boot/, eg.
    #          root (hd0,0)
    #          kernel /vmlinuz-version ro root=/dev/VolGroup00/LogVol00
    #          initrd /initrd-version.img
    default saved
    fallback=1 2
    title CentOS (2.6.18-164.10.1.el5)
         root (hd0,0)
         kernel /vmlinuz-2.6.18-164.10.1.el5 ro root=/dev/VolGroup00/LogVol00
         savedefault fallback
         initrd /initrd-2.6.18-164.10.1.el5.img
    title CentOS (2.6.18-164.9.1.el5)
         root (hd0,0)
         kernel /vmlinuz-2.6.18-164.9.1.el5 ro root=/dev/VolGroup00/LogVol00
         savedefault fallback
         initrd /initrd-2.6.18-164.9.1.el5.img
    title CentOS (2.6.18-164.el5)
         root (hd0,0)
         kernel /vmlinuz-2.6.18-164.el5 ro root=/dev/VolGroup00/LogVol00
         initrd /initrd-2.6.18-164.el5.img

    In this sample, the system would be currently running kernel version 2.6.18-164.9.1.el5 and the new kernel 2.6.18-164.10.1.el5 would need testing. Prior to rebooting, the system administrator would call grub-set-default 0 to make GRUB boot the first entry (2.6.18-164.10.1.el5). If it fails, savedefault fallback causes the next fallback entry (kernel 2.6.18-164.9.1.el5) to be booted. If it also fails, the second entry (2.6.18-164.el5) is booted. Because it is the last entry, no more fallback entries can be booted which is why the third entry uses savedefault and not savedefault fallback.

    Note that in this sample, the fallback entries follow the order in which they are defined. However, it doesn't have to be so. If you would prefer specifying the last entry (2.6.18-164.el5) as the first fallback kernel, then all you would have to do is change fallback=1 2 to fallback=2 1 when savedefault fallback is executed, it will look to the fallback parameter and set the next boot entry to entry 2 first, then entry 1.

    Now that the system can gracefully handle kernel panics while booting, an obvious question is what if a kernel panic occurs once the system is running? Fortunately, there is a sysctl parameter that can automatically reboot the system after a kernel panic:

    cat << EOF >> /etc/sysctl.conf

    # Reboot 10 seconds after a kernel panic
    kernel.panic = 10
    kernel.panic_on_oops = 10

    Your system can now gracefully handle boot failures and kernel panics.

    Preventing repeated local/unix login attempts

    pam_tally2 can be enabled to lock out users who fail to authenticate 3 times consecutively:

    touch /var/log/tallylog
    cat << 'EOF' > /etc/pam.d/system-auth
    # This file is auto-generated.
    # User changes will be destroyed the next time authconfig is run.
    auth        required      pam_env.so
    auth        sufficient    pam_unix.so nullok try_first_pass
    auth        requisite     pam_succeed_if.so uid >= 500 quiet
    auth        required      pam_deny.so
    auth        required      pam_tally2.so deny=3 onerr=fail unlock_time=60

    account     required      pam_unix.so
    account     sufficient    pam_succeed_if.so uid < 500 quiet
    account     required      pam_permit.so
    account     required      pam_tally2.so per_user

    password    requisite     pam_cracklib.so try_first_pass retry=3 minlen=9 lcredit=-2 ucredit=-2 dcredit=-2 ocredit=-2
    password    sufficient    pam_unix.so sha512 shadow nullok try_first_pass use_authtok remember=10
    password    required      pam_deny.so

    session     optional      pam_keyinit.so revoke
    session     required      pam_limits.so
    session     [success=1 default=ignore] pam_succeed_if.so service in crond quiet use_uid
    session     required      pam_unix.so

    A username can be unlocked by running:

    pam_tally2 --reset -u username

    Preventing repeated remote login attempts

    SSH: Denyhosts

    On any server with SSH exposed, it is a good idea to install Denyhosts. Denyhosts will periodically audit the /var/log/secure file periodically and ban any users over a given number of authentication failures within a set amount of time. It is extremely useful in combatting the inevitable automated brute force attacks that your system will be a victim of once it is connected to the Internet.

    yum install denyhosts
    chkconfig denyhosts on
    service denyhosts start

    Although the defaults should 'just work' for more systems, if you wish to configure the number of failed authentication attempts before a ban (or the ban time, for example) you can look in /etc/denyhosts.conf.

    Denyhosts has a whitelist file located at /var/lib/denyhosts/allowed-hosts to which you should consider adding your home IP address/hostname to to prevent yourself from getting accidentally locked out. Denyhosts will resolve domain names, so if you have a dynamic DNS account, you can add it to the file and rest easy knowing that your home IP is always whitelisted.

    POP3/IMAP: Fail2ban

    Like Denyhosts, Fail2ban also locks out malicious users once they have tried (and failed) to authenticate too many times on your system within a configurable period of time. However, unlike Denyhosts which only analyses the logs for SSH authentication failures, Fail2ban supports multiple "jail" definitions which can be configured with different log files and different regular expression patterns to match against. The code below will setup a Fail2ban jail that will lock out users with too many SASL authentication failures over POP3/IMAP, but you are also free to implement additional jails for other services if the need be.

    yum install fail2ban
    cat << EOF > /etc/fail2ban/filter.d/dovecot-pop3imap.conf
    failregex = dovecot: auth-worker\(default\): sql\(.*,\): Password mismatch
                dovecot: auth-worker\(default\): sql\(.*,\): unknown user
    ignoreregex =
    cat << EOF >> /etc/fail2ban/jail.conf
    enabled = true
    filter = dovecot-pop3imap
    action = iptables-multiport[name=dovecot-pop3imap, port="pop3,imap,pop3s,imaps,smtp", protocol=tcp]
    logpath = /var/log/maillog
    maxretry = 20
    findtime = 1200
    bantime = 1200

    Now, let's start all configured Fail2ban jails:

    chkconfig fail2ban on
    service fail2ban start

    You should consider editing /etc/fail2ban/jail.conf and add your IP address to the ignoreip parameter so you do not accidentally lock yourself out while testing. Just like Denyhosts, DNS hostnames are also resolved so feel free to use your dynamic DNS hostname.

    Hardening the system

    Below are a list of code snippets that you can execute to help harden the server with little or no effect from a user perspective. Many of these tips were found in the NSA's guide to securing RHEL 5.

    • Set /home to mount as noexec,nosuid,nodev to prevent binaries from being run
    • Update password hashing to sha512 (md5 has some well-known vulnerabilities):
      /usr/sbin/authconfig --passalgo=sha512 --update

      Note that you will need to reset any existing passwords to take advantage of the new algorithm, even choose the same password. In other words, at a minimum reset the password for root and your user account by running password username for each.

    • Nobody should be trying to plug memory stick into your server when you're not around, so allow administrators to manually insmod usb-storage but disable autoloading:
      echo "install usb-storage : " >> /etc/modprobe-nousbstorage.conf
    • Blacklist wifi modules from loading:
      rm -f /etc/modprobe.d/blacklist-wireless;for i in $(find /lib/modules/`uname -r`/kernel/drivers/net/wireless -name "*.ko" -type f) ; do echo blacklist $(basename ${i%%.ko}) >> /etc/modprobe.d/blacklist-wireless; done
    • Disable core dumps:
      sed -i'' 's/# End of file/* hard core 0\n# End of file/' /etc/security/limits.conf
      cat << EOF >> /etc/sysctl.conf

      # Disable core dumps for SUID executables
      fs.suid_dumpable = 0

    • Force the use of ExecShield (it is enabled by default):
      cat << EOF >> /etc/sysctl.conf

      # Force ExecShield to be enabled (it is on by default)"
      kernel.exec-shield = 1
      kernel.randomize_va_space = 1

    • Tweak sysctl network settings:
      cat << EOF >> /etc/sysctl.conf

      # Only needed for routers
      net.ipv4.conf.all.send_redirects = 0
      net.ipv4.conf.default.send_redirects = 0

      # Recommended settings for all hosts
      net.ipv4.conf.all.accept_source_route = 0
      net.ipv4.conf.all.accept_redirects = 0
      net.ipv4.conf.all.secure_redirects = 0
      net.ipv4.conf.all.log_martians = 1
      net.ipv4.conf.default.accept_redirects = 0
      net.ipv4.conf.default.secure_redirects = 0
      net.ipv4.icmp_echo_ignore_broadcasts = 1
      # This next one is recommended but apparently not valid
      #net.ipv4.icmp_ignore_bogus_error_messages = 1
      net.ipv4.conf.all.rp_filter = 1

    • Disable use of su command unless user is in wheel group:
      allowed_users="myusernameuser other_user1 other_user2"
      for user in $allowed_users;do
        usermod -a -G wheel $user
      sed -i'' 's/#auth\t\trequired\tpam_wheel.so use_uid/auth\t\trequired\tpam_wheel.so use_uid/' /etc/pam.d/su
    • Disable interactive boot:
      sed -i'' 's/PROMPT=yes/PROMPT=no/' /etc/sysconfig/init
    • A shell timeout will prevent anyone from accidentally leaving idle shells open:
      echo "TMOUT=900" >> /etc/profile.d/tmout.sh
      echo "readonly TMOUT" >> /etc/profile.d/tmout.sh
      echo "export TMOUT" >> /etc/profile.d/tmout.sh
      chmod 755 /etc/profile.d/tmout.sh
    • Rewrite the default system login banner so that you do not give away the OS and release version:
      echo 'Unauthorized access OR use of this system is prohibited.' > /etc/issue
    • Disable zeroconf network configuration (169.254.* IP addresses):
      echo "NOZEROCONF=yes" >> /etc/sysconfig/network
    • Some services are sensitive to the system date, so have ntpd update the system clock every 15 minutes:
      yum install ntp
      echo "15 * * * * root /usr/sbin/ntpd -q -u ntp:ntp > /dev/null" >> /etc/cron.d/ntpd
    • Disable your Apache server signature; there's no need to advertise if you are running a version with known flaws:
      sed -i'' 's/ServerSignature On/ServerSignature Off/' /etc/httpd/conf/httpd.conf
    • Disable unneeded httpd modules. Below are just some of the LoadModule lines that you can comment out /etc/httpd/httpd.conf:
      LoadModule authn_alias_module modules/mod_authn_alias.so
      LoadModule authn_anon_module modules/mod_authn_anon.so
      LoadModule authz_owner_module modules/mod_authz_owner.so
      LoadModule authz_dbm_module modules/mod_authz_dbm.so
      LoadModule auth_digest_module modules/mod_auth_digest.so
      LoadModule ldap_module modules/mod_ldap.so
      LoadModule authnz_ldap_module modules/mod_authnz_ldap.so
      LoadModule include_module modules/mod_include.so
      LoadModule dav_module modules/mod_dav.so
      LoadModule dav_fs_module modules/mod_dav_fs.so
      LoadModule status_module modules/mod_status.so
      LoadModule info_module modules/mod_info.so
      LoadModule proxy_module modules/mod_proxy.so
      LoadModule proxy_balancer_module modules/mod_proxy_balancer.so
      LoadModule proxy_ftp_module modules/mod_proxy_ftp.so
      LoadModule proxy_http_module modules/mod_proxy_http.so
      LoadModule proxy_connect_module modules/mod_proxy_connect.so

      Similarly, in /etc/httpd/conf.d/proxy_ajp.conf, you can disable:

      LoadModule proxy_ajp_module modules/mod_proxy_ajp.so
    • Disable prelinking by changing PRELINKING=yes to PRELINKING=no in /etc/sysconfig/prelink. In order to apply the changes immediately, start the prelink cron job one last time:
      sh /etc/cron.daily/prelink

    Firewall: iptables/netfilter



    Intrusion detection

    Intrusion detection software is a very useful warning system to detect an intrusion in a server. Although they will require your careful attention because any minute change on the server will result in a warning, for the same reason they can provide detailed information about exactly what was compromised during a break-in.

    yum install rkhunter aide

    Although the defaults for AIDE are fine, rkhunter needs a bit more configuration. Near line 200, you will find:


    This setting needs to match the PermitRootLogin setting in sshd_config, so therefore the value needs to be changed to no. Near line 540, you will see:


    This line needs to be commented out, as xinetd is not installed at all. Near line 610, you will find:


    rkhunter needs to be told that it is OK to be running the older versions of Apache, PHP and OpenSSL that the server will be using (Red Hat backports security patches), so change the line to:

    APP_WHITELIST="httpd:2.2.3 php:5.2.9 sshd:5.4p1"

    Now that rkhunter is fully configured, have rkhunter analyse the system:

    rkhunter --propupd

    We can now do the same for AIDE:

    /usr/sbin/aide --init
    cp /var/lib/aide/aide.db.new.gz /var/lib/aide/aide.db.gz

    If you change a configuration file, update software or add users on the server, you will need to re-run these two command snippets to recalibrate rkhunter and AIDE.

    Another trick that can help detect intrusions is to have RPM verify all installed packages nightly. It is trivial to set up a cron job to do this for us:

    cat << 'EOF' > /etc/cron.d/rpm-verify-all
    #S file Size differs
    #M Mode differs (includes permissions and file type)
    #5 MD5 sum differs
    #D Device major/minor number mis-match
    #L readLink(2) path mis-match
    #U User ownership differs
    #G Group ownership differs
    #T mTime differs
    0 2 * * * root rpm -qVa --nomtime | awk '$2!="c" {print $0}'
    chmod +x /etc/cron.d/rpm-verify-all

    Resources and further reading

  • 2 min read
  • I have been attempting to debug a poor reception problem with my iPhone 4 near my house. It has been confirmed that I reside in a dead zone (I am right in the middle of where 2 towers overlap, so I have the weakest signal from both locations) and there's no incentive from my cell provider to fix the problem since the dead zone is very small.

    In an attempt to help document the problem, I wanted to capture the exact cell IDs and frequencies used when I experience reception problems. Activating the iPhone's Field Test Mode is easy enough (dial *3001#12345#*), but I quickly realized that something was off with the frequencies listed under by the UMTS/GSM RR Info panes. It was displaying download/upload frequencies of 1037/812 respectively, which is reasonable, but at other times it would show frequencies like 437/37 which made no sense at all.

    After a bit of research, it looks like the the label for that data value is wrong; it should be channel and not frequency. Wikipedia has a list of UMTS (3G) frequency bands and the corresponding channel codes as well as the corresponding list for GSM (2G/EDGE). Channels numbers 1037/812 correspond to the 850MHz frequency band which I know Rogers, Telus & Bell all use in their new 3G network deployments. The other popular GSM band in use in Canada is 1900MHz PCS, and sure enough that's what 437/37 corresponds to. Problem solved!

  • 1 min read
  • I went to see Feed Me at the beginning of August and was really happy that he played this:

    I had heard the track before from people's footage of him playing at EDC, but wasn't sure what it was actually called/who it was by... After some research it's apparently called "Dill The Noise" (by Dillon Francis and Kill The Noise). Francis said on his twitter feed (@dillonfrancis) that it should be coming out on his next EP!

    Update 2011/08/03: It has been leaked! youtube.com/watch?v=ZmAdoAysK-w

  • 3 min read
  • This weekend AMD's Llano APUs started shipping this week and Newegg has started offering select models! While I won't be picking one up myself (sorry, I still like my discrete Radeon HD 3850 and will most probably continue to buy discrete PCIe cards), the release of this product makes me really happy because it's been too long that PC manufacturers have been skimping on graphics to lower the cost of PCs and it's really the consumer that pays the price in the end. Yes, true, you can argue "well they should have read the specs" but let's face it, your average consumer has no idea what the difference between the Intel GMA X3150, GeForce 310M or a Radeon HD 6670M is even though choosing one or the other will dramatically affect how they perceive their computer's performance. It is all too common that I see OEMs advertise their beefy and overpriced PCs as fast and all-capable and then when you look at the specs it's got something terribly outdated and underpowered like an Intel GMA X4500 or nVidia GeForce 7100 graphics processor. Seriously, why even bother with the powerful CPU if it can't handle any sort of graphics properly?

    The situation is even worse for laptops... In the past few months I've found that it is nearly impossible to find a laptop that has a nice balance between CPU and GPU power for the average consumer. OEMs seem to have their computers fall into one of three builds, and those builds are (a) low-power computing with cheap CPUs and integrated graphics (b) high performance computing with i5s/i7s and an overkill on RAM or (c) gaming computers that weigh lots, have a short battery life and are very expensive. In my opinion they are completely missing the mark on what the "average consumer" actually needs, which is a mediocre CPU with a entry level discrete graphics card and a healthy amount of RAM (3 or 4GB is acceptable) in the $600-700 range. You can think of it as targeting "basic gaming" or "media PC". Samsung has been very good at this actually, offering laptops the higher-end Core 2 Duo CPUs or i3s with the low-class nVidia GeForce 310M/320M series GPUs (some even in the $550-range). Computers like that are well priced, perfect for web & office work but won't choke if you throw 1080p at it, and you could even play Starcraft 2 or WoW on low settings.

    Anyways, all that goes to say that this integrated graphics madness should now come to an end. I can't wait until the OEMs start using the AMD chips in desktop in laptops, as it will provide the consumer with a cost-effective computer that can actually process a decent graphic payload - and that's worth a lot. Not to mention that the battery life should improve considerably, too.