I have had a 1U server co-located for some time now at iWeb Technologies' datacenter in Montreal. So far I've had no issues and it did a wonderful job hosting websites & a few other VMs, but because of my concern for its aging hardware I wanted to migrate away before disaster struck.

Modern VPS offerings are a steal in terms of they performance they offer for the price, and Linode's 4096 plan caught my eye at a nice sweet spot. Backed by powerful CPUs and SSD storage, their VPS is blazingly fast and the only downside is I would lose some RAM and HDD-backed storage compared to my 1U server. The bandwidth provided wit the Linode was also a nice bump up from my previous 10Mbps, 500GB/mo traffic limit.

When CentOS 7 was released I took the opportunity to immediately start modernizing my CentOS 5 configuration and test its configuration. I wanted to ensure full continuity for client-facing services - other than a nice speed boost, I wanted clients to take no manual action on their end to reconfigure their devices or domains.

I also wanted to ensure zero downtime. As the DNS A records are being migrated, I didn't want emails coming in to the wrong server (or clients checking a stale inboxes until they started seeing the new mailserver IP). I can easily configure Postfix to relay all incoming mail on the CentOS 5 server to the IP of the CentOS 7 one to avoid any loss of emails, but there's still the issue that some end users might connect to the old server and get served their old IMAP inbox for some time.

So first things first, after developing a prototype VM that offered the same service set I went about buying a small Linode for a month to test the configuration some of my existing user data from my CentOS 5 server. MySQL was sufficiently easy to migrate over and Dovecot was able to preserve all UUIDs, so my inbox continued to sync seamlessly. Apache complained a bit when importing my virtual host configurations due to the new 2.4 syntax, but nothing a few sed commands couldn't fix. So with full continuity out of the way, I had to develop a strategy to handle zero downtime.

With some foresight and DNS TTL adjustments, we can get near zero downtime assuming all resolvers comply with your TTL. Simply set your TTL to 300 (5 minutes) a day or so before the migration occurs and as your old TTL expires, resolvers will see the new TTL and will not cache the IP for as long. Even with a short TTL, that's still up to 5 minutes of downtime and clients often do bad things... The IP might still be cached (e.g. at the ISP, router, OS, or browser) for longer. Ultimately, I'm the one that ends up looking bad in that scenario even though I have done what I can on the server side and have no ability to fix the broken clients.

To work around this, I discovered an incredibly handy tool socat that can make magic happen. socat routes data between sockets, network connections, files, pipes, you name it. Installing it is as easy as: yum install socat

A quick script later and we can forward all connections from the old host to the new host:

And just like that, every connection made to the old server is immediately forwarded to the new one. This includes the MySQL socket (which is automatically used instead of a TCP connection a host of 'localhost' is passed to MySQL).

Note how we establish a SSH tunnel mapping a connection to localhost:3306 on the new server to port 3307 on the old one instead of simply forwarding the connection and socket to the new server - this is done so that if you have users who are permitted on 'localhost' only, they can still connect (forwarding the connection will deny access due to a connection from a unauthorized remote host).

Update: a friend has pointed out this video to me, if you thought 0 downtime was bad enough... These guys move a live server 7km through public transport without losing power or network!

Alex gave a very interesting talk at KVM Forum 2014 about the current state of VGA passthrough using KVM & VFIO:

This how-to will show you how to configure:

• Install and configure the KVM hypervisor
• Patch the kernel and QEMU for better compatibility with graphics card / VGA VFIO passthrough
• Create and configure a new virtual machine (VM) with real hardware attached to it
• Configure CPU pinning on the VM for better gaming performance

Build considerations & preparation

QEMU has several PCI passthrough techniques, the newest of which is VFIO. QEMU's normal PCI passthrough leaves much to be desired whereas VFIO takes full advantage of IOMMU, has better device support and prevents multiple access to the same device (you can read more about it in Alex Williamson's presentation here).

That said, VFIO it is relatively new and experimental technology for the purposes of passing through entire VGA cards to virtual machines. While myself and many others have had tremendous success, different hardware can produce different results and getting there may not always be straightforward.

Likely, until Fedora 21 is released you will need to patch and rebuild both the Linux kernel and QEMU; instructions for doing so will be provided in this tutorial. If you are purchasing hardware, it is also strongly recommended that you read over the KVM VGA-passthrough thread on Arch Forums to confirm that your intended hardware configuration has been reported to work by another user.

Personally, the following hardware has worked wonderfully for me:

• Motherboard: Supermicro X10SAE
• CPU: Xeon E3-1225v3
• Audio: Onboard (Intel C220 HD audio) and AMD R9 270X HDMI
• Video: AMD Radeon R9 270X
• Network: Intel I210 Gigabit

Anecdotal evidence suggests that for graphics passthrough, nVidia cards seem to fare better than AMD ones. However, success has been had on both sides on a variety of device models dating back several years. I have found that generally, problems are not inherent to the hardware but more a matter of adjusting you software stack (i.e. applying certain patches) to get a compatible passthrough. From my reading nVidia's GeForce 6xx/7xx and AMD's Radeon R9 series seem to work fairly painlessly.

For network cards, always pass through an Intel ethernet controller over a Realtek one if possible.

Common problems with VGA VFIO passthrough

Before getting to the fun part, there are several key pieces to getting a functioning VGA passthrough that need further description. Understanding these issues will be key in creating a functioning host environment for VFIO VGA passthrough.

PCIe device reset

In order to re-initialize a PCIe device, it needs to be reset. Normally the host controls this, however now that we are passing through the device to a VM, some additional work is required to get reset functioning correctly. Without this extra PCIe reset support, the machine typically freezes when starting your VM for a second time.

Fortunately for us, kernel >= 3.12 has this support and simply upgrading the kernel fixes the issue.

VGA arbiter and multiple GPUs

Passing through generic PCI devices with VFIO works pretty well. Graphics card passthrough gets put into its own category called "VGA passthrough" because of the technical challenges involved in presenting a functioning GPU for the virtual machine to initialize without things going awry.

Most computers today come with a GPU built-in to the CPU. This will be a major headache when trying to setup VGA passthrough, as VGA is a really old standard. Back when it was created, having multiple graphics cards on a single system was not a configuration they had foreseen or designed for. VGA calls can only be directed to a single device at a time, so the kernel has to use a VGA arbiter that switches the active device and directs VGA accesses to the appropriate card. I am oversimplifying this a bit, but Alex Williamson has a detailed post explaining the technical issues. In short, the x-vga=on flag passed to VFIO indicates to the VGA arbiter that the VFIO driver will need to participate in VGA arbitration, so everyone stays happy.

The problem is that the Xorg i915 driver for Intel's integrated GPUs does not participate in VGA arbitration, even though the devices claim the VGA address space. This means VGA calls get directed to the wrong card, (a) messing up your display on the host and (b) preventing the graphics card on the VM from functioning correctly. Ugh.

Fortunately, Alex has also written kernel patches to fix this, however be warned that they cripple 3D performance on the Intel GPU. Since we're building a high-performance VM for gaming, I am assuming that will not be an issue for you.

Kernel patches:

• 81b5c7bc8de3e6f63419139c2fc91bf81dea8a7d
• 6e1b4fdad5157bb9e88777d525704aba24389bee

NoSnoop

NoSnoop is a feature flag on a PCIe device that allows it to issue transitions that to bypass cache. This can cause consistency problems when passing through the card to a virtual machine.

You can check if you card has NoSnoop enabled by running lspci -vvvv as root and seeing if your graphics card lists NoSnoop+ (enabled) or NoSnoop- (disabled) under the Capabilities > DevCtl section.

Previously this required patches to the kernel, but with kernel 3.15.5 in Fedora, these patches are no longer required.

Rebuilding packages

The first step is to setup a packaging environment and download the upstream source RPMS:

It should have downloaded kernel-[version].fc20.src.rpm in your directory.

Download any of the patches listed above that may be required for your hardware configuration (as plaintext patch/diff files) and save them to ~/rpmbuild/SOURCES.

Rebuilding QEMU

Update 2014-06-09: the newest versions of QEMU in virt-preview (>= 2.0.0) have the NoSnoop patches included! No rebuilding necessary. If you previously followed this guide, remove any QEMU exclusions from yum.conf and update to the latest available version.

Rebuilding Kernel

Download any required patches and save them as plaintext files in your ~/rpmbuild/SOURCES folder. Next, add your PatchXYZ: filename.patch lines where you see other patches being declared. For example:
Next, edit ~/rpmbuild/SPECS/kernel.spec and find the lines where existing patches are listed. You should see something like this:

Add additional lines corresponding to your patchsets. The result could be something like:

I added in the Patch16000 and Patch16000, specifying the filenames of the patch files I saved into my SOURCES folder.

Now, scroll down to the %prep section where you see other patches being applied. That would look something like:

Above the line # END OF PATCH APPLICATIONS, add lines for your patch sets, for example:

Now rebuild the kernel with our patches and install it:

It may list some build dependencies. If so, install them with yum install foo and then run the rpmbuild command again. When the build is complete, it output a list of filenames that you can install. Here's a quick command to do so:

Reboot and your system will be fully patched! I suggest you add a line exclude=kernel* to /etc/yum.conf to prevent for patched packages from being upgraded.

Installing KVM

Because this is all experimental stuff, install fedora-virt-preview to get the latest and greatest software set:

Next, let's be nice to the VMs and give them some time to perform a graceful shutdown before the host powers off:

Edit the default kernel boot arguments (specified in GRUB_CMDLINE_LINUX of /etc/default/grub) and add intel_iommu=on for Intel CPUs or iommu=pt iommu=1 for AMD CPUs to turn on IOMMU functionality.

As well, the host initializes certain devices on init (e.g. graphic cards, USB controller, audio chipsets) so these devices need to be manually assigned to the PCI stub driver to prevent the host from using the device during host boot. It allows the VFIO driver to later bind to the devices and pass them to a VM. Add pci-stub.ids=PCI_IDs where PCI_IDs is a comma-separated list of PCI IDs as given by lspci -nn. For example, my GRUB_CMDLINE_LINUX looks like:

With the configuration changes in place, regenerate the GRUB configuration:

Lastly, we will now need to have VFIO bind to any device you want passed to a VM. This will include devices stubbed on boot and possibly others:

The system will now automatically attempt to bind to the devices indicated in /etc/sysconfig/vfio-bind to VFIO at bootup. The format of FULL_PCI_IDs is a little different than earlier, as it is space separated and requires a full bus address prefix as per ls /sys/bus/pci/devices. You can use lspci -nn to identify a device, and then the output from the file listing to identify its full prefix. Here's an example of my configuration:

Because QEMU normally runs sandboxed, we need to 'unhinge' it and give it root privileges so it can control hardware. In /etc/libvirt/qemu.conf, add:

As well, we need to provide QEMU with access to the VFIO device files. List all available VFIO groups like this:
ls -1 /dev/vfio

For ever number that appears, ensure its full path appears in the cgroup_device_acl configuration parameter. For example, mine looks like so:

That's it! The last step is to actually create your virtual machine. The following snippet creates a sample reference file in your home directory:

Cyberduck recently removed a particularly useful piece of information from their wiki regarding the sharing of bookmarks because it is no longer compatible with the sandboxed variant of Cyberduck available from the App Store. It is, however, still compatible with the Windows and OS X download available directly from its website.

To setup bookmark sharing between Cyberduck clients (works with both OS X or Windows), simply create a folder in your cloud sync folder and then point Cyberduck to it.

On OS X, open a Terminal and execute:

On Windows, press Super+R (Super is the key with the Windows logo on it) to open the "Run" dialog, and enter %APPDATA%. Next, open the Cyberduck.exe_Url_[some_garble]\[Version]\user.config file and modify the config file to add the new parameter:

edit 2021-04-30: Cyberduck has since upgraded to use MacOS's app containerization feature, so this hidden preference no longer exists. However, you can still sync bookmarks - you just need to symlink the "Bookmarks" folder that now lives under the app container path (see the Cyberduck FAQ) to an external directory, e.g. one in your Dropbox or Google Drive:

This how-to will show you how to configure:

• A MyDNS name server
• A database to hold the DNS record information

Before starting

Please ensure that you have followed the instructions in the getting started guide here.

If you have not setup the database server yet, please follow the database how-to first.

Installing MyDNS

Setting up the database

MyDNS uses MySQL as its backend to store record information, so it needs a database setup before it can be configured. Start by opening a root MySQL session:

Enter your MySQL root user's password and type at the mysql> prompt:

Replace new_password with a secure password. It will be used to grant MyDNS read-only access to the record database; this ensures that no exploits can result in write access to the record store (it is recommend that you setup another MySQL user for scripted write access to the database).

Next, import the default database:

The last step is to adjust the MyDNS configuration file to use the newly database user credentials:

As before, replace mydns_password with your selected MySQL user password for MyDNS.

Start the service

MyDNS is now fully configured and ready to run. The service can be started:

Administering the server

MyDNS will now serve records zones from the rr with records from the soa table. The daemon does not have to be restarted for changes to be recognized, so you can take advantage of this by using scripts to update your MyDNS database on-the-fly. Zone replication via SQL backups is another particularly handy side-effect of this feature.

As an example, included below is a small script I use to add new domains my servers:

As you can see above, it adds a zone for each domain and then sets up default CNAME aliases for www and mail to point to the main domain. The main domain gets pointed at the default shared IP using an A record.

Resources

• MyDNS reference manual