Migrating Xen instances to KVM

I started out using Xen as my virtual machine hosting software of choice; mainly because I had 32-bit server and at the time other solutions weren't readily available. Over time this situation has changed, and KVM is a somewhat more capable system for me - so I've been looking to migrate my existing Xen-based systems to KVM. This is a bit more difficult than expected, so I've documented my process here in the hopes that others will find this useful.

In particular, I'm using the following software - if you're not, this may be less relevant and even unhelpful to you:

I'm sure this could be improved. If you've got any hints about simplifications or better ways of doing this, please let me know.

Major issues

There are a few main issues which stand in the way of an easy migration. They are:

  1. Xen instances tend to have their kernel and initrd located on the host system, and their kernel is specialised to run as a Xen guest. KVM works with a standard kernel.
  2. Xen instances also have other specialisations, like libc6-xen, which - while not problematic - are sub-optimal for KVM.
  3. often, a Xen instance will just be set up as a bare partition holding the root file system. However, KVM will try to boot up like a normal PC does, from the BIOS - you need a "proper" disk layout which you probably don't have on the Xen instance.

Setting up the new KVM disk

Setting up the disk space for the new KVM system is the first port of call, because you need this setup before you can migrate content onto it. I've prefered here to create a new, blank disk and transfer the existing data in: you could also use a disk-image based approach, but this way you have the opportunity to resize the disk/etc.

First, reserve the area on the LVM volume group for the disk. Parts in italics you will want to change, they are the disk size, disk name and volume group name respectively. Make sure the disk is big enough for all the partitions you want to create (remember swap!). My naming convention is [hostname]-[mountpoint], so that I can do 'lvdisplay' and know which disks belong where on a given volume group.

lvcreate -L 10G -n test-root vgname

Then, the disk needs to be partitioned up. You can use whatever tool you would usually use to do this; I tend to use fdisk, e.g.:

fdisk /dev/vgname/test-root

Make sure to make your main partition bootable, and set the right types (Linux, Linux swap, etc.) for consistency.

At this point, we want to copy across the data from the existing system into the new one. In order to do this easily, we need to access the partitions on the new system which aren't readily available. The easiest way to do this is install 'kpartx' and do something like the following:

losetup /dev/loop0 /dev/vgname/test-root
kpartx -a /dev/loop0

You may need to use another device than loop0; 'losetup -f' will tell you the next free one. This will then make the partitions of the new disk available in your kernel, as /dev/mapper/loop0px - where 'x' is the partition number.

You can then format the main disk - for example, if your new root partition is the first one on the disk, do something like:

mkfs.ext3 /dev/mapper/loop0p1

Transfer the existing data

Unlike other operating systems, you can easily copy stuff from one system to another in Linux. I tend to prefer the use of rsync, since you can use ssh if you want to transfer over the network.

At this point, you need to bring down the Xen host, since you don't really want to copy stuff live as it will change under you (if time is crucial, clone the disk and transfer it as below, then bring the host down - the later sync will be very quick). Mount the disks on both the Xen host and the new KVM host; I'm assuming they're /mnt on both systems. For example, on the KVM host you might do:

mount /dev/mapper/loop0p1 -t ext3 /mnt

Then rsync the data across. I tend to do this from the point of view of the new host, but it's up to you and your setup really:

rsync -avz -e ssh oldhost:/mnt/ /mnt/

Ensure that you have a slash on the end of both paths. If you insist on doing it on a live system, make sure it doesn't descend into other filesystems with the -x option.

Unmount and remove the partitions from your host system thusly:

umount /mnt
kpartx -d /dev/loop0
losetup -d /dev/loop0

You'll also want to unmount the partition on the old Xen host as well.

Set up the new KVM instance

Now, the easiest thing I've found to do is to set up the new instance with an existing Debian install CD so that you can boot it up initially into rescue mode - this is useful for a number of reasons I'll get onto in a minute. Using virt-install, we can set up a new host like this:

virt-install -n test -r 256 -w bridge:br0 \
	-c debian-install.iso --accelerate --os-type=linux \
	--vnc --hvm -f /dev/vgname/test-root

Obviously, you'll need to alter a few options to taste. This command may take a little bit to run. If you don't have a graphic system set up you'll get an error like "Unable to connect to graphical console" - but that's ok; you can use virt-manager from another system to connect in and get to the console.

Configure the new KVM instance

If you've set it up with a Debian install disk, when you first connect to the new instance it will be waiting at the boot prompt. Boot into "rescue" mode: this will give you a chance to get the new system into a bootable shape.

Depending on what your system was previously setup to do, there are a number of things you may need to do, including:

Once you're happy with it, reboot and start the VM again. Hopefully, you'll be booting the system up normally and stuff is mostly working again!