Friday, June 08, 2007

Migrating XEN installation from fc5 to centos5

Scenario


  • Deployment: several XEN guests running Debian 3.1 (Sarge) over a Fedora Core 5 (fc5) host. Host & guests installed with distro-provided packages.
  • Goal: migrate host from fc5 to centos5 (held migration until centos5 got released)
  • Difficulty: a lot. :-S

UPDATE 11-Jun-2007: See 2.c: /dev/xvc0 instead of /dev/console in guest
UPDATE 13-Jun-2007: See 2.d: oneliner for easy fixed guest MAC generation

Migration

It was faaaaar... more complex than we originally thought.

1. XEN host stuff

1.a. Just one kernel-xen package

FC5 came with two kernel-xen flavor: kernel-xen0 for the Dom0 guest and kernel-xenU for the other unprivileged guests.
Centos5 (and FC6) comes with just one kernel-xen package, this is quite annoying at first (this "mix" of true hardware drivers and virt. guest ones), but it starts to make sense once you ride the wave :)
You can see it with ("front" ones are for the guests , "back" for the host):
host# rpm -ql kernel-xen | egrep /xen

1.b. XEN guest kernel doesn't have the virtual block driver

That is: the guest will plainly PANIC if used without an initrd, so now you do have to make an explicit initrd.guest.img (whatever name you'd like) and add a ramdisk= option to the xen guest config file.
That is:
host# mkinitrd --preload=xenblk --preload=xennet -f -v /boot/initrd-2.6.18-8.1.4.el5xen.guest.img 2.6.18-8.1.4.el5xen
host# vim /etc/xen/xm-guest1 ### add: ramdisk=/boot/initrd.guest.img ### see below
host# restorecon -v /boot/initrd*
The last line (restorecon ... ) is needed because xend is _correctly_ running confined by "targeted" SELinux, and mkinitrd doesn't relabel the initrd file under /boot to allow xend access.
BTW, we prefer to have a "visible" and stable guest file configuration, so we did
host# cd /boot
host# ln -sf initrd-2.6.18-8.1.4.el5xen.guest.img initrd-guest.img
host# ln -sf vmlinuz-2.6.18-8.1.4.el5xen vmlinuz-guest

1.c. [UPDATE] that nasty "4gb seg fixup, process ..." (@host)

From XenFAQ and elsewhere:
host# echo 'hwcap 0 nosegneg' > /etc/ld.so.conf.d/libc6-xen.conf

host# ldconfig -v

2. XEN guest stuff

2.a. udevd inside guests

Debian 3.1 Sarge is installed on our guest images.
Now with Centos5 XEN we do need udevd running inside guests (to correctly setup /dev), that is:
guest# apt-get install udev ### installs udev and hotplug packages

2.b. the return of the "4gb seg fixup" (guest)

That comes from the way Xen uses the CPU segmentation; for newer distros it may be solved with echo 'hwcap 0 nosegneg' > /etc/ld.so.conf.d/libc6-xen.conf , but Debian 3.1 doesn't come with this feature, so we had to:
guest# mv /lib/tls /lib/tls.DISABLED
"Offline'ing" TLS glibc implementation solved the problem, beware that you'll need to redo this everytime libc6 package is upgraded (trivially solved by a rcS script).

2.c. getty /dev/tty1 -> getty /dev/xvc0

UPDATE: /dev/console _seemed_ to work, but it lacked tty' normal signal handling such as Ctrl-C (?), putting /dev/xvc0 solved the problem.

For whatever reason (?), /dev/tty1 was working nicely as Xen guest console (xm console ), but now you should use /dev/xvc0.
That is:
guest# vim /etc/inittab ### check that following line is present:

x0:2345:respawn:/sbin/getty 38400 xvc0

guest# echo xvc0 >> /etc/securetty

2.d. [UPDATE] Debian 4.0 and network device naming

After upgrading guests from Debian 3.1 (Sarge) to 4.0 (Etch) a "nice touch" appeared: AFAICS Debian udev infrastructure tries to keep netdev naming "constant" based on device's MAC address (no, thanks |-[ ).
Given that Xen generates a non-constant MAC address each time it boots a guest, this makes each Debian 4.0 guest boot to have an increasing ethN device.

Two possible solutions:
  1. Fix MAC address in Xen guest vm config, you could use the following oneliner that uses the first 6 hexdigits from the md5 over the (unique) guestname. This 6 hexdigits are appended to the XenSource reserved MAC prefix 00:16:3E .

    guest# echo -n guestname.FQDN | md5sum | sed -r 's/(..)(..)(..).*/00:16:3E:\1:\2:\3/'

    ... or ...
  2. Just disable the correspondent udev rules by renaming:

    guest# mv /etc/udev/rules.d/{,.}z25_persistent-net.rules

... pheeuuu ... 'nuff written.

5 comments:

Unknown said...

More on Debian 4.0 interface naming and Xen at:

http://etbe.blogspot.com/2007/03/xen-and-eth-device-renaming.html

Marcelo Aguero said...

Very useful!

I have some servers that support full virtualization (Intel VT). I want something like you, i.e. CentOS 5 in dom0
AND domU's.

Do you recommend full virtualization or paravirtualization? I'll install only CentOS or another GNU/Linux in domU's.

Unknown said...

Hi Marcelo,

I've never used full virt (HVM) under Xen, but if it's Linux-over-Linux seems reasonable to just use paravirt.
About the distro choice: I like Centos5 having everything including SELinux ready for use as dom0, but I prefer Debian as domU's because it's a way more tiny and "elatic" distro (eg. not more that 300Mb for a base install, more that one version of same pkg) and an unbeatable package manager (apt).

Unknown said...

BTW, Ian Blenke has several useful and detailed posts about virtualization, Xen, HVM/PVM at http://ian.blenke.com/system/virtualization/paravirtualization/hardware/software/vmx/svm/vt/raw/iron/pv/hvm/qemu/xen/solaris/zones/brandz/jails/openvz/power5/virtual/machine/which_virtualization_is_right_for_you.html
worth a read.

Anonymous said...

Instead of xvc0 with domU kernel 2.6.18 you need to use hvc0
for recent kernels (>=2.6.25).