Migrating XEN installation from fc5 to centos5

Scenario

• Deployment: several XEN guests running Debian 3.1 (Sarge) over a Fedora Core 5 (fc5) host. Host & guests installed with distro-provided packages.
  
• Goal: migrate host from fc5 to centos5 (held migration until centos5 got released)
  
• Difficulty: a lot. :-S
  

UPDATE 11-Jun-2007: See 2.c: /dev/xvc0 instead of /dev/console in guest
UPDATE 13-Jun-2007: See 2.d: oneliner for easy fixed guest MAC generation

Migration

It was faaaaar... more complex than we originally thought.

1. XEN host stuff

1.a. Just one kernel-xen package

FC5 came with two kernel-xen flavor: kernel-xen0 for the Dom0 guest and kernel-xenU for the other unprivileged guests.
Centos5 (and FC6) comes with just one kernel-xen package, this is quite annoying at first (this "mix" of true hardware drivers and virt. guest ones), but it starts to make sense once you ride the wave :)
You can see it with ("front" ones are for the guests , "back" for the host):

host# rpm -ql kernel-xen | egrep /xen

1.b. XEN guest kernel doesn't have the virtual block driver

That is: the guest will plainly PANIC if used without an initrd, so now you do have to make an explicit initrd.guest.img (whatever name you'd like) and add a ramdisk= option to the xen guest config file.
That is:

host# mkinitrd --preload=xenblk --preload=xennet -f -v /boot/initrd-2.6.18-8.1.4.el5xen.guest.img 2.6.18-8.1.4.el5xen
host# vim /etc/xen/xm-guest1 ### add: ramdisk=/boot/initrd.guest.img ### see below
host# restorecon -v /boot/initrd*

The last line (restorecon ... ) is needed because xend is _correctly_ running confined by "targeted" SELinux, and mkinitrd doesn't relabel the initrd file under /boot to allow xend access.
BTW, we prefer to have a "visible" and stable guest file configuration, so we did

host# cd /boot
host# ln -sf initrd-2.6.18-8.1.4.el5xen.guest.img initrd-guest.img
host# ln -sf vmlinuz-2.6.18-8.1.4.el5xen vmlinuz-guest

1.c. [UPDATE] that nasty "4gb seg fixup, process ..." (@host)

From XenFAQ and elsewhere:

host# echo 'hwcap 0 nosegneg' > /etc/ld.so.conf.d/libc6-xen.conf

host# ldconfig -v

2. XEN guest stuff

2.a. udevd inside guests

Debian 3.1 Sarge is installed on our guest images.
Now with Centos5 XEN we do need udevd running inside guests (to correctly setup /dev), that is:

guest# apt-get install udev ### installs udev and hotplug packages

2.b. the return of the "4gb seg fixup" (guest)

That comes from the way Xen uses the CPU segmentation; for newer distros it may be solved with echo 'hwcap 0 nosegneg' > /etc/ld.so.conf.d/libc6-xen.conf , but Debian 3.1 doesn't come with this feature, so we had to:

guest# mv /lib/tls /lib/tls.DISABLED

"Offline'ing" TLS glibc implementation solved the problem, beware that you'll need to redo this everytime libc6 package is upgraded (trivially solved by a rcS script).

2.c. getty /dev/tty1 -> getty /dev/xvc0

UPDATE: /dev/console _seemed_ to work, but it lacked tty' normal signal handling such as Ctrl-C (?), putting /dev/xvc0 solved the problem.

For whatever reason (?), /dev/tty1 was working nicely as Xen guest console (xm console ), but now you should use /dev/xvc0.
That is:

guest# vim /etc/inittab ### check that following line is present:

x0:2345:respawn:/sbin/getty 38400 xvc0

guest# echo xvc0 >> /etc/securetty

2.d. [UPDATE] Debian 4.0 and network device naming

After upgrading guests from Debian 3.1 (Sarge) to 4.0 (Etch) a "nice touch" appeared: AFAICS Debian udev infrastructure tries to keep netdev naming "constant" based on device's MAC address (no, thanks |-[ ).
Given that Xen generates a non-constant MAC address each time it boots a guest, this makes each Debian 4.0 guest boot to have an increasing ethN device.

Two possible solutions:

1. Fix MAC address in Xen guest vm config, you could use the following oneliner that uses the first 6 hexdigits from the md5 over the (unique) guestname. This 6 hexdigits are appended to the XenSource reserved MAC prefix 00:16:3E .
   
   guest# echo -n guestname.FQDN | md5sum | sed -r 's/(..)(..)(..).*/00:16:3E:\1:\2:\3/'
   
   ... or ...
   
2. Just disable the correspondent udev rules by renaming:
   
   guest# mv /etc/udev/rules.d/{,.}z25_persistent-net.rules

... pheeuuu ... 'nuff written.