Scenario
- Deployment: several XEN guests running Debian 3.1 (Sarge) over a Fedora Core 5 (fc5) host. Host & guests installed with distro-provided packages.
- Goal: migrate host from fc5 to centos5 (held migration until centos5 got released)
- Difficulty: a lot. :-S
UPDATE 11-Jun-2007: See 2.c: /dev/xvc0 instead of /dev/console in guest
UPDATE 13-Jun-2007: See 2.d: oneliner for easy fixed guest MAC generation
Migration
It was
faaaaar... more complex than we originally thought.
1. XEN host stuff
1.a. Just one kernel-xen package
FC5 came with two kernel-xen flavor:
kernel-xen0 for the Dom0 guest and
kernel-xenU for the other unprivileged guests.
Centos5 (and FC6) comes with just one kernel-xen package, this is quite annoying at first (this "mix" of true hardware drivers and virt. guest ones), but it starts to make sense once you ride the wave :)
You can see it with (
"front" ones are for the guests ,
"back" for the host):
host# rpm -ql kernel-xen | egrep /xen
1.b. XEN guest kernel doesn't have the virtual block driver
That is: the guest will plainly
PANIC if used without an initrd, so now
you do have to make an explicit initrd.guest.img (whatever name you'd like) and add a
ramdisk= option to the xen guest config file.
That is:
host# mkinitrd --preload=xenblk --preload=xennet -f -v /boot/initrd-2.6.18-8.1.4.el5xen.guest.img 2.6.18-8.1.4.el5xen
host# vim /etc/xen/xm-guest1 ### add: ramdisk=/boot/initrd.guest.img ### see below
host# restorecon -v /boot/initrd*
The last line (
restorecon ... ) is needed because xend is _correctly_ running confined by "targeted" SELinux, and
mkinitrd doesn't relabel the initrd file under /boot to allow xend access.
BTW, we prefer to have a "visible" and stable guest file configuration, so we did
host# cd /boot
host# ln -sf initrd-2.6.18-8.1.4.el5xen.guest.img initrd-guest.img
host# ln -sf vmlinuz-2.6.18-8.1.4.el5xen vmlinuz-guest
1.c. [UPDATE] that nasty "4gb seg fixup, process ..." (@host)
From XenFAQ and elsewhere:
host# echo 'hwcap 0 nosegneg' > /etc/ld.so.conf.d/libc6-xen.conf
host# ldconfig -v
2. XEN guest stuff
2.a. udevd inside guests
Debian 3.1 Sarge is installed on our guest images.
Now with Centos5 XEN we do need
udevd running inside guests (to correctly setup
/dev), that is:
guest# apt-get install udev ### installs udev and hotplug packages
2.b. the return of the "4gb seg fixup" (guest)
That comes from the way Xen uses the CPU segmentation; for newer distros it may be solved with
echo 'hwcap 0 nosegneg' > /etc/ld.so.conf.d/libc6-xen.conf , but Debian 3.1 doesn't come with this feature, so we had to:
guest# mv /lib/tls /lib/tls.DISABLED
"Offline'ing" TLS glibc implementation solved the problem, beware that you'll need to redo this everytime libc6 package is upgraded (trivially solved by a rcS script).
2.c. getty /dev/tty1 -> getty /dev/xvc0
UPDATE: /dev/console _seemed_ to work, but it lacked tty' normal signal handling such as Ctrl-C (?), putting /dev/xvc0 solved the problem.
For whatever reason (?),
/dev/tty1 was working nicely as Xen guest console (
xm console ), but now you should use
/dev/xvc0.
That is:
guest# vim /etc/inittab ### check that following line is present:
x0:2345:respawn:/sbin/getty 38400 xvc0
guest# echo xvc0 >> /etc/securetty
2.d. [UPDATE] Debian 4.0 and network device naming
After upgrading guests from Debian 3.1 (Sarge) to 4.0 (Etch) a "nice touch" appeared: AFAICS Debian udev infrastructure tries to keep netdev naming "constant" based on device's MAC address (no, thanks |-[ ).
Given that Xen generates a non-constant MAC address each time it boots a guest, this makes
each Debian 4.0 guest boot to have an
increasing ethN device.
Two possible solutions:
- Fix MAC address in Xen guest vm config, you could use the following oneliner that uses the first 6 hexdigits from the md5 over the (unique) guestname. This 6 hexdigits are appended to the XenSource reserved MAC prefix 00:16:3E .
guest# echo -n guestname.FQDN | md5sum | sed -r 's/(..)(..)(..).*/00:16:3E:\1:\2:\3/'
... or ...
- Just disable the correspondent udev rules by renaming:
guest# mv /etc/udev/rules.d/{,.}z25_persistent-net.rules
... pheeuuu ... 'nuff written.
