Migrating XEN installation from fc5 to centos5

Scenario

• Deployment: several XEN guests running Debian 3.1 (Sarge) over a Fedora Core 5 (fc5) host. Host & guests installed with distro-provided packages.
  
• Goal: migrate host from fc5 to centos5 (held migration until centos5 got released)
  
• Difficulty: a lot. :-S
  

UPDATE 11-Jun-2007: See 2.c: /dev/xvc0 instead of /dev/console in guest
UPDATE 13-Jun-2007: See 2.d: oneliner for easy fixed guest MAC generation

Migration

It was faaaaar... more complex than we originally thought.

1. XEN host stuff

1.a. Just one kernel-xen package

FC5 came with two kernel-xen flavor: kernel-xen0 for the Dom0 guest and kernel-xenU for the other unprivileged guests.
Centos5 (and FC6) comes with just one kernel-xen package, this is quite annoying at first (this "mix" of true hardware drivers and virt. guest ones), but it starts to make sense once you ride the wave :)
You can see it with ("front" ones are for the guests , "back" for the host):

host# rpm -ql kernel-xen | egrep /xen

1.b. XEN guest kernel doesn't have the virtual block driver

That is: the guest will plainly PANIC if used without an initrd, so now you do have to make an explicit initrd.guest.img (whatever name you'd like) and add a ramdisk= option to the xen guest config file.
That is:

host# mkinitrd --preload=xenblk --preload=xennet -f -v /boot/initrd-2.6.18-8.1.4.el5xen.guest.img 2.6.18-8.1.4.el5xen
host# vim /etc/xen/xm-guest1 ### add: ramdisk=/boot/initrd.guest.img ### see below
host# restorecon -v /boot/initrd*

The last line (restorecon ... ) is needed because xend is _correctly_ running confined by "targeted" SELinux, and mkinitrd doesn't relabel the initrd file under /boot to allow xend access.
BTW, we prefer to have a "visible" and stable guest file configuration, so we did

host# cd /boot
host# ln -sf initrd-2.6.18-8.1.4.el5xen.guest.img initrd-guest.img
host# ln -sf vmlinuz-2.6.18-8.1.4.el5xen vmlinuz-guest

1.c. [UPDATE] that nasty "4gb seg fixup, process ..." (@host)

From XenFAQ and elsewhere:

host# echo 'hwcap 0 nosegneg' > /etc/ld.so.conf.d/libc6-xen.conf

host# ldconfig -v

2. XEN guest stuff

2.a. udevd inside guests

Debian 3.1 Sarge is installed on our guest images.
Now with Centos5 XEN we do need udevd running inside guests (to correctly setup /dev), that is:

guest# apt-get install udev ### installs udev and hotplug packages

2.b. the return of the "4gb seg fixup" (guest)

That comes from the way Xen uses the CPU segmentation; for newer distros it may be solved with echo 'hwcap 0 nosegneg' > /etc/ld.so.conf.d/libc6-xen.conf , but Debian 3.1 doesn't come with this feature, so we had to:

guest# mv /lib/tls /lib/tls.DISABLED

"Offline'ing" TLS glibc implementation solved the problem, beware that you'll need to redo this everytime libc6 package is upgraded (trivially solved by a rcS script).

2.c. getty /dev/tty1 -> getty /dev/xvc0

UPDATE: /dev/console _seemed_ to work, but it lacked tty' normal signal handling such as Ctrl-C (?), putting /dev/xvc0 solved the problem.

For whatever reason (?), /dev/tty1 was working nicely as Xen guest console (xm console ), but now you should use /dev/xvc0.
That is:

guest# vim /etc/inittab ### check that following line is present:

x0:2345:respawn:/sbin/getty 38400 xvc0

guest# echo xvc0 >> /etc/securetty

2.d. [UPDATE] Debian 4.0 and network device naming

After upgrading guests from Debian 3.1 (Sarge) to 4.0 (Etch) a "nice touch" appeared: AFAICS Debian udev infrastructure tries to keep netdev naming "constant" based on device's MAC address (no, thanks |-[ ).
Given that Xen generates a non-constant MAC address each time it boots a guest, this makes each Debian 4.0 guest boot to have an increasing ethN device.

Two possible solutions:

1. Fix MAC address in Xen guest vm config, you could use the following oneliner that uses the first 6 hexdigits from the md5 over the (unique) guestname. This 6 hexdigits are appended to the XenSource reserved MAC prefix 00:16:3E .
   
   guest# echo -n guestname.FQDN | md5sum | sed -r 's/(..)(..)(..).*/00:16:3E:\1:\2:\3/'
   
   ... or ...
   
2. Just disable the correspondent udev rules by renaming:
   
   guest# mv /etc/udev/rules.d/{,.}z25_persistent-net.rules

... pheeuuu ... 'nuff written.

Por fin entendí porqué SVN siempre apestó para mí.

... entonces ya puedo redimir mi pecado de no haber aprendido nunca SVN habiendo escapado directamente de CVS a GIT ;-)

Linux Torvalds no sólo habla sobre GIT, sino también sobre conceptos de SCM distribuídos & alike.
Imperdible.

Jugando con IPv6 (apto para todo IPv4-público ;)

Dale ... festejemos el día de Internet usando lo que es y será su pegamento futuro: IPv6.
Ingredientes: tan sólo 1 IP pública
Resultado: estemmm ... exactamente 1208925819614629174706176 (2^80 ;) direcciones IPv6, las cuales han estado ahí desde hace tiempo esperándote :-P
Para saborear más aún: end-to-end para las máquinas que quieras ... como en los buenos viejos tiempos, por ej. podrás hacer desde "afuera":
ssh -6 flamanteIPv6_de_esa_PC_que_no_tiene_IP_publica

El mecanismo se llama tunneling 6to4, está descripto por doquier, en particular yo escribí un mini artículo para el seminario de IPv6 que dimos en el 2005 en la UM.

Podés probar usando mi script: ipv6-setup6to4.sh , el cual sólo muestra los comandos necesarios (es decir: inofensivo :-), por ejemplo con una dir. IPv4= 65.1.2.3 da como salida:

bash$ ./ipv6-setup6to4.sh
IP4_ADDR=65.1.2.3
IP6TO4_PREF=2002:4101:203
#check you allow ipv6 encap: iptables -I INPUT -p 41 -d 65.1.2.3
ip tunnel add tun6to4 mode sit remote any local 65.1.2.3 ttl 64
ip addr flush dev tun6to4 2>/dev/null
ip link set dev tun6to4 up
ip addr add 2002:4101:203::1/16 dev tun6to4
ip route add ::/96 dev tun6to4
ip route add 2000::/3 via ::192.88.99.1 dev tun6to4 metric 1
#you may do something like:  ip -6 addr add 2002:4101:203:0001::1/64 dev eth0

#NOTHING done, use me as: ./ipv6/ipv6-setup6to4.sh   |sudo sh -x

Que lo disfrutes!

Juguemos en el bosque ... mientras los lobos no están :-S

Hoy 17 de mayo se festeja el día de "Internet" [1], veremos cuántos años más podremos hacerlo ... al menos con "la" Internet que conocemos y disfrutamos hoy.

Por si no lo sabías, los elefantes de siempre (media/content corps & alike) quieren "otra" Internet, más acorde a sus mecanismos feudales de poder [2].

Seguramente ya nos caerá el regalito a estas pampas, por ahora lo mejor que podemos hacer es tomar y hacer tomar conciencia.

Salud! y que sea por muuucho tiempo más ...

[1] http://www.itu.int/wisd/2007/index.html
[2] http://www.savetheinternet.com/

[en] From my recruiter: Job opportunities @Google

My name is Sripriya. I am part of the recruiting team for Google.com (SRE) Group.

We currently have positions available at Google that may be a good match for you. If you are open to exploring these opportunities further, please send an updated version of your resume in Word, HTML, or PDF to JuanJo's email with "Subject: job opportunity @Google: " as soon as possible

***We have multiple openings located in Dublin - Ireland and Zurich - Switzerland)***

All positions involve working in our infrastructure team, known as Google.com Engineering. Our Google.com engineers hold the beating heart of Google and are very well respected. They are responsible for keeping the google.com website up and running as well as building new automation infrastructure. We are seeking extraordinary Developers, UNIX (Linux) System Administrators, and Managers/Directors to add to our exciting team and growing organization.

If this prospect is of interest to you, please get in touch with me as I will like to share your resume with the hiring engineers to see if there is match. You may also know engineers with this skill set who are looking for fresh challenges; I will appreciate it if you will pass my contact information to them.

Please note that in addition to these positions, we also have openings as a Systems Deployment Engineer, Launch Coordination Engineer, and Engineering Project Manager.

For more information, go to:

Zurich positions:
http://www.google.ch/support/jobs/bin/topic.py?dep_id=1058&loc_id=1115

Dublin - Ireland positions:
http://www.google.ie/support/jobs/bin/topic.py?dep_id=1058&loc_id=1110

Thank you and hope to hear from you soon.

Thanks & Best Regards,
Sripriya
--
Sripriya Sampath
Technical Recruiter
Google Inc

[en] preparing yourself for the technical interview (cont'd)

Cool ... so you're already enjoying answering those tricky tech and puzzle questions from my previous post.

Time for more formal lectures, here is a list of helpful textbooks you should study before the technical interview (again, based on my profile, see my previous post).

• Networking
• Internet Core Protocols: The Definitive Guide
  E Hall - 2000 - O'Reilly & Associates.
  Great book on TCP/IP network and transport protocols, very comprehensive yet easy reading (BTW do trash the CD contents, just use wireshark )
  
  
  
• POSIX & BSD sockets API
• Our beloved APUE: Advanced Programming in the Unix Environment
  R. Stevens, S. Rago - 2004 - A. Wesley.
  From our great all-time teacher, Richard Stevens, impressive as all his books.
  
  
  
• Unix Network Programming
  WR Stevens, B Fenner, AM Rudoff - 2004 - A. Wesley.
  THE reference book for IPC & network programming
  
  
  
• POSIX.4: Programming for the Real World
  Bill 0. Gallmeister - 1995 - O'Reilly.
  Great book about multithreading POSIX API
  
  
  


• Linux sysadmin and alike: there are actually a LOT of books, but nothing can compare with a good expertise ;), so ...
  
• LPI Linux Certification in a Nutshell, 2nd Edition
  J. Dean, B. Gomes Pessanha ... - 2006 - O'Reilly.
  Nice book for autotesting your already learned sysadm knowledge & skills
  
  
• Unix/Linux internals
• Understanding the Linux Kernel, 3rd Edition
  D. P. Bovet, M. Cesati - 2005 - O'Reilly
  Too detailed, try to "grep" for concepts and graphs
  
  

'nuff written ... now enjoy the cool reading 8^)

[en] preparing yourself for the technical interview

So you've followed my suggestion, made your shinning resumè ... so the call from your future job recruiters may come in any moment.

Time to get() ready for the technical interview!

Here are some useful links I've collected while preparing my interviews, obviously considering my profile: networking, Linux/UN*X sysadm & trou, low level programming in C and shell scripting, kernel internals.

• Cool univ. courses podcasts, they even do help us non-English speakers to exercise our brain translator neurons :-):
• MIT Open CourseWare: 6.046J / 18.410J Introduction to Algorithms (SMA 5503), Fall 2005
  http://ocw.mit.edu/OcwWeb/Electrical-Engineering-and-Computer-Science/6-046JFall-2005/CourseHome/index.htm
  

• UC Berkley: CS 162 Operating Systems and System Programming
  http://webcast.berkeley.edu/course_details.php?seriesid=1906978416

• lang.shell:
• The OMNI cool Advanced Bash-Scripting Guide
  http://tldp.org/LDP/abs/html/
  
  
• lang.C:
• Advanced Test in C: The 0x10 Best Questions for C Programmers
  http://www.programmersheaven.com/articles/pathak/article2.htm
  
  
• Tech questions, puzzles and such (lot of FUN, take it!)


• Technical Interview Resources
  http://technical-interview.com/default.aspx


• UPDATED: Another similar site (thanks Nikita for the suggestion)
  http://technical-interviews.com/

Enjoy!
I assure you will ... else ask the mirror if you could be called a true geek ;-)