Migrating XEN installation from fc5 to centos5

Scenario

• Deployment: several XEN guests running Debian 3.1 (Sarge) over a Fedora Core 5 (fc5) host. Host & guests installed with distro-provided packages.
  
• Goal: migrate host from fc5 to centos5 (held migration until centos5 got released)
  
• Difficulty: a lot. :-S
  

UPDATE 11-Jun-2007: See 2.c: /dev/xvc0 instead of /dev/console in guest
UPDATE 13-Jun-2007: See 2.d: oneliner for easy fixed guest MAC generation

Migration

It was faaaaar... more complex than we originally thought.

1. XEN host stuff

1.a. Just one kernel-xen package

FC5 came with two kernel-xen flavor: kernel-xen0 for the Dom0 guest and kernel-xenU for the other unprivileged guests.
Centos5 (and FC6) comes with just one kernel-xen package, this is quite annoying at first (this "mix" of true hardware drivers and virt. guest ones), but it starts to make sense once you ride the wave :)
You can see it with ("front" ones are for the guests , "back" for the host):

host# rpm -ql kernel-xen | egrep /xen

1.b. XEN guest kernel doesn't have the virtual block driver

That is: the guest will plainly PANIC if used without an initrd, so now you do have to make an explicit initrd.guest.img (whatever name you'd like) and add a ramdisk= option to the xen guest config file.
That is:

host# mkinitrd --preload=xenblk --preload=xennet -f -v /boot/initrd-2.6.18-8.1.4.el5xen.guest.img 2.6.18-8.1.4.el5xen
host# vim /etc/xen/xm-guest1 ### add: ramdisk=/boot/initrd.guest.img ### see below
host# restorecon -v /boot/initrd*

The last line (restorecon ... ) is needed because xend is _correctly_ running confined by "targeted" SELinux, and mkinitrd doesn't relabel the initrd file under /boot to allow xend access.
BTW, we prefer to have a "visible" and stable guest file configuration, so we did

host# cd /boot
host# ln -sf initrd-2.6.18-8.1.4.el5xen.guest.img initrd-guest.img
host# ln -sf vmlinuz-2.6.18-8.1.4.el5xen vmlinuz-guest

1.c. [UPDATE] that nasty "4gb seg fixup, process ..." (@host)

From XenFAQ and elsewhere:

host# echo 'hwcap 0 nosegneg' > /etc/ld.so.conf.d/libc6-xen.conf

host# ldconfig -v

2. XEN guest stuff

2.a. udevd inside guests

Debian 3.1 Sarge is installed on our guest images.
Now with Centos5 XEN we do need udevd running inside guests (to correctly setup /dev), that is:

guest# apt-get install udev ### installs udev and hotplug packages

2.b. the return of the "4gb seg fixup" (guest)

That comes from the way Xen uses the CPU segmentation; for newer distros it may be solved with echo 'hwcap 0 nosegneg' > /etc/ld.so.conf.d/libc6-xen.conf , but Debian 3.1 doesn't come with this feature, so we had to:

guest# mv /lib/tls /lib/tls.DISABLED

"Offline'ing" TLS glibc implementation solved the problem, beware that you'll need to redo this everytime libc6 package is upgraded (trivially solved by a rcS script).

2.c. getty /dev/tty1 -> getty /dev/xvc0

UPDATE: /dev/console _seemed_ to work, but it lacked tty' normal signal handling such as Ctrl-C (?), putting /dev/xvc0 solved the problem.

For whatever reason (?), /dev/tty1 was working nicely as Xen guest console (xm console ), but now you should use /dev/xvc0.
That is:

guest# vim /etc/inittab ### check that following line is present:

x0:2345:respawn:/sbin/getty 38400 xvc0

guest# echo xvc0 >> /etc/securetty

2.d. [UPDATE] Debian 4.0 and network device naming

After upgrading guests from Debian 3.1 (Sarge) to 4.0 (Etch) a "nice touch" appeared: AFAICS Debian udev infrastructure tries to keep netdev naming "constant" based on device's MAC address (no, thanks |-[ ).
Given that Xen generates a non-constant MAC address each time it boots a guest, this makes each Debian 4.0 guest boot to have an increasing ethN device.

Two possible solutions:

1. Fix MAC address in Xen guest vm config, you could use the following oneliner that uses the first 6 hexdigits from the md5 over the (unique) guestname. This 6 hexdigits are appended to the XenSource reserved MAC prefix 00:16:3E .
   
   guest# echo -n guestname.FQDN | md5sum | sed -r 's/(..)(..)(..).*/00:16:3E:\1:\2:\3/'
   
   ... or ...
   
2. Just disable the correspondent udev rules by renaming:
   
   guest# mv /etc/udev/rules.d/{,.}z25_persistent-net.rules

... pheeuuu ... 'nuff written.

[en] From my recruiter: Job opportunities @Google

My name is Sripriya. I am part of the recruiting team for Google.com (SRE) Group.

We currently have positions available at Google that may be a good match for you. If you are open to exploring these opportunities further, please send an updated version of your resume in Word, HTML, or PDF to JuanJo's email with "Subject: job opportunity @Google: " as soon as possible

***We have multiple openings located in Dublin - Ireland and Zurich - Switzerland)***

All positions involve working in our infrastructure team, known as Google.com Engineering. Our Google.com engineers hold the beating heart of Google and are very well respected. They are responsible for keeping the google.com website up and running as well as building new automation infrastructure. We are seeking extraordinary Developers, UNIX (Linux) System Administrators, and Managers/Directors to add to our exciting team and growing organization.

If this prospect is of interest to you, please get in touch with me as I will like to share your resume with the hiring engineers to see if there is match. You may also know engineers with this skill set who are looking for fresh challenges; I will appreciate it if you will pass my contact information to them.

Please note that in addition to these positions, we also have openings as a Systems Deployment Engineer, Launch Coordination Engineer, and Engineering Project Manager.

For more information, go to:

Zurich positions:
http://www.google.ch/support/jobs/bin/topic.py?dep_id=1058&loc_id=1115

Dublin - Ireland positions:
http://www.google.ie/support/jobs/bin/topic.py?dep_id=1058&loc_id=1110

Thank you and hope to hear from you soon.

Thanks & Best Regards,
Sripriya
--
Sripriya Sampath
Technical Recruiter
Google Inc

[en] preparing yourself for the technical interview (cont'd)

Cool ... so you're already enjoying answering those tricky tech and puzzle questions from my previous post.

Time for more formal lectures, here is a list of helpful textbooks you should study before the technical interview (again, based on my profile, see my previous post).

• Networking
• Internet Core Protocols: The Definitive Guide
  E Hall - 2000 - O'Reilly & Associates.
  Great book on TCP/IP network and transport protocols, very comprehensive yet easy reading (BTW do trash the CD contents, just use wireshark )
  
  
  
• POSIX & BSD sockets API
• Our beloved APUE: Advanced Programming in the Unix Environment
  R. Stevens, S. Rago - 2004 - A. Wesley.
  From our great all-time teacher, Richard Stevens, impressive as all his books.
  
  
  
• Unix Network Programming
  WR Stevens, B Fenner, AM Rudoff - 2004 - A. Wesley.
  THE reference book for IPC & network programming
  
  
  
• POSIX.4: Programming for the Real World
  Bill 0. Gallmeister - 1995 - O'Reilly.
  Great book about multithreading POSIX API
  
  
  


• Linux sysadmin and alike: there are actually a LOT of books, but nothing can compare with a good expertise ;), so ...
  
• LPI Linux Certification in a Nutshell, 2nd Edition
  J. Dean, B. Gomes Pessanha ... - 2006 - O'Reilly.
  Nice book for autotesting your already learned sysadm knowledge & skills
  
  
• Unix/Linux internals
• Understanding the Linux Kernel, 3rd Edition
  D. P. Bovet, M. Cesati - 2005 - O'Reilly
  Too detailed, try to "grep" for concepts and graphs
  
  

'nuff written ... now enjoy the cool reading 8^)

[en] preparing yourself for the technical interview

So you've followed my suggestion, made your shinning resumè ... so the call from your future job recruiters may come in any moment.

Time to get() ready for the technical interview!

Here are some useful links I've collected while preparing my interviews, obviously considering my profile: networking, Linux/UN*X sysadm & trou, low level programming in C and shell scripting, kernel internals.

• Cool univ. courses podcasts, they even do help us non-English speakers to exercise our brain translator neurons :-):
• MIT Open CourseWare: 6.046J / 18.410J Introduction to Algorithms (SMA 5503), Fall 2005
  http://ocw.mit.edu/OcwWeb/Electrical-Engineering-and-Computer-Science/6-046JFall-2005/CourseHome/index.htm
  

• UC Berkley: CS 162 Operating Systems and System Programming
  http://webcast.berkeley.edu/course_details.php?seriesid=1906978416

• lang.shell:
• The OMNI cool Advanced Bash-Scripting Guide
  http://tldp.org/LDP/abs/html/
  
  
• lang.C:
• Advanced Test in C: The 0x10 Best Questions for C Programmers
  http://www.programmersheaven.com/articles/pathak/article2.htm
  
  
• Tech questions, puzzles and such (lot of FUN, take it!)


• Technical Interview Resources
  http://technical-interview.com/default.aspx


• UPDATED: Another similar site (thanks Nikita for the suggestion)
  http://technical-interviews.com/

Enjoy!
I assure you will ... else ask the mirror if you could be called a true geek ;-)

[en] your CV is useless

You (may) use to maintain a pretty precise Curriculum Vitae (CV) with every bit of your professional activities.
Bad news: it's useless (well, to be honest, it's perfect for filling paper files =)

At least this happened to me while applying for working at google.com , I needed to rush and write a "Resumè" , which is a sort of resumed-CV but focused on your profile, strengths, etc.

These are some useful links I've collected that helped me writing the "Resumè" (of course you could just search for "resume tips" and such, but there are plenty of results and much noise around):

1. http://www.resume-resource.com/examples.html
2. http://www.resume-resource.com/extec7.html
   
3. http://www.alec.co.uk/how-to-write-a-resume/how-to-make-a-resume.htm
4. http://www.how-to-write-a-resume.org/action_verbs.htm
5. http://lifehacker.com/5866630/how-can-i-make-sure-my-resume-gets-past-resume-robots-and-into-a-humans-hand

Don't rush when forced, do it now (it happend to be of some fun ;)

BTW Here is my Resumè: https://sites.google.com/site/juanjosec/curriculum/Resume-JuanJoseCiarlante-en.2011oct12.pdf

[UPDATED 2010-05-27: new resume url]
[UPDATED 2011-10-12: updated my resume, add link#5. ]

[en] powerful arrays in bash

Doing some experiments with advanced bash array features. I created a bash implementation of insertion sort algorithm.

UPDATE: This example is already part of the great Advanced Shell Scripting Guide :)

Enjoy :-D

#!/bin/bash
# insertion-sort.bash.sh: Insertion sort implementation in bash
#                         Heavy use of bash array features: slicing, merging, etc
# URL: http://www.lugmen.org.ar/~jjo/jjotip/insertion-sort.bash.d/insertion-sort.bash.sh
#
# Author: JuanJo Ciarlante  jjo \O/ irrigacion gov ar
# License: GPLv2
#
# Test with:   ./insertion-sort.bash.sh -t
#

: ${DEBUG:=1}  # debug, override with:  DEBUG=1 ./scriptname ..

# Global array: "list"
typeset -a list
# Load whitespace separated numbers from just stdin 1st line
if [ "$1" = "-t" ];then
     read -a list < <(od -An -w32 -t u2 /dev/urandom )
else
     read -a list
fi
numelem=${#list[*]}

# Shows the list, marking the element whose index es $1 by surrounding it with
# the two chars passed as $2; whole line prefixed with $3
showlist() { echo "$3"${list[@]:0:$1} ${2:0:1}${list[$1]}${2:1:1} ${list[@]:$1+1}; }

# loop "pivot" from 2nd elem, to end of list
for((i=1;i<numelem;i++))do
     ((DEBUG))&&showlist i "[]" " "
     # From current "pivot", back to 1st elem
     for((j=i;j;j--))do
             # search for the 1st elem less than current "pivot" ...
             [[ "${list[j-1]}" -le "${list[i]}" ]] && break
     done
     ((i==j)) && continue ## no insertion was needed for this element
     # ... move list[i] (pivot) to the left of list[j]:
     list=(${list[@]:0:j} ${list[i]} ${list[j]} ${list[@]:j+1:i-(j+1)} ${list[@]:i+1})
     #         {0,j-1}       {i}       {j}        {j+1,i-1}              {i+1,last}
     ((DEBUG))&amp;amp;amp;&showlist j "<>" "*"
done
echo $'Result:\n'${list[@]}

[en] Throughput tests WRTSL54GS v1.1 vs WRT54GS v3.0

Throughput tests WRTSL54GS v1.1 vs WRT54GS v3.0

The goal of this test is to measure and compare the NAT'ing/routing/switching behavior of these two WRT54G's [1] running OpenWRT Whiterussian RC6 [5]:

model	s/n	CPU	flash/RAM	eth switch
WRTSL54GS v1.1 [2]	CJK11xxxxxxx	Broadcomm 4704rev9 266Mhz	8/32MB	inCPU
WRT54GS v3.0 [3]	CGN5xxxxxxx	Broadcomm 5352 200Mhz	8/32MB	BCM5325

IMPORTANT: nbd ( openwrt developer) pointed a problem with "entropy gathering which fills /dev/random from wifi" in WR rc6 and suggested me trying a newer build (09test); I'll try it an repost the results (specially the wifi ones)
--jjo; Jan 10, 2007.

Testbed, methodology

• Topology



                       --------> iperf test connection------>
 WIFI-NATed    [CLIENT]+)) wifi (( +[ WRT ]+=== wan ===+[SERVER]   (iptables MASQUERADE'd)
 LAN-NATed     [CLIENT]+=== lan ===+[ WRT ]+=== wan ===+[SERVER]   (iptables MASQUERADE'd)
 LAN-routed    [CLIENT]+=== lan ===+[ WRT ]+=== wan ===+[SERVER]   (plain routing)
 LAN-switched  [CLIENT]+=== lan ===+[ WRT ]+=== lan ===+[SERVER]   (hardware switching)
           
 "lan": any LAN (1of4) port
 "wan": WAN port
 "wifi": 802.11g link at 54Mbps (aprox 2mts distance between antennas)
 iptables MASQUERADE'd: default whiterussian setup (masquerade both wifi,lan to wan)
 plain routing: accomplished with iptables full "flushing" + server route to client via WRT



• Hardware
  
  
  
  • client: IBM Thinkpad T42, lan: e1000 (10/100/1000), wifi: ipw2200
    
  • server: generic PC sempron 2200+, lan: 8139too
    
  • routers: (stated above)
    
  
  


• Software
  
  • client, server: iperf [4] (-c, -s) on linux-2.6, glibc-2.3+.
  • routers: OpenWRT WhiteRussian RC6 [5].

Results

Throughput values in megabits per second taken from iperf's output. Please note that measured rates by iperf correspond to _application_, ie: they don't consider protocol headers, etc.

CPU usage % shown by top for ksoft_IRQd from a ssh logged session, this "top" session was not considered for throughput values (separately taken) to avoid "measuring load influence": 100% CPU usage was aprox= 90% ksoft_IRQd + 8% dropbear + 2% top.

Throughput (CPU usage %)

model	WIFI-NATed	LAN-NATed	LAN-routed	LAN-switched
WRTSL54GS	23.5mbit/s (65%)	47.5mbit/s (90%+)	55.8mbit/s (90%+)	93.7mbit/s (0%)
WRT54GS	10.5mbit/s (14%)	34.2mbit/s (90%+)	40.4mbit/s (90%+)	93.8mbit/s (0%)

Analisys

WRTSL54GS

WIFI-NATed at 23.5 mbits (802.11g @54Mbps) with ~65% CPU usage denotes "reasonable" CPU availability for this 802.11g saturated link.

LAN-NATed at 47.5 mbits shows the CPU bottleneck when routing+MASQUERADING, considering that ip_conntrack table has just a few entries this is a VERY optimistic max.

LAN-routed at 55.8 mbits is pretty poor, I think that ip_conntrack processing is ALSO affecting this scenario (if loaded it gets called even for non-nated streams); it would be veryVERY useful to have ip_conntrack optionally loaded by openwrt (currently static).

LAN-switched at 93.8 mbits is expectable given that hardware only switching is involved.

WRT54GS

LAN-NATed and LAN-routed values give about 75% when compared to its newer cousin, 75% is roughly 200/266, respective CPU clocks [MHz] (this confirms CPU saturation).

WIFI-NATed at 10.5 mbits (dbl-confirmed 802.11g @54Mbps) is a bad smelling frog, I double checked openwrt setup... nothing strange found; about 1mo ago I was running RC5 getting ~20mbits+ rates (informal, from rsync/ftp rates), so it maybe something with RC6 wifi in this hardware, dunno(?).

As expected, LAN-switched gave same results.

Conclusions

Both WRT's seem reasonable for 802.11g @54Mbps even when NAT'ing traffic (common usage), obviously depending on traffic patterns, how many clients, etc ... given the constrained RAM (and CPU) for NAT+conntracking ... NOT tested here.

On the contrast, their CPU is not powerful enough to saturate _both_ (LAN+WAN) 100mbps links simultaneously, even when "only" routing (55mbit/s max on WRTSL54GS); this could be enhanced by having ip_conntrack optionally (not)loaded by openwrt.

Out of band disclaimer

Ok ... I know, you agree: blogger sucks for writing technical reports .. sheesshhhh :-P

References

[1]http://en.wikipedia.org/wiki/WRT54G
[2]http://wiki.openwrt.org/OpenWrtDocs/Hardware/Linksys/WRTSL54GS
[3]http://wiki.openwrt.org/OpenWrtDocs/Hardware/Linksys/WRT54GS
[4]Iperf bandwidth measurement tool: http://dast.nlanr.net/Projects/Iperf/
[5]http://downloads.openwrt.org/whiterussian/rc6/


This thread at openwrt forums talks about openwrt experiences on Linksys WRTSL54GS model, and was the starting point to encourage loading openwrt on WRTSL54GS v1.1 model.