Xen 3.0.2 Setup
From Mitopia
Xen 3.0.2 Setup on Fedora Core 5
Contents |
Install Xen
First we need to install Xen
Decide on installation Method
RPM or Source... pick your poison.
Install Xen via RPM
Installing via an rpm is quick and easy, but you don't have as much control over what you can do with xen ( I haven't been able to get NAT to work yet with an rpm-installed xen setup). Only one command is needed to install xen from rpm's:
# yum install xen kernel-xen0 kernel-xenU
There, you now have Xen installed. Please skip down to the Final dom0 Check-in's portion!
Install Xen via Source
Prepare System for Prereqs
This is a list of all prerequisites that are needed for installing Xen and QEMU
# yum install python-twisted SDL texmaker curl-devel bridge-utils iproute sysfsutils zlib zlib-devel python-devel ncurses ncurses-devel kernel-devel transfig
Download Xen tarball
Before anything else, we need to download the Xen source.
# mkdir /dload # mkdir /dload/xen # cd /dload/xen # wget http://www.cl.cam.ac.uk/Research/SRG/netos/xen/downloads/xen-3.0.2-src.tgz # tar -zxvf xen-3.0.2-src.tgz
Compile Xen
Now it's time to actually install Xen. During the install, it will automatically download the apropriate kernel source, so depending on your internet connection, it might take a while.
# cd xen-3.0.2 # make world # make install # ./install.sh
Setup Dom0
Now we need to setup the host domain First, lets do some system setups and tweaks.
- Disable TLS: If TLS is not disabled, an emulation mode is used within Xen which reduces performance substantially.
- (Note: Many, if not most, articles tell you to disable TLS via the command 'mv /lib/tls /lib/tls.disabled' however I've been informed that doing so isn't a good idea, as TLS would simply be reinstalled on the next TLS update. Using the command below actually tells libc to disable TLS which is a much better method. Xen may still complain that it's enabled, but that is due to a bug. If the complaint bothers you, you can move the dir as well.)
# echo "hwcap 0 nosegneg" >> /etc/ld.so.conf
- Make sure xend is set to run on startup
# /sbin/chkconfig --level 345 xend on
Compile dom0 Kernel
The only thing that is really left to be done is compile a kernel. First we'll start with copying the .config file from your old config into the xen config. (In my example I used my old kernel as I knew nothing much had changed since I upgraded it, so that it would be the closest version as possible to the xen kernel. In reality though, you can use any newer kernel's .config and the 'make ARCH=xen oldconfig' command below will remove any incompatibilities.)
- Make sure you change the directory to your existing kernel's source directory.
cd linux-2.6.11-xen0 cp /usr/src/kernels/2.6.16-1.1369_FC4-smp-i686/.config . make oldconfig
You just imported your existing kernel's config into the xen kernel config. Now run lspci to find out what type of hardware options you need, and then make sure that they are enabled by your xen kernel. After you do that, you're all set to compile!
lspci make menuconfig make <- this command compiles the kernel, it will take a while! make modules_install
That's it, you're compiled! Now just copy this kernel into your /boot (Note, the xen install puts a kernel in the /boot dir so you will have to overwrite what's there. Make a backup if you want, but in my experience the kernel it puts there wont work anyways, so there's no need to back it up).
cp vmlinuz /boot/vmlinuz-2.6.16-xen0
Create an initrd
Make an initrd with mkinitrd.
# mkinitrd /boot/initrd-xen-3.0.2.img 2.6.16-xen
If mkinitrd gives an error like this: "No module aic7xxx found for kernel 2.6.16-xen, aborting. " then it can't find a specific module (from /etc/modules.conf). If you know that it is compiled into the kernel directly, you can tell mkinitrd that, and it wont complain. So for the aic7xxx module, use this command instead:
# mkinitrd --builtin=aic7xxx /boot/initrd-xen-3.0.2.img 2.6.16-xen
The same initrd can be used for the VM by adding the following to its config file.
ramdisk = "/boot/initrd-xen-3.0.2.img"
Configure Grub
You now need to setup Grub so your host PC can boot to the xen kernel. Here is the xen portion of my grub.
title Xen 3.0.2 / XenLinux 2.6.16
root (hd0,0)
kernel /xen-3.0.gz dom0_mem=65536
module /vmlinuz-2.6.16-xen ro root=/dev/md2
module /initrd-xen-3.0.2.img
- Note to make sure that the initrd and kernel points to the correct file names
Final dom0 Check-in's
Check that the following directories exist on your system: /var/lib/xenstored/ and /var/run/xenstored/. If they have not automatically been created, you must create them manually (as a result of bug 167121):
# mkdir -p /var/lib/xenstored # mkdir -p /var/run/xenstored
Now, if all went well, you should be able to reboot your computer and come back up booted into the Xen dom0 kernel! To check to see if everything is running smoothly, run this command:
# /usr/sbin/xm list
Now we move on to setting up a guest domain!
Setup DomainU
Now we need to setup the guest domain
Compile domU Kernel
Almost identical to the compiling of the dom0 kernel, so here we go:
cd linux-2.6.11-xenU make ARCH=xen menuconfig make ARCH=xen make ARCH=xen modules_install cp vmlinuz /boot/vmlinuz-2.6.11-xenU
Install QEMU
To setup the guest domain (domU) we will be using QEMU. It's slow, but it allows for a clean install of the guest OS without relying on the host OS for anything.
- Download QEMU from their website
# cd /dload # wget http://fabrice.bellard.free.fr/qemu/qemu-0.8.0-i386.tar.gz # cd / # tar -zxvf /dload/qemu-0.8.0-i386.tar.gz
Create GuestOS Image
Now we need to create our image file and start the GuestOS install! In this example I am making a 10gb image file, you can also replace the G with M to specify the size in terms of MB.
# mkdir /guests # cd /guests # qemu-img create image-name.img 10G
Install GuestOS
And now we start the install! 128 is the amount of RAM we are designating, and of course we are booting it to an iso CD image.
- Note, to help when converting away from QEMU make sure that all data is in a single / partition, and that partition is in the ext3 format.
# qemu -m 128 -hda /guests/image-name.img -cdrom /path/to/iso -boot d
- Note, if you're doing this via ssh and get a message about SDL not initializing, then you need to install xauth and then ssh in with the -X option (not the -x option).
Follow normal installation procedures for installing your OS. To change the installation CD if needed, go into the QEMU Monitor by pressing Ctrl+Alt+2. You can then change your CD like so:
(qemu) eject cdrom (qemu) change cdrom filename.iso
or
(qemu) eject cdrom (qemu) change cdrom d:/tmp/yourcdrom.iso
You can see a status of block device.
(qemu) info block
Theoretically this works, and under the online docs this works, however I have never been able to successfully change from one iso image to another. It will always unmount the original image, but it will not mount the new image. What I have done in the past, is instead of mounting a new image, to mount the cdrom drive itself.
(qemu) eject cdrom (qemu) change cdrom /dev/cdrom
Convert QEMU Image to Xen Image
Prepare Images for Conversion
Run fdisk to find out your partition number
# fdisk -lu /path/to/img.img
Look at the output and find out which partition number it is that has your data. In my current qemu image it is partition 2, so that is what I'm going to use in this example.
- Make a couple directories to mount the images.
- Mount the QEMU image to loop1
- In order to do this, we need a little bit of pre-calculations. Looking at the fdisk output from above, take the value of the "Start" sector and multiply it by 512. Use this in place of ##### in the command below.
- Create a new sparse image file and format it (The seek number is how many MB to make the file. It will be sparse, so feel free to be liberal.)
# dd if=/dev/zero of=/path/to/img.img bs=1M count=1 seek=1024 # /sbin/mke2fs -F -j /path/to/img.img
- Mount the sparse image
# mount -o loop /path/to/sparse-image /path/to/loop2
- Copy everything over
# cp -ra /path/to/loop1 /path/to/loop2
# mkdir /mnt/loop1; mkdir /mnt/loop2
Optimize new image to work with Xen
Ok! Now that the xen image now has the OS install, it's time to optimize it so it will run properly.
- Create /dev/console and /dev/null nodes (Note, do NOT use /dev as that is different!)
# cd /mnt/loop2 # mknod dev/console c 5 1 # mknod dev/null c 1 3
- Edit etc/fstab to look like this:
# This file is edited by fstab-sync - see 'man fstab-sync' for details /dev/sda1 / ext3 defaults 1 1 /dev/devpts /dev/pts devpts gid=5,mode=620 0 0 /dev/shm /dev/shm tmpfs defaults 0 0 /dev/proc /proc proc defaults 0 0 /dev/sys /sys sysfs defaults 0 0
- Disable TLS
# echo "hwcap 0 nosegneg" >> etc/ld.so.conf
- Edit etc/inittab to look like this:
# # inittab This file describes how the INIT process should set up # the system in a certain run-level. # # Author: Miquel van Smoorenburg, <miquels@drinkel.nl.mugnet.org> # Modified for RHS Linux by Marc Ewing and Donnie Barnes # # Default runlevel. The runlevels used by RHS are: # 0 - halt (Do NOT set initdefault to this) # 1 - Single user mode # 2 - Multiuser, without NFS (The same as 3, if you do not have networking) # 3 - Full multiuser mode # 4 - unused # 5 - X11 # 6 - reboot (Do NOT set initdefault to this) # id:3:initdefault: # System initialization. si::sysinit:/etc/rc.d/rc.sysinit l0:0:wait:/etc/rc.d/rc 0 l1:1:wait:/etc/rc.d/rc 1 l2:2:wait:/etc/rc.d/rc 2 l3:3:wait:/etc/rc.d/rc 3 l4:4:wait:/etc/rc.d/rc 4 l5:5:wait:/etc/rc.d/rc 5 l6:6:wait:/etc/rc.d/rc 6 # Trap CTRL-ALT-DELETE ca::ctrlaltdel:/sbin/shutdown -t3 -r now # When our UPS tells us power has failed, assume we have a few minutes # of power left. Schedule a shutdown for 2 minutes from now. # This does, of course, assume you have powerd installed and your # UPS connected and working correctly. pf::powerfail:/sbin/shutdown -f -h +2 "Power Failure; System Shutting Down" # If power was restored before the shutdown kicked in, cancel it. pr:12345:powerokwait:/sbin/shutdown -c "Power Restored; Shutdown Cancelled" # Run gettys in standard runlevels 1:2345:respawn:/sbin/mingetty tty1 # Run xdm in runlevel 5 x:5:once:/etc/X11/prefdm -nodaemon
- Copy Modules from host
# cp -ra /lib/modules/x.y.z-blah lib/modules/
And finally, last but certainly not least, just in case you used the Fedora RPM's to install instead of building it from source, you have to add this to the /etc/modprobe.conf file in your domU:
alias eth0 xennet
(For some reason the Fedora guys decided to make xennet a module as opposed to compiled in, so you need to specify the module be loaded. I came accross this problem once, and I never did figure it out until I found it out on the mailing list.)
And we should be done preparing the image, just unmount it and move on to the next step!
# cd /mnt # umount loop1 # umount loop2
Create GuestOS Swap Partition
Ok, so just like the OS runs in a file-based partition, I'm going to create a file-based swap file partition as well. Here we're going to create and format it as swap. (again, the seek is in MB)
# dd if=/dev/zero of=/path/to/img.swap bs=1M count=1 seek=512 # mkswap /path/to/img.swap
Setup Xen Configuration File
Now time to make the Xen config file for your domU. One thing to point out, the name of the domU does not have to match the name of the config file! Just make a file to put your config in (in this example I will use itworks.xen.cfg, and then make it look something like this (I have added-in comment lines to help explain, obviously you don't have to include them if you don't need to :
# Your domain's name name = "it-works" # Kernel to use kernel = "/boot/vmlinuz-2.6.16-xen" ramdisk = "/boot/initrd-xen-3.0.2.img" # Memory in megabytes memory = 416 # Root device root = "/dev/sda1 ro" # Disk files disk = ['file:/path/to/img.img,sda1,w', 'file:/path/to/swap.swap,sda2,w'] # This 'extra' line will specify the runlevel for the domU to start in extra = "3" # Network Configuration # If you do not set the MAC variable, xend will automatically generate a random MAC address # from the range 00:16:3E:xx:xx:xx #vif = [ ] vif=[ 'ip=10.0.0.2' ] dhcp="off" netmask="255.255.0.0" gateway="10.0.0.254" hostname = "mail"
If you want to add more options, you can. They are listed in the xen manual
Startup Your Domain
Starting it up
Ok, so now you have your Xen setup pretty much complete! All we need to do now is start it up and make sure it runs properly! After that, it's just cosmetic things...
To start your Xen domU you have to do something that you would think you've just finished... you need to 'create' it. In functionality terms I would think it should be start, but oh well. Here's what we do. If you just want to start the VM and not see it's console output right now, take out the -c from this command.
- Notice that this is the config file, not the domU's name
xm create -c /path/to/itworks.xen.cfg
Connecting to the console
If things worked as planned, you should now have a message saying it started properly, but you have no idea what's going on in the virtual pc right now, so lets console in and watch it boot.
- Notice that this is the domU NAME (not config file)
xm console it-works
And now you should see the output of the virtual pc. In Xen 3.0 you can add both of these into a single command if you want, and then you can both start it and jump straight into the console. Until Xen3.0.2 comes out though, I can't use it for my needs.
Ok, so, now you have it started and are working good! In order to disconnect from that console and return to your normal console, press Ctrl + ] and you will hop back into the dom0 console.
Setting it for Auto-Startup
Once you get the domain booting up the way it's supposed to, we want to make it startup automatically. To do this, the config file has to be put into the /etc/xen/auto dir and a service set to autostartup. I started out copying the config file in there, but then I noticed that I always had to edit both if I ever wanted to edit it, so instead lets create a sym-link to the one in the /etc/xen dir.
# cd /etc/xen/auto # ln -s ../itworks.xen.cfg # /sbin/chkconfig --level 345 xendomains on # /sbin/service xendomains start
There, now this domU be started every time the PC is started.
Custom Networking
NAT
Ok so, there are times when the default bridged networking is exactly what you need, and there are times when it just doesn't fit your needs no matter how you tweak it. In one of my cases, NAT is what I needed to do, and this is how I did it.
First thing, you MUST have iptables installed and working. I think this is what was causing me problems when I first started to try to do this, I had installed Xen using the RPM's and couldn't get anything in NAT to work, even though it all worked perfectly in bridged mode. I haven't confirmed this yet, but I'm susspecting that most, if not all, of my problems were due to the fact that the RPM-installed kernel didn't have iptables included, or if it did, it was moduled and I didn't have the module running. My response to that was to remove the RPM installed copy and install from source. That is what caused me to hate life and write this HowTo (so I could follow my own steps in the future honestly). Anyways, here we go.
- To check and see if iptables is installed and working, give this command
# iptables -L
If all is working good there, then you're good to go. If that didn't work, then you need to make sure iptables is enabled and working properly in your dom0 kernel. Now that we know that iptables is working, lets setup NAT! First we need to make the following changes in the Xen config file /etc/xen/xend-config.sxp to disable bridging and enable NAT:
#(network-script network-bridge) #(vif-script vif-bridge) (network-script network-nat) (vif-script vif-nat)
That tells dom0 to use NAT, now we just have to setup the domU's to use NAT as well, and to do that we need to change their xen config file to include the following:
vif=[ 'ip=10.0.0.2' ] dhcp="off" ip="10.0.0.2" netmask="255.255.0.0" gateway="10.0.0.254" hostname="vm1"
- Note: Yes, it *MUST* be in the 10.x.x.x IP range, because that is how Xen is configured. I however chose to make the subnet 255.255.0.0, it could also be 255.0.0.0 or 255.255.255.0, any of these three subnets will work.
- An oddity that I discovered is that even though the gateway is set to 10.0.0.254, I was unable to get a ping response from that IP, and even though I didn't set any vm's to 10.0.0.1 I do get a ping response from there, leading me to believe that dom0 uses 10.0.0.1 as it's virtual-internal IP. This however is an unconfirmed guess from me. The documentation says to use .254, so that's what I used for my gateway, and it works.
Now, what's the use of using NAT if you can't access the servers from the outside world? We now need to setup some port forwarding to allow the VM's to actually do something, and this we need to do on the dom0.
- To setup some iptables rules, we need to add them to a script that is ran on boot. Add the appropriate rules you want (these are from my mail server) by typing the commands to the command line, and then save your config
iptables -A PREROUTING -t nat -p tcp -i eth0 --dport 80 -j DNAT --to 10.0.0.2:80 iptables -A PREROUTING -t nat -p tcp -i eth0 --dport 25 -j DNAT --to 10.0.0.2:25 iptables -A PREROUTING -t nat -p tcp -i eth0 --dport 443 -j DNAT --to 10.0.0.2:443 iptables -A PREROUTING -t nat -p tcp -i eth0 --dport 110 -j DNAT --to 10.0.0.2:110 iptables -A PREROUTING -t nat -p tcp -i eth0 --dport 143 -j DNAT --to 10.0.0.2:143
# service iptables save
This forwards all the ports necessary to run a mailserver and web server from the domU. Now we need to make the script executable:
# chmod 775 /etc/sysconfig/network-scripts/ifup-iptables
And that's it, make sure all your domU's are shutdown, and reboot your dom0. When it comes back up, it should work!
- Note: The first time I did this I was very annoied... I could ping IP's outside but not DNS names. I later found out, after days of troubleshooting within my DomU, that the problem was actually my dom0 firewall.
Common Commands
A list of commands that I have found to be common.
- (Note that on any of these that have <domain-name> listed, you can substitute the domain ID instead of the domU domain name. You would have to get the domain ID from the 'xm list' command however as they change every time you startup a domU.)
- xm create <config-file>
- xm create -c <config-file> <- Same as above, but opens the started domU's console immediately
- xm console <domain-name>
- xm list
- xm top
- xm shutdown <domain-name>
- xm destroy <domain-name>
Backup your VM
Ok, now that we have everything all setup, lets make a backup image of this beast so that we don't have to go through the entire mess every time you want to make a new VM. I like to do this by temporarily putting all the files into a folder named 'fresh.vm.install'
- Note: this may sound stupid, but make sure you have your domU shutdown before doing this part!
# cd /guests # mkdir fresh.vm.install # mv /path/to/image.img /guests/fresh.vm.install # mv /path/to/swap.swap /guests/fresh.vm.install # mv /etc/xen/vm.config.file /guests/fresh.vm.install
And now we tar it all up into a single file
- Note unless you want a tar file that is the size of what the vm file is supposed to be, make sure to use the -S option so it can handle the file being sparse.
# tar -Sjcvf fresh.vm.install.tar.bz2 fresh.vm.install/
You could use the -z option in the tar command to make it a gzip'd file, but I have found -j (bzip2) to compress it more, so that's what I use.
Alright, we're all done! Enjoy your new Xen setup!
Links
Links that I used to help me come up with all of this stuff... (or links that I just thought was a good alternate source of information in case you, or I, get confused)
The Perfect Xen 3.0 Setup For Debian - Page 6
