My previous blog post described how to install OpenSolaris 2008.11 on a Solaris dom0 under Xen. This also works on with a Linux dom0. However, since upstream is missing some of our dom0 fixes, it's unfortunately more complicated. In particular, we can't use virt-install, as it doesn't know about Solaris ISOs, and later on, we can't use pygrub to boot from ZFS, since it doesn't know how to read such a filesystem. Bear with me, this gets a little awkward.

This example is using a 32-bit Fedora 8 installation. Your milage is likely to vary if you're using a different version, or another Linux distribution. First some of the configuration parameters you might want to change:

export name="domu-224"
export iso="/isos/osol-2008.11.iso"
export dompath="/export/guests/2008.11"
export rootdisk="$dompath/root.img"
export unixfile="/platform/i86xpv/kernel/unix"

If you're on 64-bit Linux, set unixfile="/platform/i86xpv/kernel/amd64/unix" instead. We need to create ourselves a 10Gb root disk:

mkdir -p $dompath
dd if=/dev/zero count=1 bs=$((1024 * 1024)) seek=10230 of=$rootdisk

Now let's use the configuration we need to install OpenSolaris:

cat >/tmp/domain-$name.xml <<EOF
<domain type='xen'>
 <name>$name</name>
 <bootloader>/usr/bin/pygrub</bootloader>
 <bootloader_args>--kernel=/platform/i86xpv/kernel/unix --ramdisk=/boot/x86.microroot</bootloader_args>
 <memory>1048576</memory>
 <on_reboot>destroy</on_reboot>
 <devices>
  <interface type='bridge'>
   <source bridge='eth0' />
   <--
       If you have a static DHCP setup, add the domain's MAC address here
       <mac address='00:16:3e:1b:e8:18' />
   -->
  </interface>
  <disk type='file' device='cdrom'>
   <driver name='file' />
   <source file='$iso' />
   <target dev='xvdc:cdrom' />
  </disk>
  <disk type='file' device='disk'>
   <driver name='file' />
   <source file='$rootdisk' />
   <target dev='xvda' />
  </disk>
 </devices>
</domain>
EOF

And start up the domain:

virsh create /tmp/domain-$name.xml
virsh console $name

Now you're dropped into the domain's console, and you can use the VNC trick I described to do the install. Answer the questions, wait for the domain to DHCP, then:

domid=`virsh domid $name`
ip=`/usr/bin/xenstore-read /local/domain/$domid/ipaddr/0`
port=`/usr/bin/xenstore-read /local/domain/$domid/guest/vnc/port`
/usr/bin/xenstore-read /local/domain/$domid/guest/vnc/passwd
vncviewer $ip:$port

At this point, you can proceed with the installation as normal. Before you reboot though, we need to do some tricks, due to the lack of ZFS support mentioned above. Whilst still in the live CD environment, bring up a terminal. We need to copy the new kernel and ramdisk to the Linux dom0. We can automate this via a handy script:

#/bin/bash

dom0=$1
dompath=$2
unixfile=/platform/i86xpv/kernel/$3/unix

root=`pfexec beadm list -H |  grep ';N*R;' | cut -d \; -f 1`
mkdir /tmp/root
pfexec beadm mount $root /tmp/root 2>/dev/null
mount=`pfexec beadm list -H $root | cut -d \; -f 4`
pfexec bootadm update-archive -R $mount
scp $mount/$unixfile root@$dom0:$dompath/kernel.$root
scp $mount/platform/i86pc/$3/boot_archive root@$dom0:$dompath/ramdisk.$root
pfexec beadm umount $root 2>/dev/null
echo "Kernel and ramdisk for $root copied to $dom0:$dompath"
echo "Kernel cmdline should be:"
echo "$unixfile -B zfs-bootfs=rpool/ROOT/$root,bootpath=/xpvd/xdf@51712:a"

For example, we might do:

/tmp/update_dom0 linux-dom0 /export/guests/2008.11
or on 64-bit:
/tmp/update_dom0 linux-dom0 /export/guests/2008.11 amd64

Now, you can finish the installation by clicking the reboot button. This will shut down the domain, ready to run. But first we need the configuration file for running the domain:

cat >/$dompath/$name.xml <<EOF
<domain type='xen'>
 <name>$name</name>
 <os>
  <kernel>$dompath/kernel.opensolaris</kernel>
  <initrd>$dompath/ramdisk.opensolaris</initrd>
  <cmdline>$unixfile -B zfs-bootfs=rpool/ROOT/opensolaris,bootpath=/xpvd/xdf@51712:a</cmdline>
 </os>
 <memory>1048576</memory>
 <devices>
  <interface type='bridge'>
   <source bridge='eth0'/>
  </interface>
  <disk type='file' device='disk'>
   <driver name='file' />
   <source file='$rootdisk' />
   <target dev='xvda' />
  </disk>
 </devices>
</domain>

virsh define $dompath/$name.xml
virsh start $name
virsh console $name

It should be booting, and you're (finally) done!

Updating the guest

Unfortunately we're not quite out of the woods yet. What we have works fine, but if we update the guest via pkg image-update, we'll need to make changes in dom0 to boot the new boot environment. The update_dom0 script above will do a fine job of copying out the new kernel and ramdisk for the BE that's active on reboot, but you also need to edit the config file. For example, if I wanted to boot into the new BE called opensolaris-1, I'd replace these lines:

<kernel>$dompath/kernel.opensolaris</kernel>
<initrd>$dompath/ramdisk.opensolaris</initrd>
<cmdline>$unixfile -B zfs-bootfs=rpool/ROOT/opensolaris,bootpath=/xpvd/xdf@51712:a</cmdline>

with these:

<kernel>$dompath/kernel.opensolaris-1</kernel>
<initrd>$dompath/ramdisk.opensolaris-1</initrd>
<cmdline>$unixfile -B zfs-bootfs=rpool/ROOT/opensolaris-1,bootpath=/xpvd/xdf@51712:a</cmdline>

then re-configure the domain (whist it's shut down) via virsh undefine $name ; virsh define $dompath/$name.xml.

Yes, we're aware this is rather over-complicated. We're trying to find the time to send our changes to virt-install upstream, as well as ZFS support. Eventually this will make it much easier to use a Linux dom0.

Tags: OpenSolaris Xen Linux