Solaris Xen update

Jul 18, 2007
After an undesirably long time, I'm happy to say that another drop of Solaris on Xen is available here. Sources and other sundry parts are here. Documentation can be found at our community site, and you can read Chris Beal describe how to get started with the new bits.

As you might expect, there's been a massive amount of change since the last OpenSolaris release. This time round, we are based on Xen 3.0.4 and build 66 of Nevada. As always, we'd love to hear about your experiences if you try it out, either on the mailing list or the IRC channel.

In many ways, the most significant change is the huge effort we've put in to stabilize our codebase; a significant number of potential hangs, crashes, and core dumps have been resolved, and we hope we're converging on a good-quality release. We've started looking seriously at performance issues, and filling in the implementation gaps. Since the last drop, notable improvements include:

PAE support
By default, we now use PAE mode on 32-bit, aiding compatibility with other domain 0 implementations; we also can boot under either PAE or non-PAE, if the Xen version has 'bi-modal' support. This has probably been the most-requested change missing from our last release.
HVM support
If you have the right CPU, you can now run fully-virtualized domains such as Windows using a Solaris dom0! Whilst more work is needed here, this does seem to work pretty well already. Mark Johnson has some useful tips on using HVM domains.
New management tools
We have integrated the virt- suite of management tools. virt-manager provides a simple GUI for controlling guest domains on a single host. virt-install and virsh are simple CLIs for installing and managing guest domains respectively. Note that parts of these tools are pre-alpha, and we still have a significant amount of work to do on them. Nonetheless, we appreciate any comments...
PV framebuffer
Solaris dom0 now supports the SDL-based paravirt framebuffer backend, which can be used with domUs that have PV framebuffer support.
Virtual NIC support
The Ethernet bridge used in the previous release has been replaced with virtual NICs from the Crossbow project. This enables future work around smart NICs, resource controls, and more.
Simplified Solaris guest domain install
It's now easy to install a new Solaris guest domain using the DVD ISO. The temporary tool in the last release, vbdcfg, has disappeared now as a result. William Kucharski has a walk-through.
Better SMF usage
Several of the xend configuration properties are now controlled using the SMF framework.
Managed domain support
We now support xend-managed domain configurations instead of using .py configuration files. Certain parts of this don't work too well yet (unfortunately all versions of Xen have similar problems), but we are plugging in the gaps here one by one.
Memory ballooning support
Otherwise known as support for dynamic xm mem-set, this allows much greater flexibility in partitioning the physical memory on a host amongst the guest domains. Ryan Scott has more details.
Vastly improved debugging support
Crash dump analysis and debugging tools have always been a critical feature for Solaris developers. With this release, we can use Solaris tools to debug both hypervisor crashes and problems with guest domains. I talk a little bit about the latter feature below.
xvbdb has been renamed
To simply be xdb. This was a very exciting change for certain members of our team.

We're still working hard on finishing things up for our phase 2 putback into Nevada (where "phase 1" was the separate dboot putback). As well as finishing this work, we're starting to look at further enhancements, in particular some features that are available in other vendors' implementations, such as a hypervisor-copy based networking device, blktap support, para-virtualized drivers for HVM domains (a huge performance fix), and more.

Debugging guest domains

Here I'll talk a little about one of the more minor new features that has nonetheless proven very useful. The xm dump-core command generates an image file of a running domain. This file is a dump of all memory owned by the running domain, so it's somewhat similar to the standard Solaris crash dump files. However, dump-core does not require any interaction with the domain itself, so we can grab such dumps even if the domain is unable to create a crash dump via the normal method (typically, it hangs and can't be interacted with), or something else prevents use of the standard Solaris kernel debugging facilities such as kmdb (an in-kernel debugger isn't very useful if the console is broken).

However, this also means that we have no control over the format used by the image file. With Xen 3.0.4, it's rather basic and difficult to work with. This is much improved in Xen 3.1, but I haven't yet written the support for the new format.

To add support for debugging such image files of a Solaris domain, I modified mdb(1) to understand the format of the image file (the alternative, providing a conversion step, seemed unneccessarily awkward, and would have had to throw away information!). As you can see if you look around usr/src/cmd/mdb in the source drop, mdb(1) loads a module called mdb_kb when debugging such image files. This provides simple methods for reading data from the image file. For example, to read a particular virtual address, we need to use the contents of the domain's page tables in the image file to resolve it to a physical page, then look up the location of that page in the file. This differs considerably from how libkvm works with Solaris crash dumps: there, we have a big array of address translations, which is used directly, instead of the page table contents.

In most other respects, debugging a kernel domain image is much the same as a crash dump:

# xm dump-core solaris-domu core.domu
# mdb core.domu
mdb: warning: dump is from SunOS 5.11 onnv-johnlev; dcmds and macros may not match kernel implementation
Loading modules: [ unix genunix specfs dtrace xpv_psm scsi_vhci ufs ... sppp ptm crypto md fcip logindmux nfs ]
> ::status
debugging domain crash dump core.domu (64-bit) from sxc16
operating system: 5.11 onnv-johnlev (i86pc)
> ::cpuinfo
  0 fffffffffbc4b7f0  1b   40    9 169  yes   yes t-1408926 ffffff00010bfc80 sched
> ::evtchns
Type          Evtchn IRQ IPL CPU ISR(s)
evtchn        1      257 1   0   xenbus_intr
evtchn        2      260 9   0   xenconsintr
virq:debug    3      256 15  0   xen_debug_handler
virq:timer    4      258 14  0   cbe_fire
evtchn        5      259 5   0   xdf_intr
evtchn        6      261 6   0   xnf_intr
evtchn        7      262 6   0   xnf_intr
> ::cpustack -c 0
dispatch_hilevel+0x1f(102, 0)
do_interrupt+0x11d(ffffff00010bfaf0, fffffffffbc86f98)
xen_callback_handler+0x42b(ffffff00010bfaf0, fffffffffbc86f98)
dispatch_softint+0x38(9, 0)
do_interrupt+0x140(ffffff0001593520, fffffffffbc86048)
xen_callback_handler+0x42b(ffffff0001593520, fffffffffbc86048)
turnstile_wakeup+0x16e(fffffffec33a64d8, 0, 1, 0)
xenconswput+0x64(fffffffec42cb658, fffffffecd6935a0)
putnext+0x2f1(fffffffec42cb3b0, fffffffecd6935a0)
ldtermrmsg+0x235(fffffffec42cb2b8, fffffffec3480300)
ldtermrput+0x43c(fffffffec42cb2b8, fffffffec3480300)
putnext+0x2f1(fffffffec42cb560, fffffffec3480300)

Note that both ::cpustack and ::cpuregs are capable of using the actual register set at the time of the dump (since the hypervisor needs to store this for scheduling purposes). You can also see the ::evtchns dcmd in action here; this is invaluable for debugging interrupt problems (and we've fixed a lot of those over the past year or so!).

Currently, mdb_kb only has support for image files of para-virtualized Solaris domains. However, that's not the only interesting target: in particular, we could support mdb in live crash dump mode against a running Solaris domain, which opens up all sorts of interesting debugging possibilities. With a small tweak to Solaris, we can support debugging of fully-virtualized Solaris instances. It's not even impossible to imagine adding Linux kernel support to mdb(1), though it's hard to imagine there would be a large audience for such a feature...


Python and DTrace in build 65

May 24, 2007
A significant portion of the Xen control tools are written in Python, in particular xend. It's been somewhat awkward to observe what the daemon is doing at times, necessitating an endless cycle of 'add printf; restart' cycles. A while ago I worked on adding DTrace support to the Python packages we ship in OpenSolaris, and these changes have now made it into the latest build, 65.

As is the case with the other providers people have worked on such as Ruby and Perl, there's two simple probes for function entry and function exit. arg0 contains the filename, arg1 the function name, and arg2 has the line number. So given this simple script to trace the functions called by a particular function invocation, restricted to a given module name:

#!/usr/sbin/dtrace -ZCs

#pragma D option quiet

    /copyinstr(arg1) == $2 && strstr(copyinstr(arg0), $1) != NULL/ {
        self->trace = 1;

    /copyinstr(arg1) == $2 && strstr(copyinstr(arg0), $1) != NULL/ {
        self->trace = 0;

    /self->trace && strstr(copyinstr(arg0), $3) != NULL/ {
        printf("%s %s (%s:%d)\n", probename == "function-entry" ? "->" : "<-",
            copyinstr(arg1), copyinstr(arg0), arg2);

We can run it as follows and get some useful results:

# ./pytrace.d \"\" \"clone\" \"mercurial\" -c 'hg clone /tmp/test.hg'
-> clone (build/proto/lib/python/mercurial/
-> repository (build/proto/lib/python/mercurial/
-> _lookup (build/proto/lib/python/mercurial/
-> _local (build/proto/lib/python/mercurial/
-> __getattribute__ (build/proto/lib/python/mercurial/
-> module (build/proto/lib/python/mercurial/

Of course, this being DTrace, we can tie all of this into general system activity as usual. I also added "ustack helper" support. This is significantly more tricky to implement, but enormously useful for following the path of Python code. For example, imagine we want to look at what's causing write()s in the clone operation above. As usual:

#!/usr/sbin/dtrace -Zs

syscall::write:entry /pid == $target/
        @[jstack(20)] = count();

        trunc(@, 2);

Note that we're using jstack() to make sure we have enough space allocated for the stack strings reported. Now as well as the C stack, we can see what Python functions are involved in the user stack trace:

# ./writes.d -c 'hg clone /tmp/test.hg'
                [ build/proto/lib/python/mercurial/ (add) ]
                [ build/proto/lib/python/mercurial/ (addgroup) ]
                [ build/proto/lib/python/mercurial/ (addchangegroup) ]
                [ build/proto/lib/python/mercurial/ (pull) ]

                [ build/proto/lib/python/mercurial/ (addchangegroup) ]
                [ build/proto/lib/python/mercurial/ (pull) ]
                [ build/proto/lib/python/mercurial/ (clone) ]

Creating a ustack helper

As anyone who's come across the Java dtrace helper source will know, creating a ustack helper is rather a black art.

When a ustack helper is present, it is called in-kernel for each entry in a stack when the ustack() action occurs (source). The D instructions in the helper action are executed such that the final string value is taken as the result of the helper. Typically for Java, there is no associated C function symbol for the PC value at that point in the stack, so the result of the helper is used directly in the stack trace. However, this is not true for Python, so that's why you see a different format above: the normal stack entry, plus the result of the helper in annotated form where it returned a result (in square brackets).

The helper is given two arguments: arg0 is the PC value of the stack entry, and arg1 is the frame pointer. The helper is expected to construct a meaningful string from just those values. In Python, the PyEval_EvalFrame function always has a PyFrameObject * as one of its arguments. By having the helper look at this pointer value and dig around the structures, we can find pointers to the strings containing the file name and function, as well as the line number. We can copy these strings in, and, using alloca() to give ourselves some scratch space, build up the annotation string you see above.

Debugging helpers isn't particularly easy, since it lives and runs in probe context. You can use mdb's DTrace debugging facilities to find out what happened, and some careful mapping between the failing D instructions and the helper source can pinpoint the problem. Using this method it was relatively easy to get a working helper for x86 32-bit. Both SPARC and x86 64-bit proved more troublesome though. The problems were both related to the need to find the PyFrameObject * given the frame pointer. On amd64, the function we needed to trace was passing the arguments in registers, as defined architecturally, so the argument wasn't accessible on the stack via the frame pointer. On SPARC, the pointer we need was stored in a register that was subsequently re-used as a scratch register. Both problems were solved, rather cheesily, by modifying the way the function was called.


Booting para-virtualised OS instances

Feb 21, 2007
Whilst I'm waiting for my home directory to reappear, I thought I'd mention some of the work I've done to support easy booting of domains in Xen.

For dom0 to be able to boot a para-virtualised domU, it needs to be able to bootstrap it. In particular, it needs to be able to read the kernel file and its associated ramdisk so it can hand off control to the kernel's entry point when the domain is created. And we must somehow make these files accessible in the dom0. Previously, you had to somehow copy out the files from the domU filesystem into dom0. This was often difficult (consider getting files off an ext2 filesystem in a Solaris dom0), and was obviously prone to errors such as forgetting to update the copies when upgrading the kernel.

For a while now Xen has had support for a bootloader. This runs in userspace and is responsible for copying out the files (that specified by kernel and ramdisk in the domain's config file) to a temporary directory in dom0; the files are then passed on to the domain builder. Xen has shipped with a bootloader called pygrub. Whilst somewhat confusingly named, it essentially emulated the grub menu. It had backends for a couple of Linux filesystems written in Python and worked by searching for a grub.conf file, then presenting a lookalike grub menu for the user to interact with. When an entry was selected, the specified files would be read off the filesystem and passed back to the builder.

This worked reasonably well for Linux, but we felt there was a number of problems. First, the interactive menu only worked for first boot; subsequent reboots would automatically choose an entry without allowing user interaction (though this is now fixed in xen-unstable). Its interactive nature seemed quite a stumbling block for things like remote domain management; you really don't want to babysit domain creation. Also, the implementation of the filesystem backends wasn't ideal; there was only limited Linux filesystem support, and it didn't work very well.

We've adapted pygrub to help with some of these issues. First, we replaced the filesystem code with a C library called libfsimage. The intention here is to provide a stable API for accessing filesystem images from userspace. Thus it provides a simple interface for reading files from a filesystem image and a plugin architecture to provide the filesystem support. This plugin API is also stable, allowing filesystems past, present and future to be transparently supported. Currently there are plugins for ext2, reiserfs, ufs and iso9660, and we expect to have a zfs plugin soon. We borrowed the grub code for all of these plugins to simplify the implementation, but the API allows for any implementation.

Some people were suggesting solutions involving loopback mounts. This was problematic for us for two main reasons. First, filesystem support in the different dom0 OS's is far from complete; for example, Solaris has no ext2 support, and Linux has no (real) ZFS support. Second, and more seriously, it exposes a significant gap in terms of isolation: the dom0 kernel FS code must be entirely resilient against a corrupt domU filesystem image. If we are to consider domU's as untrusted, it doesn't make sense to leave this open as an attack vector.

Another simple change we made was to allow operation without a grub.conf at all. You can specify a kernel and ramdisk and make pygrub automatically load them from the domU filesystem. Even easier, you can leave out all configuration altogether, and a Solaris domU will automatically boot the correct kernel and ramdisk. This makes setting up your config for a domU much easier.

pygrub understands both fdisk partitions and Solaris slices, so simply specifying the disk will cause the bootloader to look for the root slice and grab the right files to boot.

There's more work we can do yet, of course.


64-bit Python in Nevada build 53

Nov 23, 2006
Coming to you in build 53 of OpenSolaris is 64-bit Python, which I worked on with Laszlo Peter of JDS fame. This means Python modules can make use of 64-bit versions of libraries, as well as 64-bit plugins. You can run the 64-bit version of Python via the path /usr/bin/amd64/python (on x86). This path isn't quite set in stone yet, so don't rely on it.

This facility didn't previously exist on any OS, so we had to make some innovations in terms of how Python lets modules build and load. In particular the Makefile used by Python previously hard-coded certain compiler flags etc. We had to make this dynamic. Also, we had to make some modifications to where Python looks for .so files when loading modules. Previously it would just assume that, say, /usr/lib/python2.4/ was of the correct word size. Now, if it's running 64-bit, it will look for /usr/lib/python2.4/64/

Similarly, building a Python module using the 64-bit Python will automagically install the .so file in the right place. Thanks to their architecture-independence, we don't need the same tricks for the .pyc files.

The need for this arose from the continuing work on Solaris dom0's running under Xen. The kernel/hypervisor interfaces provided are not 64-bit clean in the sense that 32-bit tools cannot deal with 64-bit domains; as a result, we need to run (the Python-based) xend as a native binary.

As an added bonus, Laca has also upgraded to Python 2.4.4, which finally enables the curses module on Solaris; also fixed are some niggling problems with accidental regeneration of .pyc files.


Save/restore of MP Solaris domUs

Jul 15, 2006
In honour of our new release of OpenSolaris on Xen, here's some details on the changes I've made to support save/resume (and hence migration and live migration) with MP Solaris domUs. As before, to actually see the code I'm describing, you'll need to download the sources - sorry about that.

Under Xen, the suspend process is somewhat unusual, in that only the CPU context for (virtual) CPU0 is stored in the state file. This implies that the actual suspend operation must be performed on CPU0, and we have to find some other way of capturing CPU context (that is, the register set) for the other CPUs.

In the Linux implementation, Xen suspend/resume works by using the standard CPU hotplug support for all CPUs other than CPU0. Whilst this works well for Linux, this approach is more troublesome for Solaris. Hot-unplugging the other CPUs doesn't match well with the mapping between Xen and Solaris notions of "offline" CPUs (the interested can read the big comment on line 406 of usr/src/uts/i86xen/os/mp_xen.c for a description of how this mapping currently works). In particular, offline CPUs on Solaris still participate in IPI interrupts, whilst a "down" VCPU in Xen cannot.

In addition, the standard CPU offlining code in Solaris is not built for this purpose; for example, it will refuse to offline a CPU with bound threads, or the last CPU in a partition.

However, all we really need to do is get the other CPUs into a known state which we can recover during the resume process. All the dispatcher data structures etc. associated with the CPUs can remain in place. To this end, we can use pause_cpus() on the other CPUs. By replacing the pause handler with a special routine (cpu_pause_suspend()), we can store the CPU context via a setjmp(), waiting until all CPUs have reached the barrier. We need to disable interrupts (or rather, Xen's virtualized equivalent of interrupts), as we have to tear down all the interrupts as part of the suspend process, and we need to ensure none of the CPUS go wandering off.

Once all CPUs are blocked at the known synchronisation point, we can tell Xen to "down" the other VCPUs so they can no longer run, and complete the remaining cleanup we need to do before we tell Xen we're ready to stop via HYPERVISOR_suspend().

On resume, we will come back on CPU0, as Xen stored the context for that CPU itself. After redoing some of the setup we tore down during suspend, we can move on to resuming the other CPUs. For each CPU, we call mach_cpucontext_restore(). We use the same Xen call used to create the CPUs during initial boot. In this routine, we fiddle a little bit with the context saved in the jmpbuf by setjmp(); because we're not actually returning via a normal longjmp() call, we need to emulate it. This means adjusting the stack pointer to simulate a ret, and pretending we've returned 1 from setjmp() by setting the %eax or %rax register in the context.

When each CPU's context is created, it will look as if it's just returned from the setjmp() in cpu_pause_suspend(), and will continue upon its merry way.

Inevitably, being a work-in-progress, there are still bugs and unresolved issues. Since offline CPUs won't participate in a cpu_pause(), we need to make sure that those CPUs (which will typically be sitting in the idle loop) are safe; currently this isn't being done. There are also some open issues with 64-bit live migration, and suspending SMP domains with virtual disks, which we're busy working on.


Fun with stack corruption

Apr 13, 2006
Today, we were seeing a very odd crash in some xen code. The core dump wasn't of great use, since both %eip and %ebp were zeroed, which means no backtrace. Instead I attached mdb to the process and started stepping through to see what was happening. It soon transpired that we were crashing after successfully executing a C function called xspy_introduce_domain(), but before we got back to the Python code that calls into it. After a little bit of head-scratching, I looked closer at the assembly for this function:`xspy_introduce_domain:            pushl  %ebp`xspy_introduce_domain+1:          movl   %esp,%ebp
...`xspy_introduce_domain+0x57:       subl   $0xc,%esp`xspy_introduce_domain+0x5a:       leal   -0xc(%ebp),%eax`xspy_introduce_domain+0x5d:       pushl  %eax`xspy_introduce_domain+0x5e:       leal   -0x8(%ebp),%eax`xspy_introduce_domain+0x61:       pushl  %eax`xspy_introduce_domain+0x62:       leal   -0x2(%ebp),%eax`xspy_introduce_domain+0x65:       pushl  %eax`xspy_introduce_domain+0x66:       pushl  $0xc4b13114`xspy_introduce_domain+0x6b:       pushl  0xc(%ebp)`xspy_introduce_domain+0x6e:       call   +0x43595220      
...`xspy_introduce_domain+0x11f:      leave`xspy_introduce_domain+0x120:      ret

Seems OK - we're pushing three pointers onto the stack (+0x5a-0x65) and two other arguments. Let's look at the sources:

static PyObject *xspy_introduce_domain(XsHandle *self, PyObject *args)
    domid_t dom;
    unsigned long page;
    unsigned int port;

    struct xs_handle *xh = xshandle(self);
    bool result = 0;

    if (!xh)
        return NULL;
    if (!PyArg_ParseTuple(args, "ili", &dom, &page, &port))
        return NULL;

Looking up the definition of PyArg_ParseTuple() gave me the clue as to the problem. The format string specifies that we're giving the addresses of an int, long, and int. Yet in the assembly, the offsets of the leal instructions indicate we're pushing addresses to two 32-bit storage slots, and one 16-bit slot. So when PyArg_ParseTuple() writes its 32-bit quantity, it's going to overwrite two more bytes than it should.

As it happens, we're at the very top of the local stack storage space (-0x2(%ebp)). So those two bytes actually end up over-writing the bottom two bytes of the old %ebp we pushed at the start of the function. Then we pop that corrupted value back into the %ebp register via the leave. This has no effect until our caller calls leave itself. We move %ebp into %esp, then attempt to pop from the top of this stack into %ebp again. As it happens, the memory pointed to by the corrupt %ebp is zeroed; thus, we end up setting %ebp to zero. Finally, our caller does a ret, which pops another zero, but this time into %eip. Naturally this isn't a happy state of affairs, and we find ourselves with a core dump as described earlier.

Presumably this bug happened in the first place because someone didn't notice that domid_t was a 16-bit quantity. What's amazing is that nobody else has been hitting this problem!


A brief tour of i86xen

Feb 14, 2006
In this post, I'm going to give a quick walk through the major changes we've made so far in doing our port of Solaris to the Xen "platform". As we've only supplied a tarball of the source tree so far, I can't hyperlink to the relevant bits - sorry about that. As our code is still under heavy development, you can expect some of this code organisation to change significantly; nonetheless I thought this might be useful for those interested in peeking into the internals of what we've done so far.

As you might expect, the vast majority of the changes we've made reside in the kernel. To support booting Solaris under Xen (both domU and dom0, though as we've said the latter is still in the very early stages of development), we've introduced a new platform based on i86pc called i86xen. Wherever possible, we've tried to share common code by using i86pc's sources. There's still some cleanup we can do in this area.

Within usr/src/uts/i86xen, there are a number of Xen-specific source files:

Contains the PSM ("Platform-Specific Module") module for Xen. This mirrors the PSM provided by i86pc, but deals with the hypervisor-provided features such as the clock timer and the events system.
This contains the virtual root nexus driver "xendev". All of the virtual frontend drivers are connected to this.
The virtual block driver. It's currently non-functional with the version of Xen we're working with; we're working hard on getting it functional.
The guts of the kernel/hypervisor code. Amongst other things, it provides interfaces for dealing with events in evtchn.c and hypervisor_machdep.c (the hypervisor version of virtual interrupts, which hook into Solaris's standard interrupt system), the grant table in gnttab.c (used for providing access/transfer of pages between frontend and backend, suspend/resume in xen_machdep.c, and support routines for the debugger and the MMU code (mach_kdi.c and xen_mmu.c respectively).

As mentioned we use the i86pc code where possible, occasionally using #ifdefs where minor differences are found. In particular we re-use the i86pc HAT (MMU management) code found in i86pc/vm. You can also find code for the new boot method described by Joe Bonasera in i86pc/dboot and i86pc/boot.

A number of drivers that are needed by Xen but aren't i86xen specific live under usr/src/uts/common:

common/io/xenbus_*.c common/io/xenbus/
"xenbus" is a simple transport for configuration data provided by domain0; for example, it provides a node control/shutdown which will notify the domainU that the user has requested the domain to be shutdown (or suspended) from domain0's management tools. This code provides this support.
The virtual console frontend driver.
The virtual net device frontend driver.

As you might expect, the userspace changes we've needed to make so far have been reasonably minimal. Despite supporting the new i86xen platform definition, the only significant changes have been to usr/src/cmd/mdb/, where we've added some changes to better support debugging of the Xen-style x86 MMU.


Live migration of Solaris instances

Feb 14, 2006
Today we released our current source tree for our Solaris Xen port; for more details and the downloads see the Xen community on OpenSolaris.

One of the most useful features of Xen is its ability to package up a running OS instance (in Xen terminology, a "domainU", where "U" stands for "unprivileged"), plus all of its state, and take it offline, to be resumed at a later time. Recently we performed the first successful live migration of a running Solaris instance between two machines. In this blog I'll cover the various ways you can do this.

Para-virtualisation of the MMU

Typical "full virtualisation" uses a method known as "shadow page tables", whereby two sets of pagetables are maintained: the guest domain's set, which aren't visible to the hardware via cr3, and page tables visible to the hardware which are maintained by the hypervisor. As only the hypervisor can control the page tables the hardware uses to resolve TLB misses, it can maintain the virtualisation of the address space by copying and validating any changes the guest domain makes to its copies into the "real" page tables.

All these duplicates pages come at a cost of course. A para-virtualisation approach (that is, one where the guest domain is aware of the virtualisation and complicit in operating within the hypervisor) can take a different tack. In Xen, the guest domain is made aware of a two-level address system. The domain is presented with a linear set of "pseudo-physical" addresses comprising the physical memory allocated to the domain, as well as the "machine" addresses for each corresponding page. The machine address for a page is what's used in the page tables (that is, it's the real hardware address). Two tables are used to map between pseudo-physical and machine addresses. Allowing the guest domain to see the real machine address for a page provides a number of benefits, but slightly complicates things, as we'll see.


The simplest form of "packaging" a domain is suspending it to a file in the controlling domain (a privileged OS instance known as "domain 0"). A running domain can be taken offline via an xm save command, then restored at a later time with xm restore, without having to go through a reboot cycle - the domain state is fully restored.

xm save xen-7 /tmp/domain.img

An xm save notifies the domain to suspend itself. This arrives via the xenbus watch system on the node control/shutdown, and is handled via xen_suspend_domain(). This is actually remarkably simple. First we leverage Solaris's existing suspend/resume subsystem, CPR, to iterate through the devices attached to the domain's device nexus. This calls each of the virtual drivers we use (the network, console, and block device frontends) with a DDI_SUSPEND argument. The virtual console, for example, simply removes its interrupt handler in xenconsdetach(). As a guest domain, this tears down the Xen event channel used to communicate with the console backend. The rest of the suspend code deals with tearing down some of the things we use to communicate with the hypervisor and domain 0, such as the grant table mappings. Additionally we convert a couple of stored MFN (the frame numbers of machine addresses) values into pseudo-physical PFNs. This is because the MFNs are free to change when we restore the guest domain; as the PFNs aren't "real", they will stay the same. Finally we call HYPERVISOR_suspend() to call into the hypervisor and tell it we're ready to be suspended.

Now the domain 0 management tools are ready to checkpoint the domain to the file we specified in the xm save command. Despite the name, this is done via xc_linux_save(). Its main task is to convert any MFN values that the domain still has into PFN values, then write all its pages to the disk. These MFN values are stored in two main places; the PFN->MFN mapping table managed by the domain, and the actual pages of the page tables.

During boot, we identified which pages store the PFN->MFN table (see xen_relocate_start_info()), and pointed to that structure in the "shared info" structure, which is shared between the domain and the hypervisor. This is used to map the table in xc_linux_save().

The hypervisor keeps track of which pages are being used as page tables. Thus, after domain 0 has mapped the guest domain's pages, we write out the page contents, but modify any pages that are identified as page tables. This is handled by canonicalize_pagetable(); this routine replaces all PTE entries that contain MFNs with the corresponding PFN value.

There are a couple of other things that need to be fixed too, such as the GDT.

xm restore /tmp/domain.img

Restoring a domain is essentially the reverse operation: the data for each page is written into one of the machine addresses reserved for the "new" domain; if we're writing a saved page table, we replace each PTE's PFN value with the new MFN value used by the new instance of the domain.

Eventually the restored domain is given back control, coming out from the HYPERVISOR_suspend() call. Here we need to rebuild the event channel setup, and anything else we tore down before suspending. Finally, we return back from the suspend handler and continue on our merry way.


xm migrate xen-7 remotehost

A normal save/restore cycle happens on the same machine, but migrating a domain to a separate machine is a simple extension of the process. Since our save operation has replaced any machine-specific frame number value with the pseudo-physical frames, we can easily do the restore on a remote machine, even though the actual hardware pages given to the domainU will be different. The remote machine must have the Xen daemon listening on the HTTP port, which is a simple change in its config file. Instead of writing each page's contents to a file, we can transmit it across HTTP to the Xen daemon running on a remote machine. The restore is done on that machine in the same manner as described above.

Live Migration

xm migrate --live xen-7 remotehost

The real magic happens with live migration, which keeps the time the domain isn't kept running to a bare minimum (on the order of milliseconds). Live migration relies on the empirically observed data that an OS instance is unlikely to modify a large percentage of its pages within a certain time frame; thus, by iteratively copying over modified domain pages, we'll eventually reach a point where the remaining data to be copied is small enough that the actual downtime for a domainU is minimal.

In operation, the domain is switched to use a modified form of the shadow page tables described above, known as "log dirty" mode. In essence, a shadow page table is used to notify the hypervisor if a page has been written to, by keeping the PTE entry for the page read-only: an attempt to write to the page causes a page fault. This page fault is used to mark the domain page as "dirty" in a bitmap maintained by the hypervisor, which then fixes up the domain's page fault and allows it to continue.

Meanwhile, the domain management tools iteratively transfers unmodified pages to the remote machine. It reads the dirty page bitmap and re-transmits any page that has been modified since it was last sent, until it reaches a point where it can finally tell the domain to suspend, and switch over to running it on the remote machine. This process is described in more detail in Live Migration of Virtual Machines.

Whilst transmitting all the pages takes a while, the actual time between suspension and resume is typically very small. Live migration is pretty fun to watch happen; you can be logged into the domain over ssh and not even notice that the domain has migrated to a different machine.

Further Work

Whilst live migration is currently working for our Solaris changes, there's still a number of improvements and fixes that need to be made.

On x86, we usually use the TSC register as the basis for a high-resolution timer (heavily used by the microstate accounting subsystem). We don't directly use any virtualisation of the TSC value, so when we restore a domain, we can see a large jump in the value, or even see it go backwards. We handle this OK (once we fixed bug 6228819 in our gate!), but don't yet properly handle the fact that the relationship between TSC ticks and clock frequency can change between a suspend and resume. This screws up our notion of timing.

We don't make any effort to release physical pages that we're not currently using. This makes suspend/resume take longer than it should, and it's probably worth investigating what can be done here.

Currently many hardware-specific instructions and features are enabled at boot by patching in instructions if we discover the CPU supports it. For example we discovered a domain that died badly when it was migrated to a host that didn't support the sfence instruction. If such a kernel is migrated to a machine with different CPUs, the domain will naturally fail badly. We need to investigate preventing incompatible migrations (the standard Xen tools currently do no verification), and also look at whether we can adapt to some of these changes when we resume a domain.


Generating assembly structure offset values with CTF

Jan 17, 2006
The Solaris kernel contains a fair amount of assembly, and this often needs to access C structures (and in particular know the size of such structures, and the byte offsets of their members). Since the assembler can't grok C, we need to provide constant values for it to use. This also applies to the C library and kmdb.

In the kernel, the header assym.h provides these values; for example:

#define T_STACK 0x4
#define T_SWAP  0x68
#define T_WCHAN 0x44

These values are the byte offset of certain members into struct _kthread. For each of the types we want to reference from assembly, a template is provided in one of the files. For the above, we can see in usr/src/uts/i86pc/ml/

_kthread        THREAD_SIZE
        t_pcb                   T_LABEL
        t_stk                   T_STACK
        t_lwpchan.lc_wchan      T_WCHAN
        t_flag                  T_FLAGS

This file contains structure names as well their members. Each of the members listed (which do not have to be in order, nor does the list need to be complete) cause a define to be generated; by default, an uppercase version of the member name is used. As can be seen, this can be overridden by specifying a #define name to be used. The THREAD_SIZE define corresponds to the bytesize of the entire structure (it's also possible to generate a "shift" value, which is log2(size)).

To generate the header with the right offset and size values we need, a script is used to generate CTF data for the needed types, which then uses this data to output the assym.h header. This is a Perl script called genoffsets, and the build invokes it with a command line akin to:

genoffsets -s ctfstabs -r ctfconvert cc < > assym.h

The hand-written file serves as input to the script, and it generates the header we need. The script takes the following steps:

  1. Two temporary files are generated from the input. One is a C file consisting of #includes and any other pre-processor directives. The other contains the meat of the offsets file.
  2. The C file containing all the includes is built with the compile line given (I have stripped the compiler options above for readability).
  3. ctfconvert is run on the built .o file.
  4. The pre-processor is run across the second file (the temporary offsets file)
  5. This pre-processed file is passed to ctfstabs along with the .o file.

ctfstabs reads the input offsets file, and for each entry, looks up the relevant value in the CTF data contained in the .o file passed to it. It has two output modes (which I'll come to shortly), and in this case we are using the genassym driver to output the C header. As you can see, this is a fairly simple process of processing each line of the input and looking up the type data in the CTF contained in the .o file.

A similar process is used for generating forth debug files for use when debugging the kernel via the SPARC PROM. This takes a different format of offsets file more appropriate to generating the forth debug macros, described in the forth driver.

To finish off the output header, the output from a small program called genassym (or, on SPARC, genconst) is appended. It contains a bunch of printfs of constants. A lot of those don't actually need to be there since they're simple constant defines, and the assembly file could just include the right header, but others are still there for reasons such as:

  • The macros which hide assembler syntax differences such as _MUL aren't implemented for the C compiler
  • The value is an enum type, which ctfstabs doesn't support
  • The constant is a complicated composed macro that the assembler can't grok

and other reasons. Whilst a lot of these could be cleaned up and removed from these files, it's probably not worth the development effort except as a gradual change.


Resource management of services

Nov 19, 2005
SMF introduced the notion of a service as a first-order object in the Solaris OS. Thus, you have administration interfaces capable of dealing with services (as opposed to the implicit service represented by a set of processes, for example). It doesn't seem very well known, but as Stephen Hahn mentions, this also applies to the resource management facilities of Solaris.

A service can be bound to a project (as well as a resource pool, which I won't go into here). This allows us to add resource controls to the project which will apply to the service as a whole, which is significantly more reliable and usable than trying to deal with individual daemons etc. Unfortunately, it's not as obvious to set up as it should be (of which more later), so here's a simple walkthrough.

We're going to set up a simple 'forkbomb' service, which simply runs this program:

#include <unistd.h>
#include <stdlib.h>

int main()
        int first = 1;
        while (1) {
                if (fork() > 0 && first)
                first = 0;

If you try running this program in an environment lacking resource controls, don't expect to be able to do much to your box except reboot it. Note the first parent does an exit(0) so that SMF doesn't think the service has failed (since we'll be a standard contract service). Here's the SMF manifest for our service:

<?xml version="1.0"?>
<!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1">
<service_bundle type='manifest' name='forkbomb'>
<service name='application/forkbomb' type='service' version='1'>
                <method_context project='forkbomb'>
                        <method_credential user='root' />

        <instance name='default' enabled='false' />

Note that as well as setting the project in the method context, we've set a method credential; this is a workaround for a problem I'll come to later. Now we need to create the 'forkbomb' project for the service:

# projadd -K 'project.max-lwps=(privileged,100,deny)' forkbomb

Alternatively we could create a new user for the service to use, set the method credential to use that user, then change our 'forkbomb' project to allow the user to join it. It's important to note that this still works even for root, though, so that's what we've done here.

Finally, we can import the manifest as a service, then temporarily enable it (so it won't start next time we boot!):

# svccfg import /opt/forkbomb/manifest/forkbomb.xml
# svcadm enable -t forkbomb

The forkbomb is now running flat out, but under the constraints of the resource controls we set on its project. Thus we still have a running system, and have enough resources to disable our 'mis-behaving' service. Let's have a look at prstat:

Total: 148 processes, 266 lwps, load averages: 68.06, 20.50, 10.75
 21145 root      992K  244K run      1    0   0:00:03 1.4% forkbomb/1
 21132 root      992K  244K run     49    0   0:00:03 1.2% forkbomb/1
 21128 root      992K  244K run     31    0   0:00:03 1.1% forkbomb/1
 21113 root      992K  244K run     31    0   0:00:03 1.1% forkbomb/1
 21176 root      992K  244K run     33    0   0:00:03 1.1% forkbomb/1
 21124 root      992K  244K run     53    0   0:00:03 1.1% forkbomb/1
 21119 root      992K  244K run     52    0   0:00:03 1.1% forkbomb/1
 21156 root      992K  244K run     53    0   0:00:03 1.0% forkbomb/1
 21088 root      992K  244K run     52    0   0:00:03 1.0% forkbomb/1
 21136 root      992K  244K run     43    0   0:00:03 1.0% forkbomb/1
 21133 root      992K  244K run     44    0   0:00:03 1.0% forkbomb/1
 21097 root      992K  244K run     52    0   0:00:03 1.0% forkbomb/1
 21103 root      992K  244K run     56    0   0:00:03 1.0% forkbomb/1
 21092 root      992K  244K run     52    0   0:00:03 1.0% forkbomb/1
 21183 root      992K  244K run     53    0   0:00:03 1.0% forkbomb/1
   100      100   97M   24M   0.6%   0:04:47  95% forkbomb
     1        5   11M 8268K   0.3%   0:00:00 0.0% user.root
    10        3   18M 8060K   0.3%   0:00:00 0.0% group.staff
     0       40  135M   83M   2.6%   0:00:17 0.0% system
Total: 148 processes, 266 lwps, load averages: 70.60, 21.80, 11.24

As we might expect, there's a high system load (since our fork-bomb is ignoring the errors from fork() when it hits its resource limit). Note that the 'forkbomb' project has been clamped to a maximum of 100 LWPs, as you can see in the NPROC field. But most importantly, the system is still usable, and we can stop the troublesome service:

# svcadm disable forkbomb

After a while for the stop method to finish (or time out, both of which will kill all processes in the service contract), we're done!

I mentioned above that we needed to specify a method credential to work around a bug. This is bug 5093847. The way the property lookup works currently, if the use_profile property on the service isn't found, then none of the rest of the method context is examined. Setting the method credential has the side-effect of creating this property, so things work properly. This bug would also be nice to fix since we could directly set the project property via svccfg if the properties for the method context were always created. Any interested parties are strongly encouraged to have a go at fixing it - it's not currently being worked on, and I'd happy to help :)