Xen ARM with Virtualization Extensions
The ARM v7-A and ARM v8-A architectures include optional virtualization extensions that allow a hypervisor to manage fully hardware virtualized guests. These extensions are currently available in some ARM v7 processors such as the Cortex A15 and Cortex A7.
- 1 Status
- 2 Contributing
- 3 Hardware
- 4 Requirements
- 5 Device Trees
- 6 Xen
- 7 Dom0 kernel
- 8 DomU kernel and DTS
- 9 Building Xen on ARM
- 10 Versatile Express specific configurations
- 11 Open issues, known problems and workarounds
- 12 Developer FAQs
- 13 Related pages
Both the 32-bit (arm32) and the 64-bit (arm64) ports of Xen boot dom0 and unprivileged guests can be created and destroyed using xl. See below for information on the hardware and models.
Currently only 32-bit dom0 and guests are supported, even on the 64-bit hypervisor.
The guest (including dom0) visible ABI exposed by the hypervisor is not yet set in stone and there may be changes as work progresses. See requirements (below) for specific details of matching kernel and hypervisor versions.
The arm32 port of Xen currently runs on:
- The Cortex A15 Real-time System Model's provided by ARM. See Xen ARMv7 with Virtualization Extensions/FastModels for more information on obtaining and building the necessary model.
- A Cortex-A15 Processor running on the Versatile Express.
- Here is some step to start Xen on the Arndale Development board
The arm64 port of Xen currently runs on:
- The AEMv8 Real-time System Models by ARM.
Work is also underway to support the ARM v8 Foundation Model.
- ARM Hardware or Software Model
- See above for details of the hardware and models which are supported.
- Device Tree
- A device tree in the flat device tree format (.dtb). The device tree source for unprivileged Xen ARM guests is available upstream in the Linux tree >= v3.7: xenvm-4.2.dts. The interface is also documented in the Linux tree: xen.txt. Regarding Xen and Dom0, the regular DTS for the underlying platform can be used, however it requires the additional hypervisor node described in the Linux Documentation and the command line arguments under the chosen node, see for example the Xen DTS for the Versatile Express Cortex A15 machine we use for development vexpress-v2p-ca15-tc1.dts.
- All current work is now merged into the current development branch git://xenbits.xen.org/xen.git. It is recommended to use the latest Xen master branch.
- Linux kernel for dom0
- The patches necessary to boot Linux as dom0 under Xen were merged upstream in v3.7. In order to actually start guests a few additional patches were required however these patches have now been included in the v3.8 Linux release. The latest Linus' tree has everything needed to run on Xen on ARM as dom0 and domU.
- dom0 userspace
- The developers are using the armhf port of Debian Wheezy (which is currently the testing release).
- domU kernel
- The patches necessary to boot Linux as a guest under Xen were merged upstream in v3.7.
Hypervisor ABI Compatibility
The ABI for Xen on ARM is currently not set in stone.
The most recent ABI change was xen: event channel arrays are xen_ulong_t and not unsigned long. This change was reflected in Linux commit c81611c4e96f595a80d8be9367c385d2c116428b which is already present in mainline and will be part of v3.9-rc1. The xen-arm-3.8.y branch of git://xenbits.xen.org/people/ianc/linux.git contains the current stable release + this change.
Xen needs the device trees to be in the flat device tree format (the device tree blob or DTB). This is built by the device tree compiler (dtc) from the device tree source files (.dts and .dtsi).
Version 1.3 or later of dtc is required.
Build the .dtb files with make.
When running on the model there is no bootloader to provide the DTB. Instead, it is linked with the xen image.
The DTB to use is specified by the CONFIG_DTB_FILE config variable which should be set to the absolute path of the DTB. This variable may be set in .config or on the make command line.
The Xen command line may be set with the bootargs parameter of the chosen node of the device tree. Parameters for the domain 0 kernel are passed using the xen,dom0-bootargs parameter.
Enable at least ARCH_VEXPRESS, ARCH_VEXPRESS_DT. Disable SPARSE_IRQ (it doesn't play nicely with device tree support). If ARM_APPENDED_DTB is enabled then any appended DTB will be used instead of one supplied by Xen and the kernel will crash unless the memory in the DTB matches that location/size supplied by Xen.
A working configuration is available here.
The Linux command line should be set with the xen,dom0-bootargs parameter of the chosen node of the device tree supplied to Xen. If this parameter is missing then Linux will not get a command line and the wrong console will be used (there's a bug where the kernel doesn't use the default, built-in command line if the /chosen node is missing.
DomU kernel and DTS
Unprivileged guests can be created using xl. A simple VM config file would look like this:
kernel = "/root/image" memory = 128 name = "win" vcpus = 1 disk = [ 'phy:/dev/loop0,xvda,w' ]
where image is actually a Linux zImage with an appended xenvm-4.2.dtb:
cat arch/arm/boot/zImage arch/arm/boot/dts/xenvm-4.2.dtb > /mnt/root/image
The dom0 kernel is loaded by Xen from the beginning of the flash device.
To boot the kernel natively use an updated boot-wrapper from git://xenbits.xen.org/people/dvrabel/boot-wrapper.git.
Build the device tree blobs provided with the kernel with make dtbs. A suitable DTB file for the envelope model is vexpress-v2p-aem-v7a.dtb.
For a native boot the kernel should have ARM_APPENDED_DTB and ARM_ATAG_DTB_COMPAT enabled. The kernel zImage and the DTB should be appended. e.g., cat zImage vexpress-v2p-aem-v7a.dtb > image.bin.
Building Xen on ARM
Cross compiling the hypervisor is relatively simple. For 32-bit you can find a cross compiler in some Linux distros or you can download the arm-unknown-linux-gnueabi compiler from kernel.org. For 64-bit Linaro supply a suitable cross-compiler.
Once you have a suitable cross compiler you can compile Xen with:
$ make dist-xen XEN_TARGET_ARCH=arm32 CROSS_COMPILE=arm-unknown-linux-gnueabi-
$ make dist-xen XEN_TARGET_ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu-
This assumes that the command prefix for you cross compiler is arm-unknown-linux-gnueabi- or arm-unknown-linux-gnueabi- and that the appropriate arm-unknown-linux-gnueabi-gcc or arm-unknown-linux-gnueabi-gcc
and friends are in your $PATH.
Cross compilation of the tools is described in Xen ARM with Virtualization Extensions/CrossCompiling.
In order to build the tools a native build environment is required. For 32-bit the developers mainly use the armhf port of Debian, which is present in Wheezy (the current testing distribution) running on an IMX53 based development board, although any ARMv7 development board would do. Note that the build hardware does not need to support the virtualisation extensions, since you don't have to run Xen on the same system as where you build it.
It may also be possible to build using a cross-architecture chroot running on an X86 system as described in this blog post, although this hasn't been tried yet.
Native build of the 64-bit tools has not been attempted yet.
Versatile Express specific configurations
- If you are trying to run Xen on a Versatile Express Cortex A15 machine, you might want to copy xen.bin (the Xen binary output) and the Linux Dom0 zImage to the SOFTWARE directory and add the two following stanzas to your images.txt file:
NOR3UPDATE: AUTO ;IMAGE UPDATE:NONE/AUTO/FORCE NOR3ADDRESS: 0c000000 ;Image Flash Address NOR3FILE: \SOFTWARE\Xen\xen.bin ;Image File Name NOR3LOAD: 80200000 ;Image Load Address NOR3ENTRY: 80200000 ;Image Entry Point NOR4UPDATE: AUTO ;IMAGE UPDATE:NONE/AUTO/FORCE NOR4ADDRESS: 000c0000 ;Image Flash Address NOR4FILE: \SOFTWARE\Xen\zImage ;Image File Name NOR4LOAD: 80008000 ;Image Load Address NOR4ENTRY: 80008000 ;Image Entry Point
- We recommend disabling the A7 cpus by editing the board.txt file under the SITE directory corresponding to your Cortex A15 daughterboard:
;SCC: 0x018 0x1FFFFFFF ;Reset control (CA7s running, CA15s running) - uncomment this for normal operation ;SCC: 0x018 0x1FFFF000 ;Reset control - (CA7s running, CA15s reset) - uncomment this to hold A15 cluster in reset SCC: 0x018 0x00001FFF ;Reset control - (CA7s reset, CA15s running) - uncomment this to hold A7 cluster in
- The address of the Dom0 kernel in Flash at the moment is not configurable, it is defined as KERNEL_FLASH_ADDRESS in Xen. On a Versatile Express machine this address can be determined by executing flash list images from the boot manager. You'll have to manually change it.
- In order to make your Dom0 boot sequence faster, you might want to consider passing init=/bin/bash to your kernel and then manually executing a minimal init script like this.
See this patch for a full list of projects.
- SMP support is under development in Xen ARM as we speak, we recommend exporting only 1 cpu in the DTS for the moment.
- Linux v3.8-rc2 fails to change the MMC voltage during boot on Versatile Express, a patch is needed to fix the bug.
Use the Xen ARM Dev FAQ page to cover commonly asked questions.