Getting Started with CheriBSD 23.11

Robert N. M. Watson (University of Cambridge) and Brooks Davis (SRI International)

This is a living document that describes how to get up and running with CheriBSD on Morello and CHERI-RISC-V. Topics include how to download and install prebuilt CheriBSD images, how to build your own images, how third-party packages work, and where to find further information and support.

The document describes CheriBSD as of the 23.11 release, unless explicitly stated in sections referring to earlier or later releases.

This document is a work-in-progress. Feedback and contributions are welcomed. Please see our GitHub Repository for the source code and an issue tracker. There is a rendered version on the web, which is automatically updated when the git repository is committed to.

Acknowledgements

The authors gratefully acknowledge Brian Campbell (University of Edinburgh), Jessica Clarke (University of Cambridge), George Neville-Neil (MSB Associates), and Konrad Witaszczyk (University of Cambridge) for their contributions to this document.

This work was supported by the Innovate UK project Digital Security by Design (DSbD) Technology Platform Prototype, 105694.

This material is based upon work supported by the Defense Advanced Research Projects Agency (DARPA) under Contract No. HR001122C0110 ("ETC"). Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the Defense Advanced Research Projects Agency (DARPA).

Introduction

CheriBSD is a version of the open-source FreeBSD operating system that has been extended to use CHERI architectural protections. CheriBSD is intended to:

Provide a clean, in-depth illustration and template for how CHERI support can be integrated with a general-purpose, MMU-enabled operating system;
Enable validation and evaluation of CHERI-extended processor architectures and microarchitectures;
Act as an OS research platform for how operating systems can use CHERI;
Support the development and evaluation of CHERI-enabled applications; and
Provide a foundation for CHERI-enabled software demonstrations.

This guide will support you in getting up and running with CheriBSD on the CHERI-RISC-V and Arm Morello platforms. As platforms vary substantially, from instruction-set emulators and executable formal models through to FPGA- and ASIC-based hardware implementations, there are several paths that can be taken to reach a shell prompt and start productive work. You might find that you want to build CheriBSD from scratch, or simply use one of our downloadable pre-compiled images. You may want to run directly from a live image in an emulator, or alternatively want to use an installer image to install onto an internal disk in an Arm Morello-based server or workstation.

Background

CHERI

CHERI is an extension to processor Instruction-Set Architectures (ISAs) to introduce support for fine-grained memory protection and software compartmentalization. This is done by introducing a new architectural data type, the CHERI capability, which can be used to implement pointers with strong integrity, provenance validity, spatial safety, and monotonicity. CHERI requires new microarchitectural support, including support for memory tags that track the provenance validity of capabilities in registers and memory. CHERI extensions have been specified for:

Arm's 64-bit Armv8-a ISA, with the prototype architecture known as Morello. The multi-GHz, multi-core, superscalar Morello design is a CHERI-extended version of the Arm Neoverse N1, available on an evaluation board from Arm.
The open-source 32-bit and 64-bit RISC-V ISA, with the extended ISA known as CHERI-RISC-V. CHERI-RISC-V is available via multiple open-source soft processor cores for use on FPGA.

ISA-level emulation platforms exist for both architectures (QEMU for CHERI-RISC-V and Morello, and Arm's Morello FVP for Morello). Unless you have access to a Morello board or a high-end (and supported) FPGA development platform, using QEMU-CHERI will be the easiest way to get started using CHERI. This guide covers all of these use cases.

You can learn more about CHERI by reading the technical report, An Introduction to CHERI.

CHERI C/C++

CHERI C and CHERI C++ are programming-language dialects closely tied to CHERI-based code generation and Application Binary Interfaces (ABIs) that directly support referential and spatial memory safety. These language variants compile to pure-capability code, which implements all language- and sub-language pointers using CHERI capabilities rather than architectural integers. With suitable OS support, they can also support temporal memory safety.

CHERI C/C++ support is implemented by CHERI LLVM. The Clang/LLVM compiler is able to statically identify and report as warnings many of the incompatibilites between C/C++ and CHERI C/C++ that would result in run-time CHERI exceptions. However, a correctly compiled CHERI C/C++ program might still trigger CHERI capability violations at run time (e.g., due to pointer misalignment) that can be diagnosed with GDB.

You can learn more about CHERI C/C++ and its CheriABI run-time environment by reading the technical report, CHERI C/C++ Programming Guide.

CheriBSD

CheriBSD is an extended version of the open-source FreeBSD operating system. FreeBSD is a mature UNIX-based operating system able to run on multiple hardware architectures, and widely used in industry in service, appliance, and embedded environments. CheriBSD contains various CHERI-based extensions including support for kernel and userspace memory safety, and supports multiple software compartmentalization models. CheriBSD runs on both CHERI-RISC-V and Morello with nearly identical feature sets (Morello includes hardware virtualisation extensions not present in the CHERI-RISC-V cores, and so CHERI-extended bhyve hypervisor support is not available for CHERI-RISC-V).

CheriBSD features

CheriBSD extends FreeBSD with added CHERI-enabled features, such as kernel and userspace memory safety:

Some of these features are in the main CheriBSD branch; other experimental features remain on development branches.

Kernel compilation modes

The CheriBSD kernel can be compiled either as hybrid or pure-capability code:

The hybrid kernel enables capability use in userspace while making relatively little use of capabilities in the kernel's own implementation.
The pure-capability kernel implements strong referential and spatial memory protection internally in the kernel, protecting against memory-safety vulnerabilities in components such as the network stack and system-call layer.

Userlevel process environments

The CheriBSD userspace likewise supports two different execution environments, hybrid processes and CheriABI (pure-capability) processes:

Hybrid processes provide strong binary compatibility with the non-CHERI version of the same architecture -- for example, aarch64 on Morello.
CheriABI processes implement strong referential and spatial memory protection through the system-call interface, dynamic linker, language runtime including heap memory allocators, and compiler-generated code. This protects against memory memory-safety vulnerabilities in both system services and applications. CheriABI is described in an ASPLOS 2019 paper.

Both environments can be used over either of the hybrid or pure-capability kernels.

Pre-compiled third-party software applications (packages) are provided for both ABIs, although CheriABI packages are currently considered experimental. This is discussed further in the chapter on packages.

CheriABI desktop environment (experimental)

The installer has the option to install a desktop environment using the Mali Bifrost GPU on the Morello System-on-Chip. The option installs a basic desktop environment using KDE and Wayland compiled for CheriABI with the cheri-desktop package. It also installs a hybrid ABI Chromium web browser via the cheri-desktop-hybrid-extras package.

Userlevel heap temporal memory safety (experimental)

CheriBSD implements, on an experimental development branch, support for the Cornucopia heap temporal safety algorithm, as well as successor algorithms based on load-side-barrier features present in the Morello prototype architecture and processor design. Cornucopia is described in an IEEE SSP 2020 paper.

Userlevel software compartmentalization (experimental)

CheriBSD implements two different CHERI-enabled software compartmentalization models:

Colocated processes (co-processes) compartmentalization accelerates UNIX Inter-Process Communication (IPC) and context switching by colocating multiple processes in the same address space, separating them using CHERI capabilities. Support for co-processes is maintained in an experimental development branch and is not included in current software releases. For more information see the Colocation Tutorial wiki page.
Dynamic-linker-based compartmentalization isolates shared libraries within a process using CHERI capabilities limiting the access of attackers who have achieved arbitrary code execution within a library. The linker-based library compartmentalization model has been included since the 22.12 release of CheriBSD. See the compartmentalization(7) manual page on an installed system for more information.

bhyve hypervisor (experimental)

CheriBSD implements, on an experimental development branch, CHERI extensions to FreeBSD's bhyve Type 2 hypervisor on the Morello architecture. This allows bhyve to host CHERI-enabled virtual machines, including those running CheriBSD.

Unsupported FreeBSD features

Certain FreeBSD features are currently unavailable in CheriBSD, including:

DTrace

Many other kernel features are not yet well validated, including:

Non-UFS file systems
Most optional modules, such as various firewall systems

Alpha ZFS support

The 23.11 release includes Alpha support for the ZFS file system. It is lightly tested, but works without known issues with a hybrid kernel. With the pure capability kernel, filesystems can be created and used, but it must not be the root file system as it currently hangs on boot.

Reporting bugs

Bug reports relating to these under-validated subsystems would be appreciated, and may be submitted to the CheriBSD GitHub issue tracker.

Getting CheriBSD

CheriBSD can be installed by downloading or building one of two types of disk images:

Memstick images that boot and automatically run bsdinstall, the FreeBSD installer, which can be used to prepare a filesystem on a disk, and then install CheriBSD onto it. These will typically be used on Arm Morello boards.
Live images that boot CheriBSD to a login prompt for interactive use. These will typically be used on instruction-set emulators such as QEMU and the Arm Morello FVP, as well as on FPGA.

Downloading image files

Memstick and live images can both be downloaded from the CheriBSD website. Images are compressed using the UNIX xz command, and must be decompressed before they can be used; for example by running:

unxz cheribsd-memstick-arm64-aarch64c-22.12.img.xz

Building image files

CheriBSD can be built natively on an existing CheriBSD system (e.g., on an Arm Morello box), and it can also be cross-built from FreeBSD, Linux, and macOS. Building natively from an ISA-level emulator, such as QEMU, is not recommended for performance reasons.

Native builds follow the same build procedure as used for the baseline FreeBSD operating system. Suitable git command-line substitions must be made to use CheriBSD rather than FreeBSD source code.
CheriBSD cross-build orchestration is performed by the Python-based cheribuild tool, which is able to build CHERI-enabled toolchain, operating-system stack, and various applications and other software used by the CHERI project.

Key source repositories

CheriBSD repositories currently reside in the CTSRD-CHERI GitHub Organization:

Unless you intend to modify CheriBSD, CHERI Clang/LLVM, CHERI GDB, or QEMU-CHERI, you should not generally need to manually check out or compile most of the above repositories. Instead, existing prebuilt software images/packages should be used, or the cheribuild command, which will orchestrate software cross-build for you.

Checking out cheribuild

cheribuild is a Python-based build orchestration tool that is the preferred way to cross-build CheriBSD. It can be checked out from GitHub:

git clone git@github.com:CTSRD-CHERI/cheribuild.git

Building and running CheriBSD/RISC-V in QEMU

The following command builds a CheriBSD/RISC-V live image, and boots it in QEMU-CHERI:

% ./cheribuild.py -d build-and-run-cheribsd-riscv64-purecap

Building and running CheriBSD/Morello in QEMU

The following command builds a CheriBSD/Morello live image, and boots it in QEMU-CHERI:

% ./cheribuild.py -d build-and-run-cheribsd-morello-purecap

Building CheriBSD/Morello release images

The following command builds a CheriBSD/Morello installer (memstick) image suitable to write to a USB stick:

% ./cheribuild.py -d --clean cheribsd-release-morello-purecap

The resulting image file will be generated in a file with a name along the lines of: output/cheribsd-release-morello-purecap/FreeBSD-14.0-CURRENT-arm64-aarch64c-memstick.img. This is relative to your cheribuild destination root, which is, by default, ~/cheri.

Obtaining further information

The cheribuild README.md provides detailed information on various parameters and targets its supports, as well as its dependencies.

CheriBSD on an Arm Morello board

Arm's Morello board contains a quadcore CHERI-extended Neoverse N1 processor design, as well as two management microcontrollers. CheriBSD is able to run on, and fully utilize, Morello's CHERI support to implement fine-grained kernel and userspace memory safety, and on experimental branches, also implement scalable software compartmentalization. You can install CheriBSD into the hard disk in your Morello box using a USB stick installer image. You will first need to upgrade the firmware, as older firmware revisions contain bugs or lack features essential to running the OS. This chapter explains how to:

Connect to the Morello management and system consoles via USB,
Upgrade the board's firmware, and
Install CheriBSD from a USB stick.

It is also possible to run CheriBSD from a USB stick, or netboot CheriBSD on the board, but those topics are not currently covered by this guide.

It is important that the most recent Arm-provided firmware be installed on your Morello board, as the board has most likely shipped with an older firmware version. If you are not running the latest version, you may experience hard-to-diagnose behaviour or reliability problems.

Accessing the Morello console

You can manage the Morello board, including reaching its multiple serial consoles and firmware flash storage, via a USB cable connected to a second computer -- typically a workstation or remote management server. When you plug the USB cable into your workstation, a number of different devices will attach (depending on device drivers on your workstation), including:

Arm Morello USB debug and trace
Flash storage used for firmware (an on-board SD card)
Two 4-port USB-to-serial converters

Each controller has a unique serial number ending in the letter "A" or B". The serial-port converters attach to UARTs associated with:

The MCC management microcontroller (converter B port 0)
The PCC management microcontroller (converter B port 1)
The Morello system console (converter B port 2)

In normal use, only the MCC and Morello system console ports will be used. You can connect to these using a serial access tool such as cu(1), or a full terminal emulation environment such as screen(1). The precise names of the /dev device nodes appearing on the workstation depend on the operating system being used. Often access will also require running the terminal command under su(8) or sudo(8), or adding your user to a dialer or similar group to grant access to the USB device nodes.

Connecting from FreeBSD

FreeBSD names USB device nodes in the order they probe and attach. Typical command lines to access the MCC and Morello consoles are:

MCC console: cu -l /dev/cuaU0 -s 115200
Morello console: cu -l /dev/cuaU2 -s 115200

Connecting from Apple macOS

macOS creates /dev/cu.usbserial- device nodes that embed the device serial number (including A or B) and port number. Typical command lines to access the MCC and Morello consoles are:

MCC console: cu -l /dev/cu.usbserial-00FT41683097B0 -s 115200
Morello console: cu -l /dev/cu.usbserial-00FT41683097B2 -s 115200

You will need to substitute the serial numbers of the USB converters in your specific Morello board for these commands to work.

Upgrading the Morello firmware

Newer Morello firmware versions contain important feature extensions (such as support for network booting) and also bug fixes (such as persistent variable support for UEFI, which is relied on for CheriBSD boot).

More detailed information from Arm on Morello firmware builds and configuration can be found here. The instructions in this guide are an abbreviated and slightly modified set of instructions better tuned to CheriBSD requirements.

Setting the system clock

Start by setting the system date and time in the MCC. Although the Arm documentation recommends setting your local time, we recommend using UTC with CheriBSD:

Cmd> debug
Debug> time
Debug> date
Debug> exit

Downloading prebuilt firmware

Arm provides prebuilt firmware images via the Morello GitLab instance, although you can also build your own. Start by downloading this directory tree, which can be done by performing a git clone:

git clone https://git.morello-project.org/morello/board-firmware.git

Reflashing the firmware

The Morello firmware is stored on a 2.0GiB SD Card on the Morello board within the case. The SD Card content can be exposed as a USB mass storage device accessible over the USB cable from your workstation. This may require first enabling USB support on the MCC console:

Cmd> USB_ON
Enabling debug USB...

The SD Card will attach as a USB disk containing a FAT filesystem. Depending on your workstation operating system, this may be automatically mounted (e.g., on macOS), or require manual mounting (e.g., on FreeBSD). Once mounted, delete the existing firmware and configuration, and then copy on the new version (substituting an appropriate mounted directory for /mnt below):

rm -rf /mnt/*
cp -R board-firmware/* /mnt
sync

Depending on your workstation OS and configuration, you may need to run as root, or use sudo(8), to execute these commands. Some shells may require confirmation before they will complete the rm command.

When done, you must unmount the mounted filesystem to ensure that all blocks in your workstation's filesystem buffer cache have been written back, and to avoid concurrent accesses leading to possible filesystem corruption. This must be done before issuing the MCC reboot command.

Rebooting the MCC

Complete the upgrade by rebooting the Morello board via the MCC:

Cmd> reboot
Rebooting...
Disabling debug USB..

Recovering the SD Card format

If you find that your SD Card no longer has a correct partition table (for example, if like the author you have written an installer image over it by accident) you will need to restore it to the original state with a FAT formated volume named M1SDP. The mechanism for creating such a filesystem will vary by OS. After you have created the FAT filesystem, follow the instruction above to copy the firmware into place and install it.

Linux

On Linux the following commands will create and format the FAT32 partition:

parted /dev/sdX mktable msdos
parted /dev/sdX mkpart primary fat32 0% 100%
mkfs.msdos /dev/sdX1

In this command /dev/sdX should be replaced by the path to the SD Card device.

MacOS

On MacOS the following command will create and format the M1SDP partition:

diskutil partitionDisk /dev/disk# 1 MBRFormat MS-DOS M1SDP 100%

In this command /dev/disk# should be replaced by the path to the SD Card device.

Installing on a Morello Board

Please ensure that you have upgraded the Morello board firmware before proceeding to CheriBSD installation.

Once you have a disk image, you will either need to write it to a USB stick to boot an Arm Morello system, or specify it as an argument to an emulator such as QEMU-CHERI or the Arm Morello FVP.

Writing an installer disk image to a USB stick

With appropriate substitutions of image filename and target device, the following command would write the image to a USB stick for use with a Morello board:

dd if=cheribsd-memstick-arm64-aarch64c-22.12.img of=/dev/DISK bs=1048576

It is also possible to write a live image to a USB stick, with appropriate filename substitution.

Installing an Arm Morello system

It is possible to use Arm Morello systems both by booting the live image on a USB stick, and also by installing to the internal hard disk. In this section, we describe the how to perform an installation using the memstick image.

Starting the boot

Before powering on your Morello board, ensure that the installer USB stick is inserted into one of the USB ports on the board. Use the power switch to power the board on, which should lead to firmware output on the MCC console during a multi-minute initialization and boot cycle.

Once that has completed, you will see Morello firmware output on the Morello console. When this message is presented, press Escape to enter the UEFI boot menu:

Press ESCAPE for boot options .......

At the UEFI menu, select Boot Manager using the cursor keys, and press Enter:

   Select Language            <Standard English>         This selection will
                                                         take you to the
 > Device Manager                                        Device Manager
 > Boot Manager
 > Boot Maintenance Manager

   Continue
   Reset







  ^v=Move Highlight       <Enter>=Select Entry

Select your USB stick from the Boot Manager menu using the cursor keys, and press enter. In the example below, it is the SanDisk Ultra USB 3.0 device:

/------------------------------------------------------------------------------\
|                                Boot Manager                                  |
\------------------------------------------------------------------------------/

                                                         Device Path :
   Boot Manager Menu                                     PcieRoot(0x0)/Pci(0x0,
                                                         0x0)/Pci(0x0,0x0)/Pci(
   UEFI Shell                                            0x14,0x0)/Pci(0x0,0x0)
   UEFI Misc Device                                      /USB(0x5,0x0)
   UEFI WDC WDS240G2G0A-00JH30 2143B2455605
   UEFI PXEv4 (MAC:0002F7009856)
   UEFI HTTPv4 (MAC:0002F7009856)
   UEFI HTTPv6 (MAC:0002F7009856)
   UEFI PXEv6 (MAC:0002F7009856)
   UEFI SanDisk Ultra USB 3.0
   0101cfbb32adfd6c145cd38e7d6164cd167456963c3a71b9f157
   e574567b43d96c6000000000000000000000e49dde42009f1a00
   955581079e275d95
   UEFI Non-Block Boot Device
                                                       v
/------------------------------------------------------------------------------\
|                                                                              |
| ^v=Move Highlight       <Enter>=Select Entry      Esc=Exit                   |
\------------------------------------------------------------------------------/

This will be followed by the CheriBSD boot loader and kernel output. You can press Enter to proceed, or wait for the countdown timer to finish:

    _____ _               _ ____   _____ _____
   / ____| |             (_)  _ \ / ____|  __ \
  | |    | |__   ___ _ __ _| |_) | (___ | |  | |
  | |    | '_ \ / _ \ '__| |  _ < \___ \| |  | |
  | |____| | | |  __/ |  | | |_) |____) | |__| |
   \_____|_| |_|\___|_|  |_|____/|_____/|_____/
                                                 ```                        `
                                                s` `.....---.......--.```   -/
 /---------- Welcome to CheriBSD ----------\    +o   .--`         /y:`      +.
 |                                         |     yo`:.            :o      `+-
 |  1. Boot Installer [Enter]              |      y/               -/`   -o/
 |  2. Boot Single user                    |     .-                  ::/sy+:.
 |  3. Escape to loader prompt             |     /                     `--  /
 |  4. Reboot                              |    `:                          :`
 |  5. Cons: Serial                        |    `:                          :`
 |                                         |     /                          /
 |  Options:                               |     .-                        -.
 |  6. Kernel: default/kernel (1 of 1)     |      --                      -.
 |  7. Boot Options                        |       `:`                  `:`
 |                                         |         .--             `--.
 |                                         |            .---.....----.
 \-----------------------------------------/
   Autoboot in 10 seconds. [Space] to pause

Note that seperate instances of the installer will run on both the video and serial consoles. Care should be taken to ensure that only one is used as no mechanism is implemented to prevent one from destroying the other's work.

On the serial console, you will eventually be prompted to select a terminal emulation type:

Welcome to FreeBSD!

Please choose the appropriate terminal type for your system.
Common console types are:
   ansi     Standard ANSI terminal
   vt100    VT100 or compatible terminal
   xterm    xterm terminal emulator (or compatible)
   cons25w  cons25w terminal

Console type [xterm]:

In most cases, simply hitting Enter will be the right thing to do. This will enter bsdinstall, the interactive FreeBSD installer.

bsdinstall

bsdinstall uses the libdialog text interface library. Be aware that you will sometimes need to use the space bar, and not enter key, to select menu options, which many users find confusing.

Select Install at the first menu by hitting Enter:

                     ┌────────────Welcome──────────────┐
                     │ Welcome to FreeBSD! Would you   │
                     │ like to begin an installation   │
                     │ or use the live CD?             │
                     ├─────────────────────────────────┤
                     │ <Install> < Shell > <Live CD>   │
                     └─────────────────────────────────┘

Keymap selection

If you intend only to use the serial console, and not video console, select the default keymap by hitting Enter at the next menu; otherwise, use the cursor keys and space bar to select a country-specific keyboard layout:

     ┌───────────────────────Keymap Selection──────────────────────────┐
     │ The system console driver for FreeBSD defaults to standard "US" │
     │ keyboard map. Other keymaps can be chosen below.                │
     │ ┌─────────────────────────────────────────────────────────────┐ │
     │ │>>> Continue with default keymap                             │ │
     │ │->- Test default keymap                                      │ │
     │ │( ) Armenian phonetic layout                                 │ │
     │ │( ) Belarusian                                               │ │
     │ │( ) Belgian                                                  │ │
     │ │( ) Belgian (accent keys)                                    │ │
     │ │( ) Brazilian (accent keys)                                  │ │
     │ │( ) Brazilian (without accent keys)                          │ │
     │ │( ) Bulgarian (BDS)                                          │ │
     │ │( ) Bulgarian (Phonetic)                                     │ │
     │ │( ) Canadian Bilingual                                       │ │
     │ └────↓(+)─────────────────────────────────────────────12%─────┘ │
     ├─────────────────────────────────────────────────────────────────┤
     │                   <Select>          <Cancel>                    │
     └──────────────────[Press arrows, TAB or ENTER]───────────────────┘

Hostname

Press Enter to accept the default hostname, or replace it:

           ┌───────────────────Set Hostname──────────────────────┐
           │ Please choose a hostname for this machine.          │
           │                                                     │
           │ If you are running on a managed network, please ask │
           │ your network administrator for an appropriate name. │
           │ ┌─────────────────────────────────────────────────┐ │
           │ │cheribsd                                         │ │
           │ └─────────────────────────────────────────────────┘ │
           ├─────────────────────────────────────────────────────┤
           │                     <  OK  >                        │
           └─────────────────────────────────────────────────────┘

Partitioning (automatic or manual)

Press Enter to select an automated UFS install:

          ┌─────────────────────Partitioning───────────────────────┐
          │ How would you like to partition your disk?             │
          │ ┌────────────────────────────────────────────────────┐ │
          │ │   Auto (UFS)  Guided UFS Disk Setup                │ │
          │ │   Auto (ZFS)  EXPERIMENTAL Guided Root-on-ZFS      │ │
          │ │   Manual      Manual Disk Setup (experts)          │ │
          │ │   Shell       Open a shell and partition by hand   │ │
          │ │                                                    │ │
          │ │                                                    │ │
          │ └────────────────────────────────────────────────────┘ │
          ├────────────────────────────────────────────────────────┤
          │               <  OK  >        <Cancel>                 │
          └────────────────────────────────────────────────────────┘

Partition (entire disk)

Select Entire Disk and press Enter:

                 ┌───────────────Partition──────────────────┐
                 │ Would you like to use this entire disk   │
                 │ (ada0) for FreeBSD or partition it to    │
                 │ share it with other operating systems?   │
                 │ Using the entire disk will erase any     │
                 │ data currently stored there.             │
                 ├──────────────────────────────────────────┤
                 │     <Entire Disk>   < Partition >        │
                 └──────────────────────────────────────────┘

Partition (entire disk - confirmation)

Press Enter to confirm:

                 ┌───────────────Partition──────────────────┐
                 │ Would┌──────Confirmation─────────┐disk   │
                 │ (ada0│ This will erase the disk. │  o    │
                 │ share│ Are you sure you want to  │  s?   │
                 │ Using│ proceed?                  │       │
                 │ data ├───────────────────────────┤       │
                 ├──────│     < Yes >   < No  >     │  ─────┤
                 │     <└───────────────────────────┘       │
                 └────────                             ─────┘

Partition Editor

Review the proposed partition scheme, select Finish, and then press Enter to proceed:

           ┌──────────────────Partition Editor────────────────────┐
           │ Please review the disk setup. When complete, press   │
           │ the Finish button.                                   │
           │┌────────────────────────────────────────────────────┐│
           ││ada0            224 GB  GPT                         ││
           ││  ada0p1        260 MB  efi            /boot/efi    ││
           ││  ada0p2        220 GB  freebsd-ufs    /            ││
           ││  ada0p3        3.6 GB  freebsd-swap   none         ││
           ││da0             57 GB   GPT                         ││
           ││  da0p1         33 MB   efi                         ││
           ││  da0p2         814 MB  freebsd-ufs                 ││
           ││                                                    ││
           │└────────────────────────────────────────────────────┘│
           ├──────────────────────────────────────────────────────┤
           │<Create> <Delete> <Modify> <Revert> < Auto > <Finish> │
           └──────────────────────────────────────────────────────┘

Partition editor - confirmation

Select Commit and press Enter to continue with writing out a new partition table:

           ┌──────────────────Partition Editor────────────────────┐
           │ Please review the disk setup. When complete, press   │
           │ the Finish button.                                   │
           │┌────────────────────────────────────────────────────┐│
           │┌──────────────────Confirmation─────────────────────┐││
           ││ Your changes will now be written to disk. If you  │
           ││ have chosen to overwrite existing data, it will   │
           ││ be PERMANENTLY ERASED. Are you sure you want to   │
           ││ commit your changes?                              │
           │├───────────────────────────────────────────────────┤
           ││ <   Commit    > <Revert & Exit> <    Back     >   │
           │└───────────────────────────────────────────────────┘
           │└─
           ├──────────────────────────────────────────────────────┤
           │<Create> <Delete> <Modify> <Revert> < Auto > <Finish> │
           └──────────────────────────────────────────────────────┘

Setting a root password

Enter a new root password and press Enter. Then confirm it by typing the same password again and pressing Enter.

Please select a password for the system management account (root):
Typed characters will not be visible.
Changing local password for root
New Password:

Network configuration

If desired, configure Ethernet networking by pressing Enter.

  ┌────────────────────────Network Configuration───────────────────────────┐
  │ Please select a network interface to configure:                        │
  │ ┌────────────────────────────────────────────────────────────────────┐ │
  │ │   re0  RealTek 8168/8111 B/C/CP/D/DP/E/F/G PCIe Gigabit Ethernet   │ │
  │ │                                                                    │ │
  │ │                                                                    │ │
  │ └────────────────────────────────────────────────────────────────────┘ │
  ├────────────────────────────────────────────────────────────────────────┤
  │                     <  OK  >           <Cancel>                        │
  └────────────────────────────────────────────────────────────────────────┘

Network configuration - enabling IPv4

If desired, enable IPv4 on the Ethernet interface by selecting Yes and pressing Enter:

                        ┌──Network Configuration────┐
                        │ Would you like to         │
                        │ configure IPv4 for this   │
                        │ interface?                │
                        ├───────────────────────────┤
                        │     < Yes >   < No  >     │
                        └───────────────────────────┘

Network configuration - DHCP for IPv4

If you will be using DHCP, select Yes and press Enter. Otherwise select No and press Enter to perform a manual IPv4 configuration.

                        ┌──Network Configuration────┐
                        │ Would you like to use     │
                        │ DHCP to configure this    │
                        │ interface?                │
                        ├───────────────────────────┤
                        │     < Yes >   < No  >     │
                        └───────────────────────────┘

Network configuration - enabling IPv6

If desired, enable IPv6 on the Ethernet interface by selecting Yes and pressing Enter:

                        ┌──Network Configuration────┐
                        │ Would you like to         │
                        │ configure IPv6 for this   │
                        │ interface?                │
                        ├───────────────────────────┤
                        │     < Yes >   < No  >     │
                        └───────────────────────────┘

Network configuration - SLAAC for IPv6

Press Enter to use stateless address autoconfiguration for IPv6:

                        ┌──Network Configuration────┐
                        │ Would you like to try     │
                        │ stateless address         │
                        │ autoconfiguration         │
                        │ (SLAAC)?                  │
                        ├───────────────────────────┤
                        │     < Yes >   < No  >     │
                        └───────────────────────────┘

Network configuration - resolver configuration

Enter DNS resolution configuration information, or press Enter to confirm autoconfigured DNS configuration:

   ┌───────────────────────Network Configuration──────────────────────────┐
   │ Resolver Configuration                                               │
   │ ┌──────────────────────────────────────────────────────────────────┐ │
   │ │Search                                                            │ │
   │ │IPv6 DNS #1                                                       │ │
   │ │IPv6 DNS #2                                                       │ │
   │ │IPv4 DNS #1    192.168.4.1                                        │ │
   │ │IPv4 DNS #2                                                       │ │
   │ └──────────────────────────────────────────────────────────────────┘ │
   ├──────────────────────────────────────────────────────────────────────┤
   │                     <  OK  >           <Cancel>                      │
   └──────────────────────────────────────────────────────────────────────┘

Local or UTC clock

Press Enter to select a system clock on UTC

   ┌───────────┤Select local or UTC (Greenwich Mean Time) clock├───────────┐
   │ Is this machine's CMOS clock set to UTC?  If it is set to local time, │
   │ or you don't know, please choose NO here!                             │
   │                                                                       │
   ├───────────────────────────────────────────────────────────────────────┤
   │                          [ Yes  ]   [  No  ]                          │
   └───────────────────────────────────────────────────────────────────────┘

Timezone selection

Select your continent and press Enter:

                     ┌───────┤Time Zone Selector├───────┐
                     │ Select a region                  │
                     │ ┌──────────────────────────────┐ │
                     │ │ 1 Africa                     │ │
                     │ │ 2 America -- North and South │ │
                     │ │ 3 Antarctica                 │ │
                     │ │ 4 Asia                       │ │
                     │ │ 5 Atlantic Ocean             │ │
                     │ │ 6 Australia                  │ │
                     │ │ 7 Europe                     │ │
                     │ │ 8 Indian Ocean               │ │
                     │ │ 9 Pacific Ocean              │ │
                     │ │ 0 UTC                        │ │
                     │ └──────────────────────────────┘ │
                     ├──────────────────────────────────┤
                     │        [  OK  ]   [Cancel]       │
                     └──────────────────────────────────┘

Then select your country and press Enter:

        ┌────────────────────┤Countries in Europe├────────────────────┐
        │ Select a country or region                                  │
        │ ┌─^^^─────────────────────────────────────────────────────┐ │
        │ │ 35 Poland                                               │ │
        │ │ 36 Portugal                                             │ │
        │ │ 37 Romania                                              │ │
        │ │ 38 Russian Federation                                   │ │
        │ │ 39 San Marino                                           │ │
        │ │ 40 Serbia                                               │ │
        │ │ 41 Slovakia                                             │ │
        │ │ 42 Slovenia                                             │ │
        │ │ 43 Spain                                                │ │
        │ │ 44 Svalbard and Jan Mayen                               │ │
        │ │ 45 Sweden                                               │ │
        │ │ 46 Switzerland                                          │ │
        │ │ 47 Turkey                                               │ │
        │ │ 48 Ukraine                                              │ │
        │ │ 49 United Kingdom of Great Britain and Northern Ireland │ │
        │ │ 50 Åland Islands                                        │ │
        │ └────────────────────────────────────────────────100%─────┘ │
        ├─────────────────────────────────────────────────────────────┤
        │                     [  OK  ]   [Cancel]                     │
        └─────────────────────────────────────────────────────────────┘

Confirm your choice by selecting Yes and pressing Enter:

   ┌────────────────────────────┤Confirmation├────────────────────────────┐
   │ Does the abbreviation `BST' look reasonable?                         │
   ├──────────────────────────────────────────────────────────────────────┤
   │                          [ Yes  ]   [  No  ]                         │
   └──────────────────────────────────────────────────────────────────────┘

Setting the date and time

As desired, set the date, or select Skip and press Enter if you plan to use network time synchronization:

                   ┌────────────Time & Date───────────────┐
                   │                                      │
                   │  Month            Year               │
                   │  ┌───────────────┐┌───────────────┐  │
                   │  │May            ││2022           │  │
                   │  └───────────────┘└───────────────┘  │
                   │  ┌─────↑(-)───────────────────────┐  │
                   │  │    Sun Mon Tue Wed Thu Fri Sat │  │
                   │  │ 18   1   2   3   4   5   6   7 │  │
                   │  │ 19   8   9  10  11  12  13  14 │  │
                   │  │ 20  15  16  17  18  19  20  21 │  │
                   │  │ 21  22  23  24  25  26  27  28 │  │
                   │  │ 22  29  30  31                 │  │
                   │  │                                │  │
                   │  └─────↓(+)───────────────────────┘  │
                   ├──────────────────────────────────────┤
                   │       <Set Date>    <  Skip  >       │
                   └──────────────────────────────────────┘

And, likewise, the time:

                   ┌────────────Time & Date───────────────┐
                   │                                      │
                   │           ┌──┐ ┌──┐ ┌──┐             │
                   │           │00│:│34│:│36│             │
                   │           └──┘ └──┘ └──┘             │
                   │                                      │
                   ├──────────────────────────────────────┤
                   │       <Set Time>    <  Skip  >       │
                   └──────────────────────────────────────┘

System configuration

As desired, enable any further services (e.g., ntpd and ntpdate) by selecting them and hitting Space:

  ┌────────────────────────System Configuration───────────────────────────┐
  │ Choose the services you would like to be started at boot:             │
  │ ┌───────────────────────────────────────────────────────────────────┐ │
  │ │ [ ] local_unbound  Local caching validating resolver              │ │
  │ │ [*] sshd           Secure shell daemon                            │ │
  │ │ [ ] moused         PS/2 mouse pointer on console                  │ │
  │ │ [ ] ntpdate        Synchronize system and network time at bootime │ │
  │ │ [ ] ntpd           Synchronize system and network time            │ │
  │ │ [ ] powerd         Adjust CPU frequency dynamically if supported  │ │
  │ │ [*] dumpdev        Enable kernel crash dumps to /var/crash        │ │
  │ │                                                                   │ │
  │ │                                                                   │ │
  │ └───────────────────────────────────────────────────────────────────┘ │
  ├───────────────────────────────────────────────────────────────────────┤
  │                               <  OK  >                                │
  └───────────────────────────────────────────────────────────────────────┘

Then press Enter to continue.

Add user accounts

If you plan to install the desktop environment in the next step, it is useful to create user accounts now. If desired, select Yes and press Enter to add non-root accounts. Otherwise, select No and press Enter

CHERI Desktop environment

If you wish to configure your CheriBSD system with a KDE-based desktop environment you can do so now. NOTE: You must have previously configured the network and be able to reach pkg.CheriBSD.org.

                      ┌─────────┤CHERI Desktop├────────┐
                      │ Would you like to install the  │
                      │ CHERI desktop environment      │
                      │ (requires network)?            │
                      ├────────────────────────────────┤
                      │      [ Yes  ]     [  No  ]     │
                      └────────────────────────────────┘

Select Yes to install (NOTE: this may take 20-50 minutes depending on network conditions).

Adding users to video group

In order to be able to use the keyboard and mouse when logged in via the Simple Desktop Display Manager (SDDM), users must be in the video group. If users were added earlier, you will be prompted to select ones to add to the group.

                  ┌─────────────┤CHERI Desktop├─────────────┐
                  │ Users must be in the video group to log │
                  │ in to a desktop environment. Choose any │
                  │ additional users to add to the group:   │
                  │ ┌─────────────────────────────────────┐ │
                  │ │       [ ] localuser Local User      │ │
                  │ └─────────────────────────────────────┘ │
                  ├─────────────────────────────────────────┤
                  │                [  OK  ]                 │
                  └─────────────────────────────────────────┘

Use Space to select users to add and then press Enter.

Final configuration

If there are further configuration settings to change, select an appropriate option from the menu and press Space. Otherwise, press Enter to complete the installation:

┌───────────────────────────Final Configuration──────────────────────────────┐
│ Setup of your FreeBSD system is nearly complete. You can now modify your   │
│ configuration choices. After this screen, you will have an opportunity to  │
│ make more complex changes using a shell.                                   │
│ ┌────────────────────────────────────────────────────────────────────────┐ │
│ │Exit              Apply configuration and exit installer                │ │
│ │Add User          Add a user to the system                              │ │
│ │Root Password     Change root password                                  │ │
│ │Hostname          Set system hostname                                   │ │
│ │Network           Networking configuration                              │ │
│ │Services          Set daemons to run on startup                         │ │
│ │System Hardening  Set security options                                  │ │
│ │Time Zone         Set system timezone                                   │ │
│ │Handbook          Install FreeBSD Handbook (requires network)           │ │
│ │CHERI Desktop     Install the CHERI desktop environment (requires networ│ │
│ │                                                                        │ │
│ │                                                                        │ │
│ │                                                                        │ │
│ └────────────────────────────────────────────────────────────────────────┘ │
├────────────────────────────────────────────────────────────────────────────┤
│                                 <  OK  >                                   │
└────────────────────────────────────────────────────────────────────────────┘

Manual configuration

If desired, select Yes and press Enter to drop to a shell before reboot. Otherwise, select No and press Enter.

                   ┌───────Manual Configuration──────────┐
                   │ The installation is now finished.   │
                   │ Before exiting the installer, would │
                   │ you like to open a shell in the new │
                   │ system to make any final manual     │
                   │ modifications?                      │
                   ├─────────────────────────────────────┤
                   │         < Yes >     < No  >         │
                   └─────────────────────────────────────┘

Complete

Select Reboot and press Enter:

                        ┌─────────Complete───────────┐
                        │ Installation of FreeBSD    │
                        │ complete! Would you like   │
                        │ to reboot into the         │
                        │ installed system now?      │
                        ├────────────────────────────┤
                        │   <Reboot >  <Live CD>     │
                        └────────────────────────────┘

Rebooting after installation

Remove the USB stick to prevent the installer from running after a system reboot. Due to a Morello firmware bug, OS-triggered reboot is not reliable. Until a firmware revision correcting this bug is released, you will need to type reboot on the management console to reboot after installation. CheriBSD will print messages along the following lines once it is safe to reboot:

Waiting (max 60 seconds) for system process `syncer' to stop...
Syncing disks, vnodes remaining... 24 0 0 done
All buffers synced.
Uptime: 3m12s

Logging in

...
Mounting late filesystems:.
Security policy loaded: MAC/ntpd (mac_ntpd)
Starting ntpd.
Starting cron.
Starting sendmail_submit.
Starting sendmail_msp_queue.
Performing sanity check on sshd configuration.
Starting sshd.
Starting background file system checks in 60 seconds.

Sun May  8 00:13:04 BST
CheriBSD/arm64 (cheribsd) (ttyu0)

login: root
Password:
May  8 00:13:25 cheribsd login[772]: ROOT LOGIN (root) ON ttyu0
Last login: Sat May  7 23:41:04 on ttyu0
FreeBSD 14.0-CURRENT #0 dev-d4897febcde: Tue May  3 13:34:49 BST 2022     rnw24@zeno.sec.cl.cam.ac.uk:/home/rnw24/cheri/build/cheribsd-morello-purecap-build/home/rnw24/cheri/cheribsd/arm64.aarch64c/sys/GENERIC-MORELLO

Welcome to CheriBSD!

CheriBSD extends FreeBSD to implement memory protection and software
compartmentalization features available in CHERI-extended CPUs.

We provide support via a mailing list:
  https://www.cl.cam.ac.uk/research/security/ctsrd/cheri/cheri-lists.html

And also via the CHERI-CPU Slack:
  https://www.cl.cam.ac.uk/research/security/ctsrd/cheri/cheri-slack.html

CheriBSD source may be found at:
  https://github.com/CTSRD-CHERI/cheribsd/

Find out more about about CHERI at https://cheri-cpu.org/
You have new mail.
root@cheribsd:~ #

You can now add further users, install SSH keys for remote access, install packages, and so on.

Morello known issues

The following known issues may affect the reliability and usability of CheriBSD on Morello:

If a USB hub is plugged in (including embedded hubs in USB keyboards) the system will pause on boot due to a firmware bug. Some people find they can bypass this pause by pressing Enter on a USB keyboard attached to the Morello desktop system. Others find they must unplug the hub to boot.

Sometimes the ATA controller starts timing out. and displaying messages like this on the console:

ahcich0: Timeout on slot 12 port 0
ahcich0: is 00000000 cs 00003000 ss 00000000 rs 00003000 tfd 50 serr 00000000 cmd 00718c17
(ada0:ahcich0:0:0:0): WRITE_DMA. ACB: ca 00 50 8a 8b 41 00 00 00 00 d0 00
(ada0:ahcich0:0:0:0): CAM status: Command timeout
(ada0:ahcich0:0:0:0): Retrying command, 3 more tries remain

A reboot usually cures the problem. If this happens during install, it's generally best to start over as performance is awful once timeouts start.

The HDMI transmitter on the board might fail to detect if a monitor is plugged in. In such a case, the monitor will turn blank when the CheriBSD kernel starts printing messages to the console, after the CheriBSD boot loader menu. A user can configure the kernel to ignore hot plug detection and assume that the attached monitor can receive HDMI input. To do that, add the following line to /boot/loader.conf and reboot your board:
```
hw.tda19988.broken_hpd="1"
```
While the screen itself should be working with this workaround, the user can still experience EDID reading failures described below.
The HDMI transmitter on the board might fail to detect the monitor's resolution. The CheriBSD kernel writes the following message to the console in such a case:
```
tda0: Failed to read EDID
```
Currently, there is no workaround for this issue and it is likely to occur with the HDMI transmitter issue described above.
The ZFS port is unstable with pure-capability kernels. While it does support creation of file systems it hangs while mounting a ZFS root file system either as a module or compiled in to the kernel. In general it is lightly tested and should be used with caution.
We have witnessed one apparent deadlock in ZFS during an install. Dozens of other installs have completed without issue so this is not common.

Issues with older firmware

The following issues affected users of older firmware revisions:

Due to a firmware bug, OS-triggered reboot may lead the system to hang when using older firmware. Update your firmware or use the "reboot" command on the management console to reboot the system once the OS has shut down.
Due to a firmware bug, some earlier firmware versions are not able to persist UEFI configuration variables. This affects, for example, the installation environment as the OS installer cannot instruct the firmware to boot only from the installed target, and not further USB sticks.

It is extremely important that you use the most recent firmware with your Morello board, to avoid these and other issues present in earlier firmware releases.

Third-party packages

CheriBSD on Morello ships with both hybrid ABI and CheriABI packages (compilations) of the CheriBSD ports collection, each targeting a different form of code generation and Application Binary Interface (ABI). They have different levels of completeness, maturity, security, and support.

CheriBSD ports extends FreeBSD ports, a collection of over 30,000 third-party software adaptations to FreeBSD, with CHERI- and CheriBSD-specific patches. The releng/22.12 CheriBSD ports branch contains ports matching the packages built for the current release.

Overview

The following table presents an overview of available package managers in CheriBSD that are described in more details in consecutive sections. You can also browse package repositories at pkg.CheriBSD.org to check what packages are available for a specific ABI version.

ABI	#	Manager	Install path	Suitable for	Examples
Hybrid ABI	~20,000	`pkg64`	`/usr/local64`	Day-to-day use	bash gdb-cheri llvm-base vim
CheriABI	~8,000	`pkg64c`	`/usr/local`	Experimental use	git nano rsync sudo

Package managers in CheriBSD

Note: As of this writing we only provide packages for Morello systems. We aim to add CHERI-RISC-V package sets in the near future.

CheriBSD includes two package managers:

pkg64 for hybrid ABI packages;
pkg64c for CheriABI packages.

The FreeBSD package manager pkg is not available on CheriBSD. We expect that pkg64c will be renamed to pkg in a future CheriBSD release. The intention is that, over time, the CheriABI packages will become more mature, and hence the preferred collection for day-to-day use.

The syntax of the pkg64 and pkg64c commands match the syntax of the pkg command from FreeBSD. You can find information on package manager commands in FreeBSD's pkg manual pages shipped with CheriBSD, e.g. for the commands pkg64 rquery and pkg64c rquery, execute man pkg-rquery.

Additionally, you can learn more about FreeBSD's package manager by reading the pkg(8) manual page online and the FreeBSD Handbook chapter on packages and ports.

Hybrid ABI packages

Hybrid ABI packages are compiled almost identically to packages in the baseline non-CHERI architecture (e.g., Armv8-a for Morello, and 64-bit RISC-V for RISC-V), and do not have improvements in memory protection or software compartmentalization.

These packages are considered appropriate for day-to-day use. They are intended to provide stable versions of tools necessary to develop software and use your CHERI system while more software is ported to CheriABI.

There are currently over 20,000 hybrid ABI packages available, including:

Compilers:
- llvm-morello;
  
  Includes Clang (a CHERI C/C++ compiler), LLD (a linker), and the LLVM infrastructure for the Arm Morello architecture. Binaries installed with this package have the suffix -morello.
- llvm;
  
  Adds links in /usr/local64/bin to allow the llvm-morello package to be used as the default LLVM package, without the suffix -morello, e.g. clang instead of clang-morello.
- llvm-base.
  
  Adds links in /usr/bin to allow the llvm package to be used in place of a base-system toolchain. The cc script installed with this package adds compiler flags required to natively compile code for a CHERI-enabled architecture.
Debuggers: gdb-cheri;
Editors: nano, vim;
Shells: bash, dash, fish.

The packages are installed in the /usr/local64 hierarchy. By default /usr/local64/bin and /usr/local64/sbin should be included in your PATH environment variable of a default shell shipped with CheriBSD. If you are planning to use a custom shell, remember to add these paths to PATH.

CheriABI packages

CheriABI packages are compiled using pure-capability CHERI C/C++, and employ fine-grained C/C++ memory protection.

These packages are considered appropriate for experimental use. Their primary function is to provide necessary dependencies for efforts to port software to CheriABI and to support CHERI demonstration and evaluation. They are suitable for research and development of software that benefit from spatial and temporal memory safety as well as software compartmentalisation. They can also be used to investigate potential memory safety issues in third-party software that are easier to detect and debug using a CHERI-enabled hardware-software stack.

There are currently over 8,000 CheriABI packages available, including:

Development utilities: git;
Editors: nano;
Libraries: libnghttp2, libressl, libssh2, lzlib, wolfssl;
Networking tools: rsync;
Security tools: sudo;
Shells: dash.

The packages are installed in the standard /usr/local hierarchy as they match the base system ABI.

Limitations of packages

When using or even considering third-party software delivered with CheriBSD packages, it is important to keep in mind the maturity of currently available CheriABI packages, and potential conflicts between hybrid ABI and CheriABI packages. The following sections describe some of such limitations.

Maturity of CheriABI packages

There are many projects that must still be ported to CHERI in order to provide a fully-functional CheriABI package repository. The available CheriABI packages are also not continuously tested as part of the CheriBSD package building infrastructure.

CheriABI packages compile, but many have CHERI-related warnings that have not been audited, and only a limited set have been tested. Whilst the CheriABI packages are more interesting as they use CHERI memory-safety features, they might break at run time and hence might not be suitable for critical operations. In such cases, a corresponding hybrid ABI package might be better suited than the CheriABI package.

We expect the number of available and tested packages to grow with future CheriBSD releases, but it will take quite some time until we are able to provide a CheriABI package repository that covers functionalities currently included in the hybrid ABI package repository.

We encourage everyone to report bugs related to CHERI support in the CheriBSD ports issue tracker.

Separate local bases for ABIs

Upstream FreeBSD does not have the ability to manage packages for multiple ABIs at the same time and its pkg package manager installs all packages in the default local base path (/usr/local). CheriBSD introduces a workaround for this issue by using separate local base paths for different package repositories.

The hybrid ABI packages are compiled with a local base set to /usr/local64 while the CheriABI packages use the default local base. Because the hybrid ABI packages are installed in a non-standard location, there may be bugs resulting in hybrid ABI packages looking for its dependencies in /usr/local instead of /usr/local64. For example, a CheriBSD port a package was built from might fail to configure a project's build system to use this custom local base, or an upstream project might hardcode /usr/local in its source code.

We encourage everyone to report bugs related to invalid local base paths in the hybrid ABI packages in our CheriBSD ports issue tracker.

Cross-ABI package conflicts

Hybrid ABI and CheriABI package repositories include many counterpart packages, e.g. git is available in both repositories. CheriABI packages have a higher priority than the hybrid ABI packages in default PATH environment variables in CheriBSD. If you install a CheriABI package and a hybrid ABI package with a conflicting program name, you must execute the hybrid ABI program using an absolute path by default.

When using a shell configuration not included in CheriBSD's source code, a user must remember to appropriately set the PATH environment variable to take into account both /usr/local and /usr/local64 paths. The user can set the order of the paths they prefer but it is recommended to place /usr/local before /usr/local64 in such configurations to match CheriBSD's defaults.

Cross-ABI package dependencies

At the moment, a package cannot depend at run time on another package with a different ABI, e.g. to execute a program provided by that package. Such a feature would be useful for CheriBSD ports which cannot easily be adapted to CheriABI and which include programs that could be executed by other ports already adapted to CheriABI. There are currently no plans to support this case.

Missing packages

Hybrid ABI and CheriABI package repositories might be missing third-party software due to the following reasons:

There is no FreeBSD port including that software or it is broken on AArch64;

Check FreshPorts, FreeBSD Wiki, FreeBSD Bugzilla and the freebsd-ports mailing list to find out if anyone is working on software you are interested in.
A CheriBSD port with the software is broken on Morello (e.g., it is not adapted to CheriABI), or does not respect a custom local base (/usr/local64) when compiled for the hybrid ABI.

Check the CheriBSD ports issue tracker to find out if an issue with the software is documented and the Poudriere infrastructure for CheriBSD packages to find failed build logs and skipped build causes.

Upgrading packages

It is recommended to upgrade packages in the following situations:

Before building a newer CheriBSD release;

The release might require an updated toolchain (e.g., due to ABI changes in the release that must be supported by the toolchain).
After updating your host to a newer CheriBSD release;

Package repositories are compiled for the ABI version used by the corresponding release, which might differ from the ABI of the previous release.

Where practical, we intend to support packages from the immediately prior release on the newer release, but will make no effort to ensure that package repositories from even older releases continue to work.

Additionally, package repositories for the newer release will most likely include updated third-party software versions that are tested against that release.
After finding a bug in a package (e.g., a library, a toolchain).

It might happen that the bug you discovered has already been reported and fixed. If the package upgrade does not resolve your issue, please report the bug.

Use the following commands to upgrade your packages:

CheriABI	Hybrid ABI
`pkg64c upgrade`	`pkg64 upgrade`

If you do not want perform the upgrade itself but only check if newer versions are available, use the -n flag:

CheriABI	Hybrid ABI
`pkg64c upgrade -n`	`pkg64 upgrade -n`

You can read more on the upgrade command at pkg-upgrade(8) or in your console with man pkg-upgrade.

Useful commands

The following list includes useful commands to manage packages. Replace <pkg> with pkg64c to execute a command in the context of CheriABI packages and with pkg64 in the context of hybrid ABI packages.

Most pkg commands have separate man pages describing them. For a command <pkg> foo, you can read its man page by executing man pkg-foo on a CheriBSD host.

List available package manager commands:
```
<pkg> help
```
Read more information on a package manager command:
```
<pkg> help <command>
```
Search a package repository:
```
<pkg> search <pattern>
```
Search a package repository and display full information:
```
<pkg> search --full <pattern>
```
List installed packages:
```
<pkg> info
```
Display information for an installed package:
```
<pkg> info <pkg-name>
```
Install a package:
```
<pkg> install <pkg-name>
```
Delete a package:
```
<pkg> delete <pkg-name>
```
Delete packages that are no longer required:
```
<pkg> autoremove
```
Check a new package version for an installed package:
```
<pkg> upgrade -n <pkg-name>
```
Check new package versions for all installed packages:
```
<pkg> upgrade -n
```
Upgrade a package after confirmation:
```
<pkg> upgrade <pkg-name>
```
Upgrade all packages after confirmation:
```
<pkg> upgrade
```
Display which package installed a file:
```
<pkg> which /path/to/file
```
Check a number of installed packages:
```
<pkg> stats -l
```
Check a number of available packages:
```
<pkg> stats -r
```

Compiling "Hello World"

These instructions are intended for use on an Arm Morello board, and install hybrid ABI versions of key toolchain and utilities using pkg64. The commands will (eventually) work the same on CHERI-RISC-V, but output details and registers in GDB will differ.

Toolchain installation

You will need the Morello LLVM toolchain and Morello GDB for this exercise. To install LLVM, use the command:

pkg64 install llvm-base

If this is the first time you are using pkg64 on this system, you will be prompted to bootstrap the pkg package before the package database is downloaded and you are prompted to confirm before installing llvm and its dependencies:

root@cheribsd:~ # pkg64 install llvm-base
The package management tool is not yet installed on your system.
Do you want to fetch and install it now? [y/N]: y
Bootstrapping pkg from pkg+http://pkg.CheriBSD.org/CheriBSD:20220314:aarch64, please wait...
Verifying signature with trusted certificate pkg.cheribsd.org.2022032901... done
Installing pkg-1.17.5_1...
Extracting pkg-1.17.5_1: 100%
Updating CheriBSD repository catalogue...
Fetching meta.conf: 100%    163 B   0.2kB/s    00:01
Fetching packagesite.pkg: 100%    4 MiB   1.4MB/s    00:03
Processing entries: 100%
CheriBSD repository update completed. 20954 packages processed.
All repositories are up to date.
Updating database digests format: 100%
The following 12 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
        gettext-runtime: 0.21
        indexinfo: 0.3.1
        libedit: 3.1.20210910,1
        libffi: 3.3_1
        libxml2: 2.9.13_2
        llvm: 13,1
        llvm-base: 1
        llvm-morello: 13.0.d20220502_1
        mpdecimal: 2.5.1
        perl5: 5.32.1_1
        python38: 3.8.13
        readline: 8.1.2

Number of packages to be installed: 12

The process will require 902 MiB more space.
190 MiB to be downloaded.

Proceed with this action? [y/N]: y
[1/12] Fetching llvm-base-1.pkg: 100%    3 KiB   2.8kB/s    00:01
[2/12] Fetching llvm-13,1.pkg: 100%   11 KiB  11.2kB/s    00:01
...
[12/12] Fetching libedit-3.1.20210910,1.pkg: 100%  119 KiB 121.8kB/s    00:01
Checking integrity... done (0 conflicting)
[1/12] Installing indexinfo-0.3.1...
[1/12] Extracting indexinfo-0.3.1: 100%
...
[12/12] Installing llvm-base-1...
[12/12] Extracting llvm-base-1: 100%
=====
...

Note: By default FreeBSD ships with the vi and ee editors. You may wish to install the nano or vim package to access a more familiar editor. Currently only hybrid packages installable with pkg64 are available.

Source code

#include <stdio.h>

int
main(void)
{
	printf("Hello world\n");
}

Building

To build a CheriABI Hello World program use:

cc -g -O2 -Wall -o helloworld helloworld.c

You can verify this is a CheriABI binary with the file command:

root@cheribsd:~ # file helloworld
helloworld: ELF 64-bit LSB pie executable, ARM aarch64, C64, CheriABI, version 1 (SYSV), dynamically linked, interpreter /libexec/ld-elf.so.1, for FreeBSD 14.0 (1400094), FreeBSD-style, with debug_info, not stripped

To target the Benchmark ABI, add the argument -mabi=purecap-benchmark to the cc command line:

cc -g -O2 -Wall -mabi=purecap-benchmark -o helloworld helloworld.c

You can verify this is a Benchmark ABI binary with the file command:

root@cheribsd:~ # file helloworld
helloworld: ELF 64-bit LSB pie executable, ARM aarch64, C64, CheriABI, version 1 (SYSV), dynamically linked, interpreter /libexec/ld-elf.so.1, for FreeBSD 14.0 (1400094), FreeBSD-style, pure-capability benchmark ABI, with debug_info, not stripped

Running

Run the program:

root@cheribsd:~ # ./helloworld
Hello world

Debugging

First, if it is not already installed, install the CHERI GDB debugger:

pkg64 install gdb-cheri

You can then debug helloworld by running GDB and setting a breakpoint on main:

root@cheribsd:~ # gdb ./helloworld
GNU gdb (GDB) 12.1
...
Reading symbols from ./helloworld...
...
(gdb) b main
Breakpoint 1 at 0x10a14: file helloworld.c, line 6.
(gdb) r
Starting program: /root/helloworld

Breakpoint 1, main () at helloworld.c:6
6           printf("Hello world\n");
(gdb)

If you then step into the printf (or puts depending on optimization level) call you will see that GDB prints the capability argument with expanded information including bounds and permissions:

(gdb) s
puts (s=0x1006e0 [rR,0x1006e0-0x1006ec] "Hello world")
    at /home/bed22/cheri/cheribsd/lib/libc/stdio/puts.c:60
60      /home/bed22/cheri/cheribsd/lib/libc/stdio/puts.c: No such file or directory.

The argument can also be examined directly as either an integer or capability register:

(gdb) info reg x0
x0             0x1006e0            1050336
(gdb) info reg c0
c0             0x905f400046ec06e000000000001006e0 0x1006e0 [rR,0x1006e0-0x1006ec]
(gdb)

If a capability (either in memory or a register) contains a null-derived value, it is displayed as a simple scalar value without any attributes. If a capability is not null-derived but is untagged, the string "(invalid)" is displayed after the bounds.

Capability tags in memory can also be examined by passing the "m" flag to the eXamine command. For example, compare the output of the internal FILE structure used for stdout in libc:

(gdb) p *__stdoutp
$1 = {_p = 0x0, _r = 0, _w = 0, _flags = 8, _file = 1, _bf = {_base = 0x0, 
    _size = 0}, _lbfsize = 0, 
  _cookie = 0x403a8400 [rwRWE,0x403a8230-0x403a87a0], 
  _close = 0x402bade5 <__sclose> [rxRE,0x4018e000-0x407f0000] (sentry), 
  _read = 0x402bada5 <__sread> [rxRE,0x4018e000-0x407f0000] (sentry), 
  _seek = 0x402baddd <__sseek> [rxRE,0x4018e000-0x407f0000] (sentry), 
  _write = 0x402badc1 <__swrite> [rxRE,0x4018e000-0x407f0000] (sentry), _ub = {
    _base = 0x0, _size = 0}, _up = 0x0, _ur = 0, _ubuf = "\000\000", 
  _nbuf = "", _lb = {_base = 0x0, _size = 0}, _blksize = 0, _offset = 0, 
  _fl_mutex = 0x0, _fl_owner = 0x0, _fl_count = 0, _orientation = 0, 
  _mbstate = {__mbstate8 = '\000' <repeats 127 times>, _mbstateL = 0, 
    _mbstateP = 0x0 }, _flags2 = 0}
(gdb) x/16gxm __stdoutp
<CHERI Tag 0 for range [0x403a8400,0x403a8410)>
0x403a8400:     0x0000000000000000      0x0000000000000000
<CHERI Tag 0 for range [0x403a8410,0x403a8420)>
0x403a8410:     0x0000000000000000      0x0000000000010008
<CHERI Tag 0 for range [0x403a8420,0x403a8430)>
0x403a8420:     0x0000000000000000      0x0000000000000000
<CHERI Tag 0 for range [0x403a8430,0x403a8440)>
0x403a8430:     0x0000000000000000      0x0000000000000000
<CHERI Tag 0 for range [0x403a8440,0x403a8450)>
0x403a8440:     0x0000000000000000      0x0000000000000000
<CHERI Tag 1 for range [0x403a8450,0x403a8460)>
0x403a8450:     0x00000000403a8400      0xdc5fc00047a08230
<CHERI Tag 1 for range [0x403a8460,0x403a8470)>
0x403a8460:     0x00000000402bade5      0xb05fc000bf0618e7
<CHERI Tag 1 for range [0x403a8470,0x403a8480)>
0x403a8470:     0x00000000402bada5      0xb05fc000bf0618e7

Benchmarking guidance

CheriBSD is a research operating system designed to run on experimental hardware. In this section, we provide high-level guidance on configuring CheriBSD for software benchmarking.

Framing

The expectation of this section is that a performance comparison is being made between baseline aarch64-compiled software (64-bit Arm) and memory-safe aarch64c software (CheriABI) on the Arm Morello board.

Before proceeding to the remainder of this section, it is essential to first read Early performance results from the prototype Morello microarchitecture. That document provides detailed information on how to interpret performance measurements, including documenting known limitations of the prototype Morello microarchitecture.

A key conclusion from that work is that CheriABI software being used for performance measurement should be compiled for the aarch64cb "Benchmark ABI", and not aarch64c.

Kernel configuration

Depending on the version of CheriBSD you are using, the kernel may have a number of debugging fetures enabled. These features can substantially impact system performance, including inducing disproportionate performance overhead for specific system behaviours. For example, enabling kernel lock-order checking ("WITNESS") will introduce substantial overhead, and in particular will impact kernel-centric workloads that make more intensive use of locks, such as networking. During boot, CheriBSD will add the following lines to /etc/motd, which will be displayed at login, to warn you about the "INVARIANTS" and "WITNESS" debugging features:

WARNING: INVARIANTS kernel option defined, expect reduced performance
WARNING: WITNESS kernel option defined, expect reduced performance

Performance measurements employing a hybrid kernel should use the kernel.GENERIC-MORELLO-NODEBUG configuration when kernel memory safety is not required. The following line can be added to /boot/loader.conf:

kernel="kernel.GENERIC-MORELLO-NODEBUG"

Measurements requiring a memory-safe kernel should use the kernel.GENERIC-MORELLO-PURECAP-NODEBUG configuration. The following can be added to /boot/loader.conf:

kernel="kernel.GENERIC-MORELLO-PURECAP-NODEBUG"

You will need to reboot in order for this change to take effect.

Heap temporal memory safety

As of CheriBSD 23.11, userlevel heap temporal safety is enabled by default with jemalloc as the memory allocator. This support is experimental, and has not yet been through significant performance analysis and optimization. We recommend disabling temporal safety support for the full system during benchmarking not specifically intended to capture temporal safety performance. During boot, CheriBSD will add the following line to /etc/motd, which will be displayed at login, to warn you about the temporal safety feature:

WARNING: capability revocation enabled by default, this may affect performance

The following line can be added to /boot/loader.conf:

security.cheri.runtime_revocation_default=0

You will need to reboot in order for this change to take effect.

The Benchmark ABI

As of CheriBSD 23.11, CheriBSD supports the Benchmark ABI, a modified form of code generation improving performance on the Arm Morello board. This is required due to limitations on bounds prediction in the current Morello prototype, which would be resolved in a production microarchitecture. The performance of the Benchmark ABI is more predictive of potential future CHERI microarchitectural performance than the Morello prototype running software compiled for CheriABI, making it preferable for benchmarking. However, this comes at the cost of reduce security, and so software compiled for the Benchmark ABI should not be used for security evaluation. Programs may be compiled to the Benchmark ABI using the -mabi=purecap-benchmark command-line argument to cc.

More information on compiling with the Benchmark ABI can be found in Compiling Hello World. More information on what the Benchmark ABI is, and how to interpret performance results, can be found in Early performance results from the prototype Morello microarchitecture.

Getting help

CheriBSD is a community-supported open-source research operating system. While every effort has been made to engineer to a high quality, many aspects of OS design for CHERI processors remain active research, and there are constrained engineering resources available to build, maintain, and support the system. As you gain experience with CheriBSD, your help in supporting newcomers in the community will be greatly appreciated.

GitHub issue tracker and pull requests

CheriBSD is developed and maintained in the CheriBSD GitHub repository, which includes an issue tracker in which bugs can be reported, as well as permitting pull requests to be submitted.

This document is also maintained in a GitHub repository, and your feedback and improvements would be very much appreciated.

CheriBSD Slack on cheri-cpu.slack.com

Support for CheriBSD is provided via the CHERI CPU Slack, where a number of topic-specific channels can be found:

#cheribsd;

CheriBSD, its features, releases and future plans.
#debuggers;

GDB extended to support CHERI.
#qemu;

QEMU-CHERI, QEMU system and user modes for Morello and CHERI-RISC-V.
#software-porting.

Third-party software adaptations to CheriABI and CHERI-enabled achitectures, CheriBSD ports and Poudriere.

CheriBSD mailing lists

There are also several CheriBSD email lists available:

CheriBSD announcement mailing list (low volume)
CheriBSD general discussion
CheriBSD ports and packages discussion
CheriBSD security vulnerabilities contact address cl-cheribsd-security@lists.cam.ac.uk

In-person events and meetings

You can meet with the CheriBSD developers at a number of conferences and other events, including:

UKRI Digital Security by Design (DSbD) All Hands meetings that take place in the UK twice annually to discuss the Morello architecture, board, and its use.
FreeBSD developer summits and conferences around the world, including (when it takes place) the BSDCan conference in Canada, and the EuroBSDCon conference in Europe.

There is discussion around creating further, more frequent CHERI workshops in the UK in the future.

Most CheriBSD developers are based at the University of Cambridge (UK) and SRI International (USA), and it is possible to set up meetings with them directly. Get in touch if this would be useful to you.

Resources

In addition to the CheriBSD support channels, the following resources may also be useful.