CheriBSD 23.11 new features tutorial

Robert N. M. Watson (University of Cambridge), Konrad Witaszczyk (University of Cambridge), and Jessica Man (University of Cambridge)

Acknowledgements

This work was supported by the Innovate UK project Digital Security by Design (DSbD) Technology Platform Prototype, 105694. This software was developed by SRI International, the University of Cambridge Computer Laboratory (Department of Computer Science and Technology), and Capabilities Limited under Defense Advanced Research Projects Agency (DARPA) Contract No. HR001122S0003 ("MTSS").

Introduction

During the workshop, you should be given an ssh(1) command and a password that you can use to log in to your dedicated work environment. If you do not have access to the work environment, please talk to us during the workshop.

The environment is one of FreeBSD jails on a Morello box running a recent CheriBSD dev branch snapshot that includes features that will be available in CheriBSD 23.11. The running kernel is the default kernel shipped with CheriBSD releases - the hybrid GENERIC-MORELLO kernel with INVARIANTS and WITNESS features enabled.

There are several pre-installed packages in your jail to save you time with preparing your environment:

CheriABI git, nano, tmux and chericat
Hybrid ABI llvm-base, gdb-cheri and vim

Once you are logged in, you can switch to the root user with sudo su -l and install any package you want with pkg64c, pkg64 or pkg64cb package managers, as explained in the Third-party packages section of the Getting Started with CheriBSD guide. You can also install your SSH key in ~/.ssh/authorized_keys not to be required to type in a user password to connect.

The source code of this document with example exercises are stored in the ~/tutorial directory of your jail.

You can use your jail after the workshop to experiment with CheriBSD in your free time. We will remove your access after one week, on November 15.

Benchmark ABI

The Benchmark ABI makes minor changes to pure-capability code generation to address a limitation in Morello's branch prediction when PCC bounds may change. This limitation penalizes the performance of function calls and returns where PCC bounds are required shortly after the jump, leading to instruction stalls. Benchmark ABI code uses integer jumps rather than capability jumps, and Benchmark ABI processes use a PCC with bounds covering the full address space. Further details regarding Morello performance, and the Benchmark ABI, may be found in Early performance results from the prototype Morello microarchitecture.

This activity involves source code in the benchmark subdirectory:

cd ~/tutorial/src/benchmark

Compiling code for the Benchmark ABI

Compile helloworld.c as a CheriABI binary:

cc -Wall -g -o helloworld-cheriabi helloworld.c

Compile helloworld.c as a Benchmark ABI binary:

cc -Wall -g -mabi=purecap-benchmark -o helloworld-benchmark helloworld.c

Identifying Benchmark ABI binaries

Use the file(1) command to identify the two binaries:

file helloworld-cheriabi

file helloworld-benchmark

Inspect ELF binary headers with readelf (from the base system) or llvm-readelf (from LLVM for Morello):

readelf -n helloworld-cheriabi

readelf -n helloworld-benchmark

llvm-readelf -n helloworld-cheriabi

llvm-readelf -n helloworld-benchmark

Running Benchmark ABI binaries

Run both binaries from the command line:

./helloworld-cheriabi

./helloworld-benchmark

The Benchmark ABI package manager

Using pkg64cb, which manages a complete set of third-party software packages compiled for the Benchmark ABI, list the currently installed Benchmark ABI package:

pkg64cb info

Now install the Benchmark ABI compilation of bash:

sudo pkg64cb install bash

You can read more on useful package manager commands in the Getting Started with CheriBSD guide.

Identifying Benchmark ABI processes

In one terminal window, run the Benchmark ABI version of the bash shell with

/usr/local64cb/bin/bash

and execute in the bash shell session

echo $$

to print the process's PID. Note that PID down.

In a second terminal window, run

procstat -a

to list the ABIs for all running processes.

What is shown for your Benchmark ABI bash process with the PID you noted down, and how does this differ from other processes you see?

Disassembling Benchmark ABI binaries

Use objdump to disassemble the main() functions in both binaries:

objdump -dj .text ./helloworld-cheriabi

objdump -dj .text ./helloworld-benchmark

What differences exist between the two functions, and why?

Debugging Benchmark ABI binaries

Use GDB to analyse CPU register contents just before and after returning from the main() functions in both binaries.

First, run gdb:

gdb ./helloworld-cheriabi

Once you enter a GDB session,

Disassemble the main() function:
```
disassemble main
```
Record the offset of the RET instruction at the end of disassembly, e.g. 56 in
```
0x00000000001107ec <+56>:    ret     c30
```
Set a breakpoint for the RET instruction:
```
break *main + N
```
where N is the offset you recorded, e.g.
```
break *main + 56
```
Run the program:
```
run
```
Disassemble the current function to make sure the process was suspended at the RET instruction:
```
disassemble
```
Display the C30 and PCC registers:
```
info register c30 pcc
```
Execute the RET instruction:
```
stepi
```
Display the PCC register again:
```
info register pcc
```

Repeat the GDB session for the Benchmark ABI binary:

gdb ./helloworld-benchmark

What differences exist between the two functions, and why?

Benchmarking with the Benchmark ABI

Compile the source code for two short C programs, benchmark-atoi.c and benchmark-sha256.c:

cc -Wall -g -o benchmark-atoi-cheriabi benchmark-atoi.c -lmd

cc -Wall -g -mabi=purecap-benchmark -o benchmark-atoi-benchmarkabi benchmark-atoi.c -lmd

cc -Wall -g -o benchmark-sha256-cheriabi benchmark-sha256.c -lmd

cc -Wall -g -mabi=purecap-benchmark -o benchmark-sha256-benchmarkabi benchmark-sha256.c -lmd

Using the UNIX time(1) command, run each of the resulting binaries:

time ./benchmark-atoi-cheriabi

time ./benchmark-atoi-benchmarkabi

time ./benchmark-sha256-cheriabi

time ./benchmark-sha256-benchmarkabi

How long does execution take for the CheriABI vs Benchmark ABI compilations of each program?
Review the source code for the two workloads; why do they perform the way that they do?

Heap temporal memory safety

CheriBSD 23.11 incorporates support for userlevel heap temporal memory safety based on a load-barrier extension to Cornucopia, which is inspired by garbage-collection techniques.

This feature involves a collaboration between the kernel (which provides asynchronous capability revocation with VM acceleration) and the userlevel heap allocator (which quarantines freed memory until revocation of any pointers to it) to ensure that memory cannot be reallocated until there are no outstanding valid capabilities lasting from its previous allocation. The userlevel memory allocator and the kernel revoker share an epoch counter that counts the number of completed atomic revocation sweeps. Memory added to quarantine in one epoch cannot be removed from quarantine until at least one complete epoch has passed -- i.e., the epoch counter has been increased by 2. More information on temporal memory safety support can be found in the mrs(3) man page:

man mrs

This activity involves source code in the temporal subdirectory:

cd ~/tutorial/src/temporal

Checking whether temporal safety is globally enabled

Use the sysctl(8) command to inspect the value of the security.cheri.runtime_revocation_default system MIB entry:

sysctl security.cheri.runtime_revocation_default

This sysctl sets the default policy for revocation used by processes on startup. We recommend setting this in /boot/loader.conf, which is processed by the boot loader before any user processes start.

Controlling revocation by binary or process

You can forcefully enable or disable revocations for a specific binary or process with elfctl(1) or proccontrol(1) and ignore the default policy:

elfctl -e <+cherirevoke or +nocherirevoke> <binary>

proccontrol -m cherirevoke -s <enable or disable> <program with arguments>

You can read more on these commands in the mrs(3) man page.

Exercising a use-after-free bug

Compile the program use-after-free.c:

cc -Wall -g -o use-after-free use-after-free.c

Run the program to see what happens when a use-after-free bug is exercised:

./use-after-free

Why doesn't the program crash?

Synchronous revocation

Revocation normally occurs asynchronously, with a memory quarantine preventing memory reuse until revocation of any pointers to that memory. Uncomment this line in use-after-free.c to trigger a synchronous revocation before the use-after-free memory access:

/* malloc_revoke(). */

Recompile and re-run the program, this time under GDB, and observe its behavior.

Which line in the program faults, and why?

Monitoring revocation in processes

Use the procstat cheri -v command to inspect the CHERI memory-safety state of a target process. For example:

# procstat cheri -v 923 1012
  PID COMM                C QUAR  RSTATE                              EPOCH
  923 seatd               P  yes    none                                  0
 1012 Xorg                P  yes    none                               0xd2

Both processes in this example use a pure-capability process environment, have quarantining enabled, and are not currently revoking. seatd has never needed to perform a revocation pass, as it remains in epoch 0, whereas X.org has a non-zero epoch and has performed multiple passes.

Modify use-after-free.c to await user input before and after the call to cheri_revoke() using the POSIX gets_s(3) API; run use-after-free. In a second login session, use the ps(1) command to obtain the process ID, and then use procstat cheri to inspects its protection state on either side of the revocation call.

What is EPOCH before and after calling cheri_revoke() -- and why?

Library compartmentalization

CheriBSD's library compartmentalization feature (c18n) executes each dynamic library within a compartmentalization-enabled process in its own protection domain. The non-default c18n-enabled run-time linker grants libraries capabilities only to resources (global variables, APIs) declared in their ELF linkage. Function calls that cross domain boundaries are interposed on by domain-crossing shims implemented by the run-time linker.

The adversary model for these compartments is one of trusted code but untrustworthy execution: a library such as libpng or libjpeg is trusted until it begins dynamic execution -- and has potentially been exposed to malicious data. With library compartmentalization, an adversary who achieves arbitrary code execution within the library at run time will be able to reach only the resources (and further attack surfaces) declared statically through its linkage. The programmer must then harden that linkage, and any involved APIs, to make them suitable for adversarial engagement -- but the foundation of isolation, controlled access, and controlled domain transition is provided by the c18n implementation.

In addition to a modified run-time linker, modest changes have been made to the aarch64c calling convention to avoid assumptions such as implicit stack sharing between callers and callees across library boundaries when passing variadic argument lists. This modified ABI is now used by all CheriABI binaries in CheriBSD, and so off-the-shelf aarch64c binaries and libraries can be used with library compartmentalization without recompilation to the modified ABI. More information on library compartmentalization can be found in the c18n(3) man page:

man c18n

This activity involves source code in the temporal subdirectory:

cd ~/tutorial/src/c18n

Compiling applications for library compartmentalization

To compile a main application to use library compartmentalization, add the following flags to compilation of the program binary:

-Wl,--dynamic-linker=/libexec/ld-elf-c18n.so.1

For example, compile our helloworld.c example using:

cc -Wall -g -o helloworld helloworld.c -Wl,--dynamic-linker=/libexec/ld-elf-c18n.so.1

You can confirm whether a binary uses the c18n run-time linker by inspecting its INTERP field using the readelf -l command:

readelf -l helloworld

Tracing compartment-boundary crossings

The BSD ktrace(1) command is able to trace compartment-boundary crossings. To enable this feature, set the LD_C18N_UTRACE_COMPARTMENT environmental variable, which will cause the c18n run-time linker to emit records using the utrace(2) system call. Run the program under ktrace with the -tu argument to capture only those records (and not a full system-call trace):

env LD_C18N_UTRACE_COMPARTMENT=1 ktrace -tu ./helloworld

The resulting ktrace.out file can be viewed using the kdump(1) command:

kdump

Exercise

A classic motivation for software compartmentalization is to separate less trustworthy I/O-processing routines (which are more easily subject to compromise) from keying material. We have constructed a simple application consisting of three C files:

passwordcheck.c contains the global variable 'the_password' and the function passwordcheck() that checks the offered password against the defined password.
io.c contains the API readpassword(), which uses the legacy C API fscanf().
main.c calls readpassword() followed by passwordcheck(), and if it succeeds, will print the global variable secret.

Compile the three C files as a CheriABI binary, check.cheriabi:

cc -Wall -g -o check.cheriabi main.c io.c passwordcheck.c

Run the program and enter the password Password123 to print the secret.

Next, run the program and enter the password password123456789, which will crash due to a capability bounds violation.

Why does the program crash? While CHERI memory safety is able to catch this specific vulnerability, it is relatively easy to imagine non-memory-safety vulnerabilities in I/O handling, which could lead to arbitrary code execution.
What is an example of an I/O data processing vulnerability that CHERI memory safety would not mitigate?

To understand the implications of such vulnerability, we can use objdump to see what data will be visible to a compromised main program. Run objdump --full-contents to hexdump the full program binary, whose .text and .code sections include those available to the program at run time:

objdump --full-contents check.cheriabi

This includes the hard-coded password in the password-checking routine.

To mitigate these vulnerabilities, we will recompile the program and place the I/O routine in its own library -- and hence its own compartment when using the c18n run-time linker:

cc -Wall -shared -g -o libio.so io.c

cc -Wall -g -o check.c18n main.c passwordcheck.c -Wl,--dynamic-linker=/libexec/ld-elf-c18n.so.1 -L. -lio

Now run the program:

env LD_C18N_LIBRARY_PATH=. ./check.c18n

The program will print its PID, which you can then use as an argument to chericat to dump its capability state (replace PID with the printed value):

chericat -f check.c18n.db -p PID

chericat -f check.c18n.db -c libio.so

What capabilities can the attacker reach?
How do they differ from those available to the attacker in the uncompartmentalized case?

It is important to understand, however, that simply isolating running code is almost always insufficient to achieve robust sandboxing. The competent adversary will now consider further rights and attack surfaces to explore in search of further vulnerabilities. While this increased work factor of finding additional vulnerabilities is an important part of compartmentalization, internal software APIs are rarely well suited to be security boundaries without performing additional hardening. With this in mind:

Inspecting the source code, ouput from objdump, and output from chericat, assess the robustness of this compartmentalization: How might a highly competent adversary try to escape the sandbox?
What larger software architectural steps may be required to allow library compartmentalization to be used more robustly for this kind of use case?

Library compartmentalization has the potential to significantly improve software integrity and confidentiality properties in the presence of a strong adversary. However, it is also limited by the abstraction being around the current library operational model.

What are the implications of library compartmentalization on availability?