*bsd — scrolls.anduin.net

FreeBSD: Wrapped jails

August 22, 2026

As my first post here, I'll try to describe, and perhaps rationalise, how I prefer to run most of my jails.

Background

In case anyone reading this doesn't already know, FreeBSD is a Unix-like operating system tracing its heritage back to the original Berkeley Software Distribution (BSD) from the late 70s. Fast forward to the early '00s and the release of FreeBSD 4.0, and we got a new toy to play with: Jails.

In many ways, jails were the precursor and at least partially the inspiration for several other implementations of OS-level virtualization, with Solaris and Docker being fairly well-known examples. While the implementations differ significantly, the basic idea is the same: Virtualize OS services and resources, but not the hardware, thus achieveing much more frugal virtualization than full-blown VMs.

Basics

Historically, FreeBSD jails were quite simple in their implementation: Aside from separating process spaces (grouping processes and preventing a jail from seeing processes in other jails or on the host) and a rudimentary IP address delegation mechanism, it was essentially (though perhaps not technically) an extension of the chroot mechanism that was first introduced in BSD as early as 1979 – admittedly in a much different shape than today.

Today, this has changed quite a bit, with perhaps the most interesting change being the introduction of VIMAGE, a framework for virtualizing kernel subsystems, and the VNET facility built using VIMAGE, which provides a separate, virtualized network stack to jails.

Add some other killer FreeBSD- and jail-specific features like ZFS and hierarchical, or nested, jails, the PF firewall, and you have the building blocks for some really interesting things!

The problem with classic jails

Classic jails are nice because the lack of a network stack can be considered a security feature: The processes in the jail, even those running as root, are completely isolated from touching the network stack, which means they cannot do anything useful with ifconfig(8), can't modify routing tables, and are generally strictly limited to the IP address(es) you assign to the jail.

Unfortunately, this comes with some drawbacks: Running ifconfig in the jail will reveal much about the network configuration on the host: It will basically give you the full output for the interface(s) on which the jail has IPs, minus the other IPs that are defined there. Consider the following:

$  ifconfig                                                                                                             
ix0: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
options=a538b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,HWSTATS>
        ether ac:1f:6b:62:17:1c
        media: Ethernet autoselect (10Gbase-T <full-duplex>)
        status: active
ix1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=4e53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
        ether ac:1f:6b:62:17:1d
        media: Ethernet autoselect
        status: no carrier
lo0: flags=1008049<UP,LOOPBACK,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 16384
options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        groups: lo
pflog0: flags=0 metric 0 mtu 33152
        options=0
        groups: pflog
ngeth0: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        options=28<VLAN_MTU,JUMBO_MTU>
        ether 58:9c:fc:10:64:f9
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
bridge0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        options=10<VLAN_HWTAGGING>
        ether ac:1f:6b:62:17:1c
        inet 10.1.2.3 netmask 0xffffff00 broadcast 10.1.2.255
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        bridge flags=0<>
        member: epair2a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                port 15 priority 128 path cost 2000 vlan protocol 802.1q
        member: epair1a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                port 11 priority 128 path cost 2000 vlan protocol 802.1q
        member: epair0a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                port 7 priority 128 path cost 2000 vlan protocol 802.1q
        member: ngeth0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                port 5 priority 128 path cost 20000 vlan protocol 802.1q
        member: ix0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                port 1 priority 128 path cost 2000 vlan protocol 802.1q
        groups: bridge
epair0a: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        description: jail1
        options=200019<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,RXCSUM_IPV6>
        ether 58:9c:fc:10:e6:88
        groups: epair
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
epair1a: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        description: jail2
        options=200019<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,RXCSUM_IPV6>
        ether 58:9c:fc:10:98:28
        groups: epair
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
epair2a: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        description: jail3
        options=200019<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,RXCSUM_IPV6>
        ether 58:9c:fc:10:22:09
        groups: epair
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active

This ifconfig invocation is from within a classic jail running on the same host as a number of VNET jails. Despite not being able to modify anything, I can still see a lot of detail about the host that, one would think, the jail should not be privy to: All the physical and virtual interfaces on the machine, including lo0 (localhost), the bridge0 interface and all the bridge members, the primary interface ix0 and its twin ix1, the pflog0 interface revealing we've got pf running, ngeth0 showing there's netgraph(4) in use, etc.

And perhaps worst of all: The individual epair interfaces for the VNET jails, including their descriptions which – in a well-assembled system – could give away details about what and who else exists on the host.

Clearly, this is not the prison I was hoping for.

The problem with VNET jails

Quite the opposite of classic jails, VNET jails provide a virtualised network stack. This can be connected to the outside world with a virtual cable, by way of the epair interface type, connecting it to a bridge interface on the host. Or you can assign separate physical interfaces, VLAN interfaces, or practically any other interface type. Either way, the jail controls that interface in practically every way, in addition to the routing table (or tables), packet filtering, and most other aspects of the network stack.

This is obviously very powerful, and provides some features that classic jails are (sorely?) missing:

An actual localhost interface! Classic jails don't have this, there are just some code shortcuts in the kernel that allow applications that bind to localhost to .. not fail, but there are several situations where this is far from optimal or even desired.
The ability to run stuff like nmap(1) from the jail, since it fiddles with routing tables and creates raw network packets.
Firewalling with pf(4) (or possibly others; I have not played with those). This includes basic filtering, NAT, rate limiting, and all the other goodness offered.
Several network-related sysctl(8) settings can be tuned on a per-jail basis, like datagram sizes, routing settings, TCP congestion control algorithm, etc.

However: Do you really want all the processes in your jail to be this close to all those foot cannons? A simple misconfiguration, local privilege escalation bug, or simply fat-fingering something yourself can wreck havoc on the jail's network configuration – and there is a definite possibility of inflicting damage on other jails, the host, and even other devices on your network.

The limitations of a classic jail suddenly start looking an awful lot like features, don't they?

A possible solution

Without more ado (there's enough of that above), here's how we've solved this at work: We “wrap” a classic jail inside a VNET jail.

Here's how:

1. Create the jail template

The easiest way is to fetch base.txz from the FreeBSD download servers, and estract it in a suitable directory:

# Create template directory. If you use ZFS, a dataset might be preferable:
mkdir -p /data/jails/template.151

# Fetch and extract
fetch -o - https://download.freebsd.org/ftp/releases/amd64/15.1-RELEASE/base.txz \
    | tar --unlink -xpzf - -C /data/jails/template.151

# Create a symlink for use in `fstab(5)`
ln -s /data/jails/template.151 /data/jails/current

2. Create `fstab` files

For the parent and child jails, create fstab(5) files to nullfs(4)-mount the parts of the base you need into the jails. An added benefit is that this makes the base OS read-only inside the jails, which is probably a good thing:

# /etc/fstab.testjail.parent
/data/jails/current/etc /data/jails/testjail/etc nullfs ro
/data/jails/current/bin /data/jails/testjail/bin nullfs ro
/data/jails/current/lib /data/jails/testjail/lib nullfs ro
/data/jails/current/sbin /data/jails/testjail/sbin nullfs ro
/data/jails/current/libexec /data/jails/testjail/libexec nullfs ro
/data/jails/current/usr/sbin /data/jails/testjail/usr/sbin nullfs ro

# /etc/fstab.testjail
/data/jails/current/bin /data/jails/testjail/root/bin nullfs ro
/data/jails/current/lib /data/jails/testjail/root/lib nullfs ro
/data/jails/current/usr/share /data/jails/testjail/root/usr/share nullfs ro
/data/jails/current/libexec /data/jails/testjail/root/libexec nullfs ro
/data/jails/current/sbin /data/jails/testjail/root/sbin nullfs ro
/data/jails/current/usr/bin /data/jails/testjail/root/usr/bin nullfs ro
/data/jails/current/usr/include /data/jails/testjail/root/usr/include nullfs ro
/data/jails/current/usr/lib /data/jails/testjail/root/usr/lib nullfs ro
/data/jails/current/usr/libdata /data/jails/testjail/root/usr/libdata nullfs ro
/data/jails/current/usr/libexec /data/jails/testjail/root/usr/libexec nullfs ro
/data/jails/current/usr/sbin /data/jails/testjail/root/usr/sbin nullfs ro

3. Create the directory structures for the jails

This is all greatly simplified, but it should give you a starting point. Note how the child jail lives in a subdirectory (/root) inside the parent jail:

# Create the parent jail directory structure
for dir in \
    root dev etc bin lib sbin libexec \
    usr/sbin /var/log /var/run /var/empty
do
    mkdir -p /data/jails/testjail/$dir
done

# Copy some files into the parent jail; necessary for pf to work
cp /data/jails/template.151/etc/pf.os /data/jails/testjail/etc/
cp /data/jails/template.151/etc/protocols /data/jails/testjail/etc/

# Create the child jail directory structure
for dir in \
    root dev bin lib sbin libexec tmp \
    usr/sbin usr/bin usr/include usr/lib usr/libdata \
    usr/libexec usr/share /usr/local/etc
do
    mkdir -p /data/jails/testjail/root/$dir
done

# Copy basic /etc and /var to the child jail
cp -Rp /data/jails/template.151/etc /data/jails/testjail/root/
cp -Rp /data/jails/template.151/var /data/jails/testjail/root/

Using ZFS instead

The mkdir parts for the child jail could be modified to use ZFS (which is what I'd do). This is a skeleton suggestion, but you'd do well in forming your own strategy for how to approach this with ZFS. Personally I see no point using ZFS datasets for the parent jail, as it is largely ephemeral anyway:

# Create the child jail datasets (zfs 'create -p' acts like 'mkdir -p')
for dir in \
    root usr/local usr/home tmp var etc
do
    zfs create -p data/jails/testjail/root/$dir
done

# Create directories in the datasets
for dir in \
    dev bin lib sbin libexec \
    usr/sbin usr/bin usr/include usr/lib usr/libdata \
    usr/libexec usr/share /usr/local/etc
do
    mkdir -p /data/jails/testjail/root/$dir
done

One useful trick I've learned is to keep /var/log inside the child jail in a ZFS dataset homed outside the child jail's path, and simply set its mountpoint to be inside the jail. In case you need to roll back the dataset(s) in the child jail, you can safely do so without erasing your logs. Security people and auditors may even come to appreciate this:

# Create the dataset; make sure it always has space to write logs even if the
# rest of the datasets have run out of space:
zfs create -p \
    -o reservation=64M \
    -o mountpoint=/data/jails/testjail/root/var/log \
    data/jails/testjail/logs

4. Start the jails

We start by creating the epair interfaces. Assuming there is already a bridge0 to attach the dangling cable^Winterface to:

if_a="$(ifconfig epair create)"
if_b="${if_a%a}b"
ifconfig bridge0 addm $if_a
ifconfig $if_a up

Next, we start the parent jail. Note this is crude; a complete jail.conf entry should take care to unmount filesystems on jail shutdown, avoiding mounting things twice, and various other housekeeping:

# Mount the parent filesystems
mount -a -F /etc/fstab.testjail.parent

# Start the jail - set children.max=1 to allow the child jail
jail -c path=/data/jails/testjail name=testjail children.max=1 vnet persist

# $if_a and $if_b is from the interface create script
ifconfig $if_b vnet testjail
jexec testjail ifconfig lo0 127.0.0.1/8
jexec testjail ifconfig $if_b 10.2.3.4/24
jexec testjail route add default 10.2.3.1

# Mount devfs in the jails
mount -t devfs -oruleset=4 . /data/jails/testjail/dev
mount -t devfs -oruleset=4 . /data/jails/testjail/root/dev

Now that we have the parent jail running, starting the child jail is a matter of

# Mount the child filesystems
mount -a -F /etc/fstab.testjail.parent

# Start the child jail
jexec testjail jail -c ip4=inherit path=/root name=child exec.start="/bin/sh /etc/rc autoboot"

At this point you should be able to jexec testjail.child su - and find yourself in a bare, but functional, jail. Run ifconfig to see that only network interfaces pertaining to testjail are visible, and confirm that you cannot make changes. List the jails with jls; it should show two jails – the parent and the child.

Stopping the jails

Since we're cheating and doing everything from the command line here, stopping the jails takes a few steps:

# Stop the child jail by killing all the processes
jexec testjail.child kill -9 -1

# In case there are lingering TCP sockets preventing the child jail from dying,
# use tcpdrop(8) to clean up:
jexec testjail tcpdrop -a

# Destroy the parent jail
jail -r testjail

# Destroy the epair interfaces (find the name using ifconfig if needed; both
# the 'a' and 'b' of the pair should now be visible on the host:
ifconfig $if_a destroy

# Finally, unmount the filesystems
umount -f /data/jails/testjail/root/dev
umount -f /data/jails/testjail/dev
umount -F /etc/fstab.testjail -a
umount -F /etc/fstab.testjail.parent -a

What now?

All of the above, while lengthy, is hardly elegant. Some additional steps may be desirable, e.g. to enable pf, creating a “black hole” around the child jail, and other niceties. A jail.conf taking care of these steps should also be created, which will require a helper script or two.

Enabling the packet filter

In order to use pf, we need a pf.conf file, and some additional device nodes and optionally the pflog interface in the parent jail.

The pf.conf file really only needs a single line for the purpose of this demonstration. We're adding log here for pflog to have any work to do. Place it in the root of the parent jail, like so:

echo "pass log all" > /data/jails/testjail/pf.conf

Then we need to unhide a couple of device nodes in the parent jail, load the rules, and start pflogd. This should be done after starting the parent jail, but before starting the child jail:

devfs -m /data/jails/testjail/dev rule apply path pf unhide
devfs -m /data/jails/testjail/dev rule apply path bpf unhide
jexec testjail pfctl -F all -vf /pf.conf && pfctl -e

# Starting pflogd, excluding some particularly noisy traffic
jexec testjail pflogd 'not host 224.0.0.18'

At this point, if you plan to keep a configuration like this, you may want to make sure you rotate the pflogd files automatically, by adding this to /usr/local/etc/newsyslog.d/jails.conf (or anywhere else you prefer): /data/jails/testjail/var/log/pflog root:wheel 640 7 * $D0 B /data/jails/nextcloud/var/run/pflogd.pid Ask me how I know. :-P

The Black Hole

One of the more “interesting” possibilities with this approach is that after the child jail has been started, there really is no reason to keep the parent jail around. Of course it needs to exist, but most of the filesystems (i.e binaries and libraries) are really not necessary. While nowhere near an actual black hole (you can still access the child jail's files), it has a non-zero value as an obfuscation.

Following startup of the child jail and all the parent jail services (pflogd, etc.), you can simply unmount all the parent filesystems: umount -F /etc/fstab.testjail.parent

However, a side effect of this is that you'll no longer be able to modify the running configuration (update pf rules, etc.). You can no longer simply jexec into the jail – no libraries or binaries are available in their expected locations.

To get around this, you can use some variation of the following: LD_LIBRARY_PATH=/root/lib jexec testjail /root/libexec/ld-elf.so.1 /root/sbin/pfctl -F all -f /pf.conf This will set LD_LIBRARY_PATH to the child jail's path, and invoke ld-elf.so.1 directly (with the binary as argument), since it doesn't exist in /libexec any longer.

Be aware, however, that doing this will expose you to potentially tainted files in the child jail. An alternate approach would be to mount and unmount the parent filesystems, e.g.

mount -F /etc/fstab.testjail.parent -a
jexec testjail <command>
umount -F /etc/fstab.testjail.parent -a

Then again, since these base OS directories are read-only mounted from the template, this should be a minimal risk. If you're attempting to run programs/libraries that live in read/write-mounted directories in the child jail, however – beware.

Putting it all together (sort of)

What's missing now is of course the jail.conf entry. With this in place, you can use the regular service jail start [<name>] and service jail stop [<name>] commands to manage your jails, and you can have them automatically start on system boot.

Correctly assembling such a configuration is a fairly involved task, but I'll leave you with an example that should help you get started. Good luck!

# /etc/jail.conf
## Some defaults
# The default devfs ruleset - this is a default /etc/defaults/devfs.rules
devfs_ruleset = 4;
# We want the venerable rc(8) to manage the child jails
exec.start = "/bin/sh /etc/rc autoboot";
# Set a high securelevel for the child jails (see security(7))
$child_securelevel = 4;
# Disallow modifying hostnames after creating jails
allow.set_hostname = 0;
# Clean the environment
exec.clean;

## Our test jail starts here
testjail {
    # Set a variable for use later - a run directory for some housekeeping
    $rundir = "/var/run/jails/testjail";

    # Basic settings for the parent jail
    host.hostname = "testjail.PARENT";
    path = "/data/jails/testjail";
    vnet;
    # Setting a high value here, so we can use service jails (see rc.conf(5))
    children.max = 21;

    # The fstab for the parent, will be automatically mounted on startup. Also
    # make sure we can mount filesystems inside the parent
    mount.fstab = "/etc/fstab.bbs";
    allow.mount;
    allow.mount.devfs;
    # Allow inspecting ZFS snapshots from within the child jail (optional)
    zfs.mount_snapshot = 1;
    # In case you want PostgreSQL or something
    allow.sysvipc = 1;
    # Keep a console log for the jail
    exec.consolelog = "/var/log/jail_${name}_console.log";

    # Cleanup before starting the jail (In case a previous attempt failed)
    exec.prepare = "umount -f -F /etc/fstab.testjail -a || :";
    exec.prepare += "umount -f -F /etc/fstab.testjail.parent -a || :";
    exec.prepare += "umount -f ${path}/root/dev || :";
    exec.prepare += "umount -f ${path}/dev || :";

    ### Starting the jail
    ## Pre-start commands
    # Create rundir on host
    exec.prestart = "mkdir -p ${rundir}";
    # Create the epair interface; put the names of each end in files in $rundir
    exec.prestart += "ifconfig epair create > ${rundir}/if_a"
    exec.prestart += "sed 's/a$/b/' < ${rundir}/if_a > ${rundir}/if_b"
    # Mount parent jail filesystems
    exec.prestart += "mount -F /etc/fstab.${name}.parent -a || :";
    # Mount devfs in parent jail
    exec.prestart += "mount -t devfs -oruleset=${devfs_ruleset} . ${path}/dev";
    # Unhide pf and bpf device in parent jail
    exec.prestart += "devfs -m ${path}/dev rule apply path pf unhide";
    exec.prestart += "devfs -m ${path}/dev rule apply path bpf unhide";

    ## Post-create commands (run after jail created, but before exec.start)
    # Mount devfs in the child jail with appropriate ruleset
    exec.created = "mount -t devfs -oruleset=${devfs_ruleset} . ${path}/root/dev";
    # Assign jail-side interface to jail
    exec.created += "ifconfig `cat ${rundir}/if_b` vnet ${name}";
    # Assign IP address to jail interface
    exec.created += "jexec ${name} ifconfig $(cat ${rundir}/if_b) inet 10.2.3.4/24";
    # Add default route
    exec.created += "jexec ${name} route add default 10.2.3.1";

    ## Startup commands
    # Assign IP to lo0-interface
    exec.start = "ifconfig lo0 127.0.0.1/8";
    # Load PF rules for parent jail and enable
    exec.start += "pfctl -F all -vf /pf.conf && pfctl -e";
    # Start pflogd
    exec.start += "/sbin/pflogd 'not host 224.0.0.18'";
    # Wait for ping to jail (interface plumbing); assuming this file is
    # autogenerated and the IP includes the netmask
    exec.start += "ip='10.1.2.3/24' ; ping -n -c 15 -t 20 -o \${ip%/*}";
    # Set sysctls (if any are configured)
    exec.start += "(test -s /sysctl.conf && sysctl -f /sysctl.conf) || :";

    ## Create child jail
    exec.start += "jail -c ip4=inherit path=/root host.hostname=testjail \
                   name=child zfs.mount_snapshot=1 children.max=20 \
                   securelevel=${child_securelevel} \
                   sysvmsg=\"new\" sysvsem=\"new\" sysvshm=\"new\" \
                   exec.start=\"/bin/sh /etc/rc autoboot\"";

    ## Post-start commands
    # Unount parent jail filesystems
    exec.poststart += "umount -F /etc/fstab.${name}.parent -a -f || :";

    ### Stopping the jail
    ## Pre-stop commands
    # Mount parent jail filesystems
    exec.prestop = "mount -F /etc/fstab.${name}.parent -a";

    ## Shutdown commands
    # Shut down child jail cleanly, if possible
    exec.stop = "jexec child /bin/sh /etc/rc.shutdown ; sleep 1";
    # Forcibly remove child jail, just in case
    exec.stop += "jls -d name|chroot /root grep -q '^${name}.child$' && jail -Rd ${name}.child || :";
    # Remove any lingering TCP sockets
    exec.stop += "tcpdrop -a || :";

    ## Post-stop commands
    # Unount parent jail filesystems
    exec.poststop = "umount -F /etc/fstab.${name}.parent -a -f || :";
    # Unmount devfs
    exec.poststop += "umount -f ${path}/root/dev || :";
    # Unount parent jail dev filesystem
    exec.poststop += "umount -f ${path}/dev || :";
    # Finally, destroy jail interfaces
    exec.poststop += "ifconfig `cat ${rundir}/if_b` destroy";

    # Clean up rundir
    exec.poststop += "rm -Rf ${rundir}";

~ love over gold ~
The Floppy Museum	About the Museum