Wait… What just happened?

I recently encountered an interesting issue: I had a power failure and had to shut down my Fedora server. When power was restored, none of my LXC containers auto-started.

Needless to say, this confused me.

Investigating the problem

I double checked the config files for my LXC containers, and the ones that I expected to auto start all still had the following line:

lxc.start.auto = 1

So that wasn’t the problem. I checked that the LXC service had started and appeared to be functioning normally. All was well.

I checked logs and found no mention of my containers (error or otherwise).

I tried manually starting the LXC containers. No problem — they all started just fine.

Google to the rescue?

Next, I did some Google searching. I found lots of info on how to configure containers to auto-start, and a few threads on problems auto-starting containers that all seemed to be a result of network or other simple misconfigurations.

A couple threads mentioned that the actual auto start of LXC containers is performed by lxc-autostart, so I shut down all my containers and tried running that manually. No joy. Not a single container started.

Ah-hah!

I checked the man page for lxc-autostart and suddenly had a sudden realization when I found this in the initial description:

By default only containers without a lxc.group set will be affected.

I have recently been migrating most of my containers from Fedora to Ubuntu, and wanted an easy way to keep track of which containers were running what. In order to do that, I wrote a quick script that would scan all my containers and add them into LXC groups (e.g., “fedora-39”, “fedora-40”, “ubuntu-20-04”, “ubuntu-24-04”, etc.). These groups will show up when you run “lxc-ls -f”, making it very easy to tell what’s running what.

What I didn’t realize is that this would effectively make the lxc-autostart program completely ignore all my containers on a system reboot.

Running “lxc-autostart -a”, which processes all containers regardless of their LXC group, started my containers and confirmed the problem.

Solving the problem

So… how do I fix this? Further investigation determined that the lxc-autostart program is run during boot by the /usr/libexec/lxc/lxc-containers script, which includes the following:

# BOOTGROUPS - What groups should start on bootup?
#       Comma separated list of groups.
#       Leading comma, trailing comma or embedded double
#       comma indicates when the NULL group should be run.
# Example (default): boot the onboot group first then the NULL group
BOOTGROUPS="onboot,"

So this left me with two easy solutions. I could either add all my groups into this script, or I could add all my containers into the onboot group. Since I didn’t want to have to keep editing this script as new OS versions come out, I decided to add all my containers into the onboot group. LXC containers can be in multiple groups, so this was as easy as adding the following line to each of my containers:

lxc.group = onboot

Running “lxc-ls -f” isn’t quite as pretty, but this solved my problems.