When sleep infinity Is PID 1: How Zombies Stall docker exec on cgroup freeze

While building a runnable lab for Linux syscall notes, I kept a long-lived Debian container under OrbStack, bind-mounted the notes tree, and drove everything through docker exec: compile, run, kill on timeout. The container’s main command started as the laziest possible option:

docker run -d --name syscall-lab ... sleep infinity

It worked for a while. After enough --repeat runs, timeout kills, and crashing demos, the next docker exec began to hang and eventually failed with:

waiting for the cgroup to freeze

Restarting the container bought temporary relief; without a restart, the hang came back. This post is the postmortem. The symptom looks like resource exhaustion. The mechanism is different.

1. First guess: what ran out?

A freezer / cgroup error naturally points at:

• memory or CPU hitting a cgroup limit;
• process count hitting pids.max / pid_max;
• the OrbStack / Docker Desktop Linux VM being under heavy pressure.

In this lab, none of those were the real bottleneck. The workload is tiny; even a pile of zombies is usually nowhere near pid_max. What failed was a synchronization condition that never became true.

2. What was actually happening

2.1 Long-lived container, many execs

This container is not a one-shot job. It stays up so run.sh can docker exec into it repeatedly:

host: run.sh
   └─ docker exec → gcc / demos / timeout kills inside the container

Each exec creates more child processes. After a timeout or crash they exit; if nobody wait()s, they become zombies (Z).

2.2 sleep infinity is a bad PID 1

Without --init, sleep infinity is PID 1 inside the container.

On Linux, orphaned children are reparented to PID 1, and init is expected to wait and reap them. sleep does not do that — it only sleeps. So:

timeout kill / crash / --repeat
        ↓
children exit and become zombies
        ↓
they hang under sleep (PID 1) forever
        ↓
the process tree gets dirtier over time

Zombies barely cost CPU or memory, but they occupy process-table slots — and more importantly, they mean process lifecycle management is already broken.

3. Why docker exec freezes the cgroup

docker exec does not ask the container’s PID 1 to fork your command. Roughly: runc on the host (or VM) setnss into the container namespaces, then fork/execves the command you asked for. The two processes are roommates, not parent and child.

When joining or updating cgroups and applying some resource rules, the runtime often freezes the container cgroup first, to avoid races between rule changes and still-running tasks. The state machine looks like:

write freezer.state = FROZEN
        ↓
kernel: FREEZING (in progress)
        ↓
wait until every task in the group is quiet
        ↓
FROZEN  ← runc polls for this
        ↓
then THAWED when done

So waiting for the cgroup to freeze literally means:

stuck in FREEZING, never reaching FROZEN, until userspace times out.

4. Stuck in FREEZING: what is actually missing

Kernel docs are blunt: freezing can be incomplete. If some task is busy doing something that cannot be frozen right now, the whole group stays in FREEZING.

Common blockers:

Blocker	Why
D state (uninterruptible sleep)	task is stuck in the kernel waiting on I/O / locks; the freezer cannot pin it down
fork/exec racing a freeze	a new task joining the cgroup can bounce FROZEN back to FREEZING
concurrent exec / update / pause	multiple parties fighting the freezer leave it half-frozen

One intuition to correct:

Zombies themselves are usually not the tasks that refuse to freeze. They are already dead and not scheduled. What makes the wait time out is more often a D-state runc init, a half-finished exec, or a freeze↔exec race sitting next to that dirty tree.

A more accurate causal chain:

PID 1 does not reap
  → zombie pile-up = dirty process lifecycle
  → more half-finished runtime work / D-state / freeze races
  → next docker exec writes FROZEN, stalls in FREEZING
  → timeout: "waiting for the cgroup to freeze"

What ran out was not a memory quota. It was this condition failing to hold within the timeout:

∀ task ∈ cgroup: that task is in a decidable frozen state

runc only retries / waits for a bounded time (on the order of ~10s in practice). After the timeout it errors; if the cgroup is left in half-frozen FREEZING, later freezer-touching operations keep hanging too.

Useful upstream breadcrumbs:

• containerd#3286 — later docker exec hangs after an earlier exec left zombies
• runc#2753 / runc#2774 — incomplete freeze, stuck FREEZING, races with concurrent exec
• Kernel freezer docs — FREEZING can remain incomplete; userspace must retry or thaw

5. Fix: give the container a real init

Docker’s --init injects tini as PID 1 in front of your command:

docker run -d --name syscall-lab --init ... sleep infinity

The tree becomes:

tini (PID 1)
  └─ sleep infinity
  └─ (plus adopted orphans / leftovers from exec)

tini’s job is to be a tiny init: reap. Exited children get waited; they no longer accumulate under sleep.

The critical line in the lab script:

docker run -d --name "$CONTAINER" --init --privileged \
  -v "$SCRIPT_DIR:/work" "$IMAGE" sleep infinity

On startup it also checks whether an existing container has HostConfig.Init; if not, it recreates — an old bare-sleep container will bring the bug back unchanged.

6. Prevention plus self-healing

--init cuts the recurrence rate. It does not guarantee OrbStack / Docker Desktop never hit a freezer timeout. So docker exec is wrapped with recovery:

1. detect waiting for the cgroup to freeze;
2. first attempt: docker restart;
3. still failing: delete and recreate the --init container;
4. if even docker rm -f times out (shim unresponsive), restart the whole OrbStack engine.

That is the engineering fallback: keep the process tree clean at the source, then auto-recover the known hang path so you are not clicking the menu-bar icon by hand every time.

7. Building the self-healing wrapper: three traps

Writing that recovery logic surfaced three failure modes that have nothing to do with the freezer, but looked exactly like the same bug from the outside — and cost more debugging time than the original hang.

7.1 Multiple runtimes installed side by side

The machine had both Docker Desktop and OrbStack installed. docker resolves through whichever docker context is active — here it was orbstack, not desktop-linux. The first version of the recovery logic assumed Docker Desktop, and on a stuck container it force-quit and relaunched the Docker Desktop app. It “worked,” in the sense that docker info came back quickly every time — because OrbStack, the engine actually running the container, was never touched. Docker Desktop was just sitting there idle, burning memory, entirely unrelated to the container that was stuck.

Lesson: before writing automation that “restarts Docker” as a recovery step, check docker context ls / docker context show. If more than one container runtime is installed, verify which one the CLI actually talks to — restarting the wrong one is indistinguishable from doing nothing, except slower and more confusing.

7.2 A hand-rolled timeout that silently killed the script

macOS ships no timeout (and no gtimeout unless coreutils is installed via Homebrew). A portable replacement looks straightforward:

run_with_timeout() {
  local secs="$1"; shift
  "$@" &
  local cmd_pid=$!
  ( sleep "$secs"; kill -9 "$cmd_pid" 2>/dev/null ) &
  local watcher_pid=$!
  local status
  wait "$cmd_pid" 2>/dev/null; status=$?
  ...
}

Under set -e, this is broken. wait "$cmd_pid"; status=$? is two separate statements joined by ;, not guarded by &&, ||, or an if. The instant wait returns non-zero — exactly the case this function exists to handle — the shell exits right there, before status=$? even runs. No error message, no return value, just a script that stops.

wait "$cmd_pid" 2>/dev/null || status=$?

turns the failure into the tested branch of an ||, which set -e exempts. The fix is one token; the failure mode it produces — a script vanishing with a bare exit code and no explanation — looks enough like the original freezer hang that it was mistaken for a continuation of the same bug for a while.

7.3 Escalating on the wrong condition

The first recovery draft treated “docker rm -f failed” as “the container is stuck, restart the whole engine,” without checking why it failed. docker rm -f on a container that simply doesn’t exist yet (the very first run, or right after a clean removal) also returns non-zero. Left unguarded, every fresh run triggered a full engine restart before it had done anything wrong.

The fix is to inspect the actual error text before escalating:

out="$(run_with_timeout 20 docker rm -f "$CONTAINER" 2>&1)" && return 0
printf '%s' "$out" | grep -q "No such container" && return 0   # nothing to do
# only past this point is it a real stuck-shim situation

Escalation should be driven by matching the specific failure signature (waiting for the cgroup to freeze, did not receive an exit event) — the same way the detection loop in section 6 does — not by “any non-zero exit from a cleanup step.”

8. Takeaways

1. For long-lived containers driven by docker exec, do not let sleep be PID 1. Use --init (or any init that reaps).
2. waiting for the cgroup to freeze is usually not “quota exhausted.” It is the freezer state machine failing to reach FROZEN.
3. Zombies are a strong signal of a dirty environment, not necessarily the freeze blocker itself. The real blockers are often D-state tasks or freeze/exec races; unreaped children make those failures more likely and more repeatable.
4. When debugging hangs, separate two classes: resource exhaustion (memory / pids) vs synchronization timeout (freezer). Both can look like a stuck exec; the mechanisms are unrelated.
5. Self-healing code needs its own postmortem discipline. A recovery wrapper that assumes the wrong runtime, swallows its own failures under set -e, or escalates on any error instead of the specific one it targets can hide behind the exact symptom it was written to fix.

I started this syscall lab looking for a convenient Linux environment. The container runtime delivered an extra lesson for free: PID 1, zombies, and the cgroup freezer — all mechanisms the notes were meant to teach. Sometimes the toolchain breaking is the best textbook.