15. Link, monitor, supervision

Chapter 14 gave you a failure domain and a way to hear about its death. It did not say what to do about the death, and that omission was deliberate: the answer is a policy, policies belong in programs, and wolf puts exactly two primitives under the policy so that everything above them is code you can read.

Here are both of them, in the two programs that differ by one line.

15.1 Two primitives

monitor makes a death into information:

fn shard() -> !int { Corrupt }
fn main() -> !int {
    let index = spawn proc shard()
    let m = index.monitor()
    select {
        exit(reason) from m => { print("the shelf heard about it: {reason}") },
        timeout(1.s) => { print("no word from the shard") },
    }
    print("the shelf is still serving")
    0
}

$ lupin monitored.lu
the shelf heard about it: error(Corrupt)
the shelf is still serving

link makes it contagion:

fn shard() -> !int { Corrupt }
fn main() -> !int {
    let index = spawn proc shard()
    index.link()
    let idle = channel[int](0)
    let v = idle.recv() else |err| { print("the shelf noticed: {err}"); return 7 }
    print("still serving {v}")
    0
}

$ lupin linked.lu
$ echo $?
1

Nothing printed. Read that carefully, because the silence is the whole lesson: the else |err| handler exists, it is on the blocking call, and it did not run. A link does not deliver an error to the linked proc — it ends it. The shelf died at its recv with the shard’s failure as its own exit reason, and the exit code is 1 because that is what a proc that died of a failure hands the process.

So the choice between the two is not a matter of taste, and it is not “link is a monitor with a shorter handler.” It is a question you can answer about any pair of procs in your design:

Is this failure information, or is it fate? If the survivor has something useful to do — retry, reroute, degrade, log and carry on — the failure is information and monitor is the primitive. If the survivor has nothing useful to do, link says so in one word and makes it true even while the survivor is blocked. Code written below a link is code written in the knowledge that it may never run.

Two shapes of link, because fate has two shapes. child.link() couples the child to the proc that is asking, which is the one above. Two procs that are peers couple to each other:

fn parked() -> int {
    let idle = channel[int](0)
    let v = idle.recv()?
    v
}
fn main() -> !int {
    let a = spawn proc parked()
    let b = spawn proc parked()
    a.link(b)
    let m = b.monitor()
    a.kill()
    select {
        exit(reason) from m => { print("b went with a: {reason}") },
        timeout(1.s) => { print("b survived") },
    }
    0
}

$ lupin pair.lu
b went with a: killed

a.link(b) is symmetric and idempotent per pair: linking twice is linking once, and either one dying takes the other. The compressor that has no uploader is doing work nobody will receive; the uploader with no compressor has nothing to send. That pair wants a link, in either spelling, and main above is watching from outside the pair, which is the position a supervisor occupies.

Coming from Erlang: these are Erlang’s two primitives with Erlang’s two meanings, and the lineage is deliberate. Two differences are worth stating. Erlang’s link is trappable — process_flag(trap_exit, true) converts an incoming exit signal into a message, which is how OTP supervisors are built on top of links. Wolf splits that at the primitive instead: if you want the message, you asked for monitor, and a link is not convertible into one. The other difference is monitor’s delivery channel. Erlang’s monitor sends a message to the mailbox, where it competes with every other message; wolf’s monitor hands back a channel, so the exit arrives on its own route and a select arm can wait on it without the service’s other traffic interleaving. One less pattern to get right, one more channel to declare.

Exercise 15-1 (comprehension · lupin) — Chapter 14 showed a proc returning a value: is_normal() was true. This proc returns an error instead. Predict both fields — there are three possible exit shapes and this line can only show you two booleans:

fn boom() -> !int { Bad }
fn main() -> !int {
    let w = spawn proc boom()
    let m = w.monitor()
    select {
        exit(reason) from m => {
            print("normal={reason.is_normal()} killed={reason.is_killed()}")
        },
        timeout(1.s) => { return 1 },
    }
    0
}

Exercise 15-2 (comprehension · lupin) — monitor delivers a message; link shares fate. This program links to a proc that fails, then blocks on an empty channel. Two prints are written. Predict what appears on stdout, and what echo $? shows:

fn boom() -> !int { Bad }
fn main() -> !int {
    let w = spawn proc boom()
    w.link()
    let ch = channel[int](0)
    let v = ch.recv() else |err| { print("recv failed"); return 7 }
    print("got {v}")
    0
}

Exercise 15-3 (design) — A pipeline proc feeds a compressor proc, which feeds an uploader proc. For each of the three pairs, choose link or monitor and defend the choice with the failure you are designing for. One of the three answers should be “neither” — which, and what replaces it?

15.2 A supervisor in forty lines

There is no supervisor in the language. There is a loop, a monitor, a judgment about the reason, and a budget, and if you have read the two sections above you already have all four. Here is the whole thing, on one page, before anything hands you a library:

fn shard(attempt: int, out: channel[str]) -> !int {
    if attempt < 3 { return Corrupt }
    out.send("shard {attempt} serving")
    0
}

fn supervise(name: str, budget: int, out: channel[str]) -> !int {
    var attempt = 1
    while attempt <= budget {
        let child = spawn proc shard(attempt, out)
        let m = child.monitor()
        select {
            exit(reason) from m => {
                if reason.is_normal() { return attempt }
                out.send("{name} attempt {attempt} died: {reason}")
                attempt += 1
            },
            timeout(1.s) => { return Stuck },
        }
    }
    Exhausted
}
fn main() -> !int {
    let out = channel[str](8)
    let n = supervise("index", 4, out) else |err| {
        print("the supervisor gave up: {err}")
        return 1
    }
    out.close()
    for line in out { print(line) }
    print("the shard came up on attempt {n}")
    0
}

$ lupin supervisor.lu
index attempt 1 died: error(Corrupt)
index attempt 2 died: error(Corrupt)
shard 3 serving
the shard came up on attempt 3

Thirty-two lines including the child and the driver. Walk the four decisions in it, because they are the four decisions every supervisor makes and no library removes them:

What counts as failure. reason.is_normal() and nothing else. A shard that returns is done; a shard that errors or is killed gets another try. That is the coarse policy and it is the right default: a supervisor that distinguishes error kinds is a supervisor that has to be updated whenever a child grows a new error.

What restarting means. spawn proc shard(attempt, out) — a fresh proc, a fresh region, and the arguments recomputed at the spawn site. No state carries over, which is the point. If the child needs state to survive its own restart, that state belongs to the supervisor or behind a channel, and the choice of where is the actual design work.

When to stop. budget. Without it a deterministic crash loops forever and “restarting” degrades from recovery into denial — the same failure repeated at whatever rate the scheduler allows, with a log that says it is working on it. The budget makes the supervisor able to fail, and a supervisor that cannot fail cannot be supervised.

What failing means. Exhausted, an ordinary error value in supervise’s row, which the caller reads with else |err|. The supervisor’s own failure is a value handed up to whoever spawned it, which is how a tree gets more than one level. Give the shard above a budget it cannot meet and the last line becomes the supervisor gave up: Exhausted at exit 1 — exercise 15-5 is that run, and the off-by-one in the budget check is worth predicting on paper.

The error kernel

The reason the loop above is so short is that it does nothing except decide. It holds no shard state, parses no queries, and touches no memory the shard touched — so the set of bugs that can kill the supervisor is much smaller than the set that can kill a shard, which is the property the whole design rests on.

That is the error kernel: the part of the system whose failure is not recoverable, kept small on purpose. Everything above it can crash and be restarted; the kernel itself is the code you review twice, keep boring, and never grow. Armstrong’s phrasing is that you push the risky work out to the leaves and keep the trunk simple. In wolf the trunk is the supervisor loops and the leaves are the procs that do the work, and the size of the trunk is a number you can count in lines.

Which is also the argument for flat trees over deep ones. Each level of supervision adds a policy that has to be right, a budget that has to be tuned, and a hop that a failure has to climb before anything useful happens. A shard pool under one supervisor under the root is two levels and is usually enough. Deep trees look like careful engineering and behave like a telephone game.

Choosing a restart policy

One supervisor, one policy, chosen for the failure you expect. The three that cover almost everything, and what each one is for:

policy            on a child's death              use it when
one-for-one       restart that child alone       children are independent
all-for-all       restart every child            children share a protocol
escalate          fail, let the level above act  the budget is exhausted

The loop above is one-for-one with a budget of four, spelled by having one child. all-for-all is the same loop with a list of children and a kill for each survivor before the respawn, and it is what you want when the children hold state about each other: a planner that died mid-query leaves a listener holding connections whose answers will never come, so restarting the pair together resets them to a consistent nothing. escalate is the Exhausted return, and it is a policy rather than an accident — the level above knows something this level does not, which is whether “the index cannot come up” means retry the world or shut down.

What none of these is, is a way to keep a broken program running. A supervisor converts an unpredicted failure into a predicted one and buys time for a human. If the crash is deterministic, the budget expires and the escalation reaches somebody who can read a log, which is the correct outcome and the reason the budget exists.

Exercise 15-4 (fingers · lupin) — Build the smallest supervisor: spawn a worker that fails on its first attempt and succeeds on its second; monitor it; on an abnormal exit, print a line and respawn with the next attempt number; stop after three attempts. Run it and keep the output.

Exercise 15-5 (comprehension · lupin) — Same supervisor, but the worker fails every time. Predict the full output and the exit code before running — including how many times the worker actually runs.

Exercise 15-6 (extension · lupin) — Change the worker to fail twice and succeed on the third attempt — the flappy dependency pattern. Predict the output, run it, and then answer: your budget is 3. What single-character change makes this worker’s recovery impossible, and what does the output become?

15.3 The root supervisor

Chapter 11 built a background refresher and then refused to build a daemon, on the grounds that work whose lifetime is the process wants a failure domain rather than a scope. Here it is, and it is the supervisor of §15.2 with its exit condition moved outside:

fn shard(attempt: int) -> !int {
    if attempt == 1 { return Corrupt }
    let idle = channel[int](0)
    let v = idle.recv()?
    v
}

fn roster(stop: channel[int], log: channel[str]) -> int {
    var attempt = 1
    var child = spawn proc shard(attempt)
    var running = true
    while running {
        let m = child.monitor()
        select {
            exit(reason) from m => {
                attempt += 1
                log.send("shard restarted after {reason}")
                child = spawn proc shard(attempt)
            },
            _ from stop => {
                child.cancel()
                log.send("the roster stood down")
                running = false
            },
        }
    }
    attempt
}
fn main() -> !int {
    let stop = channel[int](1)
    let log = channel[str](8)
    let r = spawn proc roster(stop, log)
    print(log.recv() else |_| { return 1 })
    stop.close()
    print(log.recv() else |_| { return 1 })
    0
}

$ lupin roster.lu
shard restarted after error(Corrupt)
the roster stood down

The shard crashes, comes back, and runs until told to stop; the roster runs until told to stop; the close is the telling, exactly as it was for chapter 11’s refresher. What makes this a daemon rather than a background task is the failure domain: the shard’s death does not reach main, and the roster survives it because the roster’s job is to survive it.

Notice which verb stands the shard down. child.cancel() — the polite one — because the shard is not misbehaving, it is finished, and §14.2’s table says that is the verb whose defers run. A shutdown path that kills its children skips their cleanup, and does it silently.

Nothing is detached

Every proc in that program is reachable from the top. Ask the interpreter, with three procs live:

wolf> fn shard(id: int) -> int { let idle = channel[int](0); let v = idle.recv() else |_| { return 0 }; v }
defined fn `shard`
wolf> fn roster(n: int) -> int { var i = 0; while i < n { let c = spawn proc shard(i); i += 1 }; let idle = channel[int](0); let v = idle.recv() else |_| { return 0 }; v }
defined fn `roster`
wolf> let tree = spawn proc roster(2)
wolf> let pause = channel[int](0)
wolf> select { v from pause => { v }, timeout(2.ms) => { 0 }, }
0 : i32
wolf> :mem
regions:
  #0 `program` arena state=open objects=0
  #1 `proc:roster` arena state=open objects=0
  #2 `proc:shard` arena state=open objects=0
  #3 `proc:shard` arena state=open objects=0
tasks and procs:
  proc#0 `program` root=task0 state=live
  proc#1 `roster` root=task1 state=live
  proc#2 `shard` root=task2 state=live
  proc#3 `shard` root=task3 state=live
  task#0 `main` Running
  task#1 `proc:roster` Blocked
  task#2 `proc:shard` Blocked
  task#3 `proc:shard` Blocked
wolf> :quit

proc#0 is the program itself — a proc, named program, with main as its root task. It is not a special case bolted on so the forest has a root; it is the same kind of thing as the other three, which is why the process’s own death is describable in the same vocabulary as a shard’s. Every proc the program has spawned is in that list, with its own region and its own state, and there is no way to spawn one that is not: there is no detached spawn proc in wolf, exactly as there is no detached task.

The consequence for operating a wolf program is the one worth keeping. “What is running right now” is a question with a printed answer, and “who is supposed to notice if this dies” is a question with a code answer — the supervisor that spawned it, in the loop you can read. A service that appears in neither list does not exist.

Exercise 15-7 (design) — Every proc in wolf lives under the root supervisor; there is no unsupervised spawn. Sketch the supervision tree for the chapter 14 log-search service (listener, planner, one proc per shard), choosing for each internal node: restart the child alone, restart all children, or escalate. Name the failure scenario that made you pick each policy.

Exercise 15-8 (spelunking · corpus) — The corpus checks the kill rule with this directive header:

//! check: run(exit=0, stdout="released")

Explain why the stdout= clause — not the exit code — is the part of this header that actually verifies the rule “defers in a killed proc do not run.” What would a conforming-looking run that violates the rule produce, and which field would catch it?

Exercise 15-9 (design) — A teammate proposes: “monitors are strictly better — a link is a monitor whose handler calls exit, so the language should ship only monitors.” Take the other side using 15-2’s observed behavior: name two properties of link that the monitor-plus-handler encoding does not provide.