14. Procs: the unit of failure

Everything in this part so far has been about work that succeeds. A scope joins its children, a channel carries their answers, a select waits on several things at once — and when a child failed, chapter 10 sent the error up to the join and chapter 6’s machinery read it. That is the right answer for a computation that can fail. It is the wrong answer for a service that can fail, and the shelf is about to show why:

struct Doc { title: str, words: int }
fn shard() -> !int {
    var docs = List[Doc]()
    (mut docs).push(Doc { title: "regions", words: 900 })
    (mut docs).push(Doc { title: "moves", words: 640 })
    Corrupt
}
fn main() -> !int {
    let index = spawn proc shard()
    let m = index.monitor()
    select {
        exit(reason) from m => { print("the shard is down: {reason}") },
        timeout(1.s) => { print("no word from the shard") },
    }
    print("the shelf is still serving")
    0
}

$ lupin shard.lu
the shard is down: error(Corrupt)
the shelf is still serving

Two documents were allocated and then the shard gave up. Nobody freed them, nobody unwound anything, and main did not handle an error — it was told one, by name, and went on serving. A proc is the thing that died: a failure domain, with its own memory, whose death is an event other procs can subscribe to instead of an error they have to catch.

14.1 Armstrong’s argument, one page

The design above is not new and this book will not pretend it is. It is Joe Armstrong’s, from the 2003 thesis that came out of building the AXD301 switch at Ericsson, and it is worth compressing to one page because the argument is short and every clause of it does work.

Start from the observation that software fails in ways you did not enumerate. Not the errors in your row — those are chapter 6’s, and they are values because you thought of them. The other kind: the assertion that should have held, the index that was not there, the invariant two threads broke between them. You cannot handle what you did not predict, so the only sound response is to stop the part of the system that is now in an unknown state, and restart it from a known one.

That is only sound if “the part” is a real boundary. If the failing code shares mutable state with the code that survives, restarting it does not restore a known state — it leaves half-updated data behind and a supervisor that believes it fixed something. So the argument’s real content is a constraint on architecture, and Armstrong states it as an alignment: the unit of service, the unit of failure, and the unit of ownership are the same unit. Draw those three boundaries on top of each other and “let it crash” is not recklessness, it is arithmetic. Draw them apart and no amount of supervision helps.

Wolf spells that unit proc. What crosses between procs is messages; what does not cross is a writable reference to anything. The rule is chapter 10’s D14 rule, unchanged and pointed at a bigger boundary: a proc’s function is an ordinary function taking ordinary arguments, and what it may receive is what a channel may carry.

Three ways to stop

A proc’s death is reported, not thrown, and the report is a value with a shape worth knowing before anything else. Here are the three shapes on one page:

fn ok() -> int { 3 }
fn bad() -> !int { Corrupt }
fn parked() -> int {
    let idle = channel[int](0)
    let v = idle.recv() else |_| { return 0 }
    v
}
fn reason_of(p: Proc, label: str) {
    let m = p.monitor()
    select {
        exit(reason) from m => { print("{label}: {reason}") },
        timeout(1.s) => { print("{label}: no word") },
    }
}
fn main() -> !int {
    reason_of(spawn proc ok(), "returned")
    reason_of(spawn proc bad(), "failed")
    let k = spawn proc parked()
    k.kill()
    reason_of(k, "killed")
    0
}

$ lupin reasons.lu
returned: normal(3)
failed: error(Corrupt)
killed: killed

normal(3) carries the value the proc’s function returned — the result rides along inside the reason, which is why a proc that computes something does not need a channel to hand it back. error(Corrupt) is a failure the proc reported about itself, with the tag from its own error row. killed is what happens when somebody else decides. Three shapes, three predicates (is_normal, is_error, is_killed), and a supervisor that only asks the first one treats the other two the same, which is usually the policy it wants.

Read the signature of reason_of before moving on, because it is the whole ergonomics of this chapter. A proc handle is a Proc: an ordinary value, passed to an ordinary function, with nothing in the signature announcing that a failure domain is involved. monitor is a method on it that hands back a channel of exit messages, and select’s exit(reason) from m arm is how you read one. There is no callback, no registry, and no supervisor type in the language — which is the point of §15.2.

Per-proc accounting, closed

Now the part of the argument Armstrong’s thesis leaves open, and states that it leaves open. Erlang’s isolation is per-process, but its accounting is not: a process that grows a huge mailbox or a huge heap takes memory from the node, and the node is shared. The folklore remedies — poll message_queue_len, set max_heap_size, watch the node — are all measurements after the fact, because the runtime has no structural place to put “this process’s memory.”

Wolf has one, and chapter 8 built it. A proc owns regions. Ask the interpreter what exists after a proc is spawned:

wolf> struct Doc { title: str, words: int }
defined type `Doc`
wolf> fn shard() -> !int { var xs = List[Doc](); (mut xs).push(Doc { title: "regions", words: 900 }); Corrupt }
defined fn `shard`
wolf> let s = spawn proc shard()
wolf> :mem
regions:
  #0 `program` arena state=open objects=0
  #1 `proc:shard` arena state=open objects=0
tasks and procs:
  proc#0 `program` root=task0 state=live
  proc#1 `shard` root=task1 state=live
  task#0 `main` Running
  task#1 `proc:shard` Ready

Region #1 is the shard’s, created with the shard and named after it. Every allocation the shard makes without naming a region lands there, which means “how much memory does this service hold” is a question about one region rather than a question about a heap. §14.2 is what happens to that region when the shard dies.

The design target

The thesis reports a number about AXD301 worth carrying into your own programs: around 92% of its functions contained no concurrency at all. The claim is not that 92% of the code was straightforward; it is that 92% of it was sequential, written and tested as ordinary functions, with the concurrency confined to a thin layer that started things and watched them.

Treat that as a target rather than a curiosity. The chapters in this part give you seven or eight constructs, and the measure of a design that uses them well is how little of the program has to know they exist. The shelf’s shard above is an ordinary function that builds a list and returns an error; nothing in it is concurrent, and it became a failure domain because one line somewhere else said spawn proc.

The one thing both tools refuse

spawn proc f() takes an argument list, always, and the compiler and the interpreter agree about it at the same character:

fn shard() -> !int { 0 }
fn main() -> !int {
    let index = spawn proc shard
    0
}

error[E0201]: expected `(…)` proc arguments
 --> ./s3.lu:6:33
  |
6 |     let index = spawn proc shard
  |                                 ^
  |

$ lupin noparens.lu
noparens.lu: E0201: expected `(`, found an end of line [gram.expr.conc] at 77..77
$ echo $?
2

The rule is small and the reason it is grammar rather than inference is not. A proc is spawned by calling something, so the arguments it will own are written at the spawn site, in one place, where a reader looking for “what does this service start with” finds them. There is no spawn proc f that means “spawn f with whatever is around.”

Exercise 14-1 (comprehension · lupin) — A proc’s function returns 3. Before running, predict both fields of the line this prints:

fn worker() -> int { 3 }
fn main() -> !int {
    let w = spawn proc worker()
    let m = w.monitor()
    select {
        exit(reason) from m => {
            print("normal={reason.is_normal()} killed={reason.is_killed()}")
        },
        timeout(1.s) => { return 1 },
    }
    0
}

Exercise 14-2 (design) — A log-search service has three concerns: an HTTP listener, a query planner, and one index shard per disk. Argue where the proc boundaries go. For each boundary you draw, name the failure it isolates and the state that dies with it; for one boundary you chose not to draw, name what shared fate you accepted.

14.2 Crash means bulk-free

A proc’s death frees its regions. That sentence is the reason this chapter is in a book that spent a whole part on memory, and it is observable: ask what exists, kill the shard, ask again.

wolf> struct Doc { title: str, words: int }
defined type `Doc`
wolf> fn shard() -> !int { var xs = List[Doc](); (mut xs).push(Doc { title: "regions", words: 900 }); Corrupt }
defined fn `shard`
wolf> let s = spawn proc shard()
wolf> :mem
regions:
  #0 `program` arena state=open objects=0
  #1 `proc:shard` arena state=open objects=0
tasks and procs:
  proc#0 `program` root=task0 state=live
  proc#1 `shard` root=task1 state=live
  task#0 `main` Running
  task#1 `proc:shard` Ready
wolf> let m = s.monitor()
wolf> select { exit(reason) from m => { print("shard down: normal={reason.is_normal()}") }, timeout(1.s) => { print("timeout") }, }
shard down: normal=false
wolf> :mem
regions:
  #0 `program` arena state=open objects=0
tasks and procs:
  proc#0 `program` root=task0 state=live
  proc#1 `shard` root=task1 state=error
  task#0 `main` Running
  task#1 `proc:shard` Done

Region #1 is not closed, not emptied, and not queued for anything. It is gone, and no line of the program freed it. The proc is still in the forest, because a supervisor may want to ask about it, and its state is error — the record of the death outlives the memory the death released.

That is the whole cleanup story for a crashed proc, and it is worth stating what is absent from it. No destructor ran. No finalizer queue drained. No garbage collector noticed. There is no traversal of the shard’s data proportional to how much data it had, because a region is freed as one act regardless of how many objects are in it (§8.8). Erlang’s answer here is the same shape — a dead process’s heap goes back whole — and wolf gets it in a language with no garbage collector at all, because the granule was already in the type system.

Kill and cancel are different verbs

Which raises the question chapter 10 §10.4 deliberately left standing. A cancelled task runs its defers on the way out. What does a proc do?

It depends on which verb you used, and the difference is decided rather than incidental:

fn shard() -> !int {
    defer print("the shard closed its cursor")
    let queries = channel[str](0)
    let q = queries.recv()?
    print("serving {q}")
    0
}
fn reason_of(p: Proc, label: str) {
    let m = p.monitor()
    select {
        exit(reason) from m => { print("{label}: {reason}") },
        timeout(1.s) => { print("{label}: no word") },
    }
}
fn main() -> !int {
    let polite = spawn proc shard()
    polite.cancel()
    reason_of(polite, "cancelled")
    let rude = spawn proc shard()
    rude.kill()
    reason_of(rude, "killed")
    0
}

$ lupin verbs.lu
the shard closed its cursor
cancelled: cancelled
killed: killed

One cursor was closed, out of two shards that both registered a defer to close one. cancel is chapter 10’s cooperative cancellation raised to proc granularity: it is delivered at a blocking point, the blocked recv hands back an error, the ? returns through the frame, and returning is what runs a defer. The exit reason is cancelled, which is a fourth shape and the one that means “asked to stop, and did.”

kill is not that. It destroys the proc where it stands and frees its regions, and the defer does not run.

The asymmetry looks harsh until you ask what the alternative would be. A kill has to be safe to issue against a proc in any state, including a proc that is spinning in a loop with no blocking point, and including a proc whose invariants are already broken — which is the usual reason somebody reached for kill. Running that proc’s cleanup code would mean running arbitrary user code inside a failure domain that is being destroyed for misbehaving. So the language declines, and the compensation is the region: nothing leaks, because the memory was never the defer’s job.

The design consequence is a rule for your own code. Anything that must be released even if its owner is killed does not belong to the proc that might be killed. It belongs to the other side of a channel, or to the supervisor above it — which is chapter 15.

verb        delivered at      defers run   regions      exit reason
cancel      a blocking point  yes          freed        cancelled
kill        immediately       no           freed        killed

Both rows free the regions. That is the column that makes the rest of the table a choice about code rather than a choice about leaks.

Exercise 14-3 (comprehension · lupin) — build_then_crash allocates a hundred integers into a region, then returns an error. Predict what the monitor reports and, separately, what happened to the hundred integers — then say which line of code freed them.

Exercise 14-4 (comprehension · lupin) — sleeper registers a defer and then blocks forever on an empty channel; the owner kills it. Two prints are written in this program: defer-skipped in the proc and released in the owner. Predict which of them appear, and in what order:

fn sleeper() -> int {
    defer print_raw("defer-skipped")
    let ch = channel[int](0)
    let v = ch.recv() else |_| { return 1 }
    v
}
fn main() -> !int {
    let w = spawn proc sleeper()
    let m = w.monitor()
    w.kill()
    select {
        exit(reason) from m => {
            if reason.is_killed() { print_raw("released") } else { print_raw("wrong") }
        },
        timeout(1.s) => { print_raw("timeout") },
    }
    0
}

Exercise 14-5 (comprehension · lupin) — The same shape at task granularity. One sibling blocks on a channel with a defer registered; the other fails. Predict the output — and then state, in one sentence each, why this defer runs when 14-4’s did not:

fn fail_fast() -> !int { Boom }
fn race_them() -> !int {
    let ch = channel[int](0)
    scope s {
        s.spawn(fn() {
            defer print("sibling cleanup ran")
            let v = ch.recv()?
            v
        })
        s.spawn(fn() { fail_fast() })
    }
    0
}
fn main() -> !int {
    let r = race_them() else |_| { 42 }
    if r == 42 { print("caught") }
    0
}

14.3 Mailboxes

A proc that owns state serves it to everybody else through a queue, and in wolf that queue is chapter 12’s channel with no additions:

struct Doc { title: str, words: int }
fn shelver(inbox: channel[Doc]) -> int {
    var words = 0
    for d in inbox { words += d.words }
    words
}
fn main() -> !int {
    let inbox = channel[Doc](4)
    let s = spawn proc shelver(inbox)
    let m = s.monitor()
    inbox.send(Doc { title: "regions", words: 900 })
    inbox.send(Doc { title: "moves", words: 640 })
    inbox.close()
    select {
        exit(reason) from m => { print("the shelver finished: {reason}") },
        timeout(1.s) => { print("the shelver is still working") },
    }
    0
}

$ lupin shelver.lu
the shelver finished: normal(1540)

The mailbox is inbox, the receive loop is for d in inbox, and the answer comes back in the exit reason because the proc’s whole job was to produce one number. Everything in that program is vocabulary you already have. words is a local of one proc, so nothing synchronizes it and no lock appears anywhere; the close is the sender’s promise that there is no third document; and the count is the same under every schedule, which the explorer confirms because addition does not care what order it happens in.

Two message kinds want two channels and a select:

fn desk(shelving: channel[int], queries: channel[int], out: channel[int]) -> int {
    var served = 0
    while served < 2 {
        select {
            n from shelving => { out.send(n) },
            n from queries => { out.send(0 - n) },
        }
        served += 1
    }
    served
}
fn main() -> !int {
    let shelving = channel[int](1)
    let queries = channel[int](1)
    let out = channel[int](2)
    let d = spawn proc desk(shelving, queries, out)
    shelving.send(900)
    queries.send(1)
    let x = out.recv() else |_| { return 1 }
    let y = out.recv() else |_| { return 1 }
    print("the desk handled two kinds of message: {x + y}")
    0
}

$ lupin desk.lu
the desk handled two kinds of message: 899

That is the whole mailbox design: typed channels, one per conversation, multiplexed by select. What it does not have is a feature Erlang programmers will look for immediately, so here is why it is missing.

Why there is no selective receive

Erlang’s receive matches patterns against the mailbox and takes the first message that matches, leaving the rest queued. It is genuinely expressive: a call-and-response over one mailbox is four lines, because the reply is picked out by pattern while unrelated traffic waits its turn.

The cost is a scan. Every selective receive may walk the queue, so a receive that matches nothing walks all of it — and a message class that never matches is never removed. The failure mode is a mailbox that grows silently until the node dies, and it is common enough that Erlang folklore has a name for it and Erlang tooling has a counter for it. The expensive version is worse: a proc whose queue is long spends time proportional to the queue on each receive, so the system gets slower exactly as it falls behind, which is the shape of every queueing collapse.

Wolf’s answer is to make the topology explicit instead. One channel per conversation, each with a capacity you chose, and pattern matching happens on the channel rather than on the contents of one queue. The reply channel in the program above is that trade, made once: the desk answers on out because out is the answer’s route, not because somebody wrote a pattern that happens to match answers.

You pay for it in expressiveness. A protocol that Erlang writes as one mailbox and three patterns, wolf writes as three channels; the three channels are declared, bounded, and visible to the scheduler, and none of them can grow without limit while nobody looks. Less clever per receive, and every queue in the program has a name and a bound.

Handlers are atomic and non-blocking

One posture, stated plainly because it decides the shape of every service you write. A proc’s receive loop handles one message at a time to completion. There is no reentrancy, no interleaving of two handlers, and therefore no lock inside a proc: the proc is the lock, and its state is private by construction.

Which means a handler that blocks stops the service. A recv in the middle of handling a message is a service that has stopped serving everybody else in order to wait for one thing, and the fix is never a timeout inside the handler — it is to make the waiting somebody else’s job. Spawn a task for the slow part and let it send the result back as another message, or spawn a proc per conversation. The handler’s contract is that it finishes.

Exercise 14-6 (fingers · lupin) — Build a counting service: a proc that reads commands from a channel, where 0 means “reply with the total” and any other value adds to it. Drive it with 5, 2, then a report, and print what comes back.

Exercise 14-7 (extension · lupin) — Grow the protocol: -1 resets the counter. Report the total, reset, add 3, and report again. Predict both numbers first, then answer: what ordering guarantee makes your prediction safe, and which chapter taught it?

Exercise 14-8 (design) — Erlang mailboxes offer selective receive: a proc can pluck the first message matching a pattern, leaving the rest queued. Wolf’s mailbox is a FIFO channel plus select over multiple channels. State one protocol that selective receive expresses more directly, then argue wolf’s side: what does a skipped-over message cost in Erlang that wolf’s design refuses to pay?

What a proc costs

Three facts, which are contract rather than implementation detail.

A proc is not an OS process. Isolation at v1 is in-process: procs share an address space, and what separates them is the type system’s refusal to let a writable reference cross, not a page table. The consequence is honest and worth knowing — a proc that reaches the unsafe tier (chapter 9) and corrupts memory can corrupt another proc’s memory, because the membrane there is an audited unsafe block rather than hardware. Safe wolf has no such reach, and the isolation is exactly as strong as the audit surface §9.7 taught you to grep for.

A proc costs a region and a task. The region is created with the proc and freed with it; the task is chapter 10’s task, scheduled the same way, and a proc’s root task blocks and wakes like any other. Nothing about spawn proc is a different scheduler.

A proc’s death is ordered with respect to its monitors. Everything the proc did happens before the exit message its monitor receives, which is what makes the two :mem dumps in §14.2 evidence rather than a race: by the time the reason arrives, the memory is already gone, and no monitor can observe a half-freed proc.

Exercise 14-9 (comprehension + schedule play · lupin) — Two client tasks each send two increments to the counting proc; the scope joins, then main asks for the total. Run it under seeds 0, 1, 5, 9. Predict first: does the total vary with the schedule, and why not — and name the thing that does vary between those runs even though no output shows it.