33. The serving loop

Every program in this book so far has been handed its question by you. It read a file you named, or took an argument you typed, and printed the answer where you could see it. A server is that same program with the question arriving from somewhere else, and the difference is a door:

struct Doc { title: str, words: int }
fn words_in(docs: List[Doc], title: str) -> str {
    for d in docs {
        if d.title == title { return "{d.words}" }
    }
    "unknown"
}
fn main() -> !int {
    var docs = List[Doc]()
    (mut docs).push(Doc { title: "regions", words: 900 })
    let door = net_listen("127.0.0.1:0")?
    let caller = net_connect("127.0.0.1:{net_port(door)?}")?
    net_write(caller, "regions")?
    let conn = net_accept(door)?
    let ask = net_read(conn, 64)?
    net_write(conn, words_in(docs, ask))?
    print("{ask}: {net_read(caller, 64)?}")
    net_close(conn)?
    net_close(caller)?
    net_close(door)?
    0
}

The shelf has a door now, and the caller behind it is in the same process, which is why the program fits on a page and why CI runs it on three operating systems. Yours will be somewhere else. Everything else on this page is what a real one does.

33.1 A door of your own

Read the handles first. net_listen answers an int, and so do net_connect and net_accept. A socket in wolf is a number the runtime owns: you cannot construct one, you can only be given one, and net_close gives it back. There is no socket type to learn, no builder, and no destructor. The door is a number, and it stops being a door when you close it.

Read the address next. 127.0.0.1 is loopback (the machine talking to itself), and the 0 asks the machine to choose a free port, which net_port then reports. Every program in this chapter is written that way, and it is the discipline to keep in your own tests. A fixed port is a program that fails on the one machine where something else got there first, and a public address is a test that leaves the building.

Read the errors last, because they are the ones chapter 6 already taught. Nothing here traps. net_connect returns int ! {refused, timeout, io}, and every member of that row is a value you can match on:

fn main() -> !int {
    let door = net_listen("127.0.0.1:0")?
    let addr = "127.0.0.1:{net_port(door)?}"
    net_close(door)?
    var why = "connected"
    let sock = net_connect(addr) else |e| match e {
        refused => {
            why = "refused"
            -1
        },
        timeout => {
            why = "timeout"
            -1
        },
        io => {
            why = "io"
            -1
        },
    }
    if sock >= 0 { net_close(sock)? }
    print("nobody home: {why}")
    0
}

The address that program dials is real and the door behind it is gone, so the answer is refused and not a coarser word. That matters more on a socket than anywhere else in the language: a network error is the ordinary case, arriving several times a second on a busy machine, and a row you can name is the difference between a log line that says which of your peers is down and one that says something failed. net_read is str ! {closed, timeout, utf8, io} for the same reason (the peer hung up, your budget ran out, the bytes were not text, or the host said no), and ? propagates any of them the way it propagates anything else.

Exercise 33-1 (fingers · lupin). Change the cold open’s request from "regions" to a title the shelf does not hold, predict the printed line before you run it, then run it. Then move net_close(door) above net_accept and say, in one sentence, which of the two programs on this page you have written.

33.2 Waiting on the whole set

One connection is not a server. A server holds a door and some number of open conversations, all of which may go quiet for an hour and any of which may speak next, and the question it asks the machine over and over is: which of these has something for me?

fn main() -> !int {
    let door = net_listen("127.0.0.1:0")?
    var watched = List[int]()
    (mut watched).push(door)

    print("nobody there: {net_wait(watched, 20)?.len} ready")
    let caller = net_connect("127.0.0.1:{net_port(door)?}")?
    let ready = net_wait(watched, 5000)?
    print("someone there: {ready.len} ready, and it is the door: {ready[0] == door}")
    print("looking took nothing: {net_wait(watched, 0)?.len} still ready")
    let conn = net_accept(door)?
    print("taken: {net_wait(watched, 20)?.len} ready")

    net_close(conn)?
    net_close(caller)?
    net_close(door)?
    0
}

$ lupin wait.lu
nobody there: 0 ready
someone there: 1 ready, and it is the door: true
looking took nothing: 1 still ready
taken: 0 ready

net_wait takes a list of handles and a deadline in milliseconds, and answers the sublist of them that can be read without blocking: a door whose net_accept will not block, a connection whose net_read will answer bytes or say closed. It is the only call in the family that waits on more than one thing, and it is what a program that spawns nothing has instead of a scheduler. Its clause is [os.net.wait].

Four lines, four rules, and the first is the one that costs people days. An empty answer is the deadline expiring with nothing ready. It is an answer, not a failure: the first line above is a healthy wait that found the world quiet, and a loop that treats it as an error will spend its life reporting that nothing happened. The row net_wait does declare is io, and it means a handle in the set was forged or already closed, or the set was empty with no deadline to end the wait.

The second line is the answer’s shape. What comes back is the subset, in the order you passed it, so the loop that reads it is a for over the answer and not a scan of the set.

The third line is readiness itself: asking consumes nothing. Two waits over the same set answer the same thing until the program actually reads or accepts, which is why the second net_wait (with a deadline of 0, which asks and returns at once) still names the door. Readiness follows the state of the socket, not the history of your looking.

The fourth line is that state changing. The connection was taken, so the door has nothing pending, so the set goes quiet again. The same rule read from the other end is how a loop learns to let go: a connection whose peer has closed is ready, every time you ask, and the net_read that follows is what answers closed. Readiness is the invitation; the read is the news.

Exercise 33-2 (comprehension · lupin). Delete the net_accept line and the conn variable from the program above, leaving the fourth net_wait in place. Predict the fourth line before you run it, then run it, and explain the answer using one sentence of this section.

33.3 The loop

Now put those together, and notice what is missing from the result: there is no spawn, no scope, no channel, and no task. This is one process serving two callers at once.

struct Doc { title: str, words: int }
fn words_in(docs: List[Doc], title: str) -> str {
    for d in docs {
        if d.title == title { return "{d.words}" }
    }
    "unknown"
}

Two callers arrive before the shelf opens its eyes, which is the ordinary case for a server and the reason a listener has a backlog at all: the machine holds arriving connections for you until you take them.

fn main() -> !int {
    var docs = List[Doc]()
    (mut docs).push(Doc { title: "regions", words: 900 })
    (mut docs).push(Doc { title: "moves", words: 640 })
    let door = net_listen("127.0.0.1:0")?
    let addr = "127.0.0.1:{net_port(door)?}"
    var callers = List[int]()
    (mut callers).push(net_connect(addr)?)
    (mut callers).push(net_connect(addr)?)
    net_write(callers[0], "regions")?
    net_write(callers[1], "moves")?

And here is the loop the chapter is named for. Build the set, wait on it, and answer whatever the wait names: the door by accepting, a connection by reading it and writing back.

    var open = List[int]()
    var served = 0
    while served < 2 {
        var watched = List[int]()
        (mut watched).push(door)
        for c in open { (mut watched).push(c) }
        for h in net_wait(watched, 1000)? {
            if h == door {
                (mut open).push(net_accept(door)?)
            } else {
                net_write(h, words_in(docs, net_read(h, 64)?))?
                served = served + 1
            }
        }
    }
    print("served {served}: {net_read(callers[0], 64)?} and {net_read(callers[1], 64)?}")
    0
}

That is a serving loop: listen, wait, take what is ready, answer it, go around. Six of its lines are bookkeeping and three are the program, and the shape does not change as the number of connections does. Add a thousand callers and the loop is the same loop with a longer list.

The cost is the point, because the alternative is the thing people write when the wait is missing. Without a call that waits on a set, a single-loop server has to take turns: arm a short deadline on the door, try to accept, arm another on each open connection, try to read it, come back around. It works, and it spends its life waking up to learn that nothing happened. Hold one connection open and idle for five seconds and the two shapes are not close. The loop that takes turns (a 25 ms budget on the door, a 12 ms one on the connection) goes around 130 times and is descheduled by the operating system about 1,560 times to do it. The loop above goes around once and is descheduled about ten times, which is what a program that does nothing but sleep for those five seconds costs.

That accounting is about a loop that is waiting. The other half is a loop that is not. When net_wait has just named a socket ready, the read that follows makes its syscall and waits for nothing, and it did not always. Until wolf 0.2.8 every parking call in this family went to the runtime’s reactor first — a waiter cell, a lock handshake, one kernel call to arm it and another to wake it, a condvar, two context switches — and only then made the syscall, on a socket the program’s own net_wait had already reported ready. A web server written in wolf paid that round trip three times per request. The same source built against the two compilers, measured on one macOS arm64 machine in one session six minutes apart: 53.3 microseconds per request before, 21.2 after, and 18,766 requests per second becoming 47,162. The share of the serving thread’s time parked in a condvar fell from 40.7% to 1.1%, and in the second run the reactor thread never started at all, because the reactor is lazy and a loop that never parks never asks for one.

The other thing to notice is what this loop is not. Chapter 12’s select waits on channels, and §12.2 costed an idle connection in the concurrency runtime, where an arm in a wait set is a registration, not a task. net_wait is that same argument about a different machine: the runtime is the kernel, the registration is a descriptor, and the program holding them spawns nothing at all. The two compose (net_wait neither parks a task nor disturbs the reactor, so a program may mix it with spawn freely), but a serving loop does not need to.

Exercise 33-3 (extension · lupin). The loop above never lets go of a connection. Give it the other half: when net_read answers the closed row, close that handle and remove it from open instead of writing back. Then say why the program cannot learn that a peer has gone without reading it.

33.4 Many hands on one door

One process on one core answers a great many connections with the loop above, and the day it does not, the answer is more processes, not a bigger loop. Two calls size that decision and one arranges it.

fn main() -> !int {
    let hands = os_cpus()?
    let door = net_listen("127.0.0.1:0")?
    net_deadline(door, 50)?
    var came_back = "no"
    let conn = net_accept(door) else |e| match e {
        timeout => {
            came_back = "timeout"
            -1
        },
        io => -1,
    }
    if conn >= 0 { net_close(conn)? }
    net_close(door)?
    print("hands: {hands >= 1}, accept came back: {came_back}")
    0
}

os_cpus answers how many cores this process may actually be scheduled on, which is not how many the machine has. On Linux it honors a control group’s cpu quota and a cpu affinity mask, so a container given two cpus of quota on a sixty-four-core host answers 2 — where counting the lines of /proc/cpuinfo, which is what a program had to do instead, answers 64 and starts sixty-two workers that will never get a core between them. Its signature is os_cpus() -> int ! {io}: a program resolving a configuration that says “one worker per core” has to be able to say it did not learn the number, so there is no silent 1. Its clause is [os.cpus].

net_deadline arms a budget on a handle, and every parking call on that handle afterward answers its row’s timeout when the budget fires first. The program above arms fifty milliseconds on a door nobody dials, and net_accept comes back inside it with the row instead of parking forever. That is the guarantee the next paragraph rests on. Arm it even when you think the call cannot block: a serving loop that can only be left by a connection arriving is a serving loop that cannot be told to stop.

The accept race, and why it is teachable

Several processes can hold one listening socket. A parent binds a door and hands it to children it spawns: os_spawn_with takes a list of handles to pass down, and the child picks them up with net_adopt_listener, numbered 3, 4, and onward in the order the parent gave them, which is the contract [os.proc.inherit] states. The other route is net_listen_with(addr, reuse_port, backlog), where several processes each bind the same address with the option set and the kernel gives the group one port; that is [os.net.listen.opts].

fn hand() -> !int {
    let door = net_adopt_listener(3)?
    net_deadline(door, 5000)?
    let conn = net_accept(door)?
    let ask = net_read(conn, 64)?
    net_write(conn, "a hand answered {ask}")?
    net_close(conn)?
    net_close(door)?
    0
}

A hand is a whole program that never binds anything. It picks up descriptor 3, arms a budget on it, and serves. The parent is the half that decides how many of those to start, binds the one door they will share, and hands it down.

This pair runs under wolf. Handing an open socket to a child is something a compiled program does and an interpreted one cannot: the child would belong to the interpreter, not to the program it is running. Every other program in this chapter runs on both machines.

fn main() -> !int {
    for a in env_args() {
        if a == "hand" { return hand() }
    }
    let door = net_listen_with("127.0.0.1:0", false, 16)?
    var argv = List[str]()
    (mut argv).push("hand")
    var handed = List[int]()
    (mut handed).push(door)
    let kid = os_spawn_with(os_exe()?, argv, handed)?
    let caller = net_connect("127.0.0.1:{net_port(door)?}")?
    net_write(caller, "regions")?
    print("{net_read(caller, 64)?}, and it exited {os_wait(kid)?}")
    net_close(caller)?
    net_close(door)?
    0
}

The child is the same program, re-executed with a marker argument and holding a door it never bound. Start several of them and you have the oldest server architecture there is, and its oldest question with it: several hands are waiting on one door, one connection arrives, and every one of them wakes up. Readiness is not exclusivity. One hand takes the connection and the others find nothing there.

What happens to the hands that lost is the part of this that changed, and the history is short. Until wolf 0.2.6 a hand that lost the race went back into a blocking accept with its budget already spent, and stayed there until the next connection arrived. On a busy server that is microseconds. On a quiet one it is never: the process is alive, using no cpu, answering nothing, and looking dead to whatever supervises it. The honest advice for that arrangement was to not build it.

Now a listener is non-blocking underneath, the take after a wake finds the connection or finds nothing, and a hand that finds nothing waits again against the budget it already armed, the one computed when the call began, and never in the kernel’s queue. With a budget armed, net_accept returns within it. That is the relation this section’s first program demonstrates on one hand, and what the clause [os.net.accept] promises to many. A lost race carries no error row of its own: from the program’s side it is indistinguishable from the connection never having arrived. So the budget you arm on the door is the most a lost race can cost the loop.

Under load the change buys nothing and costs nothing, and the toolchain’s own measurement of it says so: three hands sharing one door, six thousand requests at thirty-two concurrent, about 30,000 requests per second before the change and about 30,000 after. That is the expected answer, because under load the next connection unparks every loser microseconds later. The change buys the server nobody is talking to.

What your machine does

Two of the calls in this section are the host’s to serve or to decline, and both decline with the reason named. On Windows, reuse_port is refused: that operating system has no equivalent option, and the one that is spelled similarly means something else — it lets a second process take a port the first is using and then quietly delivers every connection to the first, which is a worse answer than a refusal. Passing descriptors to a child is refused on the same host, because a Windows socket is not a small stable number a parent can hand over by position.

The rest of this chapter serves everywhere wolf runs, and that is most of it: net_listen, net_connect, net_accept, net_read, net_write, net_close, net_deadline, os_cpus, and net_wait, the one call in the whole set that names no refusal on any host, because every host it runs on can answer the question it asks. A loop that waits on a set, arms budgets, and answers what is ready is portable. Sharing one door among processes is the part to check on the machine you are shipping to, and os_spawn_with with nothing to hand down is served everywhere, so a prefork master’s spawn is not the half in question.

Exercise 33-4 (comprehension · lupin). net_wait declares one row, io, and one of the things it means is a set that could never become ready. Write the four-line program that produces it (an empty list and a deadline that never expires) and predict the exit code before you run it. Then say what a program that meant to sleep should have called instead.

Exercises

Exercise 33-5 (spelunking · prose). §33.4 says a lost accept race is not an error row. Read [os.net.accept] in the specification and write down the two candidate rows it rejects and the one sentence of reasoning that rejects each. Then say which of the two would have been the more tempting mistake.

Exercise 33-6 (design). A configuration file says workers auto. Write down the rule your program follows: what it starts when os_cpus answers, what it starts when os_cpus answers io, and what it logs in each case. Then defend the number you chose for the second case against the two obvious alternatives.

Exercise 33-7 (extension · lupin). §33.3’s loop answers one question per connection and forgets it. Make the shelf answer two: keep serving a connection until its read answers closed, and have each caller ask twice before hanging up. Which of the loop’s data structures had to change, and which did not?