2. Strings, honestly

"Skin and Bones."— Cage the Elephant

A wolf string is skin and bones: bytes, and a length. Everything in this chapter follows from refusing to pretend otherwise.

Chapter 1’s receipt guessed its column width. Here it measures:

fn comma(row: str) -> int {
    var i = 0
    while row[i..i + 1] != "," { i += 1 }
    i
}
fn main() -> !int {
    let rows = """
        espresso,340
        pastry,275
        tip,100
        """
    var width = 0
    for row in rows.lines() {
        let i = comma(row)
        if i > width { width = i }
    }
    for row in rows.lines() {
        let i = comma(row)
        print("{row[..i]}{" ".repeat(width - i + 2)}{row[i + 1..]:>5}")
    }
    0
}

$ lupin align.lu
espresso    340
pastry      275
tip         100

Two passes: one to learn the widest name, one to print. The columns come from the data instead of from a number you typed and will forget to change. By the end of this chapter every operation in it is one you can account for (§2.5 prices the slices).

2.1 Literals, methods, interpolation

A string literal is written between double quotes and holds text in UTF-8. Braces inside it interpolate: the expression between the braces is evaluated and formatted into place.

fn main() -> !int {
    let item = "tip"
    print("[{item}] [{item:<8}] [{item:>8}] [{item:^8}]")
    print("[{item:.<8}] [{100:0>5}] [{2 + 2}]")
    print("[{{braces}}] [{item.upper()}]")
    0
}

$ lupin specs.lu
[tip] [tip     ] [     tip] [  tip   ]
[tip.....] [00100] [4]
[{braces}] [TIP]

Interpolation keeps the expression where its value lands, so the pair cannot drift apart during a refactor; there is one place to look, and no printf-style format string with an argument list on the other side. Any expression works, not only a name: {2 + 2} and {item.upper()} are as ordinary as {item}. To print a literal brace, double it.

After a colon comes the format spec: an optional fill character, an alignment (< left, > right, ^ centered), and a width. {item:.<8} means left-aligned in eight columns, padded with dots; {100:0>5} means right-aligned in five, padded with zeros. Interpolation belongs to the literal and not to print, so the same braces work in a multiline literal, in a string you never print, and anywhere else a literal is legal, except the raw literal §2.2 introduces.

A dot after the width introduces a precision, which rounds a number and truncates a string:

$ lupin eval '"{3.14159:>8.2}"'
    3.14 : str

The full spec is [[fill]align][+][0][width][.precision][type], and the three remaining pieces are quickly named. + prints a sign on positive numbers ({340:+} is +340). 0 pads with zeros after the sign, so {7:08} is 00000007. And a trailing type letter changes the base or the notation: x and X for hexadecimal, e and E for exponent form ({12345.6789:e} is 1.234568e+04). A float with no precision prints the shortest text that reads back as the same value, which is why a whole-valued f64 prints 12 and not 12.0.

Strings answer questions about themselves by method call:

wolf> "  espresso  ".trim()
espresso : str
wolf> "tip".upper()
TIP : str
wolf> "340".to_int()
340 : i64
wolf> "tip,100".starts_with("tip")
true : bool
wolf> "awoo".repeat(2)
awooawoo : str
wolf> "tip" + "," + "100"
tip,100 : str

upper and lower know Unicode ("é".upper() is "É"); trim removes whitespace from both ends; contains and starts_with answer yes or no; is_empty is the readable spelling of len == 0; + and += join. split cuts a string on a separator and hands back the pieces, find and rfind report where a separator is, and replace swaps one run of bytes for another. §2.4 sets split beside the byte scan that does the same work by hand.

Exercise 2-2 (fingers · lupin). Using format specs, print a two-column table: names left-aligned in 10 columns, numbers right-aligned in 4.

Exercise 2-12 (comprehension + extension · lupin). Center "DEN LOG" in twenty columns of stars, twice: once with a fill-align spec, once by hand with repeat and len. The padding is thirteen, which does not halve. Predict which side gets the extra star under each spelling before you run them, then state the rule.

2.2 Multiline and raw

Text with newlines in it goes between triple quotes, and the common indentation of the block is removed, so a literal can sit where it belongs in the code rather than jammed against the left margin:

fn main() -> !int {
    let who = "reader"
    let note = """
        dear {who},
          this line keeps its two spaces
        and this one does not
        """
    print("[{note}]")
    print("{note.len} bytes, {note.lines().count()} lines")
    0
}

$ lupin note.lu
[dear reader,
  this line keeps its two spaces
and this one does not
]
68 bytes, 3 lines

Three rules are visible in that output. The newline after the opening """ is not part of the string. The indentation of the closing """ sets the margin, and every line is dedented by that much; relative indentation survives, which is why the middle line keeps its two spaces. And the last line ends with a newline, because there is one before the closing delimiter, which is why the ] lands on its own line. Braces interpolate here exactly as they do in a one-line literal.

The margin is not a convention. A layout mistake inside a """ block does not announce itself — it hands you a string whose contents are not what you typed — so the lexer refuses the three ways to make one, a code each. A """ delimiter stands alone on its line, opening and closing alike: text sharing a line with either one is E0103. The closing delimiter’s column is the margin, so a content line that sits left of it has bytes the margin would eat:

fn main() -> !int {
    let note = """
        dear reader
      lost
        """
    print(note)
    0
}

error[E0104]: this line starts left of the string's margin
 --> ./s4.lu:7:1
  |
7 |       lost
  | ^^^^^^ the line starts here
8 |         """
  | -------- the margin is set here
  |
  = note: the whitespace before the closing `"""` is the margin stripped from every content line.
    Indent this line at least that far, or move the closing `"""` left.

The refusal points at both ends of the comparison: the line that is short, and the delimiter whose column decided how short. E0105 is the same comparison read byte for byte rather than by eye — a line indented with tabs against a closing """ indented with spaces is a mismatch even when the two render identically in your editor, because wolf never guesses a tab’s width. And a plain "…" string owes its closure the same way: { opens an interpolation, and an interpolation still open when the line ends is E0102, the string that never closed. The note names both repairs — } to close the hole, {{ for a literal brace.

When the text is full of backslashes, escape archaeology gets old. r"…" turns the escape table off:

fn main() -> !int {
    let path = r"C:\logs\{today}\wolf.log"
    print(path)
    print("{path.len} bytes")
    0
}

$ lupin raw.lu
C:\logs\{today}\wolf.log
24 bytes

In a raw literal nothing is special: \n is a backslash and an n, and the braces are two more characters, not an interpolation. Count the bytes on the page and you have the length.

Exercise 2-5 (comprehension · lupin REPL). Predict all three lengths before evaluating: "\n".len, r"\n".len, r"C:\temp".len.

Exercise 2-13 (fingers · lupin). Make the invisible visible: print "howl\tat\nthe moon" as one line in which each tab shows as <tab> and each newline as <nl>. Two replace calls chain. Why must the program spell its markers in ordinary literals but would have needed raw ones to match \t if the text had carried a real backslash?

2.3 Bytes, honestly

One honesty up front, because it will bite the Python refugee within the hour:

wolf> "é".len
2 : i64
wolf> let row = "tip,100"
wolf> row[..3]
tip : str
wolf> row[4..]
100 : str
wolf> row[3..4]
, : str

len counts bytes. Indexing takes a byte range, half-open: row[..3] is the first three bytes, row[4..] is everything from byte four to the end, and row[3..4] is the one byte between them. An omitted end fills in the boundary.

There is no s[i] that hands you the ith character. The subscript wants a range, always, and a program that asks for a single index does not run.

That absence is a design decision, not a missing feature. After thirty years of Unicode there is no cheap answer to what a character is — only answers that lie at different speeds. A byte is not a code point; a code point is not a grapheme; the letter your reader sees may be several of each. Wolf gives you byte offsets, checked slices, and iterators that spell their unit (bytes(), chars(), words(), lines(), each one named for what it hands you; §2.4 walks them), and it declines to hand you an s[i] whose meaning changes with the alphabet.

A range that runs off the end is a fault, not undefined behavior (§1.5 showed one). A range that stops partway through a character is a fault too:

fn main() -> !int {
    let s = "héllo"
    print("{s.len}")
    print(s[..3])
    print(s[..2])
    0
}

$ lupin utf8.lu
6
hé
utf8.lu: trap(bounds): byte range 0..2 splits a UTF-8 code point [mem.ub.defined] at 5:11

Six bytes for five letters, [..3] is hé because é is two bytes wide, and [..2] stops in the middle of é, so the run stops instead of handing you half a letter. This is the trade wolf makes in the open: you count in bytes, and every slice you take is checked to be a string.

Coming from Python: len("é") is 1 in Python 3 and 2 here, and s[i] gives you a one-character string there and nothing at all here. Python bought that with a representation it chooses per string and a cost model you cannot see. Wolf keeps one representation, tells you the unit, and refuses the operation whose answer depends on which definition of “character” you had in mind; the price is that you count bytes, and §2.5 puts what each operation costs on the page.

bytes() is the iterator that spells this section’s unit, and what it hands you is not a number. A byte is one octet — eight bits, unsigned, 0 through 255, one byte of storage — and it is a type of its own, spelled byte:

fn main() -> !int {
    let a = 65 as byte
    let hi = 200 as byte
    print("{a} {a as int} {hi as int}")
    print("{255 as byte} {256 as byte} {300 as byte} {-1 as byte}")
    print("{hi + hi} {a + 1} {(a + 1) as byte}")
    0
}

$ wolf run byte.lu
65 65 200
255 0 44 255
400 66 66

The posture is the one §2.4 gives char, and for the same reason: a width is a claim, so the type carries it and nothing widens behind your back. No literal adopts it. let b: byte = 65 does not compile, and neither does b == 10 on a byte the string handed you; 65 as byte is the spelling, in every position, a match arm included. Widening is what you write when a byte meets a number, and it cannot fail: every octet is an int, so b as int is total.

Narrowing is the interesting direction. int as byte keeps the value’s low eight bits, and it is the one narrowing cast in this language that does not trap: 255 as byte is 255, 256 as byte is 0, 300 as byte is 44, and -1 as byte is 255. §3.3’s arithmetic traps on overflow and this does not, because a byte exists to hold the low octet of whatever produced it — a mask, a shift, a checksum step — and a trap there would put a ceremony on every one of those lines.

The third line of output is the rule that catches people. Arithmetic on a byte is int’s arithmetic: hi + hi is 400, not a wrapped 144; a + 1 is an int; and a byte at zero minus one is -1. Nothing overflows eight bits because nothing stays in eight bits, and narrowing back is spelled — (a + 1) as byte. That is also why there is no b += 1: the right-hand side of it is an int.

The mask in that list is two operators. & keeps the bits two integers share, and ! on an integer flips every bit of it at its own width ([type.int.not]): !n is -n - 1, total, never a trap. A byte widens to int first, as it does under every bitwise operator, so !b is an int, and the octet’s own complement is spelled with the cast:

fn main() -> !int {
    let n = 300
    let b = 200 as byte
    let low = n & 0xff
    let high = n & !0xff
    let flipped = (!b) as byte
    print("{!n} {low} {high} {!b} {flipped}")
    0
}

$ wolf run mask.lu
-301 44 256 -201 55

n & 0xff is the low octet, the same 44 that 300 as byte keeps; n & !0xff is everything above it. ! on a bool is still logical not, and on a float it is refused.

Then why have the type at all, when every octet fits an int with seven bytes to spare? Because the seven bytes are the price, and they are not the whole of it. The same 65,536 octets held as List[int] charge a region at least seven times what they charge as List[byte] — eight for the width, and again for the buffer a growing list leaves behind it — and §8.9 measures the whole of it on both machines, where the multiple lands at sixteen on one and thirty-two on the other. That is why every builtin handing you raw octets is declared over List[byte]: bytes() here, and fs_read_bytes, fs_read_chunk and str_from_utf8 when a program wants a file’s bytes rather than its text.

The file your program is written in is bytes too, and it has one tolerance worth knowing. A byte order mark at the very start of a .lu file is stripped and is never a diagnostic — an editor that insists on writing one cannot break your build, and the formatter leaves it where it is. That formatter is wolf fmt, and Appendix E is its page. Anywhere else in the file those same three bytes are a stray character, E0107, like any other.

Exercise 2-1 (comprehension · lupin REPL). Predict all three before evaluating: "wolf".len, "é".len, "🐺".len.

Exercise 2-3 (comprehension · lupin). "wolf" has four bytes. Predict the exact behavior of:

fn main() -> !int {
    let s = "wolf"
    let t = s[2..9]
    print(t)
    0
}

Exercise 2-6 (comprehension · lupin REPL). "wolf" has four bytes. Predict each of these, precisely, as a value or an event: "wolf"[..2], "wolf"[2..], "wolf"[4..4], "wolf"[3..2].

2.4 Iterating meaning

When you want units rather than bytes, ask for the unit by name. lines() walks lines; words() walks whitespace-separated words. The method’s name is the definition of what you get, which is the point:

fn main() -> !int {
    let log = """
          espresso,340
        pastry,275

        tip,100
        """
    var rows = 0
    for line in log.lines() {
        let row = line.trim()
        if row.is_empty() { continue }
        rows += 1
        print("{rows}: [{row}] {row.words().count()} word(s)")
    }
    0
}

$ lupin iterate.lu
1: [espresso,340] 1 word(s)
2: [pastry,275] 1 word(s)
3: [tip,100] 1 word(s)

espresso,340 is one word: words() splits on whitespace and has no opinion about commas. The blank line was skipped by trim and is_empty, in the two places you can see, and by nothing else. count() counts what an iterator yields, so lines().count() and words().count() are line and word counts with no argument about which unit was meant.

One unit is missing from that list, and §2.3 told you why it is hard: the character. Wolf’s answer is a type. A char is one Unicode scalar value, the unit UTF-8 encodes, which is exactly why it can hold what a str holds and nothing a str could not. chars() is the iterator that spells it:

fn main() -> !int {
    let word = "naïve"
    print("{word.len} bytes, {word.chars().len} chars")
    for c in word.chars() {
        print("{c} = {c as int}")
    }
    let composed = "é"
    let decomposed = "e\u{301}"
    print("{composed.chars().len} {decomposed.chars().len} {composed == decomposed}")
    0
}

$ wolf run chars.lu
6 bytes, 5 chars
n = 110
a = 97
ï = 239
v = 118
e = 101
1 2 false

The output shows three decisions. Six bytes, five chars: a char is wider than a byte when it needs to be (ï is one char, two bytes), and the byte tier stays where §2.3 put it, in len and the checked slices. Each c prints as the character it is, never as a number, because a char is its own type, not an integer: let c: char = 65 refuses to compile, there is no arithmetic on chars, and when the scalar’s number is what you want you say so; c as int is the spelled-out direction, and it is total.

The cast back is not. n as char is checked the way every arithmetic operation in this language is checked, and it traps when n names no scalar: negative, past 0x10FFFF, or inside the surrogate gap that UTF-16 reserved and UTF-8 refuses.

fn main() -> !int {
    let n = 55296
    let c = n as char
    print("{c}")
    0
}

$ lupin cast.lu
cast.lu: trap(overflow): `55296 as char`: the value is 0xD800, inside the surrogate gap 0xD800..=0xDFFF — it names no character, and admitting it would mint a `char` that cannot be UTF-8-encoded (D24), so the cast traps ([type.char.cast], D56's family) [type.char.cast] at 3:13

The trap line says why for itself: admitting 0xD800 would mint a char no str could encode. Comparisons follow scalar-value order: 'a' < 'b', and also 'z' < 'é', because 122 is less than 233. That is order, not alphabetization; sorting for a human reader is a collation library’s job.

The last line of the chars-walk output is the third decision, and it is the one the type cannot spare you: a char is not a grapheme, not “what the reader sees”. é typed precomposed is one scalar; the same glyph typed as e plus a combining accent is two, in a string three bytes long, and the two strings do not compare equal. A char holds exactly one scalar, so the literal form refuses the combining spelling at the lexer, with the note that explains the whole trade:

fn main() -> !int {
    let g = 'e\u{301}'
    print("{g}")
    0
}

error[E0110]: a `char` literal holds exactly one Unicode scalar value
 --> ./s12.lu:5:13
  |
5 |     let g = 'e\u{301}'
  |             ^^^^^^^^^^ the second one starts at byte 2
  |
  = note: a `char` is one scalar, not a grapheme: an accented letter typed as base + combining
    accent is TWO scalars even when it renders as one glyph (spell it precomposed, or keep
    it in a `str`). For text, use a string literal `"…"`.

The refusal names both repairs. Spell the glyph precomposed and 'é' is one char and legal; keep the text in a str and it can hold as many scalars as it likes. What you cannot do is put two scalars in a type whose definition is “one scalar”. The lexer holds the line so the runtime never has to.

Splitting on a separator you choose is row.split(","), and the comma function at the head of this chapter is the same job written by hand: walk forward a byte at a time, comparing single-byte slices, and stop where the separator is. Both look at every byte once, so the scan costs what the method costs; what the scan buys is that you can see the byte offsets, which is the arithmetic every checked slice in §2.5 is made of. Write it once for the understanding, then reach for split.

Exercise 2-4 (extension · lupin). Extend the word counter to also report lines and bytes for a """ multiline block. Predict which of the three numbers is a byte count before running.

Exercise 2-7 (extension · lupin). Write encode, a run-length encoder over bytes: "aaabcc" becomes "a3b1c2". Walk the string with byte slices and equality only. What does your encoder do with the empty string, and did you have to write a special case for it?

Exercise 2-9 (comprehension · wolf). Predict all six numbers before running: for each of "wolf", "é", and "e\u{301}", both .len and .chars().len. Then the pointed half: which of the six could §2.3 have told you, and which one needed this section?

Exercise 2-10 (fingers · lupin). Reverse each word of "the wolf runs", keeping the words in order. Build each reversal with chars() and string joins alone. Then say why reversing the bytes instead would wreck "éland" while this version does not.

Exercise 2-11 (fingers · lupin). Squeeze: delete from "howling at the moon" every character that appears in a second string, here "aeiou". One pass, contains, and a growing result. Which of the two strings does your loop walk, and what happens to the program’s cost if you walk the other one?

Exercise 2-14 (fingers · lupin). Detab: replace each tab in "a\tbb\tccc" with the spaces that carry the column to the next tab stop, stops every four columns. Print the result in brackets so the spacing is checkable. Why is “replace each tab with four spaces” wrong, and which input proves it?

2.5 What the machine does

A slice is a view: two machine words, a pointer and a length, aimed at bytes that already exist. Taking one copies nothing.

fn main() -> !int {
    let row = "espresso,340"
    let name = row[..8]
    let cents = row[9..]
    print("{name} + {cents}")
    print("{row.len} {name.len} {cents.len}")
    0
}

$ lupin view.lu
espresso + 340
12 8 3

You can check the no-copy claim from the values alone. row is still twelve bytes after two slices were taken out of it, because nothing was taken out of it: name and cents are two pointers into the same twelve bytes, with lengths of 8 and 3.

wolf> let row = "espresso,340"
wolf> let name = row[..8]
wolf> name
espresso : str
wolf> name.len
8 : i64
wolf> row.len
12 : i64

The consequence is a cost model you can hold: row[..8] costs the same whether row is twelve bytes or twelve megabytes, so a parser that walks a large input in slices allocates once, at the input, and never again. That is why §2.4’s scan is not the slow way to split a row, and why nothing here needed a second string type: slicing does not build. Building is what + and += do, and building copies; remember that before you put one in a loop over a large file.

The second half of the cost model is the interpolation itself. A format string in C is data the program parses at runtime, matched against an argument list by convention and by hope. Wolf resolves the braces when it compiles the literal, into the formatting calls they name: no format string is left to parse at runtime, so the call and its arguments cannot disagree about how many %s you wrote. The machinery is the compile-time evaluator chapter 18 takes apart, where f-strings turn out to be its most-used customer and the one nobody notices. Whether the result beats C’s printf on your machine is a measurement, and this book does not make measured claims without the measurement: chapter 21 runs the comparison, prints the numbers CI produced, and names the cases where C still wins.