Machine Readable Zoning: Compiling Toronto Bylaw 569-2013 into a Verifiable Clause Graph

A bylaw is not a list of values; it is a graph of clauses that override, exclude, and derive from one another. This project compiles Toronto Zoning Bylaw 569-2013 (129 source files, 5,058 enumerated provisions) into a clause graph with typed edges, and grades the result on a fixed 100 question comprehension test. Asked cold, the model scores 20/100. Given the raw extracted standards, 60. With override relations folded next to each value, 81. With coverage complete, 97. A final exam of 50 questions the system had never seen, written after it was frozen and taken once, scores 43/50; six of its seven failures turned out to be errors in the exam's own answer key. The system never issues a pass/fail verdict: where a determination depends on information no building model holds, it returns candidate clauses and names the missing inputs.

You can make a language model meaningfully better at zoning without touching the model. You change what it reads.

When a measure becomes a target, it ceases to be a good measure.Marilyn Strathern, "Improving Ratings," 1997, restating Goodhart's law.

The common way to put a bylaw in front of an AI is to paste it into a chat window, or to chunk it into a retrieval index (Lewis et al., 2020). Both sever what makes a bylaw mean what it means: the relations between clauses. A clause read alone is frequently the opposite of the law, so a system that retrieves the general rule with high confidence quotes the bylaw correctly and answers wrongly. The failure is measured, not hypothetical: commercial legal research tools hallucinate on 17 to 33% of queries (Magesh et al., 2024), and general purpose models at least 58% of the time on direct legal questions (Dahl et al., 2024).

This project treats the bylaw as a compilation target instead. The source text is enumerated deterministically, an extraction loop lifts values and relations into a graph, a citation gate keeps the loop honest, and deterministic code, not the model, resolves which value survives. One constraint governs the whole design: the LLM never decides pass/fail. A zoning determination is a legal act. The model extracts and explains; the graph stores; the resolver computes; a person decides.

The contribution is not the pipeline. It is the measurements: what a fixed test did as the representation changed, what coverage means once the denominator is honest, and what an audit against the source found.

The corpus is the office consolidation of Bylaw 569-2013 (version date July 31, 2024): 129 files, 2.1 MB. Chapter 900, site specific exceptions that only apply once an address is known, is out of scope. Two independent paths leave the source: a deterministic enumerator that establishes what exists, and an unattended LLM extraction loop that fills the graph through a citation gate.

3.1 Typed edges: how a bylaw overrides itself

Wikilinks between clauses look like a graph but cannot be queried, so every relation is a typed edge carrying the source phrasing that justifies it. Four types cover what the bylaw actually does to itself.

The loader unfolds this graph before any number is compared: for each concept it computes one effective value, and if the graph does not resolve to a single value, no number is passed downstream. In the representation handed to the model, the same relations are folded flat, next to the value they affect:

side_yard_setback:
  value: 1.2
  unit: m
  direction: min
  clause: "10.10.40.70(2)"
  overridden_by: ["10.10.40.70(5)(A)"]   # override folded next to the value; see 4.1
  answer_status: resolved

Field names are real; the clause numbers in this snippet are illustrative.

3.2 The denominator problem

Coverage was first measured by section, and the pipeline scored 750/750: 100%. The number was worthless. A section is not the unit a zoning question resolves at; an item is. Section 10.10.40.70 contains 43 items; section 50.10.20.100 contains 112. Landing 1.1 items per section was enough to score 100% by section, and by item the same graph covered 25%.

The denominator itself took three attempts. A regex count gave 426 items; nearly all were cross references to other clauses. The enumerator instead reconstructs full item paths, 10.10.40.70(5)(A)(i)(b), from each section's nesting and removes 1,524 duplicates, fixing the denominator at 5,058. Every coverage figure here uses it. The built graph holds 5,349 clause nodes against those 5,058 items; the two counts measure different units, and keeping them apart is the point of this section.

3.3 The citation gate

The loop runs unattended, so it needs a truth test that does not involve trusting the model: a node's quote must exist verbatim in the source, inside the quoting clause's own context. The naive version of "context", a fixed 6,000 character window, broke in both directions: it blocked 160 legitimate items in long sections and admitted neighbouring sentences in short ones. The loop stalled at 96.8% coverage on a gate bug, not a model failure.

The correct boundary follows the text's own structure: open toward ancestors, because the bylaw puts meaning in introductions, and closed toward children and siblings, because an absorbed child sentence silently reassigns its numbers. Rebuilding the gate flagged 467 existing nodes; all were re-extracted from the source, none rewritten. The stall also forced a policy decision: the 160 blocked items appeared in no test question, so stopping at 96.8% would have cost nothing on the benchmark. Coverage went to 100% anyway: the benchmark is not the deliverable.

4.1 The representation experiment

The same corpus was rewritten into five representations and each was scored on the same 100 question test. The spread everyone expects, that some clever format wins, did not appear: the two leaders tied at 81 despite one being 2.6× the size of the other, and the format that moved the override relations out of the values into a separate rule list scored exactly the baseline, 60.

Of the 21 point gain, +17 came from folding 224 override edges next to the values they affect, so that finding a value and learning its exceptions is one lookup instead of two files, in line with what long context studies report: models read what is near and miss what is far (Liu et al., 2024). +4 came from suppressing excessive abstention. The conclusion is not that one format is better. It is: regardless of format, writing the override relations next to the value raises comprehension, and defeasible-rules landing exactly on the baseline is the counter evidence that isolates the cause.

The experiment also had a ceiling nobody had measured: 83, because seventeen questions' answering clauses were missing from the corpus. The five formats were competing over a 2 point band. After coverage completed, the same test scored 97/100.

4.2 Verification against the source

Comprehension scores measure the reader. A second audit measured the corpus: 168 extracted standards from Chapter 10 were compared item by item against the published provisions, each verdict pinned to a sentence lifted mechanically from the source.

The headline is not the 122; it is the shape of the errors. Not one number was transcribed wrongly. Every failure is a condition attached to the wrong place, like the dropped unless in Figure 2. And the operative finding sits outside the match column: of 39 items used as baseline values in cover sheet checks, only 14 were free of defects, so four of the five downstream rules could not be promoted to verified. The values are right. What is missing is the problem.

4.3 An exam the system had never seen

The 100 question test was reused throughout development: the corpus improved while looking at the same questions, so 97 by itself is a practice score. (Scoring, on both tests, is mechanical: the reader's JSON answer is compared against a key pinned to quoted source sentences. The reader is Claude, invoked through the CLI at its default model, the opus alias on the run date; the practice key was fixed by hand.)

To close that gap, the system took a final exam on August 24, 2026, a held out test in testing terms. The corpus was frozen and its hash recorded. Then 50 brand new questions were drawn by seeded random sampling from sections the practice test never used (33 from Chapter 10, 17 from Chapters 15 and 200, which the practice test never covered at all), authored from the source text alone with every answer pinned to a quote, and taken once with the same reader and grader. Graded against the sealed key: 43/50, 86%. Chapter 10 came back 33/33.

The seven failures are the finding. Six were errors in the exam's own answer key: the reader cited clause 200.5.1.10 where the key said 200.5.10.1, and the source text sides with the reader. The key's author, reading narrow excerpts, attached six items to a neighbouring section, the same context severing failure the audit in 4.2 documents; the graph reader out cited its own examiner. With the key corrected, the exam stands at 49/50, with one genuine miss, a parking rate in a Chapter 200 table the reader could not find. The as run number stays 43/50, and the protocol, sampling record, question set, and raw outputs are kept with the project.

Refusing to output pass/fail is not caution for its own sake. Three inputs of a zoning determination are structurally unavailable to a model reading a building file.

The baseline is not in the model. Setbacks are measured from lot lines, and lot lines are survey data. A BIM file can be geometrically perfect and contain none of them.

The definitions live elsewhere. Height is measured from established grade, defined in Chapter 800; floor area excludes attic and mechanical space under 10.5.40.40. The modelled number and the bylaw's number are different quantities with the same name.

Which clause wins is decided outside the file. A front yard setback of 3.0 m or 6.0 m can turn on abutting a major street, being a through lot, or lot depth under 36 m.

The audit put a number on this: of the 39 or 40 concepts a cover sheet checks per zone, only about half resolve from the bylaw text alone.

So the system computes what it can, shows candidate clauses with their quotes, and returns the names of the inputs it does not have. The unresolved case is a normal output, not a failure mode.

One bylaw, one city. Chapter 900's site specific exceptions are excluded by design, and prior bylaws that still govern some parcels are not modelled. The corpus is the live toronto.ca text fetched August 5, 2026: its pages say "Version Date: July 31, 2024" but already carry 2025 amendment markers in 97 places, so neither "the 2024 edition" nor "current law" describes it exactly. Amendments after the fetch date, including the 2026 multi tenant house changes, are absent either way, and the City notes a consolidation is a convenience copy, not the certified text. The test is a progress instrument, not a standardized benchmark; the ceiling incident in 4.1 shows what an unexamined test hides. The audit covers Chapter 10; the other chapters have the same gate but not yet the same review. And nothing here is a legal instrument: the published text and a qualified reviewer remain the authority.