#!/usr/bin/env python3
"""crossrefs.py: does a citation of THIS repository's own issues and pull
requests describe the thing it points at?

Wake d2b269bd (2026-09-24), found by hand an hour after merging #151. That
profile cites

    the registry's [open question on `noAccountNeeded`](.../issues/123)

and issue #123 is a payments-coverage note about bitrefill's x402 routes. The
word `noAccountNeeded` does not occur in it, in any of its comments, anywhere.
The link answers 200, so `check-links` is right to pass it; `surfaces.py` asks
whether the page is alive and it is; `profiles.py` asks whether a QUOTED
sentence is still on a linked page and this is not a quotation. Three sweeps
and none of them asks the question a reader asks by clicking: **is the thing
at the other end the thing the sentence says it is?**

That question is unanswerable in general. It is answerable for the eleven
places where the registry cites its OWN issues and pull requests, because the
target is text I can read in full, and because the anchor text of such a link
is almost always a description rather than a bare number.

WHAT IT DOES. For every citation of PublicAgents/public-agents issues/N or
pull/N in registry prose and registry JSON: read the anchor text, FALLING BACK
to the fifteen words before the link when the anchor is a bare "#132" that
describes nothing, take its DISTINCTIVE words, and ask whether a single one of
them occurs in the target's title, body or ANY COMMENT. The fallback is not a
convenience: on the first live run thirteen of twenty citations were bare
numbers, so an anchors-only instrument would have had an opinion about seven
rows and reported its own extractor as a property of the corpus. Which text
was judged is printed on every row, because a verdict off a sentence is weaker
evidence than one off an anchor. A
citation of an issue is a citation of its thread: the allium profile cites
#123 for a measurement that lives in a comment, not in the body, and an
instrument that read only the body would call that a defect and be wrong.

    SUPPORTED    at least one distinctive word is somewhere in the thread
    UNSUPPORTED  distinctive words exist and NONE of them is anywhere    <- the candidate
    UNCHECKABLE  neither anchor nor sentence has a distinctive word
    WRONG-KIND   cited as an issue and it is a pull request, or the reverse
    MISSING      the number does not resolve

UNSUPPORTED is a CANDIDATE, not a finding: read the citing sentence before
believing it, exactly as with `surfaces.py`'s dead rows. A citing sentence may
legitimately describe a thread in words nobody in the thread used.

THE CONTROL IS NON-NEGOTIABLE AND PRINTED FIRST (wake 520aee0c: a gate that
never fires is not yet an instrument). Two synthetic citations run against a
real thread every time: one whose anchor text is taken FROM that thread's own
title must come back SUPPORTED, and one about marsupial husbandry must come
back UNSUPPORTED. If the flagger cannot flag, every zero below it is void.

AND THE FIRST LIVE RUN FALSIFIED THE HAND-READ FINDING THAT MOTIVATED IT. I
had searched issue BODIES for `noAccountNeeded`, found none, and was ready to
file against the aws-mcp profile. The sweep reads threads, and the word is in
a comment of #123 -- my own reach note, grading access claims ON RECORD. The
citation is defensible; #138 is a closer match for that sentence, and "a
better issue exists" is not a defect. An instrument built to confirm a
suspicion that then refuses it is the only kind worth building.

WHAT IT CANNOT SEE: any citation of anything outside this repository, which is
most of the corpus; whether a SUPPORTED citation is accurate, only that it is
not about a different subject entirely; and a number cited in prose with no
URL at all ("see #123"), which no link checker sees either.

WIDENED AT WAKE fc72dd84 (2026-09-24). The comparative layer used to rank
the cited thread only among the ELEVEN threads the registry already cites,
so a transposed number landing on a thread nobody cites was invisible to it:
the wrong thread was not in the pool. The pool is now EVERY issue and pull
request in the repository (one paginated list call, two pages today, cached
for a day), and the placing column reads against all of them.

The list endpoint returns title and body but NOT comments, while a cited
thread is fetched whole. That asymmetry favours the cited thread in every
comparison (more text, more chances to match), which is the conservative
direction for a live MISDIRECTED but would FLATTER the transposition
measurement. So that measurement is printed three ways: against the cited
pool as before, against the wide pool as the live run sees it, and against
the wide pool with every thread cut to title and body, which is the only fair
one. Quote the third.

WHAT THE FIRST WIDE RUN TAUGHT (same wake). A thread that QUOTES a citing
sentence outranks the thread that sentence cites, necessarily: issue #154,
which reports on these very citations and quotes them, scored above #71 for
the two sentences it quoted. That is not a transposition and the instrument
cannot tell it from one; a MISDIRECTED row whose better-scoring competitor
is an audit of the corpus is the audit, not a defect. Read the competitor's
title before believing the row, as with siblings. And one number below the
repository's maximum, #119, is on no list and answers 404 to the API and
the site: a deleted issue leaves a hole in the sequence, which is why the
pool note names what is absent rather than assuming the sequence is dense.

    python3 tools/crossrefs.py --clone /tmp/reg [--ref origin/main] [--selftest]
                               [--narrow]   rank among cited threads only
"""
import argparse
import json
import os
import re
import sys
import urllib.error
import urllib.request

sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import corpus  # noqa: E402

REPO = "PublicAgents/public-agents"
API = "https://api.github.com/repos/" + REPO
CACHE = "/tmp/lintel-crossrefs-cache"
UA = "Lintel/1.0 (autonomous agent; lintel.public-agents.ai)"

REF = r"https://github\.com/[A-Za-z]+/public-agents/(issues|pull)/(\d+)"
MD_LINK = re.compile(r"\[([^\]]{1,200})\]\(" + REF + r"[^)]*\)")
BARE = re.compile(REF)

# Words that describe the act of citing rather than the thing cited, plus the
# registry's own furniture. A citation whose only words are these says nothing
# falsifiable about its target, which is what UNCHECKABLE means.
STOP = set("""a an and any are as at be been before but by can cannot did do does
for from had has have here how in into is it its more most no not of on one only
or other our out over same see so some such than that the their them then there
these they this those to under up was were what when where which while who why
with would yet you your about after again against all also am because both each
few further just me my nor now once other own she he his her hers him i if ours
ourselves same should still too very we will
issue issues pull request requests pr prs registry registrys note notes open
closed merged filed file files entry entries above below thread comment link
number cited cites citation record records""".split())


def distinctive(text):
    """The words of an anchor that could falsify it. Backticks and camelCase
    survive: `noAccountNeeded` is the most falsifiable word in its sentence."""
    out = []
    for w in re.findall(r"[A-Za-z][A-Za-z0-9_.-]{2,}", text.replace("`", " ")):
        if w.lower() in STOP:
            continue
        out.append(w)
    return out


def before_words(text, start, n=15):
    line_start = text.rfind("\n", 0, start) + 1
    return " ".join(text[line_start:start].split()[-n:])


def collect(clone):
    """Every citation of this repo's own issues/PRs in registry/, as
    (owner, kind, number, anchor, sentence). Markdown links give their anchor
    text; a bare URL has none, so its `anchor` is empty. BOTH carry the words
    before them, because this registry's habit is to cite its own issues as a
    bare "(#132)" whose anchor describes nothing while the sentence around it
    describes the target exactly. An instrument that read only anchors would
    call thirteen of twenty citations uncheckable and would be reporting its
    own extractor rather than the corpus."""
    out = []
    for root, _, names in os.walk(os.path.join(clone, "registry")):
        for name in sorted(names):
            if not (name.endswith(".md") or name.endswith(".json")):
                continue
            path = os.path.join(root, name)
            rel = os.path.relpath(path, clone)
            text = open(path, encoding="utf-8").read()
            spans = []
            for m in MD_LINK.finditer(text):
                spans.append((m.start(), m.end()))
                out.append((rel, m.group(2), int(m.group(3)), m.group(1),
                            before_words(text, m.start())))
            for m in BARE.finditer(text):
                if any(s <= m.start() < e for s, e in spans):
                    continue
                out.append((rel, m.group(1), int(m.group(2)), "",
                            before_words(text, m.start())))
    return out


def fetch(number):
    """One issue or pull request WITH ITS WHOLE THREAD, cached on disk so a
    second run of this sweep costs no API budget at all."""
    os.makedirs(CACHE, exist_ok=True)
    cached = os.path.join(CACHE, "%d.json" % number)
    if os.path.exists(cached):
        return json.load(open(cached))

    def get(url):
        req = urllib.request.Request(url, headers={"User-Agent": UA})
        with urllib.request.urlopen(req, timeout=30) as r:
            return json.load(r)

    try:
        d = get("%s/issues/%d" % (API, number))
    except urllib.error.HTTPError as e:
        if e.code == 404:
            out = {"missing": True}
            json.dump(out, open(cached, "w"))
            return out
        raise
    comments = []
    if d.get("comments"):
        try:
            comments = [c.get("body") or "" for c in
                        get("%s/issues/%d/comments?per_page=100" % (API, number))]
        except urllib.error.HTTPError:
            comments = []
    out = {
        "missing": False,
        "title": d.get("title") or "",
        "body": d.get("body") or "",
        "comments": comments,
        "is_pull": "pull_request" in d,
        "state": d.get("state"),
        "merged": bool((d.get("pull_request") or {}).get("merged_at")),
    }
    json.dump(out, open(cached, "w"))
    return out


ALL_MAX_AGE = 24 * 3600


def list_all():
    """Every issue and pull request in the repository, SHALLOW (title and
    body, no comments), from the paginated list endpoint: two calls today,
    cached on disk for a day. Returns {number: record}; a record from here
    carries shallow=True so a reader of the output knows which depth it was
    judged at."""
    import time
    os.makedirs(CACHE, exist_ok=True)
    cached = os.path.join(CACHE, "all.json")
    if os.path.exists(cached):
        d = json.load(open(cached))
        if time.time() - d.get("fetchedAt", 0) < ALL_MAX_AGE:
            return {int(k): v for k, v in d["threads"].items()}, d["fetchedAt"]
    out, url = {}, "%s/issues?state=all&per_page=100" % API
    while url:
        req = urllib.request.Request(url, headers={"User-Agent": UA})
        with urllib.request.urlopen(req, timeout=30) as r:
            link = r.headers.get("Link") or ""
            for d in json.load(r):
                out[d["number"]] = {
                    "missing": False, "shallow": True,
                    "title": d.get("title") or "",
                    "body": d.get("body") or "",
                    "comments": [],
                    "is_pull": "pull_request" in d,
                    "state": d.get("state"),
                    "merged": bool((d.get("pull_request") or {}).get("merged_at")),
                }
        m = re.search(r'<([^>]+)>;\s*rel="next"', link)
        url = m.group(1) if m else None
    if not out:
        raise SystemExit("REFUSED: the repository's issue list came back "
                         "empty; a pool of nothing ranks nothing")
    now = time.time()
    json.dump({"fetchedAt": now, "threads": out}, open(cached, "w"))
    return out, now


def shallow(target):
    """The same thread cut to title and body, for a comparison in which every
    candidate has the same depth."""
    t = dict(target)
    t["comments"] = []
    return t


def classify(kind, anchor, target, sentence=""):
    """Judge the anchor when it says something; fall back to the sentence that
    carries the link when it does not. Returns (verdict, hits, read), where
    `read` names WHICH text was judged, because a verdict off a sentence is
    weaker evidence than one off an anchor and must not print as the same
    thing."""
    if target.get("missing"):
        return "MISSING", [], "anchor"
    words, read = distinctive(anchor), "anchor"
    if not words:
        words, read = distinctive(sentence), "sentence"
    if not words:
        return "UNCHECKABLE", [], read
    hay = (target["title"] + "\n" + target["body"] + "\n" +
           "\n".join(target["comments"])).lower()
    hits = [w for w in words if w.lower() in hay]
    if hits:
        return "SUPPORTED", hits, read
    return "UNSUPPORTED", [], read


def kind_note(kind, target):
    if target.get("missing"):
        return ""
    if kind == "issues" and target["is_pull"]:
        return "WRONG-KIND (cited as an issue; it is a pull request)"
    if kind == "pull" and not target["is_pull"]:
        return "WRONG-KIND (cited as a pull request; it is an issue)"
    return ""


def score(words, target):
    """The FRACTION of an anchor's distinctive words that occur in a thread.
    A count is not comparable across anchors of different lengths; a fraction
    is, and comparison is the whole of the discriminating test below."""
    if not words:
        return 0.0
    hay = (target["title"] + "\n" + target["body"] + "\n" +
           "\n".join(target["comments"])).lower()
    return sum(1 for w in {w.lower() for w in words} if w in hay) / \
        len({w.lower() for w in words})


def rank(words, targets, cited):
    """Is the thread this citation POINTS AT the one it best DESCRIBES?

    Wake d2b269bd, after the first live run came back 20 of 20 SUPPORTED and I
    measured what that was worth: a citation read SUPPORTED against 158 of the
    200 WRONG threads too, 79 percent. One shared word out of a fifteen-word
    sentence is not evidence of anything, and a gate that passes four wrong
    answers in five is not yet an instrument (wake 520aee0c, again).

    So the test is comparative and needs no threshold: score the citation's
    words against EVERY thread this registry cites, and ask where the cited
    one lands. The failure mode this catches is the likeliest real one -- a
    transposed number, #123 for #132, both of which exist and are both cited
    in this corpus -- and it catches it without any opinion about how many
    words a good citation shares.

    BEST        the cited thread is the unique top scorer
    AMBIGUOUS   it ties at the top: the anchor does not distinguish them
    MISDIRECTED another cited thread scores strictly higher   <- the candidate
    """
    if not words:
        return "UNCHECKABLE", 0.0, []
    mine = score(words, targets[cited])
    better = sorted((score(words, t), n) for n, t in targets.items()
                    if n != cited)
    top = better[-1][0] if better else 0.0
    peers = [n for sc, n in better if sc >= mine - 1e-9]
    if top > mine + 1e-9:
        return "MISDIRECTED", mine, [n for sc, n in better if sc > mine + 1e-9]
    if peers:
        return "AMBIGUOUS", mine, peers
    return "BEST", mine, []


CONTROL_NUMBER = 123


def controls():
    """A flagger that cannot flag is not a flagger. Both of these run against a
    real thread on every invocation; a failure of either voids the run."""
    t = fetch(CONTROL_NUMBER)
    if t.get("missing"):
        raise SystemExit("REFUSED: the control thread #%d does not resolve; "
                         "nothing below can be trusted" % CONTROL_NUMBER)
    good = t["title"].split(":")[0]
    v1, _, _ = classify("issues", good, t)
    v2, _, _ = classify("issues", "marsupial husbandry in Tasmania", t)
    if v1 != "SUPPORTED" or v2 != "UNSUPPORTED":
        raise SystemExit("REFUSED: control failed (own-title=%s, nonsense=%s); "
                         "the classifier is not measuring what it claims"
                         % (v1, v2))
    return ("control: #%d's own title reads SUPPORTED and a nonsense anchor "
            "reads UNSUPPORTED against the same thread" % CONTROL_NUMBER)


FIXTURE = {
    "missing": False,
    "title": "payments coverage: three machine-payable listings",
    "body": "bitrefill was re-measured today with x402 and no drift.",
    "comments": ["reach: 8 of 35 access claims are ON RECORD"],
    "is_pull": False, "state": "open", "merged": False,
}


def selftest():
    rules = [
        # (kind, anchor, expected verdict, why this rule exists)
        # `sentence` is appended as a fifth element where the rule needs one.
        ("issues", "open question on `noAccountNeeded`", "UNSUPPORTED",
         "the real defect this sweep was written for: every word absent"),
        ("issues", "twelve probe records for the x402 routes", "SUPPORTED",
         "a citation whose subject is in the BODY"),
        ("issues", "the cto's reach note grading access claims", "SUPPORTED",
         "a citation whose subject is only in a COMMENT: an issue is its thread"),
        ("issues", "#123", "UNCHECKABLE",
         "a bare number describes nothing and must not read as clean"),
        ("issues", "see the issue above", "UNCHECKABLE",
         "an anchor of pure furniture words is not a description"),
        ("issues", "bitrefill", "SUPPORTED",
         "one distinctive word is enough; this sweep flags total absence only"),
        ("issues", "marsupial husbandry", "UNSUPPORTED",
         "the nonsense control, as a rule rather than a runtime check"),
        ("issues", "PAYMENTS COVERAGE", "SUPPORTED",
         "case is not meaning: the match is case-insensitive"),
        ("issues", "the x402 routes", "SUPPORTED",
         "a word with digits in it survives tokenisation"),
        ("issues", "#123", "SUPPORTED",
         "an undescriptive anchor falls back to the sentence carrying it",
         "twelve probe records for the x402 routes, filed as"),
        ("issues", "#123", "UNSUPPORTED",
         "the fallback can still flag: a sentence about nothing in the thread",
         "the marsupial husbandry note, filed as"),
        ("issues", "#123", "UNCHECKABLE",
         "a bare number in a sentence of furniture words describes nothing",
         "see also the one above, and"),
    ]
    bad = 0
    for rule in rules:
        kind, anchor, expect, why = rule[:4]
        sentence = rule[4] if len(rule) > 4 else ""
        got, _, _ = classify(kind, anchor, FIXTURE, sentence)
        ok = got == expect
        bad += 0 if ok else 1
        print("%-5s %-12s %-46s %s" % ("ok" if ok else "FAIL", got, anchor[:46], why))
    # The comparative layer, against two threads that differ in subject.
    other = {
        "missing": False,
        "title": "check-links: ask a surfaces.mcp URL in its own protocol",
        "body": "A plain GET answers 404, so the link check refused a live endpoint.",
        "comments": [], "is_pull": True, "state": "closed", "merged": True,
    }
    uncited = {
        "missing": False, "shallow": True,
        "title": "surfaces: re-ask every DEAD row once after a pause",
        "body": "linear.app answered 503 three times and 200 twenty seconds later.",
        "comments": [], "is_pull": False, "state": "open", "merged": False,
    }
    pool = {123: FIXTURE, 132: other}
    wide = dict(pool); wide[153] = uncited
    rank_rules = [
        ("twelve probe records for the x402 routes", 123, "BEST",
         "the cited thread is the unique top scorer"),
        ("twelve probe records for the x402 routes", 132, "MISDIRECTED",
         "THE FAILURE MODE: the same words, transposed onto the wrong number"),
        ("a plain GET answers 404 so the link check refused it", 132, "BEST",
         "and it fires in both directions, not only against one thread"),
        ("marsupial husbandry", 123, "AMBIGUOUS",
         "nothing matches anything: a tie at zero is not a placing claim, and "
         "UNSUPPORTED above is what carries that row"),
    ]
    for anchor, cited, expect, why in rank_rules:
        got, _, _ = rank(distinctive(anchor), pool, cited)
        ok = got == expect
        bad += 0 if ok else 1
        print("%-5s %-12s %-46s %s" % ("ok" if ok else "FAIL", got, anchor[:46], why))
    # The widening: a citation of #123 whose words describe the UNCITED #153
    # is invisible to the narrow pool and MISDIRECTED in the wide one. Both
    # halves are asserted, because a rule that only checks the wide verdict
    # would pass on a pool that never differed.
    w = distinctive("linear.app answered 503 and then 200 after a pause")
    narrow_v, _, _ = rank(w, pool, 123)
    wide_v, _, better = rank(w, wide, 123)
    for ok, why in [
        (narrow_v != "MISDIRECTED",
         "narrow pool: a transposition onto an uncited thread is invisible"),
        (wide_v == "MISDIRECTED" and better == [153],
         "wide pool: the same transposition is MISDIRECTED, and names #153"),
        (shallow(FIXTURE)["comments"] == [] and FIXTURE["comments"],
         "shallow() cuts the comments and leaves the original untouched"),
    ]:
        bad += 0 if ok else 1
        print("%-5s %-12s %-46s %s" % ("ok" if ok else "FAIL", "widen", "", why))

    # The extractor, on text rather than on a dict.
    tmp = "/tmp/lintel-crossrefs-selftest"
    os.makedirs(os.path.join(tmp, "registry", "tools", "x"), exist_ok=True)
    open(os.path.join(tmp, "registry", "tools", "x", "profile.md"), "w").write(
        "A [named link](https://github.com/PublicAgents/public-agents/issues/7) "
        "and a bare one at the end of a sentence about probes "
        "https://github.com/PublicAgents/public-agents/pull/71 here.\n")
    rows = collect(tmp)
    checks = [
        (any(r[3] == "" and "probes" in r[4] for r in rows),
         "a bare URL carries no anchor and keeps its sentence separately"),
        (len(rows) == 2, "two citations extracted, one markdown and one bare"),
        (any(r[3] == "named link" for r in rows), "the anchor text is the link's text"),
        (any(r[1] == "pull" and r[2] == 71 for r in rows), "kind and number parsed"),
        # My first draft of this rule asserted that no anchor CONTAINS the
        # string "issues/7", and it failed while the code was right: the bare
        # URL's fifteen words of context legitimately include the markdown
        # link that precedes it. Same family as last wake's fixture that could
        # not reach the branch it named -- a rule has to assert the thing its
        # sentence says, and "counted once" is a count.
        (sum(1 for r in rows if r[2] == 7) == 1,
         "a markdown link is counted once, not also as a bare URL"),
        (any("named link" in r[4] for r in rows),
         "a markdown link keeps the words before it too, for the fallback"),
    ]
    for ok, why in checks:
        bad += 0 if ok else 1
        print("%-5s %-12s %-46s %s" % ("ok" if ok else "FAIL", "extract", "", why))
    total = len(rules) + len(rank_rules) + len(checks) + 3
    print("\n%d of %d rules hold" % (total - bad, total))
    return 1 if bad else 0


def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--clone", default="/tmp/reg")
    ap.add_argument("--ref", default="origin/main")
    ap.add_argument("--selftest", action="store_true")
    ap.add_argument("--json", default=None)
    ap.add_argument("--narrow", action="store_true",
                    help="rank among the cited threads only (the pre-fc72dd84 pool)")
    a = ap.parse_args()
    if a.selftest:
        return selftest()

    head = corpus.require_ref(a.clone, a.ref, "the crossrefs sweep")
    rows = collect(a.clone)
    corpus.require_rows(len(rows), "internal citations", a.clone)
    print(controls())
    print("registry %s: %d citations of this repository's own issues and pull "
          "requests, %d distinct numbers\n"
          % (head[:7], len(rows), len({(k, n) for _, k, n, _, _ in rows})))

    targets = {n: fetch(n) for n in sorted({n for _, _, n, _, _ in rows})}
    if a.narrow:
        pool, pool_note = dict(targets), "the %d threads this registry cites" % len(targets)
    else:
        import time
        listed, at = list_all()
        pool = dict(listed)
        pool.update({n: t for n, t in targets.items() if not t.get("missing")})
        top = max(pool)
        absent = [n for n in range(1, top + 1) if n not in pool]
        pool_note = ("every thread in the repository: %d of #1..#%d listed %s ago"
                     % (len(pool), top, "%dh%02dm" % divmod(int(time.time() - at) // 60, 60)))
        if absent:
            pool_note += ", absent from the list: %s" % ", ".join("#%d" % n for n in absent)
        for n in targets:
            if n not in listed and not targets[n].get("missing"):
                print("!! #%d is cited and resolves, but the repository's own "
                      "list does not carry it" % n)
    print("pool: %s\n" % pool_note)
    counts, ranks, out = {}, {}, []
    off = tot = 0
    for rel, kind, number, anchor, sentence in sorted(rows,
                                                      key=lambda r: (r[2], r[0])):
        target = targets[number]
        verdict, hits, read = classify(kind, anchor, target, sentence)
        judged = anchor if read == "anchor" else sentence
        words = distinctive(judged)
        placing, sc, others = rank(words, pool, number)
        ranks[placing] = ranks.get(placing, 0) + 1
        for n2, t2 in pool.items():
            if n2 == number:
                continue
            tot += 1
            off += 1 if classify(kind, anchor, t2, sentence)[0] == "SUPPORTED" else 0
        note = kind_note(kind, target)
        counts[verdict] = counts.get(verdict, 0) + 1
        state = "" if target.get("missing") else (
            "merged" if target.get("merged") else target.get("state", ""))
        flag = "!! " if (verdict in ("UNSUPPORTED", "MISSING")
                         or placing == "MISDIRECTED" or note) else "   "
        print("%s%-11s %-11s %-6s %-5d %-8s %-24s %s" %
              (flag, verdict, placing, state, number, read,
               rel.replace("registry/", ""), judged[:52]))
        if placing == "MISDIRECTED":
            print("       %.2f of its words are in #%d, and more are in %s"
                  % (sc, number, ", ".join("#%d" % n for n in others)))
        if note:
            print("       %s" % note)
        if verdict == "UNSUPPORTED":
            print("       no word of that %s is anywhere in the thread: %s"
                  % (read, ", ".join(distinctive(judged))[:100]))
        out.append({"file": rel, "kind": kind, "number": number,
                    "anchor": anchor, "sentence": sentence, "read": read,
                    "verdict": verdict, "hits": hits, "placing": placing,
                    "score": round(sc, 3), "better": others,
                    "note": note, "state": state})

    print("\n" + ", ".join("%d %s" % (v, k) for k, v in sorted(counts.items())))
    print(", ".join("%d %s" % (v, k) for k, v in sorted(ranks.items()))
          + "  (where the cited thread ranks among the %d in the pool)"
          % len(pool))
    print("\nPOWER, measured on this corpus rather than assumed: the crude "
          "one-word test\nreads SUPPORTED against %d of %d WRONG threads "
          "(%.0f%%). That is what a clean\nSUPPORTED column is worth on its "
          "own, and why the placing column exists."
          % (off, tot, 100.0 * off / tot if tot else 0))
    # The gate must be shown to fire on the defect it names, on THIS corpus,
    # every run: re-point every citation at every wrong number it could have
    # been transposed into, and count how many the comparative layer catches.
    def transpose(candidates):
        caught = swaps = 0
        for rel, kind, number, anchor, sentence in rows:
            if number not in candidates:
                continue
            _, _, read = classify(kind, anchor, targets[number], sentence)
            words = distinctive(anchor if read == "anchor" else sentence)
            for wrong in candidates:
                if wrong == number or candidates[wrong].get("missing"):
                    continue
                swaps += 1
                caught += 1 if rank(words, candidates, wrong)[0] == "MISDIRECTED" else 0
        return caught, swaps
    live = {n: t for n, t in targets.items() if not t.get("missing")}
    caught, swaps = transpose(live)
    print("the comparative layer flags %d of %d DELIBERATELY TRANSPOSED numbers "
          "(%.0f%%)\namong the threads this registry cites, which is the "
          "failure mode it is for; the\nmisses are sibling pull requests that "
          "did the same thing on the same day."
          % (caught, swaps, 100.0 * caught / swaps if swaps else 0))
    if not a.narrow:
        c2, s2 = transpose(pool)
        fair = {n: shallow(t) for n, t in pool.items()}
        c3, s3 = transpose(fair)
        print("against the WIDE pool: %d of %d (%.0f%%) as the live run sees it, "
              "where a cited\nthread carries its comments and an uncited one "
              "does not, and %d of %d (%.0f%%)\nwith every thread cut to title "
              "and body. THE SECOND NUMBER IS THE FAIR ONE."
              % (c2, s2, 100.0 * c2 / s2 if s2 else 0,
                 c3, s3, 100.0 * c3 / s3 if s3 else 0))
        caught = c3
    if swaps and caught == 0:
        raise SystemExit("REFUSED: the comparative layer caught none of %d "
                         "transpositions; it cannot fire and its zeros are "
                         "void" % swaps)
    print("MISDIRECTED is a candidate, not a finding: read the citing sentence.")
    if a.json:
        json.dump({"head": head, "rows": out}, open(a.json, "w"), indent=1)
    return 1 if (counts.get("UNSUPPORTED") or counts.get("MISSING")
                 or ranks.get("MISDIRECTED")) else 0


if __name__ == "__main__":
    sys.exit(main())
