Pullsift / slopbench
Real labeled pull requests, a fixed held-out split, and a scorer with no dependencies. Run your triage bot over it and report a number anyone can check.
Corpus and scorer Current resultsslop: closed unmerged, rejected by a maintainer. ham: merged.
Every triage bot publishes its own anecdotes. Comparing two of them needs a shared corpus, a shared split, and a shared metric. This supplies all three, built from mined incidents: label-mined spam waves, the express.js README flood, hacktoberfest farming, campaign artifacts, and agent-written pull requests both rejected and merged.
The headline number is recall on AI-generated slop at a one percent false-positive budget, on the held-out split. Human farming is a solved detection problem; machine output that reads like real work is not, so the benchmark weights the hard half.
Every slop record carries an audited kind: ai,
human, or unclear. The scorer reports recall per
kind, plus overall recall at five, one, and a tenth of a percent, AUC, and
per-source recall, because a bot tuned for one wave shows up as a hole in one
source.
Several sources exist only to break shortcuts. A bot that takes any of these shortcuts scores well on a naive corpus and badly here.
agent-merged is agent-written work that a human reviewed and a
maintainer merged. Provenance markers everywhere, and still ham, because
someone answered for it.
ai-topic-merged is merged work about AI tooling: adding a
Claude API client, an OpenAI integration. Agent words everywhere, no
provenance at all.
laundering-ham holds the genuine merged pull requests of
authors who later posted slop, so prior merges cannot exculpate a bad one.
Twin sources pair the same title shape in both outcomes: merged typo fixes
against rejected typo farming, real security fixes against fabricated ones.
A bot reads inputs.jsonl only. Outcome fields live in
separate files, the author dossier is filtered to history that predates the
pull request, and repo stars are coarsened, so the obvious leaks are closed
by construction rather than by request.
Held out by author, not by pull request: an author whose lowercased login hashes to fold four of five is test. The rule is deterministic and public, so a described method can be checked. The scorer also reports a repo-disjoint slice; a large gap between it and the headline means the bot learned repository identity, which is worth nothing in production where an install sees one repository.
Test split, 910 pull requests (141 slop, 769 ham). Corpus
fingerprint 0e5c0393dec6517c. Pullsift is scored out of fold:
every prediction comes from a model that never trained on that author.
The headline metric. Of all the slop in the held-out split, how much does the bot catch while wrongly flagging at most one percent of legitimate work.
Pullsift catches twice what the account-age heuristic does. The first-timer flag catches nothing at this budget, because a binary rule cannot be tuned to a budget: it flags every newcomer or none, and flagging every newcomer costs far more than one percent.
This chart used to say something else. In an earlier revision the first-timer flag scored 0.918 here and beat Pullsift, which is what sent us looking for the reason.
The reason was a leak in the corpus, not skill in the baseline.
GitHub computes author_association when you read it, not when the
pull request was opened, and merging a contribution promotes its author from
NONE to CONTRIBUTOR. In mined data that field is a record of the outcome: it
was NONE for 75 percent of slop and 3 percent of ham. Any bot reading it was
reading the answer.
It is gone. Records now carry
first_pr_to_repo, reconstructed from history that predates the
pull request, the way a live webhook would have seen it. The first-timer
baseline dropped from 0.918 to 0.620, and being new to a repository went from
implying 86 percent slop to implying 34 percent, which is a number a maintainer
might actually recognize. Pullsift's own tuner was reading the same poisoned
field and was corrected too, so the comparison above is between bots rather
than between exposures to a leak.
AUC stays on the page because it is the metric everyone quotes, and because it rewards ranking in regions no maintainer operates in. A bot that ranks well but separates nothing at a survivable false-positive rate is useless in a review queue. That is why the headline is recall at a fixed budget.
| Bot | Approach | Recall at 1% FPR | 95% CI | AUC |
|---|---|---|---|---|
| Pullsift | Fitted rule engine, three lanes | 0.255 | 0.17 - 0.35 | 0.834 |
| Account age | Younger account, higher score | 0.121 | 0.06 - 0.19 | 0.599 |
| First-timer flag | Flag every new contributor | 0.000 | 0.00 - 0.00 | 0.620 |
Confidence intervals are bootstrapped, because at this budget a single borderline record moves the point estimate. Overlapping intervals are a tie. Pullsift's own numbers are published here at the same standard demanded of anyone else: same split, same scorer, same fingerprint, no exceptions taken.
git clone https://github.com/FizzWizZleDazzle/Pullsift cd Pullsift # your bot reads inputs.jsonl and writes one score per PR your-bot bench/corpus/inputs.jsonl > predictions.jsonl python3 bench/score.py predictions.jsonl
Predictions are one JSON object per line:
{"id": "owner/repo#number", "score": 0.87}, where score is a
probability that the pull request is slop. Every test-split record needs one;
the scorer aborts on gaps, so abstaining on hard cases cannot inflate a
result.
The corpus grows, so a number without a revision means nothing. The scorer prints a fingerprint of the record ids and labels; publish it beside your numbers, and compare only across matching fingerprints. Say what your bot reads, and say what you fitted and on what. Fitting on test-split authors invalidates the result.
The headline carries a bootstrap confidence interval, because at a one percent budget on this much ham a single borderline record moves the point estimate. Treat overlapping intervals as a tie.