Pullsift

Decides whether a pull request is worth a maintainer's first look, and keeps the rest out of the review queue.

Source Benchmark

The problem

Maintainers lose hours to pull requests nobody should have to read. Agent output that no human checked before submitting. Tutorial floods where hundreds of new accounts send the same change. Drive-by accounts with a trail of closed-as-invalid work elsewhere. Pull requests against repositories that accept none.

Existing tools score one pull request, or one account, inside one repository. Pullsift is built around sharing: repositories in the network exchange campaign signatures and corroborated author verdicts, so the first repository hit by a wave inoculates the rest.

What it decides

Tier thresholds sit at measured false-positive rates, so the cost of being wrong is a number rather than a hope. Every install starts in dry run: it annotates what it would have done and acts on nothing.

How it decides

Every signal is a rule with a weight fitted on real pull requests, and every verdict carries the evidence that produced it. Three lanes feed one score:

• Campaign signatures. Diff and prose similarity cluster arrivals across repositories, with burst detection, so a flood is caught as one campaign even when every author is a fresh account.
• Author history. Whether the account answers for its work: abandonment after review, provenance markers, submission patterns, and corroborated verdicts from other installations.
• Content. Whether the body describes the diff, whether it references code that exists, and a self-hosted text model that scores prose for machine authorship.

AI policy is yours to set

Whether AI assistance is welcome is a project's call, not a detector's. Pullsift measures whether a human answers for the work; you decide how much the tooling itself matters.

# .github/pullsift.yml
ai_policy: welcome    # AI involvement carries no weight
ai_policy: neutral    # the default: markers count for what the data says
ai_policy: disclose   # fine when disclosed; undisclosed AI prose is penalized
ai_policy: forbid     # any provenance marker escalates

Under welcome, an agent-written documentation fix from an account that responds to review passes untouched. Under any policy, no single text signal can reach the close tier alone.

slopbench

slopbench is the benchmark Pullsift ships with. Triage bots publish anecdotes. Comparing them needs a shared corpus, a shared split, and a shared metric. Pullsift ships all three: real labeled pull requests, a fixed author-held-out test split, and a scorer with no dependencies.

The headline metric is recall on AI-generated slop at a one percent false-positive budget. Human farming is a solved detection problem; machine output that reads like work is not.

Slop caught at a one percent false-positive budget, on the held-out split, scored out of fold. Numbers are pinned to a corpus revision by fingerprint. The baselines ship with the corpus so a new bot has something honest to beat. Full results and method.

Install

Point the App at a repository and leave dry run on for a week. Read what it would have done. Then decide.

dry_run: false
challenge: true
exempt_users: [trusted-bot]
protected_paths: [README.md, LICENSE]

Free for the first hundred scored pull requests per month, per install. Beyond that it is paid, because the cost of running it scales with volume. Public and private repositories cost the same.

Limitations

• Federation between installations is designed and tested, not yet switched on. Each install currently learns alone.
• Text detectors are weak evidence and biased against writers whose first language is not English. Pullsift gives them one weight among many and never lets them decide alone.
• Labels in the benchmark are maintainer decisions, not ground truth. A closed pull request is slop because a maintainer said so.