AI evaluation / Paper breakdown

Agents can write docs. They can't tell when to stop.

DOGBENCH tests whether AI agents produce docs that meet expert standards, and whether they know when docs don't need updating at all.

Problem

AI agents increasingly write user-facing docs, but no benchmark tested whether their output meets expert standards, or whether agents know when docs don't need updating.

Approach

DOGBENCH (an evaluation framework) gives agents 292 real open-source tasks, each a pre-change repo plus a trigger (a pull request or reported gap). The agent must either patch the docs or abstain.

Real tasks

292 open-source repos, each frozen before the change, paired with the pull request or reported gap that should trigger a docs update.

Patch or abstain

The agent decides whether the docs need changing. Knowing when not to edit counts as much as the edit itself.

Expert rubrics

Maintainer-validated rubrics evaluate the work, and any critical failure caps the score at 60.

Evidence

Best agent score: 47.3 / 100
Fabrication: 6.1% (rare)
Task-completion gaps: 45.5% (common)

Agents stopped investigating early, guessed instead of checking evidence, and edited only the first plausible page.

Caveats

Single runs, noisy abstention labels, and two authors work at a docs-agent company.

Lens

An evaluation paper whose real insight is about agent behavior: agents lack a stopping rule. Docs also matter for distribution, since AI agents now read them when choosing products.

Agents lack a stopping rule.