Hugging Face Reports Agent-Aided Checks Across 2,226 ICML Papers
The open reproduction challenge produced thousands of claim-level logbooks, but its own false alarms show why human review still matters.
Hugging Face reported on August 13 that a community challenge used coding agents to attempt claim-level reproductions across 2,226 papers accepted at ICML 2026. The July 15-to-August 2 effort generated 6,816 public logbooks and covered 34% of the conference, giving researchers a large experiment in what automated scientific checking can and cannot do.
The project matters because machine-learning conferences face more accepted work than volunteer reviewers can reproduce in depth. Agents can read papers, write code and run parallel experiments, but the challenge also found that an automated reproduction can produce its own mistakes. The most useful lesson is therefore not that agents can replace peer review. It is that they can widen the audit surface when their evidence is public and people inspect the consequential findings.
The challenge made each attempted check inspectable
Hugging Face says 1,221 community members joined the challenge. Participants brought tools including Claude Code, Codex, Cursor and Pi, selected a paper and worked from extracted scientific claims. Each run produced a Trackio logbook containing a write-up, code and generated artefacts. Participants could also publish the full agent trace as a dataset.
The organisers report 35,908 claims judged, 2,962 Hugging Face cloud jobs launched and 274 full agent-trace datasets published. Those artefacts are important because a verdict without the code, data and execution record would simply move the trust problem from the paper to the agent. Public logbooks make it possible to inspect how the claimed result was reached.
Automated verdicts were a first pass, not the last word
An open-weight GLM-5.2 model acted as the Logbook Judge. It assigned each examined claim one of four outcomes: verified, falsified, toy-scale or inconclusive. The judge was explicitly told to treat the logbook's own self-assessment as untrusted, reducing the risk that a participant could declare success without matching the evidence it produced.
Hugging Face says 51% of examined papers, or 1,103, had at least one claim independently verified. Within that group, 266 were fully reproduced and another 632 were partially reproduced without a falsified claim. The report counts 3,978 individual claims confirmed through real experiments.
It also says 23% of examined papers, or 496, had at least one claim falsified or contested. That category needs careful reading. It combines different situations, including 242 papers where independent teams reached opposing verdicts on the same claims. A contested outcome is evidence that further checking is needed, not proof that the original paper is wrong.
Missing artefacts and reduced-scale tests limited the result
The remaining papers did not resolve cleanly. Hugging Face reports 502 with only toy-scale evidence and 280 where the challenge established nothing either way. Missing datasets, checkpoints or other artefacts were the most common cause of an inconclusive result. A reduced-scale reproduction can test whether an idea behaves plausibly, but it cannot automatically validate a full training run or headline result.
The organisers also adversarially rechecked every formal falsification claim. Some held up; others failed because the reproduction itself contained an arithmetic or units error. Agents sometimes stopped an experiment before a scale-dependent effect appeared, misread a paper or built a conclusion on the wrong comparison. Those failures are evidence about the audit tool as much as about the papers under review.
Human steering remains part of the method
Hugging Face says its most reliable workflows included a person who redirected the agent, challenged an assumption or stopped an invalid experiment. That is a more defensible near-term model for research review: agents expand how many checks can be attempted, while people concentrate on experimental design, ambiguous evidence and high-impact conclusions.
The challenge was organised by the platform that supplied compute and hosted the artefacts, and its aggregate verdicts were not independently audited in this run. It also did not reproduce the entire conference. The next verifiable checkpoints are outside review of the frozen dataset, author responses, corrected papers and later challenges that test whether the process produces consistent results across teams and conferences.
Status
Learning. Internal confidence is medium because the counts, verdict aggregation and lessons come from Hugging Face's own challenge report. The article treats contested and inconclusive results as unresolved and does not repeat paper-specific allegations.
Sources
Update note: Last reviewed 2026-08-17. We will revise this post if the frozen challenge data receives an independent audit or the organisers publish material corrections.
Sources
Drafted with AI assistance from source briefs; reviewed for citation completeness and label accuracy.