AI Agents Run Experiments but Fail Original Research - Princeton
A multi-institution study led by Princeton University researchers found that AI agents could handle the engineering tasks of AI research but produced papers rejected by expert reviewers as nowhere near conference quality.
👁 137 reads
A multi-institution research team led by Peter Kirgis and Sayash Kapoor at Princeton University has found that AI agents can complete the engineering work of AI research but fail at the open-ended thinking required to produce original, publishable science - a finding that challenges optimistic timelines for recursive self-improvement.
- AI agents given six days, $3,000 in API credits and a GPU budget failed to produce papers worthy of a top ML conference
- The study used a new 'shadow evaluation' method, asking agents to answer research questions from two unpublished NeurIPS 2026 papers
- Agents ran hundreds of experiments competently but showed poor judgment, tested hypotheses on tiny synthetic datasets, and made no novel contributions
The study introduced a method called “shadow evaluation,” in which Anthropic’s Claude Opus 4.8, running on open-source software called OpenClaw, was asked to answer research questions drawn from two unpublished papers submitted to the machine-learning conference NeurIPS 2026. Because the papers had not been made public, the agent could not retrieve answers from training data or the web. Each question addressed a genuinely open problem: one asked whether a model’s behavioral “personas” can be controlled by editing its weights; the other asked how to build a detector for when a tabular prediction model becomes unreliable.
The agents were given six days, $3,000 in Anthropic API credits, a GPU budget, virtual computers, and open web access. The original paper authors then graded the output as they would a conference submission. Both papers were rejected.
Reviewers found the agents technically capable - they surveyed literature and ran hundreds of experiments - but deeply flawed as researchers. “The agents were unambiguously bad at carrying out the research itself,” Kapoor said. They ran experiments on tiny synthetic datasets, struggled to write coherently about their findings, and contributed nothing new to either field. “The papers were nowhere close to the mark when it came to being at the quality of a top AI conference,” he added. The full study coverage notes that most prior benchmarks only tested agents on narrow, checkable tasks - not the kind of judgment-heavy, hypothesis-driven work the new evaluation demanded.
Practitioner takeaway: For developers and builders using AI agents today, the study is a useful calibration. Agents are reliable for structured, engineering-style subtasks - running code, reviewing literature, compiling results - but should not be trusted to set research direction, choose meaningful experiments, or judge whether an approach is worth pursuing. Treating agent output as a starting draft that needs expert human review, rather than a finished product, remains the practical baseline.
Google · Your sourcesQuantslant higher in your Google - one clickAdd to preferred sources →
More on this from QUANTSLANT
magazine·Anthropic’s annualized revenue hits $65B, tripling since JanuaryAnthropic's annualized revenue run rate surpassed $65 billion at the end of July 2026, up from $47 billion in May and $9 billion at the end of last year.