The hackathon, organized by the computing and research community platforms Hugging Face and alphaXiv, posed an ambitious question: Can AI agents make it possible to check scientific work at a scale that’s increasingly impossible for human reviewers?

The contest was the largest attempt to date to use AI to replicate papers presented at a major technical conference. Many of the participants are not AI experts by training. Nonetheless, they ultimately tackled a total of more than 2000 papers – and uncovered hundreds of results that the AI agents could not reproduce or found unsupported by evidence. So far, the hackathon organizers have confirmed that at least a dozen papers contain real errors that the conference’s human reviewers missed.

In recent years, computer science conferences have faced an explosion of submissions. This year, Conference on Machine Learning (ICML) accepted some 6000 papers, nearly twice as many as it did the year before. But the pool of human reviewers has not grown nearly as fast, an imbalance that has brought the peer-review system close to a breaking point. Whether AI can aid peer review is the subject of fierce debate, as the process often requires judgments about novelty or significance.
 

Source https://www.science.org/content/article/researchers-presented-6000-papers-major-meeting-could-ai-reproduce-their-findings

Another high-profile case has interested scientists. Read now