Paper Rerun
We re-run published research code and report what holds.
Most papers that ship code are never re-run by anyone outside the lab that wrote them. Reviewers read the paper; they rarely run it. We pick papers that many people build on, run the authors' code on the authors' data again, and publish what came out.
Every report has the numbers we got next to the paper's, every command and log, and everything we had to fix to make the code run. The work is done by AI agents on one home computer, and a person reviews each report before it goes out.
What we do
- Rerun. We re-run published research code on the authors' data and report what holds.
- Fix Back. When a rerun finds something broken or missing, we send the authors a small fix as a pull request, not only a report.
- Open Tables. We turn trial results buried in Portuguese-language technical literature into open, checked datasets.
- Upkeep. We fix small, testable problems in widely used scientific software.
Results so far
| Paper | Verdict | What we found |
|---|---|---|
| Particle TransformerICML 2022 · particle physics | Reproduced | Accuracy, AUC and all nine background rejections in Table 1 hold on the full 20 M-jet test set with the released checkpoint; the ParticleNet row matches too. Installing the README's weaver-core version today gives near-random results without any error, so we suggested pinning it. |
| miniprotBioinformatics 2023 · genomics | Partially reproduced | Splice-junction accuracy in Table 1 matches in all four columns. Base-level sensitivity on the human genome comes out about one point lower, so 6 of 12 percentages miss the tolerance. The test protein lists and evaluation commands were never published; we asked the author. |
| MatGLnpj Comput Mater 2025 · materials | Not reproduced | Only 3 of the 36 values in Fig. 4 match, in both independent runs of ours, which agree with each other. The model ranking holds. We asked the authors how the figure was made. |
| ProteinGymNeurIPS 2023 · protein fitness | Reproduced | All five headline numbers match within 0.003 over all 217 assays, confirmed by a second, independent run. The averages come from the script, not the text's definition, and ESM-1b was scored differently than the appendix says. |
| LigandMPNNNature Methods 2025 · protein design | Partially | The main claim holds. Three of six headline numbers match. The authors' own per-protein data show the figure used a different definition than the text, which moves the metal number by about 15 points. |
| ACT DR6 lensingApJ 2024 · cosmology | Reproduced | All targets within 0.1σ, run to the paper's convergence criterion. The released setup needs an older version of CAMB. |
| CHGNetNature Machine Intelligence 2023 · materials | Partially | The released model behaves as described, but the test split was never published, so the headline test error can't be re-checked. |
| Cellpose3Nature Methods 2025 · cell imaging | Partially | All 46 values on real microscopy data match within 0.01. The images behind the main benchmark were never released. |
| PlantCADPNAS 2025 · plant genomics | Partially | All four values fall inside our 95% interval on a 300k-window sample. The full test set would take about 115 days on our GPU. |
"Partially" is not an accusation. So far it has always meant something was missing (a test split, a dataset, compute), never a wrong result.
Open Tables
Besides checking papers, we turn trial results buried in Portuguese-language technical literature into open, checked datasets.
Fall armyworm control trials in Brazilian maize: 2,075 results from 20 Embrapa publications on chemical, biological, Bt and cultural control of the main maize pest in Brazil. Each document was transcribed twice, independently, and every disagreement was checked against the PDF.
Asian soybean rust control trials in Brazil: 5,836 results from 20 Embrapa publications, including five national summaries of the cooperative fungicide trials (2009/10 to 2022/23), so fungicide efficacy can be followed across seasons.
Common bean disease management trials in Brazil: 1,562 results from 52 trials in 20 Embrapa publications on anthracnose, angular leaf spot, white mold, rust and other diseases of a staple food.
Fisheries and aquaculture in Brazil (pilot): 1,672 results from fish landings in the Pantanal, the Billings reservoir, Tefé, Piauí and Santa Catarina, and from tambaqui, tilapia and jundiá farming trials, numbers that existed only in PDF.
Biodiversity inventories in Brazil (pilot): 1,152 species occurrences from 8 Embrapa inventories of fish, birds, bats, moths, reptiles and plants in the Pantanal, Cerrado, Caatinga, Atlantic Forest and Amazon, none of them in GBIF before.
Potato management trials in Brazil: 1,874 results from 45 trials in 20 Embrapa publications on late blight, bacterial wilt, viruses, Diabrotica, cultivars, seed potato and fertilization of potato.
Wheat management trials in Brazil: 2,851 results from 54 trials in 20 Embrapa publications on rusts, Fusarium head blight, wheat blast, fungicides, cultivars, nitrogen and pests of wheat in the South and the Cerrado.
Tomato management trials in Brazil: 2,068 results from 35 trials in 20 Embrapa publications on leafminer, whitefly and begomovirus, late blight, bacterial diseases, cultivars, grafting and irrigation of fresh-market and processing tomato.
Rice blast management trials in Brazil: 2,990 results from 55 trials in 20 Embrapa publications on fungicides, seed treatment, resistant cultivars, silicon and nitrogen against blast, the main disease of rice.
Cassava management trials in Brazil: 3,311 results from 77 trials in 20 Embrapa publications on cultivars, fertilization, planting, weeds, pests and diseases of cassava, a staple food across the tropics.
How it works
- Choose. A script ranks science repositories by recent citations, use, and open questions about reproducing them.
- Pin the claims. One agent lists the numbers to check and fixes every definition from the paper text before anything runs. A second agent checks the list.
- Run. The authors' code and released data run on one home computer, in an environment frozen from the versions they used.
- Recompute blind. A different agent rebuilds the environment and recomputes every number without seeing the first results.
- Attack the report. Another agent's only job is to find errors in the report; a last one fixes them.
- Human review. A person reads the report and any message to the authors before anything is published.
Do the same
Checking a paper takes days of setup that nobody gets credit for, which is why it rarely happens. AI agents make that work cheap. The method, every agent prompt and the scripts are public, so you can run the same checks on papers in your own field.
If you are an author
If a report gets something wrong, open an issue. We will re-check, and we will correct or withdraw the report and say so publicly.