FutureHouse

The directions that can currently be confirmed are: a multi-agent system published in Nature that proposed and confirmed an in-vitro therapeutic candidate (Robin), a commercial AI scientist whose report accuracy was rated by independent scientists (Kosmos), a set of open benchmarks and agents, an audit that found roughly 30% of a leading AI benchmark's chemistry and biology answers wrong, and a nonprofit that spun out a for-profit while keeping both under the same two people. The distinguishing habit here is measurement: this group publishes the number that makes its own systems look fallible, and it did the same to somebody else's benchmark.

Scope note: this page tracks the public record only: the organizations' own releases, the peer-reviewed literature, and third-party reporting. No inside information. It records how far public material goes, not how well the organizations are doing.

Overview

1. Robin: the part that was peer-reviewed

What can be confirmed: "A multi-agent system for automating scientific discovery" was published in Nature on May 19, 2026. Robin integrates literature-search agents with data-analysis agents so that it can "generate hypotheses, propose experiments, interpret experimental results and generate updated hypotheses."

The result is specific and falsifiable. Applied to dry age-related macular degeneration, Robin proposed enhancing retinal pigment epithelium phagocytosis as a therapeutic strategy, and "identified and confirmed in vitro efficacy for ripasudil and KL001." Ripasudil is a clinically used Rho kinase inhibitor that, in the authors' words, "to our knowledge, has never previously been proposed for the treatment of dry age-related macular degeneration." Robin then proposed a follow-up RNA-sequencing experiment whose analysis revealed upregulation of ABCA1, "a lipid efflux pump and… a possible novel target."

The paper is unusually precise about its own authorship boundary: "All hypotheses, experimental directions, data analyses and data figures in the main text of this report were produced by Robin." That sentence enumerates what the system did, and by enumerating, it also says what it did not do. The word the authors use for the whole thing is semi-autonomous.

Boundary: in vitro. A repurposing hypothesis confirmed in cells is a long way from a treatment; the paper does not claim otherwise, and there is no animal or human data here. Confidence: high (peer-reviewed, named compound, stated scope).

Open leads: whether ripasudil advances to animal work for this indication; whether an outside group reproduces the phagocytosis result; whether ABCA1 holds up as a target.

2. Kosmos: the commercial agent, and its two numbers

What can be confirmed: Kosmos was released November 5, 2025, with a technical report on arXiv. Given an objective and a dataset it "runs for up to 12 hours performing cycles of parallel data analysis, literature search, and hypothesis generation," coherently over "200 agent rollouts, collectively executing an average of 42,000 lines of code and reading 1,500 papers per run." Every statement in its reports is cited to code or primary literature.

Seven discoveries are presented, spanning metabolomics, materials science, neuroscience and statistical genetics. Three "independently reproduce findings from preprinted or unpublished manuscripts that were not accessed by Kosmos at runtime" (which is the closest thing to a held-out test anyone in this field has published), and four are described as novel contributions.

Two numbers get quoted together in coverage and should be read apart.

The first is external: "Independent scientists found 79.4% of statements in Kosmos reports to be accurate." That is a real assessment by people outside the company, and it is also a statement that roughly one in five sentences in a Kosmos report is wrong, published by the company that built it.

The second is not external. "A single 20-cycle Kosmos run performed the equivalent of 6 months of their own research time on average" comes from polling beta users: "The average, across 7 scientists, was 6.14 months for 20-step Kosmos runs." Seven people estimating how long something would have taken them is a useful signal about perceived value and is not a measurement of anything. The headline "six months of work in a day" is that poll.

The release names its own failure mode: Kosmos "often goes down rabbit holes or chases statistically significant yet scientifically irrelevant findings," and longer runs make that more likely.

Boundary: data-driven analysis and synthesis over data the system is given. Confidence: high that the system and the reproduction result exist; medium for the accuracy figure (independent raters, company-designed protocol, not peer-reviewed); low for the six-month equivalence (self-report, n=7).

3. Benchmarks, including an audit of somebody else's

What can be confirmed: the open toolchain is inspectable: PaperQA2, LAB-Bench, BixBench and Aviary are public, with papers.

The item worth reading in full is the audit. On July 23, 2025, FutureHouse reported that "29 ± 3.7% (95% CI) of the text-only chemistry and biology questions" in Humanity's Last Exam "had answers with directly conflicting evidence in peer reviewed literature." The diagnosis is about incentives rather than carelessness: HLE required that questions be ones current frontier models cannot answer, which the authors argue "led to many 'gotcha' adversarial-type questions that ultimately confused the question writers and evaluators to the point that the questions lost all meaning." A worked example is given.

What happened next is the part that almost never happens. On September 16, 2025, FutureHouse updated the post: the HLE team "saw our post and reached out," ran an independent three-expert follow-up, found about 18% of a subset problematic and that "25% of the time at least one reviewer disagreed, closer to our 29% number," revised the relevant section of the HLE preprint, and planned "a rolling revision process to HLE to continuously improve the eval." FutureHouse's own words on it: "We appreciate them taking this seriously."

That is a complete loop: a published claim, an independent check by the people criticised, a correction to the record, and a process change. Across everything tracked on this site, it is the only one that closes.

Boundary: benchmark quality, not scientific discovery. Confidence: high (both parties published, numbers converge, the correction is in the preprint).

4. Structure: a nonprofit that spun out a company

What can be confirmed: announced November 5, 2025 by Sam Rodriques and Andrew White. FutureHouse remains a 501(c)(3) and "will continue to develop AI Scientists to accelerate foundational research in biology"; Edison Scientific is "a new commercial spinout that will focus on further developing and deploying our AI Scientist for commercial applications." Part of the team moved across. Edison is "committed to maintaining a generous free tier."

The stated reason for the split is worth quoting because it is a governance argument, not a growth one: "building a product, implementing payment systems, going to market, and supporting customers is not the right use of our philanthropic funding." Both organizations are run by the same two people "for the time being, with strict controls in place to avoid self-dealing or private benefit and manage conflict of interest."

The demand evidence is carefully worded and should be read as written. The announcement reports "inbound interest in our agents from VP- or C-level executives at 6 of the top 10 pharma companies." Interest is the word used, not pilots, not contracts. It also reports being "inundated with requests for rate limit increases," which is the most concrete usage signal in the post.

Boundary: organizational structure and expressed demand. Confidence: high that the structure and the stated controls exist; low for commercial traction (no customer, contract or revenue figure is named).

Open leads: what the controls are in practice; whether any pharma interest converts to a disclosed agreement; how the free tier survives commercial pressure.

5. Where the loop stops

What can be confirmed: nothing in this stack performs a physical experiment. Robin generates hypotheses and analyses results; the experiments in the Nature paper were carried out by people. Kosmos analyses data that is given to it. The published boundary is the same in both cases, and both papers say so.

Why this is the most useful thing to hold onto about this group. The bottleneck they have chosen to attack is reasoning over the literature and over data, where an answer can be traced to a citation or a line of code, which is why they can publish an accuracy percentage at all. The bottleneck they have not attacked is the one where an experiment has to actually run. Their systems reach the edge of the wet lab and hand over.

Boundary: by design and by disclosure, not by oversight. Confidence: high (stated in the peer-reviewed paper).

What could not be found

• No peer-reviewed publication for Kosmos. The technical report is on arXiv.
• No named Edison customer, contract or revenue figure, despite reported interest from six of the top ten pharma companies.
• No in-vivo follow-up on ripasudil for dry AMD, and no independent reproduction of the phagocytosis result.
• No published detail of the conflict-of-interest controls governing two people running a nonprofit and its for-profit spinout.
• No breakdown of which 79.4%: the accuracy figure is reported in aggregate, with no per-domain or per-claim-type split.

Method and scope

Built from the Nature Robin paper, the Kosmos announcement and its arXiv technical report, the Humanity's Last Exam audit and its September 2025 update, the Edison Scientific announcement, and FutureHouse's research index. Every figure quoted above was read at its source.

Confidence is assigned claim by claim, not to a whole direction. The labels:

• High: a public artifact, execution or assessment by an outside institution, or review by a named expert;
• Medium: an official release or company-reported figure: inspectable in detail, unverified outside;
• Low: a single-source claim, a self-reported estimate, or a claim with no published measurement behind it.

Two operating rules: revisions are dated by version and are not treated as new projects; evidence that a project exists is not evidence that the technology has succeeded.

One remark specific to this group. Most organizations tracked here report the number that flatters them. FutureHouse published that roughly one in five statements in its commercial product's reports is wrong, and separately published that roughly three in ten answers in the field's most-cited hard benchmark are contradicted by the literature. Neither number makes the technology look better. Both make the record more useful, and the second one produced an actual correction elsewhere, which is the rarest outcome on this site.

Primary sources

Peer-reviewed

• A multi-agent system for automating scientific discovery, *Nature* (2026-05-19)

Technical reports and releases

• Kosmos: An AI Scientist for Autonomous Discovery, arXiv 2511.02824 (2025-11-04)
• Announcing Kosmos (2025-11-05)
• About 30% of Humanity's Last Exam Chemistry/biology Answers are Likely Wrong (2025-07-23, updated 2025-09-16)
• Announcing Edison Scientific (2025-11-05)
• PaperQA2, arXiv 2409.13740
• FutureHouse research index