In August 2024 a research lab released a system it called the AI Scientist: an agent that proposes an idea, writes the code, runs the experiment, writes the paper and reviews it, end to end, for about fifteen dollars a paper [2]. It is the most ambitious shape a language-model system can take, because it does not answer a question. It originates. For anyone selling or buying a system that is supposed to think for itself, what happened when independent researchers ran it is the most instructive evaluation of the last two years [1].
What the evaluation found
Three researchers, Joeran Beel, Min-Yen Kan and Moritz Baumgart, ran the system as shipped and read everything it produced [1]. The headline figures are plain. Of the experiments the agent designed for itself, 42 percent failed outright, with coding errors it could not recover from. Of those that ran, the authors found results that were “logically flawed or misleading,” and manuscripts in which numerical results had been invented rather than measured [1].
The literature review, which is where a scientist decides whether an idea is new, was the weakest part. The system classed ideas as novel that are standard textbook techniques; the evaluators give the example of micro-batching for stochastic gradient descent, a method any graduate student would recognize, which the agent presented as a contribution [1]. Papers came back with missing figures, repeated sections, and placeholder text left in the body.
The authors are careful. Their title ends with a question mark, and they call the approach promising. The sample is small; the 42 percent is five experiments of twelve. The point of the evaluation is not that the system is bad. It is that nothing in it stops a wrong result from becoming a confident paragraph.
The same defect, in a bolder system
The failure is the one described in September 2025 by researchers at the company that builds the most-used models: a generative system produces the claim, the evidence for the claim and its own confidence in the claim by one operation, sampling, and nothing inside it checks one against the other [3]. In a chat assistant that produces a fake case citation. In a system that originates, it produces a fake experiment, and then a fake number to report from it, and then a sentence calling it novel. The ambition is higher, so the invention is larger. The mechanism is identical.
This matters for a buyer because “it reasons on its own” is the next thing vendors will sell, after “it cites its sources” has been found wanting. A system that forms its own hypotheses is more useful than one that answers questions, and it is more dangerous in exactly the proportion that it is more useful, unless something outside the model tests the hypothesis before it renders.
What a gate looks like for a system that originates
The evaluation makes the specification easy to write, because every failure it lists is a missing check.
An experiment that failed to run should produce no result, not a reported number. That is a gate on the data: a figure may appear in the paper only if a run produced it, and the run is attached. An idea may be called novel only if a search of the prior literature came back empty, with the search attached. A conclusion may be stated only if the result that supports it is in the paper, quoted, and a reader can follow the line from the claim to the measurement.
None of those checks is a model’s judgment. Each is a condition code can test: does the artifact exist, does the quote appear in it, does the chain close. A system that originates can be held to that standard without losing the thing that makes it useful. The hypothesis is still bold. It is simply not asserted until each sub-claim it rests on has been grounded, and when the grounding is missing, the hypothesis is reported as what it is: open, with the proof it still owes named in plain words. That is the difference between a leap and a hallucination, and it is the difference the AI Scientist’s architecture did not draw.
Why a lawyer should care about a science experiment
Because the next legal tool will be built this way. Investigation, not answering, is what a defense file needs: what does this record turn on, what would defeat the charge, what should we go and find. A tool that does that is worth having. The evaluation above is what it looks like when that ambition ships without a gate, and ABA Formal Opinion 512 is the reminder that the lawyer who signs the result owns every invented number in it [4].
Ask any vendor who says their system “reasons” or “investigates” the questions the evaluators asked. What happens to a result when the step that should have produced it failed? What stops a conclusion from being stated when its evidence is missing? Can I follow every claim back to the thing that produced it? If the honest answer is “the model is pretty good about that,” you have the AI Scientist, and the evaluation tells you how that goes.