IBM Tests Open-Model Agent Factory for Chip Design
A quantum-error-correction experiment found a much cheaper open-model search, while the faster proprietary result and single-run design limit broader conclusions.
Edited by Tyronne Panaino
IBM Research published an October 13 demonstration in which an AI agent factory searched for hardware improvements to a quantum-error-correction decoder. The study used open frontier models served through llm-d and compared their observed result with a proprietary frontier-model configuration under the same search budget.
IBM reported that the all-open search produced a 52-times speedup over its baseline in high-level synthesis estimates for about $26 of model inference. The proprietary configuration produced a larger 204-times speedup but cost $299, while the open configuration found the smallest-area design. For chip teams, the useful result is not that one unnamed model class won. It is that model choice, serving efficiency and the amount of verified exploration bought by a budget can change the engineering trade-off.
The experiment turned design work into a bounded search
The workload was Relay-BP, a decoder for quantum error correction using a bivariate bicycle code. IBM's agent-factory setup assigned optimization work across agents that modified the decoder implementation and used high-level synthesis feedback to guide later attempts. Each compared configuration received a budget of 100 hypotheses.
That structure matters because hardware optimization is not simply code generation. A proposed change has to preserve correctness and then be evaluated against hardware objectives. In this study, the public result is an automated design-space search bounded by a fixed number of hypotheses, not an autonomous system delivering a production chip.
The experiment therefore offers a practical unit of comparison: what different model-and-infrastructure choices found within the same search allowance. It does not remove the need for engineers to define the decoder, constraints, objectives and acceptance process.
Lower inference cost bought a different kind of result
IBM says the open configuration's inference cost was 11.5 times lower than the all-proprietary setup. The open search used approximately 241 million tokens, compared with about 216 million for the proprietary search, so the lower bill did not come from doing less token work. It came from the economics of the open model and the serving system used for those tokens.
The performance result points in the other direction. The proprietary configuration found the greatest estimated speedup, while the open configuration found a less aggressive speedup and a compact design. That makes the comparison a cost-quality trade-off rather than a claim that open models matched the best observed optimization.
Doubling the open search from 100 to 200 hypotheses increased its reported speedup from 52 times to 60 times. The extra search still helped, but the smaller marginal gain is a warning against assuming that cheaper inference automatically converts into proportional hardware improvement. Better search allocation may matter as much as a larger budget.
llm-d reduced repeated serving work
The open models ran on premises through llm-d, an open-source distributed inference framework. IBM describes cache-aware routing that sends requests toward servers holding reusable context and a design that separates prompt processing from token generation. Those mechanisms suit an agent factory because many attempts revisit the same source code, constraints, prior changes and synthesis reports.
For this workload, IBM reported an approximately 98% prefix-cache hit rate and a blended serving cost of about $0.11 per million tokens. The deployment also served other agent workloads on shared infrastructure. Those are useful observations about the reported setup, but they are not a price guarantee for another cluster, model, utilization pattern or hardware-design task.
The evidence is promising but deliberately narrow
IBM evaluated each configuration in one search run and explicitly says the results do not establish a universal model ranking. The speedups are high-level synthesis estimates, and the stated costs cover model inference rather than the complete hardware-development workflow. Synthesis, verification, engineering time and eventual implementation costs sit outside that figure.
The study also comes from IBM, which developed the demonstration and describes the supporting infrastructure. No independent replication was fetched in this run. That is enough to explain the experiment and its observed trade-offs, but not to generalize the numbers across designs or declare an industry benchmark.
The next useful checkpoint is a repeated evaluation with multiple runs per configuration, full synthesis and verification cost accounting, and results on additional hardware blocks. Evidence from implemented hardware would further separate improvements that survive engineering validation from gains visible only in an early synthesis estimate.
Status
Learning. IBM documented a new chip-design experiment with measurable cost and optimization outcomes. Internal confidence is medium because the only fetched evidence is IBM's first-party account of a single-run comparison.
Sources
Update note: Last reviewed 2026-10-14. We will revise this post if IBM publishes the underlying study in full, independent teams reproduce it or implemented-hardware results become available.
Sources
Drafted with AI assistance from source briefs; reviewed for citation completeness and label accuracy.