Google WikiProfile Study Finds Factual Recall Is the Bottleneck
A 2,150-fact benchmark separates whether a model can reveal a fact in familiar context from whether it can produce that fact when directly questioned.
Google Research published a study on August 12 that separates two reasons a language model can answer a factual question incorrectly: the benchmark may find no behavioural evidence that the model encoded the fact, or the model may expose the fact in a familiar context but fail to recall it when asked directly. On WikiProfile, the authors report that recall was the larger bottleneck for the frontier models they tested.
The distinction matters to model developers and people evaluating factuality because a single accuracy score collapses different failure modes. If a fact is absent under every probe, broader training coverage may help. If the model can complete the fact in a familiar context but cannot retrieve it across question forms, post-training, prompting or inference-time reasoning may be the more relevant intervention.
WikiProfile tests facts through several behaviours
WikiProfile contains 2,150 facts derived from Wikipedia. Each fact is paired with ten tasks covering proposition completion, contextual questions, direct and reverse questions, and multiple-choice recognition. Google says the questions went through automated generation, refinement and search-grounded filtering, followed by final manual validation.
The framework uses behavioural definitions. A fact counts as encoded when the model can reproduce it in a context similar to material it might have seen during pretraining. A fact counts as known when the model answers semantically equivalent questions across different phrasings and directions. Recall means producing an encoded fact in those question tests.
That wording is important: the study does not inspect a model's parameters and locate a literal stored record. It infers a knowledge profile from model behaviour under designed prompts. The result depends on whether those probes reliably distinguish a missing fact from one that is present but hard to access.
The evaluation sampled about 4.5 million responses
Google evaluated 13 language models with and without a thinking step. For every model, fact and task, the researchers sampled eight responses. Prompted language-model autoraters graded approximately 4.5 million outputs.
For Gemini 3 Pro and GPT-5, the authors report that 95% to 98% of benchmark facts met the study's encoding test. The same models still failed to recall 26% to 34% of facts directly. With thinking enabled, they continued to miss 11% to 12%. Those percentages apply to this benchmark and its behavioural definitions; they are not general hallucination rates for all topics or product settings.
The study also reports that thinking-optimised models recovered roughly 40% to 65% of facts judged encoded but not directly known. For facts that did not meet the encoding test, thinking helped only 5% to 15% of the time. The authors interpret that gap as evidence that a reasoning step often helps retrieve available information more than it constructs an answer from facts the model lacks.
Rare facts and reverse questions exposed the gap
The paper finds that rare facts were encoded at rates closer to popular facts than their direct-recall rates would suggest. It also reports a directionality gap: models could recognise answers to reverse questions in multiple-choice form more reliably than they could generate the answers unaided. In both cases, changing the retrieval conditions altered performance even when another test suggested the fact was available.
For practitioners, that supports evaluating more than one phrasing and more than one response mode. A model that recognises the correct answer among options may still fail in open-ended use. Conversely, a model that succeeds after an extended reasoning step may be more expensive or slower than its direct-answer score implies.
The benchmark does not settle factual reliability
WikiProfile is built from Wikipedia facts, and its construction relies heavily on prompted models. Responses are also graded by LLM autoraters. Although Google describes filtering and manual validation, the fetched article does not provide an independently replicated error analysis for every generated task or grade. The benchmark therefore offers a diagnostic lens, not proof that nearly all real-world facts are stored by frontier models.
The next useful checkpoints are external replication, human audits of the benchmark labels and tests across domains where facts are newer, contested, specialised or absent from Wikipedia. Product evaluations will also need to ask whether recall improvements remain reliable under real prompts, tool use and retrieval systems.
Status
Learning. Internal confidence is medium because the benchmark design and results come from Google Research and were not independently replicated in the evidence reviewed for this article.
Sources
Update note: Last reviewed 2026-08-17. We will revise this post if an external team audits WikiProfile or reproduces the reported recall profiles.
Sources
Drafted with AI assistance from source briefs; reviewed for citation completeness and label accuracy.