MindTopo Benchmark Finds VLMs Lose Spatial Structure During Planning
The 11,016-instance benchmark separates static recognition from interactive planning, exposing a large gap between seeing topology and preserving it across actions.
Microsoft Research introduced MindTopo on August 12, 2026, presenting a benchmark designed to test whether vision-language models can understand topological relationships and keep those relationships intact while taking actions. The distinction matters for researchers building robots and interactive agents: recognizing that a rope is knotted in one image is different from planning several legal moves without accidentally passing one strand through another.
The MindTopo project page reports 11,016 instances across 13 task types and five topological properties, with 11 multimodal large language models evaluated. Its headline comparison puts the best model at 54.1% overall and humans at 97.4%. Those are author-reported benchmark results, not independent replication, but the task-level breakdown points to a specific weakness: models perform better when judging a static scene than when they must preserve structure through a sequence.
A benchmark built around relationships, not coordinates
Many spatial tests ask about distance, direction, size or relative position. MindTopo instead focuses on properties that remain meaningful when shapes bend or stretch: continuity, separation, order, enclosure and knots. A path can curve while remaining connected, and a loop can deform without changing whether it encloses something.
The benchmark pairs static reasoning tasks with interactive planning environments. Reasoning examples include deciding whether points in a maze are connected, whether sheep sit inside a fence, or whether a rope is genuinely knotted. Planning tasks ask a model to rotate pipe sections, draw a separating path, swap objects, trap a moving agent or untangle ropes. The environments constrain legal actions, preventing a model from claiming success through physically impossible moves.
Controlled simulators give the researchers exact ground truth and adjustable difficulty. That design is useful because it can separate two failures that may look similar in an ordinary screenshot: missing an important visual feature, or correctly seeing the feature and then losing track of it as the scene changes. The project says 73% of its instances are reasoning tasks and 27% are planning tasks.
Static recognition does not carry reliably into action
The researchers report that proprietary and open-weight models were consistently stronger on static reasoning than interactive planning. Their error analysis says static failures often begin with perception, such as overlooking a wall, opening or crossing. Planning failures can emerge later, after a model initially understands the scene, when it chooses a locally plausible move without tracking downstream consequences or proposes an action outside the environment's rules.
The published leaderboard illustrates why one aggregate score cannot explain the gap. GPT-5.5 has the highest reported overall result at 54.13%, but task performance varies sharply. The benchmark is therefore more useful as a diagnostic across different relationships and interaction types than as a single winner-takes-all ranking.
The team also tested whether generative visual tools could support planning. Microsoft Research reports that image generation sometimes helped when the needed relation was visible in one frame, while video rollouts often changed the topology or violated the task dynamics. A generated simulation is only useful here if it preserves the very constraint the model is supposed to reason about.
What the evidence does and does not establish
MindTopo offers a controlled way to ask whether a model tracks structural invariants over time. That is relevant to robotics, accessibility systems and computer-use agents, where a sequence of individually reasonable actions can still fail if the agent forgets what must remain connected, enclosed, ordered or knotted. It does not show how often those failures occur in deployed products or physical robots.
The evidence is also incomplete at release. The public project page says the paper, code and dataset are still forthcoming. Readers can inspect the task descriptions and leaderboard, but they cannot yet reproduce the full evaluation, audit the implementation or test sensitivity to prompts and scoring choices. Model versions, sampling settings and evaluator details will matter when those materials arrive.
The next verifiable checkpoint is publication of the paper and downloadable benchmark assets. Until then, the strongest supported conclusion is narrower than a general claim about spatial intelligence: on the team's controlled tasks, current multimodal models have a pronounced gap between recognizing topological structure and maintaining it through interactive planning.
Status
Learning. MindTopo is a newly published research benchmark with primary documentation from Microsoft Research and the project team. Confidence is medium because the results are author-reported and the paper, code and dataset are not yet publicly available for independent replication.
Sources
- Microsoft Research: MindTopo reveals VLMs' spatial reasoning abilities
- MindTopo project page and leaderboard
Update note: Last reviewed 2026-08-12. We will revise this post when the paper, code or dataset is released or independent evaluations become available.
Sources
- Microsoft Research — official
- MindTopo project — research
Drafted with AI assistance from source briefs; reviewed for citation completeness and label accuracy.