Vision-Language Models Keep Failing to See the World From Someone Else's Eyes

Three separate research papers converge on the same uncomfortable conclusion: today's vision-language models (VLMs) struggle to reason about visual information the way a human casually does, whether that means placing objects correctly, judging what another viewer can actually see, or imagining a scene from someone else's point of view.

The most striking evidence comes from FlipSet, a diagnostic benchmark built to test Level-2 visual perspective taking (L2 VPT), the ability to mentally rotate a scene as if standing in another agent's position. The task itself sounds almost trivial: simulate a 180-degree rotation of a 2D character string as another agent would see it. Yet across 103 VLMs evaluated, the vast majority scored below chance. Roughly three-quarters of the errors weren't random noise either; they systematically reproduced the model's own camera-centric viewpoint instead of the rotated one, pointing to a deep-seated egocentric bias baked into how these systems process images. The paper earned the Best Undergraduate Student Paper award at CogSci 2026.

A second line of work tackles a related but distinct failure mode: models placing objects incorrectly within structured visual tasks. A new MaxSAT-based feedback method, accepted at EPIA 2026 (the 25th Conference on Artificial Intelligence), reframes candidate VLM placements, tested using Sudoku-style grid puzzles, as soft clauses in a partial MaxSAT formulation, while the actual Sudoku rules stay fixed as hard clauses. In practice, this lets a solver take a model's uncertain or partially wrong guesses and reconcile them against rigid logical constraints, then feed corrected guidance back to the model. Researchers ran this feedback loop across a mix of open-source and closed-access VLMs, suggesting the placement problem isn't confined to any single model family or training approach.

The third paper asks a more architectural question: how much of an image does a VLM actually need to "see" to get the answer right? Researchers introduce the Visual Access Boundary (VAB), defined as the minimal region of an image a model must attend to in order to preserve task accuracy. To find that boundary, they use a technique called Visual Access Sweep, a causal intervention that selectively masks attention flowing from generated-token queries to image-token keys, varying the mask across both layer depth and generation timestep. Tested on Qwen2.5-VL-32B and two InternVL3 variants (14B and 38B), the sweep effectively maps out exactly which visual regions and which processing stages actually matter for a given answer, and which are functionally ignored despite being technically "visible" to the model.

Taken together, these findings sketch a consistent picture rather than three isolated quirks. Whether the task is rotating a viewpoint, placing a symbol on a grid, or simply attending to the right pixels at the right layer, current VLMs default to shortcuts tied to their own training-time viewpoint rather than genuine spatial reasoning. That gap matters for any application, from robotics to assistive AI, where a system needs to reason about a scene the way another observer, human or otherwise, actually experiences it.