Why Vision Models Fail to Generalize, and Why Locality Might Fix It

A recurring failure mode in language models, the tendency to latch onto shortcuts rather than learn generalizable reasoning, turns out to have a direct analogue in vision. A new paper, "On Locality and Length Generalization in Visual Reasoning," accepted at ECCV 2026, traces this weakness to how vision models process spatial information, and proposes a fix rooted in restricting how much of an image a model is allowed to see at once.

The researchers, who submitted the work to arXiv on July 10, 2026, built simple visual tasks that require aggregating local information scattered across an image, think of counting objects, tracing connected regions, or combining evidence from multiple small patches into a single judgment. These tasks are deliberately basic, but they are structured so that a model's true reasoning ability can be separated from its ability to memorize patterns tied to a fixed input size.

Global Shortcuts, Local Reasoning

The central finding is that vision models with unrestricted, global access to an image tend to discover shortcuts that work for training-scale inputs but collapse once task length or complexity increases. This is functionally the same failure that has been documented extensively in transformer-based language models, where systems trained on short sequences often fail to extrapolate to longer ones despite the underlying task logic being identical. The paper's contribution is showing that this isn't a language-specific artifact of tokenized sequences: it's a broader property of how models trained with unconstrained access to context tend to optimize for the training distribution rather than the task's actual structure.

The proposed remedy is architectural rather than purely data-driven. By constraining models to recurrent, strictly local perception, meaning the network processes an image piece by piece and must integrate information sequentially rather than attending to the whole scene at once, the researchers observed meaningfully better generalization as task length and complexity scaled beyond training conditions. The implication is that local attention isn't just a computational shortcut for efficiency; it may be a structural requirement for compositional generalization, forcing the model to build up an answer step by step rather than pattern-match against a globally visible input.

Why It Matters

This has practical weight for anyone building vision systems expected to handle inputs more complex than what they were trained on, a common scenario in robotics, satellite imagery analysis, and industrial inspection, where the "training distribution" rarely captures every real-world edge case. If global vision architectures are systematically prone to shortcut learning, then scaling data alone may not close the generalization gap, and architectural choices around locality and recurrence could matter as much as dataset size.

The paper stops short of claiming local models are strictly superior across all vision tasks; global context remains useful for many problems where holistic scene understanding is the point. But it draws a sharper line around when locality becomes necessary rather than optional: specifically, in tasks demanding compositional aggregation of evidence, where the ability to generalize over length and complexity is the actual test of whether a model has learned to reason at all.