Why AI Agents Keep Misplacing Their Blocks: A 2.5-D Fix Pushes Structural Accuracy Past 94%

Ask a large language model to build a stone tower with a window on the third row, and there's a good chance it will get the geometry wrong. Not because it misunderstands the request, but because it struggles to translate natural language into precise three-dimensional coordinates. That gap is the subject of a new paper by researcher Paul Whitten, submitted in its first version on May 8, 2026, which identifies systematic coordinate errors as the main bottleneck in AI-driven structure building, and proposes a fix that closes much of that gap.

The paper centers on Build What I Mean, a benchmark designed to test whether an AI "architect" agent can translate a plain-language building instruction into an accurate voxel or block structure. On this benchmark, the best prior competing system tops out at 76.3% accuracy. GPT-4o alone reaches 90.3%, a solid baseline but still far from reliable when structures require exact spatial reasoning like symmetry, layering, or overhangs.

The 2.5-D Trick

Whitten's core contribution is a decomposition technique that breaks full 3-D placement into a 2.5-D representation, essentially reasoning about height maps and layer-by-layer footprints rather than asking the model to reason in raw x, y, z coordinates all at once. Ablation studies show this decomposition alone accounts for a 28.7 percentage point jump in accuracy, which is the single largest factor in the paper's reported gains.

Layered on top of this pipeline, GPT-4o-mini reaches 94.6% mean structural accuracy across 160 evaluation rounds. Swap in Nemotron-3 120B running on an NVIDIA Jetson Thor AGX edge device, and accuracy climbs slightly further to 96.0%, using the identical pipeline. That's a notable result: a smaller, edge-deployable setup nearly matches or exceeds a larger cloud model once the coordinate-handling burden is offloaded from the language model itself.

The paper also reports a ceiling of 97.6% accuracy on the benchmark, but notes that this ceiling isn't a limitation of the building pipeline. It's imposed by errors made by the "architect" agent itself, the component responsible for interpreting the initial instruction, suggesting that further gains will require improving instruction parsing rather than spatial execution.

Speed Versus Accuracy Tradeoffs

Accuracy gains aren't free of latency costs, and the paper addresses this directly. Expanding the system prompt with four targeted few-shot examples cuts mean per-request latency threefold, down to 19.7 seconds, while maintaining 95.6% accuracy, a practical tradeoff for real-time or interactive applications where waiting minutes per structure isn't viable.

To test whether these findings generalize beyond a single synthetic benchmark, Whitten ran a transfer experiment across 500 tasks drawn from IGLU, an existing collaborative building and instruction-following dataset. The improvement held, reinforcing that the 2.5-D decomposition isn't simply overfit to Build What I Mean's specific test conditions.

For teams building embodied AI agents, whether in gaming, simulation, or robotics-adjacent planning tasks, the takeaway is straightforward: raw 3-D coordinate reasoning remains a weak spot for even frontier-class LLMs, but restructuring the problem representation, rather than scaling model size, can close most of that gap.