A New Format for Chain-of-Thought Could Cut LLM Reasoning Costs by 85%

Eight percent more accuracy for a fraction of the memory footprint: that is the trade researchers are proposing with a new method called Structured Thoughts, detailed in a paper submitted to arXiv on July 11, 2026.

The core problem the paper addresses is a familiar one for anyone running reasoning-heavy LLMs in production. Chain-of-thought prompting lets models work through problems step by step, and it reliably improves accuracy on math, logic, and multi-hop tasks. But the reasoning traces these models generate tend to sprawl. Models re-derive intermediate steps, restate context, and pack their scratchpad with redundant tokens, all of which balloons context length and drives up inference cost without a proportional gain in output quality.

Structured Thoughts tackles this by imposing a format on the reasoning process itself. Instead of letting a model free-write its chain of thought, the framework organizes it into alternating <try> and <outcome> blocks. Each <try> block represents an attempted step or hypothesis, and each <outcome> block records the result of that attempt before the model moves on. The structure effectively turns an unstructured stream of tokens into a sequence of discrete, labeled reasoning units.

That labeling turns out to matter for two reasons. First, when researchers fine-tuned models on data formatted this way, performance on reasoning benchmarks improved by as much as 8.08% compared to standard supervised fine-tuning on the same tasks. The gain suggests that forcing a model to separate "what I'm attempting" from "what happened" produces cleaner, more learnable reasoning patterns than letting the model narrate its thought process in free-form prose.

Second, and arguably more consequential for real-world deployment, the <try>/<outcome> structure makes it possible to prune context aggressively. Because each block is self-contained, a system can discard earlier <try> blocks once their <outcome> has been recorded, keeping only the information still relevant to the next step. In the paper's tests, this context pruning approach delivered an average of 85% savings in memory and context usage across mathematical reasoning tasks, at the cost of an 8.67% drop in performance.

That trade-off is the headline number for infrastructure teams. An 85% reduction in the context a model has to carry through a long reasoning chain translates directly into lower KV-cache pressure, cheaper inference, and the ability to run longer or more complex reasoning sessions within existing hardware budgets. Losing under 9 percentage points of accuracy in exchange is the kind of trade many production systems, especially those bottlenecked by context window limits or GPU memory rather than raw accuracy, would readily accept.

The two results point in different directions and that is arguably the more interesting finding. Structured Thoughts as a fine-tuning target improves accuracy outright, while Structured Thoughts as a pruning mechanism sacrifices some accuracy for large efficiency gains. The framework's real value may lie in letting teams choose a point on that curve rather than being locked into either extreme, an appealing property for anyone deploying long-chain reasoning models at scale where both cost and correctness are under constant pressure.