ATLAS Cuts LLM Benchmarking Down to Size, Using 41 Questions Instead of 5,600

Forty-one questions. That is all a new evaluation framework called ATLAS needs to estimate a large language model's ability on HellaSwag, a benchmark that normally runs 5,600 items, and still land within 0.157 mean absolute error of the full-scale score.

Described in a paper posted to arXiv, ATLAS is an adaptive testing framework that applies Item Response Theory (IRT), a statistical approach long used in human educational assessment, to the problem of evaluating LLMs. Rather than running every model against every item in a benchmark, ATLAS selects questions dynamically based on Fisher information, a measure of how much a given item can narrow down uncertainty about a model's underlying ability at any point in the test. Each answer updates the system's estimate, and the next question is chosen to be maximally informative given what's already been learned.

The efficiency gains claimed are substantial. According to the paper, ATLAS can cut the number of benchmark items needed by as much as 90% without sacrificing measurement precision. On HellaSwag specifically, that means compressing a 5,600-item evaluation down to 41 targeted questions while still matching the ability estimate the full benchmark would produce.

Why Raw Accuracy May Be Misleading

The more consequential finding may be what happens to model rankings once ATLAS's ability estimates replace simple accuracy scores. Testing across more than 3,000 evaluated models, the researchers found that 23% to 31% of them shift by more than 10 rank positions when scored through ATLAS instead of raw accuracy. That is not a rounding error: it suggests a meaningful share of published leaderboard positions may reflect noise in benchmark design as much as genuine differences in capability.

Standard benchmarks treat every question as equally diagnostic, when in practice many items barely discriminate between weak and strong models. IRT-based scoring corrects for that by weighting items according to how much information they actually carry about ability, which is also what allows ATLAS to discard the vast majority of a benchmark's questions without losing accuracy.

Practical Implications

For teams running frequent model evaluations, whether for internal fine-tuning checkpoints, vendor comparisons, or research ablations, the cost of exhaustive benchmarking adds up quickly in compute and API spend. A framework that reaches statistically comparable conclusions with a fraction of the items could meaningfully cut evaluation overhead, particularly for organizations testing many model variants or benchmark suites at scale.

The authors have released both the code and the calibrated item banks publicly, meaning other researchers can apply ATLAS to their own benchmarks rather than starting calibration from scratch. Given the scale of the rank shifts observed, the work also raises a broader question for the field: how many current leaderboard comparisons would hold up if scored this way instead.