AdaStop Cuts DNN Testing Costs by Knowing When to Stop Looking for Bugs

Testing a deep neural network for hidden faults is expensive, and most teams have no principled way to know when they've tested enough. A new framework called AdaStop, detailed in a paper submitted to arXiv on July 6, 2026 by lead author Bonan Shen, tackles that exact problem by treating DNN testing as a cost-benefit calculation rather than an open-ended search.

The core idea is deceptively simple. Instead of exhaustively labeling test inputs until a budget runs out or engineers get tired, AdaStop continuously tracks the marginal rate at which new faults are being uncovered as testing proceeds. It compares that rate against a threshold, defined as τ = c/v, where c represents the cost of labeling another test sample and v represents the value of finding one more fault. Once the estimated discovery rate drops below that threshold, the framework calls it quits, reasoning that further labeling is no longer worth its price.

That threshold-based stopping rule turns fault detection into an economic decision. Rather than asking "have we tested enough to feel confident," AdaStop asks "is the next test sample likely to be worth what it costs to label." The distinction matters for teams operating under real budget and time constraints, where labeling data, especially for specialized domains or safety-critical systems, can be the single most expensive part of the validation pipeline.

The Efficiency Numbers

According to the paper, experiments run across multiple datasets, network architectures, and test-input selection strategies show AdaStop recovering between 65% and 84% of the faults that exhaustive testing would have found, while consuming only 9% to 31% of the total labeling budget. That's a substantial compression: in the best cases, a team could theoretically capture roughly two-thirds to four-fifths of a model's discoverable bugs by spending less than a third, and sometimes less than a tenth, of what full-scale testing would require.

The range in those figures (65-84% of faults, 9-31% of budget) reflects how performance shifts depending on the dataset and selection strategy paired with AdaStop. That variability is expected in adaptive testing research, since the method's efficiency depends heavily on how faults are distributed across the input space and how good the underlying selection strategy is at surfacing them early. Still, the consistency of the pattern across multiple experimental setups, aggressive savings paired with a strong majority of faults caught, is what makes the results notable rather than anecdotal.

For engineering teams building and validating neural networks, the appeal of a framework like AdaStop isn't just speed. It's the ability to justify, in economic terms, exactly when testing has reached diminishing returns. As labeling costs and model complexity both continue to climb, having a formal stopping rule instead of a gut-feel cutoff could become a standard part of the DNN validation toolkit rather than a research curiosity.