New Benchmark Exposes How Compiler Optimizations Quietly Break LLM Reverse Engineering

Seven contemporary large language models were asked to do something security analysts spend hours on daily: figure out what a decompiled binary function actually does and give it a sensible name. The results, detailed in a new paper called REFORGE, show that as compiler optimization increases, model confidence and accuracy erode substantially, a finding with direct implications for anyone deploying LLMs in offensive-security or malware-analysis pipelines.

REFORGE (accepted for presentation at the 23rd International Conference on Applied Computing 2026 in Lisbon this October) is not itself a new model. It is a provenance-tracked benchmarking pipeline built specifically to test how well LLMs perform at reverse engineering decompiled binary functions, a task that underpins vulnerability research, malware triage, and software forensics. The nine-page paper, illustrated with four figures, tackles a problem that has dogged prior benchmarks in this space: establishing reliable ground truth for what a compiled, then decompiled, function was originally meant to do.

Building Ground Truth From Source to Binary

The pipeline's core contribution is methodological. REFORGE constructs its function-level ground truth by starting from original C source code, compiling it, then extracting both DWARF debug information and syntactic structure. It aligns these extractions against the decompiled output before evaluation begins. This chain of custody matters because sloppy or approximate ground truth has historically made it hard to trust benchmark scores in binary analysis research. By tracking provenance at every step, REFORGE aims to give researchers a cleaner signal of what models actually got right or wrong, rather than an artifact of noisy labeling.

On top of this pipeline, the authors layer what they call an eight-gate confidence funnel with three-tier stratification, a structured way of operationalizing how uncertain a model's naming decision is, rather than treating every output as a simple pass or fail.

The Optimization Cliff

The headline result comes from a controlled micro-benchmark: high-confidence yield, the proportion of function names produced with strong model confidence, drops from 87.2% at lower optimization levels to just 65.9% as compiler optimization increases. That's a steep falloff, and it matters because optimized binaries are the norm in real-world software, not the exception. Compiler optimizations strip away structural cues, inline functions, reorder instructions, and generally make the decompiled output further removed from the programmer's original intent, exactly the kind of noise that degrades an LLM's ability to infer semantic meaning.

The paper's framing situates this within a broader concern: LLMs are already being folded into live offensive-security workflows, where reverse engineering speed and accuracy have real operational consequences. A benchmark showing that model reliability drops sharply under realistic optimization conditions is a caution flag for teams treating LLM-assisted binary analysis as a solved problem. REFORGE doesn't propose a fix for the optimization cliff itself, but by quantifying it with a rigorous, provenance-tracked methodology, it gives the security and AI research communities a sharper tool for measuring the gap between LLM capability and production-grade demands.