Three New Benchmarks Put LLMs to Work on Industrial CAD and Systems Engineering

A trio of research papers submitted to arXiv in July 2026 converge on a single question that matters far beyond academic circles: can large language models actually design things, not just describe them? Together, the papers push generative AI into territory long dominated by specialized CAD software and systems-engineering tools, testing whether foundation models can produce mechanically valid parts and structurally sound design documentation.

The most technically ambitious of the three introduces ArtisanCAD, a skill-guided agent built for generating industrial components. Rather than asking a model to output raw geometry, ArtisanCAD relies on an intermediate layer called CAD-IR, a procedural representation that captures parameters, ordered operations, tool bindings, dependencies, and verification rules before anything gets built. That representation is executed through a CATIA-MCP backend, which turns the plan into production-ready B-Rep models, the boundary-representation format used throughout professional CAD pipelines. On the Text2CAD benchmark, this approach cut the mean Chamfer Distance (a standard measure of shape deviation) from 14.83 down to 9.88 when generating from intermediate prompts, a meaningful jump in geometric fidelity.

Benchmarking Foundation Models Head to Head

A companion paper, accepted as a book chapter in Springer's "Advances in Global Applied Artificial Intelligence" series, takes a broader comparative approach. It evaluates seven foundation models, DeepSeek-V3.2, Qwen3-235B-A22B, Llama-3.3-70B, Gemma-3-27B, GLM-4.5, MiniMax-M2.1, and INTELLECT, using a framework called LLMForge. The system combines JSON-schema validation, analytic feature scoring, mesh synthesis, and iterative refinement across four geometry families: plates with bolt circles, multi-feature boxes, flanged cylinders, and L-brackets.

Two evaluation modes reveal how much the choice of critic matters. Under IterTracer, which relies on a Phong-shaded ray-tracer for analytic visual scoring, the top four models clustered tightly, scoring between 0.885 and 0.890 with a 98.97% mesh success rate. Switching to IterVision, which uses Qwen2.5-VL-72B as a vision-language critic, produced a different signal entirely: the leading model achieved 100% watertight mesh generation, suggesting semantic, human-like critique surfaces different strengths than geometric analysis alone.

Beyond Geometry: Testing Design Structure Matrices

The third paper, authored by Theo Hofman and submitted on July 7, 2026, shifts focus from shapes to systems engineering. It proposes a black-box framework for scoring how well LLMs generate Design Structure Matrices (DSMs) from technical documentation, comparing model-generated matrices against manually validated ground truth. The evaluation blends structural metrics like Completeness and Coupling Density, classification metrics such as Selective Accuracy, and stability measures including entropy and Fleiss' kappa, rolled into a single Composite Quality Score.

Tested on both a fictive abstract system and a real refrigerator decomposition, the framework found that LLMs generate structurally plausible, reproducible DSMs when inputs are well organized, but falter under ambiguous or inconsistently defined dependencies. That mirrors earlier findings from Auto-DSM research published in March 2026 in Research in Engineering Design, where GPT-4 hit 0.981 accuracy on plain-language dependency descriptions and a perfect 1.000 on patent-claim-style inputs, reinforcing that input structure, not raw model capability, is often the deciding factor.

Taken together, the three studies suggest the bottleneck in AI-driven engineering design is shifting from geometry generation toward representation quality: how information is structured, verified, and critiqued before a model ever touches a part.