In the Weights: Redefining AI Model Recall in a Post-Search Landscape

Thomas Dimson and Joey Flynn, former OpenAI researchers, have launched "In the Weights," a novel platform engineered to assess the intrinsic knowledge of various large language models (LLMs). This initiative aims to shift the paradigm from conventional "vanity searches" on web engines to evaluating how deeply AI models have absorbed information about individuals and concepts within their training data, without recourse to real-time internet queries.

Dimson and Flynn, who joined OpenAI following the acquisition of their design startup Global Illumination, developed In the Weights with the explicit belief that traditional Google vanity searches are becoming an outdated metric in an era where an increasing volume of information retrieval is mediated by LLMs. Their platform queries a diverse array of prominent AI models, including iterations of GPT, Grok, Gemini, Claude, and Llama, to gauge their internal recall capabilities.

Methodology and Insights

The core methodology of In the Weights involves a standardized prompt: "Who is ? Give up to 10 results, each with a short description and confidence." Based on the models' responses, the platform assigns a "strength score" to each queried name. This score quantifies how robustly a person or entity is recognized and described by the AI models without external lookups.

Early results from In the Weights have revealed intriguing patterns. Notably, figures such as actor Macaulay Culkin and opera legend Luciano Pavarotti both achieved an impressive strength score of 988, indicating a high degree of consistent recall across the tested LLMs. The platform's interface itself leans into a Nintendo-inspired retro aesthetic, contrasting with its forward-looking analytical purpose.

By providing a clear metric for internal model knowledge, In the Weights offers developers and AI engineers a new tool to understand the nuances of different LLMs' foundational recall. This distinct approach highlights the evolving landscape of information accessibility and the growing importance of an AI model's embedded understanding as opposed to its real-time search capabilities.