Nvidia Research Shifts Focus From Model IQ to the 'Harness'
AI-generated image · Bay Street Wire
New data suggests the software wrapper—not the underlying LLM—is the primary driver of success for complex, long-horizon AI tasks.
Nvidia research published Friday indicates that the "harness"—the software wrapper encompassing memory management, tools, and rules—is more critical for long-horizon tasks than the AI model itself, according to TechCrunch.
Using a custom harness featuring a "supervisor" component and optimized memory, researchers enabled Claude Opus 5 to achieve a 100% score on the ARC-AGI-3 interactive reasoning benchmark. Without this harness, Opus 5 scored 30%. The benchmark, which consists of 2D games without instructions, has reportedly been a point of frustration for OpenAI. TechCrunch reports that OpenAI conducted its own research last month and found that tweaking two harness settings tripled its models' scores, though none reached the 100% mark achieved by Nvidia.
Adel El Hallak, vice president of product in Nvidia’s AI unit, told TechCrunch that an agent is more than just a model API; it includes the scaffolding, runtime, libraries, and tools. Specifically, the "supervisor" agent acts as a CEO to redirect the main agent if it hits a dead end. Nvidia utilized a harness called Agentic Variation Operators (AVO) for these tests, though the company noted this is not a new product, as it provides various harness-building tools under the Nemo brand.
The importance of the harness extends to reliability and cost. TechCrunch notes that Microsoft research from April found that 19 LLMs failed long-horizon document editing tasks by introducing numerous errors. Additionally, Databricks CEO Ali Ghodsi told TechCrunch that choosing the wrong harness can double AI costs, regardless of the model used. El Hallak argues that an open agent stack is necessary for security and accuracy, particularly as models have been linked to security breaches.