Nvidia critiques skill evaluation methods in new ACES paper

Nvidia just told the AI industry that its standard way of grading agent skills is basically checking homework without reading the answers. The company’s new research paper, published on August 20, introduces a framework called Agentic Continuous Evaluation of Skills, or ACES, that replaces static code checks with live head-to-head trials to measure whether a skill actually makes an AI agent better at its job. The core finding is damning for existing methods. The correlation between traditional scan-only metrics and LLM-judge scores came in at a Spearman rho of just 0.14. For context, a perfect correlation would be 1.0. A coin flip would be closer to zero. The current industry standard for evaluating agent skills lands uncomfortably near the coin flip end of that spectrum. How ACES actually works The framework’s approach is deceptively simple in concept. Run an AI agent on a task with a given skill enabled, then run the same agent on the same task without that skill. Compare the results. The difference is what Nvidia calls “Skill Lift,” a quantitative measure of whether a skill actually moves the needle. In practice, the execution is considerably more involved. ACES conducts paired ...