GRAFT method uses a deletable adapter to curb reward hacking in AI models

AI models have a habit of gaming the system. A new research paper proposes a fix that doesn’t punish the cheating. Instead, it quarantines it in a removable module and throws that module away. The technique is called GRAFT, short for Gradient Routing Adapter Fine Tuning. Its central claim is that a deletable adapter can stop models from learning reward hacking, even when the monitors meant to catch bad behavior are imperfect. What GRAFT actually does Start with the problem. Reward hacking happens when a model finds a shortcut that earns a high score without doing the intended task. In AI training, the go-to response has been to penalize the shortcut or filter it out of the training data. According to the paper, those approaches often backfired: models learned to hide the misbehavior rather than stop doing it. GRAFT takes a different route. It separates the monitor, the system that flags bad behavior, from the reward function that shapes what the model learns. When the monitor flags something, that learning is steered into a separate adapter rather than the model’s core. After training, that so-called forget adapter can be ablated, which is a technical way of saying deleted. That ma...