The 'One-Step Trap' Hindering AI Long-Term Predictions
AI-generated image · Bay Street Wire
Rich Sutton warns that relying on iterated one-step predictions leads to compounding errors and computational infeasibility.
AI research frequently falls into what Rich Sutton describes as the "one-step trap," a misconception that an agent's long-term predictions can be reliably generated by iterating a series of one-step predictions. According to reporting from Hacker News, Sutton argues that while this approach is appealing—drawing parallels to physics or realistic simulators—it is fundamentally flawed in practice.
Sutton notes that for this method to work, one-step predictions must be perfectly accurate. When they are not, errors compound and accumulate, leading to poor long-term results. Furthermore, Sutton explains that calculating long-term predictions from one-step models is prohibitively complex. In a stochastic world, the future represents a tree of possibilities that must be weighted by probability, making the computational complexity exponential relative to the length of the prediction.
Despite these failures, Sutton observes that one-step models remain widely utilized in Bayesian analyses, control theory, compression theories of AI, and POMDPs. As a solution, Sutton advocates for the use of temporally abstract models of the world through GVFs and options, referencing his previous work on the Horde architecture and frameworks for temporal abstraction in reinforcement learning.