AI Scaling and System One vs. System Two Thinking

AI Progress & Scaling

• AI's progress in the last 5 years is largely due to increased data scale and compute, rather than algorithmic breakthroughs.
• The Transformer architecture (2017) and training methods similar to those of 2019 still form the basis of current AI.
• Training GPT-2 in 2019 cost about $5,000, but current frontier models can cost hundreds of millions of dollars.
• The high costs of scaling raise concerns that AI progress may plateau.

System One vs. System Two Thinking

• Training an AI to play poker showed the importance of slower, more methodical "system two" thinking, as opposed to fast, intuitive "system one" thinking.
• In poker, giving the AI 20 seconds to "think" improved performance as much as scaling up the model and training by 100,000x.
• Daniel Kahneman's "Thinking, Fast and Slow" describes System 1 as fast and intuitive, while System 2 is slower and more methodical.

Poker AI Redesign and Success

• The poker AI was redesigned to emphasize scaling up system two thinking, in addition to system one.
• The redesigned AI beat top poker professionals by a large margin in a 120,000-hand competition with a $200,000 prize in 2017.
• The AI's dominance was so clear that betting on the competition was stopped.

Thinking Time in Games

• IBM's Deep Blue (1997) thought for a couple of minutes before each move when it beat Garry Kasparov in chess.
• DeepMind's AlphaGo (2016) also took time to think before moves when it defeated Lee Sedol in Go.
• A study showed that in games, increasing thinking time by 10x was roughly equivalent to scaling up model size and training by 10x.

Implications for Language Models

• Training frontier language models costs hundreds of millions of dollars, but querying them costs fractions of a penny.
• Scaling up system two thinking (spending more per query) is a viable alternative to scaling up system one training (spending billions on a model).
• OpenAI's 01 series of language models think before responding, which benefits from increased thinking time.
• Scaling up system two thinking is relatively untapped compared to system one training.

01 Impact and Future

• 01 takes longer and costs more than current models but can be worthwhile for important problems.
• 01 has already saved users, including researchers, days of work.
• The future of AI progress involves scaling up system two thinking, which is just beginning.