Limits of AI Intelligence

AI's Limit: The $100 Million Math Equation

• A limit exists to AI intelligence that remains difficult for current tech to overcome.
• AI is exceptional at math and data storage, yet lacks real intelligence.
• The AI boom is built on the idea that more GPUs make for smarter models, leading to increased AI usage and exponentially smarter AI models.
• One Chinese startup claimed it matched ChatGPT's performance at a much lower compute cost; Deep seek got more downloads than ChatGPT.
• The main issue is that improving AI model intelligence is getting harder.

How AI Models (GPT) Think

• GPT is good at predicting the next word in a sentence, performing well on standardized tests.
• GPT struggles with math as it spots patterns from large datasets instead of calculating.
• Models need many parameters, obtained by consuming billions of texts.

Parameters Explained

• Parameters play a key role in training AI models.
• GPT models use pre-trained Transformers for text generation.
• The model divides the input sentence into tokens and classifies each based on its meaning.
• Words are classified by grouping fractions of words called tokens.
• GPT-3 classifies words into 12,288 dimensions, grouping similar meanings together.
• OpenAI's GPT-3 uses about 50,000 tokens mapped into a 12,000-dimension space, needing around 600 million parameters.

Embedding and Transforming

• Embedding converts words into a format computers can understand and process.
• Transforming adjusts a word's meaning based on sentence context, shifting its placement in the 12,000-dimension space.
• Each Transformer in GPT-3 has about 1.8 billion parameters (600 million in the attention layer, 1.2 billion in the feed forward network layer).
• GPT-3 uses 96 Transformers, for almost 174 billion parameters total.
• The output layer un-embeds words, listing possible words with probabilities to predict the next word.

Training and Limitations

• Machine learning adjusts parameters through algorithms to reduce errors by trial and error on a large scale.
• Connections between values create neurons, forming a neural network like human neurons.
• GPT models can understand relationships between words and follow instructions, but have limits.
• The 175 billion parameter GPT-3 first struggled with human-like writing, counting, and limited context processing.
• For GPT-4, OpenAI scaled data and parameters, using 1.8 trillion parameters and 25,000 GPUs for 3 months.

The Wall and the Plateau

• OpenAI faces diminishing returns, where more size doesn't improve performance much.
• The amount of data for training AI to near perfection exceeds the available amount.
• The Transformer model marks a scientific breakthrough, enabling models to predict protein shapes and generate images.
• Models are very powerful, reverse engineering 3D worlds from training data, creating implications for 3D artists.

Potential Solutions and Reasoning

• Deep seek's efficiency suggests a possible workaround to data and parameter limits.
• The next stage involves connecting AI brains to other systems, providing sight, hearing, and physical abilities.
• New models aim to interpret questions, divide them into subtasks, solve them sequentially, and analyze the response, similar to human thought.
• This iterative thinking (Chain of Thought) helps models outperform humans on general intelligence tests and in writing code.

Real-World Challenges and the Future

• AI has difficulty with common sense, creativity, real-world decision-making, and some advanced mathematics.
• Processing limits affect decision-making speed and capabilities.
• As the line blurs between human and computer skills, this raises questions about the future of work and the nature of thinking.