Researchers Discover 'Natural Ungrokking' in Language Model Pretraining
Reported by arXiv cs.AI: Natural Ungrokking: Asymmetric Control of Which Rules Survive Pretraining. Analysis and context written by TickrWire.
Researchers discovered 'natural ungrokking'—where a small language model forgets learned rules (e.g., pronoun-gender resolution) mid-pretraining despite no loss curve changes, with survival tied to corpus statistics.
- Natural ungrokking describes the mid-pretraining reversal of learned rules in small language models, with no trace in the loss curve.
- A model initially learned pronoun-gender rules (94% accuracy) but later forgot them (near 0% accuracy) despite unchanged training data.
- Rule survival is predictable from corpus statistics, particularly the frequency of rule application in the training stream.
- This phenomenon challenges traditional assumptions about how models retain knowledge during pretraining.
- The study suggests corpus biases play a critical role in determining which rules models retain.
A new study reveals an unexpected phenomenon in language model pretraining called 'natural ungrokking,' where a model initially learns and applies rules (e.g., pronoun-gender resolution) but later discards them without any detectable change in the loss curve. For example, a model trained on a corpus with gendered pronouns initially achieves 94% accuracy on held-out probes by step 925 but scores near zero by step 3,500, despite the rule's evidence remaining in the training data. The survival of learned rules is predictable based on corpus statistics, specifically how often the training stream shows the rule's application. This challenges assumptions about how models retain knowledge during pretraining and highlights the role of implicit corpus biases in shaping model behavior.
Highlights the need for better monitoring of model behavior during pretraining to detect rule forgetting, even when loss curves remain stable.
Raises concerns about the reliability of language models in production, as learned rules may degrade unpredictably over training.
Identifies a potential risk in AI model training pipelines, which could impact investments in companies relying on pretrained models.
Provides insight into the complexities of language model training and the factors influencing model behavior beyond loss minimization.
Challenges public perception of AI models as static after training, showing that their behavior can change unpredictably during development.
- natural ungrokking
- The mid-pretraining reversal of learned rules in language models without changes in the loss curve.
- pretraining
- The initial phase of training a language model on a large corpus of text to learn general patterns and rules.
- loss curve
- A graph showing the model's error rate during training; used to gauge learning progress.
- held-out probes
- Test cases not seen during training, used to evaluate the model's generalization ability.
- corpus statistics
- Quantitative measures of the training data, such as frequency of specific patterns or rules.
AI bias estimate: Neutral presentation of empirical findings; no overt opinion or sensationalism detected. (Automated estimate, not a definitive judgement.)
Don’t mistake chatbot intelligence for consciousness - The Economist
Biological AI models: new paradigms to leverage the languages of life - joint-research-centre.ec.europa.eu
China’s Military Says AI Can’t Replace Commanders. Xi Is Testing That - War on the Rocks
SPADE: Self-Play in Adaptive Synthetic Executable Environments
Beyond Teacher Likelihood: Group-Calibrated On-Policy Distillation for Long-Context Reasoning
AI ToolsMeta AI’s new Mac app wants you to talk to your apps
Meta released a new Mac application that lets users control apps and dictate text using voice commands powered by its Muse Spark AI model.
New White House strategy clarifies military tech priorities: undersea, outer space and AI - Breaking Defense
The White House released a new strategy prioritizing military investments in artificial intelligence, space systems and undersea technologies to counter emerging threats.
AI in an iron grip: How dictatorships use artificial intelligence to strengthen their rule - theins.press
A new report examines how authoritarian governments deploy AI for surveillance, censorship, and propaganda to reinforce their power.
Stripe, OpenRouter finally strike a deal - Banking Dive
Stripe and OpenRouter have partnered to integrate Stripe's payment processing with OpenRouter's AI model aggregation platform.
How one Philadelphia school is using AI to strengthen student learning, not replace teachers - CBS News
A Philadelphia school is integrating AI tools to support teachers and improve student outcomes, focusing on collaboration rather than replacement.
Exclusive-How a Texas student blew the whistle on a rogue AI hacking attempt - The Mighty 790 KFGO
A Texas student uncovered an AI-powered hacking attempt targeting local systems, prompting a swift law enforcement response.