Natural Ungrokking: How Language Models Forget Rules During Training

Researchers discovered 'natural ungrokking'—where a small language model forgets learned rules (e.g., pronoun-gender resolution) mid-pretraining despite no loss curve changes, with survival tied to corpus statistics.

One continuously updated timeline instead of dozens of separate articles. New developments are appended as the story evolves.

1. UpdateJun 24, 2026, 05:27 PM 88%

   Researchers uncover 'natural ungrokking'—language models forget learned rules mid-pretraining without loss curve changes.

   Researchers discovered 'natural ungrokking'—where a small language model forgets learned rules (e.g., pronoun-gender resolution) mid-pretraining despite no loss curve changes, with survival tied to corpus statistics.

   Read the full story →