Teach a neural network to tell apples from pears. Then teach the same network to tell cats from dogs. Go back and ask about apples, and it flounders. It has forgotten what it learned — not gradually, but almost completely.
The name for this is catastrophic forgetting. It has been known for over thirty years and remains one of the most stubborn problems in machine learning. Nothing like it happens in people: learning to drive does not erase knowing how to ride a bicycle. It happens in networks because everything a network "knows" lives in the same set of weights. When a new task pulls those weights toward its own purposes, the trace of the old one is written over.
Same digits, rotating worlds
Measuring this forgetting takes a clean setup: the tasks must be similar enough for comparison to mean something, yet different enough to strain the network. The common solution is elegant. Take MNIST, the classic dataset of handwritten digits, and rotate the images by a fixed angle for each task.
Same digits, same classes, same difficulty. The only thing that changes is the tilt of the world. A network that has learned the zero-degree task meets a genuinely new distribution when it moves to forty-five degrees — and starts to lose the first one.
Four strategies against forgetting
The literature approaches this with four distinct philosophies. All of them manage the same tension: the network must stay flexible enough to learn the new thing and rigid enough to keep the old.
Methods Four answers to one problem
- EWC — First measures which weights were critical to the old task, then makes moving them expensive. A kind of selective rigidity.
- SI — Makes the same idea continuous: importance is updated step by step throughout learning. A weight that stops mattering is released.
- A-GEM — Keeps a small sample of examples from earlier tasks. Before each step it checks whether the step would hurt performance on those examples; if so, it changes direction.
- LwF — Stores no data at all. Instead it records the answers the network's earlier self gave, and requires the new network to stay faithful to them. The network becomes the teacher of its own past.
The real variable is not the algorithm
So far the question has been: which method is best? This study asked a more interesting one. How much does the order of the tasks matter?
Answering it requires being able to say how hard one task is, and how different two tasks are. A method called Task2Vec does exactly that: it runs a dataset through a pre-trained network and produces a vector representing the task. Once tasks are vectors, the distance between them becomes measurable.
Two measures follow. Total complexity is the sum of the individual task difficulties — how heavy is the sequence as a whole? Sequential heterogeneity is the sum of the gaps between consecutive tasks — how sharp are the turns along the way?
Both measures rise clearly together with the final error rate. How much a sequence will make a model forget can be estimated from the structure of the sequence itself, independently of which algorithm you pick.
Who is good where?
The algorithms also reveal distinct temperaments. A-GEM does very well with a small number of tasks and smooth transitions, but struggles when the jumps between tasks are abrupt — the small memory it keeps is not enough to represent a very different task.
EWC and SI are strikingly indifferent to sequence structure: eight tasks or eighteen, ordered or scattered, they return comparable results. LwF comes into its own on long sequences; variety is not a problem for it, and arguably an advantage.
What does this mean?
The practical upshot: if you are building a system that learns continually, half the job is choosing the right algorithm, and the other half is deciding in what order to show it the tasks. The curriculum matters as much as the teaching method.
The limits are equally clear. The work used a single dataset and a single family of tasks. Rotated digits are a clean laboratory, not the mess of the real world. The vectors Task2Vec produces also depend on the network chosen to produce them — the measuring instrument carries an assumption of its own. None of this invalidates the result; it marks how far it travels.
What is this article based on?
This piece draws on Apprentissage Continu de Tâches Structurellement Reliées, carried out at Université d'Évry Paris-Saclay (Nadjib Atamnia, Selim Kusursuz, Omar Gharib; supervisor: Prof. Massinissa Hamidi, 2024–2025).
- Kirkpatrick et al., Overcoming catastrophic forgetting in neural networks, PNAS, 2017.
- Zenke, Poole, Ganguli, Continual learning through synaptic intelligence, 2017.
- Chaudhry et al., Efficient lifelong learning with A-GEM, 2019.
- Li, Hoiem, Learning without forgetting, 2017.
- Achille et al., Task2Vec: Task embedding for meta-learning, ICCV, 2019.
- Nguyen et al., Toward understanding catastrophic forgetting in continual learning, 2019.
Selim Kusursuz
Studies Computer Network Systems at Paris-Saclay, on the Autonomous Systems track. Works on mathematics and computer science. Speaks French, English and Turkish well.
See all their articles
Nadjib Atamnia
Works on autonomous systems; co-author on the continual learning and multi-agent simulation projects.
See all their articles
