What Tenet Can Teach Us About How AI Learns From The Past

In Tenet, a machine called a turnstile flips the entropy of people and objects, so they move backward through time while everyone else moves forward.

The film is hard to follow on a first watch, and Nolan never claimed it was real physics.

The more I thought about it, though, the more it looked like a description of how AI models learn.

A model learns from a frozen past, gets shaped by feedback from the people who use it later, and then has to operate in a present it has never seen.

That's a lot of time travel for a piece of software.

Training Runs Backward

The first parallel is almost literal.

A neural network learns through backpropagation.

Data flows forward through the model to produce a prediction, then the error flows backward through every layer to adjust the weights.

Each training step is a forward pass followed by an inverted one, which is a lot like watching a scene in Tenet from both directions.

The model only improves because the mistake travels back to the place where it started.

The Temporal Pincer

Tenet's climax uses a temporal pincer.

One team moves forward through the battle while another moves backward from its end, and each uses what the other has already learned.

That's close to how modern models are built.

Pretraining is the forward team, reading an enormous record of the past in order.

Post-training is the team coming back from the future, where people's feedback about which answers were helpful is used to reshape the model's behaviour.

Neither team alone produces a good assistant.

The pretrained model knows a lot but behaves strangely, and post-training without that knowledge has nothing to work with.

Breathing Through A Mask

Inverted characters in Tenet need respirators, because the air around them is moving in the opposite direction.

A model has a similar problem with its knowledge cutoff.

It's frozen at the moment training data was collected, and everything after that is air it can't breathe on its own.

Web search and retrieval tools are its respirator.

They let a model frozen in the past work in the present, as long as it knows it needs them.

The dangerous case is a model that forgets it's wearing a mask and treats its frozen past as the current day.

What's Happened, Happened

Neil's line, "What's happened, happened," describes a training corpus surprisingly well.

Once a model has learned from data, that data is part of its weights, and removing it later is hard.

The opposite problem is worse.

Whatever wasn't in the record can't be learned at all.

Pew found that 38% of webpages that existed in 2013 were no longer accessible a decade later, and 54% of sampled Wikipedia pages had at least one broken reference link.

The past that future models learn from keeps shrinking.

A lot of human knowledge was never online in the first place.

English makes up about 44% of Common Crawl, while Hindi makes up 0.2% despite being spoken by 7.5% of the world.

One study of medicinal plant uses across three regions found that more than 75% were known in only one local language.

The Algorithm Is Scattered

In the film, the Algorithm is split into nine pieces hidden in different places and times.

Human knowledge looks a lot like that.

Some sits in digitized books, some in private archives, some in oral traditions and some in the habits of experts who never wrote anything down.

A model trained only on what's easy to collect ends up with a few pieces of the Algorithm and the confidence of someone who has all nine.

When The Future Poisons The Past

Tenet's villain receives weapons and instructions from the future.

AI has its own version of that loop.

Models are now writing a growing share of the content that will become training data for the next generation of models.

A 2024 Nature paper found that training on recursively generated data causes "model collapse," where the "tails of the original content distribution disappear."

The rare and unusual parts of human knowledge vanish first.

And the supply of fresh human writing is limited.

Epoch AI estimated that models will fully use the stock of public human-generated text between 2026 and 2032.

I've written about AI agents creating their own programming language, which would add another layer to this, with future data that people can't easily read or check.

Time Lives In The Weights

One of my favourite findings here comes from a 2023 paper by Kai Nylund, Suchin Gururangan and Noah A. Smith.

They fine-tuned a model on text from a single year, then subtracted the original weights to get what they called a "time vector".

Time vectors for neighbouring years sit close together, and blending them improved performance on the years in between, and sometimes on future periods, without extra training.

That's the closest thing I've seen to a turnstile for AI.

It suggests a model's sense of time can be adjusted, instead of being fixed forever by its training date.

How To Fix The Blind Spots

Keeping more records helps, but it isn't enough on its own.

These are the approaches I'd prioritise.

The first is to label data with time and origin.

If every document carried when it was written and whether a person or a model wrote it, training could weight sources properly and avoid feeding models their own output by accident.

The second is to protect the tails on purpose.

Labs and governments could fund digitization of local archives, oral histories and low-resource languages, then deliberately upweight that material during training.

The third is to keep a clean reserve of human data.

I picture something like a seed vault, a preserved snapshot of the human-written web from before AI content took over, that every new model can learn from.

The fourth is to teach models about their own time.

That means training them to know their cutoff, to recognize when a question is about the present and to reach for retrieval tools instead of guessing.

The fifth is to test from the future.

Labs can measure how models handle events after their cutoff, the same way Tenet's teams send information back from the end of the battle.

Conclusion: We Are The Past Of Future Models

The twist in Tenet is that the Protagonist set up the whole mission from the future.

I think we're in a similar position.

What we write, archive and delete today becomes the past that future models learn from.

If we let the record shrink and fill it with machine-written content, those models will inherit a smaller, flatter version of us.

If we protect the rare parts and label what's real, we can hand them a much better past.

My Take: AI can't change what's happened, but we still decide what becomes the record, and that may be the most important training decision of the next decade.

Sources

Pages I opened for the facts in this post:

• The Business Standard: Tenet explained
• Science & Film: Entropy and the end of the world in Tenet
• Pew Research Center: When online content disappears
• Slashdot: AI systems miss vast bodies of human knowledge
• Oxford OATML: AI models collapse when trained on recursively generated data
• Epoch AI: Will we run out of data?
• arXiv: Time is encoded in the weights of finetuned language models