Could a superintelligence never be wrong?

In films and headlines, superintelligence is usually drawn the same way: a machine that knows everything, never errs and keeps making itself bigger. It is a striking picture. But how much of it do mathematics and physics actually allow?

The short answer cuts both ways. A system that learns far faster than we do, predicts better and improves its own algorithms contradicts no known law. By contrast, "never making a mistake", "knowing everything" and "computing without limit" cannot be defended within those laws. This piece goes through where those limits come from, one by one.

Superhuman is not the same as unlimited

The first trap in the debate is mixing up two words. A system can be far more capable than a human and still run on finite energy, in finite time, with incomplete observations. Being superhuman does not mean being unlimited.

So two levels need to be kept apart. The first is effective intelligence: the ability to produce better models, better algorithms and better plans with the same resources. The second is physical computing capacity: the upper limit on the information processing a system can do with its energy, mass, volume and time.

They are not the same thing, because operations per second are not intelligence. Picture one method that solves a problem in 2ⁿ steps and another that solves it in n² steps. When n is fifty, the first needs about a quadrillion steps; the second, two thousand five hundred. A better algorithm, found without touching the hardware, closes a gap that the fastest processor never could.

Where intelligence cannot see

The first obstacle to never erring has nothing to do with intelligence and everything to do with information. No system sees the world as it is; it sees it through measurements, and measurements are incomplete or noisy.

If the same observation fits two different states of the world, thinking alone cannot settle for certain which one is real. A fact that no sensor reaches does not turn into knowledge just because intelligence goes up. This is not a shortfall of reason but a limit of access.

Questions no algorithm can answer

The second obstacle comes from mathematics itself. In 1936 Alan Turing showed that there can be no universal algorithm that correctly says, for every program, whether that program will eventually stop. This is not a hurdle that faster computers will clear; it is an impossibility built into the question. So a system that "always decides correctly about every program" cannot be promised.

In 1931 Kurt Gödel showed that every consistent formal system strong enough to express arithmetic, and whose axioms can be listed mechanically, contains statements that are true yet cannot be proved inside that system.

Gödel's result should not be shortened to "AI can never know some truths". A system can move to stronger axioms. The limit it sets is a different one: there is no single, closed system that captures every truth.

The energy bill of thinking

The first limit on the physics side is about heat. In 1961 Rolf Landauer showed that irreversibly erasing a bit is not free: the operation must release at least k_BT ln 2 of energy into its surroundings, where k_B is Boltzmann's constant and T the temperature.

At room temperature, 300 kelvin, that comes to about 2.87 × 10⁻²¹ joules. With one joule, under ideal conditions, roughly 3.5 × 10²⁰ bits could be erased. As the temperature drops, the cost drops in the same proportion.

It matters not to misread this number. It is not "the amount of thinking that fits in a joule"; it is only the lower limit for operations that lose information. Approaches that compute reversibly, without erasing information, can avoid part of this cost. Real devices, meanwhile, spend far more than this limit on cooling, error correction, clock signals and communication.

No thinking faster than light

Does making a mind bigger make it faster? Relativity puts a brake on the idea: information cannot travel faster than light. A message between the two ends of a system of size d takes at least d/c to arrive.

Beyond a certain point, adding more computing matter does not produce one fast mind; it produces a network of agents linked with delays. The parts of a mind spread across a galaxy would have to wait tens of thousands of years to agree with one another. More matter does not mean a proportionally faster single thought.

How much information a kilogram can hold

Can infinite information fit into a finite region? In 1981 Jacob Bekenstein said no: a system of given radius and energy has a universal upper limit on the entropy, and therefore the information, it can contain.

By the review's calculation, a one-kilogram mass squeezed into a radius of ten centimetres has a limit of about 2.6 × 10⁴² bits. That is not usable memory but an extremely loose theoretical ceiling. What it says is simple: a finite place cannot hold unlimited information.

Speed has a limit too. In 1998 Norman Margolus and Lev Levitin showed that the shortest time a quantum system needs to move into a completely distinct state is bounded by its energy. In 2000 Seth Lloyd combined these limits in a thought experiment called the "ultimate laptop": one kilogram of mass in one litre of volume.

Counting all of its mass as energy, this imaginary device could perform about 5.4 × 10⁵⁰ elementary operations per second and hold roughly 10³¹ bits. These are not engineering forecasts; they are limit calculations that mark the extreme point physics allows.

A search the universe cannot afford

In 2002 Lloyd went a step further and counted the operations the observable universe could have performed since its beginning: about 10¹²⁰. The information that ordinary matter can record comes to about 10⁹⁰ bits.

These numbers are staggeringly large, yet they are not enough for a single search. Trying every combination of just a thousand yes-or-no choices means 2¹⁰⁰⁰, about 1.07 × 10³⁰¹ possibilities. That is 10¹⁸¹ times more than the operations the universe has performed in its entire history.

The advantage of the most powerful intelligence cannot be to try everything one by one. It lies in finding out what is not worth trying.

Does a self-improving machine go on forever?

Behind the idea of an "intelligence explosion" lies a simple model: if a system improves itself by a fixed rate at every step, its growth is exponential and keeps accelerating.

The review argues that this model is not physically realistic in the long run. Energy, communication, manufacturing capacity, data and algorithmic saturation eventually brake growth. A better model is logistic growth: a curve that looks exponential at first and then slows as it approaches the ceiling set by a given regime of resources.

There is one more detail: improving oneself does not always mean getting better. A new architecture can bring bugs, security holes or a drift in values. A realistic self-improvement mechanism has to include verification, and the ability to return to an earlier version when needed, as much as performance.

The real question: what is it optimising?

An agent being "autonomous" does not mean it has free will in the philosophical sense. It can choose from a wide set of options according to its own model and goal, yet that choice is still a product of its algorithm, its memory and its observations.

The crucial point is the goal. The more optimising power grows, the more it matters what is being optimised. The review gives a simple example: human welfare, knowledge gained, system safety and respect for human autonomy weighed positively, risk weighed negatively. But choosing those weights is not a mathematical task; it is a question of ethics and governance.

That is why, in the superintelligence debate, the question "how intelligent is it?" should not crowd out the question "what is it optimising?"

Four ceilings

All of this shows that "maximum intelligence" cannot be expressed as a single number. There are at least four separate ceilings:

1. The computability ceiling: some problems have no general algorithmic solution.
2. The information ceiling: certainty cannot be produced about an unobserved state; a finite system carries finite information.
3. The physical ceiling: energy, entropy, the speed of light and quantum dynamics limit computation.
4. The algorithmic ceiling: more raw operations cannot always make up for a poor representation or a poor algorithm.

Within this frame, the review suggests thinking of intelligence as an efficiency: verifiable, useful results obtained per unit of energy, time and computation spent. This is not a standard scientific measure but a way of thinking. Still, it captures something true: an advanced system's edge may lie not in doing more operations but in discovering which operations are unnecessary.

Not a god machine, but a finite agent

The idea of superintelligence is not self-contradictory. But it makes sense not as an all-knowing machine standing outside physics; rather as a finite information processor working with extraordinary efficiency inside the physical universe. An agent that works with well-calibrated uncertainty instead of absolute certainty, that reduces its errors systematically instead of pretending to eliminate them, and that takes the limits of energy, heat, communication and information into account as it grows.

Never making a mistake is not a defensible goal. What is defensible is making errors very small and correction very fast, and doing so safely within the resource limits that physics sets.

Selim Kusursuz

Founder and writer 5 pieces

Studies Computer Network Systems at Paris-Saclay, on the Autonomous Systems track. Works on mathematics and computer science. Speaks French, English and Turkish well.

See all their articles