On a recent visit to my mum’s place, I searched through my old stuff for something my children might like. One book that caught my eyes was (the German edition of) James Trefil’s Dark Side of the Universe. It’s about cosmology, the Big Bang and the expansion of the universe, Einstein’s theory of general relativity, and speculations about what dark matter might be.

Trefil’s book was published in 1989. Back then, I found it tremendously exciting. But much of it could be published in 2021 without change – we’d just have to add that the cosmological constant is back. Then again, the cosmological constant was Einstein’s idea, so it’s not exactly new.

For much of the past century, scientific advances led to technological progress that furthered science, which in return led to more technological progress, and so on. It was a virtuous cycle that rapidly raised our standard of living. But in the foundations of physics, this virtuous cycle broke in the mid 1980s. Since then, we have been in a phase of stagnation.

This stagnation has befallen not only cosmology but also the rest of the foundations of physics: quantum gravity, particle physics, and quantum foundations. You have certainly noticed this yourself: popular science articles that cover these areas just regurgitate the same topics.

Nowadays, headlines covering the foundations of physics won’t tell you about new discoveries, but merely what “might be” or “could be”.

We’ve known of dark matter since the 1930s, when Fritz Zwicky’s study of galactic clusters led him to infer the existence of a substance he called dunkle Materie. But we still don’t know what it is made of: in fact, we don’t know whether it’s made of anything – it could just be we use the wrong theory for gravity. We’ve known similarly long that general relativity is incompatible with quantum theory, yet we still don’t know how to combine them. We’ve debated the beginning of the universe for even longer.

It’s not like physicists are short on ideas for solving those problems. It’s just that so far all their ideas were falsified: theories of grand unification, supersymmetric particles, extra dimensions, various fifth forces, and countless dark matter particles have been ruled out. The only ideas that survived are those that are unfalsifiable to begin with because they can be arbitrarily amended when conflicting evidence arises.

Nowadays, headlines covering the foundations of physics won’t tell you about new discoveries, but merely what “might be” or “could be”. The phrase “physicists say” is all too frequently followed by speculations about multiverses, non-existent particles, or fifth forces that we have no evidence of. Sometimes I’m embarrassed to be associated with this discipline.

But the worst part is that most of my colleagues think this situation perfectly okay.

For starters, they would probably disagree that we have a problem in the foundations of physics at all. They’d tell you about lots of exciting papers that have been published in recent years. At present the biggest fad is throwing artificial intelligence at everything, closely followed by claiming that quantum simulations or quantum computing is the way forward. About this I can only say that scientific progress isn’t measured by how many papers have been written.

But this illusion of progress is the minor problem. Worse is that they seem resigned to the idea that foundational work in physics is detached from experiment and technological application. Quantum mechanics is the theory on which all modern electronics build – from semi-conductors, to LEDs, lasers, and digital cameras, all the way to quantum computers. If this theory was revised, it would almost certainly impact technological development, and far more so than some new particle or dark matter possibly could.

Those working in the foundations of physics believe that whatever is left to find, it’ll be irrelevant to everyday life. I think they’re wrong – but I’m afraid their belief might become a self-fulfilling prophecy. That’s why I want to tell you why I think they’re wrong.

How we got here

To understand what happened in the foundations of physics, it’s worth a quick look at the history of the field. Physics is one of the oldest disciplines of science, acknowledged by the Greek philosopher Aristotle in the fourth century BC. For almost two millennia, physics and philosophy remained closely intertwined. But after the scientific revolution of the 16^th and 17^th centuries, scientific disciplines increasingly specialised, and the link between physics and natural philosophy faded away. The title PhD – “philosophiae doctor” – still bears witness to our common past.

Owing to the success of science and the growing number of researchers, physics developed sub-disciplines, of which we count today about a dozen. In most of them, theory-development is still closely linked to experimental test: for example, plasma physics, quantum information, statistical mechanics, photonics, optics and acoustics, quantum optics, condensed matter physics, nuclear physics, and much of astrophysics. These areas have remained comparably healthy. But in the foundations of physics – those areas concerned with the most fundamental laws of nature: particle physics, quantum foundations, quantum gravity, and cosmology – theory-development has decoupled from experimental test. And in the absence of reality checks, pointless speculation became accepted norm.

Let me be clear: it’s not that experiments have stopped in the foundations of physics. It’s just that – one after the other – experiments have confirmed theories we already had half a century ago and ruled out any ideas put forward after that. The confirmation that neutrinos have masses, that gravitational waves are real, and the detection of the Higgs are recent examples of remarkable experimental achievements in the foundations of physics. But the predictions of these phenomena all date back to before the 1970s.

The misgivings that philosophers had about quantum mechanics, it turned out, weren’t entirely irrelevant after all. If physicists hadn’t been so dismissive of philosophy, they might have seen that sooner.

This makes me worry it’s only a matter of time until experimental progress stalls in other areas of physics, too. That’s because for much of the history of physics, better observations led to a better understanding of natural laws, which led to better technologies, which led to better observations and so on. This virtuous cycle broke in the middle of the past century when foundational research hit the wall.

The progress we currently see in the non-foundational areas of physics is largely due to more computing power and miniaturisation well into the quantum regime – all progress driven by those foundational breakthroughs from the beginning of the 20^th century. There is still much potential in pushing this trend further, but if we don’t make new discoveries in the foundations soon, this progress will eventually stagnate, too. And that won’t only affect physics, it’ll affect all of society.

Where we are now

This sounds rather grim, and yet I am hopeful. My hope springs, oddly enough, from a conversation I recently had with the American science writer John Horgan. Twenty-five years ago, John wrote The End of Science, in which he conjectured that the rapid scientific progress of recent history is a blissful – but ultimately transient – epoch in societal development. There’s only so much to discover about natural laws, John argues, and we’re almost done. There’s nothing left to find. John is not a scientist by training. His book is a report of what he heard from scientists – interesting, well-written, and sparkling with wit – but in the end still a report.

The book didn’t make John many friends among scientists, but I can’t blame him for having arrived at this conclusion. Physicists have declared we’re close to a final “theory of everything” since the 1970s, but widely acknowledged that this success would be largely cosmetic – like the cherry topping the tart, aesthetically pleasing but not of high nutritious value. That’s because they believe the currently open questions, once answered, won’t be useful for technological applications anyway. Quantum gravity and dark matter, which have attracted most of the attention, are far too feeble phenomena to be good for every-day gadgets.

Then the COVID pandemic came and, like many of us, John began to spend too much time on YouTube. At the same time, with half of the world going into lockdown, I saw a marked increase of interest in my YouTube channel. And so, one day in May 2020, John stumbled across one of my videos: an introduction to quantum mechanics. He didn’t understand it and vowed to finally properly learn quantum mechanics. I recommended some books.

One year later, John has learned enough about quantum mechanics to complain it’s a hopelessly confused patchwork, as he told me when we recently spoke.

This sounds rather grim, and yet I am hopeful.

Quantum mechanics has been remarkably successful. But ever since its conception a century ago, quantum mechanics has given headaches to philosophers for screwing up our notion of reality. The central ingredient of quantum mechanics – the “wave-function”, which supposedly describes everything we can possibly observe – can’t be observed itself. An entire research program of “foundations of quantum mechanics”, populated mostly by philosophers, sprung up and offered various ways of interpreting quantum math to make more sense of it.

Physicists ignored the philosophers. But in the past 10 years or so, physicists themselves have put forward a flood of new arguments, theorems even, that bring the problems with quantum mechanics into focus. They are past debating whether quantum mechanics is inconsistent; instead they argue in which way it is inconsistent – and what to do about it.

The most remarkable of those recent arguments is perhaps the Frauchinger-Renner paradox, which demonstrates that quantum mechanics cannot consistently describe the use of itself. If you imagine observers observing observers, Daniela Frauchinger and Renato Renner showed that in some cases the observers cannot agree on what happened – if quantum mechanics is correct. It’s simply not fit to be a fundamental theory of nature.

Another milestone has been a no-go theorem for theories that may underlie quantum mechanics. In 2012, Matthew Pusey, Jonathan Barrett, and Terry Rudolph proved that certain completions of quantum mechanics – that is, theories from which quantum mechanics might derive – are impossible. Now, this may sound like a negative result, but no-go theorems are incredibly helpful for theory development because they narrow down possible options.

The misgivings that philosophers had about quantum mechanics, it turned out, weren’t entirely irrelevant after all. If physicists hadn’t been so dismissive of philosophy, they might have seen that sooner.

What’s next?

I believe that physicists made a big mistake in the 1980s when they banked on new physics to appear on shorter distances and built a series of increasingly larger and increasingly more expensive particle colliders. Yes, they did confirm a couple of previously predicted particles. This is nice alright, but there’s not much you can do with particles that decay in a nanosecond or less. Frankly, I think, politicians back then should have asked a little more loudly what all this particle slamming would do for society. But they let themselves be shamed into silence. This is why the virtuous cycle that linked scientific and technological progress broke. This is why the foundations of physics have reached a dead end today.

In hindsight, physicists should have focused on the problem in front of their eyes, the one they’ve seen in myriad experiments: the measurement problem of quantum mechanics.

I am more optimistic today that we will finally make progress in the foundations of physics than I was 10 years ago.

In quantum mechanics, the outcome of a measurement cannot be predicted with certainty. We can merely calculate the probability of obtaining a particular outcome. Einstein was famously unhappy about this and argued that “God does not play dice”. But the randomness in and by itself is not the problem. The problem is that a theory which describes nature on the fundamental level shouldn’t rely on vague terms like “measurement” – it should instead explain what a measurement is. That quantum mechanics can’t do this is the reason for paradoxes like that of Frauchinger-Renner, and also why quantum mechanics is decried as strange, weird, and impossible to understand.

At last, it seems the “shut up and calculate” doctrine, which has dominated quantum mechanics for half a century, is losing its grip on the community. And this is why I am more optimistic today that we will finally make progress in the foundations of physics than I was 10 years ago.

If John, the-end-of-science guy, can see that quantum mechanics isn’t the end of the story, I believe physicists will finally see this too. And I hope that by the time my children have children, the physics books of today will be in dire need of revision.

To hear more from Sabine Hossenfelder, visit her YouTube channel “Science without the gobbledygook“.