A hundred years ago, in 1917, Albert Einstein had a problem. He had just come up with a beautiful new theory of gravity called general relativity. But the theory predicted that the universe should either expand or contract. Even for Einstein, this was a bridge too far. The universe, though it might have had a beginning, certainly did not appear to be changing.
General relativity is a rigid theory; little of it can be changed without destroying its elegant mathematical structure. The only wiggle room was a single quantity, which Einstein called the cosmological constant. If he assumed this quantity was zero, the equations looked particularly simple, but they required the universe to be dynamical.
The gravity of ordinary matter (such as galaxies) is attractive. Introducing a positive cosmological constant adds a repulsive counterforce. By setting the cosmological constant to a particular nonzero value at which the two tendencies cancel each other, Einstein thought he could get his theory to spit out a static, unchanging universe.
Einstein later called this idea his “biggest blunder,” an accurate assessment. First, his move doesn’t actually accomplish the task: Einstein’s static universe is unstable, like a pencil balanced on its tip. Because matter is distributed unevenly, the opposing forces couldn’t possibly be arranged to balance out everywhere. Individual regions would soon begin to expand or contract. Worse, Einstein missed out on a spectacular prediction. Had he believed his own equations, he could have anticipated the 1929 discovery that galaxies are in fact receding from one another.
This dramatic turn of events proved that the universe was not static. But an expanding universe didn’t imply that the cosmological constant was necessarily zero! As quantum field theory triumphed in the second half of the 20th century as a description of elementary particles, physicists recognized that the cosmological constant “wants to be there.” The very theories predicting with unprecedented accuracy the behavior of small particles also implied that empty space should have some weight, or “vacuum energy.” This kind of energy happens to be indistinguishable from a cosmological constant, as far as the equations of general relativity are concerned.
So it was no longer an option to set the cosmological constant to zero. Rather, it had to be calculated. Estimates indicated an enormous value—save for some unlikely precise cancellation between large positive and negative contributions from different particles. But a huge cosmological constant would show up as a repulsive force that would blow up the entire universe in a split second (or, alternatively, would cause it to collapse instantly, if the constant happened to come out negative). Evidently this was not what the universe was doing.
This drastic conflict between theory and observation is the “cosmological constant problem.” It remains the most serious problem in theoretical physics. It has contributed to the development of revolutionary ideas, particularly the multiverse and the “landscape” of string theory.
The string landscape solves the cosmological constant problem by a strategy similar to throwing many darts randomly: Some will hit the bullseye by accident. In string theory there are many different ways of making empty space, and they would all get realized as vast regions in different parts of the universe. In some regions, the universe “hits the bullseye”—the cosmological constant is accidentally small. There, spacetime does not quickly explode or collapse, so structure and observers are more likely to evolve in these lucky regions.
A crucial prediction of this approach was pointed out by Steven Weinberg in 1987: If an accidental near-cancellation is the reason the cosmological constant is so small, then there’s no particular reason for it to be exactly zero. Rather, it should be large enough to have a just-noticeable effect.
In 1998, astronomers did find such an effect. By observing distant supernovae, we can tell that the universe is not just expanding but accelerating (expanding ever more rapidly). Other observations, such as the history of galaxy formation and the present rate of expansion, have since provided independent evidence for the same conclusion. Empty space is filled with vacuum energy. In other words, there’s a positive cosmological constant of a particular value, which we have now measured.
The cause of the acceleration is sometimes described more dramatically as a “mysterious dark energy.” But in science we shouldn’t embrace mystery where there is none. If it walks like a duck and quacks like a duck, we call it a duck. In this case, it accelerates the expansion and affects galaxy formation precisely like a cosmological constant, so we should call it by its name.
One of the most fascinating consequences of a positive cosmological constant is that we’ll never see much more than the present visible universe. In fact, billions of years from today the most distant galaxies will begin to disappear, accelerated out of sight too far for light from them to reach us. Eventually our local group of galaxies will hover alone in a vast emptiness filled with nothing but vacuum energy.