A hundred years ago, in 1917, Albert Einstein had a problem. He had just come up with a beautiful new theory of gravity, called General Relativity. But the theory predicted that the universe should either expand or contract. Even for Einstein, this was a bridge too far. The universe, though it might have had a beginning, certainly did not appear to be changing.
General Relativity is a very rigid theory: little can be changed about it without entirely destroying its elegant mathematical structure. The only wiggle room was a single quantity which Einstein called the “cosmological constant.” If he assumed that this quantity was zero, the equations looked particularly simple. But then they also required the universe to be dynamical.
The gravity of ordinary matter (such as galaxies) is attractive. Introducing a positive cosmological constant adds a repulsive counterforce. By setting the cosmological constant to a particular, nonzero value, at which the two tendencies precisely cancel each other, Einstein thought he could get his theory to spit out a static, unchanging universe.
Einstein later called this idea his “biggest blunder,” an accurate assessment. First, his move does not actually accomplish the task: Einstein's static universe is unstable, like a pencil balanced on its tip. Because matter is distributed unevenly, the opposing forces could not possibly be arranged to balance out everywhere. Individual regions would soon begin to expand or contract. Worse, Einstein missed out on a spectacular prediction. Had he believed his own equations, he could have anticipated the 1929 discovery that galaxies are, in fact, receding from one another.
This dramatic turn of events proved that the universe was not static. But of course, an expanding universe did not imply that the cosmological constant was necessarily zero! As quantum field theory triumphed as a description of elementary particles, the second half of the twentieth century led physicists to recognize that the cosmological constant “wants to be there.” The very theories that predicted the behavior of small particles with unprecedented accuracy also implied that empty space should have some weight, or “vacuum energy.” This kind of energy happens to be indistinguishable from a cosmological constant, as far as the equations of General Relativity are concerned.
So it was no longer an option to set the cosmological constant to zero. Rather, it had to be calculated. Estimates indicated an enormous value—save for some unlikely, precise cancellation between large positive and negative contributions from different particles. But a huge cosmological constant would show up as a repulsive force that would blow up the entire universe in a split second (or alternatively, would cause it to collapse instantly, if the constant happened to come out negative). Evidently, this was not what the universe was doing.
This drastic conflict between theory and observation is the “cosmological constant problem.” It remains the most serious problem in theoretical physics. It has led to important developments, particularly the “multiverse” and the landscape of string theory.
The idea is similar to throwing many darts randomly: some will hit the bullseye by accident. Similarly, in string theory there are many different ways of making empty space, and they would all get realized as vast regions in different parts of the universe. In some regions, the universe “hits the bullseye”: the cosmological constant is accidentally small. There, spacetime does not quickly explode or collapse, so structure and observers are more likely to evolve in these lucky regions.
A crucial prediction of this approach was pointed out by Steve Weinberg in 1987: if an accidental near-cancellation is the reason why the cosmological constant is so small, then there is no particular reason for it to be exactly zero. Rather, it should be large enough to have a just-noticeable effect.
In 1998, astronomers did find such an effect. By observing distant supernovae, we can tell that the universe is not just expanding, but “accelerating” (expanding ever more rapidly). Other important observations, such as the history of galaxy formation and the present rate of expansion, have since provided independent evidence for same conclusion. Empty space is filled with vacuum energy. In other words, there is a positive cosmological constant of a particular value, which we have now measured.
The cause of the acceleration is sometimes described more dramatically as a “mysterious dark energy.” But in science we should not embrace mystery where there is none. If it walks and quacks like a duck, we call it a duck. In this case, it accelerates the expansion and affects galaxy formation precisely like a cosmological constant, so we should call it by its name.
One of the most fascinating consequences of a positive cosmological constant is that we will never be able to see much more than the present visible universe. In fact, billions of years from today, other galaxies will begin to disappear, accelerated out of sight, too far for light from them to ever reach us again. Eventually, our local group of galaxies will hover alone in a vast emptiness filled with nothing but vacuum energy.