Spiral galaxy in Coma Cluster [Courtesy NASA]

What is the cosmological constant paradox?

David H. Bailey
Updated 2 January 2021 (c) 2021

The cosmological constant paradox

Perhaps the most startling "cosmic coincidence" that modern scientists have noted in the structure of our universe is the fine-tuning of the cosmological constant [Vilenkin2006, pg. 121-126]. The paradox derives from the fact that when one calculates, based on known principles of quantum mechanics, the "zero-point mass density" or the "vacuum energy density" of the universe, focusing for the time being on the electromagnetic force, one obtains the incredible result that empty space "weighs" 1093 grams per cc. Since the actual average mass density of the universe is 10-28 grams per cc [Susskind2005, pg. 70-78], this prediction is roughly 120 orders of magnitude greater than the measured value. As Stephen Hawking has quipped, this is arguably the most spectacular failure of a physical theory in history [Davies2007, pg. 147]. The "cosmological constant'" of Einstein's general relativity equations is linearly related to the zero-point mass density. Einstein originally posited a nonzero value for the cosmological constant, but after the expansion of the universe was discovered, he lamented that this was his greatest blunder and set the constant to zero [Davies2007, pg. 58].

Hoped-for explanations

Physicists, who have fretted over this paradox for decades, have noted that calculations such as the above involve only the electromagnetic force, and so perhaps when the contributions of the other known forces are included (bosons give rise to positive terms, whereas fermions give rise to negative terms), all terms will cancel out to exactly zero, as a consequence of some unknown, yet-to-be-discovered fundamental principle of physics. When "supersymmetry" was theorized in the 1970s, it was thought that it would meet this requirement, but when it was later discovered that our universe is not precisely supersymmetric, this explanation was abandoned. In any event, until recently physicists remained hopeful that some yet-to-be-discovered principle would imply that the positive and negative terms of the zero-point mass density (and thus the cosmological constant) precisely cancel out to zero.

These hopes were shattered with the 1998 discovery that the expansion of the universe is accelerating, which implies that the cosmological constant (and the zero-point mass density) must be slightly nonzero. This "dark energy," which is the unknown force accelerating the universe, also appears to be just what is needed to fill the 70% "missing mass" of the universe, namely the mass needed to explain the observed fact that space is very nearly flat (i.e., locally it appears to be almost perfectly rectilinear) [Panek2011]. But this means that physicists are left to explain the startling fact that the positive and negative contributions to the cosmological constant cancel to 120-digit accuracy, yet fail to cancel beginning at the 121-st digit. This is an even stranger paradox! Curiously, this observation is in accord with a prediction made by physicist Steven Weinberg in 1987, who argued from basic principles that the cosmological constant must be zero to within one part in roughly 10120, or else the universe either would have dispersed too fast for stars and galaxies to have formed, or else would have recollapsed upon itself long ago [Susskind2005, pg. 80-82; Weinberg1989].

The anthropic principle

Numerous "solutions" have been proposed for the cosmological constant paradox (Lewis and Barnes mention eight [Lewis2016, pg. 163-164]), but they all fail, most rather miserably.

One "solution" has been proposed by physicists working in string theory, who for 30 years have been earnestly seeking a complete and unique "theory of everything" that encompasses all known physical laws. One outgrowth of their theory is the possible existence of other universes, numbering (by one reckoning) more than 10500. This stupendous number is, for instance, incomparably larger that the total number of atoms in the universe, which is a mere 1080. The vast majority of these universes without doubt are utterly hostile to any conceivable form of long-lived information-rich structure, much less life, and thus, if they truly exist somewhere, must be completely devoid of observers. But with so many universes to choose from, so these physicists now theorize, inevitably one of them (ours) beats the 1-in-10120 odds and is life-friendly.

Needless to say, such "anthropic" reasoning constitutes a dramatic departure in philosophy from the traditional program of physics (and science in general), and it has sharply divided the physics and astronomy communities (for further discussion on the anthropic principle, see Anthropic). Edward Witten, the "father" of string theory, is still optimistic that theoretical work will narrow down the number of possible universes to one -- ours. David Gross, paraphrasing Winston Churchill, urges fellow theorists to "never, never, never give up" [Susskind2005, pg. 241]. Paul Steinhardt is in this camp and is a vehement foe of the anthropic principle: "Decades from now, I hope that physicists will be pursuing once again their dreams of a truly scientific 'final theory' and will look back at the current anthropic craze as millennial madness." [Susskind2005, pg. 353]. But others see the anthropic principle as the solution. Andre Linde, a leading theoretical physicist, says "Those who dislike anthropic principle are simply in denial." [Susskind2005, pg. 353]. Still others, such as Lee Smolin, argue that the discovery of the nonzero yet breathtakingly small cosmological constant, together with the derivation of 10500 universes when a single unique system was sought, constitutes a fatal reductio ad absurdum of the entire string theory approach to modern physics, and we may need to start anew to formulate a coherent theory [Smolin2009b].

Summary

In short, the recent discovery of the accelerating expansion of the universe and the implied slightly positive value of the cosmological constant constitutes, in the words of physicist Leonard Susskind, a "cataclysm," a "stunning reversal of fortunes" [Susskind2005, pg., 22, 154]. It is literally shaking the entire field of theoretical physics, astronomy and cosmology to its foundations. It is still too early to see how this controversy will pan out, but it is certain to be one of the most interesting debates of modern science.

For additional discussion, see Anthropic principle, Cosmic coincidences, Fine-tuned and Multiverse .

References

[See Bibliography].