Are physicists just making up dark energy?S

In this week's "Ask a Physicist," I get defensive about dark energy. It dominates the universe; it's completely absurd; and it's apparently absolutely necessary.

I've been putting off talking about dark energy for a while. I was afraid that you'd judge me, and all cosmologists, as being charlatans. But I can't put it off any longer, so for this week's "Ask a Physicist" our (metaphorical) shame will be revealed by Donovan, who asks:

I've never seen someone try to explain dark energy as anything other than a feature of our universe. Instead of explaining dark energy as a mysterious force inside of our universe that is pushing out and inflating us, has anyone ever tried to explain it by postulating some kind of "vacuum" outside of the universe that is "pulling" us out?

You see? This is why I'm so defensive. The idea of dark energy is so ridiculous that almost every question is based on trying to make it go away. And believe me, I share your concerns. I don't want to believe in dark energy, but I have no choice.

When I was a student in the mid 1990's, cosmologists thought we had it all figured out. Dark matter had been known (or at least suspected) since the 1920's, and most respectable physicists simply assumed that the universe was made up of a combination of ordinary and dark matter. Since gravity is attractive, this should presumably act to pull the universe together.

At the same time, several groups were observing distant supernova explosions. Supernovas (Type Ia's, if you must be precise) are really useful probes of the universe because a) they are very bright, which means that you can see them from very far away, and b) they are "standard candles" which means that if you understand them well enough, you can figure out exactly how bright they really are, and from that, you can determine their distance. As I discussed previously, we can measure the redshifts of the supernovas to tell how much the universe has expanded since they blew up. From the combination of expanding universe and distance — voila! We can determine how quickly the universe is slowing down.

Only it isn't.

Are physicists just making up dark energy?S

In 1998, the High-z Supernova Search Team followed quickly by the Supernova Cosmology Project announced that based on their observations, the universe is, in fact, accelerating. Subsequent observations have confirmed this, and the culprit has been dubbed "Dark Energy." Like "Dark Matter," the name is meant to obscure the fact that we have no frakking clue what it really is.

Dark energy, as you almost certainly know if you're an avid pop-sci reader, is a "mysterious substance" (it's always called mysterious) which causes the universe to accelerate. This is not as ridiculous as it would seem at first blush, since when it comes to gravity, you've probably been lied to. You probably already know that mass creates a gravitational field, but general relativity shows that any form of energy (including a big box of photons) will do the trick. Stranger still is that gravity gets an extra power-up if there is pressure involved. Under normal circumstances, we don't notice this, since even in the center of the sun, the pressure is tiny compared to the energy density.

Dark energy is a weird case. The idea is that the pressure is negative — kind of like elastic — which means that the net gravity is repulsive. This being io9, I'd be remiss if I didn't point out that dark energy is the closest thing that we have to anti-gravity. It's not anti-gravity, mind you, but if you have your heart set on writing it into your story, it's the best you're going to do.

And there's a lot of it; our current best estimate is that dark energy accounts for about 70% of the total energy in the universe. Take a moment for that to sink in. For all of you who would give me grief about dark matter simply because we've never captured a particle of the stuff, consider the fact that at least dark matter has the good grace to behave something like everyday particles. With dark energy, we have three times as much, and even our best models don't involve particles that we might see in a detector. But that doesn't mean that we've never seen anything like dark energy.

Quantum Electrodynamics is, besides a potentially awesome name for a band, one of the most successful theories ever. It basically unifies electromagnetism and quantum mechanics, and has predicted everything from the detailed structure of the atom to the magnetic strength of the electron to fantastic precision. It also has a well-earned bad reputation for producing lots of infinities in calculations. This is bad, by and large, but we can normally get around it by subtracting one infinity from another. Yes, it's a cheat. Yes, it makes me feel dirty. But it also works, and I guarantee that you're not going to make me feel any worse about myself than my quantum field theory students did when I distracted them with puppies and quickly just erased the infinities.

One of the infinities that pop out of this theory is related to the particles and antiparticles that are constantly being created out of "the vacuum." There are two cool things and one really crappy thing about the "vacuum energy" of these temporary particles. The first cool thing is that it has exactly the negative pressure needed to make dark energy. The second cool thing is that it isn't just made up. We can observe the "Casimir Effect" in a lab. Two metal plates in a vacuum will be pulled together because there are more vacuum fluctuations outside the plates than between them.

The really crappy thing, however, is that any realistic calculation of the vacuum energy gives an energy density about 10100 times larger than the density actually measured by the accelerations of supernovas (even if you round down the "infinity" a bit). This is not a small problem. As far as I am concerned, it is the worst problem in physics, and one of the reasons that Donovan and other readers have posed questions trying to get around the whole thing.

One possibility is to suggest that the universe isn't accelerating at all. Perhaps the most distant supernovas are just different from the ones near by, and that effect oddly makes the universe look like it's accelerating? I had this thought myself, at first, but the supernova evidence isn't the only reason to be a believer.

Are physicists just making up dark energy?

One of the best pieces of evidence that we have is the measurement of the Cosmic Microwave Background, the radiation from when the universe was only about 380,000 years old. We can observe the bumps and wiggles of hot and cold which behaved like water waves do today. Since the light from these peaks had to travel for almost the entire history of the universe before reaching our telescopes, we can use this information to figure out the shape of the universe to incredible accuracy. It's "flat," by the way. What this means for us is that since we know that dark and ordinary matter combined only add up to about 30% of the energy needed to make our universe flat, the rest must be made up by something else: dark energy. But that's not all. From the distribution of galaxies, to gravitational lensing, to the number of distant galaxy clusters, every piece of evidence points to a model with this dark energy feature in it.

Basically, if you want to get rid of dark energy, you have to get rid of relativity. You're welcome to try, but for my money, it seems like a better bet to try to figure out what dark energy is.

The problem is that we honestly don't know. One possibility is that Einstein's "cosmological constant" was right all along, and for some reason, it's simply hard-wired into our theory of gravity. Another is that the dark energy is a fluid, with a pressure that might change from place to place, and time to time. So far, there's no evidence that this is the case, but the errorbars are still pretty big.

But if the vacuum energy is so large, why is dark energy so small? We just don't know, which is why I got all defensive early on. One possibility (and this is a possibility likely to piss off many physicists) is that it's simply the anthropic principle at work. Maybe all universes have more or less dark energy, but the ones with more than us accelerated so quickly that no stars or complex structures could form. This would mean, of course, no stars, which means no human beings, or ALF for that matter, to be having this conversation.

So back to Donovan's original question: Couldn't the acceleration of the universe be real, but basically be a pull form the outside? The problem is that the universe doesn't have an outside. Wherever you go, there you are. But I'll play along. Let's imagine that our universe does have a crispy crust, and some cosmic pizza-maker is pulling it from all sides. What happens then? The edges may be accelerating away from you, but the nearby mushrooms would more or less sit still. That's because gravity is a local curvature in space-time, and it takes something — let's call the dark energy yeast — to push all the mushrooms away. Hopefully the whole pizza digression will distract you from the fact that while physicists are pretty damn sure that something is playing the role of dark energy, we have no idea what it is.

And now you know our shameful, shameful secret.

Dave Goldberg is the author, with Jeff Blomquist, of "A User's Guide to the Universe: Surviving the Perils of Black Holes, Time Paradoxes, and Quantum Uncertainty." (follow us on facebook or twitter.) He is an Associate Professor of Physics at Drexel University. Feel free to send email to askaphysicist@io9.com with any questions about the universe.