Has the Light From Behind a Black Hole Been Seen? Does the Claim Ring True?

Back in 2019, the Event Horizon Telescope (EHT) made history as its scientists used it to create an image of a huge black hole — or rather, of the “accretion disk” of material surrounding a black hole — at the center of the galaxy M87. The dark central gap reveals where the disk’s material vanishes from view, as it presumably flows toward and disappears into the black hole.  

What the image actually shows is a bit complicated, because there is not only “light” (actually, radio waves, an invisible form of light, which is what EHT measures) from the disk that travels directly to us but also (see the Figure below) light that travels around the back of the black hole.  That light ends up focused into a sharp ring, an indirect image of the accretion disk.  (This is an oversimplication, as there are additional rings, dimmer and close together, from light that goes round the black hole multiple times. But it will be a decade before we can hope to image anything other than the first ring.)

Regrettably, that striking bright and narrow “photon ring” can’t be seen in the EHT image, because EHT, despite its extraordinary capabilities, doesn’t yet have good enough focus for that purpose.  Instead, the narrow ring is completely blurred out, and drowned in the direct image of the light from the wider but overall brighter accretion disk. (I should note that EHT originally seemed to claim the image did show the photon ring, but backed off after a controversy.) All that can be observed in the EHT image at the top of this post is a broad, uneven disk with a hole in it.

The news this week is that a group within EHT is claiming that they can actually detect the photon ring, using new and fancy statistical techniques developed over a year ago.  This has gotten a lot of press, and if it’s true, it’s quite remarkable. 

However, having looked at the paper, I’m skeptical of this claim, at least so far.  Here’s why.

1. Normally, if you claim to have detected something for the first time, you make it clear to what extent you’ve ruled out the possibility it actually isn’t there… i.e., if there’s only a 0.01% chance that it’s absent, that’s a strong argument that it’s present. I don’t see this level of clarity in the paper.
2. Almost everyone is pretty darn sure that in reality the photon ring is actually present. That introduces a potential bias when you search for it; at least unconsciously, you’re not weighing the present vs. absent options equally. For this reason, it’s important to demonstrate that you’ve eliminated that bias. I don’t see that the authors have done this.
3. Simulations of black hole surroundings and theoretical estimates both suggest that the photon ring should have significantly less overall brightness than the broad accretion disk. However, the ring measured in this paper has the majority of the total light (60%). The authors explain this by saying this is typical of their method: it combines some of the disk light near the photon ring (i.e., background) with the actual photon ring (i.e. signal). But normally one doesn’t claim to have detected a signal until one has measured and effectively subtracted the background. Without doing so, how can we be sure that the ring that the authors claim to have measured isn’t entirely background, or estimate how statistically significant is their claim of detection?

I’ve included more details on the following section, but the bottom line is that I’d like a lot more information before I’d believe the photon ring’s really been detected.

Reasons for Possible Concern

Let me be very clear: I am no expert on image processing, and cannot directly evaluate the techniques used by these scientists in this effort.  Even if the tools employed by the researchers (which are not yet public) were made available to me, I myself could not use them.  I am also not an expert in advanced statistical analysis.

Nevertheless, I am reasonably educated in methods of consistency tests and basic statistics, and have seen many wrong claims appear and disappear over the years.  Applying basic reasoning to the paper, I find myself unconvinced, at least so far.

A) If you want to claim that you have observed — “detected” — something, one way to do it, and the clearest, in my view, is to

1. assume the null hypothesis: i.e., that’s there’s no photon ring at all, and then
2. give the statistical evidence showing that the null hypothesis is wrong.

In other words, “we detect a photon ring” means “we show the photon ring is very unlikely to be absent.” It’s not merely that “we show the photon ring is more likely present than absent.” Since I see no statistical argument about this in the paper, I don’t know if “photon ring absence” is ruled out convincingly (5 standard deviations), moderately (3 standard deviations) or merely suggestively (1.5 standard deviations or so.) I’d never accept a result from the Large Hadron Collider that claimed a discovery without such a statistical statement, and the lack of one here is worrying.  [More on this in section (C) below.]

B) Granted, the photon ring is almost certainly present. It is almost unavoidable if Einstein’s gravity (general relativity) and our basic understanding of a black hole’s surroundings are correct.  But that raises an issue. When you’re looking for something that everyone’s confident should be there, there’s a serious risk of bias: you don’t worry as much as other experimenters/observers that you might be making an error, and so you tend not to be as careful.  This is especially true when you’re using new methods that aren’t made public; after all, how can someone show you’re wrong? and besides, even if your methods are wrong, the photon ring’s probably there anyway, so how wrong can you be? That lack of fear can cause over-optimism and overreach even among first-rate scientists.

Because of this, the bar for discovery claims of this type should be higher than usual (as they were for the Higgs boson). In particular, it behooves the researchers to prove that, in simulated data with and without a photon ring, the method they use always gives the right answer, never finding a ring that isn’t there, and always finding a ring that is there. (More precisely, one would want a study of the contexts in which dim photon rings become undetectable.)  In the paper where they presented their methodology, they only showed that the method could find photon rings when they are present; they did not show that the method never finds fake rings in data where they are absent.

C) For various reasons, one expects the total light coming from the photon ring to be a rather small fraction of the total light in the image.  Apparently (so says the new paper) simulations done by the EHT team as part of their 2019 work suggest the photon ring should be emitting 10%-30% of the total light. (This is actually larger than I would expect based on this theoretical paper, which suggests a smaller number closer to 5-15%; I don’t yet understand the discrepancy.) However, as one can see from Table 2 of the new paper, the amount of light that the researcher’s method associates to the M87 black hole’s photon ring comes out to be about 60% of the total light, far too high. 

Why is this? Well, the authors say, that’s just the way the method works; and they refer back to their methodology paper, where the same issue is observed in that paper’s Table 2. The method finds the photon ring in the right place, but it’s always significantly too bright. No explanation for this is given there, but in the recent paper, without giving evidence, the authors write that their method scoops up some of the light from the accretion disk (background!) and adds it to the photon ring (signal!). In other words, only a fraction of the thin ring of light that the method identifies actually comes from the photon ring signal — typically less than half of it.

Here’s a quote from this week’s paper:

• We find that the ring component contains between ∼54%–64% of the total flux in the image. This range exceeds the fraction of flux contained in the narrow ringlike features in [earlier EHT] simulations, from which we anticipate only ∼10%–30% of the total image flux to be contained in the narrow ring. The measured fluxes are, however, consistent with the results from [the methodology paper] for hybrid image reconstructions of simulated data. The excess ring flux appears to be a consequence of the absorption of a portion of the surrounding direct emission into the ring component.

So… wait a minute. Aren’t the authors saying, then, that their method is particularly good at identifying the signal because it cleverly adds some of the background to the signal?

That sounds a little fishy to me. And it leaves me with questions:

• How do we know the method can’t add a ring of background to a signal with ZERO flux? Where’s the evidence that this cannot happen? How do we know this didn’t happen in the data?
• The uncertainties on the measured flux numbers in the measured ring are small, see their Table 2. But how large is the uncertainty on the actual photon ring signal, after the background is removed? How many standard deviations away from zero is it?
• More generally, how can one claim to have detected a signal before determining and subtracting all known backgrounds from the overall measurement?

I don’t know how to answer these questions based on what the papers say.

I find myself imagining how I would have reacted if, back in late 2011, someone had come to me with a claim to have discovered the Higgs boson, and the argument went like this:

• We are confident, from theory, that the Higgs particle is almost certainly in the data, and that it should appear as a sharp bump over a smooth background;
• So let’s use that knowledge to try to look for that bump more efficiently;
• Here’s a method that’s very sophisticated (beyond even what most experts can understand) that’s really good at finding sharp, super-narrow bumps;
• It has the feature that when we test it on simulated data that has a Higgs particle bump in it, it always finds the bump, identifying it as 2 to 3 times larger than it actually is;
• The reason is that it sucks some of the background into the bump;
• Applying this to the real data, we do indeed find a bump; it appears 3 to 4 times larger than we think it truly is, but that’s just a feature of the method;
• On this basis, we claim we’ve discovered the Higgs particle.

I think I would have been pretty unconvinced by that, even if, in the end, they were lucky and their claimed detection turned out to be in the right place. I’d have argued that the authors of the method should do a double-blind test, where other experts submit simulated data with and without Higgs particle signals in it, to see whether the method always finds true signals, and never creates fake ones where none is truly present. I would not be satisfied relying on the authors’ assurances.

In the present context, I hope the authors of this new paper would accept such a challenge, to demonstrate their confidence in their method. (In fact, perhaps they already did this study internally, though I’m not sure why they didn’t tell the rest of it about it.)

Final Thoughts (For Now)

First: A sociological remark.  Although the signatories on the paper are EHT members, the number of signatories is far smaller than on the 2019 papers that presented the M87 black hole image. This could be for various reasons. For instance, many EHT members may feel they didn’t participate in this effort and shouldn’t take credit for the result. Alternatively, many may lack confidence in its results and don’t want their names to be associated with it. There could be a combination of the two. I don’t have any way to know; I’ve seen both happen. But here’s something to watch: as this paper is critiqued and criticized (as all claims of an important discovery inevitably and rightly are), will EHT experts who didn’t sign the paper publicly defend its results? Or will they remain silent?

Second: So far I have not spoken to any signatories on the paper. I expect they will rebut my arguments point by point, and I hope I’ll have the opportunity to understand and perhaps present their perspective soon.  It may turn out that my concerns are unwarranted, and that their claims will hold up over time. 

But for now, I’ll remain skeptical and open-minded as this paper undergoes some intense scientific scrutiny.