Well, I hope some of you attempted yesterday’s exercise, which involved looking at a bunch of plots with simulated data, and trying to figure out in each plot
- is there a signal of the “Higgs particle” in the plot?
- what roughly is the mass of the “Higgs particle” (which I assured you lies, for each plot, in the range 122-127 GeV/c²?)
So now let’s see how well you did, and also what the implications are for the 3 GeV/c² discrepancy between the two measurements by the ATLAS experiment of the Higgs particle’s mass — a discrepancy which led to a big discussion (which ATLAS did not encourage or generate, mind you). Some speculated (and others chose to report this speculation in the news media) that there might be evidence in this data for two Higgs particles. (This is contradicted by CMS’s result from November, but hey, who’s counting?) But the substantive question you might ask is: Is this such a big discrepancy that either there are two Higgs particles or ATLAS has made a big mistake?
With that in the back of our minds, let’s take a look at the exercise I proposed yesterday, and see how things go. Note, however, that this exercise does not have direct or precise implications for ATLAS’s discrepancy. First, these plots are made in crude fashion and without simulating real Higgs particles, though the signal and the background are about the right size to match ATLAS’s current data on a Higgs decaying to two lepton/anti-lepton pairs. Second, ATLAS uses very sophisticated methods to measure the Higgs mass, much more powerful than you could employ by eye, and much more than these graphs could convey. Rather, the point here is to convey limitations of human psychology, and illustrate that humans are not naturally skilled at evaluating the statistical properties of small amounts of data.
Which Plots Have a Higgs Signal and Which Do Not?
15 of the 20 plots shown yesterday (in a larger figure that will be easier for you to read) have a Higgs signal; the others, marked with a red line in Figure 1, do not. That probably wasn’t too hard, except for the uppermost plot in the right column, because the plots without a Higgs have close to 40 events while those with a Higgs have close to 60; and the discrepancy is especially big in the range 120-130 GeV/c2 where I told you that the Higgs is to be found. Some people would have been tripped up by the peak in the bottom left plot, were it not for the fact that I told you the Higgs particle couldn’t be there.
Now on to the next question: What, roughly, is the Higgs mass in each of the 15 plots that have a signal.
Estimate the Higgs’ Mass
This is generally what trips people up the most. The human brain is an absolutely terrible statistician. It takes years of practice to unlearn what your brain wants to tell you — graduate students, and even senior theoretical physicists who haven’t stared at enough data, do an awful job with a problem like this.
For example, take Figure 2, which is one of the plots I showed yesterday. [Recall that the bins are 1 GeV/c² wide, and the bin just to the right of the 2 in “125” is the bin running from 125 to 126 GeV/c².] Your eye wants to tell you there’s a big peak at 126-127 GeV/c² and so that’s where the Higgs mass should be. But you’ve been warned that the Higgs peak is 4 GeV/c² wide at half its maximum height — so it shouldn’t just be a one-bin wide spike! Still, your brain forgets this and your eye misleads you. If you look carefully, there are rather few events to the right of the highest bin, and more to the left. The real Higgs mass is quite a bit lower than 126-127 in this case.
Now what about Figure 3? This looks like it has a double peak! It has one peak between 123 and 124, and another between 126 and 127! But I assure you this was generated by a signal that has a single peak. Your eye, yet again, is fooled. The real Higgs mass in this case lies between the two peaks.
And Figure 4? This case doesn’t look too hard; there are several events in the 125-126 bin and several in the 126-127 bin, just 2 in the 127-128 bin, and none in the 124-125 bin. So clearly the Higgs mass should be between 125 and 127, probably 126. But in fact the actual Higgs mass in this case lies in the empty bin!
The Truth is Revealed
Well, as the experts who are looking closely will already have guessed, every single one of these 15 plots was generated with a Higgs mass of 124.7 GeV/c2. Every one.
If there were 400 times as many events as in each of the plots from yesterday, here’s what you would see; three of the plots in Figure 5 show what the data looks like with a Higgs peak, and the fourth shows what the random background looks like.
I assure you there was no funny business in generating the plots from yesterday. No bias was introduced. Try it yourself if you don’t believe me! The computer program was designed to sample this flat background plus smooth peak at random. When you pick only 20 or so events from such a peak, and another 40 or so from the flat background, you’ll get something that looks quite squiggly. The peak in the resulting plot can easily lie one or even two GeV/c2 away from the true mass, as in Figure 2; there can appear to be two peaks, as in Figure 3; the bin with the true mass can actually be empty, as in Figure 4. And the probability of the plot looking weird in some way, given this small amount of data, should not be underestimated — it’s maybe one in four or five.
The Implications
What’s the point? The photon-based and lepton-based measurements of the Higgs mass at ATLAS differ by 3 GeV/c2, which is bigger than we’ve seen here. So what? The photon-based mass measurement is 126.6±0.3±0.7 — the first uncertainty number is due to random statistics, the second is called “systematic” and includes experimental defects and other problems. So it has a systematic uncertainty of nearly a GeV/c2 (which is always ignored by the press, as though uncertainties don’t matter in interpreting what you actually know). Systematic uncertainties are often not random; they may give an overall shift to all the data, moving the peak uniformly up or down. And the lepton-based measurement is 123.5±0.9+0.4-0.2 GeV/c², which has an upward systematic uncertainty of almost half a GeV/c2. So if ATLAS got unlucky and their lepton-based result got pulled down by a couple of GeV/c2 purely due to the statistical effects we’ve seen today, and if they have small systematic problems of nearly a GeV/c2 in one or both of their measurements, they could potentially get a 3 GeV/c2 discrepancy.
Is such a big discrepancy likely? No; it is certainly possible, but it is certainly not very likely.
But here you have to remember how statistics works. If instead of asking “is this particular weird phenomenon likely” you had instead asked “is it likely that somewhere, in all of the measurements that ATLAS is making about the Higgs, one or two of them would have turned out weird”, the answer is “very likely indeed!” I’m writing and you’re reading a post about the mass measurement — and so are other bloggers and Scientific American and all the rest — because currently this is the one that looks odd. It could instead have been that there was a double peak in the photon-based plot, or that the lepton-based plot was showing no peak at all (simply because the signal rate fluctuated down from 20 events to 11, which is just 2 standard deviations.) Or the measurements of the spin and parity of the Higgs could be looking strange. In that case we’d be talking about those things instead. In July people were talking about the hint that the Higgs didn’t seem to be decaying to tau leptons; well that’s over and done.
In other words, there’s a tremendous bias both among scientists and the news media. We always talk about the outliers, the things that look odd to us. We make a big deal about them, and of course, to some extent we should. But we often forget that although these outliers look unlikely,
- they aren’t generally as unlikely as they look to our brains, and
- the probability of something being an outlier, when many things are being measured, is not small.
I personally think it would be enormously helpful if scientists would remind the news media of this well-known fact, and if the news media would convey it to the public. It would help the public understand how science is really done — and explain why it is so very common that the big story that you read about on the science pages simply disappears, after its 15 minutes of fame, without leaving a trace.
The most interesting outlier is ATLAS’s hint that the Higgs decays more often than expected to two photons; on this everyone (including the Scientific American journalist) agrees. We had that hint last December, in July, and still now. However, although CMS saw something somewhat similar in July, they’ve (somewhat disturbingly) delayed making their most recent data public on this measurement. So right now there’s still no way to know if this is more than a fluke, because the ATLAS result by itself is still not statistically significant.
14 Responses
Thank you for your comment, I like your honesty and openness in this case.
But why are you referring to a Z particle as a proof against a “simple” explanation like thunderstorms? They produce very low frequency radio waves, due to gamma rays (Fermi detected them).
And rf waves have an effect on the confinement beam (http://prst-ab.aps.org/abstract/PRSTAB/v14/i9/e092802). CMS is build on wet clay, attracting lightning.
Z particles are extremely short lived. You said “.. well measured, such as the Z particle”.
We likely won’t see much of a “well measured” effect in reality, I suppose.
Sorry, meant Antonio Ereditato above. Sergio Bertolucci is the tnew director of the Italian INFN’s National Laboratories in Frasca.
Whoever took the courage – it is OK in my opinion.
Ereditato had nothing to do with anything specifically happening at CERN. He was only in charge of the OPERA experiment itself, based at Gran Sasso laboratory.
How many events are we talking here about, and within which time period?
I do not understand why there is no specific data – about exactly when these effects took place, and at which time & date precisely and how many occurrences of energy excess have been measured.
If there is any local interference, like for example low energy photons from thunderstorms (producing radio waves with low frequencies) this would have an influence on the measurements. This would also explain the difference in derived Higgs’ mass in CMS and ATLAS.
But without data about actual detections there is no way to tell if there is a flaw like the superluminal neutrinos here.
Yes, this maybe farfetched – but a loose cable would have been also – some months ago, with a 6 sigma confidence from CMS.
If someone is a scientist within an established community with a known general commitment one should be afraid to admit there is not a real proof for a Higgs here.
There is a clear surplus of energy, but an absolute proof of a Higgs boson needs more precise information.
But there is no control over this information stream from any other independent party. Since the debacle with those speedy neutrinos CERN is also really cautious in releasing too much information to the public.
They are trying hard (and quite successfully until now) to release only the information that is harmless to their cause.
With a 6 billion euros investment that is not really assuring, imho.
I would really like more openness in the delivered data.
In my personal opinion it was a good idea to inform the people about the problems with the superluminal neutrinos. It is a pity that Sergio Bertolucci, CERN’s head of research who took this brave step is no longer in command.
I’m afraid you’re somewhat confused about the facts.
I described the numbers of events and showed the data as of July back in my series http://profmattstrassler.com/articles-and-posts/the-higgs-particle/the-discovery-of-the-higgs/ . Since that time the number of events has roughly doubled.
You are confused about the fact that the “loose cable” that caused the mistaken neutrino measurements was from OPERA — which is not located at CERN, but in Gran Sasso. CMS is a completely different experiment, with a completely different organizational structure, much better suited for preventing such errors.
Also, the measurements being done to discover the Higgs and measure its mass are not so susceptible to something as simple as a loose cable or thunderstorm, because such problems would also affect similar measurements of particles that are already known and well measured, such as the Z particle. CMS and ATLAS data on the Z particle looks just fine.
Calibration methods done at ATLAS and CMS are looked at by dozens of people within those collaborations, and carefully reviewed before any results are released.
You say “If someone is a scientist within an established community with a known general commitment one should be afraid to admit there is not a real proof for a Higgs here.
There is a clear surplus of energy, but an absolute proof of a Higgs boson needs more precise information.”
This is partially incorrect. There is not a “clear surplus of energy”; there is no surplus of energy at all. There is a surplus of the number of collisions in which certain classes of processes occurs, all of them corresponding to roughly the same amount of invariant mass.
It is correct, however, that the evidence, while very strong, is still not at the point of being so overwhelming that all uncertainties are gone. And no one is afraid to say that. There is clearly something new in the data; exactly what it is, time will tell. We have a strong prejudice as to what it is; if that turns out to be wrong, that will be very exciting. And there’s not much to worry about, since the LHC has still only produced 5% or so of its total data set, and at less than its maximum energy. We will learn a great deal more over coming years.
As for transparency in the data; I do wish there were more of it, with something this important. I do think we will see a lot more in 2013, once the machine is shut down for repairs and upgrades, and the experimentalists are no longer working 29 hour days, 8 days a week.
Matt, your plots with the simulated data are a good demonstration that one should not rely on visual inspection when it comes to statistics. In general I agree with your reasoning about the statistical effects.
What still leaves me a bit puzzled is that for the real LHC data of ATLAS and CMS the error bars were given as about 1 GeV/c2 or a little bit more, while the discrepancy as a total is about 3 GeV/c2. If there is good reason to believe that all these experiments measure the same particle, then, taking all LHC experiments together, the error bar should be more like 2 or 3 GeV/c2. The surprising thing is not so much that three different experiments show slightly different data; this can be expected for such a new and complex machinery. But the problem in the public perception turns out when for each experiment a precision is claimed of about 1 GeV/c2 or even better, but the results differ notably more.
It is possible, but unlikely, that the large difference of 3 GeV/c2 is solely caused by statistical fluctuations (that should reduce or vanish if more data come in from more measurements of the same type).
I think we all agree that 3 GeV (which is a 2.7 standard deviation) is probably too big for a purely statistical fluctuation. That is, purely random chance is very unlikely to generate something that big.
However, systematic shifts are not random. If there happens to be a technical problem that shifts the mass for a certain class of events down by 0.5 GeV/c2, then that’s not a random fluctuation; it affects all of the events in that class in the same way.
So what we are probably dealing with is a combination of the two; a systematic shift from an experimental technicality (which is not random, and therefore doesn’t cost anything in probability — it is not a question of whether it is unlikely, it is either there or it is not) combined with a statistical fluke that happens, by random chance, to have gone in the same direction.
Again, the combination of these two things is not likely, but it’s probably not as unlikely as humans tend to think it is.
I cannot count the number of times that I have seen effects disappear or be explained away that naively were at 3 standard deviations from expectations.
If I was looking at an x ray diff graph – a single spike like that would indicate the presence of a particular type of atom or metal oxide etc. But not in this case…? So what does the single spike in the bottom left actually determine? ( system noise ). And the so called Higgs only apparent when the returns are formed into a ‘group of peaks’?
No, it wouldn’t indicate that… not if there were only 5 events over a background of one or two per bin. It’s purely random.
Also, in this particular case (as shown above) the sought signal is known to be a wide peak, not a narrow one, relative to the bin size.
As I said above — the brain is a terrible statistician.
i don’t have the background or been taught how to read your data graphs so i don’t know what the rules are…Other than looking for a wide peak! as i you have just indicated so i suppose i have learned something. And i am guessing the width then is relevant to the Gev result?
The bottom left has a pronounced peak – is this eliminated as it is a single spike?
It’s simply random. That kind of thing happens! Try it yourself! If you run a random number generator 40 times, making numbers between 0 and 1, and you make a histogram of the result, you will see strange things occasionally. And remember, your eye is drawn to the strangest thing on the plot.
thanks Matt – what do you mean by strange? – you mean strange patterns = patterns with a pattern?