This one (sticky) blogpost will serve for accumulating everything related to the statistical analysis of the OPERA experiment (that found faster-than-light neutrinos).
PS: There is a rumor that OPERA has repeated its experiment with very short pulses of protons and the faster-than-light result persists. If the rumor is true, then everything under the fold is irrelevant to the issue of whether OPERA is right or wrong.
PPS: OPERA confirms its result! See Tommaso Dorigo for a discussion, waiting for the pre-print to show up.
PPPS: the updated pre-print is here. Almost all the objections that had been raised about the original preprint have been addressed. If there is an error, it is inaccessible to the remote observer. The way forward is to repeat the experiment elsewhere.
Unstick-ing this post.
PPPPS: Further comments from Tommaso Dorigo.
{Last update: Nov 18, 1:07 PM}
A lot of commentators on the OPERA experiment explain the faster-than-light neutrinos in terms of exotic new physics. Another set of commentators seek an explanation in neglected relativistic effects arising from the earth's motion, the gravitational field, in the synchronization of clocks using satellites, and so on. These set of authors I completely ignore.
The set of commentators of interest to me, whose papers I've tried to gather in the links below, accept the veracity of the OPERA data, but challenge its statistical interpretation. If I was a physicist, I'd belong to this school of thought.
Notes:
External links that have a bearing on the statistical analysis:
PS: There is a rumor that OPERA has repeated its experiment with very short pulses of protons and the faster-than-light result persists. If the rumor is true, then everything under the fold is irrelevant to the issue of whether OPERA is right or wrong.
PPS: OPERA confirms its result! See Tommaso Dorigo for a discussion, waiting for the pre-print to show up.
PPPS: the updated pre-print is here. Almost all the objections that had been raised about the original preprint have been addressed. If there is an error, it is inaccessible to the remote observer. The way forward is to repeat the experiment elsewhere.
Unstick-ing this post.
PPPPS: Further comments from Tommaso Dorigo.
{Last update: Nov 18, 1:07 PM}
A lot of commentators on the OPERA experiment explain the faster-than-light neutrinos in terms of exotic new physics. Another set of commentators seek an explanation in neglected relativistic effects arising from the earth's motion, the gravitational field, in the synchronization of clocks using satellites, and so on. These set of authors I completely ignore.
The set of commentators of interest to me, whose papers I've tried to gather in the links below, accept the veracity of the OPERA data, but challenge its statistical interpretation. If I was a physicist, I'd belong to this school of thought.
Notes:
- The OPERA experiment simply assumes that the neutrino detection rate is proportional to a time-delayed version of the proton rate. Several authors have pointed out that even a small effect at the leading and trailing edges of the pulse can eliminate the superluminal effect. Various proposals are made. For instance, when turning up and turning down the proton current, the beam geometry varies slightly from the steady state configuration of the beam. The graphite target that the proton beam hits has a lot of energy deposited and heats up considerably, possibly changing its density, and hence the collision rate. And so on. Various statistical tests of the fundamental assumption that OPERA makes are proposed, but I think without the actual data from the experiment, can be nothing but inconclusive.
External links that have a bearing on the statistical analysis:
- arxiv:1109.4897, Measurement of the neutrino velocity with the OPERA detector in the CNGS beam,The OPERA Collaboration
- Why OPERA’s claim for faster-than-light neutrinos is not wrong, John P. Costella.
- arXiv:1109.5727, A possible statistical mechanism of anomalous neutrino velocity in OPERA experiment?, Robert Alicki
- arXiv:1110.0239, A simple explanation of OPERA results without strange physics, Gilles Henri
- arXiv:1110.0595, Is there a neutrino speed anomaly?, Juergen Knobloch (also arXiv:1111.3284)
- arXiv:1110.3783, A potential issue for the OPERA neutrino velocity measurement, Antonio Palazzo
- arXiv:1110.4781, Narrowing of the neutrino light curve in the OPERA experiment, Maurice H.P.M. van Putten
- arXiv:1110.4805, Some light on "Measurement of the neutrino velocity with the OPERA detector in the CNGS beam", I. Area, X. Prado
- arXiv:1110.5275, About Statistical Questions Involved in the Data Analysis of the OPERA Experiment, H. Bergeron
- arXiv:1110.6291, Comment on OPERA neutrino velocity measurement, Pierluigi Frabetti, Leonid Chernenko (also arXiv:1111.3116)
- arxiv:1110.6408, Possible Origin Of The Neutrino Speed Anomaly Reported By OPERA, Shlomo Dado, Arnon Dar
- arxiv:1111.0282, Statistical model uncertainty and OPERA-like time-of-flight measurements,Oliver Riordan, Alex Selby
My stuff:
Nov 6: Workbooks 3 & 4. These suggest that an excess of neutrinos at the edges of w[t] may invalidate the OPERA results; a deficit is not so important.
Nov 5: A simulation in Mathematica.
Nov 4: Workbook 1, with fewer events. The variance increases more than expected.
Nov 4: Workbook 2: A second idealized example.
Nov 3: (corrected, expanded) Workbook 1: The numbers from a simple idealized example (PDF)
Nov 3: A simple model (PDF)
Logging you in...
jpd · 700 weeks ago
jpd · 700 weeks ago
Bert Morrien · 699 weeks ago
I'm not a physicist but a retired techician who, after scrutinising the OPERA report, spent the last 1.5 month on debunking OPERA's analysis and designing a correct one.
In fact, I designed two of them, but one has superior properties.
I am sure the error is in ths summing of the PEWs. This effectively removes the timing info in the PEWs ad replaces it with noise. The MLA is not matching two event distributions, but it is matching noise with the neutrino detection distribution.
The error is fundamental. PEWs without a corresponding event were removed, because these would invalidate the analysis.
Constituents of the PEW without a corresponding event must also be removed; this was not done and these were made part of the PDF data. In fact, it is easly shown that the shape of the PDF is virtually entirely caused by the invalid PEW parts alone.
The MLA cannot cope with that and it's result cannot be trusted.
About my design:
Summing the PEWs is OK, but first the PEWs must be time-shifted, so that the events are aligned.
After time shifting, the the relevant timing informations in the PEWs are aligned too.
It can be proved that these have an average value that is twice the average value of the remainder of the PEW.
Scanning the sumed PEWs will show a single maximum at the start time. The aligned events form the stop time.
Since the noise is mostly ignored at the correct start time, and because the noise in the timing information of the PEW is surprisingly low, a useful result can be expected with less events.
This analysis method is not sensitive to efficiency variations in the neutrino production, hence PEWs from first and second extractions can be used as they are.
It is also not sensitive to uneven event spreading, because the evants are aligned first.
Finally, it is not sensitive to the precise placement of the window in which the PEW is measured.
This is important, because the current analysis is very sensitive to the exact position of this window.
In fig.4 of the OPERA paper, the green-blue arrowed line seems to indicate a window on the PEW; the leading edge falls outside this window.
If this was realy true, events caused by neutrinos produced in the leading edge would shift the real start time toward this edge, leading to a measured start time that is later, resulting in a sorter TOF.
Currently 2 articles are prepared, one about debunking the OPERA analysis and the other about the new analysis method.
Both these articles are mainly qualitative reasonings and need qualitative support.
I would like these articles to be reviewed so that they can be published on arXiv.
I gave up hope forwarding my ideas to the appropriate authorities of CERN or OPERA.
Bert Morrien,
Eemnes,
The Netherlands.
Bert Morrien · 699 weeks ago
This site is hosted by my ADSL provider, I'm not sure that it is available world-wide and I was not supposed to cause a lot of traffic to this site.
Apart from that, what is your position in using invalid data in experiments, i.e. using a response for which there is no corresponding stimulus, or, using a stimulus for which there is no corresponding result?
The latter is the case in the current analysis.
If you allow that, we don't have anything to discuss.
Bert
macgupta 81p · 699 weeks ago
I wanted to point out this from the OPERA paper (section 5.)
"The OPERA data acquisition system (DAQ) time-tags the detector TT hits with 10 ns quantization with respect to the UTC".
This essentially wipes out all the fine detail that is in the proton extraction waveform.
---
If I understand Figure 1 in AA.pdf, this is the Kolmogorov-Smirnov test. http://en.wikipedia.org/wiki/Kolmogorov%E2%80%93S...
I think this test is superior to the Maximum Likelihood computation done by OPERA, but haven't shown it yet. That is, if I sample a probability distribution function in an OPERA-like way, then just the finiteness and randomness of the sample induces a false shift that the maximum likelihood procedure detects but the K-S procedure doesn't (or is much less susceptible to.) This I think can be demonstrated by simulations.
--
Some further issues on the measurement of the proton extraction waveform are raised by Juergen Knobloch,
Is there a neutrino speed anomaly?, linked in the blog post.
--
The Antonio Palazzo preprint linked above also touches upon the construction of the proton extraction waveform, and is worth a look see.
--
Bert Morrien · 699 weeks ago
The Opera paper explicitely states that they are matching two event distributions.
I deny that the PDF is an event distribution in the sense that it has nothing to do with the available neutrino detection distribution, because the PDF is only an average of the proton distribution, which is garbage.
Macgupta.
I know about the time uncertainties, but these do not wipe out all the fine detail in the PEW, it only means that the detail of interest has a 10 ns accuracy.
For the set of PEWs, this means the start time information forms a Gaussian distribution.
The result of the summing of the aligned PEWS is a proper PDF in the form of the mentioned Gaussian distribution, in which the noise is only a small fraction from the noise that is present in the sum of the unaligned PEWs.
A disturbing factor is that the 200 MHz component in the PEWs is responsible for a number of Gaussian shapes, symmetrically positioned with respect to the shape at the correct start time. Because the other components have better autocorrelation properties, the extra shapes are expected to be lower.
Monte Carlo techniques can be used to lessen the effect of the time uncertainites if the maximum is ambiguous.
A qualitative analysis of IA.pdf would be welcome.
Since the proposed algorithm is simple, the analysis could be done on any high end home PC.
If only OPERA could publish the data, the IA could be easily tested.
I don't understand http://en.wikipedia.org/wiki/Kolmogorov%E2%80%93S... because I am not familiar with the used notation.
I searched for graphical visualisations, but I can't see the method in AA.pdf is the same.
The method in AA.pdf is a special case of a more general solution, that allows for events with less time uncertainty.
In that case the PDF can be split in equal probable parts and for each event the MLA is done within its own part.
However, the method presented in IA.pdf is superior as soon as the result becomes unambiguous.
The MLA in AA.pdf was only meant as a demonstration of a possible correct mechanism, but it has important drawbacks.
Nevertheless, I contacted the patent office of my former employer to see if method in AA.pdf and the one of IA.pdf are new.
Like I said, I do not believe in the validity of the PDF because of all the noise in it.
I am not interested in what OPERA thougt what they were douing, I am observing what they actually did.
Leaving invalid data in the PEW is a mortal sin against first principles.
Proposing an alternative analysis method that may not be perfect is better than using one that is wrong.
Bert
Arun · 699 weeks ago
The average of the proton distribution is entirely garbage only if the various individual instances vary widely. Imagine the repeated proton distributions appearing on an oscilloscope. If it produces a meaningful display, it isn't garbage. Yes, it introduces noise, but - depending on the actual data set - may retain considerable signal.
Bert Morrien · 699 weeks ago
I thought the same to get rid of the 200 MHz, however, increasing the bandwidth of BCT and digitiser would do a better job by obscuring it, i.e. making the autocorrelating property of the PEW better; in this way the intrinsic precision is not compromised.
A quantitative analysis is needed.
You did not understand my meaning of garbage: only a part of the PEW has a corresponding event.
The rest is garbage because it predicted events that did not happen.
Only the part of the PEW that has a corresponding event is information about the start time of the neutrino that caused the event.
You must take great care that the garbage does not disturb this information.
Summing the PEWs is a very effective way to replace the information with the garbage.
It's like a bunch of old lottery tickets without a prize, except for one.
It's like having a drop of pure water and diluting it with thousands of drops of salt water
If you don't agree, that's the end of the discussion for me.
Bert
macgupta 81p · 699 weeks ago
I don't agree. As a simple example, you cannot get how the rate of neutrino events scales with the proton rate unless you include the "garbage". As far as timing is concerned, you need to time the edge of the PEW (from which relatively few neutrino events result) in order to have any meaning for the timing of the piece of PEW that resulted in the neutrino event, as well as to have any meaning for the timing of the neutrino event.
So I guess, end of discussion.
Bert Morrien · 699 weeks ago
OK, thanks, maybe there comes a time you understand.
Bert
Bert Morrien · 699 weeks ago
Feynman once said:
"We don't have the experiments and thus we do not know which results to
calculate".
Arun · 699 weeks ago
Bert Morrien · 698 weeks ago
Sorry about my ultimatum, that's not a good way to communicate.
Please read my documents and then state why a repeatable PEW is necessary.
If I did not make myself clear, please ask me specific questions; I am confident I have the answers.
But, to come back to the validity of OPERA's analysis method, you make apparently the following assumptions.
1. Only the leading and trailing edges are useful for determining the start time.
2. The leading and trailing edge of the PDF are correct.
In radar, long pulses are used to measure distances by comparing the reflected pulse with the original pulse.
If the pulse was not modulated, the leading and trailing edges would become important and the remaining part a waste of energy.
However, the pulse is modulated so that it resembles the delayed original pulse only if it is delayed with exactly the correct echo time. In this way, every part of the pulse is useful for the timing.
See also http://en.wikipedia.org/wiki/Pulse_compression
Another example:
In the 1960's I was involved in the development of bit- and frame synchronisers for the telemetry system of ELDO, the European Launcher
Development Organisation.
A frame synchroniser determines the position of the first bit of a frame in a data stream.
A frame is a block of e.g. 256 consecutive bits.
For this purpose one frame was filled with a so-called pseudo-random sequence of bits
The received data stream was shifted through a frame-wide register, e.g. a 256 bit shift register.
The outputs of the register were summed, so that if the pseudo-random sequence was present, the sum would be 256; in all other situations it was around half this value.
In this example a 256 bit sequence is used to achieve a 1 bit accuracy.
This worked also in the presence of big amounts of noise, which was important, because if the ground station was synchronised immediately when the satellite emerged above the horizon, valid data could be recorded as soon as the signal to noise ratio allowed this.
The rich structure of the PEW makes it excellent for timing purposes and so the first assumption is wrong.
The edges of the PDF would be correct only if these were determined by PEWs with a corresponding event at the leading of trailing edge.
However, most corresponding events are located in the remaining part of the PDF, i.e. most PEWs have no corresponding event at the leading or trailing edge.
This invalidates the second assumption, as is shown by the following example.
A shot is fired at t1 and hits the target at t2; given the distance to the target the velocity of the bullet can be calculated.
Now, a shot is fired and it misses the target, so only t1 is available; using t1 without t2 is nonsense, but that was done with the OPERA
experiment.
Due to the summing of the PEWs, they used almost entirely PEW parts without a corresponding event.
The PDF would be valid only if all parts of all PEWs had a corresponding event.
As it is, only one in 10,000 parts of the PDF has a corresponding event, which makes it invalid.
Bert
Bert Morrien · 698 weeks ago
Using a 1 ns window as I propose in the improved analysis works better without much effort.
The time uncertainties would only result in a wider and lower bell-shape, but I expect a single convincing maximun when 8000 events and corresponding PEWs are analysed.
It's a pity OPERA has not published the data; I think it fits on a single 700 MB CD and it can be analysed with any reasonable home PC.
I have sometimes the idea that I am a kind of illusionist trying to explain my simple trick while everybody thinks it is very complex.
OPERA makes it complex. At this moment, nobody could explain to me how OPERA's analysis exactly works in a convincing way.
Several scientists tried it, but they couldn't without using murky reasonings.
The reason is simple: it cannot be explained; maybe it can be excused.
David Brown · 697 weeks ago
The -1/2 in Einstein’s field equations should be replaced by -1/2 + dark-matter-compensation-constant. If nature provides a curling-up mechanism for M-theory, then the dark-matter-compensation-constant is an approximate, heuristic mechanism that represents superpartners found in the supersymmetry predicted by Seiberg-Witten M-theory. If nature operates according to modified M-theory with Wolfram’s automaton, then the dark-matter-compensation-constant is a totally precise constant that can be calculated using Planck’s constant, Newton’s gravitational constant, the speed of light in a vacuum, and the Fredkin-Wolfram constant. Dark matter might be thought of as D-brane reinforcement of the Einsteinian gravitational signal. The Pioneer anomaly and Fernandez-Rañada’s idea of anomalous gravitational acceleration of clocks yield the value sqrt((60±10)/4) * 10^-5 for the dark-matter-compensation-constant. http://en.wikipedia.org/wiki/Dark_matter http://en.wikipedia.org/wiki/Pioneer_anomaly
Is the OPERA neutrino anomaly actually a confirmation of M-theory via the Rañada-Milgrom effect?
Bert Morrien · 697 weeks ago
This result was compatible with the earlier finding, and so was the result of a new experiment with much shorter pulses.
This means, Opera’s current analysis must be valid.
This means also that Opera knew exactly what they were doing.
Consequently, the PDF obtained by summing the PEWs is valid, despite the lack of PEW parts with a corresponding event.
This is because with enough events, the event distribution resembles the shape of the PDF sufficiently for trusting the outcome of a maximum likelihood analysis.
It is regrettable that this point never became clear to me before.
The lesson learned is that declaring the PDF and Opera’s analysis invalid is a good example of narrow minded reasoning; a humble apology is in order here.
Bert
Bert Morrien · 697 weeks ago
I noticed also that they modified fig.4. The old version suggested a window that did not include the rising edge of the proton spill.
If that was real, events associated with the leading edge would be considered as if they were fired within the window, causing a shift towards a shorter TOF. In the new version, the blue/green range indicator is replaced with a red one, now correctly covering the whole proton spill. Ofcourse, the wrong window was not real, but it illustrates the fact that using windows is dangerous, because bias is easily introduced..
If the alternative analysis worked the way as explained at http://www.science20.com/alpha_meme/faster_light_...
this window placement was not critical at all, because evenit outside the window would be ignored, as they cannot be associated to a PEW part.
Moreover, because the PEWs are adressed relative to the event time, there is no senstiivity for uneven event spreading.
The PDF is a Gaussian distribution, the events form a Poisson didtribution, that effectively imposes multiple small windows on the PDF which cannot be controlled. Only after many events, their distribution will resemble a Gaussian distribution, but even then it is necessary to revert to binning to show a resemblance with the PDF in fig. 13 and 14. Note that the time resolution of these figures is insufficient to allow firm conclusions.
This all added to my suspicion that Opera's method might not precise enough. Ofcouse, the new result proved me wrong and I am glad for it, because those guys apperently knew what they were doing when analysing the result, despite this method was no routine for determining the TOF, because it was only done twice before.
Bert