When searching for a new particle, a discovery is claimed if a large enough incompatibility between data and the null hypothesis is observed. If no deviation is observed, an upper limit on the test hypothesis is extracted.
The profile likelihood ratio ˜λ(µ) (Eq. 3.7) ranges from 0 < λ <˜ 1 with ˜λ close to 1 implying good agreement between the data and the hypothesized value of µ. Although it is more commonly used the form −2 ln ˜λ(µ) for reasons explained in the following Sections.
3.4.1 Test statistics q0 and qµ
The procedure to discover a new signal is to reject the background-only hypothesis. The statistic to do so is qµ whenµ= 0 defined as:
q0 =
−2 ln ˜λ(0) µˆ≥0
0 µ <ˆ 0. (3.11)
3.4. DISCOVERY AND UPPER LIMIT
Figure 3.1: (Left) Log-normal distributions with κ = 1.10, 1.20, 1.33 and 1.50. (Right) Gamma distribution with the number of events in a sample B = 100,25,9 and 4. Ref. [41].
An increase of event yields above the expected background translates into an increase of ˆµ, thus ˆµ ≥ 0 means that a new signal could be present.
If this is the case, larger q0 values will correspond to an increase level of incompatibility between data and the background-only hypothesis. Since only upward fluctuations of the background are considered, by construction if ˆµ < 0 then q0 = 0, indicating a compatibility with the background-only hypothesis.
If no discovery has been made, one can define upper limit on the signal strength parameter µ. The test statistic is defined differently compared to the discovery case:
qµ=
−2 ln ˜λ(µ) µˆ≤µ
0 µ > µˆ =
−2 lnL(µ,θ(µ))ˆˆ
L(0,θ(0))ˆˆ µ <ˆ 0
−2 lnL(µ,θ(µ))ˆˆ
L(ˆµ,θ)ˆ 0≤µˆ≤µ
0 µ > µ.ˆ
(3.12)
In this case,qµis set to 0 if ˆµ > µbecause no upper limit could be set if the observed signal strength is larger than the tested one. As for the discovery case, higher values ofqµrepresent greater incompatibility between the data and the tested value ofµ.
CHAPTER 3. STATISTICAL MODEL
3.4.2 Probabilities p0 and pµ
The compatibility of data with the background-only hypothesis is given by thep-value:
p0= Z ∞
q0,obs
f(q0|0,θ(0))dqˆˆ 0, (3.13) where q0,obs is the observed value of the test statistic and f(q0|µ) denotes theq0 pdf under the assumption of the signal strength µ= 0 andθ(0). Inˆˆ the presence of a signal, the p0 is generally converted to the corresponding Gaussian significance,Z, defined as the Gaussian distributed variable with upper-tail probability equal to thep0 which correspond toZ standard devi-ations above its means. That isZ = Φ−1(1−p0), where Φ−1 is the quantile (inverse of the cumulative distribution) of the standard Gaussian. In parti-cle physics, it has been adopted the convention that if the background-only hypothesis is rejected with a p0 = 1.3×10−3, equivalent to a significance Z = 3, an evidence is announced. A discovery is claimed for Z = 5, corre-sponding top0 = 2.9×10−7.
If no signal is present in data, the signal+background hypothesis can be excluded with a certain Confident-Level, and an upper limits can be set on its cross section. This is done by calculating the probability that theqµtest statistic for a given signal strength µassumes a value equal or higher than the observed valueqµ,obs using thep-value expression:
pµ= Z ∞
qµ,obs
f(qµ|µ,θ(µ))dqˆˆ µ. (3.14) The upper limitµup at 95 % CL is defined as the highest value of the signal strengthµ for which the probabilitypµis still higher or equal to 5 %.
3.4.3 CLS method
When downward fluctuations on the observed number of background events are produced, upper limits computed usingpµcan lead to unphysical exclu-sion of small µvalues to which the search is not a priori sensitive. This is overcome in the analysis by using theCLS method [42] defined as a ratio of probabilities
where ps+b and pb quantify the compatibilities between the data and the signal+background and background-only hypotheses, respectively. The pb
is defined as
pb = 1− Z ∞
qµ,obs
f(qµ|0,θ(0))dqˆˆ µ. (3.16) A downward background fluctuation in data will lead to small values of 1−pb increasing theCLS upper limit and avoiding the exclusion of too small cross
3.4. DISCOVERY AND UPPER LIMIT
sections. In LHC Higgs searchesCLS upper limits at 95 % confidence level are obtained by computing theµ for whichCLS gives 0.05.
Figure 3.2 shows the pdfs for the test statistics q1. It also illustrates the computation of the signal+background and the background-only hypotheses probabilities used in theCLS method for an observationqobs obtained from data.
Figure 3.2: The distribution of the statistic q under the hypotheses sig-nal+background and background-only and qobs obtained from data. Green and yellow areas correspond to ps+b and pb probabilities defined with Eq. 3.14 and Eq. 3.16 integrals respectively. This illustrative Figure uses the Tevatron Collider test statistic which is defined with a different likeli-hood ratio. Figure obtained from Ref. [43].
1In this caseq represents the Tevatron Collider test statistic which is defined using a different likelihood ratio.
CHAPTER 3. STATISTICAL MODEL
3.5 Approximate sampling distributions and the Asimov data set
The computation of a p-value associated with a hypothesis needs the full distribution of the test statistic as shown in Eq. 3.13 and Eq. 3.14. In particular, when trying to make a discovery, the background only hypothesis needs to be tested usingf(q0|0,θ(0)). Similarly, when computing an upperˆˆ limit using theCLS method, thef(qµ|µ,θ(µ)) to describe theˆˆ qµdistribution is also needed.
The estimation of qµ and q0 distributions could be done with Monte Carlo methods. These methods are computationally heavy specially when calculating upper limits. As an example, if a discovery with p0 ∼ 10−7 wants to be claimed, an order 108 pseudo-experiments toys would need to be simulated. For this reason, an approximation [43] valid in the large sample limit is normally used to describe the profile likelihood ratio instead.
If the assumption is that the data are distributed according to a strength parameterµ0, one can write using Wald’s approximation [44]
−2 ln ˜λ(µ) = (µ−µ)ˆ 2
σ2 +O(1/√
N), (3.17)
where ˆµfollows a Gaussian distribution with a meanµ0 and standard devia-tionσ, andN accounts for the data sample size. The value ofσ is estimated from an artificial data set known as ‘Asimov data set’ [43].
The Asimov data set takes its name after I. Asimov’s short story Fran-chise, where one most representative voter was chosen to act as the whole electorate. To construct it, the pseudo-data is forced to be equal to its ex-pectation value ni,A = E[ni] = µ0si(θ) +bi(θ) and all NPs are set to the values obtained from the best fit to data.
The test statistics qµ and q0 can be rewritten using Eq. 3.17 approxi-mating the pdfs f(qµ) and f(q0) to χ2 distributions [43]. This asymptotic approximation of the profile likelihood ratio works very well and reduces the computational time since calculations are done analytically. An example of distributions obtained with this method is shown in In Figure 3.2 where the histograms are from Monte Carlo and the solid curves are the predictions of the asymptotic approximation.
In Ref. [43] approximate equations are derived for the significance and upper limits using the asymptotic approximation. The significance can be written as
Z =√
q0, (3.18)
and the upper limit onµ can be written as
µup= ˆµ+σΦ−1(1−α), (3.19)