1.2.2 Information theory
Information theory is a mathematical framework created to study the trans-mission of information through communication systems. It was primarily developed by Claude Shannon, with a first publication in 1948 . This paper was focused on the transmission of information in the presence of noise in the channel of communication. It also introduced the concept of entropy, which we will describe in detail later in this Introduction. Soon after this publication, the scientific community recognized the significance and flexibility of this method and researchers started to apply information theory to fields outside of its original scope, for example in statistics, bi-ology, behavioural science or statistical mechanics, and also, obviously, in Neuroscience 
The way in which neurons transmit information was debated at the time the paper from Shannon was published. Information theory thus appeared
to be the perfect tool to answer this question . The first works using information theory in Neuroscience were published in the 50s and indeed showed that neurons are able to relay large quantities of information [71–73].
These works were the starting points for many subsequent studies trying to better characterize information flow in specific neural systems. Information theory has been used to address many different Neuroscience questions and in a number of different contexts. It has led to a number of advances and developments of theories for brain functions, and it is still used today in order to better understand how neural cells process information. The interested reader can see references , ,  and  for reviews on the subject.
Information theory in Neuroscience
Information can have different meanings in different contexts . In Neu-roscience, information is often evoked when discussing stimulus encoding (information encoding), decision-making (information processing) or memory (information storage). Information of a neural variable is generally associated with a reduction in uncertainty of another variable or stimulus . Informa-tion is measured in bits, which should allow for straightforward comparisons between cells, brain regions, tasks or subjects.
Data obtained from Neuroscience experiments are frequently noisy and often represent systems with non-linear interactions. They also can be composed of numerous variables (e.g. physiologic, behavioural or stimulation data) that interact. Information theory offers a number of multivariate analysis tools, it can be applied to different types of data, it can capture non-linear interactions and it is model-independent. Because of its general applicability, information theory is widely used in Neuroscience, for example on analysis of electroencephalography (EEG), magnetoencephalography (MEG), functional magnetic resonance imaging (fMRI) and trial-based data, or on studies on connectivity or sensory coding . It can detect a wide range of interactions and structure, even in large or complex systems.
The results of information theoretic analysis can quantify the uncertainty of variables, the dependencies between variables and the influence some variables have on others . It thus can be used to quantify encoding (how much information a neuron conveys) [80, 81], or complex encoding relationships (how much information two neurons convey together) [82, 83].
It is important to note that information theory can not produce models that describe how the system works (but it can be used to restrict the space of possible models) .
The following sections present the metrics of information theory used in the present thesis. Other tools to measure information exists, for more details on those, see for example reference .
The concept of entropy is something physicists are utterly familiar with.
Developed in Boltzmann’s theories about statistical mechanics and thermo-dynamics, entropy measures the disorder in a system. The entropy used in information theory and developed by Shannon has a similar mathematical formulation but the reason why Shannon chose to also call it entropy is attributed to John van Neumann :" The theory was in excellent shape, except that he needed a good name for "missing information". "Why don’t you call it entropy", von Neumann suggested. "In the first place, a mathematical development very much like yours already exists in Boltzmann’s statistical mechanics, and in the second place, no one understands entropy very well, so in any discussion you will be in a position of advantage. "
The entropy is the fundamental quantity in information theory. It mea-sures the uncertainty contained in a variable. For a variableXwith individual statesx, the entropyH(X) is defined as [63, 79]:
H(X) = X
This ensures H(X) ≥ 0, as a negative uncertainty would not have any meaning. Moreover, systems with one absolutely certain state haveH(X) = 0, as there is no uncertainty over their state. In this thesis, we will concentrate on discrete distributions, but it is interesting to note that there also exists an extension of entropy for continuous distributions .
For systems with two variables,X with statesxandY with statesy, the joint entropyH(X, Y) is defined as: The average uncertainty in a variable, given the state of another variable, is calculated with the conditional entropyH(X|Y) (also called noise entropy) and is defined as:
H(X|Y) = X
p(x, y)log2 1
Conditional entropy can also be used to rewrite the joint entropy:
H(X, Y) =H(X) +H(Y|X). (1.4) Figure 1.7 gives a more visual interpretation of Equations 1.1 to 1.4, where it can be seen that the total area ofH(X, Y) is composed by the areas of H(X) andH(Y|X), or equivalentlyH(Y) andH(X|Y).
Figure 1.7:Venn diagram of entropy and mutual information. The entropiesH(X) andH(Y) each are a circular area. The area formed with the combination ofH(X) andH(Y) is the joint entropyH(X, Y). The conditional entropyH(X|Y) is the area formed by removing fromH(X) the part ofH(X) that intersects withH(Y). The other conditional entropyH(Y|X) is the area formed by removing fromH(Y) the part ofH(Y) that intersects withH(X). The area at the intersection betweenH(X) andH(Y) is the mutual informationI(X;Y). It can also be constructed as the area ofH(X) minus the area ofH(X|Y), or equivalentlyH(Y) minusH(Y|X).
The entropy measures the uncertainty in one variable. If two variables are dependent from each other (for example the output spike trains of two connected neurons), then learning the state of one variable reduces the uncertainty in the other variable. This means that one variable will provide information about the other variable. This can also be written as:
H(X) =H(X|Y) +I(X;Y), (1.5) whereH(X|Y) is the entropy that remains in X, given knowledge aboutY, andI(X;Y) is the information provided by Y aboutX measured in bits.
This is also illustrated in Figure 1.7. Equation 1.5 can be rewritten as:
p(x)p(y). (1.6) I(X;Y) is called the mutual information . It corresponds to the reduction of uncertainty inX, given knowledge of the state ofY.H(X)≥0 implies that I(X;Y)≤H(X).
By symmetry,I(X;Y) can also be rewritten as  (see also Figure 1.7):
This means that the information Y provides about X is the same as the informationX provides aboutY. Furthermore, if we combine Equation 1.7 with Equation 1.4, we obtain the alternate form:
I(X;Y) =H(X) +H(Y)−H(X, Y). (1.8) It is possible to expand the mutual information to take into account more than two variables . An alternate form of mutual information that is very helpful for causal relations is the mutual information between two variables conditioned on a third variable, the conditional mutual information:
I(X;Y|Z) =H(X|Z)−H(X|Y, Z)
The transfer entropy was first introduced by Schreiber in 2000  as a tool to measure the statistical coherence between systems evolving in time.
It is defined as a conditional mutual information with assumptions about temporal order:
This quantity measures the information about the future state of a variable Y+ provided by another variable in the past X− given the information provided by the past state of the first variable Y−. The transfer entropy measures the changes caused inY fromX that cannot be derived only by the past states ofY, like it is the case for example ifX was an inhibitory neuron impinging on neuronY, i.e. the past of X is more informative about the future state ofY than the past ofY itself.
In a Neuroscience context, and in this thesis in particular, when applied to spike trains, transfer entropy measures the information flow between
neurons, and mutual information measures the encoding of stimulus and behavioural information shared by individual neurons. Mutual information is symmetric under the exchange of the variables. On the contrary, transfer entropy measures the direction of information flow and is, by definition, non-symmetric. Thus, transfer entropy can be seen as a conditional mutual information with assumptions about temporal structure. This means that depending on the temporal structure of the data, mutual information and transfer entropy could evaluate similar quantities. This is interesting, because even if the quantity evaluated is similar, the estimation methods will differ.
The accuracy of those estimation methods seems then to be an important point to study.
Application and limitations of those metrics
The basic idea when conducting a Neuroscience experiment aiming to measure the information flow or encoding is to send a stimulus to a neuron (input) and to record the response of the neuron (output). Information theory and, specifically, the metrics introduced above, can then be used to calculate the information flow between the recorded input and output. But, to do that, the data used need to be discretized, even though biological processes are continuous. In a Neuroscience experiment, the recording on the computer of continuous biological processes ends up being discrete with some sampling rate determined by the experimenter and dependent on the equipment used.
The recording is usually sampled at a higher frequency than necessary for data analysis with information theory, and must be downsampled by using binning or discretization methods . The choice of the method used is important for a correct evaluation of the probability distributions. This is a very system-specific procedure, and a poor choice of discretization will lead to a wrong evaluation of the information theory metrics .
Estimating probability distributions is necessary to any information theoretic analysis, and, in an experimental context, these distributions must be determined from raw experimental data. In 1998, De Ruyteret al.
developed a relatively straightforward way of estimating information carried by a spike sequence, as well as a new method for the application of information theory to neural spike trains . This new method represented an advance from a methodological point of view and has led to many applications.
The issue one will encounter when evaluating the information from probabilities reconstructed from experimental data is the bias that appears due to a finite sample size. To calculate the information it is necessary to reconstruct the probability of each possible state (as it can be seen in the above Sections, every metrics introduced is based on probabilities). A state can be, for example, a word formed as a string of bins, such as 100 010 001 101 011 110 111 000 for words of length 3. The probability of a given word
is reconstructed by counting every occurrence of this word in the data and dividing this number by the total length of the data. But, if the dataset is not big enough, there is a risk to poorly evaluate those probabilities. For example, let us consider a very unlikely word, with a very small (but still bigger than zero!) probability of occurrence. In a small dataset, chances are that a very unlikely word is never observed. The probability of this state would thus be evaluated to be zero, an underestimation. Of course, because the sum of all the probabilities must, by definition, be equal to 1, this implies that the estimates of the other probabilities will also be incorrect, some of them will be overestimated.
This then propagates in the estimation of the entropies, resulting in the entropies being underestimated . Intuitively, this bias can be understood because the entropy is a measure of variability in the dataset, which means that a smaller dataset is less likely to fully sample the full range of possi-ble responses. A more mathematical understanding can be achieved with the help of Figure 1.8, where we can see that miss-evaluating very small probabilities will impact the evaluation of the functionh(x) =p(x)logp(x)1 the most. Indeed, a small probability underevaluated leads to a big error in the evaluation ofh(x) (as shown with the red arrows in Figure 1.8), as a higher probability overevaluated leads to a smaller error inh(x) (see the orange arrows in Figure 1.8). This means that the entropy, calculated as P
xh(x) (see Equation 1.1) will be underestimated. This graph also shows that the largest error will be for probabilities close to zero, which are the most difficult to correctly sample.
If we take the case of the evaluation of the mutual information (see Equation 1.6), the entropy and the conditional entropy will both suffer from this bias and be underestimated. Interestingly, the conditional entropy will often be underestimated by a larger amount that the entropy. The estimation of the entropy necessitates the evaluation of all the probabilities p(x) (see Equation 1.1). For words of lengthL for instance, there are 2L possible combinations ofp(x) that need to be evaluated. On the other hand, conditional entropy necessitates the evaluation of 22L possible combinations of p(x|y) (see Equation 1.3). This means that in a limited dataset, p(x) will be better sampled than p(x|y), resulting in the conditional entropy being more significantly underestimated than the entropy. This usually results in the mutual information (entropy minus conditional entropy) being overestimated.
Poor evaluation of the probabilities will bias the estimation of entropies and mutual information, but enough observations can help adequately sample the space of states and reduce this bias . This is however not always possible from an experimental point of view. To alleviate this issue, several correction methods for biases have been developed. The first theoretical (as in non-empirical) correction method was developed by Treves and Panzeri 
Figure 1.8:Plot of the functionh(x) =p(x)logp(x)1 . The arrows show two examples were the probabilities are slightly miss-evaluated. The red arrows show a small probability (>0) evaluated to be zero (underevaluation), leading to a big impact in the evaluation ofh(x). The orange arrows show a much larger probability slightly overevaluated, leading to a relatively smaller impact in the evaluation ofh(x). Both errors in the evaluation of the probabilities have the same magnitude.
in 1995. A few years later, in 1998, Strong and colleagues developed the so-called direct method , which is relatively easy to implement and is thus widely used by experimentalists. For a more thorough review on correction methods, see . It is important to note that all these correction methods are designed for the mutual information. Nonetheless, because transfer entropy is also the subtraction of two entropies, its estimation could suffer from similar biases too.
This Introduction focuses on the direct method, as it is the method used in this thesis for the evaluation of the mutual information. It was also used in this thesis to adapt a correction method for the transfer entropy (see Chapter 2 for more details).
The direct method
The direct method applies two distinct corrections for the evaluation of both entropies entering the calculation of the mutual information. The first correction aims to correct the bias due to a limited dataset. The idea is to extrapolate the entropy (total entropy or conditional (noise) entropy) to an infinite dataset by plotting the entropy for fractions of the dataset.
Figure 1.9 shows an example of how this correction is applied in the original work from Strong and colleagues, for words of length 10 bins. The total entropyHtotis computed for the whole dataset, as well as fractions of the dataset ranging from 12 to 15. The points are then fitted with a function
of the formHtot=H0+sizeH1 +sizeH22, wheresizedenotes the inverse data fraction. The intercept at the fraction 10,H0, gives the extrapolated value of the entropy for an infinite dataset.
Figure 1.9: First correction of the direct method of Strong. Words used in this example have a length of 10 bins. Total entropy is evaluated over different fractions of the whole dataset. To extrapolate the value of the entropy for an infinite dataset, the points obtained is fitted with a function of the formHtot=H0+sizeH1 + H2
size2. The interceptH0is thus the value for an infinite dataset. Adapted from reference .
In addition to correcting the bias occurring for a small dataset, the direct method also offers a way to correct the bias due to finite word lengths.
Ideally, the entropies should be estimated with an infinite word length to accurately take into account possible long-range correlations or stereotypical temporal patterns. Correlations and patterns reduce the information content of a spike train, but their time scales are usually not known. An infinite word length would thus ensure that they are all taken into account. For a small dataset, long words will particularly suffer form the bias described above, creating what is referred to as a sampling disaster by Strong and colleagues . A sampling disaster occurs for relatively long words, when the first correction is not sufficient to correct the bias, leading to a strong underestimation of the entropy (see Figure 1.11C below). Interestingly, when the correlations have a limited range, their contribution to the entropy is usually a constant and emerges before the sampling disaster . This means that entropies calculated for word lengths before the sampling disaster occurs (i.e. for relatively short words) can be used to extrapolate the estimates of the entropy and conditional entropy for words of infinite length (again, see Figure 1.11C).
A good example of the application of the direct method (and specifically the second correction) on experimental measurements can be observed in the work of Reinagel and Reid . In this work, the authors used the direct method to calculate the stimulus-dependent information conveyed by single neurons in the cat lateral geniculate nucleus in response to randomly modulated visual stimuli (Figure 1.10 shows an example of how this kind of experiment is conducted). Their data were discretized accordingly.
Figure 1.10:Example of a typical experiment on the cat visual pathway. A randomly modulated visual stimuli is shown to a cat and electrodes are used to record the responses of the target neurons. Adapted from reference .
Figure 1.11 shows how they evaluated mutual information for their mea-surements. As defined in the direct method, they evaluated the probability distributions for several word lengths. The entropy is evaluated from the responses to unique stimuli (see Figure 1.11A) and the conditional entropy is evaluated from the responses to a repeating stimuli (see Figure 1.11B). The mutual information is then calculated as the entropy minus the conditional entropy (see Equation 1.6).
One of the very powerful ideas introduced by the direct method (apart from the corrections) is how the conditional (or noise) entropy itself is evaluated (before applying the corrections). Instead of evaluating the joint and conditional probabilities for states between the input and output (like Equation 1.3 suggests), the conditional entropy is evaluated by comparing several outputs (called repetitions) generated by the same repeated input.
The repetitions are aligned and a value of conditional entropy is calculated for each particular word position in the outputs across the repetitions (as can be seen in Figure 1.11B). The estimate of the conditional entropy is then evaluated as the average over all particular word positions. This allows the estimation of the conditional entropy to be decorrelated from the transmission processes between the input and the output. Those can indeed be a source of issues in estimating the information flow, or simply
be experimentally inaccessible. It is nonetheless important to note that this does not remove the conditionality on the input, as every repetition of the output is aligned so that each bin in every repetition corresponds to the same bin in the input (as shown in Figure 1.11B).
Figure 1.11:Example of the application of the direct method to estimate the mutual information. A: Probability distribution of words of lengthL= 8 for time bins of 1msused for the calculation of the entropy. The inset is a sample of responses to a non repeated stimuli, with some words of length 8 highlighted. B: Probability distribution of words of lengthL= 8 for time bins of 1msused for the calculation of the conditional entropy at one particular word position. The inset is a sample of responses to a repeated stimuli, with words with a fixed timing highlighted. C:
Estimated entropy rate (top) and conditional entropy rate (bottom) as a function of the inverse word length 1/L. The dashed line is the extrapolation from the linear part
Estimated entropy rate (top) and conditional entropy rate (bottom) as a function of the inverse word length 1/L. The dashed line is the extrapolation from the linear part