Intra-individual variability in accuracy scores: When biased coefficients always tell the same story

(1)

Poster

Reference

GOLAY, Philippe, FAGOT, Delphine, LECERF, Thierry

GOLAY, Philippe, FAGOT, Delphine, LECERF, Thierry. Intra-individual variability in accuracy scores: When biased coefficients always tell the same story. In: 16th European Society for Cognitive Psychology Conference, Krakow, 2nd-5th September, 2009

Available at:

http://archive-ouverte.unige.ch/unige:29436

Disclaimer: layout of this document may differ from the published version.

1 / 1

(2)

0 10 20 30 40 50 60 70 80 90 100

0 1 2 3 4 5 6 7 8 9 10

Maximum Value of Coefficient

Mean Reaction Time

Reaction times

0 1 2 3 4 5 6 7 8 9 10

Maximum Value of Coefficient

Average Number of Correct Answers

Accuracy Scores

0 1 2 3 4 5 6

0 1 2 3 4 5 6 7 8 9 10

Coefficient Value

Average number of correctly recalled elements (Max = 10)

Simulated Data, N = 1000

P PD

x n xi

Max ⁿ

i

2 1

)2

1 (

Maximum iSD as a function of Performance (average

number of stimuli correctly recalled)

P = Performance

D = level of Difficulty

(e.g. 10 stimuli to remember in each item)

INTRODUCTION AND OBJECTIVE

METHOD

CONCLUSIONS RESULTS

Contact: [email protected] ESCoP 2009, XVI Conference | Krakow, Poland, 2nd – 5th September, 2009 Supported by the Swiss National Science Foundation Grant N°100011-107764

Intraindividual variability in accuracy scores: When biased coefficients always tell the same story

Philippe Golay, Delphine Fagot & Thierry Lecerf

Faculty of Psychology and Educational Sciences, University of Geneva, Geneva, Switzerland

• Intraindividual variability (IIV) across trials or occasions is frequently measured with Intraindividual Standard Deviation (iSD) and/or with Intraindividual Coefficient of Variation (iCV = iSD divided by Individual Mean, iM);

• iSD measures IIV on an absolute scale (same unit of measurement, e.g. ms)

• iCV measures IIV on a relative scale (ICV is not related to a metric/dimensionless number)

• IIV is mostly studied with Reaction Times (RTs), but can be investigated with accuracy measures (e.g. number/percentage of correct responses in a memory task);

• With accuracy measures, limits of the domain of possible scores for iCV introduce strong linear dependency between that coefficient and Mean performance;

• iCV has a strong bias towards depicting participants with lower scores as being more variable;

• iCV forces IIV and performance to appear as the same source of variance;

• Quadratic relationship between Mean and Maximum Variance (

²

) :

• No valuable reason to divide iSD by individual mean;

• No valuable reason to use iCV with accuracy measures;

 To get an insight on variability relative to one’s performance, we recommend to divide iSD by its respective maximal value. This “corrected iSD” reflects IIV relative to the level of variability attainable at that specific level of performance.

• 5x5 matrix: Free recall of black cells (computerized task)

• Simultaneous presentation (presentation time = number of black cells x 1 sec)

VISUOSPATIAL WM TASK: The MATRIX TASK



Level 5

ADAPTATIVE PROCEDURE

• Span Level determination (N; ascending procedure)

• 10 items at level N (span level)

• 10 items at level N+1 (supra-span level)

DEPENDENT VARIABLES

• Locations correctly recalled

- Individual Mean (iM; proportion)

- Intraindividual Standard Deviation (iSD)

- Intraindividual Coefficient of Variation (iCV)

SAMPLE DESCRIPTION

 RTs have a lower finite bound (e.g. a minimum RT)

 Accuracy measures have an upper and a lower bound (0 to 100 %)

• iSD

• iCV

RTs :

• Range of iSD increases with mean

• Range of iCV has a constant upper limit

Accuracy measures:

• Range of iSD increases and then decreases between two limits (min & max score)

• Range of iCV tends to zero as mean increase

• iSD

• iCV

TOTAL SAMPLE N = 201

9 year N=50

10 year N=50

11 year N=51

12 year N=50

% N % N % N % N

Gender Female 38 19 38 19 52.9 27 54 27

Male 62 31 62 31 47.1 24 46 23

R = .00

R= -.77

Span Level (N)

Mean x iSD : R = -.73, p<.01 Mean x iCV : R = -.88, p<.01

Supra-span Level (N+1)

Mean x iSD : R = -.68, p<.01 Mean x iCV : R = -.85, p<.01

ADDITIONAL RESULTS: THE READING SPAN TEST

• IIV is negatively correlated with performance for both level of difficulty;

• The magnitude of the correlation between IIV and Mean is increased when computed with coefficient iCV instead of iSD;

• When correlation > .80, iCV and Mean almost appear as a unique source of variance.

• Computerized task (adapted from Daneman & Carpenter, 1980; Delaloye et al., 2007)

• Span level determination (with ascending procedure)

• 20 items at Span level (N) & 20 Supra-span items (N+1)

• Results

^:

• Span Level (N) :

• Mean x iSD : R = -.69, p<.01

• Mean x iCV : R = -.87, p<.01

• Supra-span Level (N+1) :

• Mean x iSD : R = -.09, n.s

• Mean x iCV : R = -.72, p<.01

• Accuracy measures : Maximum Variance ( ²) can be described as a quadratic function of the Mean

• Values of iCV (delimited in red in the figure) will tend to decrease when the mean increase

• Uncorrelated data (in blue, iSD) demonstrate strong linear dependency after division by the mean (in red, iCV);

• In this example, variability and performance now share about 87% of variance;

• To illustrate this phenomenon, analyses were conducted on a Visuospatial Working Memory (WM) task and in an additional way on a classical verbal WM task (the Reading Span Test).

 Same effect : Mean x iCV always stronger than iSD x Mean

 When iSD x Mean is close to 0, iCV and Mean still share >50% of variance

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8

0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Coefficient Value

Average proportion of cells correctly reccalled

DIFFICULTY : SPAN LEVEL (N)

iSD iCV

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8

0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Coefficient Value

Average proportion of cells correctly recalled

DIFFICULTY : SUPRA-SPAN LEVEL (N+1)

iSD iCV