Searching for replicable associations between cortical thickness and psychometric variables in healthy adults.

(1)

Shahrzad Kharabian Masouleh

1

_{, Simon B. Eickhoff}

1,2

_{, Eliana Nicolaisen-Sobesky}

2

_{, Sarah Genon}

1,2

1_{Institute of Neuroscience and Medicine (INM-7: Brain and Behaviour), Research Centre Jülich, Jülich, Germany} 2_{Institute of Systems Neuroscience, Heinrich Heine University Düsseldorf, Düsseldorf, Germany;}

s.kharabian@fz-juelich.de@fz-juelich.de

Searching for replicable associations between cortical

thickness and psychometric variables in healthy adults:

empirical facts.

Participants:

HCP2_{: 420 healthy young adults (50% female, age: 28}

3.7 ).

Behavioral data:

- Age for validation.

- 34 standard tests in the categories of Personality, emotion and cognition (cognitive flexibility, episodic and working memory. Composite scores, assessing crystallised and fluid intelligence, early childhood and general measure of cognitive functioning). In-scanner tasks : two-back working memory, language task as well as relational processing task.

Structural data: CT estimates derived using FreeSurfer

v5.3-HCP pipeline, registered to fsaverage surface and smoothed using a gaussian kernel of 15 mm FWHM.

Statistical analysis:

Replicability of whole brain exploratory SBB:

Association between each behavioral score and CT is assessed by fitting a linear model at each vertex, controlling for confounding effects of age, sex, education.

Results

using general linear model in PALM3 _{with 1000}

permutations. Inference is made at cluster-level, using TFCE (p < 0.05).

• This procedure is applied on 100 randomly generated subsamples, of same size (e.g. 50% of the original cohort) and binary maps of significant clusters are aggregated to identify rate of spatial overlap of significant findings for each behavioral score.

Replicability of ROI-based SBB:

For every subsample, an independent matched-sample is generated from the main cohort. Partial-correlation of behavioral score and average CT in the significant clusters are compared between the original and matched-subsamples. Replicated effects are defined based on three criteria:

• Same direction in the original and replication sample. • Same direction + Significant (p<0.05)

• Bayes factor4 _(BF

10) ≥ 3

(H0: absence of SBB; H1: presence of SBB in the same direction as original effect.)

• Similar to our findings with GMV, for most of the tested behavioral measures, we did not find any significant association with CT in more than 90% of exploratory analyses.

• ROI-based analysis: Low rate of “significant” replication; Lack of clear association between original and replication effect sizes; Over-estimation of effects size derived from

exploratory analysis in small samples. This questions usefulness of small pilot studies with to define the expected effect size and create and estimation of sample size5_.

• Our empirical assessments show that irrespective of the chosen morphological marker, robust associations between brain structure and normal variation in behavior are very unlikely.

• Importantly, composite scores of cognitive functioning, as a measure of general intelligence, did not show robust associations with CT.

• Considering the publication bias6_{, these results are alarming and demonstrates that} _harking7 _{(hypothesizing after the results are known) in such context can result in serious}

misleading conclusions about the true neurobiological associations

• Thus reported SBB associations within small to medium samples of healthy individuals should be interpreted with caution, before their replication is demonstrated in independent datasets.

Introduction

Acknowledgments: This study was supported by the Deutsche Forschungsgemeinschaft (DFG, GE 2835/1-1, EI 816/11-1), the National Institute of Mental Health (R01-MH074457), the Helmholtz Portfolio Theme "Supercomputing and Modeling for the Human Brain"

and and the European Union’s Horizon 2020 Research and Innovation Programme under Grant Agreement 785907 (HBP SGA2).

References: [1] Kharabian Masouleh, 2019. [2] http://www.humanconnectome.org/ [3] https://fsl.fmrib.ox.ac.uk/fsl/fslwiki/PALM. [4] Boekel, et, al., 2015. [5] Button et. al., 2013. [6] Dwan et. al., 2008. [7] Kerr et. al., 1998.

Methods

Conclusions

Fig1. Frequency of spatial overlap (density plots and aggregate maps) of significant findings from exploratory analysis over 100 random subsamples, calculated for three different sample sizes (X-axis). Here the top five behavioral scores with the highest frequency of overlapping results for all tested sample sizes are demonstrated.

Exploratory whole-brain SBB 100 1 80 60 40 20 % O verl ap

Fig2. Upper row: Summary of ROI-based SBB-replication (% of ROIs) using three different criteria: Inner layer: “direction”. Middle layer: direction + Significant. Outer layer: BF₁₀. Lower row: Original versus replication effects sizes for all ROIs from 100 splits; (replication defined using “direction” only) and size of each point is proportional to the power of replication. Original and replication samples have equal size(n ≈ 208).

ROI-based SBB re pl ica te d no t r ep lica te d In ne r

not replicated [direction] replicated [direction]

Mi

ddl

e

not replicated [direction + p<0.05] replicated [direction + p<0.05]

Ou

te

r Moderate evidence for H1_BF

10 ≥ 3

Anecdotal evidence for H1

1 ≤ BF10< 3

Moderate evidence for H0

BF10 < 1/3

1/3 ≤ BF10 < 1

Colors in each layer

Age (negative associations) 0 20 40 60 80 100 % O ve rl ap Agreeableness (negative associations)

Delay discounting (AUC-40K)

(positive associations)

Relational task (ACC%)

(negative associations) _{(negative associations)}Openness _{(positive associations)}Conscientiousness

70% Discovery/ 30% Test 50% Discovery/ 50% Test 30% Discovery/ 70% Test

Agreeableness Delay discounting ( AUC-40K ) Age

Inner replicated [sign]

not replicated [sign]

M

iddle _{replicated [sign + p<0.05]}not replicated [sign + p<0.05]

O

ut

er

Moderate-to-strong evidence for H1 Anecdotal evidence for H1

Moderate-to-strong evidence for H0 Colors in each layer

0 1/3 1 3 inf BF01 Openness 17.4 3.4 8.5 6.9 15.4 99.7 84.3 63.5 10.9 5.6 1.9 74.9 91.8 99.3 7.5 0.7 0.7 0.7 6.0 10.6 79.4 91.9 100 7.5 1.9_0.6 8.1 Conscientiousness 50.5 72.3 92.1 7.9 19.8 9.9 7.9 7.9 7.9 21.8 2.0 46.5 74.7 89.1 10.9 10.9 10.9 4.5 14.4 8.4 1.5 28.2 2.1 13.7 _13.7 29.5 21.1 2.1 2.1 2.1 68.4 97.9 47.3 43.6 83.6 65.5 34.4 76.6 57.8 18.8 6.2 3.1 9.4 23.4 23.4 23.4 23.4 16.7 29.2 8.3 4.2 8.3 4.2 4.2 4.2 58.3 66.7 95.8 10.9 7.3 16.4 16.4 16.4 18.2 21.8 42.1 48.7 51.3 6.6 48.7 48.7 48.7 1.3 1.3 2.6 _53.4 60.4 _67.4 7.0 32.6 32.6 32.6 7.0 4.7 2.3 62.5 66.7 66.7 4.2 33.3 33.3 33.3 43.3 83.3 63.3 16.7 16.7 16.7 20.0 20.0 10.0 6.7 3.3 15.5 15.5 39.4 8.4 7.0 7.0 7.0 11.3 93.0 42.3 53.6 51.0 66.7 88.2 15.7 11.8 11.8 11.8 11.8 21.5 5.8 3.9 0.3 0.3 0.3 7.4 66.7 66.7 66.7 33.3 33.3 25.9 50.0 55.6 55.6 44.4 44.4 44.4 5.6 40.6 40.6 40.6 53.1 59.4 59.4 6.2 Relational task (ACC% )

M

O

ut

er

0 1/3 1 3 inf BF01 Openness 17.4 3.4 8.5 6.9 15.4 99.7 84.3 63.5 10.9 5.6 1.9 74.9 91.8 99.3 7.5 0.7 0.7 0.7 6.0 10.6 79.4 91.9 100 7.5 1.9 0.6 8.1 Conscientiousness 50.5 72.3 92.1 7.9 19.8 9.9 7.9 7.9 7.9 21.8 2.0 46.5 74.7 89.1 10.9 10.9 10.9 4.5 14.4 8.4 1.5 28.2 2.1 13.7 _13.7 29.5 21.1 2.1 2.1 2.1 68.4 97.9 47.3 43.6 83.6 65.5 34.4 76.6 57.8 18.8 6.2 3.1 9.4 23.4 23.4 23.4 23.4 16.7 29.2 8.3 4.2 8.3 4.2 4.2 4.2 58.3 66.7 95.8 10.9 7.3 16.4 16.4 16.4 18.2 21.8 42.1 48.7 51.3 6.6 48.7 48.7 48.7 1.3 1.3 2.6 _53.4 60.4 _67.4 7.0 32.6 32.6 32.6 7.0 4.7 2.3 62.5 66.7 66.7 4.2 33.3 33.3 33.3 43.3 83.3 63.3 16.7 16.7 16.7 20.0 20.0 10.0 6.7 3.3 15.5 15.5 39.4 8.4 7.0 7.0 7.0 11.3 93.0 42.3 53.6 51.0 66.7 88.2 15.7 11.8 11.8 11.8 11.8 21.5 5.8 3.9 0.3 0.3 0.3 7.4 66.7 66.7 66.7 33.3 33.3 25.9 50.0 55.6 55.6 44.4 44.4 44.4 5.6 40.6 40.6 40.6 53.1 59.4 59.4 6.2 Relational task (ACC% )

M

O

ut

er

M

O

ut

er

0 1/3 1 3 inf BF01 Openness 17.4 3.4 8.5 6.9 15.4 99.7 84.3 63.5 10.9 5.6 1.9 74.9 91.8 99.3 7.5 0.7 0.7 0.7 6.0 10.6 79.4 91.9 100 7.5 1.9_0.6 8.1 Conscientiousness 50.5 72.3 92.1 7.9 19.8 9.9 7.9 7.9 7.9 21.8 2.0 46.5 74.7 89.1 10.9 10.9 10.9 4.5 14.4 8.4 1.5 28.2 2.1 13.7 _13.7 29.5 21.1 2.1 2.1 2.1 68.4 97.9 47.3 43.6 83.6 65.5 34.4 76.6 57.8 18.8 6.2 3.1 9.4 23.4 23.4 23.4 23.4 16.7 29.2 8.3 4.2 8.3 4.2 4.2 4.2 58.3 66.7 95.8 10.9 7.3 16.4 16.4 16.4 18.2 21.8 42.1 48.7 51.3 6.6 48.7 48.7 48.7 1.3 1.3 2.6 _53.4 60.4 _67.4 7.0 32.6 32.6 32.6 7.0 4.7 2.3 62.5 66.7 66.7 4.2 33.3 33.3 33.3 43.3 83.3 63.3 16.7 16.7 16.7 20.0 20.0 10.0 6.7 3.3 15.5 15.5 39.4 8.4 7.0 7.0 7.0 11.3 93.0 42.3 53.6 51.0 66.7 88.2 15.7 11.8 11.8 11.8 11.8 21.5 5.8 3.9 0.3 0.3 0.3 7.4 66.7 66.7 66.7 33.3 33.3 25.9 50.0 55.6 55.6 44.4 44.4 44.4 5.6 40.6 40.6 40.6 53.1 59.4 59.4 6.2 Relational task (ACC% )

M

O

ut

er

M

O

ut

er

0 1/3 1 3 inf BF01 Openness 17.4 3.4 8.5 6.9 15.4 99.7 84.3 63.5 10.9 5.6 1.9 74.9 91.8 99.3 7.5 0.7 0.7 0.7 6.0 10.6 79.4 91.9 100 7.5 1.9_0.6 8.1 Conscientiousness 50.5 72.3 92.1 7.9 19.8 9.9 7.9 7.9 7.9 21.8 2.0 46.5 74.7 89.1 10.9 10.9 10.9 4.5 14.4 8.4 1.5 28.2 2.1 13.7 _13.7 29.5 21.1 2.1 2.1 2.1 68.4 97.9 47.3 43.6 83.6 65.5 34.4 76.6 57.8 18.8 6.2 3.1 9.4 23.4 23.4 23.4 23.4 16.7 29.2 8.3 4.2 8.3 4.2 4.2 4.2 58.3 66.7 95.8 10.9 7.3 16.4 16.4 16.4 18.2 21.8 42.1 48.7 51.3 6.6 48.7 48.7 48.7 1.3 1.3 2.6 _53.4 60.4 _67.4 7.0 32.6 32.6 32.6 7.0 4.7 2.3 62.5 66.7 66.7 4.2 33.3 33.3 33.3 43.3 83.3 63.3 16.7 16.7 16.7 20.0 20.0 10.0 6.7 3.3 15.5 15.5 39.4 8.4 7.0 7.0 7.0 11.3 93.0 42.3 53.6 51.0 66.7 88.2 15.7 11.8 11.8 11.8 11.8 21.5 5.8 3.9 0.3 0.3 0.3 7.4 66.7 66.7 66.7 33.3 33.3 25.9 50.0 55.6 55.6 44.4 44.4 44.4 5.6 40.6 40.6 40.6 53.1 59.4 59.4 6.2 Relational task (ACC% ) R ep licat io n ef fect s

Original effects Original effects Original effects Original effects Original effects Original effects

Age Delay discounting (AUC-40K) Relational task (ACC%) Agreeableness Openness Conscientiousness

• Search for underlying neuronal structural correlates of variability in behavior (SBB) has a long history and its impact goes beyond the research community.

• Previously, we demonstrated that

correlations between inter-individual

variations in gray matter volume (GMV) and normal variation in behavior are predominantly spurious and could not be robustly evidenced within moderately sized

healthy samples1_.

• However, GMV is shown to be an impure marker of gray matter structural changes,

which may explain the spurious

correlations.

• Here we use estimates of cortical thickness (CT) as marker of structural variations an

assess empirical replication rates of