Appendix B: Results of the Ad-hoc Robust Task

Appendix B

Results of the Core Tracks Ad-hoc Robust Tasks

Prepared by:

Giorgio Maria Di Nunzio and Nicola Ferro

{dinunzio, ferro}@dei.unipd.it

Department of Information Engineering University of Padua

Italy

Introduction

1.

2.

3.

- - - -

- - -

- - -

- - -

- - - - - - -

The following pages contain the results and graphs for all the experiments that have been officially submitted to the CLEF 2006 campaign for the Ad-Hoc Robust track.

This document is divided in three main parts:

List of submitted experiments Track overview results and graphs Individual experiment results and graphs 1. List of Submitted Experiments

This section gives a listing of all experiments and their characteristics:

2. Track Overview Results and Graphs

For each track/task graphs and tables are shown in order to compare the experiments.

The graphs and tables contain the following information:

Mandatory experiments title + description (TD) of at most top five participants Interpolated recall vs precision averages plot

Average precision comparison to median plot All experiments

Average precision box plot Average precision Tukey t-test plot

Mandatory experiments title + description (TD) of at most top five participants Document cutoff levels (DCL) vs precision at DCL plot

R-Precision comparison to median plot All experiments

R-Precision box plot R-Precision Tukey t-test plot

A table with descriptive statistics of performance figures for each topic 3. Individual Experiment Results and Graphs

This section provides the individual results for each official experiment. For each experiment the following tables and graphs are shown:

Overall statistics and information

Interpolated recall vs precision averages plot Average precision statistics and box plot Average precision comparison to median plot Document cutoff levels vs precision at DCL plot R-Precision statistics and box plot

R-Precision comparison to median plot

Results for CLEF 2006 Ad-Hoc Robust Tracks

Participant: the name of the participant who submitted the experiment.

Country: country of the participant.

Identifier: unique identifier for each experiment.

Task: track/task to which the experiment belongs.

Topic language: language of the topics used to create the experiment (ISO identifiers for language).

Topic fields: identifies the parts of the topics used to create the experiment (T = title, D = Description, N = Narrative).

Query constr.: identifies how the query has been constructed from topic fields (manual/automatic).

Pool: specifies if experiment was used for relevance assessment pooling.

List of Submitted Experiments

Lang. Fields Construction

colesir Spain CoLesIRdeTst AH-ROBUST-MONO-DE-CLEF2006 de TD AUTOMATIC no

daedalus Spain deFSdeR2S AH-ROBUST-MONO-DE-CLEF2006 de TD AUTOMATIC no

daedalus Spain deFSdeR3S AH-ROBUST-MONO-DE-CLEF2006 de TDN AUTOMATIC no

daedalus Spain dex021deRFW3FS3 AH-ROBUST-MONO-DE-CLEF2006 de TDN AUTOMATIC no

daedalus Spain dex011deRFW3FS3 AH-ROBUST-MONO-DE-CLEF2006 de TDN AUTOMATIC no

hummingbird Canada humDE06Rtde AH-ROBUST-MONO-DE-CLEF2006 de TD AUTOMATIC no

hummingbird Canada humDE06Rtd AH-ROBUST-MONO-DE-CLEF2006 de TD AUTOMATIC no

colesir Spain CoLesIRenTst AH-ROBUST-MONO-EN-CLEF2006 en TD AUTOMATIC no

daedalus Spain enFSenR2S AH-ROBUST-MONO-EN-CLEF2006 en TD AUTOMATIC no

daedalus Spain enFSenR3S AH-ROBUST-MONO-EN-CLEF2006 en TDN AUTOMATIC no

daedalus Spain enFSenR5S AH-ROBUST-MONO-EN-CLEF2006 en TDN AUTOMATIC no

daedalus Spain enyenRFS3456 AH-ROBUST-MONO-EN-CLEF2006 en TDN AUTOMATIC no

dcu Ireland dcunarreng1005 AH-ROBUST-MONO-EN-CLEF2006 en TDN AUTOMATIC no

dcu Ireland dcudesceng12075 AH-ROBUST-MONO-EN-CLEF2006 en TD AUTOMATIC no

hildesheim Germany uhienmo2 AH-ROBUST-MONO-EN-CLEF2006 en TD AUTOMATIC no

hildesheim Germany uhienmo1 AH-ROBUST-MONO-EN-CLEF2006 en TD AUTOMATIC no

hummingbird Canada humEN06Rtd AH-ROBUST-MONO-EN-CLEF2006 en TD AUTOMATIC no

hummingbird Canada humEN06Rtde AH-ROBUST-MONO-EN-CLEF2006 en TD AUTOMATIC no

reina Spain reinaENttest AH-ROBUST-MONO-EN-CLEF2006 en T AUTOMATIC no

reina Spain reinaENtdtest AH-ROBUST-MONO-EN-CLEF2006 en TD AUTOMATIC no

colesir Spain CoLesIResTst AH-ROBUST-MONO-ES-CLEF2006 es TD AUTOMATIC no

daedalus Spain esx011esRFW3FS3 AH-ROBUST-MONO-ES-CLEF2006 es TDN AUTOMATIC no

daedalus Spain esFSesR2S AH-ROBUST-MONO-ES-CLEF2006 es TD AUTOMATIC no

daedalus Spain esx021esRFW3FS3 AH-ROBUST-MONO-ES-CLEF2006 es TDN AUTOMATIC no

daedalus Spain esFSesR3S AH-ROBUST-MONO-ES-CLEF2006 es TDN AUTOMATIC no

dcu Ireland dcu1500narrsp1005.qy1 AH-ROBUST-MONO-ES-CLEF2006 es TDN AUTOMATIC no

dcu Ireland dcudescsp12075 AH-ROBUST-MONO-ES-CLEF2006 es TD AUTOMATIC no

hummingbird Canada humES06Rtde AH-ROBUST-MONO-ES-CLEF2006 es TD AUTOMATIC no

hummingbird Canada humES06Rtd AH-ROBUST-MONO-ES-CLEF2006 es TD AUTOMATIC no

reina Spain reinaEStdtest AH-ROBUST-MONO-ES-CLEF2006 es TD AUTOMATIC no

reina Spain reinaESttest AH-ROBUST-MONO-ES-CLEF2006 es T AUTOMATIC no

colesir Spain CoLesIRfrTst AH-ROBUST-MONO-FR-CLEF2006 fr TD AUTOMATIC no

daedalus Spain frFSfrR2S AH-ROBUST-MONO-FR-CLEF2006 fr TD AUTOMATIC no

daedalus Spain fryfrRFS3456 AH-ROBUST-MONO-FR-CLEF2006 fr TDN AUTOMATIC no

daedalus Spain frx011frRFW3FS3 AH-ROBUST-MONO-FR-CLEF2006 fr TDN AUTOMATIC no

daedalus Spain frFSfrR3S AH-ROBUST-MONO-FR-CLEF2006 fr TDN AUTOMATIC no

dcu Ireland dcudescfr12075 AH-ROBUST-MONO-FR-CLEF2006 fr TD AUTOMATIC no

dcu Ireland dcudescfr12075 AH-ROBUST-MONO-FR-CLEF2006 fr TD AUTOMATIC no

dcu Ireland dcunarrfr1005 AH-ROBUST-MONO-FR-CLEF2006 fr TDN AUTOMATIC no

hildesheim Germany uhifrmo2 AH-ROBUST-MONO-FR-CLEF2006 fr TD AUTOMATIC no

hildesheim Germany uhifrmo1 AH-ROBUST-MONO-FR-CLEF2006 fr TD AUTOMATIC no

hildesheim Germany uhifrmo1 AH-ROBUST-MONO-FR-CLEF2006 fr TD AUTOMATIC no

hummingbird Canada humFR06Rtde AH-ROBUST-MONO-FR-CLEF2006 fr TD AUTOMATIC no

hummingbird Canada humFR06Rtd AH-ROBUST-MONO-FR-CLEF2006 fr TD AUTOMATIC no

reina Spain reinaFRtdtest AH-ROBUST-MONO-FR-CLEF2006 fr TD AUTOMATIC no

reina Spain reinaFRttest AH-ROBUST-MONO-FR-CLEF2006 fr T AUTOMATIC no

unine Switzerland UniNEfrr1 AH-ROBUST-MONO-FR-CLEF2006 fr TD AUTOMATIC no

unine Switzerland UniNEfrr3 AH-ROBUST-MONO-FR-CLEF2006 fr TD AUTOMATIC no

unine Switzerland UniNEfrr2 AH-ROBUST-MONO-FR-CLEF2006 fr T AUTOMATIC no

colesir Spain CoLesIRitTst AH-ROBUST-MONO-IT-CLEF2006 it TD AUTOMATIC no

daedalus Spain itFSitR4S AH-ROBUST-MONO-IT-CLEF2006 it TDN AUTOMATIC no

daedalus Spain ityitRFS3456 AH-ROBUST-MONO-IT-CLEF2006 it TDN AUTOMATIC no

daedalus Spain itFSitR2S AH-ROBUST-MONO-IT-CLEF2006 it TD AUTOMATIC no

daedalus Spain itFSitR6S AH-ROBUST-MONO-IT-CLEF2006 it TDN AUTOMATIC no

dcu Ireland dcu1500narrit1005.qy1 AH-ROBUST-MONO-IT-CLEF2006 it TDN AUTOMATIC no

dcu Ireland dcudescit1005 AH-ROBUST-MONO-IT-CLEF2006 it TD AUTOMATIC no

hummingbird Canada humIT06Rtd AH-ROBUST-MONO-IT-CLEF2006 it TD AUTOMATIC no

hummingbird Canada humIT06Rtde AH-ROBUST-MONO-IT-CLEF2006 it TD AUTOMATIC no

reina Spain reinaITttest AH-ROBUST-MONO-IT-CLEF2006 it T AUTOMATIC no

reina Spain reinaITtdtest AH-ROBUST-MONO-IT-CLEF2006 it TD AUTOMATIC no

colesir Spain CoLesIRnlTst AH-ROBUST-MONO-NL-CLEF2006 nl TD AUTOMATIC no

daedalus Spain nlFSnlR4S AH-ROBUST-MONO-NL-CLEF2006 nl TDN AUTOMATIC no

daedalus Spain nlynlRFS3456 AH-ROBUST-MONO-NL-CLEF2006 nl TDN AUTOMATIC no

daedalus Spain nlx011nlRFW4FS4 AH-ROBUST-MONO-NL-CLEF2006 nl TDN AUTOMATIC no

daedalus Spain nlFSnlR2S AH-ROBUST-MONO-NL-CLEF2006 nl TD AUTOMATIC no

hummingbird Canada humNL06Rtde AH-ROBUST-MONO-NL-CLEF2006 nl TD AUTOMATIC no

hummingbird Canada humNL06Rtd AH-ROBUST-MONO-NL-CLEF2006 nl TD AUTOMATIC no

colesir Spain CoLesIRmultTst AH-ROBUST-MULTI-CLEF2006 en TD AUTOMATIC no

daedalus Spain ml4XRSFSen4S AH-ROBUST-MULTI-CLEF2006 en TDN AUTOMATIC no

daedalus Spain ml5XRSFSen4S AH-ROBUST-MULTI-CLEF2006 en TDN AUTOMATIC no

daedalus Spain ml6XRSFSen4S AH-ROBUST-MULTI-CLEF2006 en TDN AUTOMATIC no

daedalus Spain mlRSFSen2S AH-ROBUST-MULTI-CLEF2006 en TD AUTOMATIC no

jaen Spain ujamlblr AH-ROBUST-MULTI-CLEF2006 en TD AUTOMATIC no

jaen Spain ujamlrsv2 AH-ROBUST-MULTI-CLEF2006 en TD AUTOMATIC no

jaen Spain ujamllr AH-ROBUST-MULTI-CLEF2006 en TD AUTOMATIC no

reina Spain reinaES2mtdtest AH-ROBUST-MULTI-CLEF2006 es TD AUTOMATIC no

reina Spain reinaES2mttest AH-ROBUST-MULTI-CLEF2006 es T AUTOMATIC no

colesir Spain CoLesIRendeTst AH-ROBUST-BILI-X2DE-CLEF2006 en TD AUTOMATIC no

daedalus Spain deFSdeRSen2S AH-ROBUST-BILI-X2DE-CLEF2006 en TD AUTOMATIC no

daedalus Spain deFSdeRSen4S AH-ROBUST-BILI-X2DE-CLEF2006 en TDN AUTOMATIC no

daedalus Spain deFSdeRSen3S AH-ROBUST-BILI-X2DE-CLEF2006 en TDN AUTOMATIC no

daedalus Spain deydeRSenFS3456 AH-ROBUST-BILI-X2DE-CLEF2006 en TDN AUTOMATIC no

daedalus Spain esx011esRLitFW3FS3 AH-ROBUST-BILI-X2ES-CLEF2006 it TDN AUTOMATIC no

daedalus Spain esFSesRLit2S AH-ROBUST-BILI-X2ES-CLEF2006 it TD AUTOMATIC no

daedalus Spain esFSesRLit3S AH-ROBUST-BILI-X2ES-CLEF2006 it TDN AUTOMATIC no

daedalus Spain esx021esRLitFW3FS3 AH-ROBUST-BILI-X2ES-CLEF2006 it TDN AUTOMATIC no

dcu Ireland dcuitqynarrsp1005 AH-ROBUST-BILI-X2ES-CLEF2006 it TDN AUTOMATIC no

dcu Ireland dcuitqydescsp12075 AH-ROBUST-BILI-X2ES-CLEF2006 it TD AUTOMATIC no

reina Spain reinaIT2EStdtest AH-ROBUST-BILI-X2ES-CLEF2006 it TD AUTOMATIC no

reina Spain reinaIT2ESttest AH-ROBUST-BILI-X2ES-CLEF2006 it T AUTOMATIC no

daedalus Spain nlFSnlRLfr2S AH-ROBUST-BILI-X2NL-CLEF2006 fr TD AUTOMATIC no

daedalus Spain nlFSnlRLfr6S AH-ROBUST-BILI-X2NL-CLEF2006 fr TDN AUTOMATIC no

daedalus Spain nlx021nlRLfrFW4FS6 AH-ROBUST-BILI-X2NL-CLEF2006 fr TDN AUTOMATIC no daedalus Spain nlx011nlRLfrFW4FS6 AH-ROBUST-BILI-X2NL-CLEF2006 fr TDN AUTOMATIC no

Track Overview Results and Graphs

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Interpolated Recall

Average Precision

Ad−Hoc Robust Monolingual German track Top 5 Participants − Interpolated Recall vs Average Precision hummingbird [humDE06Rtde; MAP 48.30%; Not Pooled]

colesir [CoLesIRdeTst; MAP 37.21%; Not Pooled]

daedalus [deFSdeR2S; MAP 34.06%; Not Pooled]

041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085

−1

−0.5 0 0.5 1

Topic Identifier

Difference

Ad−Hoc Robust Monolingual German track Top 5 Participants − Comparison to Median Mean Average Precision by Topic (Topics 041 to 085)

hummingbird [humDE06Rtde; MAP 48.30%; Not Pooled]

colesir [CoLesIRdeTst; MAP 37.21%; Not Pooled]

daedalus [deFSdeR2S; MAP 34.06%; Not Pooled]

086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130

−1

−0.5 0 0.5 1

Topic Identifier

Difference

Ad−Hoc Robust Monolingual German track Top 5 Participants − Comparison to Median Mean Average Precision by Topic (Topics 086 to 130)

hummingbird [humDE06Rtde; MAP 48.30%; Not Pooled]

colesir [CoLesIRdeTst; MAP 37.21%; Not Pooled]

daedalus [deFSdeR2S; MAP 34.06%; Not Pooled]

131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165

−1

−0.5 0 0.5 1

Topic Identifier

Difference

Ad−Hoc Robust Monolingual German track Top 5 Participants − Comparison to Median Mean Average Precision by Topic (Topics 131 to 165)

hummingbird [humDE06Rtde; MAP 48.30%; Not Pooled]

colesir [CoLesIRdeTst; MAP 37.21%; Not Pooled]

daedalus [deFSdeR2S; MAP 34.06%; Not Pooled]

−0.5 0 0.5 1

Difference

Ad−Hoc Robust Monolingual German track Top 5 Participants − Comparison to Median Mean Average Precision by Topic (Topics 166 to 200)

hummingbird [humDE06Rtde; MAP 48.30%; Not Pooled]

colesir [CoLesIRdeTst; MAP 37.21%; Not Pooled]

daedalus [deFSdeR2S; MAP 34.06%; Not Pooled]

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

deFSdeR2S [MAP 34.06%; Not Pooled]

CoLesIRdeTst [MAP 37.21%; Not Pooled]

deFSdeR3S [MAP 38.03%; Not Pooled]

dex021deRFW3FS3 [MAP 38.31%; Not Pooled]

dex011deRFW3FS3 [MAP 38.63%; Not Pooled]

humDE06Rtd [MAP 42.45%; Not Pooled]

humDE06Rtde [MAP 48.30%; Not Pooled]

Mean Average Precision

Experiments

Ad−Hoc Robust Monolingual German track − Box Plot of the Topics

0.55 0.6 0.65 0.7 0.75 0.8 deFSdeR2S

CoLesIRdeTst deFSdeR3S dex021deRFW3FS3 dex011deRFW3FS3 humDE06Rtd humDE06Rtde

Ad−Hoc Robust Monolingual German track − Tukey T test with "top group" highlighted

arcsin(sqrt(Mean average precision))

Experiments

5 10 15 20 30 100 200 500 1000 0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Retrieved Documents (logarithmic scale)

R−Precision

Ad−Hoc Robust Monolingual German track Top 5 Participants − Retrieved documents vs Precision hummingbird [humDE06Rtde; R−Prec 46.34%; Not Pooled]

colesir [CoLesIRdeTst; R−Prec 36.49%; Not Pooled]

daedalus [deFSdeR2S; R−Prec 34.80%; Not Pooled]

041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085

−1

−0.5 0 0.5 1

Topic Identifier

Difference

Ad−Hoc Robust Monolingual German track Top 5 Participants − Comparison to Median R−Precision by Topic (Topics 041 to 085)

hummingbird [humDE06Rtde; R−Prec 46.34%; Not Pooled]

colesir [CoLesIRdeTst; R−Prec 36.49%; Not Pooled]

daedalus [deFSdeR2S; R−Prec 34.80%; Not Pooled]

086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130

−1

−0.5 0 0.5 1

Topic Identifier

Difference

Ad−Hoc Robust Monolingual German track Top 5 Participants − Comparison to Median R−Precision by Topic (Topics 086 to 130)

hummingbird [humDE06Rtde; R−Prec 46.34%; Not Pooled]

colesir [CoLesIRdeTst; R−Prec 36.49%; Not Pooled]

daedalus [deFSdeR2S; R−Prec 34.80%; Not Pooled]

131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165

−1

−0.5 0 0.5 1

Topic Identifier

Difference

Ad−Hoc Robust Monolingual German track Top 5 Participants − Comparison to Median R−Precision by Topic (Topics 131 to 165)

hummingbird [humDE06Rtde; R−Prec 46.34%; Not Pooled]

colesir [CoLesIRdeTst; R−Prec 36.49%; Not Pooled]

daedalus [deFSdeR2S; R−Prec 34.80%; Not Pooled]

166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200

−1

−0.5 0 0.5 1

Topic Identifier

Difference

Ad−Hoc Robust Monolingual German track Top 5 Participants − Comparison to Median R−Precision by Topic (Topics 166 to 200)

hummingbird [humDE06Rtde; R−Prec 46.34%; Not Pooled]

colesir [CoLesIRdeTst; R−Prec 36.49%; Not Pooled]

daedalus [deFSdeR2S; R−Prec 34.80%; Not Pooled]

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

deFSdeR2S [R−Prec 34.80%; Not Pooled]

CoLesIRdeTst [R−Prec 36.49%; Not Pooled]

deFSdeR3S [R−Prec 37.76%; Not Pooled]

dex021deRFW3FS3 [R−Prec 38.15%; Not Pooled]

dex011deRFW3FS3 [R−Prec 38.76%; Not Pooled]

humDE06Rtd [R−Prec 41.68%; Not Pooled]

humDE06Rtde [R−Prec 46.34%; Not Pooled]

R−Precision

Experiments

Ad−Hoc Robust Monolingual German track − Box Plot of the Topics

0.5 0.55 0.6 0.65 0.7 0.75 0.8 0.85 0.9 deFSdeR2S

CoLesIRdeTst deFSdeR3S dex021deRFW3FS3 dex011deRFW3FS3 humDE06Rtd humDE06Rtde

Ad−Hoc Robust Monolingual German track − Tukey T test with "top group" highlighted

arcsin(sqrt(R Precision))

Experiments

Average Precision R-Precision

Topic Minimum 1st Q. Median 3rd Q. Maximum Mean Std Minimum 1st Q. Median 3rd Q. Maximum Mean Std 041 0.6119 0.6119 0.6392 0.7928 0.8497 0.6971 0.1026 0.5000 0.5000 0.5714 0.7321 0.7500 0.6071 0.1166 042 0.7170 0.7170 0.7248 0.7355 0.7423 0.7262 0.0105 0.6792 0.6804 0.6934 0.6934 0.6934 0.6880 0.0069 043 0.0239 0.0368 0.0368 0.3310 0.4167 0.1488 0.1836 0.0000 0.0000 0.0000 0.0000 0.5000 0.0714 0.1890 044 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 045 0.5134 0.5497 0.6660 0.6756 0.6756 0.6179 0.0716 0.5377 0.5684 0.6226 0.6415 0.6415 0.6038 0.0432 046 0.0492 0.0520 0.0866 0.1383 0.1383 0.0940 0.0432 0.0000 0.0000 0.0000 0.2000 0.2000 0.0857 0.1069 047 0.0674 0.1205 0.1277 0.1342 0.1589 0.1234 0.0278 0.0417 0.1094 0.1250 0.1875 0.1875 0.1369 0.0549 048 0.0805 0.1093 0.1093 0.1895 0.2142 0.1374 0.0529 0.1455 0.1818 0.1818 0.2318 0.2909 0.2052 0.0478 049 0.1405 0.1567 0.1936 0.2371 0.2912 0.2014 0.0534 0.1111 0.1111 0.1667 0.2083 0.2222 0.1587 0.0500 060 0.1931 0.1989 0.2204 0.3034 0.3471 0.2537 0.0625 0.2807 0.2807 0.2982 0.4254 0.5439 0.3584 0.1027 061 0.0385 0.0385 0.0455 0.3483 0.4612 0.1760 0.1841 0.0233 0.0233 0.0233 0.4477 0.5814 0.2126 0.2477 062 0.4278 0.4718 0.5187 0.5522 0.5621 0.5068 0.0495 0.3333 0.4444 0.4444 0.6667 0.6667 0.5238 0.1393 063 0.4154 0.4729 0.5422 0.6704 0.7915 0.5709 0.1355 0.3421 0.4934 0.5526 0.5921 0.6842 0.5376 0.1073 064 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.0000 065 0.0553 0.0719 0.1086 0.1193 0.1327 0.0990 0.0292 0.1538 0.1692 0.2000 0.2154 0.2154 0.1912 0.0264 066 0.7684 0.7736 0.7736 0.8336 0.8415 0.7984 0.0333 0.6667 0.6667 0.6863 0.7010 0.7059 0.6835 0.0176 067 0.1578 0.1630 0.2006 0.3604 0.6823 0.2858 0.1952 0.1800 0.2600 0.2600 0.3850 0.6600 0.3314 0.1616 068 0.1586 0.2085 0.2872 0.3144 0.3297 0.2605 0.0660 0.2045 0.3239 0.3864 0.3864 0.4091 0.3474 0.0703 069 0.0754 0.0754 0.0828 0.1257 0.1346 0.0983 0.0270 0.0000 0.0000 0.1429 0.2500 0.2857 0.1224 0.1285 080 0.0707 0.4341 0.4834 0.5020 0.5077 0.4213 0.1574 0.1167 0.4896 0.5500 0.5667 0.5667 0.4821 0.1645 081 0.3859 0.5184 0.5280 0.6194 0.6450 0.5481 0.0893 0.4200 0.4800 0.5800 0.5800 0.5800 0.5286 0.0672 082 0.3365 0.5521 0.5521 0.6782 0.8460 0.5946 0.1562 0.2963 0.5926 0.5926 0.7037 0.7778 0.6085 0.1555 083 0.1620 0.1784 0.1825 0.2081 0.2658 0.1955 0.0350 0.1111 0.1111 0.1111 0.3056 0.3333 0.1905 0.1057 084 0.0022 0.0022 0.1000 0.1661 0.2167 0.0874 0.0903 0.0000 0.0000 0.2000 0.2000 0.2000 0.1143 0.1069 085 0.0725 0.1645 0.2417 0.3969 0.3969 0.2635 0.1342 0.0405 0.2230 0.2703 0.3919 0.3919 0.2780 0.1293 086 0.3530 0.4205 0.4457 0.4811 0.4824 0.4423 0.0470 0.4414 0.4685 0.4685 0.4842 0.4865 0.4710 0.0154 087 0.2292 0.2872 0.3249 0.3451 0.3708 0.3154 0.0480 0.2353 0.3088 0.3529 0.3529 0.3529 0.3277 0.0463 088 0.5302 0.5869 0.6077 0.6110 0.6410 0.5981 0.0348 0.5517 0.5517 0.5517 0.5517 0.5690 0.5542 0.0065 089 0.5092 0.5134 0.5209 0.6142 0.6554 0.5552 0.0643 0.4966 0.5310 0.5310 0.6431 0.6759 0.5734 0.0705 090 0.1004 0.1147 0.1569 0.1786 0.2216 0.1539 0.0427 0.1250 0.1406 0.1875 0.1875 0.3125 0.1875 0.0625 091 0.1020 0.1133 0.1237 0.1919 0.3210 0.1597 0.0799 0.1562 0.1562 0.1562 0.1797 0.3750 0.1920 0.0815 092 0.1654 0.1729 0.1999 0.1999 0.2349 0.1935 0.0235 0.1765 0.1765 0.2353 0.2353 0.2941 0.2185 0.0445 093 0.7219 0.8344 0.8672 0.8949 0.8949 0.8493 0.0621 0.7222 0.7500 0.7500 0.7986 0.8056 0.7659 0.0315 094 0.7555 0.7648 0.7799 0.7854 0.7985 0.7770 0.0148 0.7188 0.7500 0.7500 0.8125 0.8125 0.7723 0.0392 095 0.0171 0.0421 0.1214 0.1214 0.1294 0.0876 0.0473 0.0455 0.0773 0.1455 0.1659 0.2455 0.1364 0.0688 096 0.1941 0.2067 0.2067 0.2330 0.3245 0.2279 0.0447 0.2222 0.2222 0.2593 0.2963 0.4444 0.2804 0.0796 097 0.3134 0.3628 0.5065 0.6579 0.6579 0.5164 0.1538 0.3333 0.3833 0.5333 0.5333 0.6000 0.4857 0.1069 098 0.3035 0.3688 0.3948 0.4263 0.4368 0.3881 0.0459 0.4400 0.4800 0.4800 0.5200 0.5200 0.4914 0.0302 099 0.1350 0.2300 0.2655 0.2751 0.5475 0.2819 0.1271 0.2584 0.2978 0.3820 0.3820 0.5393 0.3676 0.0912 110 0.4242 0.4564 0.4564 0.5131 0.5639 0.4822 0.0476 0.5417 0.5417 0.5417 0.5833 0.6250 0.5655 0.0328 111 0.0042 0.0077 0.0077 0.0499 0.1671 0.0390 0.0598 0.0000 0.0000 0.0000 0.0938 0.1250 0.0357 0.0610 112 0.7102 0.7602 0.7602 0.8229 0.8729 0.7845 0.0540 0.6429 0.7143 0.7143 0.7143 0.8571 0.7245 0.0643 113 0.0336 0.0337 0.0881 0.1278 0.1538 0.0821 0.0511 0.0800 0.0800 0.1200 0.1600 0.2400 0.1314 0.0598 114 0.0946 0.1393 0.2368 0.2521 0.2607 0.2006 0.0673 0.1071 0.1964 0.2500 0.2768 0.2857 0.2296 0.0647 115 0.1332 0.1434 0.1639 0.1916 0.2682 0.1765 0.0460 0.1667 0.2500 0.2500 0.2812 0.3750 0.2619 0.0623 116 0.4683 0.4848 0.5688 0.5783 0.5903 0.5364 0.0530 0.4516 0.4758 0.5484 0.5484 0.5806 0.5253 0.0517 117 0.1785 0.1839 0.1918 0.2237 0.4553 0.2314 0.1005 0.2222 0.2333 0.3111 0.3111 0.5333 0.3111 0.1058 118 0.0632 0.1875 0.2448 0.2646 0.2683 0.2114 0.0733 0.0000 0.2222 0.2222 0.4167 0.4444 0.2698 0.1553 119 0.6202 0.6206 0.6297 0.7161 0.7250 0.6619 0.0498 0.6667 0.6667 0.6771 0.7318 0.7396 0.6949 0.0339 130 0.5621 0.6112 0.7276 0.7843 0.7843 0.6952 0.0970 0.6000 0.6000 0.6667 0.6667 0.7333 0.6476 0.0504

Average Precision R-Precision

Topic Minimum 1st Q. Median 3rd Q. Maximum Mean Std Minimum 1st Q. Median 3rd Q. Maximum Mean Std 131 0.0047 0.0062 0.0093 0.0218 0.0283 0.0132 0.0098 0.0152 0.0303 0.0455 0.0606 0.0606 0.0433 0.0184 132 0.7359 0.7359 0.7907 0.7975 0.8465 0.7766 0.0425 0.6071 0.6071 0.7143 0.7411 0.7857 0.6837 0.0756 133 0.5730 0.5863 0.6338 0.6338 0.6563 0.6180 0.0318 0.5909 0.6227 0.6455 0.6455 0.6636 0.6351 0.0237 134 0.8452 0.8607 0.8839 0.9235 0.9571 0.8920 0.0411 0.7143 0.7143 0.7143 0.8214 0.8571 0.7551 0.0697 135 0.0630 0.1126 0.1161 0.3416 0.6049 0.2404 0.1959 0.1609 0.2184 0.2184 0.4167 0.5862 0.3136 0.1537 136 0.4581 0.4880 0.5118 0.5668 0.5959 0.5208 0.0509 0.4286 0.4286 0.4286 0.5357 0.5714 0.4694 0.0697 137 0.0000 0.0000 0.0000 0.0716 0.1429 0.0393 0.0557 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 138 0.4094 0.4668 0.6590 0.7109 0.7197 0.5944 0.1320 0.3548 0.4113 0.6452 0.7419 0.7419 0.5853 0.1731 139 0.0106 0.0106 0.0153 0.1305 0.1610 0.0625 0.0677 0.0357 0.0357 0.0357 0.1071 0.1071 0.0663 0.0382 140 0.0799 0.0832 0.1369 0.3537 0.4962 0.2181 0.1676 0.1852 0.1852 0.2716 0.4198 0.5679 0.3157 0.1498 141 0.7500 0.7500 0.7500 0.8520 0.9705 0.8097 0.0849 0.6250 0.7500 0.7500 0.7500 0.8750 0.7500 0.0722 142 0.3251 0.3398 0.3398 0.8916 0.8984 0.5750 0.2980 0.4000 0.4000 0.4000 0.8115 0.8462 0.5802 0.2252 143 0.5072 0.5608 0.5768 0.6218 0.6941 0.5931 0.0592 0.5873 0.5873 0.5873 0.5992 0.6667 0.6009 0.0296 144 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 145 0.0833 0.1812 0.3281 0.4349 0.7874 0.3472 0.2322 0.1000 0.2500 0.4000 0.4750 0.6000 0.3714 0.1704 146 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 147 0.0297 0.0308 0.0319 0.2085 0.3384 0.1106 0.1314 0.0345 0.0345 0.0690 0.2241 0.3793 0.1281 0.1404 148 0.0941 0.1141 0.1345 0.1416 0.1666 0.1303 0.0235 0.0833 0.1042 0.2500 0.2500 0.2500 0.1905 0.0793 149 0.5387 0.8176 0.8468 0.8648 0.9338 0.8131 0.1269 0.5000 0.7708 0.8333 0.8333 0.9167 0.7857 0.1349 160 0.1000 0.1571 0.2500 0.2500 0.2500 0.2061 0.0619 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 161 0.0068 0.0160 0.0714 0.8333 1.0000 0.3491 0.4589 0.0000 0.0000 0.0000 0.7500 1.0000 0.2857 0.4880 162 0.3220 0.5773 0.7497 0.8033 0.8921 0.6863 0.1971 0.4167 0.5833 0.7083 0.7708 0.7917 0.6667 0.1382 163 0.1284 0.5627 0.5864 0.6165 0.6920 0.5373 0.1856 0.1702 0.5106 0.5745 0.5904 0.6596 0.5198 0.1620 164 0.0706 0.0937 0.0973 0.2839 0.3965 0.1745 0.1327 0.1389 0.1389 0.1667 0.3333 0.4444 0.2262 0.1317 165 0.4091 0.4091 0.4483 0.4633 0.6500 0.4645 0.0855 0.2500 0.2500 0.2500 0.2500 0.5000 0.2857 0.0945 166 0.0130 0.0170 0.0170 0.0292 0.0476 0.0238 0.0121 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 167 0.5738 0.6266 0.6750 0.6821 0.8042 0.6711 0.0720 0.5000 0.5000 0.5000 0.5000 0.7500 0.5357 0.0945 168 0.0190 0.0238 0.0238 0.0654 0.1512 0.0518 0.0477 0.0000 0.0588 0.0588 0.1029 0.2353 0.0840 0.0748 169 0.0038 0.0628 0.1389 0.1632 0.1632 0.1085 0.0644 0.0000 0.1000 0.2000 0.2000 0.2000 0.1429 0.0787 170 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 171 0.0271 0.0596 0.0596 0.3628 0.5888 0.2135 0.2180 0.0000 0.0000 0.0000 0.3438 0.5000 0.1607 0.2130 172 0.0000 0.0000 0.0000 0.8333 1.0000 0.3333 0.4714 0.0000 0.0000 0.0000 0.7500 1.0000 0.2857 0.4880 173 0.9184 0.9672 0.9733 0.9790 0.9809 0.9665 0.0219 0.8000 0.8250 0.9000 0.9000 0.9000 0.8714 0.0488 174 0.2760 0.6322 0.6322 0.8692 0.8936 0.6846 0.2172 0.3056 0.6389 0.6389 0.7986 0.8333 0.6627 0.1789 175 0.5282 0.5282 0.6036 0.8001 0.8742 0.6610 0.1492 0.5455 0.5455 0.5455 0.6364 0.8182 0.6104 0.1012 176 0.9703 0.9753 0.9753 0.9820 0.9877 0.9777 0.0060 0.8919 0.9054 0.9054 0.9189 0.9324 0.9112 0.0132 177 0.0000 0.0000 0.0007 0.2074 0.2881 0.0905 0.1258 0.0000 0.0000 0.0000 0.2500 0.2857 0.1020 0.1359 178 0.1027 0.1027 0.2404 0.4007 0.4377 0.2513 0.1523 0.2381 0.2381 0.3810 0.4286 0.4286 0.3401 0.0969 179 0.5368 0.6265 0.6265 0.6762 0.8069 0.6501 0.0824 0.5306 0.5510 0.5510 0.6378 0.7551 0.5977 0.0805 190 0.0708 0.1284 0.5493 0.5726 0.5726 0.3739 0.2421 0.1754 0.2281 0.5614 0.5789 0.5789 0.4185 0.1955 191 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 192 0.3487 0.4851 0.4851 0.7007 0.8497 0.5676 0.1724 0.4615 0.5000 0.5000 0.6154 0.7308 0.5604 0.0964 193 0.5460 0.6167 0.6296 0.6346 0.6458 0.6185 0.0335 0.5294 0.5368 0.5588 0.5809 0.6176 0.5630 0.0314 194 0.8333 0.8333 0.8333 1.0000 1.0000 0.9048 0.0891 0.5000 0.5000 0.5000 1.0000 1.0000 0.7143 0.2673 195 0.2174 0.2351 0.2467 0.4384 0.5345 0.3322 0.1288 0.2414 0.2414 0.2759 0.4569 0.5862 0.3498 0.1385 196 0.8733 0.9479 0.9583 0.9583 0.9762 0.9468 0.0337 0.8333 0.8333 0.8333 0.8333 0.8333 0.8333 0.0000 197 0.5766 0.5793 0.5856 0.6534 0.7321 0.6228 0.0586 0.5820 0.5820 0.5820 0.6373 0.6885 0.6101 0.0423 198 0.1028 0.1091 0.1195 0.1378 0.1429 0.1228 0.0157 0.0833 0.1667 0.1667 0.1667 0.1667 0.1548 0.0315 199 0.7229 0.7400 0.7831 0.7831 0.8227 0.7713 0.0349 0.7083 0.7344 0.7500 0.7500 0.7917 0.7470 0.0253 200 0.1002 0.2354 0.2405 0.3256 0.6916 0.3003 0.1874 0.1061 0.3068 0.3182 0.3864 0.5909 0.3377 0.1445 ALL 0.3406 0.3742 0.3831 0.4149 0.4830 0.3957 0.0457 0.3480 0.3680 0.3815 0.4095 0.4634 0.3914 0.0381

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Interpolated Recall

Average Precision

Ad−Hoc Robust Monolingual English track Top 5 Participants − Interpolated Recall vs Average Precision hummingbird [humEN06Rtde; MAP 47.63%; Not Pooled]

reina [reinaENtdtest; MAP 43.66%; Not Pooled]

dcu [dcudesceng12075; MAP 43.48%; Not Pooled]

daedalus [enFSenR2S; MAP 39.69%; Not Pooled]

colesir [CoLesIRenTst; MAP 37.64%; Not Pooled]

041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085

−1

−0.5 0 0.5 1

Topic Identifier

Difference

Ad−Hoc Robust Monolingual English track Top 5 Participants − Comparison to Median Mean Average Precision by Topic (Topics 041 to 085)

hummingbird [humEN06Rtde; MAP 47.63%; Not Pooled]

reina [reinaENtdtest; MAP 43.66%; Not Pooled]

dcu [dcudesceng12075; MAP 43.48%; Not Pooled]

daedalus [enFSenR2S; MAP 39.69%; Not Pooled]

colesir [CoLesIRenTst; MAP 37.64%; Not Pooled]

086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130

−1

−0.5 0 0.5 1

Topic Identifier

Difference

Ad−Hoc Robust Monolingual English track Top 5 Participants − Comparison to Median Mean Average Precision by Topic (Topics 086 to 130)

hummingbird [humEN06Rtde; MAP 47.63%; Not Pooled]

reina [reinaENtdtest; MAP 43.66%; Not Pooled]

dcu [dcudesceng12075; MAP 43.48%; Not Pooled]

daedalus [enFSenR2S; MAP 39.69%; Not Pooled]

colesir [CoLesIRenTst; MAP 37.64%; Not Pooled]

131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165

−1

−0.5 0 0.5 1

Topic Identifier

Difference

Ad−Hoc Robust Monolingual English track Top 5 Participants − Comparison to Median Mean Average Precision by Topic (Topics 131 to 165)

hummingbird [humEN06Rtde; MAP 47.63%; Not Pooled]

reina [reinaENtdtest; MAP 43.66%; Not Pooled]

dcu [dcudesceng12075; MAP 43.48%; Not Pooled]

daedalus [enFSenR2S; MAP 39.69%; Not Pooled]

colesir [CoLesIRenTst; MAP 37.64%; Not Pooled]

166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200

−1

−0.5 0 0.5 1

Topic Identifier

Difference

Ad−Hoc Robust Monolingual English track Top 5 Participants − Comparison to Median Mean Average Precision by Topic (Topics 166 to 200)

hummingbird [humEN06Rtde; MAP 47.63%; Not Pooled]

reina [reinaENtdtest; MAP 43.66%; Not Pooled]

dcu [dcudesceng12075; MAP 43.48%; Not Pooled]

daedalus [enFSenR2S; MAP 39.69%; Not Pooled]

colesir [CoLesIRenTst; MAP 37.64%; Not Pooled]

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

uhienmo2 [MAP 7.12%; Not Pooled]

uhienmo1 [MAP 7.98%; Not Pooled]

CoLesIRenTst [MAP 37.64%; Not Pooled]

enFSenR2S [MAP 39.69%; Not Pooled]

reinaENttest [MAP 40.66%; Not Pooled]

humEN06Rtd [MAP 42.80%; Not Pooled]

enFSenR3S [MAP 42.89%; Not Pooled]

enFSenR5S [MAP 43.02%; Not Pooled]

enyenRFS3456 [MAP 43.38%; Not Pooled]

dcudesceng12075 [MAP 43.48%; Not Pooled]

reinaENtdtest [MAP 43.66%; Not Pooled]

humEN06Rtde [MAP 47.63%; Not Pooled]

dcunarreng1005 [MAP 49.12%; Not Pooled]

Mean Average Precision

Experiments

Ad−Hoc Robust Monolingual English track − Box Plot of the Topics

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 uhienmo2

uhienmo1 CoLesIRenTst enFSenR2S reinaENttest enFSenR5S enFSenR3S enyenRFS3456 humEN06Rtd dcudesceng12075 reinaENtdtest humEN06Rtde dcunarreng1005

Ad−Hoc Robust Monolingual English track − Tukey T test with "top group" highlighted

arcsin(sqrt(Mean average precision))

Experiments

5 10 15 20 30 100 200 500 1000 0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Retrieved Documents (logarithmic scale)

R−Precision

Ad−Hoc Robust Monolingual English track Top 5 Participants − Retrieved documents vs Precision hummingbird [humEN06Rtde; R−Prec 44.80%; Not Pooled]

reina [reinaENtdtest; R−Prec 40.74%; Not Pooled]

dcu [dcudesceng12075; R−Prec 40.51%; Not Pooled]

daedalus [enFSenR2S; R−Prec 36.69%; Not Pooled]

colesir [CoLesIRenTst; R−Prec 36.21%; Not Pooled]

041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085

−1

−0.5 0 0.5 1

Topic Identifier

Difference

Ad−Hoc Robust Monolingual English track Top 5 Participants − Comparison to Median R−Precision by Topic (Topics 041 to 085)

hummingbird [humEN06Rtde; R−Prec 44.80%; Not Pooled]

reina [reinaENtdtest; R−Prec 40.74%; Not Pooled]

dcu [dcudesceng12075; R−Prec 40.51%; Not Pooled]

daedalus [enFSenR2S; R−Prec 36.69%; Not Pooled]

colesir [CoLesIRenTst; R−Prec 36.21%; Not Pooled]

086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130

−1

−0.5 0 0.5 1

Topic Identifier

Difference

Ad−Hoc Robust Monolingual English track Top 5 Participants − Comparison to Median R−Precision by Topic (Topics 086 to 130)

hummingbird [humEN06Rtde; R−Prec 44.80%; Not Pooled]

reina [reinaENtdtest; R−Prec 40.74%; Not Pooled]

dcu [dcudesceng12075; R−Prec 40.51%; Not Pooled]

daedalus [enFSenR2S; R−Prec 36.69%; Not Pooled]

colesir [CoLesIRenTst; R−Prec 36.21%; Not Pooled]

131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165

−1

−0.5 0 0.5 1

Topic Identifier

Difference

Ad−Hoc Robust Monolingual English track Top 5 Participants − Comparison to Median R−Precision by Topic (Topics 131 to 165)

hummingbird [humEN06Rtde; R−Prec 44.80%; Not Pooled]

reina [reinaENtdtest; R−Prec 40.74%; Not Pooled]

dcu [dcudesceng12075; R−Prec 40.51%; Not Pooled]

daedalus [enFSenR2S; R−Prec 36.69%; Not Pooled]

colesir [CoLesIRenTst; R−Prec 36.21%; Not Pooled]

−0.5 0 0.5 1

Difference

Ad−Hoc Robust Monolingual English track Top 5 Participants − Comparison to Median R−Precision by Topic (Topics 166 to 200)

hummingbird [humEN06Rtde; R−Prec 44.80%; Not Pooled]

reina [reinaENtdtest; R−Prec 40.74%; Not Pooled]

dcu [dcudesceng12075; R−Prec 40.51%; Not Pooled]

daedalus [enFSenR2S; R−Prec 36.69%; Not Pooled]

colesir [CoLesIRenTst; R−Prec 36.21%; Not Pooled]

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

uhienmo2 [R−Prec 5.78%; Not Pooled]

uhienmo1 [R−Prec 7.29%; Not Pooled]

CoLesIRenTst [R−Prec 36.21%; Not Pooled]

enFSenR2S [R−Prec 36.69%; Not Pooled]

reinaENttest [R−Prec 39.47%; Not Pooled]

enFSenR3S [R−Prec 39.70%; Not Pooled]

humEN06Rtd [R−Prec 39.78%; Not Pooled]

enFSenR5S [R−Prec 40.01%; Not Pooled]

enyenRFS3456 [R−Prec 40.19%; Not Pooled]

dcudesceng12075 [R−Prec 40.51%; Not Pooled]

reinaENtdtest [R−Prec 40.74%; Not Pooled]

humEN06Rtde [R−Prec 44.80%; Not Pooled]

dcunarreng1005 [R−Prec 45.72%; Not Pooled]

R−Precision

Experiments

Ad−Hoc Robust Monolingual English track − Box Plot of the Topics

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 uhienmo2

uhienmo1 enFSenR2S CoLesIRenTst enFSenR3S enyenRFS3456 humEN06Rtd enFSenR5S dcudesceng12075 reinaENttest reinaENtdtest humEN06Rtde dcunarreng1005

Ad−Hoc Robust Monolingual English track − Tukey T test with "top group" highlighted

arcsin(sqrt(R Precision))

Experiments

Average Precision R-Precision

Topic Minimum 1st Q. Median 3rd Q. Maximum Mean Std Minimum 1st Q. Median 3rd Q. Maximum Mean Std 041 0.0000 0.1644 0.1794 0.2168 0.2749 0.1724 0.0855 0.0000 0.1250 0.2500 0.2500 0.3750 0.1923 0.1096 042 0.0000 0.2546 0.2982 0.3555 0.3870 0.2689 0.1298 0.0000 0.3088 0.3529 0.4118 0.4510 0.3122 0.1462 043 0.0000 0.4425 0.6002 0.8193 0.8349 0.5441 0.2916 0.0000 0.4091 0.5455 0.7273 0.8182 0.5105 0.2664 044 0.0000 0.0668 0.5200 0.5255 0.5294 0.3348 0.2296 0.0000 0.0000 0.5000 0.5000 0.5000 0.3462 0.2402 045 0.0000 0.5857 0.6450 0.7025 0.7445 0.5604 0.2542 0.0000 0.5909 0.6364 0.6667 0.6970 0.5361 0.2410 046 0.0000 0.3113 0.3610 0.3819 0.3913 0.3020 0.1391 0.0000 0.3333 0.4444 0.5556 0.5556 0.3761 0.1900 047 0.0000 0.5638 0.6192 0.6614 0.7132 0.5374 0.2430 0.0000 0.5244 0.5854 0.6098 0.6585 0.4972 0.2250 048 0.0000 0.0528 0.1114 0.2479 0.3611 0.1467 0.1187 0.0000 0.0000 0.1515 0.3106 0.3636 0.1515 0.1464 049 0.0000 0.1512 0.1687 0.2286 0.3154 0.1740 0.0946 0.0000 0.1429 0.1429 0.2857 0.3571 0.1813 0.1076 060 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 061 0.0000 0.3811 0.5614 0.6084 0.7200 0.4694 0.2328 0.0000 0.3500 0.4000 0.6000 0.8000 0.4308 0.2428 062 0.0000 1.0000 1.0000 1.0000 1.0000 0.8462 0.3755 0.0000 1.0000 1.0000 1.0000 1.0000 0.8462 0.3755 063 0.0000 0.0032 0.0175 0.0274 0.5000 0.0534 0.1349 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 064 0.0000 0.2255 0.3066 0.5725 0.8354 0.3710 0.2529 0.0000 0.2000 0.3000 0.7000 0.9000 0.4077 0.2957 065 0.0000 0.2726 0.3751 0.4718 0.6051 0.3470 0.1905 0.0000 0.3000 0.3333 0.5333 0.6000 0.3538 0.1970 066 0.0000 0.7014 0.7548 0.7722 0.8184 0.6352 0.2846 0.0000 0.7273 0.7273 0.7273 0.8182 0.6224 0.2774 067 0.0000 0.0250 0.1262 0.2514 0.2557 0.1264 0.1018 0.0000 0.0000 0.0000 0.5000 0.5000 0.1538 0.2402 068 0.0000 0.1804 0.3788 0.4100 0.8016 0.3274 0.2196 0.0000 0.2857 0.4286 0.4643 0.7143 0.3736 0.2224 069 0.0000 0.2376 0.3377 0.4254 0.6135 0.3103 0.1756 0.0000 0.2500 0.3750 0.3750 0.6250 0.2981 0.1657 080 0.0000 0.2674 0.3938 0.4497 0.4709 0.3219 0.1735 0.0000 0.3214 0.3929 0.4286 0.4821 0.3352 0.1675 081 0.0000 0.7543 0.8058 0.8590 0.9308 0.6992 0.3153 0.0000 0.5938 0.7500 0.7812 0.8750 0.6250 0.2976 082 0.0000 0.4122 0.6241 0.6458 0.7985 0.5100 0.2501 0.0000 0.4000 0.6000 0.6000 0.8000 0.4923 0.2431 083 0.0000 0.1279 0.2319 0.2854 0.3649 0.2033 0.1232 0.0000 0.0000 0.2000 0.2000 0.4000 0.1385 0.1261 084 0.0000 0.5751 0.6788 0.7027 0.7416 0.5605 0.2593 0.0000 0.5667 0.6000 0.6667 0.6667 0.5231 0.2386 085 0.0000 0.1880 0.4541 0.4823 0.5948 0.3621 0.2010 0.0000 0.1719 0.4375 0.4688 0.5000 0.3317 0.1860 086 0.0000 0.3222 0.4457 0.5230 0.5542 0.3882 0.1919 0.0000 0.3872 0.4878 0.5244 0.5610 0.4081 0.1923 087 0.0000 0.4672 0.5852 0.6157 0.7753 0.4957 0.2414 0.0000 0.2917 0.5000 0.5000 0.6667 0.3846 0.2056 088 0.0000 0.1349 0.2000 0.6250 1.0000 0.3644 0.3824 0.0000 0.0000 0.0000 0.2500 1.0000 0.2308 0.4385 089 0.0000 0.2429 0.6451 0.7738 1.0000 0.5341 0.3456 0.0000 0.2500 0.5000 0.5625 1.0000 0.4615 0.3203 090 0.0000 0.0509 0.3086 0.3845 0.6236 0.2675 0.1974 0.0000 0.0625 0.2500 0.2708 0.5833 0.2308 0.1805 091 0.0000 0.0773 0.2076 0.5009 0.5922 0.2731 0.2223 0.0000 0.1071 0.2857 0.4286 0.4286 0.2637 0.1735 092 0.0000 0.5872 0.7298 0.7452 0.7972 0.6017 0.2760 0.0000 0.5741 0.6667 0.6667 0.7407 0.5527 0.2510 093 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 094 0.0000 0.6987 0.7739 0.7869 0.8194 0.6454 0.2896 0.0000 0.6458 0.6667 0.6875 0.7500 0.5769 0.2601 095 0.0000 0.2525 0.3171 0.4055 0.4750 0.2890 0.1482 0.0000 0.2630 0.3333 0.4401 0.5208 0.3149 0.1651 096 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 097 0.0000 1.0000 1.0000 1.0000 1.0000 0.8462 0.3755 0.0000 1.0000 1.0000 1.0000 1.0000 0.8462 0.3755 098 0.0000 0.2500 0.5000 1.0000 1.0000 0.5385 0.4147 0.0000 0.0000 0.0000 1.0000 1.0000 0.3846 0.5064 099 0.0000 0.1059 0.3562 0.5062 0.6208 0.3162 0.2094 0.0000 0.2500 0.2857 0.4643 0.5714 0.3187 0.1948 110 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 111 0.0000 0.1421 0.3483 0.3987 0.4869 0.2831 0.1706 0.0000 0.2000 0.4000 0.4125 0.4500 0.3154 0.1625 112 0.0000 0.4418 0.5422 0.5856 0.6257 0.4546 0.2219 0.0000 0.3750 0.5000 0.5250 0.6000 0.4231 0.2048 113 0.0000 0.0026 0.0084 0.0373 0.3244 0.0397 0.0872 0.0000 0.0000 0.0000 0.0000 0.2500 0.0192 0.0693 114 0.0000 0.4356 0.5306 0.6154 0.6408 0.4610 0.2178 0.0000 0.4286 0.5714 0.7143 0.7143 0.4835 0.2442 115 0.0000 0.1473 0.1716 0.1883 0.2181 0.1475 0.0693 0.0000 0.1600 0.2000 0.2400 0.2400 0.1754 0.0825 116 0.0000 0.3248 0.3869 0.4091 0.4984 0.3328 0.1554 0.0000 0.2727 0.3636 0.4545 0.4545 0.3287 0.1641 117 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 118 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 119 0.0000 0.6730 0.7685 0.7864 0.8384 0.6415 0.2927 0.0000 0.5938 0.6786 0.7009 0.7500 0.5714 0.2589 130 0.0000 0.3799 0.4739 0.5688 0.6709 0.4231 0.2092 0.0000 0.4167 0.5079 0.5714 0.6032 0.4371 0.2047

Average Precision R-Precision

Topic Minimum 1st Q. Median 3rd Q. Maximum Mean Std Minimum 1st Q. Median 3rd Q. Maximum Mean Std 131 0.0000 0.3712 0.4469 0.4868 0.6263 0.3943 0.1976 0.0000 0.3250 0.4250 0.4937 0.6500 0.3885 0.1989 132 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 133 0.0000 0.1354 0.2261 0.2747 0.5552 0.2147 0.1514 0.0000 0.1000 0.3000 0.3000 0.5000 0.2462 0.1506 134 0.0000 0.7135 0.7500 0.7513 0.7541 0.6241 0.2799 0.0000 0.7500 0.7500 0.7500 0.7500 0.6346 0.2817 135 0.0000 0.0828 0.1720 0.3379 0.4126 0.2028 0.1481 0.0000 0.1500 0.4000 0.4000 0.4000 0.2769 0.1739 136 0.0000 0.2500 0.3333 0.5000 0.5000 0.3462 0.1827 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 137 0.0000 0.1406 0.5000 0.5000 0.5000 0.3189 0.2141 0.0000 0.0000 0.5000 0.5000 0.5000 0.3077 0.2532 138 0.0000 0.2724 0.4129 0.5655 0.5833 0.3769 0.2028 0.0000 0.3750 0.5000 0.5000 0.5000 0.3942 0.1829 139 0.0000 0.0397 0.0422 0.0434 0.0489 0.0361 0.0162 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 140 0.0000 0.2601 0.3509 0.4544 0.6148 0.3415 0.2028 0.0000 0.3393 0.4429 0.5179 0.6000 0.3912 0.1978 141 0.0032 1.0000 1.0000 1.0000 1.0000 0.8469 0.3738 0.0000 1.0000 1.0000 1.0000 1.0000 0.8462 0.3755 142 0.6250 0.7714 0.8874 0.9778 1.0000 0.8609 0.1323 0.6250 0.8438 0.8750 0.8750 1.0000 0.8462 0.1159 143 0.0155 0.7136 0.8170 0.8517 0.9366 0.6997 0.3103 0.0256 0.6538 0.7436 0.7756 0.8718 0.6391 0.2729 144 0.0556 0.5056 0.5868 0.7056 0.8667 0.5436 0.2323 0.0000 0.3333 0.3333 0.6667 0.6667 0.4615 0.2168 145 0.0000 0.4162 0.5909 0.6685 0.7901 0.5019 0.2761 0.0000 0.4276 0.5789 0.6447 0.7368 0.4858 0.2602 146 0.0035 0.2885 0.5026 0.5082 0.5135 0.3900 0.1976 0.0000 0.3750 0.5000 0.5000 0.5000 0.3846 0.2193 147 0.0989 0.4559 0.7804 0.7946 0.8744 0.6034 0.2647 0.1176 0.4755 0.6667 0.7304 0.7843 0.5581 0.2272 148 0.0166 0.0564 0.0727 0.1966 0.2062 0.1068 0.0721 0.0000 0.1667 0.1667 0.1667 0.1667 0.1410 0.0626 149 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 160 0.0407 0.0554 0.0694 0.1049 0.1667 0.0837 0.0377 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 161 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 162 0.1429 0.3125 0.5000 0.6250 1.0000 0.5156 0.3055 0.0000 0.0000 0.0000 0.2500 1.0000 0.2308 0.4385 163 0.0000 0.8142 0.8645 0.8776 0.8973 0.7258 0.3239 0.0000 0.7172 0.7869 0.7971 0.8197 0.6551 0.2936 164 0.0360 0.0911 0.1278 0.1676 0.2255 0.1281 0.0581 0.0000 0.1019 0.1111 0.1481 0.3333 0.1339 0.0836 165 0.2000 1.0000 1.0000 1.0000 1.0000 0.9000 0.2517 0.0000 1.0000 1.0000 1.0000 1.0000 0.8462 0.3755 166 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 167 0.0000 0.2391 0.3067 0.3142 0.3490 0.2482 0.1150 0.0000 0.2000 0.4000 0.4000 0.4000 0.2923 0.1553 168 0.3465 0.3922 0.4598 0.5070 0.7185 0.4721 0.1022 0.3333 0.3889 0.4444 0.5139 0.6667 0.4615 0.0972 169 0.0000 0.0100 0.0262 0.0582 0.0765 0.0314 0.0250 0.0000 0.0000 0.0000 0.0000 0.1667 0.0256 0.0626 170 0.1250 0.8125 1.0000 1.0000 1.0000 0.8109 0.3603 0.0000 0.7500 1.0000 1.0000 1.0000 0.7692 0.4385 171 0.0000 0.0809 0.1114 0.1265 0.1844 0.0973 0.0518 0.0000 0.0526 0.0526 0.1053 0.1579 0.0769 0.0509 172 0.0199 0.0687 0.1108 0.1480 0.1896 0.1052 0.0553 0.0000 0.0000 0.0000 0.1786 0.2857 0.0879 0.1242 173 0.0000 1.0000 1.0000 1.0000 1.0000 0.8462 0.3755 0.0000 1.0000 1.0000 1.0000 1.0000 0.8462 0.3755 174 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.0000 175 0.0000 0.6787 0.7492 0.8665 0.9071 0.6573 0.3051 0.0000 0.5357 0.7143 0.8571 0.8571 0.6044 0.2983 176 0.0032 0.9143 0.9330 0.9428 0.9471 0.7902 0.3488 0.0256 0.7949 0.8462 0.8526 0.8718 0.7140 0.3066 177 0.0158 0.3153 0.3724 0.4313 0.6169 0.3681 0.1614 0.0000 0.3462 0.3846 0.4808 0.5385 0.3846 0.1538 178 0.0000 0.0263 0.0875 0.1423 0.2722 0.1010 0.0913 0.0000 0.0000 0.0000 0.2500 0.2500 0.1154 0.1297 179 0.0188 0.0258 0.6407 0.8133 0.8954 0.5012 0.3683 0.0000 0.0341 0.5909 0.7386 0.8636 0.4615 0.3396 190 0.0006 0.2130 0.5424 0.7292 0.7531 0.4555 0.2942 0.0000 0.1944 0.4444 0.6667 0.7778 0.4274 0.2827 191 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 192 0.0186 0.1604 0.4730 0.5822 0.7722 0.4027 0.2452 0.0000 0.2917 0.3333 0.5417 0.8333 0.3974 0.2504 193 0.2270 0.3874 0.4637 0.5252 0.5371 0.4373 0.1003 0.1429 0.2857 0.4286 0.4286 0.5714 0.3736 0.1242 194 0.0000 0.0153 0.1667 0.2500 1.0000 0.1949 0.2687 0.0000 0.0000 0.0000 0.0000 1.0000 0.0769 0.2774 195 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 196 0.5000 0.5000 0.5000 1.0000 1.0000 0.6538 0.2402 0.0000 0.0000 0.0000 1.0000 1.0000 0.3077 0.4804 197 0.1193 0.2598 0.7609 0.8422 0.9288 0.5944 0.2950 0.1000 0.2850 0.6800 0.7450 0.8600 0.5415 0.2697 198 0.0096 0.0243 1.0000 1.0000 1.0000 0.5571 0.4985 0.0000 0.0000 1.0000 1.0000 1.0000 0.5385 0.5189 199 0.4867 0.6248 0.6553 0.6804 0.7108 0.6374 0.0622 0.4000 0.5000 0.6000 0.7000 0.7000 0.5846 0.0987 200 0.0162 0.2380 0.3615 0.5911 0.6509 0.3689 0.2199 0.0000 0.3333 0.4444 0.5833 0.6667 0.4274 0.2215 ALL 0.0712 0.3918 0.4289 0.4352 0.4912 0.3762 0.1367 0.0578 0.3657 0.3978 0.4057 0.4572 0.3515 0.1296

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Interpolated Recall

Average Precision

Ad−Hoc Robust Monolingual Spanish track Top 5 Participants − Interpolated Recall vs Average Precision hummingbird [humES06Rtde; MAP 45.66%; Not Pooled]

reina [reinaEStdtest; MAP 44.01%; Not Pooled]

dcu [dcudescsp12075; MAP 42.14%; Not Pooled]

daedalus [esFSesR2S; MAP 40.40%; Not Pooled]

colesir [CoLesIResTst; MAP 40.17%; Not Pooled]

041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085

−1

−0.5 0 0.5 1

Topic Identifier

Difference

Ad−Hoc Robust Monolingual Spanish track Top 5 Participants − Comparison to Median Mean Average Precision by Topic (Topics 041 to 085)

hummingbird [humES06Rtde; MAP 45.66%; Not Pooled]

reina [reinaEStdtest; MAP 44.01%; Not Pooled]

dcu [dcudescsp12075; MAP 42.14%; Not Pooled]

daedalus [esFSesR2S; MAP 40.40%; Not Pooled]

colesir [CoLesIResTst; MAP 40.17%; Not Pooled]

086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130

−1

−0.5 0 0.5 1

Topic Identifier

Difference

Ad−Hoc Robust Monolingual Spanish track Top 5 Participants − Comparison to Median Mean Average Precision by Topic (Topics 086 to 130)

hummingbird [humES06Rtde; MAP 45.66%; Not Pooled]

reina [reinaEStdtest; MAP 44.01%; Not Pooled]

dcu [dcudescsp12075; MAP 42.14%; Not Pooled]

daedalus [esFSesR2S; MAP 40.40%; Not Pooled]

colesir [CoLesIResTst; MAP 40.17%; Not Pooled]

131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165

−1

−0.5 0 0.5 1

Topic Identifier

Difference

Ad−Hoc Robust Monolingual Spanish track Top 5 Participants − Comparison to Median Mean Average Precision by Topic (Topics 131 to 165)

hummingbird [humES06Rtde; MAP 45.66%; Not Pooled]

reina [reinaEStdtest; MAP 44.01%; Not Pooled]

dcu [dcudescsp12075; MAP 42.14%; Not Pooled]

daedalus [esFSesR2S; MAP 40.40%; Not Pooled]

colesir [CoLesIResTst; MAP 40.17%; Not Pooled]

−0.5 0 0.5 1

Difference

Ad−Hoc Robust Monolingual Spanish track Top 5 Participants − Comparison to Median Mean Average Precision by Topic (Topics 166 to 200)

hummingbird [humES06Rtde; MAP 45.66%; Not Pooled]

reina [reinaEStdtest; MAP 44.01%; Not Pooled]

dcu [dcudescsp12075; MAP 42.14%; Not Pooled]

daedalus [esFSesR2S; MAP 40.40%; Not Pooled]

colesir [CoLesIResTst; MAP 40.17%; Not Pooled]

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

reinaESttest [MAP 38.93%; Not Pooled]

CoLesIResTst [MAP 40.17%; Not Pooled]

esFSesR2S [MAP 40.40%; Not Pooled]

humES06Rtd [MAP 42.00%; Not Pooled]

dcudescsp12075 [MAP 42.14%; Not Pooled]

reinaEStdtest [MAP 44.01%; Not Pooled]

esx021esRFW3FS3 [MAP 44.46%; Not Pooled]

esx011esRFW3FS3 [MAP 45.09%; Not Pooled]

humES06Rtde [MAP 45.66%; Not Pooled]

dcu1500narrsp1005.qy1 [MAP 45.81%; Not Pooled]

esFSesR3S [MAP 45.84%; Not Pooled]

Mean Average Precision

Experiments

Ad−Hoc Robust Monolingual Spanish track − Box Plot of the Topics

0.55 0.6 0.65 0.7 0.75 0.8 reinaESttest

CoLesIResTst esFSesR2S humES06Rtd dcudescsp12075 reinaEStdtest esx021esRFW3FS3 humES06Rtde esx011esRFW3FS3 dcu1500narrsp1005.qy1 esFSesR3S

Ad−Hoc Robust Monolingual Spanish track − Tukey T test with "top group" highlighted

arcsin(sqrt(Mean average precision))

Experiments

5 10 15 20 30 100 200 500 1000 0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Retrieved Documents (logarithmic scale)

R−Precision

Ad−Hoc Robust Monolingual Spanish track Top 5 Participants − Retrieved documents vs Precision hummingbird [humES06Rtde; R−Prec 43.99%; Not Pooled]

reina [reinaEStdtest; R−Prec 42.96%; Not Pooled]

dcu [dcudescsp12075; R−Prec 40.58%; Not Pooled]

daedalus [esFSesR2S; R−Prec 40.15%; Not Pooled]

colesir [CoLesIResTst; R−Prec 40.03%; Not Pooled]

041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085

−1

−0.5 0 0.5 1

Topic Identifier

Difference

Ad−Hoc Robust Monolingual Spanish track Top 5 Participants − Comparison to Median R−Precision by Topic (Topics 041 to 085)

hummingbird [humES06Rtde; R−Prec 43.99%; Not Pooled]

reina [reinaEStdtest; R−Prec 42.96%; Not Pooled]

dcu [dcudescsp12075; R−Prec 40.58%; Not Pooled]

daedalus [esFSesR2S; R−Prec 40.15%; Not Pooled]

colesir [CoLesIResTst; R−Prec 40.03%; Not Pooled]

086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130

−1

−0.5 0 0.5 1

Topic Identifier

Difference

Ad−Hoc Robust Monolingual Spanish track Top 5 Participants − Comparison to Median R−Precision by Topic (Topics 086 to 130)

hummingbird [humES06Rtde; R−Prec 43.99%; Not Pooled]

reina [reinaEStdtest; R−Prec 42.96%; Not Pooled]

dcu [dcudescsp12075; R−Prec 40.58%; Not Pooled]

daedalus [esFSesR2S; R−Prec 40.15%; Not Pooled]

colesir [CoLesIResTst; R−Prec 40.03%; Not Pooled]

131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165

−1

−0.5 0 0.5 1

Topic Identifier

Difference

Ad−Hoc Robust Monolingual Spanish track Top 5 Participants − Comparison to Median R−Precision by Topic (Topics 131 to 165)

hummingbird [humES06Rtde; R−Prec 43.99%; Not Pooled]

reina [reinaEStdtest; R−Prec 42.96%; Not Pooled]

dcu [dcudescsp12075; R−Prec 40.58%; Not Pooled]

daedalus [esFSesR2S; R−Prec 40.15%; Not Pooled]

colesir [CoLesIResTst; R−Prec 40.03%; Not Pooled]

166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200

−1

−0.5 0 0.5 1

Topic Identifier

Difference

Ad−Hoc Robust Monolingual Spanish track Top 5 Participants − Comparison to Median R−Precision by Topic (Topics 166 to 200)

hummingbird [humES06Rtde; R−Prec 43.99%; Not Pooled]

reina [reinaEStdtest; R−Prec 42.96%; Not Pooled]

dcu [dcudescsp12075; R−Prec 40.58%; Not Pooled]

daedalus [esFSesR2S; R−Prec 40.15%; Not Pooled]

colesir [CoLesIResTst; R−Prec 40.03%; Not Pooled]

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

reinaESttest [R−Prec 38.82%; Not Pooled]

CoLesIResTst [R−Prec 40.03%; Not Pooled]

esFSesR2S [R−Prec 40.15%; Not Pooled]

dcudescsp12075 [R−Prec 40.58%; Not Pooled]

humES06Rtd [R−Prec 41.41%; Not Pooled]

reinaEStdtest [R−Prec 42.96%; Not Pooled]

dcu1500narrsp1005.qy1 [R−Prec 43.96%; Not Pooled]

humES06Rtde [R−Prec 43.99%; Not Pooled]

esx021esRFW3FS3 [R−Prec 44.20%; Not Pooled]

esx011esRFW3FS3 [R−Prec 44.57%; Not Pooled]

esFSesR3S [R−Prec 44.77%; Not Pooled]

R−Precision

Experiments

Ad−Hoc Robust Monolingual Spanish track − Box Plot of the Topics

0.55 0.6 0.65 0.7 0.75 0.8 reinaESttest

dcudescsp12075 CoLesIResTst esFSesR2S humES06Rtd reinaEStdtest humES06Rtde dcu1500narrsp1005.qy1 esx021esRFW3FS3 esx011esRFW3FS3 esFSesR3S

Ad−Hoc Robust Monolingual Spanish track − Tukey T test with "top group" highlighted

arcsin(sqrt(R Precision))

Experiments

Average Precision R-Precision

Topic Minimum 1st Q. Median 3rd Q. Maximum Mean Std Minimum 1st Q. Median 3rd Q. Maximum Mean Std 041 0.2119 0.3289 0.3587 0.6256 0.7200 0.4352 0.1859 0.3125 0.3750 0.3750 0.5312 0.6250 0.4432 0.1099 042 0.2214 0.5869 0.6385 0.7111 0.7130 0.6067 0.1469 0.3295 0.5757 0.6130 0.6580 0.6705 0.5869 0.1001 043 0.0142 0.1644 0.3497 0.4127 0.4168 0.2794 0.1497 0.0556 0.2222 0.3889 0.3889 0.4444 0.3030 0.1300 044 0.1362 0.2067 0.2440 0.2874 0.3388 0.2436 0.0571 0.1250 0.2063 0.2500 0.2875 0.3750 0.2455 0.0650 045 0.4512 0.5621 0.6076 0.6347 0.6900 0.6019 0.0670 0.4921 0.5707 0.5864 0.6283 0.6911 0.5945 0.0553 046 0.0144 0.0252 0.0287 0.0391 0.1111 0.0377 0.0260 0.0000 0.0000 0.0000 0.0667 0.1333 0.0364 0.0458 047 0.1043 0.1221 0.1486 0.1896 0.2034 0.1529 0.0361 0.1759 0.1875 0.2222 0.2500 0.2500 0.2180 0.0300 048 0.0104 0.0760 0.1065 0.1656 0.1788 0.1151 0.0581 0.0104 0.1068 0.1562 0.1562 0.2292 0.1383 0.0588 049 0.0984 0.1050 0.1163 0.1784 0.2533 0.1474 0.0539 0.1579 0.1579 0.2105 0.3026 0.3684 0.2249 0.0784 060 0.2266 0.3994 0.4879 0.5977 0.7475 0.4851 0.1607 0.3067 0.4367 0.5067 0.5900 0.6800 0.4994 0.1163 061 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 062 0.2067 0.2141 0.2405 0.2633 0.3564 0.2475 0.0429 0.1333 0.1750 0.2000 0.2500 0.4333 0.2273 0.0941 063 0.7162 0.7267 0.8102 0.8584 0.8785 0.7992 0.0636 0.6441 0.6568 0.7627 0.7797 0.8136 0.7319 0.0660 064 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.0000 065 0.1248 0.1615 0.1796 0.2467 0.2536 0.1915 0.0478 0.2297 0.2736 0.3243 0.3716 0.3784 0.3206 0.0510 066 0.8744 0.8931 0.9334 0.9341 0.9415 0.9145 0.0256 0.7778 0.8207 0.8586 0.8687 0.8687 0.8411 0.0341 067 0.1054 0.1137 0.1289 0.2022 0.3570 0.1682 0.0796 0.0612 0.1429 0.1633 0.2959 0.4286 0.2152 0.1198 068 0.0637 0.0742 0.0904 0.1120 0.2473 0.1055 0.0522 0.0000 0.0000 0.0000 0.1111 0.2222 0.0606 0.0911 069 0.5964 0.7843 0.8750 0.8833 0.8958 0.8175 0.1042 0.6667 0.7083 0.8333 0.8333 0.8333 0.7879 0.0778 080 0.3238 0.4122 0.4554 0.4735 0.4738 0.4364 0.0469 0.3714 0.4612 0.4898 0.4990 0.5306 0.4772 0.0427 081 0.6315 0.7640 0.7872 0.8205 0.8619 0.7770 0.0670 0.6552 0.7356 0.7816 0.7931 0.8046 0.7544 0.0545 082 0.3925 0.7348 0.8242 0.8438 0.9823 0.7628 0.1711 0.4211 0.6908 0.7632 0.7829 0.9474 0.7201 0.1364 083 0.4168 0.4657 0.5234 0.5544 0.6094 0.5151 0.0561 0.4444 0.4444 0.4444 0.4444 0.5556 0.4646 0.0449 084 0.1701 0.2233 0.3115 0.3139 0.3783 0.2851 0.0667 0.1250 0.2812 0.3750 0.3750 0.3750 0.3295 0.0843 085 0.7224 0.7505 0.8022 0.8373 0.8389 0.7908 0.0444 0.7009 0.7360 0.7570 0.7570 0.7850 0.7511 0.0234 086 0.2468 0.2690 0.3012 0.3149 0.3207 0.2916 0.0262 0.2500 0.2902 0.3214 0.3750 0.4107 0.3312 0.0488 087 0.2637 0.3319 0.3912 0.5962 0.7650 0.4531 0.1679 0.2955 0.3750 0.4205 0.5881 0.7159 0.4659 0.1356 088 0.2805 0.3063 0.3424 0.4144 0.4956 0.3629 0.0755 0.1429 0.2857 0.4286 0.4286 0.4286 0.3636 0.0982 089 0.4379 0.4421 0.4483 0.4777 0.6824 0.4768 0.0725 0.5000 0.5000 0.5625 0.5625 0.6250 0.5398 0.0421 090 0.0204 0.0527 0.0750 0.1164 0.1257 0.0809 0.0376 0.0667 0.0667 0.0667 0.1333 0.1333 0.0970 0.0348 091 0.0480 0.3211 0.3410 0.3828 0.4404 0.3331 0.1021 0.0753 0.3226 0.3441 0.3763 0.4194 0.3304 0.0906 092 0.1920 0.2449 0.2706 0.2881 0.3494 0.2677 0.0421 0.1860 0.2587 0.3023 0.3256 0.3837 0.2960 0.0603 093 0.1303 0.1769 0.2620 0.2819 0.3491 0.2376 0.0676 0.2857 0.2857 0.2857 0.3571 0.4286 0.3247 0.0491 094 0.7511 0.8031 0.8686 0.8723 0.8904 0.8428 0.0480 0.8182 0.8182 0.8182 0.8523 0.8636 0.8306 0.0212 095 0.0045 0.0548 0.1005 0.1907 0.2649 0.1200 0.0861 0.0343 0.1456 0.2212 0.3014 0.3988 0.2198 0.1081 096 0.0593 0.0926 0.1338 0.1955 0.2279 0.1431 0.0599 0.0215 0.0645 0.2043 0.2796 0.3011 0.1789 0.1112 097 0.3764 0.6586 0.7088 0.7335 0.7731 0.6677 0.1134 0.4800 0.6800 0.7200 0.7200 0.7600 0.6873 0.0776 098 0.3493 0.4589 0.6727 0.8030 0.9000 0.6401 0.1840 0.4000 0.4000 0.6000 0.7500 0.8000 0.5636 0.1748 099 0.0058 0.1463 0.2670 0.2926 0.3892 0.2292 0.1101 0.0357 0.1964 0.3214 0.3571 0.5357 0.2890 0.1292 110 0.0962 0.6131 0.6642 0.6723 0.7401 0.6065 0.1747 0.1429 0.5714 0.5714 0.5714 0.7143 0.5455 0.1402 111 0.0005 0.0028 0.0046 0.0107 0.0221 0.0080 0.0074 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 112 0.1176 0.2779 0.3909 0.4245 0.4772 0.3469 0.1098 0.0000 0.2812 0.3750 0.4688 0.5000 0.3523 0.1460 113 0.0887 0.2207 0.2367 0.2781 0.3446 0.2423 0.0695 0.1139 0.2690 0.2785 0.3323 0.3861 0.2854 0.0756 114 0.0140 0.0552 0.0703 0.4795 0.7827 0.2354 0.2677 0.0000 0.0222 0.0667 0.4667 0.6889 0.2101 0.2558 115 0.1421 0.1669 0.2118 0.2920 0.2982 0.2234 0.0605 0.2105 0.2105 0.2632 0.3158 0.3158 0.2584 0.0497 116 0.1102 0.1457 0.2405 0.2732 0.3746 0.2171 0.0807 0.0000 0.1429 0.1429 0.2500 0.2857 0.1558 0.1001 117 0.0772 0.2087 0.2366 0.2435 0.3043 0.2252 0.0574 0.1250 0.2857 0.3214 0.3705 0.3750 0.3052 0.0760 118 0.2577 0.3287 0.4015 0.4280 0.4496 0.3710 0.0680 0.2000 0.2000 0.4000 0.4000 0.4000 0.3273 0.1009 119 0.6469 0.7111 0.7333 0.7501 0.8063 0.7302 0.0401 0.5619 0.6667 0.6667 0.6976 0.7238 0.6667 0.0463 130 0.6425 0.6873 0.7150 0.7605 0.8421 0.7240 0.0566 0.5385 0.5385 0.6154 0.6731 0.6923 0.6084 0.0639