Appendix B: Results for Core Tracks

(1)

Topic language: language of the topics used to create the experiment (ISO identifiers for language).

Topic fields: identifies the parts of the topics used for query creation (combination of T=title, D=description and N=narrative).

Run Type: type of experiment (automatic/manual).

Pool: specifies if experiment was used for relevance assessment pooling.

The following ten pages compare the top entries for every track/task. Shown are the recall/precision curves for at most five groups. Chosen are the best entries by the respective groups, with the topic fields fixed to title+description (mandatory experiment for every participant) and only automatic experiments used. For the multilingual, bilingual to English, bilingual to Russian and GIRT experiments, the top entries regardless of topic language are presented.

The remainder of the pages is made up by the individual results for every official experiment. Each experiment is presented on one page, with the following details given:

1. A table providing the following information:

• Average precision figures for every individual query. This allows comparison of system performance for single queries, which is important since variation of performance across queries is often very high and can be significant.

• Overall statistics, giving:

- the total number of documents retrieved by the system

- the total number of overall relevant documents in the collection, and - the total number of relevant documents actually found by the system - interpolated precision averages at specific recall levels

- non-interpolated average precision over all queries

- precision numbers after inspecting a specific number of documents - R-precision: precision after the last relevant document was retrieved.

2. Two graphs consisting of:

• a recall/precision graph, providing a plot of the precision values for various recall levels. This is the standard statistic and is the one most commonly reported in the literature.

• a comparison to median performance. For each query, the difference in average precision, when compared to the median performance for the given task, is plotted. This graph gives valuable insight into which type of queries is handled well by different systems.

The results page for a specific experiment can be most quickly located by using the table of contents. This table is sorted by group, task, topic language and topic fields. The individual pages are sorted by task and run tag.

More information on the interpretation of the standard measures used for scoring experiments (average precision, recall levels, precision/recall graphs, etc.) can be found e.g. in:

Martin Braschler, Carol Peters: CLEF 2002 Methodology and Metrics, CLEF 2002 (Carol Peters, Martin

Braschler, Julio Gonzalo, Michael Kluck (Eds.), Advances in Cross-Language Information Retrieval: Results of

the CLEF 2002 Evaluation Campaign. Lecture Notes in Computer Science, Vol. 2785, Springer, 2003.

(2)

CLIPS/IMAG France 2003CLIPSMONFR01 Monolingual FR TN Auto Y

CMU USA cmuI2Scomb Bilingual_ES IT TD Auto

CMU USA cmuI2Scombfb Bilingual_ES IT TD Auto

CMU USA cmuI2Spara Bilingual_ES IT TD Auto

CMU USA cmuI2Sparafb Bilingual_ES IT TD Auto

CMU USA cmuG2Icomb Bilingual_IT DE TD Auto

CMU USA cmuG2Icombfb Bilingual_IT DE TD Auto

CMU USA cmuG2Ipara Bilingual_IT DE TD Auto

CMU USA cmuG2Iparafb Bilingual_IT DE TD Auto Y

CMU USA cmuM4fb Multi-4 EN TD Auto Y

CMU USA cmuM4fbre Multi-4 EN TD Auto

CMU USA cmuM4lowfb Multi-4 EN TD Auto

CMU USA cmuM4lowfbre Multi-4 EN TD Auto

Daedalus Spain spenQTdoc Bilingual_EN ES TD Auto Y

Daedalus Spain spenQTor1 Bilingual_EN ES TD Auto

Daedalus Spain spenQTor3 Bilingual_EN ES TD Auto

Daedalus Spain spenQTor3full Bilingual_EN ES TD Auto

Daedalus Spain frenQTdoc Bilingual_EN FR TD Auto Y

Daedalus Spain frenQTor3full Bilingual_EN FR TDN Auto Y

Daedalus Spain itspQTdoc Bilingual_ES IT TD Auto

Daedalus Spain itspQTor1 Bilingual_ES IT TD Auto

Daedalus Spain itspQTor3 Bilingual_ES IT TD Auto

Daedalus Spain itspQTor3full Bilingual_ES IT TD Auto

Daedalus Spain gegeQdoc Monolingual DE TD Auto Y

Daedalus Spain gegeQor Monolingual DE TD Auto Y

Daedalus Spain gegeQorand Monolingual DE TD Auto

Daedalus Spain gegeQorlem Monolingual DE TD Auto Y

Daedalus Spain enenQdoc Monolingual EN TD Auto

Daedalus Spain enenQor Monolingual EN TD Auto

Daedalus Spain enenQorlem Monolingual EN TD Auto

Daedalus Spain enenQorrf Monolingual EN TD Auto Y

Daedalus Spain spspQdoc Monolingual ES TD Auto

Daedalus Spain spspQor Monolingual ES TD Auto

Daedalus Spain spspQorlem Monolingual ES TD Auto

Daedalus Spain spspQorrf Monolingual ES TD Auto Y

Daedalus Spain frfrQdoc Monolingual FR TD Auto Y

Daedalus Spain frfrQor Monolingual FR TD Auto

Daedalus Spain frfrQorand Monolingual FR TD Auto

Daedalus Spain frfrQorlem Monolingual FR TD Auto

Daedalus Spain spxxQTdoc Multi-4 ES TD Auto

Daedalus Spain spxxQTor3 Multi-4 ES TD Auto

(3)

Fernuni Hagen Germany FUHpTD1 GIRT_DE DE TD Auto Y

Fernuni Hagen Germany FUHpTD4 GIRT_DE DE TD Auto Y

Fernuni Hagen Germany FUHpTD5 GIRT_DE DE TD Auto Y

Fondazione Ugo Bordoni Italy fub03es3 Monolingual ES TD Auto Fondazione Ugo Bordoni Italy fub03es4 Monolingual ES TD Auto Fondazione Ugo Bordoni Italy fub03esAV Monolingual ES TD Auto Fondazione Ugo Bordoni Italy fub03esB Monolingual ES TD Auto Y Fondazione Ugo Bordoni Italy fub03fr3 Monolingual FR TD Auto Fondazione Ugo Bordoni Italy fub03fr4 Monolingual FR TD Auto Fondazione Ugo Bordoni Italy fub03frAV Monolingual FR TD Auto Y Fondazione Ugo Bordoni Italy fub03frB Monolingual FR TD Auto Y Fondazione Ugo Bordoni Italy fub03it3 Monolingual IT TD Auto Fondazione Ugo Bordoni Italy fub03it4 Monolingual IT TD Auto Fondazione Ugo Bordoni Italy fub03itAV Monolingual IT TD Auto Fondazione Ugo Bordoni Italy fub03itB Monolingual IT TD Auto Y

Hummingbird Canada humDE03td Monolingual DE TD Auto Y

Hummingbird Canada humDE03tde Monolingual DE TD Auto

Hummingbird Canada humES03td Monolingual ES TD Auto

Hummingbird Canada humES03tde Monolingual ES TD Auto

Hummingbird Canada humFI03td Monolingual FI TD Auto Y

Hummingbird Canada humFI03tde Monolingual FI TD Auto Y

Hummingbird Canada humFR03td Monolingual FR TD Auto Y

Hummingbird Canada humFR03tde Monolingual FR TD Auto

Hummingbird Canada humIT03td Monolingual IT TD Auto

Hummingbird Canada humIT03tde Monolingual IT TD Auto

Hummingbird Canada humNL03td Monolingual NL TD Auto Y

Hummingbird Canada humNL03tde Monolingual NL TD Auto

Hummingbird Canada humRU03t Monolingual RU T Auto Y

Hummingbird Canada humRU03te Monolingual RU T Auto Y

Hummingbird Canada humRU03tm Monolingual RU T Auto Y

Hummingbird Canada humRU03td Monolingual RU TD Auto Y

Hummingbird Canada humRU03tde Monolingual RU TD Auto Y

Hummingbird Canada humRU03tdm Monolingual RU TD Auto Y

Hummingbird Canada humRU03tdn Monolingual RU TDN Auto Y

Hummingbird Canada humRU03tdne Monolingual RU TDN Auto Y

Hummingbird Canada humRU03tdnm Monolingual RU TDN Auto Y

Hummingbird Canada humSV03td Monolingual SV TD Auto Y

Hummingbird Canada humSV03tde Monolingual SV TD Auto Y

IMS/U Padua Italy IMSBGZ0 Bilingual_EN DE TD Auto

IMS/U Padua Italy IMSBGZ3 Bilingual_EN DE TD Auto Y

IMS/U Padua Italy IMSBSZ0 Bilingual_EN ES TD Auto

(4)

IMS/U Padua Italy IMSMSSP2 Monolingual ES TD Auto

IMS/U Padua Italy IMSMSST1 Monolingual ES TD Auto

IMS/U Padua Italy IMSMFPO Monolingual FR TD Auto

IMS/U Padua Italy IMSMFSP1 Monolingual FR TD Auto

IMS/U Padua Italy IMSMFST1 Monolingual FR TD Auto Y

IMS/U Padua Italy IMSMIPO Monolingual IT TD Auto

IMS/U Padua Italy IMSMISP1 Monolingual IT TD Auto

IMS/U Padua Italy IMSMIST2 Monolingual IT TD Auto Y

IMS/U Padua Italy IMSMDPO Monolingual NL TD Auto

IMS/U Padua Italy IMSMDSP3 Monolingual NL TD Auto Y

IMS/U Padua Italy IMSMDST1 Monolingual NL TD Auto Y

IRST Italy IRSTit2es_1 Bilingual_ES IT TD Auto Y

IRST Italy IRSTit2es_2 Bilingual_ES IT TD Auto

IRST Italy IRSTit2es_3 Bilingual_ES IT TD Auto

IRST Italy IRSTit2es_4 Bilingual_ES IT TD Auto

IRST Italy IRSTde2it_1 Bilingual_IT DE TD Auto Y

IRST Italy IRSTde2it_2 Bilingual_IT DE TD Auto

IRST Italy IRSTde_1 Monolingual DE TD Auto Y

IRST Italy IRSTes_1 Monolingual ES TD Auto

IRST Italy IRSTfr_1 Monolingual FR TD Auto Y

IRST Italy IRSTit_1 Monolingual IT TD Auto Y

IRST Italy IRSTen2xx_1 Multi-4 EN TD Auto Y

IRST Italy IRSTen2xx_2 Multi-4 EN TD Auto

IRST Italy IRSTen2xx_3 Multi-4 EN TD Auto

IRST Italy IRSTen2xx_4 Multi-4 EN TD Auto

JHU/APL USA aplbifidea Bilingual_DE FI TD Auto Y

JHU/APL USA aplbifideb Bilingual_DE FI TD Auto Y

JHU/APL USA aplbiitesa Bilingual_ES IT TD Auto

JHU/APL USA aplbiitesb Bilingual_ES IT TD Auto

JHU/APL USA aplbideita Bilingual_IT DE TD Auto

JHU/APL USA aplbideitb Bilingual_IT DE TD Auto

JHU/APL USA aplbifrnla Bilingual_NL FR TD Auto Y

JHU/APL USA aplbifrnlb Bilingual_NL FR TD Auto Y

JHU/APL USA aplmodea Monolingual DE TD Auto Y

JHU/APL USA aplmodeb Monolingual DE TD Auto

JHU/APL USA aplmoena Monolingual EN

JHU/APL USA aplmoenb Monolingual EN

JHU/APL USA aplmoesa Monolingual ES TD Auto Y

JHU/APL USA aplmoesb Monolingual ES TD Auto

JHU/APL USA aplmofia Monolingual FI TD Auto Y

JHU/APL USA aplmofib Monolingual FI TD Auto Y

(5)

JHU/APL USA aplmuen4a Multi-4 EN TD Auto Y

JHU/APL USA aplmuen4b Multi-4 EN TD Auto

JHU/APL USA aplmuen8a Multi-8 EN TD Auto Y

JHU/APL USA aplmuen8b Multi-8 EN TD Auto

Kermit France/UK xrce_ML4_run2 Multi-4 EN TDN Auto Y

Kermit France/UK kcca300 Multi-4 EN TDN Auto Y

Medialab Netherlands Medialab_b Monolingual NL TD Auto Y

Medialab Netherlands Medialab_geo_1 Monolingual NL TD Auto Medialab Netherlands Medialab_geo_2 Monolingual NL TD Auto Medialab Netherlands Medialab_geo_3 Monolingual NL TD Auto Medialab Netherlands Medialab_geo_4 Monolingual NL TD Auto Medialab Netherlands Medialab_geo_5 Monolingual NL TD Auto Medialab Netherlands Medialab_htr_1 Monolingual NL TD Auto Y Medialab Netherlands Medialab_htr_2 Monolingual NL TD Auto Y Medialab Netherlands Medialab_htr_3 Monolingual NL TD Auto Medialab Netherlands Medialab_htr_4 Monolingual NL TD Auto Medialab Netherlands Medialab_htr_5 Monolingual NL TD Auto

NII Japan NiiDic01 Bilingual_IT DE D Auto Y

NII Japan NiiDic02 Bilingual_IT DE D Auto

NII Japan NiiMt01 Bilingual_IT DE D Auto Y

NTU Taiwan NTUm4Topn Multi-4 EN TD Auto

NTU Taiwan NTUm4TopnLinear Multi-4 EN TD Auto

NTU Taiwan NTUm4TopnTp Multi-4 EN TD Auto Y

NTU Taiwan NTUm4TopnTpCw Multi-4 EN TD Auto

Océ Netherlands oce03DEnoXbm Monolingual DE TD Auto

Océ Netherlands oce03DEnoXpr Monolingual DE TD Auto Y

Océ Netherlands oce03ESnoXbm Monolingual ES TD Auto

Océ Netherlands oce03ESnoXpr Monolingual ES TD Auto Y

Océ Netherlands oce03FInoXbm Monolingual FI TD Auto Y

Océ Netherlands oce03FInoXpr Monolingual FI TD Auto Y

Océ Netherlands oce03FRnoXbm Monolingual FR TD Auto

Océ Netherlands oce03FRnoXpr Monolingual FR TD Auto Y

Océ Netherlands oce03ITnoXbm Monolingual IT TD Auto

Océ Netherlands oce03ITnoXpr Monolingual IT TD Auto Y

Océ Netherlands oce03NLnoXbm Monolingual NL TD Auto Y

Océ Netherlands oce03NLnoXpr Monolingual NL TD Auto Y

Océ Netherlands oce03st01 Monolingual NL TD Auto

Océ Netherlands oce03st02 Monolingual NL TD Auto

Océ Netherlands oce03SEnoXbm Monolingual SV TD Auto Y

Océ Netherlands oce03SEnoXpr Monolingual SV TD Auto Y

Ricoh/USL Japan rdetde03 Monolingual DE TD Auto Y

(6)

SICS Sweden sicsSVtad Monolingual SV TD Auto Y

SICS Sweden sicsSVtadaRR Monolingual SV TD Auto Y

SICS Sweden sicsSVtmd Monolingual SV TD Auto Y

SINAI Group Spain uja03ShortRR Multi-4 EN TD Auto

SINAI Group Spain uja03ShortRRPrf Multi-4 EN TD Auto

SINAI Group Spain uja03ShortRSV2 Multi-4 EN TD Auto

SINAI Group Spain uja03ShortRSV2m Multi-4 EN TD Auto Y

SINAI Group Spain uja03LargeRR Multi-8 EN TD Auto

SINAI Group Spain uja03LargeRRPrf Multi-8 EN TD Auto

SINAI Group Spain uja03LargeRSV2 Multi-8 EN TD Auto Y

SINAI Group Spain uja03LargeRSV2m Multi-8 EN TD Auto

Tagmatica France KWSynonyms Monolingual FR TD Auto

Tagmatica France KWSynonymsSL Monolingual FR TD Auto

Tagmatica France basicKeyWords Monolingual FR TD Auto

Tagmatica France patterns Monolingual FR TD Auto Y

U Alicante Spain IRn-ites -exp Bilingual_ES IT TD Auto

U Alicante Spain IRn-ites -noexp Bilingual_ES IT TD Auto

U Alicante Spain IRn-al-exp-nsp Monolingual DE TD Auto Y

U Alicante Spain IRn-al-exp-s p Monolingual DE TD Auto Y

U Alicante Spain IRn-al-nexp-nsp Monolingual DE TD Auto

U Alicante Spain IRn-al-nexp-s p Monolingual DE TD Auto Y

U Alicante Spain IRn-es -exp Monolingual ES TD Auto Y

U Alicante Spain IRn-es -noexp Monolingual ES TD Auto

U Alicante Spain IRn-fr-exp Monolingual FR TD Auto Y

U Alicante Spain IRn-fr-noexp Monolingual FR TD Auto

U Alicante Spain IRn-it-exp Monolingual IT TD Auto Y

U Alicante Spain IRn-it-noexp Monolingual IT TD Auto

U Alicante Spain IRn-m-exp-nsp Multi-4 EN TD Auto

U Alicante Spain IRn-m-exp-s p Multi-4 EN TD Auto Y

U Alicante Spain IRn-m-nexp-nsp Multi-4 EN TD Auto

U Alicante Spain IRn-m-nexp-sp Multi-4 EN TD Auto

U Alicante Spain IRn-mi-exp-sp Multi-4 EN TD Auto

U Buffalo USA UBmonoDErf1 Monolingual DE TD Auto

U Buffalo USA UBmonoDErf2 Monolingual DE TD Auto Y

U Buffalo USA UBmonoENrf1 Monolingual EN TD Auto

U Buffalo USA UBmonoENrf2 Monolingual EN TD Auto Y

U Buffalo USA UBmonoESrf1 Monolingual ES TD Auto

U Buffalo USA UBmonoESrf2 Monolingual ES TD Auto Y

U Buffalo USA UBmonoFIrf1 Monolingual FI TD Auto Y

U Buffalo USA UBmonoFIrf2 Monolingual FI TD Auto Y

U Buffalo USA UBmonoFRrf1 Monolingual FR TD Auto

(7)

U Buffalo USA UBENmultirf2 Multi-8 EN TD Auto Y

U Buffalo USA UBENmultirf3 Multi-8 EN TD Auto

U Buffalo USA UBESmultishort2 Multi-8 ES T Auto Y

U Buffalo USA UBESmultirf1 Multi-8 ES TD Auto

U Buffalo USA UBESmultirf2 Multi-8 ES TD Auto Y

U Buffalo USA UBESmultirf3 Multi-8 ES TD Auto

U Exeter UK exeitqdsptcpro Bilingual_ES IT TD Auto

U Exeter UK exeitqspdbipro Bilingual_ES IT TD Auto

U Exeter UK exeitqydspbipro Bilingual_ES IT TD Auto

U Exeter UK exeitqydspbisys Bilingual_ES IT TD Auto

U Exeter UK exedeqdittcpro Bilingual_IT DE TD Auto

U Exeter UK exedeqitdbipro Bilingual_IT DE TD Auto Y

U Exeter UK exedeqyditbipro Bilingual_IT DE TD Auto

U Exeter UK exedeqyditbisys Bilingual_IT DE TD Auto

U Exeter UK exedemonopro Monolingual DE TD Auto Y

U Exeter UK exedemonosys Monolingual DE TD Auto Y

U Exeter UK exedetcmono Monolingual DE TD Auto

U Exeter UK exespmonopro Monolingual ES TD Auto

U Exeter UK exespmonosys Monolingual ES TD Auto Y

U Exeter UK exesptcmono Monolingual ES TD Auto

U Exeter UK exefrmonopro Monolingual FR TD Auto

U Exeter UK exefrmonosys Monolingual FR TD Auto Y

U Exeter UK exefrtcmono Monolingual FR TD Auto

U Exeter UK exeitmonopro Monolingual IT TD Auto

U Exeter UK exeitmonosys Monolingual IT TD Auto

U Exeter UK exeittcmono Monolingual IT TD Auto

U Exeter UK exemult4d Multi-4 EN TD Auto

U Exeter UK exemult4p Multi-4 EN TD Auto

U Exeter UK exemult4s Multi-4 EN TD Auto Y

U Exeter UK exemult4u Multi-4 EN TD Auto

U Exeter UK exemulttc Multi-4 EN TD Auto

U Oviedo/AIC Spain ex25x25 Monolingual ES TD Auto

U Oviedo/AIC Spain ex50x50 Monolingual ES TD Auto Y

U Sunderland UK sundb Bilingual_ES IT T Auto

U Sunderland UK sundrc Bilingual_ES IT T Auto Y

U Tampere Finland UTAmul1 Multi-8 EN TD Auto Y

U Tampere Finland UTAmul2 Multi-8 EN TD Auto

U Tampere Finland UTAmul3 Multi-8 EN TD Auto

U Tampere Finland UTAmul4 Multi-8 EN TD Auto

U Tampere Finland UTAmul5 Multi-8 EN TD Auto

U Twente Netherlands ut3 Monolingual NL TD Auto Y

(8)

UC Berkeley USA BKRUBLER2 Bilingual_RU EN TDN Auto Y

UC Berkeley USA BKRUMLEE2 Bilingual_RU EN TDN Auto Y

UC Berkeley USA BKGRMLGG1 GIRT_DE DE TD Auto Y

UC Berkeley USA BKGRMLGG2 GIRT_DE DE TD Auto Y

UC Berkeley USA BKGRBLEG1 GIRT_DE EN TD Auto Y

UC Berkeley USA BKGRBLRG1 GIRT_DE RU TD Auto Y

UC Berkeley USA BKGRBLRG2 GIRT_DE RU TD Auto Y

UC Berkeley USA BKGRBLGE1 GIRT_EN DE TD Auto Y

UC Berkeley USA BKGRMLEE1 GIRT_EN EN TD Auto Y

UC Berkeley USA BKGRMLEE2 GIRT_EN EN TD Auto Y

UC Berkeley USA BKGRBLRE1 GIRT_EN RU TD Auto Y

UC Berkeley USA bkmonode1 Monolingual DE TD Auto Y

UC Berkeley USA bkmonoes1 Monolingual ES TD Auto

UC Berkeley USA bkmonofi1 Monolingual FI TD Auto Y

UC Berkeley USA bkmonofi2 Monolingual FI TD Auto Y

UC Berkeley USA bkmonofr1 Monolingual FR TD Auto Y

UC Berkeley USA bkmonoit1 Monolingual IT TD Auto

UC Berkeley USA bkmononl1 Monolingual NL TD Auto Y

UC Berkeley USA BKRUMLRR1 Monolingual RU TD Auto Y

UC Berkeley USA BKRUMLRR3 Monolingual RU TD Auto Y

UC Berkeley USA BKRUMLRR2 Monolingual RU TDN Auto Y

UC Berkeley USA BKRUMLRR4 Monolingual RU TDN Auto Y

UC Berkeley USA bkmonosv1 Monolingual SV TD Auto Y

UC Berkeley USA bkmonosv2 Monolingual SV TD Auto Y

UC Berkeley USA bkmul4en1 Multi-4 EN TD Auto

UC Berkeley USA bkmul4en2 Multi-4 EN TD Auto

UC Berkeley USA bkmul4en3 Multi-4 EN TD Auto Y

UC Berkeley USA bkmul8en1 Multi-8 EN TD Auto

UC Berkeley USA bkmul8en2 Multi-8 EN TD Auto

UC Berkeley USA bkmul8en3 Multi-8 EN TD Auto Y

Univ. A Coruna/COLE Group Spain coleTDlemZP03 Monolingual ES TD Auto

Univ. A Coruna/COLE Group Spain coleTDNlemSM03 Monolingual ES TDN Auto

Univ. A Coruna/COLE Group Spain coleTDNlemZP03 Monolingual ES TDN Auto

Univ. A Coruna/COLE Group Spain coleTDNpdsSM03 Monolingual ES TDN Auto Y

Univ. Amsterdam/LIT Netherlands UAmsC03ItSp4GiSb Bilingual_ES IT TD Auto

Univ. Amsterdam/LIT Netherlands UAmsC03GeIt4GiSb Bilingual_IT DE TD Auto Y

Univ. Amsterdam/LIT Netherlands UAmsC03FrDu4GiSb Bilingual_NL FR TD Auto Y

Univ. Amsterdam/LIT Netherlands UAmsC03FrDu4Gr Bilingual_NL FR TD Auto Y

Univ. Amsterdam/LIT Netherlands UAmsC03FrDuSblSS Bilingual_NL FR TD Auto Y

Univ. Amsterdam/LIT Netherlands UAmsC03EnRu4GiSb Bilingual_RU EN TD Auto Y

Univ. Amsterdam/LIT Netherlands UAmsC03EnRu4Gr Bilingual_RU EN TD Auto Y

(9)

Univ. Amsterdam/LIT Netherlands UAmsC03ItIt4GiSb Monolingual IT TD Auto

Univ. Amsterdam/LIT Netherlands UAmsC03DuDu4GiSb Monolingual NL TD Auto Y

Univ. Amsterdam/LIT Netherlands UAmsC03DuDuSS3w Monolingual NL TD Auto

Univ. Amsterdam/LIT Netherlands UAmsC03RuRu4GiSb Monolingual RU TD Auto Y

Univ. Amsterdam/LIT Netherlands UAmsC03RuRu4Gr Monolingual RU TD Auto Y

Univ. Amsterdam/LIT Netherlands UAmsC03RuRuSbl Monolingual RU TD Auto Y

Univ. Amsterdam/LIT Netherlands UAmsC03RuRuWrd Monolingual RU TD Auto Y

Univ. Amsterdam/LIT Netherlands UAmsC03SwSw4GSbO Monolingual SV TD Auto Y

Univ. Amsterdam/LIT Netherlands UAmsC03SwSw4GiSb Monolingual SV TD Auto Y

Univ. Amsterdam/LIT Netherlands UAmsC03EnM44GiSb Multi-4 EN TD Auto Y

Univ. Amsterdam/LIT Netherlands UAmsC03EnM44Gr Multi-4 EN TD Auto

Univ. Amsterdam/LIT Netherlands UAmsC03EnM4SS4G Multi-4 EN TD Auto

Univ. Amsterdam/LIT Netherlands UAmsC03EnM84GiSb Multi-8 EN TD Auto Y

Univ. Amsterdam/LIT Netherlands UAmsC03EnM84Gr Multi-8 EN TD Auto

Univ. Amsterdam/LIT Netherlands UAmsC03EnM84Gr6 Multi-8 EN TD Auto

Univ. Amsterdam/LIT Netherlands UAmsC03EnM8SS4G Multi-8 EN TD Auto

Univ. Amsterdam/LIT Netherlands UAmsC03EnM8SS4G6 Multi-8 EN TD Auto

Universität Hildesheim/AIS Germany UHImnenR1 Monolingual EN TD Auto Y

Universität Hildesheim/AIS Germany UHImnenR2 Monolingual EN TD Auto

Universität Hildesheim/AIS Germany UHImlt4R1 Multi-4 EN TD Auto Y

Universität Hildesheim/AIS Germany UHImlt4R2 Multi-4 EN TD Auto

Université de Neuchâtel Switzerland UniNEde Monolingual DE TD Auto Y

Université de Neuchâtel Switzerland UniNEde2 Monolingual DE TD Auto

Université de Neuchâtel Switzerland UniNEsp Monolingual ES TD Auto

Université de Neuchâtel Switzerland UniNEsp2 Monolingual ES TD Auto

Université de Neuchâtel Switzerland UniNEfi Monolingual FI TD Auto Y

Université de Neuchâtel Switzerland UniNEfi2 Monolingual FI TD Auto Y

Université de Neuchâtel Switzerland UniNEfr Monolingual FR TD Auto Y

Université de Neuchâtel Switzerland UniNEfr2 Monolingual FR TD Auto

Université de Neuchâtel Switzerland UniNEit Monolingual IT TD Auto

Université de Neuchâtel Switzerland UniNEit2 Monolingual IT TD Auto

Université de Neuchâtel Switzerland UniNEnl Monolingual NL TD Auto Y

Université de Neuchâtel Switzerland UniNEnl2 Monolingual NL TD Auto

Université de Neuchâtel Switzerland UniNEru1 Monolingual RU TD Auto Y

Université de Neuchâtel Switzerland UniNEru2 Monolingual RU TD Auto Y

Université de Neuchâtel Switzerland UniNEru Monolingual RU TDN Auto Y

Université de Neuchâtel Switzerland UniNEru3 Monolingual RU TDN Auto Y

Université de Neuchâtel Switzerland UniNEsv Monolingual SV TD Auto Y

Université de Neuchâtel Switzerland UniNEsv2 Monolingual SV TD Auto Y

Université de Neuchâtel Switzerland UniNEms Multi-4 EN TD Manual Y

Université de Neuchâtel Switzerland UniNEms1 Multi-4 EN TD Auto

(10)

(11)

0.0 0.2 0.4 0.6 Recall

0.0

0.2

0.4

0.6 0.8 1.0

Precision

0.3733 UC Berkeley 0.3285 Université de Neuchâtel 0.2884 Univ. Amsterdam/LIT 0.2420 JHU/APL 0.1870 U Tampere

Multilingual−8; Recall−Precision Graph Top performing experiments by the best five groups; TD topic fields; Automatic

(12)

0.0 0.2 0.4 0.6 Recall

0.0

0.2

0.4

0.6 0.8 1.0

Precision

0.4376 U Exeter 0.4260 UC Berkeley 0.3920 Université de Neuchâtel 0.3773 CMU 0.3377 U Alicante

Multilingual−4; Recall−Precision Graph Top performing experiments by the best five groups; TD topic fields; Automatic

(13)

0.0 0.2 0.4 0.6 0.8 1.0 Recall

0.0 0.2 0.4 0.6

Precision

0.0 0.2 0.4 0.6 0.8 1.0

Recall 0.0

0.2 0.4 0.6 0.8 1.0

Precision

0.3688 Daedalus

0.2026 IMS/U Padua

Bilingual X−>EN; Recall−Precision Graph

Top performing experiments by the best five groups; TD topic fields; Automatic

(14)

0.0 0.2 0.4 0.6 0.8 1.0 Recall

0.0 0.2 0.4 0.6

Precision

0.0 0.2 0.4 0.6 0.8 1.0

Recall 0.0

0.2 0.4 0.6 0.8 1.0

Precision

0.4603 JHU/APL

0.4148 U Exeter

0.4117 CMU

0.3859 UC Berkeley

0.3830 Univ. Amsterdam/LIT

Bilingual DE−>IT; Recall−Precision Graph

Top performing experiments by the best five groups; TD topic fields; Automatic

(15)

0.0 0.2 0.4 0.6 0.8 1.0 Recall

0.0 0.2 0.4 0.6

Precision

0.0 0.2 0.4 0.6 0.8 1.0

Recall 0.0

0.2 0.4 0.6 0.8 1.0

Precision

0.2809 UC Berkeley

0.2270 Univ. Amsterdam/LIT

Bilingual X−>RU; Recall−Precision Graph

Top performing experiments by the best five groups; TD topic fields; Automatic

(16)

0.0 0.2 0.4 0.6 0.8 1.0 Recall

0.0 0.2 0.4 0.6

Precision

0.0 0.2 0.4 0.6 0.8 1.0

Recall 0.0

0.2 0.4 0.6 0.8 1.0

Precision

0.5553 Fondazione Ugo Bordoni

0.5380 Université de Neuchâtel

0.5375 IRST

0.5289 Hummingbird

0.5174 Ricoh/USL

Monolingual ES; Recall−Precision Graph

Top performing experiments by the best five groups; TD topic fields; Automatic

(17)

0.0 0.2 0.4 0.6 0.8 1.0 Recall

0.0 0.2 0.4 0.6

Precision

0.0 0.2 0.4 0.6 0.8 1.0

Recall 0.0

0.2 0.4 0.6 0.8 1.0

Precision

0.5450 Université de Neuchâtel

0.5434 Hummingbird

0.5377 Fondazione Ugo Bordoni

0.5339 IRST

0.5323 UC Berkeley

Monolingual FR; Recall−Precision Graph

Top performing experiments by the best five groups; TD topic fields; Automatic

(18)

0.0 0.2 0.4 0.6 0.8 1.0 Recall

0.0 0.2 0.4 0.6

Precision

0.0 0.2 0.4 0.6 0.8 1.0

Recall 0.0

0.2 0.4 0.6 0.8 1.0

Precision

0.5315 Hummingbird

0.5304 UC Berkeley

0.5138 Univ. Amsterdam/LIT

0.5065 Université de Neuchâtel

0.4817 JHU/APL

Monolingual NL; Recall−Precision Graph

Top performing experiments by the best five groups; TD topic fields; Automatic

(19)

0.0 0.2 0.4 0.6 0.8 1.0 Recall

0.0 0.2 0.4 0.6

Precision

0.0 0.2 0.4 0.6 0.8 1.0

Recall 0.0

0.2 0.4 0.6 0.8 1.0

Precision

0.5465 UC Berkeley

0.4903 Université de Neuchâtel

0.4560 JHU/APL

0.4556 Univ. Amsterdam/LIT

0.4363 Hummingbird

Monolingual SV; Recall−Precision Graph

Top performing experiments by the best five groups; TD topic fields; Automatic

(20)

0.0 0.2 0.4 0.6 0.8 1.0 Recall

0.0 0.2 0.4 0.6

Precision

0.0 0.2 0.4 0.6 0.8 1.0

Recall 0.0

0.2 0.4 0.6 0.8 1.0

Precision

0.5192 UC Berkeley

0.1992 ENEA/La Sapienza

GIRT X−>EN; Recall−Precision Graph

Top performing experiments by the best five groups; TD topic fields; Automatic

(21)

JHU/APL, Multilingual-8 English, Auto, TD Average precision (individual queries): Query 141: 0.2332 Query 142: 0.6987 Query 143: 0.4319 Query 144: 0.0663 Query 145: 0.2343 Query 146: 0.0396 Query 147: 0.1049 Query 148: 0.2480 Query 149: 0.0331 Query 150: 0.2002 Query 151: 0.1924 Query 152: 0.2379 Query 153: 0.1018 Query 154: 0.2065 Query 155: 0.0337 Query 156: 0.0294 Query 157: 0.1505 Query 158: 0.4180 Query 159: 0.6023 Query 160: 0.0557 Query 161: 0.0035 Query 162: 0.2950 Query 163: 0.5771 Query 164: 0.0066 Query 165: 0.0816 Query 166: 0.0006 Query 167: 0.3561 Query 168: 0.1885 Query 169: 0.0111 Query 170: 0.4069 Query 171: 0.0673 Query 172: 0.0415 Query 173: 0.6978 Query 174: 0.4804 Query 175: 0.1716 Query 176: 0.7694 Query 177: 0.1162 Query 178: 0.0216 Query 179: 0.3474 Query 180: 0.0084 Query 181: 0.3548 Query 182: 0.3768 Query 183: 0.2411 Query 184: 0.2853 Query 185: 0.0321 Query 186: 0.0000 Query 187: 0.0042 Query 188: 0.2758 Query 189: 0.4421 Query 190: 0.0573 Query 191: 0.0769 Query 192: 0.7788 Query 193: 0.2018 Query 194: 0.0087 Query 195: 0.3593 Query 196: 0.2233 Query 197: 0.3601 Query 198: 0.0517 Query 199: 0.6875 Query 200: 0.5487 Overall statistics (for 60 queries): Total number of documents over all queries Retrieved: 60000 Relevant: 10020 Rel_ret: 5998 Interpolated Recall - Precision Averages: at 0.00 0.6329 at 0.10 0.4373 at 0.20 0.3776 at 0.30 0.3258 at 0.40 0.2937 at 0.50 0.2427 at 0.60 0.1989 at 0.70 0.1521 at 0.80 0.1073 at 0.90 0.0496 at 1.00 0.0087 Avg. prec. (non-interpolated) for all rel. documents 0.2389 Precision: At 5 docs: 0.4300 At 10 docs: 0.3883 At 15 docs: 0.3756 At 20 docs: 0.3683 At 30 docs: 0.3600 At 100 docs: 0.3007 At 200 docs: 0.2426 At 500 docs: 0.1542 At 1000 docs: 0.1000 R-Precision (prec. after R (=num_rel) docs. retrieved): Exact: 0.2789

0.00.20.4 Recall

0.0

0.2

0.4

0.6

0.81.0

Precision

Run aplmuen8a

Recall−Precision Values 141150160170 Topic number

−1.0

−0.50.0

+0.5

+1.0

Difference (average precision)

Run aplmuen8a

Comparison to median by topic

(22)

JHU/APL, Multilingual-8 English, Auto, TD Average precision (individual queries): Query 141: 0.1129 Query 142: 0.7962 Query 143: 0.4088 Query 144: 0.0913 Query 145: 0.3962 Query 146: 0.0507 Query 147: 0.1822 Query 148: 0.1532 Query 149: 0.0626 Query 150: 0.2193 Query 151: 0.2546 Query 152: 0.2129 Query 153: 0.0591 Query 154: 0.2798 Query 155: 0.0581 Query 156: 0.0464 Query 157: 0.1427 Query 158: 0.2874 Query 159: 0.5324 Query 160: 0.0570 Query 161: 0.0068 Query 162: 0.3768 Query 163: 0.6627 Query 164: 0.0039 Query 165: 0.1833 Query 166: 0.0022 Query 167: 0.2238 Query 168: 0.1064 Query 169: 0.0208 Query 170: 0.2356 Query 171: 0.0452 Query 172: 0.0286 Query 173: 0.7540 Query 174: 0.3609 Query 175: 0.5153 Query 176: 0.8333 Query 177: 0.1662 Query 178: 0.0121 Query 179: 0.3436 Query 180: 0.0169 Query 181: 0.3656 Query 182: 0.4185 Query 183: 0.2253 Query 184: 0.2406 Query 185: 0.0386 Query 186: 0.0001 Query 187: 0.0107 Query 188: 0.2314 Query 189: 0.6164 Query 190: 0.0517 Query 191: 0.0909 Query 192: 0.4751 Query 193: 0.2078 Query 194: 0.1115 Query 195: 0.1170 Query 196: 0.4963 Query 197: 0.5233 Query 198: 0.1128 Query 199: 0.6263 Query 200: 0.2578 Overall statistics (for 60 queries): Total number of documents over all queries Retrieved: 60000 Relevant: 10020 Rel_ret: 5899 Interpolated Recall - Precision Averages: at 0.00 0.7140 at 0.10 0.4664 at 0.20 0.3844 at 0.30 0.3312 at 0.40 0.2882 at 0.50 0.2267 at 0.60 0.1913 at 0.70 0.1334 at 0.80 0.0868 at 0.90 0.0465 at 1.00 0.0058 Avg. prec. (non-interpolated) for all rel. documents 0.2420 Precision: At 5 docs: 0.5067 At 10 docs: 0.4467 At 15 docs: 0.4400 At 20 docs: 0.4208 At 30 docs: 0.3900 At 100 docs: 0.2892 At 200 docs: 0.2315 At 500 docs: 0.1508 At 1000 docs: 0.0983 R-Precision (prec. after R (=num_rel) docs. retrieved): Exact: 0.2853

0.00.20.4 Recall

0.0

0.2

0.4

0.6

0.81.0

Precision

Run aplmuen8b

−1.0

−0.50.0

+0.5

+1.0

Difference (average precision)

Run aplmuen8b

(23)

UC Berkeley, Multilingual-8 English, Auto, TD

Average precision (individual queries): Query 141: 0.6954 Query 142: 0.8050 Query 143: 0.4553 Query 144: 0.2653 Query 145: 0.4556 Query 146: 0.2843 Query 147: 0.2262 Query 148: 0.1389 Query 149: 0.6475 Query 150: 0.2368 Query 151: 0.0717 Query 152: 0.3495 Query 153: 0.3724 Query 154: 0.2447 Query 155: 0.0495 Query 156: 0.0373 Query 157: 0.1808 Query 158: 0.3308 Query 159: 0.2540 Query 160: 0.1605 Query 161: 0.0001 Query 162: 0.6094 Query 163: 0.4760 Query 164: 0.0184 Query 165: 0.3728 Query 166: 0.0000 Query 167: 0.3989 Query 168: 0.2833 Query 169: 0.0176 Query 170: 0.4119 Query 171: 0.0803 Query 172: 0.0277 Query 173: 0.5032 Query 174: 0.7000 Query 175: 0.2859 Query 176: 0.8628 Query 177: 0.2920 Query 178: 0.0314 Query 179: 0.2982 Query 180: 0.5795 Query 181: 0.2399 Query 182: 0.5842 Query 183: 0.0688 Query 184: 0.3444 Query 185: 0.0758 Query 186: 0.2151 Query 187: 0.6148 Query 188: 0.4790 Query 189: 0.7112 Query 190: 0.2109 Query 191: 1.0000 Query 192: 0.3985 Query 193: 0.2714 Query 194: 0.0147 Query 195: 0.1899 Query 196: 0.4165 Query 197: 0.5084 Query 198: 0.0298 Query 199: 0.7064 Query 200: 0.3100 Overall statistics (for 60 queries): Total number of documents over all queries Retrieved: 60000 Relevant: 10020 Rel_ret: 6342 Interpolated Recall - Precision Averages: at 0.00 0.8052 at 0.10 0.6173 at 0.20 0.5294 at 0.30 0.4557 at 0.40 0.3979 at 0.50 0.3431 at 0.60 0.2610 at 0.70 0.1944 at 0.80 0.1321 at 0.90 0.0913 at 1.00 0.0314 Avg. prec. (non-interpolated) for all rel. documents 0.3317 Precision: At 5 docs: 0.5700 At 10 docs: 0.5450 At 15 docs: 0.5100 At 20 docs: 0.5017 At 30 docs: 0.4794 At 100 docs: 0.3688 At 200 docs: 0.2778 At 500 docs: 0.1700 At 1000 docs: 0.1057 R-Precision (prec. after R (=num_rel) docs. retrieved): Exact: 0.3814

0.00.20.4 Recall

0.0

0.2

0.4

0.6

0.81.0

Precision

Run bkmul8en1

−1.0

−0.50.0

+0.5

+1.0

Difference (average precision)

Run bkmul8en1

(24)

UC Berkeley, Multilingual-8 English, Auto, TD

0.00.20.4 Recall

0.0

0.2

0.4

0.6

0.81.0

Precision

Run bkmul8en2

−1.0

−0.50.0

+0.5

+1.0

Difference (average precision)

Run bkmul8en2

(25)

UC Berkeley, Multilingual-8 English, Auto, TD

0.00.20.4 Recall

0.0

0.2

0.4

0.6

0.81.0

Precision

Run bkmul8en3

−1.0

−0.50.0

+0.5

+1.0

Difference (average precision)

Run bkmul8en3

(26)

Univ. Amsterdam/LIT, Multilingual-8 English, Auto, TD Average precision (individual queries): Query 141: 0.6697 Query 142: 0.8436 Query 143: 0.2981 Query 144: 0.2007 Query 145: 0.3353 Query 146: 0.1359 Query 147: 0.1812 Query 148: 0.1840 Query 149: 0.5247 Query 150: 0.2009 Query 151: 0.1472 Query 152: 0.3226 Query 153: 0.1281 Query 154: 0.1829 Query 155: 0.0422 Query 156: 0.1220 Query 157: 0.2936 Query 158: 0.3810 Query 159: 0.2069 Query 160: 0.0356 Query 161: 0.0028 Query 162: 0.3365 Query 163: 0.6123 Query 164: 0.0096 Query 165: 0.4288 Query 166: 0.0038 Query 167: 0.3901 Query 168: 0.2573 Query 169: 0.0193 Query 170: 0.2696 Query 171: 0.0365 Query 172: 0.0159 Query 173: 0.5692 Query 174: 0.7790 Query 175: 0.5975 Query 176: 0.8928 Query 177: 0.1095 Query 178: 0.0723 Query 179: 0.3464 Query 180: 0.4818 Query 181: 0.2249 Query 182: 0.4968 Query 183: 0.2328 Query 184: 0.4207 Query 185: 0.0656 Query 186: 0.1355 Query 187: 0.3113 Query 188: 0.4542 Query 189: 0.5730 Query 190: 0.1165 Query 191: 0.3333 Query 192: 0.0093 Query 193: 0.1507 Query 194: 0.1487 Query 195: 0.0941 Query 196: 0.5991 Query 197: 0.3798 Query 198: 0.0500 Query 199: 0.6356 Query 200: 0.1288 Overall statistics (for 60 queries): Total number of documents over all queries Retrieved: 60000 Relevant: 10020 Rel_ret: 5662 Interpolated Recall - Precision Averages: at 0.00 0.8084 at 0.10 0.5776 at 0.20 0.4738 at 0.30 0.4048 at 0.40 0.3273 at 0.50 0.2732 at 0.60 0.2063 at 0.70 0.1547 at 0.80 0.0887 at 0.90 0.0438 at 1.00 0.0064 Avg. prec. (non-interpolated) for all rel. documents 0.2871 Precision: At 5 docs: 0.6400 At 10 docs: 0.5900 At 15 docs: 0.5411 At 20 docs: 0.5158 At 30 docs: 0.4806 At 100 docs: 0.3303 At 200 docs: 0.2490 At 500 docs: 0.1506 At 1000 docs: 0.0944 R-Precision (prec. after R (=num_rel) docs. retrieved): Exact: 0.3253

0.00.20.4 Recall

0.0

0.2

0.4

0.6

0.81.0

Precision

Run UAmsC03EnM84GiSb

−1.0

−0.50.0

+0.5

+1.0

Difference (average precision)

Run UAmsC03EnM84GiSb

(27)

0.00.20.4 Recall

0.0

0.2

0.4

0.6

0.81.0

Precision

Run UAmsC03EnM84Gr

−1.0

−0.50.0

+0.5

+1.0

Difference (average precision)

Run UAmsC03EnM84Gr

(28)

0.00.20.4 Recall

0.0

0.2

0.4

0.6

0.81.0

Precision

Run UAmsC03EnM84Gr6

−1.0

−0.50.0

+0.5

+1.0

Difference (average precision)

Run UAmsC03EnM84Gr6

(29)

0.00.20.4 Recall

0.0

0.2

0.4

0.6

0.81.0

Precision

Run UAmsC03EnM8SS4G

−1.0

−0.50.0

+0.5

+1.0

Difference (average precision)

Run UAmsC03EnM8SS4G

(30)

0.00.20.4 Recall

0.0

0.2

0.4

0.6

0.81.0

Precision

Run UAmsC03EnM8SS4G6

−1.0

−0.50.0

+0.5

+1.0

Difference (average precision)

Run UAmsC03EnM8SS4G6

(31)

U Buffalo, Multilingual-8 English, Auto, TD

0.00.20.4 Recall

0.0

0.2

0.4

0.6

0.81.0

Precision

Run UBENmultirf1

−1.0

−0.50.0

+0.5

+1.0

Difference (average precision)

Run UBENmultirf1

(32)

U Buffalo, Multilingual-8 English, Auto, TD

0.00.20.4 Recall

0.0

0.2

0.4

0.6

0.81.0

Precision

Run UBENmultirf2

−1.0

−0.50.0

+0.5

+1.0

Difference (average precision)

Run UBENmultirf2

(33)

U Buffalo, Multilingual-8 English, Auto, TD

0.00.20.4 Recall

0.0

0.2

0.4

0.6

0.81.0

Precision

Run UBENmultirf3

−1.0

−0.50.0

+0.5

+1.0

Difference (average precision)

Run UBENmultirf3

(34)

U Buffalo, Multilingual-8 English, Auto, T

0.00.20.4 Recall

0.0

0.2

0.4

0.6

0.81.0

Precision

Run UBENmultishort2

−1.0

−0.50.0

+0.5

+1.0

Difference (average precision)

Run UBENmultishort2

(35)

U Buffalo, Multilingual-8 English, Auto, T

0.00.20.4 Recall

0.0

0.2

0.4

0.6

0.81.0

Precision

Run UBENmultishort3

−1.0

−0.50.0

+0.5

+1.0

Difference (average precision)

Run UBENmultishort3

(36)

U Buffalo, Multilingual-8 Spanish, Auto, TD

0.00.20.4 Recall

0.0

0.2

0.4

0.6

0.81.0

Precision

Run UBESmultirf1

−1.0

−0.50.0

+0.5

+1.0