Appendix B: Results of the GeoCLEF Tracks

(1)

Appendix B

Results of the GeoCLEF Tracks

Prepared by:

Giorgio Maria Di Nunzio and Nicola Ferro Department of Information Engineering

University of Padua

(2)

(3)

Introduction

(4)

(5)

The following pages contain the results for all valid experiments for the GeoCLEF bilingual and monolingual tasks that were officially submitted to the organizers of the CLEF 2005 campaign.

This document is divided in three main parts:

1. List of submitted runs 2. Overview graphs 3. Individual results

1. List of submitted runs

This section gives a listing of all runs and their characteristics:

2. Overview Graphs

For each track/task two front-back pages are produced to compare the top experiments. The first page shows the best entries only for the mandatory title + description (TD) automatic run of at most top five paricipant. The second page shows the best entries among all runs (T/TD/TDN, automatic/manual) submitted of at most top five participants.

The two pages contain the following information:

- Front

- Average precision vs Recall plot

- R-Precision vs number of retrieved documents plot - Back

- Comparison to median plot

- A table with best/median/worst performance figures for each topic

3. Individual Results

This section provides the individual results for each official experiments. Each experiment is presented in one page containing a set of tables and graphs:

- Overall statistics and info

- Interpolated recall vs Precision averages - Document cutoff levels vs Precision at DCL - Individual topic results

Results for CLEF 2005 GeoCLEF Tracks

Participant: the name of the participant responsible for run.

Country: country of participant.

Run Tag unique identifier for each experiment.

Task: track/task to which the experiment belongs.

Topic language: language of the topics used to create the experiment (ISO identifiers for language).

Query type: identifies the parts of the topics used to create the experiment (T = title, D = Description, N = Narrative).

Run Type: type of experiment (automatic/manual).

Pool: specifies if experiment was used for relevance assessment pooling.

(6)

(7)

List of Submitted Runs

(8)

(9)

Participant Country Run Tag Task Topic Lang.

Query Type

Run Type Pool

alicante Spain irua-de-gaz GC-Monolingual-DE DE TD automatic yes

alicante Spain irua-de-ner GC-Monolingual-DE DE TD automatic yes

alicante Spain irua-de-titledesc GC-Monolingual-DE DE TD automatic yes

alicante Spain irua-de-titledescgeotags GC-Monolingual-DE DE TD automatic yes

alicante Spain irua-deen-gaz GC-Bilingual-X2EN DE TD automatic yes

alicante Spain irua-deen-ner GC-Bilingual-X2EN DE TD automatic yes

alicante Spain irua-deen-titledesc GC-Bilingual-X2EN DE TD automatic yes alicante Spain irua-deen-titledescgeotags GC-Bilingual-X2EN DE TD automatic yes

alicante Spain irua-en-gaz GC-Monolingual-EN EN TD automatic yes

alicante Spain irua-en-ner GC-Monolingual-EN EN TD automatic yes

alicante Spain irua-en-syn GC-Monolingual-EN EN TD automatic yes

alicante Spain irua-en-titledesc GC-Monolingual-EN EN TD automatic yes

alicante Spain irua-en-titledescgeotags GC-Monolingual-EN EN TD automatic yes

alicante Spain irua-ende-gaz GC-Bilingual-X2DE EN TD automatic yes

alicante Spain irua-ende-ner GC-Bilingual-X2DE EN TD automatic yes

alicante Spain irua-ende-syn GC-Bilingual-X2DE EN TD automatic yes

alicante Spain irua-ende-titledesc GC-Bilingual-X2DE EN TD automatic yes alicante Spain irua-ende-titledescgeotags GC-Bilingual-X2DE EN TD automatic yes

alicante Spain irua-esde-gaz GC-Bilingual-X2DE ES TD automatic yes

alicante Spain irua-esde-ner GC-Bilingual-X2DE ES TD automatic yes

alicante Spain irua-esde-titledesc GC-Bilingual-X2DE ES TD automatic yes alicante Spain irua-esde-titledescgeotags GC-Bilingual-X2DE ES TD automatic yes

alicante Spain irua-esen-gaz GC-Bilingual-X2EN ES TD automatic yes

alicante Spain irua-esen-ner GC-Bilingual-X2EN ES TD automatic yes

alicante Spain irua-esen-titledesc GC-Bilingual-X2EN ES TD automatic yes alicante Spain irua-esen-titledescgeotags GC-Bilingual-X2EN ES TD automatic yes

alicante Spain irua-ptde-gaz GC-Bilingual-X2DE PT TD automatic yes

alicante Spain irua-ptde-ner GC-Bilingual-X2DE PT TD automatic yes

alicante Spain irua-ptde-titledesc GC-Bilingual-X2DE PT TD automatic yes alicante Spain irua-ptde-titledescgeotags GC-Bilingual-X2DE PT TD automatic yes

alicante Spain irua-pten-gaz GC-Bilingual-X2EN PT TD automatic yes

alicante Spain irua-pten-ner GC-Bilingual-X2EN PT TD automatic yes

alicante Spain irua-pten-titledesc GC-Bilingual-X2EN PT TD automatic yes alicante Spain irua-pten-titledescgeotags GC-Bilingual-X2EN PT TD automatic yes

berkeley USA BERK1BLDEENLOC01 GC-Bilingual-X2EN DE TD automatic yes

berkeley USA BERK1BLDEENNOL01 GC-Bilingual-X2EN DE TD automatic yes

berkeley USA BERK1BLENDELOC01 GC-Bilingual-X2DE EN TD automatic yes

berkeley USA BERK1BLENDENOL01 GC-Bilingual-X2DE EN TD automatic yes

berkeley USA BERK1MLDELOC02 GC-Monolingual-DE DE TD automatic yes

berkeley USA BERK1MLDELOC03 GC-Monolingual-DE DE TD automatic yes

berkeley USA BERK1MLDENOL01 GC-Monolingual-DE DE TD automatic yes

berkeley USA BERK1MLENLOC02 GC-Monolingual-EN EN TD automatic yes

berkeley USA BERK1MLENLOC03 GC-Monolingual-EN EN TD automatic yes

berkeley USA BERK1MLENNOL01 GC-Monolingual-EN EN TD automatic yes

berkeley-2 USA BKGeoD1 GC-Monolingual-DE DE TD automatic yes

berkeley-2 USA BKGeoD2 GC-Monolingual-DE DE TD automatic yes

berkeley-2 USA BKGeoD3 GC-Monolingual-DE DE TDN automatic yes

berkeley-2 USA BKGeoD4 GC-Monolingual-DE DE TD manual yes

(10)

Query Type

csu-sanmarcos USA csusm1 GC-Monolingual-EN EN TD automatic yes

csu-sanmarcos USA csusm2 GC-Monolingual-EN EN TD automatic yes

csu-sanmarcos USA csusm3 GC-Bilingual-X2EN ES TD automatic yes

csu-sanmarcos USA csusm4 GC-Bilingual-X2EN ES TD automatic yes

hagen Germany FUHinstd GC-Monolingual-DE DE TD automatic yes

hagen Germany FUHo10td GC-Monolingual-DE DE TD automatic yes

hagen Germany FUHo10tdl GC-Monolingual-DE DE TD automatic yes

hagen Germany FUHo14td GC-Monolingual-DE DE TD automatic yes

hagen Germany FUHo14tdl GC-Monolingual-DE DE TD automatic yes

linguit Germany LCONCPHR GC-Monolingual-EN EN TD manual no

linguit Germany LCONCPHRALL GC-Monolingual-EN EN TD manual no

linguit Germany LCONCPHRANY GC-Monolingual-EN EN TD manual no

linguit Germany LCONCPHRMOST GC-Monolingual-EN EN TD manual no

linguit Germany LCONCPHRSPAT GC-Monolingual-EN EN TD automatic no

linguit Germany LCONCPHRSPATALL GC-Monolingual-EN EN TD manual no

linguit Germany LCONCPHRSPATANY GC-Monolingual-EN EN TD manual no

linguit Germany LCONCPHRSPATMOST GC-Monolingual-EN EN TD manual no

linguit Germany LCONCPHRWNMN GC-Monolingual-EN EN TD manual no

linguit Germany LCONCPHRWNMNALL GC-Monolingual-EN EN TD manual no

linguit Germany LCONCPHRWNMNANY GC-Monolingual-EN EN TD manual no

linguit Germany LCONCPHRWNMNMOST GC-Monolingual-EN EN TD manual no

linguit Germany LTITLE GC-Monolingual-EN EN T automatic no

linguit Germany LTITLEALL GC-Monolingual-EN EN T automatic no

linguit Germany LTITLEANY GC-Monolingual-EN EN T automatic no

linguit Germany LTITLEMOST GC-Monolingual-EN EN T automatic no

metacarta USA run0 GC-Monolingual-EN EN T automatic yes

metacarta USA run1 GC-Monolingual-EN EN D automatic yes

miracle Spain GCdeCS GC-Monolingual-DE DE TD automatic yes

miracle Spain GCdeNCS GC-Monolingual-DE DE TD automatic yes

miracle Spain GCdeNOR GC-Monolingual-DE DE TD automatic yes

miracle Spain GCenCS GC-Monolingual-EN EN TD automatic yes

miracle Spain GCenNCS GC-Monolingual-EN EN TD automatic yes

miracle Spain GCenNOR GC-Monolingual-EN EN TD automatic yes

miracle Spain LGCdeCS GC-Monolingual-DE DE TD automatic yes

miracle Spain LGCdeNCS GC-Monolingual-DE DE TD automatic yes

miracle Spain LGCenCS GC-Monolingual-EN EN TD automatic yes

miracle Spain LGCenNCS GC-Monolingual-EN EN TD automatic yes

nicta Australia i2d2Run1 GC-Monolingual-EN EN T automatic yes

nicta Australia i2d2Run2 GC-Monolingual-EN EN TDN automatic yes

talp Spain geotalpIR1 GC-Monolingual-EN EN TD automatic yes

u.valencia Spain dsic_gc051 GC-Monolingual-EN EN TD automatic yes

u.valencia Spain dsic_gc052 GC-Monolingual-EN EN TD automatic yes

xldb Portugal XLDBDEManTD GC-Monolingual-DE DE TD manual yes

(11)

Query Type

xldb Portugal XLDBENManTDL GC-Monolingual-EN EN TD manual yes

xldb Portugal XLDBPTAutMandTD GC-Bilingual-X2EN PT TD automatic yes

xldb Portugal XLDBPTAutMandTDL GC-Bilingual-X2EN PT TD automatic yes

xldb Portugal XLDBPTManTDGKBm3 GC-Bilingual-X2EN PT TD automatic yes

xldb Portugal XLDBPTManTDGKBm4 GC-Bilingual-X2EN PT TD automatic yes

(12)

(13)

Overview Graphs

(14)

(15)

GeoCLEF Overview Graphs GC-Bilingual-X2DE

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Interpolated Recall

Average Precision

CLEF 2005 − Top 3 participants of GeoCLEF Bilingual X2DE − Interpolated Recall vs Average Precision berkeley−2 [Avg. Prec. 17.88%; Run BKGeoED2, TD Auto, Pooled]

alicante [Avg. Prec. 17.52%; Run irua−ende−syn, TD Auto, Pooled]

berkeley [Avg. Prec. 7.77%; Run BERK1BLENDENOL01, TD Auto, Pooled]

30%

40%

50%

60%

70%

80%

90%

100%

R−Precision

CLEF 2005 − Top 3 participants of GeoCLEF Bilingual X2DE − Retrieved documents vs Precision berkeley−2 [Exact Prec. 19.03%, Run BKGeoED2, TD Auto, Pooled]

alicante [Exact Prec. 18.48%, Run irua−ende−syn, TD Auto, Pooled]

berkeley [Exact Prec. 10.17%, Run BERK1BLENDENOL01, TD Auto, Pooled]

(16)

1 2 3 4 5 6 7 8 9 10 11 12 13

−1

−0.5 0 0.5 1

Topic Number

Difference (average precision)

CLEF 2005 − Top 3 participants of GeoCLEF Bilingual X2DE − Comparison to median by topic (topics 1 to 13)

berkeley−2 [Run BKGeoED2, TD Auto, Pooled]

alicante [Run irua−ende−syn, TD Auto, Pooled]

berkeley [Run BERK1BLENDENOL01, TD Auto, Pooled]

14 15 16 17 18 19 20 21 22 23 24 25

−1

−0.5 0 0.5 1

Topic Number

# of Relevant Retrieved Docs @ 100 # of Relevant Retrieved Docs @ 1000 Average Precision %

Topic ID Best Median Worst Best Median Worst Best Median Worst

001 0 0 0 0 0 0 00.00 00.00 00.00

002 5 4 0 5 4 0 05.56 01.65 00.00

003 37 13 1 58 31 2 41.96 06.79 00.11

004 12 2 0 16 6 0 59.12 02.73 00.00

005 9 8 3 11 10 6 72.66 59.45 06.71

006 2 0 0 6 1 0 03.19 00.20 00.00

007 12 2 0 28 13 2 19.67 00.81 00.01

008 3 1 0 5 3 0 03.33 00.43 00.00

009 45 22 3 63 34 6 63.77 21.02 00.51

010 40 5 0 47 21 2 65.16 02.69 00.04

011 39 2 0 51 20 0 40.32 02.08 00.00

012 21 10 5 29 25 7 30.96 13.51 05.30

013 31 20 11 39 33 24 56.82 34.09 09.57

014 25 13 0 53 30 5 20.96 08.47 00.05

015 45 40 23 144 136 104 33.08 26.58 13.35

016 12 7 1 14 13 2 36.50 15.49 01.60

017 35 13 11 64 52 49 31.00 05.82 05.23

018 1 0 0 1 1 1 03.03 00.76 00.11

019 50 34 1 59 45 7 77.21 45.69 00.62

020 0 0 0 0 0 0 00.00 00.00 00.00

021 9 6 1 21 19 5 23.43 06.59 01.17

022 0 0 0 0 0 0 00.00 00.00 00.00

023 1 1 0 2 2 0 05.73 01.42 00.00

024 2 0 0 3 2 0 05.96 00.65 00.00

025 0 0 0 0 0 0 00.00 00.00 00.00

(17)

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Interpolated Recall

Average Precision

CLEF 2005 − Top 3 participants of GeoCLEF Bilingual X2DE − Interpolated Recall vs Average Precision berkeley−2 [Avg. Prec. 17.88%; Run BKGeoED2, TD Auto, Pooled]

alicante [Avg. Prec. 17.52%; Run irua−ende−syn, TD Auto, Pooled]

berkeley [Avg. Prec. 7.77%; Run BERK1BLENDENOL01, TD Auto, Pooled]

30%

40%

50%

60%

70%

80%

90%

100%

R−Precision

CLEF 2005 − Top 3 participants of GeoCLEF Bilingual X2DE − Retrieved documents vs Precision berkeley−2 [Exact Prec. 19.03%, Run BKGeoED2, TD Auto, Pooled]

alicante [Exact Prec. 18.48%, Run irua−ende−syn, TD Auto, Pooled]

berkeley [Exact Prec. 10.17%, Run BERK1BLENDENOL01, TD Auto, Pooled]

(18)

1 2 3 4 5 6 7 8 9 10 11 12 13

−1

−0.5 0 0.5 1

Topic Number

14 15 16 17 18 19 20 21 22 23 24 25

−1

−0.5 0 0.5 1

Topic Number

(19)

GeoCLEF Overview Graphs GC-Bilingual-X2EN

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Interpolated Recall

Average Precision

CLEF 2005 − Top 5 participants of GeoCLEF Bilingual X2EN − Interpolated Recall vs Average Precision berkeley−2 [Avg. Prec. 37.15%; Run BKGeoDE2, TD Auto, Pooled]

csu−sanmarcos [Avg. Prec. 35.60%; Run csusm3, TD Auto, Pooled]

alicante [Avg. Prec. 31.78%; Run irua−deen−ner, TD Auto, Pooled]

berkeley [Avg. Prec. 27.53%; Run BERK1BLDEENLOC01, TD Auto, Pooled]

xldb [Avg. Prec. 16.45%; Run XLDBPTAutMandTDL, TD Auto, Pooled]

30%

40%

50%

60%

70%

80%

90%

100%

R−Precision

CLEF 2005 − Top 5 participants of GeoCLEF Bilingual X2EN − Retrieved documents vs Precision berkeley−2 [Exact Prec. 37.87%, Run BKGeoDE2, TD Auto, Pooled]

csu−sanmarcos [Exact Prec. 37.17%, Run csusm3, TD Auto, Pooled]

alicante [Exact Prec. 34.40%, Run irua−deen−ner, TD Auto, Pooled]

berkeley [Exact Prec. 27.85%, Run BERK1BLDEENLOC01, TD Auto, Pooled]

xldb [Exact Prec. 20.42%, Run XLDBPTAutMandTDL, TD Auto, Pooled]

(20)

1 2 3 4 5 6 7 8 9 10 11 12 13

−1

−0.5 0 0.5 1

Topic Number

CLEF 2005 − Top 5 participants of GeoCLEF Bilingual X2EN − Comparison to median by topic (topics 1 to 13)

berkeley−2 [Run BKGeoDE2, TD Auto, Pooled]

csu−sanmarcos [Run csusm3, TD Auto, Pooled]

alicante [Run irua−deen−ner, TD Auto, Pooled]

berkeley [Run BERK1BLDEENLOC01, TD Auto, Pooled]

xldb [Run XLDBPTAutMandTDL, TD Auto, Pooled]

14 15 16 17 18 19 20 21 22 23 24 25

−1

−0.5 0 0.5 1

Topic Number

001 13 11 0 14 14 0 70.70 51.62 00.00

002 7 5 2 11 7 2 23.38 09.58 06.34

003 7 6 0 10 8 0 38.33 20.49 00.00

004 19 7 0 31 23 0 19.55 04.29 00.00

005 25 23 1 27 27 1 73.45 60.39 03.70

006 10 5 0 12 10 0 53.93 17.02 00.00

007 19 5 1 63 28 16 08.92 01.85 00.36

008 8 6 0 10 9 0 18.33 09.07 00.00

009 14 6 1 15 12 1 59.58 08.16 02.63

010 12 10 8 12 12 8 75.86 56.88 19.43

011 8 5 1 15 10 3 17.70 11.22 00.78

012 35 30 14 75 67 36 32.10 25.10 10.86

013 7 7 2 7 7 2 51.76 22.94 05.95

014 39 18 0 43 39 0 64.51 14.55 00.00

015 78 75 68 110 110 68 82.09 67.94 46.75

016 14 12 0 15 15 0 83.57 62.86 00.00

017 62 45 40 129 129 49 53.96 38.50 30.43

018 27 4 1 42 10 2 30.86 01.64 00.17

019 45 29 10 97 71 36 39.29 14.73 05.54

020 7 4 0 9 6 0 55.40 18.54 00.00

021 27 18 0 29 25 0 69.71 41.01 00.00

022 35 26 19 46 39 19 55.12 41.16 17.39

023 14 3 0 33 19 0 10.32 01.44 00.00

024 65 38 0 105 87 0 59.14 32.06 00.00

025 3 3 0 3 3 0 100.00 53.17 00.00

(21)

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Interpolated Recall

Average Precision

CLEF 2005 − Top 5 participants of GeoCLEF Bilingual X2EN − Interpolated Recall vs Average Precision berkeley−2 [Avg. Prec. 37.15%; Run BKGeoDE2, TD Auto, Pooled]

alicante [Avg. Prec. 31.78%; Run irua−deen−ner, TD Auto, Pooled]

berkeley [Avg. Prec. 27.53%; Run BERK1BLDEENLOC01, TD Auto, Pooled]

xldb [Avg. Prec. 16.45%; Run XLDBPTAutMandTDL, TD Auto, Pooled]

30%

40%

50%

60%

70%

80%

90%

100%

R−Precision

CLEF 2005 − Top 5 participants of GeoCLEF Bilingual X2EN − Retrieved documents vs Precision berkeley−2 [Exact Prec. 37.87%, Run BKGeoDE2, TD Auto, Pooled]

alicante [Exact Prec. 34.40%, Run irua−deen−ner, TD Auto, Pooled]

berkeley [Exact Prec. 27.85%, Run BERK1BLDEENLOC01, TD Auto, Pooled]

xldb [Exact Prec. 20.42%, Run XLDBPTAutMandTDL, TD Auto, Pooled]

(22)

1 2 3 4 5 6 7 8 9 10 11 12 13

−1

−0.5 0 0.5 1

Topic Number

14 15 16 17 18 19 20 21 22 23 24 25

−1

−0.5 0 0.5 1

Topic Number

(23)

GeoCLEF Overview Graphs GC-Monolingual-DE

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Interpolated Recall

Average Precision

CLEF 2005 − Top 5 participants of GeoCLEF Monolingual DE − Interpolated Recall vs Average Precision berkeley−2 [Avg. Prec. 16.08%; Run BKGeoD2, TD Auto, Pooled]

alicante [Avg. Prec. 12.27%; Run irua−de−titledescgeotags, TD Auto, Pooled]

miracle [Avg. Prec. 11.63%; Run GCdeNOR, TD Auto, Pooled]

hagen [Avg. Prec. 10.53%; Run FUHo14td, TD Auto, Pooled]

berkeley [Avg. Prec. 5.35%; Run BERK1MLDELOC02, TD Auto, Pooled]

30%

40%

50%

60%

70%

80%

90%

100%

R−Precision

CLEF 2005 − Top 5 participants of GeoCLEF Monolingual DE − Retrieved documents vs Precision berkeley−2 [Exact Prec. 17.30%, Run BKGeoD2, TD Auto, Pooled]

alicante [Exact Prec. 14.83%, Run irua−de−titledescgeotags, TD Auto, Pooled]

miracle [Exact Prec. 15.29%, Run GCdeNOR, TD Auto, Pooled]

hagen [Exact Prec. 12.01%, Run FUHo14td, TD Auto, Pooled]

berkeley [Exact Prec. 8.46%, Run BERK1MLDELOC02, TD Auto, Pooled]

(24)

1 2 3 4 5 6 7 8 9 10 11 12 13

−1

−0.5 0 0.5 1

Topic Number

CLEF 2005 − Top 5 participants of GeoCLEF Monolingual DE − Comparison to median by topic (topics 1 to 13)

berkeley−2 [Run BKGeoD2, TD Auto, Pooled]

alicante [Run irua−de−titledescgeotags, TD Auto, Pooled]

miracle [Run GCdeNOR, TD Auto, Pooled]

hagen [Run FUHo14td, TD Auto, Pooled]

berkeley [Run BERK1MLDELOC02, TD Auto, Pooled]

14 15 16 17 18 19 20 21 22 23 24 25

−1

−0.5 0 0.5 1

Topic Number

berkeley−2 [Run BKGeoD2, TD Auto, Pooled]

berkeley [Run BERK1MLDELOC02, TD Auto, Pooled]

001 0 0 0 0 0 0 00.00 00.00 00.00

002 7 0 0 9 2 0 15.06 00.18 00.00

003 45 24 1 58 55 1 67.13 23.36 00.19

004 15 1 0 16 3 0 67.56 06.27 00.00

005 8 2 0 11 6 0 56.41 09.88 00.00

006 6 1 0 7 2 0 39.29 00.66 00.00

007 14 7 0 26 16 4 19.07 05.39 00.05

008 6 0 0 6 1 0 58.64 00.02 00.00

009 52 42 3 64 56 3 62.73 52.15 01.93

010 46 7 0 47 29 0 79.36 07.82 00.00

011 24 1 0 49 6 0 23.42 00.41 00.00

012 21 6 0 30 23 1 29.56 10.07 00.02

013 31 18 0 43 33 0 56.82 24.66 00.00

014 48 14 0 54 30 0 72.99 07.17 00.00

015 52 44 5 150 128 5 36.30 23.50 01.66

016 14 7 0 16 10 0 44.39 09.39 00.00

017 33 10 0 81 40 0 25.44 04.20 00.00

018 1 0 0 1 1 0 11.11 00.87 00.00

019 50 12 0 60 14 0 64.88 12.70 00.00

020 0 0 0 0 0 0 00.00 00.00 00.00

021 11 5 0 19 19 0 11.23 07.44 00.00

022 0 0 0 0 0 0 00.00 00.00 00.00

023 1 0 0 2 0 0 16.82 00.00 00.00

024 3 0 0 3 3 0 04.10 00.86 00.00

025 0 0 0 0 0 0 00.00 00.00 00.00

(25)

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Interpolated Recall

Average Precision

CLEF 2005 − Top 5 participants of GeoCLEF Monolingual DE − Interpolated Recall vs Average Precision berkeley−2 [Avg. Prec. 20.42%; Run BKGeoD3, TDN Auto, Pooled]

alicante [Avg. Prec. 12.27%; Run irua−de−titledescgeotags, TD Auto, Pooled]

miracle [Avg. Prec. 11.63%; Run GCdeNOR, TD Auto, Pooled]

xldb [Avg. Prec. 11.23%; Run XLDBDEManTDGKBm3, TD Manual, Pooled]

hagen [Avg. Prec. 10.53%; Run FUHo14td, TD Auto, Pooled]

30%

40%

50%

60%

70%

80%

90%

100%

R−Precision

CLEF 2005 − Top 5 participants of GeoCLEF Monolingual DE − Retrieved documents vs Precision berkeley−2 [Exact Prec. 21.15%, Run BKGeoD3, TDN Auto, Pooled]

alicante [Exact Prec. 14.83%, Run irua−de−titledescgeotags, TD Auto, Pooled]

miracle [Exact Prec. 15.29%, Run GCdeNOR, TD Auto, Pooled]

xldb [Exact Prec. 12.52%, Run XLDBDEManTDGKBm3, TD Manual, Pooled]

hagen [Exact Prec. 12.01%, Run FUHo14td, TD Auto, Pooled]

(26)

1 2 3 4 5 6 7 8 9 10 11 12 13

−1

−0.5 0 0.5 1

Topic Number

berkeley−2 [Run BKGeoD3, TDN Auto, Pooled]

xldb [Run XLDBDEManTDGKBm3, TD Manual, Pooled]

14 15 16 17 18 19 20 21 22 23 24 25

−1

−0.5 0 0.5 1

Topic Number

berkeley−2 [Run BKGeoD3, TDN Auto, Pooled]

xldb [Run XLDBDEManTDGKBm3, TD Manual, Pooled]

(27)

GeoCLEF Overview Graphs GC-Monolingual-EN

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Interpolated Recall

Average Precision

CLEF 2005 − Top 5 participants of GeoCLEF Monolingual EN − Interpolated Recall vs Average Precision berkeley−2 [Avg. Prec. 39.36%; Run BKGeoE1, TD Auto, Pooled]

alicante [Avg. Prec. 34.95%; Run irua−en−ner, TD Auto, Pooled]

berkeley [Avg. Prec. 29.24%; Run BERK1MLENLOC03, TD Auto, Pooled]

miracle [Avg. Prec. 26.53%; Run GCenNOR, TD Auto, Pooled]

30%

40%

50%

60%

70%

80%

90%

100%

R−Precision

CLEF 2005 − Top 5 participants of GeoCLEF Monolingual EN − Retrieved documents vs Precision berkeley−2 [Exact Prec. 38.87%, Run BKGeoE1, TD Auto, Pooled]

alicante [Exact Prec. 37.74%, Run irua−en−ner, TD Auto, Pooled]

berkeley [Exact Prec. 31.31%, Run BERK1MLENLOC03, TD Auto, Pooled]

miracle [Exact Prec. 29.67%, Run GCenNOR, TD Auto, Pooled]

(28)

1 2 3 4 5 6 7 8 9 10 11 12 13

−1

−0.5 0 0.5 1

Topic Number

CLEF 2005 − Top 5 participants of GeoCLEF Monolingual EN − Comparison to median by topic (topics 1 to 13)

berkeley−2 [Run BKGeoE1, TD Auto, Pooled]

alicante [Run irua−en−ner, TD Auto, Pooled]

berkeley [Run BERK1MLENLOC03, TD Auto, Pooled]

miracle [Run GCenNOR, TD Auto, Pooled]

14 15 16 17 18 19 20 21 22 23 24 25

−1

−0.5 0 0.5 1

Topic Number

001 14 10 0 14 12 0 71.05 47.28 00.00

002 7 3 0 11 4 0 22.88 06.68 00.00

003 7 4 0 10 6 0 44.55 05.04 00.00

004 28 14 0 40 27 0 36.42 13.98 00.00

005 25 17 0 27 25 0 72.74 50.08 00.00

006 10 5 0 13 8 0 37.98 14.79 00.00

007 44 15 0 80 58 0 44.13 10.39 00.00

008 6 4 0 10 7 0 15.62 06.10 00.00

009 12 8 0 16 11 0 47.96 28.31 00.00

010 12 9 0 13 12 1 81.83 37.67 00.10

011 15 4 0 19 10 0 25.29 05.11 00.00

012 35 15 0 76 38 0 26.54 13.76 00.00

013 7 3 0 7 6 0 55.58 17.12 00.00

014 36 13 0 43 29 0 56.80 10.08 00.00

015 80 65 4 110 110 4 83.40 63.51 03.59

016 15 7 0 15 13 0 88.64 12.85 00.00

017 67 42 1 129 122 1 62.06 37.02 00.78

018 30 12 0 46 25 2 43.24 09.14 00.03

019 43 17 0 99 61 0 37.65 10.30 00.00

020 7 5 0 9 7 0 35.55 22.20 00.00

021 26 19 1 29 25 6 60.50 30.86 03.09

022 36 15 0 46 31 0 55.12 15.76 00.00

023 23 3 0 32 9 0 22.38 01.13 00.00

024 63 37 0 105 86 0 56.70 32.94 00.00

025 3 3 0 3 3 0 55.26 13.70 00.00

(29)

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Interpolated Recall

Average Precision

CLEF 2005 − Top 5 participants of GeoCLEF Monolingual EN − Interpolated Recall vs Average Precision berkeley−2 [Avg. Prec. 39.36%; Run BKGeoE1, TD Auto, Pooled]

alicante [Avg. Prec. 34.95%; Run irua−en−ner, TD Auto, Pooled]

berkeley [Avg. Prec. 29.24%; Run BERK1MLENLOC03, TD Auto, Pooled]

miracle [Avg. Prec. 26.53%; Run GCenNOR, TD Auto, Pooled]

30%

40%

50%

60%

70%

80%

90%

100%

R−Precision

CLEF 2005 − Top 5 participants of GeoCLEF Monolingual EN − Retrieved documents vs Precision berkeley−2 [Exact Prec. 38.87%, Run BKGeoE1, TD Auto, Pooled]

alicante [Exact Prec. 37.74%, Run irua−en−ner, TD Auto, Pooled]

berkeley [Exact Prec. 31.31%, Run BERK1MLENLOC03, TD Auto, Pooled]

miracle [Exact Prec. 29.67%, Run GCenNOR, TD Auto, Pooled]

(30)

1 2 3 4 5 6 7 8 9 10 11 12 13

−1

−0.5 0 0.5 1

Topic Number

14 15 16 17 18 19 20 21 22 23 24 25

−1

−0.5 0 0.5 1

Topic Number

(31)

Individual Results

(32)

(33)

alicante irua-ende-gaz Overall statistics for 25 queries :

Total number of documents over all queries

Retrieved 25,000

Relevant 785

Relevant retrieved 496

General info:

Track= GeoCLEF Bilingual English to German RunType= automatic

QueryType= TD Pooled= true

Interploated Recall (%) Precision Averages (%)

0 45.73

10 32.60

20 26.59

30 20.93

40 14.13

50 5.74

60 3.92

70 3.18

80 1.49

90 0.53

100 0.21

A v e r a g e p r e c i s i o n ( n o n - i n t e r p o l a t e d ) f o r a l l r e l e v a n t d o c u m e n t s ( a v e r a g e d o v e r q u e r i e s )

12.81

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Interpolated Recall

Average Precision

CLEF 2005 GC−Bilingual−X2DE − Interpolated Recall vs Average Precision irua−ende−gaz

Docs Cutoff Levels Precision at DCL (%)

5 docs 28.80

10 docs 27.60

15 docs 24.00

20 docs 21.20

30 docs 17.33

100 docs 9.36

200 docs 6.18

500 docs 3.29

1000 docs 1.98

R-Precision (precision after R document retrieved, where R = Relevant retrieved)

16.90

5 10 15 20 30 100 200 500 1000

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Retrieved Documents (logarithmic scale)

R−Precision

CLEF 2005 GC−Bilingual−X2DE − Retrieved documents vs Precision irua−ende−gaz

Precision averages (%) for individual queries

Topic 001 0.00 Topic 002 1.65 Topic 003 41.96 Topic 004 13.94 Topic 005 45.39 Topic 006 0.20 Topic 007 17.87 Topic 008 0.25 Topic 009 21.02 Topic 010 0.76 Topic 011 20.77 Topic 012 8.22 Topic 013 34.78

Topic 014 3.45 Topic 015 31.86 Topic 016 25.17 Topic 017 5.82 Topic 018 3.03 Topic 019 24.97 Topic 020 0.00 Topic 021 10.72 Topic 022 0.00 Topic 023 2.38 Topic 024 5.96 Topic 025 0.00

1 2 3 4 5 6 7 8 9 10 11 12 13

−1

−0.5 0 0.5 1

Topic Number

CLEF 2005 GC−Bilingual−X2DE − Comparison to median by topic (topics 1 to 13)

irua−ende−gaz

0.5 1

irua−ende−gaz

GC-Bilingual-X2DE

(34)

alicante irua-ende-ner Overall statistics for 25 queries :

Retrieved 25,000

Relevant 785

General info:

0 46.22

10 30.75

20 27.01

30 22.34

40 19.00

50 16.19

60 11.48

70 6.76

80 3.69

90 1.21

100 0.07

15.60

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Interpolated Recall

Average Precision

CLEF 2005 GC−Bilingual−X2DE − Interpolated Recall vs Average Precision irua−ende−ner

5 docs 28.00

10 docs 26.80

15 docs 22.93

20 docs 22.00

30 docs 18.27

100 docs 11.12

200 docs 7.62

500 docs 4.03

1000 docs 2.38

18.20

5 10 15 20 30 100 200 500 1000

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

R−Precision

CLEF 2005 GC−Bilingual−X2DE − Retrieved documents vs Precision irua−ende−ner

1 2 3 4 5 6 7 8 9 10 11 12 13

−1

−0.5 0 0.5 1

Topic Number

irua−ende−ner

0.5 1

irua−ende−ner

(35)

alicante irua-ende-syn Overall statistics for 25 queries :

Retrieved 25,000

Relevant 785

General info:

0 44.87

10 32.20

20 29.63

30 27.23

40 22.32

50 19.71

60 14.06

70 8.41

80 5.03

90 1.58

100 0.24

17.52

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Interpolated Recall

Average Precision

CLEF 2005 GC−Bilingual−X2DE − Interpolated Recall vs Average Precision irua−ende−syn

5 docs 28.80

10 docs 28.00

15 docs 24.27

20 docs 22.80

30 docs 20.00

100 docs 12.08

200 docs 7.96

500 docs 4.10

1000 docs 2.43

18.48

5 10 15 20 30 100 200 500 1000

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

R−Precision

CLEF 2005 GC−Bilingual−X2DE − Retrieved documents vs Precision irua−ende−syn

1 2 3 4 5 6 7 8 9 10 11 12 13

−1

−0.5 0 0.5 1

Topic Number

irua−ende−syn

0.5 1

irua−ende−syn

(36)

alicante irua-ende-titledesc Overall statistics for 25 queries :

Retrieved 25,000

Relevant 785

General info:

0 47.20

10 33.02

20 29.02

30 25.02

40 20.01

50 16.54

60 10.17

70 7.02

80 4.42

90 1.75

100 0.60

16.42

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Interpolated Recall

Average Precision

CLEF 2005 GC−Bilingual−X2DE − Interpolated Recall vs Average Precision irua−ende−titledesc

5 docs 30.40

10 docs 27.20

15 docs 24.27

20 docs 22.00

30 docs 18.67

100 docs 11.40

200 docs 7.72

500 docs 4.10

1000 docs 2.42

18.50

5 10 15 20 30 100 200 500 1000

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

R−Precision

CLEF 2005 GC−Bilingual−X2DE − Retrieved documents vs Precision irua−ende−titledesc

1 2 3 4 5 6 7 8 9 10 11 12 13

−1

−0.5 0 0.5 1

Topic Number

irua−ende−titledesc

0.5 1

irua−ende−titledesc