Web search engines

Web search engines

Outline

Information Retrieval Text Preprocessing Inverted Index

Answering Keyword Queries PageRank

Conclusion

Information Retrieval, Search

Problem

How toindexWeb content so as to answer (keyword-based) queries efficiently?

Context: set oftext documents

d₁ The jaguar is a New World mammal of the Felidae family.

d₂ Jaguar has designed four new engines.

d₃ For Jaguar, Atari was keen to use a 68K family device.

d₄ The Jacksonville Jaguars are a professional US football team.

d₅ Mac OS X Jaguar is available at a price of US $199 for Apple’s new “family pack”.

d₆ One such ruling family to incorporate the jaguar into their name is Jaguar Paw.

d It is a big cat.

Outline

Information Retrieval Text Preprocessing Inverted Index

Answering Keyword Queries PageRank

Conclusion

Text Preprocessing

Initial textpreprocessingsteps Number of optional steps

Highly depends on theapplication

Highly depends on thedocument language(illustrated with English)

Tokenization

Principle

Separate text intotokens(words) Not so easy!

In some languages (Chinese, Japanese), wordsnot separated by whitespace

Dealconsistentlywith acronyms, elisions, numbers, units, URLs, emails, etc.

Compound words:hostname,host-nameandhost name. Break into two tokens or regroup them as one token? In any case, lexicon and linguistic analysis needed! Even more so in other

Tokenization: Example

d₁ the₁jaguar₂is₃a₄new₅world₆mammal₇of₈the₉felidae₁₀ family₁₁ d₂ jaguar₁has₂designed₃four₄new5engines₆

d₃ for₁jaguar₂atari₃was₄keen₅to₆use₇a₈68k₉family₁₀ device₁₁ d₄ the₁jacksonville₂jaguars₃are₄a₅professional₆us₇football₈team₉ d5 mac₁os₂x₃jaguar₄is5available₆at7a₈price₉of₁₀ us₁₁ $199₁₂

for₁₃ apple’s₁₄new₁₅ family₁₆ pack₁₇

d₆ one₁such₂ruling₃family₄to₅incorporate₆the₇jaguar₈into₉ their₁₀ name₁₁ is₁₂ jaguar₁₃ paw₁₄

d₇ it₁is₂a₃big₄cat₅

Stemming

Principle

Mergedifferent forms of the same word, or of closely related words, into a singlestem

Not in all applications!

Useful for retrieving documents containinggeesewhen searching forgoose

Various degreesof stemming

Possibility of building different indexes, with different stemming

Stemming schemes (1/2)

Morphological stemming (lemmatization).

Removebound morphemesfrom words:

plural markers gender markers

tense or mood inflections etc.

Can be linguisticallyvery complex, cf:

Les poules du couvent couvent.

[The hens of the monastery brood.]

In English, somewhateasy:

Remove final -s, -’s, -ed, -ing, -er, -est Take care of semiregular forms (e.g., -y/-ies) Take care of irregular forms (mouse/mice) But still someambiguities: cf rose

Stemming schemes (2/2)

Lexical stemming.

Mergelexically relatedterms of various parts of speech, such aspolicy,politics,politicalorpolitician For English,Porter’s stemming[Porter, 1980]; stem universityanduniversaltounivers: not perfect!

Possibility of coupling this withlexiconsto merge (near-)synonyms

Phonetic stemming.

Mergephonetically relatedwords: search proper names with different spellings!

For English,Soundex[US National Archives and

Stemming Example

d₁ the₁jaguar₂be₃a₄new₅world₆mammal₇of₈the₉felidae₁₀ family₁₁ d₂ jaguar₁have₂design₃four₄new5engine₆

d₃ for₁jaguar₂atari₃be₄keen₅to₆use₇a₈68k₉family₁₀ device₁₁ d₄ the₁jacksonville₂jaguar₃be₄a₅professional₆us₇football₈team₉ d5 mac₁os₂x₃jaguar₄be5available₆at7a₈price₉of₁₀us₁₁$199₁₂

for₁₃ apple₁₄new₁₅ family₁₆ pack₁₇

d₆ one₁such₂rule₃family₄to₅incorporate₆the₇jaguar₈into₉ their₁₀ name₁₁ be₁₂jaguar₁₃paw₁₄

d₇ it₁be₂a₃big₄cat₅

Stop Word Removal

Principle

Removeuninformativewords from documents, in particular to lower the cost of storing the index

determiners: a,the,this, etc.

function verbs: be,have,make, etc.

conjunctions: that,and, etc.

etc.

Stop Word Removal Example

d₁ jaguar₂new₅world₆mammal₇felidae₁₀ family₁₁ d₂ jaguar₁design₃four₄new5engine₆

d₃ jaguar₂atari₃keen₅68k₉family₁₀device₁₁

d₄ jacksonville₂jaguar₃professional₆us₇football₈team₉ d5 mac₁os₂x₃jaguar₄available₆price₉us₁₁$199₁₂apple₁₄

new₁₅ family₁₆ pack₁₇

d₆ one₁such₂rule₃family₄incorporate₆jaguar₈their₁₀ name₁₁ jaguar₁₃ paw₁₄

d₇ big₄cat₅

Outline

Information Retrieval Text Preprocessing Inverted Index

Answering Keyword Queries PageRank

Conclusion

Inverted Index

After all preprocessing, construction of aninverted index:

Index ofall terms, with the list of documents where this term occurs

Small scale: disk storage, withmemory mapping(cf.mmap) techniques; secondary index for offset of each term in main index Large scale: distributed on acluster of machines; hashing gives the machine responsible for a given term

Updating the index costly, so onlybatch operations(not one-by-one addition of term occurrences)

Inverted Index Example

family d₁,d₃,d₅,d₆ football d₄

jaguar d₁,d₂,d₃,d₄,d₅,d₆ new d₁,d₂,d₅

rule d₆ us d₄,d₅ world d₁ . . .

Storing positions in the index

phrase queries, NEAR operator: need to keepposition information in the index

just add it in the document list!

family d₁/11,d₃/10,d₅/16,d₆/4 football d₄/8

jaguar d₁/2,d₂/1,d₃/2,d₄/3,d₅/4,d₆/8+13 new d₁/5,d₂/5,d₅/15

rule d₆/3

us d₄/7,d₅/11 world d₁/6

. . .

TF-IDF Weighting

Some term occurrences have moreweightthan others:

Terms occurringfrequentlyin agiven document: morerelevant Terms occurringrarelyin thedocument collectionas a whole: more informative

AddTerm Frequency—Inverse Document Frequencyweighting to occurrences;

tfidf(t,d) = n_t,d

∑︀

t^′n_t^′_,d ·log |D|

⃒

⃒

{︀d^′ ∈D|n_t,d^′ >0^}︀⃒⃒

n_t,d number of occurrences oft ind D set of all documents

TF-IDF Weighting Example

family d₁/11/.13,d₃/10/.13,d₆/4/.08,d₅/16/.07 football d₄/8/.47

jaguar d₁/2/.04,d₂/1/.04,d₃/2/.04,d₄/3/.04,d₆/8+13/.04,d₅/4/.02 new d₂/5/.24,d₁/5/.20,d₅/15/.10

rule d₆/3/.28

us d₄/7/.30,d₅/11/.15 world d₁/6/.47

. . .

Outline

Information Retrieval Text Preprocessing Inverted Index

Answering Keyword Queries PageRank

Conclusion

Answering Boolean Queries

Single keyword query: just consult the index and return the documents in index order.

Boolean multi-keyword query

(jaguarANDnewAND NOTfamily) ORcat

Same way! Retrieve document lists from all keywords and apply adequate set operations:

AND intersection OR union AND NOT difference

Global score: some function of the individual weight (e.g., addition for conjunctive queries)

Position queries: consult the index, and filter by appropriate

Answering Top-k Queries

t₁AND . . . ANDtn

t₁OR . . . ORtn

Problem

Find thetop-k results(for some givenk) to the query, without retrieving all documents matching it.

Notations:

s(t,d) weight oft ind (e.g., tfidf)

g(s , . . . ,s ) monotonous function that computes the global score

Fagin’s Threshold Algorithm [Fagin et al., 2001]

(with an additional direct index givings(t,d)) 1. LetRbe the empty list andm= +∞.

2. For each 1≤i≤n:

2.1 Retrieve the documentd⁽ⁱ⁾containing termt_i that has thenext largests(t_i,d⁽ⁱ⁾).

2.2 Compute its global scoreg_d(i) =g(s(t₁,d⁽ⁱ⁾), . . . ,s(tn,d⁽ⁱ⁾))by retrieving alls(tj,d⁽ⁱ⁾)withj̸=i.

2.3 IfRcontains less thank documents, or ifg_d(i)is greater than the minimum of the score of documentsinR, addd⁽ⁱ⁾toR(and remove the worst element inRif it is full).

3. Letm=g(s(t₁,d⁽¹⁾),s(t₂,d⁽²⁾), . . . ,s(tn,d⁽ⁿ⁾)).

4. IfRcontainsk documents, and the minimum of the score of the documents inRisgreater than or equaltom, returnR.

5. Redo step 2.

Outline

Information Retrieval PageRank

Conclusion

The Web Graph

The World Wide Web seenas a (directed) graph:

Vertices: Web pages Edges: hyperlinks

Same for otherinterlinkedenvironments:

dictionaries encyclopedias scientific publications social networks

PageRank (Google’s Ranking [Brin and Page, 1998])

Idea

Importantpages are pages pointed to byimportantpages.

{︃g_ij =0 if there is no link between pageiandj;

g_ij = _n¹

i otherwise, withn_i the number of outgoing links of pagei.

Definition (Tentative)

Probabilitythat the surfer following therandom walkinGhas arrived on pagei at some distant given point in the future.

(︂ )︂

Illustrating PageRank Computation

0.100 0.100

0.100

0.100

0.100 0.100

0.100

0.100

0.100

0.100

Illustrating PageRank Computation

0.033 0.317

0.075

0.108

0.058

0.083

0.150

0.033

Illustrating PageRank Computation

0.036 0.193

0.108

0.163

0.079 0.090

0.074

0.154

0.094

0.008

Illustrating PageRank Computation

0.054 0.212

0.093

0.152

0.051

0.108

0.149

0.026

Illustrating PageRank Computation

0.051 0.247

0.078

0.143

0.053 0.062

0.097

0.153

0.099

0.016

Illustrating PageRank Computation

0.048 0.232

0.093

0.156

0.067

0.087

0.138

0.018

Illustrating PageRank Computation

0.052 0.226

0.092

0.148

0.058 0.064

0.098

0.146

0.096

0.021

Illustrating PageRank Computation

0.049 0.238

0.088

0.149

0.063

0.095

0.141

0.019

Illustrating PageRank Computation

0.050 0.232

0.091

0.149

0.060 0.066

0.094

0.143

0.096

0.019

Illustrating PageRank Computation

0.050 0.233

0.091

0.150

0.064

0.095

0.142

0.020

Illustrating PageRank Computation

0.050 0.234

0.090

0.148

0.058 0.065

0.095

0.143

0.097

0.019

Illustrating PageRank Computation

0.049 0.233

0.091

0.149

0.065

0.095

0.142

0.019

Illustrating PageRank Computation

0.050 0.233

0.091

0.149

0.058 0.065

0.095

0.143

0.097

0.019

Illustrating PageRank Computation

0.050 0.234

0.091

0.149

0.065

0.095

0.142

0.019

PageRank with Damping

May not always converge, or convergence may not be unique.

To fix this, the random surfer can at each steprandomly jumpto any page of the Web with some probabilityd (1−d:damping factor).

pr(i) = (︂

k→+∞lim ((1−d)G^T +dU)^kv )︂

i

whereU is the matrix with all _N¹ values withN the number of vertices.

Using PageRank to Score Query Results

PageRank: globalscore, independent of the query Can be used to raise the weight ofimportantpages:

weight(t,d) =tfidf(t,d)×pr(d), This can be directly incorporatedin the index.

Iterative Computation of PageRank

1. ComputeG(often stored as its adjacency list). Make sure lines sum to 1.

2. Letu be the uniform vector of sum 1,v =u,w thezerovector.

3. Whilev isdifferent enoughfromw:

Setw =v.

Setv = (1−d)G^Tv+du.

Outline

Information Retrieval PageRank

Conclusion

Conclusion

What you should remember

The inverted index model, associated tools and techniques Main ideas behind Fagin’s TA

PageRank, and its iterative computation Software

Most DBMS have text indexing capabilities (e.g., MySQL’s FULLTEXTindexes)

Apache Lucene, free software library for information retrieval To go further

Good textbooks [Manning et al., 2008, Chakrabarti, 2003]

Very influential papers on top-k algorithms and PageRank: [Fagin et al., 2001, Brin and Page, 1998]

Bibliography I

Sergey Brin and Lawrence Page. The anatomy of a large-scale hypertextual Web search engine. Computer Networks, 30(1–7):

107–117, April 1998.

Soumen Chakrabarti. Mining the Web: Discovering Knowledge from Hypertext Data. Morgan Kaufmann, San Fransisco, USA, 2003.

Ronald Fagin, Amnon Lotem, and Moni Naor. Optimal aggregation algorithms for middleware. InProc. PODS, Santa Barbara, USA, May 2001.

Christopher D. Manning, Prabhakar Raghavan, and Hinrich SchÃijtze.

Introduction to Information Retrieval. Cambridge University Press, Cambridge, United Kingdom, 2008. Available online at

http://informationretrieval.org.

Bibliography II

US National Archives and Records Administration. The Soundex indexing system.

http://www.archives.gov/genealogy/census/soundex.html, May 2007.

Licence de droits d’usage

Contexte public} avec modifications

Par le téléchargement ou la consultation de ce document, l’utilisateur accepte la licence d’utilisation qui y est attachée, telle que détaillée dans les dispositions suivantes, et s’engage à la respecter intégralement.

La licence confère à l’utilisateur un droit d’usage sur le document consulté ou téléchargé, totalement ou en partie, dans les conditions définies ci-après et à l’exclusion expresse de toute utilisation commerciale.

Le droit d’usage défini par la licence autorise un usage à destination de tout public qui comprend : – le droit de reproduire tout ou partie du document sur support informatique ou papier,

– le droit de diffuser tout ou partie du document au public sur support papier ou informatique, y compris par la mise à la disposition du public sur un réseau numérique,

– le droit de modifier la forme ou la présentation du document,

– le droit d’intégrer tout ou partie du document dans un document composite et de le diffuser dans ce nouveau document, à condition que : – L’auteur soit informé.

Les mentions relatives à la source du document et/ou à son auteur doivent être conservées dans leur intégralité.

Le droit d’usage défini par la licence est personnel et non exclusif.

Tout autre usage que ceux prévus par la licence est soumis à autorisation préalable et expresse de l’auteur :sitepedago@telecom-paristech.fr