April20th,2015 TovaMilo PierreSenellart AntoineAmarilli YaelAmsterdamer Top- k QueryingofIncompleteDataunderOrderConstraints

(1)

Introduction Approach Complexity results Conclusion

Top-k Querying of Incomplete Data under Order Constraints

Antoine Amarilli¹ Yael Amsterdamer² Tova Milo² Pierre Senellart^1,3

1Télécom ParisTech, Paris, France

2Tel Aviv University, Tel Aviv, Israel

3National University of Singapore

April 20th, 2015

(2)

Introduction

Taxonomy of items for a store

with categories. Ask the crowd to classify items

Products

Clothing Electronics

Shoes Cell Phones

TVs Watches Diving Gear

Wearable Devices

Smartphones Diving Watches

Sports

Monotonicity: compatibility increases as we go up. Best categories? Naiveanswer... Clever answer...

2/16

(3)

Introduction

Taxonomy of items for a store withcategories.

Ask the crowd to classify items

Products

Shoes Cell Phones

Wearable Devices

Sports

(4)

Introduction

Ask the crowd to classify items Products

Shoes Cell Phones

Wearable Devices

Sports

2/16

(5)

Introduction

Shoes Cell Phones

Wearable Devices

Sports

(6)

Introduction

Shoes Cell Phones

Wearable Devices

Sports

2/16

(7)

Introduction

Shoes Cell Phones

Wearable Devices

Sports

.1

(8)

Introduction

Shoes Cell Phones

Wearable Devices

Sports

.1

2/16

(9)

Introduction

Shoes Cell Phones

Wearable Devices

Sports

.1

.9

(10)

Introduction

Shoes Cell Phones

Wearable Devices

Sports

.1

.9

2/16

(11)

Introduction

Shoes Cell Phones

Wearable Devices

Sports

.1

.9

.5

(12)

Introduction

Shoes Cell Phones

Wearable Devices

Sports

.1

.9

.5

2/16

(13)

Introduction

Shoes Cell Phones

Wearable Devices

Sports

.1

.9

.5

(14)

Introduction

Products

Shoes Cell Phones

Wearable Devices

Sports

.1

.9

.5

Monotonicity: compatibility increases as we go up.

Best categories? Naiveanswer... Clever answer...

2/16

(15)

Introduction

Products

Shoes Cell Phones

Wearable Devices

Sports

.1

.9

.5

Best categories?

Naiveanswer... Clever answer...

(16)

Introduction

Products

Shoes Cell Phones

Wearable Devices

Sports

.1

.9

.5

Best categories? Naiveanswer...

Clever answer...

2/16

(17)

Introduction

Products

Shoes Cell Phones

Wearable Devices

Sports

.1

.9

.5

Best categories? Naiveanswer...

Clever answer...

(18)

Introduction

Products

Shoes Cell Phones

Wearable Devices

Sports

.1

.9

.5

2/16

(19)

Introduction

Products

Shoes Cell Phones

Wearable Devices

Sports

.1

.9

.5

=.5

(20)

Introduction

Products

Shoes Cell Phones

Wearable Devices

Sports

.1

.9

.5

=.5

≥.9

2/16

(21)

Introduction

Products

Shoes Cell Phones

Wearable Devices

Sports

.1

.9

.5

=.5

≥.9

≥.5

(22)

Introduction

Products

Shoes Cell Phones

Wearable Devices

Sports

.1

.9

.5

=.5

≥.9

≥.5

2/16

(23)

Problem statement

Taxonomy:

Partial order, i.e., directed acyclic graph Someend categoriesdistinguished

Compatibility values:

Tosimplify, assume0≤ • ≤1

Monotonicitywith respect to the taxonomy Some valuesknown, otherunknown

→ How to completethe missing values?

→ What are the top-k and their expected values?

→ What is our confidence in the answer?

(24)

Problem statement

Taxonomy:

Partial order, i.e., directed acyclic graph Someend categoriesdistinguished

Compatibility values:

Tosimplify, assume0≤ • ≤1

Monotonicitywith respect to the taxonomy Some valuesknown, otherunknown

→ How to completethe missing values?

→ What are the top-k and their expected values?

→ What is our confidence in the answer?

3/16

(25)

Admissible polytope

Each unknown value has one variable

Consider thespace of all possible assignments It is a polytope(linear constraints)

Example:

0≤x≤.8,.2≤y≤1

y≤x

5/16

(27)

Admissible polytope

Example:

0≤x≤.8,.2≤y≤1

y≤x

0 0 1

1 y

.2 x

.8

(28)

Admissible polytope

Example:

0≤x≤.8,.2≤y≤1 y≤x

0 0 1

1 y

.2 x

.8

5/16

(29)

Probabilistic formalization

Consider theadmissible polytope Take theuniform distribution on it

(Intuitively, all possible assignments areequiprobable)

→ What is the average value of each variable?

(Possible extensions: variance, marginal distribution...)

(30)

Probabilistic formalization

6/16

(31)

Probabilistic formalization

(32)

Easy case: total order

0 ≤ • ≤ • ≤ .3 ≤ • ≤ 1

How to complete this? Any ideas? ...

→ Linear interpolation!

(Formarginal distribution: order statistics, Beta distribution)

7/16

(33)

Easy case: total order

0 ≤ • ≤ • ≤ .3 ≤ • ≤ 1 How to complete this? Any ideas? ...

(34)

Easy case: total order

0 ≤ • ≤ • ≤ .3 ≤ • ≤ 1 How to complete this? Any ideas? ...

7/16

(35)

Easy case: total order

0 ≤ .1 ≤ .2 ≤ .3 ≤ .65 ≤ 1 How to complete this? Any ideas? ...

(36)

Easy case: total order

0 ≤ .1 ≤ .2 ≤ .3 ≤ .65 ≤ 1 How to complete this? Any ideas? ...

7/16

(37)

General case

Consider thetaxonomy: partial order

Consider all possible total orders (Ties can be madenegligible) Solve each total orderas before

Take theweighted average of the orders Total order weight: probabilityof this order

→ Gives the averagefor the actual taxonomy!

(38)

General case

Consider thetaxonomy: partial order Consider all possible total orders

(Ties can be madenegligible) Solve each total orderas before

8/16

(39)

General case

Consider thetaxonomy: partial order Consider all possible total orders (Ties can be madenegligible)

Solve each total orderas before

(40)

General case

Consider thetaxonomy: partial order Consider all possible total orders (Ties can be madenegligible) Solve each total orderas before

8/16

(41)

General case

Take theweighted average of the orders

Total order weight: probabilityof this order

(42)

General case

8/16

(43)

General case

(44)

Example

y y^′

z

x

.3

Possibility 1: 0≤x≤y≤y^′ ≤z≤1

→ Expected values: x=.1,y=.2,z=.65 Probability:

→ Volume of0≤x≤y≤.3 times volume of.3≤z≤1

→ ^.3_2!² and ^1−.3_1!

Possibility 2: 0≤x≤y^′ ≤y≤z≤1

→ Expected values of y: .2 and.533

→ Normalized probabilities: .3 and.7

→ Final result: yhas expected value .43

9/16

(45)

Example

y y^′

z

x

.3

→ Expected values: x=.1,y=.2,z=.65 Probability:

→ ^.3_2!² and ^1−.3_1!

(46)

Example

y y^′

z

x

.3

→ Expected values: x=.1,y=.2, z=.65

Probability:

→ ^.3_2!² and ^1−.3_1!

9/16

(47)

Example

y y^′

z

x

.3

→ Expected values: x=.1,y=.2, z=.65 Probability:

→ ^.3_2!² and ^1−.3_1!

(48)

Example

y y^′

z

x

.3

→ ^.3_2!² and ^1−.3_1!

9/16

(49)

Example

y y^′

z

x

.3

→ ^.3_2!² and ^1−.3_1!

(50)

Example

y y^′

z

x

.3

→ ^.3_2!² and ^1−.3_1!

9/16

(51)

Example

y y^′

z

x

.3

→ ^.3_2!² and ^1−.3_1!

(52)

Example

y y^′

z

x

.3

→ ^.3_2!² and ^1−.3_1!

9/16

(53)

Example

y y^′

z

x

.3

→ ^.3_2!² and ^1−.3_1!

(54)

Complexity of the bruteforce algorithm

Complexityof the previous algorithm:

PTIME in the number of compatible total orders (aka.linear extensions)

They can be enumeratedin PTIME in their number

However there may beexponentially many

→ Volume computation for convex polytopes is #P-hard

→ Can we show hardness of our problems?

(56)

Complexity of the bruteforce algorithm

They can be enumeratedin PTIME in their number However there may beexponentially many

11/16

(57)

Complexity of the bruteforce algorithm

(58)

Complexity of the bruteforce algorithm

11/16

(59)

Completeness results

Existing results [Brightwell and Winkler, 1991]

→ Counting the number oflinear extensionsis #P-hard

→ Expected rankcomputation is #P-hard

→ Computing the expected value in our setting is #P-hard

→ Connection between expectedrankandvalue

→ Computing the top-kis #P-hard evenwithout values!

→ Binary searchagainst known values to find expected value

→ Uses scheme forrational search[Papadimitriou, 1979]

→ FP^#P-membership of our problems

→ Non-trivial as polytope volume computation isnotin FP^#P!

(60)

Completeness results

12/16

(61)

Completeness results

(62)

Completeness results

12/16

(63)

Tractable cases

Intractable for arbitrary taxonomies Are theretractable subcases?

Common situation: taxonomy is a tree

→ PTIME expected value computation

(Compute the marginal distributions as piecewise polynomials)

(64)

Tractable cases

Intractable for arbitrary taxonomies Are theretractable subcases?

Common situation: taxonomy is a tree

→ PTIME expected value computation

(Compute the marginal distributions as piecewise polynomials)

13/16

(65)

Conclusion

Formal definitionof top-k queries on incomplete data Also generalizes linear interpolationto partial orders

Principled algorithmfor top-k Hardness results for these problems

Tractable subcases for tree-shaped taxonomies Open questions:

→ Is this theright definition?

→ Are thereother tractable cases?

→ What aboutchoosing the next queries?

Thanks for your attention!

Smartwatch photo on slide 2: Bostwickenator, CC-BY-SA 3.0 https://en.wikipedia.org/wiki/File:WimmOneInBand.jpg

15/16

(67)

Conclusion

Formal definitionof top-k queries on incomplete data Also generalizes linear interpolationto partial orders Principled algorithmfor top-k

Hardness results for these problems

(68)

Conclusion

15/16

(69)

Conclusion

Tractable subcases for tree-shaped taxonomies

Open questions:

(70)

Conclusion

15/16

(71)

Conclusion

(72)

References I

Brightwell, G. and Winkler, P. (1991).

Counting linear extensions.

Order, 8(3):225–242.

Papadimitriou, C. H. (1979).

Efficient search for rationals.

Information Processing Letters, 8(1):1–4.

16/16

April20th,2015 TovaMilo PierreSenellart AntoineAmarilli YaelAmsterdamer Top- k QueryingofIncompleteDataunderOrderConstraints

Top-k Querying of Incomplete Data under Order Constraints

Introduction

Introduction

Introduction

Introduction

Introduction

Introduction

Introduction

Introduction

Introduction

Introduction

Introduction

Introduction

Introduction

Introduction

Introduction

Introduction

Introduction

Introduction

Introduction

Introduction

Introduction

Problem statement

Problem statement

Table of contents

Admissible polytope

Admissible polytope

Admissible polytope

Probabilistic formalization

Probabilistic formalization

Probabilistic formalization

Easy case: total order

Easy case: total order

Easy case: total order

Easy case: total order

Easy case: total order

General case

General case

General case

General case

General case

General case

General case

Example

Example

Example

Example

Example

Example

Example

Example

Example

Example

Table of contents

Complexity of the bruteforce algorithm

Complexity of the bruteforce algorithm

Complexity of the bruteforce algorithm

Complexity of the bruteforce algorithm

Completeness results

Completeness results

Completeness results

Completeness results

Tractable cases

Tractable cases

Table of contents

Conclusion

Conclusion

Conclusion

Conclusion

Conclusion

Conclusion

References I