December7th,2015 AntoineAmarilli ,SilviuManiu UncertainDataManagementRelationalProbabilisticDatabaseModels

(1)

Probabilistic instances TID BID pc-tables Conclusion

Uncertain Data Management

Relational Probabilistic Database Models

Antoine Amarilli¹, Silviu Maniu²

1Télécom ParisTech

2LRI

December 7th, 2015

(2)

Uncertain instances

Remember fromlast class:

Fix a finite set of possible tuplesof same arity A possible world: a subset of the possible tuples An uncertain relation: set ofpossible worlds

U₁

date teacher room 04 Silviu C017 04 Antoine C017 04 Antoine C47 11 Silviu C017 11 Silviu C47 11 Antoine C017

U₂

(4)

Uncertain instances

Fix a finite set of possible tuplesof same arity A possible world: a subset of the possible tuples An uncertain relation: set ofpossible worlds

U₁

U₂

(5)

Probabilistic instances

Support U: uncertain relation

Probability distributionπ onU: FunctionfromU to reals in[0,1] It mustsum upto1: ∑

I∈Uπ(I) = 1

U₁

π(U₁) = 0.8

U₂

π(U₂) = 0.2

(6)

Probabilistic instances

Support U: uncertain relation Probability distributionπ onU:

FunctionfromU to reals in[0,1] It mustsum upto1: ∑

I∈Uπ(I) = 1

U₁

π(U₁) = 0.8

U₂

π(U₂) = 0.2

(7)

Probabilistic instances

FunctionfromU to reals in[0,1]

It mustsum upto1: ∑

I∈Uπ(I) = 1

U₁

π(U₁) = 0.8

U₂

π(U₂) = 0.2

(8)

Probabilistic instances

I∈Uπ(I) = 1

U₁

π(U₁) = 0.8

U₂

π(U₂) = 0.2

(9)

Probabilistic instances

I∈Uπ(I) = 1

U₁

U₂

(10)

What about NULLs?

Remember that last time we saw:

Codd-tables andv-tablesandc-tables, with NULLs Boolean c-tables, with NULLsonly in conditions

→ Boolean variables

→ We focus for probabilities on models like Boolean c-tables

→ Easier to define probabilities on a finite space!

(11)

What about NULLs?

Remember that last time we saw:

Codd-tables andv-tablesandc-tables, with NULLs Boolean c-tables, with NULLsonly in conditions

→ Boolean variables

→ We focus for probabilities on models like Boolean c-tables

→ Easier to define probabilities on a finite space!

(12)

Relational algebra on uncertain instances

Extend relational algebra operators to uncertain instances The possible worldsof the result should be...

take allpossible worldsin the supports of the inputs apply the operation and get thepossible outputs

U₁ 04 S. C017 11 S. C47

U₂ 11 A. C017

∪

V₁

V₂ 11 A. C017

=

04 S. C017 11 S. C47 04 S. C017 11 S. C47 11 A. C017 11 A. C017

(13)

Relational algebra on uncertain instances

U₁ 04 S. C017 11 S. C47

U₂ 11 A. C017

∪

V₁

V₂ 11 A. C017

=

04 S. C017 11 S. C47 04 S. C017 11 S. C47 11 A. C017 11 A. C017

(14)

Relational algebra on uncertain instances

U₁ 04 S. C017 11 S. C47

U₂ 11 A. C017

∪

V₁

V₂ 11 A. C017

=

04 S. C017 11 S. C47 04 S. C017 11 S. C47 11 A. C017 11 A. C017

(15)

Relational algebra on uncertain instances

U₁ 04 S. C017 11 S. C47

U₂ 11 A. C017

∪

V₁

V₂ 11 A. C017

=

04 S. C017 11 S. C47 04 S. C017 11 S. C47 11 A. C017 11 A. C017

(16)

Relational algebra on uncertain instances

U₁ 04 S. C017 11 S. C47

U₂ 11 A. C017

∪

V₁

V₂ 11 A. C017

=

04 S. C017 11 S. C47 04 S. C017 11 S. C47 11 A. C017 11 A. C017

(17)

Relational algebra on uncertain instances

U₁ 04 S. C017 11 S. C47

U₂ 11 A. C017

∪

V₁

V₂ 11 A. C017

=

04 S. C017 11 S. C47 04 S. C017 11 S. C47 11 A. C017

(18)

Relational algebra on probabilistic instances

Let’s adapt relational algebra to probabilistic instances The possible worldsof the result should be...

take allpossible worldsof the inputs

apply the operation and get apossible output The probabilityof each possible world should be...

considerall input possible worldsthat give it sum up theirprobabilities

(19)

Relational algebra on probabilistic instances

apply the operation and get apossible output

The probabilityof each possible world should be... considerall input possible worldsthat give it sum up theirprobabilities

(20)

Relational algebra on probabilistic instances

apply the operation and get apossible output The probabilityof each possible world should be...

considerall input possible worldsthat give it sum up theirprobabilities

(21)

Example of relational algebra on probabilistic instances

U₁ 04 S. C017 11 S. C47

π(U₁) = 0.8

U₂ 11 A. C017

π(U₂) = 0.2

∪

V₁

π(V₁) = 0.9

V₂ 11 A. C017

π(V₂) = 0.1

=

W₁ 04 S. C017 11 S. C47

π(W₁) = 0.8·0.9

W₂ 04 S. C017 11 S. C47 11 A. C017

π(W₁) = 0.8·0.1

W₃ 11 A. C017

π(W₁) = 0.2·0.9 +0.2·0.1

(22)

Example of relational algebra on probabilistic instances

U₁ 04 S. C017 11 S. C47

π(U₁) = 0.8

U₂ 11 A. C017

π(U₂) = 0.2

∪

V₁

π(V₁) = 0.9

V₂ 11 A. C017

π(V₂) = 0.1

=

W₁ 04 S. C017 11 S. C47

π(W₁) = 0.8·0.9

W₂ 04 S. C017 11 S. C47 11 A. C017

π(W₁) = 0.8·0.1

W₃ 11 A. C017

π(W₁) = 0.2·0.9 +0.2·0.1

(23)

Example of relational algebra on probabilistic instances

U₁ 04 S. C017 11 S. C47

π(U₁) = 0.8

U₂ 11 A. C017

π(U₂) = 0.2

∪

V₁

π(V₁) = 0.9

V₂ 11 A. C017

π(V₂) = 0.1

=

W₁ 04 S. C017 11 S. C47

π(W₁) = 0.8·0.9

W₂ 04 S. C017 11 S. C47 11 A. C017

π(W₁) = 0.8·0.1

W₃ 11 A. C017

π(W₁) = 0.2·0.9 +0.2·0.1

(24)

Example of relational algebra on probabilistic instances

U₁ 04 S. C017 11 S. C47

π(U₁) = 0.8

U₂ 11 A. C017

π(U₂) = 0.2

∪

V₁

π(V₁) = 0.9

V₂ 11 A. C017

π(V₂) = 0.1

=

W₁ 04 S. C017 11 S. C47

π(W₁) = 0.8·0.9

W₂ 04 S. C017 11 S. C47 11 A. C017

π(W₁) = 0.8·0.1

W₃ 11 A. C017

π(W₁) = 0.2·0.9 +0.2·0.1

(25)

Example of relational algebra on probabilistic instances

U₁ 04 S. C017 11 S. C47

π(U₁) = 0.8

U₂ 11 A. C017

π(U₂) = 0.2

∪

V₁

π(V₁) = 0.9

V₂ 11 A. C017

π(V₂) = 0.1

=

W₁ 04 S. C017 11 S. C47

π(W₁) = 0.8·0.9

W₂ 04 S. C017 11 S. C47 11 A. C017

π(W₁) = 0.8·0.1

W₃ 11 A. C017

π(W₁) = 0.2·0.9 +0.2·0.1

(26)

Example of relational algebra on probabilistic instances

U₁ 04 S. C017 11 S. C47

π(U₁) = 0.8

U₂ 11 A. C017

π(U₂) = 0.2

∪

V₁

π(V₁) = 0.9

V₂ 11 A. C017

π(V₂) = 0.1

=

W₁ 04 S. C017 11 S. C47

π(W₁) = 0.8·0.9

W₂ 04 S. C017 11 S. C47 11 A. C017

π(W₁) = 0.8·0.1

W₃ 11 A. C017

π(W₁) = 0.2·0.9 +0.2·0.1

(27)

Example of relational algebra on probabilistic instances

U₁ 04 S. C017 11 S. C47

π(U₁) = 0.8

U₂ 11 A. C017

π(U₂) = 0.2

∪

V₁

π(V₁) = 0.9

V₂ 11 A. C017

π(V₂) = 0.1

=

W₁ 04 S. C017 11 S. C47

π(W₁) = 0.8·0.9 W₂ 04 S. C017 11 S. C47 11 A. C017

π(W₁) = 0.8·0.1

W₃ 11 A. C017

π(W₁) = 0.2·0.9 +0.2·0.1

(28)

Example of relational algebra on probabilistic instances

U₁ 04 S. C017 11 S. C47

π(U₁) = 0.8

U₂ 11 A. C017

π(U₂) = 0.2

∪

V₁

π(V₁) = 0.9

V₂ 11 A. C017

π(V₂) = 0.1

=

W₁ 04 S. C017 11 S. C47

π(W₁) = 0.8·0.9

W₂ 04 S. C017 11 S. C47 11 A. C017

π(W₁) = 0.8·0.1 W₃ 11 A. C017

π(W₁) = 0.2·0.9 +0.2·0.1

(29)

Example of relational algebra on probabilistic instances

U₁ 04 S. C017 11 S. C47

π(U₁) = 0.8

U₂ 11 A. C017

π(U₂) = 0.2

∪

V₁

π(V₁) = 0.9

V₂ 11 A. C017

π(V₂) = 0.1

=

W₁ 04 S. C017 11 S. C47

π(W₁) = 0.8·0.9

W₂ 04 S. C017 11 S. C47 11 A. C017

π(W₁) = 0.8·0.1

W₃ 11 A. C017

π(W₁) = 0.2·0.9 +0.2·0.1

(30)

Example of relational algebra on probabilistic instances

U₁ 04 S. C017 11 S. C47

π(U₁) = 0.8

U₂ 11 A. C017

π(U₂) = 0.2

∪

V₁

π(V₁) = 0.9

V₂ 11 A. C017

π(V₂) = 0.1

=

W₁ 04 S. C017 11 S. C47 π(W₁) = 0.8·0.9

W₂ 04 S. C017 11 S. C47 11 A. C017

π(W₁) = 0.8·0.1

W₃ 11 A. C017

π(W₁) = 0.2·0.9 +0.2·0.1

(31)

Example of relational algebra on probabilistic instances

U₁ 04 S. C017 11 S. C47

π(U₁) = 0.8

U₂ 11 A. C017

π(U₂) = 0.2

∪

V₁

π(V₁) = 0.9

V₂ 11 A. C017

π(V₂) = 0.1

=

W₁ 04 S. C017 11 S. C47 π(W₁) = 0.8·0.9

W₂ 04 S. C017 11 S. C47 11 A. C017 π(W₁) = 0.8·0.1

W₃ 11 A. C017

π(W₁) = 0.2·0.9 +0.2·0.1

(32)

Example of relational algebra on probabilistic instances

U₁ 04 S. C017 11 S. C47

π(U₁) = 0.8

U₂ 11 A. C017

π(U₂) = 0.2

∪

V₁

π(V₁) = 0.9

V₂ 11 A. C017

π(V₂) = 0.1

=

W₁ 04 S. C017 11 S. C47 π(W₁) = 0.8·0.9

W₂ 04 S. C017 11 S. C47 11 A. C017 π(W₁) = 0.8·0.1

W₃ 11 A. C017 π(W₁) = 0.2·0.9

+0.2·0.1

(33)

Example of relational algebra on probabilistic instances

U₁ 04 S. C017 11 S. C47

π(U₁) = 0.8

U₂ 11 A. C017

π(U₂) = 0.2

∪

V₁

π(V₁) = 0.9

V₂ 11 A. C017

π(V₂) = 0.1

=

W₁ 04 S. C017 11 S. C47 π(W₁) = 0.8·0.9

W₂ 04 S. C017 11 S. C47 11 A. C017 π(W₁) = 0.8·0.1

W₃ 11 A. C017 π(W₁) = 0.2·0.9

(34)

Representation system

Remember that if we have Npossible tuples

→ there are2^N possible instances

→ there are2²^N possible uncertain instances

→ writing outan uncertain instance isexponential

→ Last time we saw Boolean c-tablesas a concise way to representuncertain instances

For probabilistic instances:

→ there areinfinitely manypossible instances

→ writing outa probabilistic instance is stillexponential

→ How to representprobabilistic instances?

(35)

Representation system

(36)

Representation system

(37)

Representation system

(38)

Representation system

(39)

Representation system

(40)

Representation system

(41)

Tuple-independent databases

The simplestmodel: tuple-independent databases Annotate each instance factwith a probability

U

date teacher room 04 Silviu C017

0.8

04 Antoine C017

0.2

11 Silviu C017

1

(43)

Tuple-independent databases

U

date teacher room

04 Silviu C017

0.8

04 Antoine C017

0.2

11 Silviu C017

1

(44)

Tuple-independent databases

U

04 Silviu C017 0.8

04 Antoine C017 0.2

11 Silviu C017 1

(45)

Semantics of TID

Each tuple is kept or discardedwith the probability Probabilistic choices are independentacross tuples

U

04 Silviu C017 0.8

04 Antoine C017 0.2

11 Silviu C017 1

U

04 Silviu C017

04 Antoine C017

11 Silviu C017

What’s theprobabilityof this outcome? 0.8×(1−0.2)×1

(46)

Semantics of TID

U

04 Silviu C017 0.8

04 Antoine C017 0.2

11 Silviu C017 1

U

04 Silviu C017

04 Antoine C017

11 Silviu C017

(47)

Semantics of TID

U

04 Silviu C017 0.8

04 Antoine C017 0.2

11 Silviu C017 1

U

04 Silviu C017

04 Antoine C017

11 Silviu C017

(48)

Semantics of TID

U

04 Silviu C017 0.8

04 Antoine C017 0.2

11 Silviu C017 1

U

04 Silviu C017

04 Antoine C017

11 Silviu C017

(49)

Semantics of TID

U

04 Silviu C017 0.8

04 Antoine C017 0.2

11 Silviu C017 1

U

04 Silviu C017

04 Antoine C017

11 Silviu C017

(50)

Semantics of TID

U

04 Silviu C017 0.8

04 Antoine C017 0.2

11 Silviu C017 1

U

04 Silviu C017

04 Antoine C017

11 Silviu C017

(51)

Semantics of TID

U

04 Silviu C017 0.8

04 Antoine C017 0.2

11 Silviu C017 1

U

04 Silviu C017

04 Antoine C017

11 Silviu C017

What’s theprobabilityof this outcome?

0.8×(1−0.2)×1

(52)

Semantics of TID

U

04 Silviu C017 0.8

04 Antoine C017 0.2

11 Silviu C017 1

U

04 Silviu C017

04 Antoine C017

11 Silviu C017

0.8

×(1−0.2)×1

(53)

Semantics of TID

U

04 Silviu C017 0.8

04 Antoine C017 0.2

11 Silviu C017 1

U

04 Silviu C017

04 Antoine C017

11 Silviu C017

0.8×

(1−0.2)×1

(54)

Semantics of TID

U

04 Silviu C017 0.8

04 Antoine C017 0.2

11 Silviu C017 1

U

04 Silviu C017

04 Antoine C017

11 Silviu C017

0.8×(1−0.2)

×1

(55)

Semantics of TID

U

04 Silviu C017 0.8

04 Antoine C017 0.2

11 Silviu C017 1

U

04 Silviu C017

04 Antoine C017

11 Silviu C017

0.8×(1−0.2)×1

(56)

Getting a probability distribution

Thesemanticsof a TID instance is a probabilistic instance...

→ thepossible worlds are the subsets

→ always keeping tuples withprobability 1 Formally, for a TID instance I, theprobability of J:

we must haveJ⊆I

product of p_t for each tuple tkept in J

product of 1−p_t for each tuple t not keptin J

(57)

Getting a probability distribution

→ always keeping tuples withprobability 1

Formally, for a TID instance I, theprobability of J: we must haveJ⊆I

(58)

Getting a probability distribution

→ always keeping tuples withprobability 1 Formally, for a TID instanceI, theprobability ofJ:

we must haveJ⊆I

(59)

Getting a probability distribution

→ always keeping tuples withprobability 1 Formally, for a TID instanceI, theprobability ofJ:

we must haveJ⊆I

product of 1−p_t for each tuple tnot kept in J

(60)

Is it a probability distribution?

Do the probabilities alwayssum to 1?

0 tuples: vacuous 1 tuple: p+ (1−p) = 1

2 tuples: p₁p₂+p₁(1−p₂) + (1−p₁)p₂+ (1−p₁)(1−p₂)

→ p₁(p₂+ (1−p₂)) + (1−p₁)(p₂+ (1−p₂))

→ (p1+ (1−p1))(p2+ (1−p2))

→ p2+ (1−p2)

More tuples: p₁(· · ·) + (1−p₁)(· · ·)andinduction hypothesis

(61)

Is it a probability distribution?

0 tuples: vacuous

1 tuple: p+ (1−p) = 1

2 tuples: p₁p₂+p₁(1−p₂) + (1−p₁)p₂+ (1−p₁)(1−p₂)

→ p₁(p₂+ (1−p₂)) + (1−p₁)(p₂+ (1−p₂))

→ (p1+ (1−p1))(p2+ (1−p2))

→ p2+ (1−p2)

(62)

Is it a probability distribution?

2 tuples: p₁p₂+p₁(1−p₂) + (1−p₁)p₂+ (1−p₁)(1−p₂)

→ p₁(p₂+ (1−p₂)) + (1−p₁)(p₂+ (1−p₂))

→ (p1+ (1−p1))(p2+ (1−p2))

→ p2+ (1−p2)

(63)

Is it a probability distribution?

2 tuples: p₁p₂+p₁(1−p₂) + (1−p₁)p₂+ (1−p₁)(1−p₂)

→ p₁(p₂+ (1−p₂)) + (1−p₁)(p₂+ (1−p₂))

→ (p1+ (1−p1))(p2+ (1−p2))

→ p2+ (1−p2)

(64)

Is it a probability distribution?

2 tuples: p₁p₂+p₁(1−p₂) + (1−p₁)p₂+ (1−p₁)(1−p₂)

→ p1(p2+ (1−p2)) + (1−p1)(p2+ (1−p2))

→ (p1+ (1−p1))(p2+ (1−p2))

→ p2+ (1−p2)

(65)

Is it a probability distribution?

2 tuples: p₁p₂+p₁(1−p₂) + (1−p₁)p₂+ (1−p₁)(1−p₂)

→ p1(p2+ (1−p2)) + (1−p1)(p2+ (1−p2))

→ (p1+ (1−p1))(p2+ (1−p2))

→ p2+ (1−p2)

(66)

Is it a probability distribution?

2 tuples: p₁p₂+p₁(1−p₂) + (1−p₁)p₂+ (1−p₁)(1−p₂)

→ p1(p2+ (1−p2)) + (1−p1)(p2+ (1−p2))

→ (p1+ (1−p1))(p2+ (1−p2))

→ p2+ (1−p2)

(67)

Is it a probability distribution?

2 tuples: p₁p₂+p₁(1−p₂) + (1−p₁)p₂+ (1−p₁)(1−p₂)

→ p1(p2+ (1−p2)) + (1−p1)(p2+ (1−p2))

→ (p1+ (1−p1))(p2+ (1−p2))

→ p2+ (1−p2)

(68)

Strong representation system

Uncertain instance: set of possible worlds

Uncertainty framework: concise way to represent uncertain instances Query language: here, relational algebra Definition (Strong representation system) For any query in the language,

on uncertain instances represented in the framework, the uncertain instance obtained by evaluating the query can also be represented in the framework.

→ Are TID instances a strong representation system?

(69)

Strong representation system

Uncertain instance: set of possible worlds Uncertainty framework: concise way to represent

uncertain instances

Query language: here, relational algebra Definition (Strong representation system) For any query in the language,

(70)

Strong representation system

uncertain instances Query language: here, relational algebra

Definition (Strong representation system) For any query in the language,

(71)

Strong representation system

uncertain instances Query language: here, relational algebra Definition (Strong representation system) For any query in the language,

(72)

Strong representation system

on uncertain instances represented in the framework,

the uncertain instance obtained by evaluating the query can also be represented in the framework.

(73)

Strong representation system

on uncertain instances represented in the framework, the uncertain instance obtained by evaluating the query

can also be represented in the framework.

(74)

Strong representation system

(75)

Strong representation system

(76)

Implementing select

U

04 Silviu C47 0.8

04 Antoine C47 0.2

11 Silviu C47 1

σteacher=“Silviu”(U) date teacher room

04 Silviu C47 0.8

11 Silviu C47 1

→ Is this correct?

→ So far, so good.

(77)

Implementing select

U

04 Silviu C47 0.8

04 Antoine C47 0.2

11 Silviu C47 1

04 Silviu C47 0.8

11 Silviu C47 1

(78)

Implementing select

U

04 Silviu C47 0.8

04 Antoine C47 0.2

11 Silviu C47 1

04 Silviu C47 0.8

11 Silviu C47 1

(79)

Implementing select

U

04 Silviu C47 0.8

04 Antoine C47 0.2

11 Silviu C47 1

04 Silviu C47 0.8

11 Silviu C47 1

(80)

Implementing select

U

04 Silviu C47 0.8

04 Antoine C47 0.2

11 Silviu C47 1

04 Silviu C47 0.8

11 Silviu C47 1

→

(81)

Implementing project

U

04 Silviu C47 0.8

04 Antoine C47 0.2

11 Silviu C47 1

11 Antoine C47 0.1

18 Silviu C47 0.9

π_date(U) date

04

1−(1−0.2)·(1−0.8)

11

1

18

0.9

→ So far, so good

December7th,2015 AntoineAmarilli ,SilviuManiu UncertainDataManagementRelationalProbabilisticDatabaseModels

Uncertain Data Management

Relational Probabilistic Database Models

Table of contents

Uncertain instances

Uncertain instances

Probabilistic instances

Probabilistic instances

Probabilistic instances

Probabilistic instances

Probabilistic instances

What about NULLs?

What about NULLs?

Relational algebra on uncertain instances

∪

=

Relational algebra on uncertain instances

∪

=

Relational algebra on uncertain instances

∪

=

Relational algebra on uncertain instances

∪

=

Relational algebra on uncertain instances

∪

=

Relational algebra on uncertain instances

∪

=

Relational algebra on probabilistic instances

Relational algebra on probabilistic instances

Relational algebra on probabilistic instances

Example of relational algebra on probabilistic instances

∪

=

Example of relational algebra on probabilistic instances

∪

=

Example of relational algebra on probabilistic instances

∪

=

Example of relational algebra on probabilistic instances

∪

=

Example of relational algebra on probabilistic instances

∪

=

Example of relational algebra on probabilistic instances

∪

=

Example of relational algebra on probabilistic instances

∪

=

Example of relational algebra on probabilistic instances

∪

=

Example of relational algebra on probabilistic instances

∪

=

Example of relational algebra on probabilistic instances

∪

=

Example of relational algebra on probabilistic instances

∪

=

Example of relational algebra on probabilistic instances

∪

=

Example of relational algebra on probabilistic instances

∪

=

Representation system

Representation system

Representation system

Representation system

Representation system

Representation system

Representation system