Leveraging the structure of uncertain data

(1)

Leveraging the structure of uncertain data

Antoine Amarilli May 26, 2017

(2)

Example application: Subway routing

(3)

(4)

(5)

(6)

(7)

Database theory and query evaluation

13 6

23 14

Database

• (Hyper)graph

• Collection of ground facts

G(aa1, ab2), G(ab2, ac3), S(aa1, m4), S(ab2, rB), ...

(8)

13 6

23 14

Query

?

+

Database

• (Hyper)graph

G(aa ab G(ab ac

• Regular path

∀X(rm ∈ X ∧ ∀xy (Metro|RER)*

• Logic formula

|(Bus|Tram)*

(9)

13 6

23 14

Query

?

+

Database

• (Hyper)graph

G(aa1, ab2), G(ab2, ac3), S(aa1, m4), S(ab2, rB), ...

• Regular path

∀X(rm ∈ X ∧ ∀xy

i _Result

• TRUE/FALSE

↔ Model checking (Metro|RER)*

• Logic formula

(x ∈ X ∧ G(x, y) → y ∈ X)) → gn ∈ X

|(Bus|Tram)*

(10)

Probabilistic query evaluation

13 6

23 14

(11)

(12)

(13)

(14)

(15)

(16)

(17)

13 6

23 14

(18)

13 6

23 ^95% 14

5% 60

98%

2% 60

(19)

• (Hyper)graph

Probabilistic database

+ independent probabilities

13 6

23 ^95%14

5% 60

98%

2% 60

(20)

Query

?

+

• (Hyper)graph

• Regular path

• Logic formula

|(Bus|Tram)*

Probabilistic database

+ independent

13 6

23 ^95%14

5% 60

98%

2% 60

(21)

Query

?

+

• (Hyper)graph

• Regular path

• Logic formula

(x ∈ X ∧ G(x, y) → y ∈ X)) → gn ∈ X

|(Bus|Tram)*

Probabilistic database

+ independent probabilities

Probabilistic i

Result

• Probability according to the input distribution

13 6

23 ^95%14

5% 60

98%

2% 60

proba to be on time: 98%

(22)

Computational complexity

•

Computing paths on a large graph:

→ #P-hardcomputational complexity in thedatabase

(23)

50%

90%

42%

37%

90%

83%

78%

72%

•

Computing paths on a largeprobabilisticgraph:

→ ???

(24)

50%

90%

42%

37%

90%

83%

78%

72%

•

(25)

50%

90%

42%

37%

90%

83%

78%

72%

•

→ Exponentialnumber of possibilities

(26)

Idea: use the structure of data

→ Shortest path:very easyon alarge tree

(27)

(28)

(29)

(30)

(31)

(32)

(33)

Leveraging the structure of uncertain data

Does query evaluation on probabilistic data havelower complexity when thestructureof the data isclose to a tree?

In this talk:

•

Existing results onnon-probabilistic data:

• Tree automata, to evaluate queries on trees

• Treewidth, formalizes the notion of being “close to a tree”

• Courcelle’s theorem

•

^Introduce^{new tools}^and^results^:

• Provenance circuitsof tree automata on uncertain trees

• Applications toprobabilistic query evaluation

•

Other applications:Counting, enumeration, provenance...

(34)

In this talk:

•

(35)

In this talk:

•

(36)

In this talk:

•

(37)

In this talk:

•

(38)

In this talk:

•

(39)

In this talk:

•

(40)

In this talk:

•

(41)

In this talk:

•

(42)

Table of contents

Introduction Existing tools

Provenance circuits and probabilistic query evaluation Other applications

(43)

Non-probabilistic query evaluation on trees

Database: atreeT where nodes have a color from an alphabet

?

^Query^Q: asentencein monadic second-order logic (MSO)

•P (x)means “xis blue”

•x→ymeans “xis the parent ofy”

“Is there both a pink and a blue node?”

∃x y P (x)∧P (y)

i

Result: TRUE/FALSE indicating if the treeT satisfies the queryQ

Computational complexityas a function ofT (the queryQisfixed)

(44)

?

^Query^Q^{: a}^sentence^{in monadic}

second-order logic (MSO)

∃x y P (x)∧P (y)

i

(45)

?

∃x y P (x)∧P (y)

i

(46)

?

∃x y P (x)∧P (y)

i

(47)

Tree automata Tree alphabet:

•

Bottom-up deterministictree automaton

•

“Is there both a pink and a blue node?”

•

^States:^{⊥,^B,^P,^>}

•

Final states:{>}

•

Initial function: ⊥ P B

•

Transitions(examples): P

P ⊥

> B P

⊥

⊥ Theorem [Thatcher and Wright, 1968]

MSOandtree automatahave the sameexpressive poweron trees Corollary

Non-probabilistic query evaluation of MSO on trees is inlinear time.

(48)

Tree automata

Tree alphabet:

•

^States:^{⊥,^B,^P,^>}

•

Final states:{>}

•

P ⊥

> B P

⊥

(49)

Tree automata

Tree alphabet:

•

^States:^{⊥,^B,^P,^>}

•

Final states:{>}

•

P ⊥

> B P

⊥

(50)

Tree automata

Tree alphabet:

•

^States:^{⊥,^B,^P,^>}

•

Final states:{>}

•

P ⊥

> B P

⊥

(51)

Tree automata

Tree alphabet:

•

^States:^{⊥,^B,^P,^>}

•

Final states:{>}

•

P ⊥

> B P

⊥

(52)

B

⊥ P ⊥

•

^States:^{⊥,^B,^P,^>}

•

Final states:{>}

•

P ⊥

> B P

⊥

(53)

B

⊥ P ⊥

•

^States:^{⊥,^B,^P,^>}

•

Final states:{>}

•

Transitions(examples):

P P ⊥

>

B P

⊥

Theorem [Thatcher and Wright, 1968]

(54)

B

⊥ P

P ⊥

•

^States:^{⊥,^B,^P,^>}

•

Final states:{>}

•

P P ⊥

>

B P

⊥

(55)

>

B

⊥ P

P ⊥

•

^States:^{⊥,^B,^P,^>}

•

Final states:{>}

•

P P ⊥

>

B P

⊥

(56)

>

B

⊥ P

P ⊥

•

^States:^{⊥,^B,^P,^>}

•

Final states:{>}

•

P P ⊥

>

B P

⊥

MSOandtree automatahave the sameexpressive poweron trees

Corollary

(57)

>

B

⊥ P

P ⊥

•

^States:^{⊥,^B,^P,^>}

•

Final states:{>}

•

P P ⊥

>

B P

⊥

(58)

?

∃x y P (x)∧P (y)

i

Result: TRUE/FALSE indicating ifTsatisfies the queryQ

Computational complexityas a function of thetreeT (the queryQisfixed)

(59)

Non-probabilistic query evaluation on treelike data

Database: atreelike databaseT

???

?

(Metro|RER)^∗

|(Bus|Tram)^∗

i

Result: TRUE/FALSE indicating ifTsatisfies the queryQ

Computational complexityas a function of thetreeT (the queryQisfixed)

(60)

Treewidth

Treewidth by example:

•

^Treeshave treewidth1

•

^Cycleshave treewidth2

•

^k-cliques^and^(k⁻^1)-gridshave treewidthk−1

→ Treelike: thetreewidthis bounded by aconstant

(61)

Treewidth

•

(62)

Treewidth

•

(63)

Treewidth

•

(64)

Treewidth

•

(65)

Treewidth

•

(66)

Treewidth

•

(67)

Treewidth

•

(68)

Treewidth

•

(69)

Treewidth

•

(70)

Treewidth

•

(71)

Treewidth

•

(72)

Treewidth

•

(73)

Treewidth

•

(74)

Treewidth

•

^k-cliques ^and^(k⁻^1)-gridshave treewidthk−1

(75)

Treewidth

•

^k-cliques ^and^(k⁻^1)-gridshave treewidthk−1

(76)

Courcelle’s theorem

(RER|metro)*

|(bus|tram)*

MSO query Treelike data

Theorem [Courcelle, 1990]

For any fixed Boolean MSO queryQandk∈N, given a databaseDof treewidth≤k,

we can compute inlinear timeinDwhetherDsatisfiesQ

(77)

Tree automaton (RER|metro)*

|(bus|tram)*

(78)

Tree automaton Tree encoding

(RER|metro)*

|(bus|tram)*

linear

(79)

(RER|metro)*

|(bus|tram)*

Query answer

TRUE

linear linear

(80)

(RER|metro)*

|(bus|tram)*

Query answer

TRUE

(81)

Table of contents

Introduction Existing tools

Provenance circuits and probabilistic query evaluation Other applications

(82)

Probabilistic query evaluation on treelike data

•

^Database^D^with^treewidth^≤^k

for some constantk

•

Probabilityof each fact ofD to be actually present in the data (independently from other facts)

?

(Metro|RER)^∗

|(Bus|Tram)^∗

i

Result: Probabilitythat the databaseDsatisfies queryQ

Computational complexityas a function of thedatabaseD (the queryQisfixed)

(83)

•

for some constantk

• ?

(Metro|RER)^∗

|(Bus|Tram)^∗

i

(84)

•

for some constantk

• ?

(Metro|RER)^∗

|(Bus|Tram)^∗

i

(85)

•

for some constantk

• ?

(Metro|RER)^∗

|(Bus|Tram)^∗

i

(86)

Roadmap

(RER|metro)*

|(bus|tram)*

linear

(87)

Roadmap

(RER|metro)*

|(bus|tram)*

Provenance circuit

(88)

Roadmap

Tree automaton Uncertain tree encoding

(RER|metro)*

|(bus|tram)*

MSO query Probabilistic treelike data

Probability

95%

linear

+probabilities

(89)

Uncertain trees

1

5

6 7 2

3 4

Avaluationof a tree decides whether to keep(1) ordiscard(0) node labels Valuation:{2,3,77→1, ∗ 7→0}

A:“Is there both a pink and a blue node?”

(90)

Uncertain trees

1

5

6 7 2

3 4

Avaluationof a tree decides whether to keep(1) ordiscard(0) node labels

Valuation:{2,3,77→1, ∗ 7→0}

(91)

Uncertain trees

1

5

6 7 2

3 4

(92)

Uncertain trees

1

5

6 7 2

3 4

Avaluationof a tree decides whether to keep(1) ordiscard(0) node labels Valuation:{27→1, ∗ 7→0}

(93)

Uncertain trees

1

5

6 7 2

3 4

Avaluationof a tree decides whether to keep(1) ordiscard(0) node labels Valuation:{2,77→1, ∗ 7→0}

(94)

Uncertain trees

1

5

6 7 2

3 4

(95)

Uncertain trees

1

5

6 7 2

3 4

The tree automatonAaccepts

(96)

Uncertain trees

1

5

6 7 2

3 4

Avaluationof a tree decides whether to keep(1) ordiscard(0) node labels Valuation:{27→1, ∗ 7→0}

The tree automatonArejects

(97)

Uncertain trees

1

5

6 7 2

3 4

The tree automatonAaccepts

(98)

Boolean circuit

∨

¬

x

∧

y

•

Directed acyclic graph ofgates

•

^Output^gate:

•

^Variable^gates: ^x

•

^Internal^gates: ^∨ ^∧ ^¬

•

^Valuation: function from variables to{0,1} Example:ν ={x7→0, y7→1}... mapped to1

(99)

Boolean circuit

∨

¬

x

∧

y

•

^Output^gate:

•

^Variable^gates: ^x

•

(100)

Boolean circuit

∨

¬

x

∧

y

•

^Output^gate:

•

^Variable^gates: ^x

•

(101)

Boolean circuit

∨

¬

x

∧

y

•

^Output^gate:

•

^Variable^gates: ^x

•

(102)

Boolean circuit

∨

¬

x

∧

y

•

^Output^gate:

•

^Variable^gates: ^x

•

^Valuation: function from variables to{0,1}

Example:ν ={x7→0, y7→1}...

mapped to1

(103)

Boolean circuit

∨

¬

x

∧

y

•

^Output^gate:

•

^Variable^gates: ^x

•

Example:ν ={x7→0, y7→1}...

mapped to1

(104)

Boolean circuit

∨

¬

x

∧

y

•

^Output^gate:

•

^Variable^gates: ^x

•

Example:ν ={x7→0, y7→1}...

mapped to1

(105)

Boolean circuit

∨

¬

x

∧

y

•

^Output^gate:

•

^Variable^gates: ^x

•

Example:ν ={x7→0, y7→1}... mapped to1

(106)

Provenance circuit 1

5 6 7 2

4 3

Query:Is there both a pink and a blue node?

Provenance circuit:

∧

∨ 7

2 3

Formally:

•

Tree automatonA, uncertain treeT, circuitC

•

Variable gatesofC: nodes ofT

•

^Condition:^Let^νbe a valuation ofT, thenν(C)iffAacceptsν(T)

(107)

5 6 7 2

4 3

Provenance circuit:

∧

∨ 7

2 3

Formally:

•

(108)

5 6 7 2

4 3

Provenance circuit:

∧

∨ 7

2 3

Formally:

•

(109)

Building provenance circuits on trees

Theorem

For any bottom-uptree automatonAand inputtreeT, we can build aprovenance circuitofAonT inO(|A| × |T|)

•

^Alphabet:

•

^Automaton:“Is there both a pink and a blue node?”

•

^States:

{⊥,B,P,>}

•

^Final: ^{>}

•

Transitions:

> P ⊥

P P ⊥

n

∨ ∨ ∨ ∨

⊥ B P >

∨ ∨ ∨ ∨

⊥ B P >

∨ ∨ ∨ ∨

⊥ B P >

∧

¬ ∧

(110)

Theorem

•

^Alphabet:

•

^States:

{⊥,B,P,>}

•

^Final: ^{>}

•

Transitions:

>

P ⊥

P P ⊥

n

∨ ∨ ∨ ∨

⊥ B P >

∨ ∨ ∨ ∨

⊥ B P >

∨ ∨ ∨ ∨

⊥ B P >

∧

¬ ∧

(111)

Theorem

•

^Alphabet:

•

^States:

{⊥,B,P,>}

•

^Final: ^{>}

•

Transitions:

>

P ⊥

P P ⊥

n

∨ ∨ ∨ ∨

⊥ B P >

∨ ∨ ∨ ∨

⊥ B P >

∨ ∨ ∨ ∨

⊥ B P >

∧

¬ ∧

(112)

Theorem

•

^Alphabet:

•

^States:

{⊥,B,P,>}

•

^Final: ^{>}

•

Transitions:

>

P ⊥

P P ⊥

n

∨ ∨ ∨ ∨

⊥ B P >

∨ ∨ ∨ ∨

⊥ B P >

∨ ∨ ∨ ∨

⊥ B P >

∧

¬ ∧

(113)

Theorem

•

^Alphabet:

•

^States:

{⊥,B,P,>}

•

^Final: ^{>}

•

Transitions:

>

P ⊥

P P ⊥

n

∨ ∨ ∨ ∨

⊥ B P >

∨ ∨ ∨ ∨

⊥ B P >

∨ ∨ ∨ ∨

⊥ B P >

∧

¬ ∧

(114)

Theorem

•

^Alphabet:

•

^States:

{⊥,B,P,>}

•

^Final: ^{>}

•

Transitions:

>

P ⊥

P P ⊥

n

∨ ∨ ∨ ∨

⊥ B P >

∨ ∨ ∨ ∨

⊥ B P >

∨ ∨ ∨ ∨

⊥ B P >

∧

¬ ∧

(115)

Theorem

•

^Alphabet:

•

^States:

{⊥,B,P,>}

•

^Final: ^{>}

•

Transitions:

>

P ⊥

P P ⊥

n

∨ ∨ ∨ ∨

⊥ B P >

∨ ∨ ∨ ∨

⊥ B P >

∨ ∨ ∨ ∨

⊥ B P >

∧

¬ ∧

(116)

Theorem

•

^Alphabet:

•

^States:

{⊥,B,P,>}

•

^Final: ^{>}

•

Transitions:

>

P ⊥

P P ⊥

n

∨ ∨ ∨ ∨

⊥ B P >

∧

¬ ∧

(117)

Tree automaton Uncertain tree encoding

(RER|metro)*

|(bus|tram)*

MSO query Probabilistic treelike data

Probability

95%

linear

+probabilities

(118)

Details of the approach

Probabilistic treelike data

Each fact can disappear with some probability

→ How to computeefficientlythe probability of the circuit?

(119)

Uncertain tree encoding Probabilistic

treelike data

Each node label can disappear with the probability of the coded fact

(120)

treelike data

Each variable can be true with the probability of the coded fact

Provenance circuit +probabilities

(121)

treelike data

Probability

95%

Probability that the circuit evaluates to true

(122)

treelike data

Probability

95%

Probability that the circuit evaluates to true

(123)

Computing the circuit probability The circuit is ad-DNNF...

... so probability computation iseasy!

•

^¬ gates only have variablesas inputs

¬ g

g⁰

P(g) :=1−P(g⁰)

•

^∨ gates always have mutually exclusiveinputs

∨ g

g⁰₁ g⁰₂

P(g) :=P(g⁰₁) +P(g⁰₂)

•

^∧ gates are all on independentinputs

∧ g

g⁰₁ g⁰₂

P(g) :=P(g⁰₁)×P(g⁰₂)

Theorem [Amarilli et al., 2015]

For anyfixedBoolean MSO queryQandk∈N,

given a databaseDoftreewidth≤kwithindependent probabilities, we can compute inlinear timethe probability thatDsatisfiesQ

(124)

•

¬ g

g⁰

P(g) :=1−P(g⁰)

•

∨ g

g⁰₁ g⁰₂

P(g) :=P(g⁰₁) +P(g⁰₂)

•

∧ g

g⁰₁ g⁰₂

P(g) :=P(g⁰₁)×P(g⁰₂)

(125)

•

¬ g

g⁰

P(g) :=1−P(g⁰)

•

∨ g

g⁰₁ g⁰₂

P(g) :=P(g⁰₁) +P(g⁰₂)

•

∧ g

g⁰₁ g⁰₂

P(g) :=P(g⁰₁)×P(g⁰₂)

(126)

•

¬ g

g⁰

P(g) :=1−P(g⁰)

•

∨ g

g⁰₁ g⁰₂

P(g) :=P(g⁰₁) +P(g⁰₂)

•

∧ g

g⁰₁ g⁰₂

P(g) :=P(g⁰₁)×P(g⁰₂)

(127)

Computing the circuit probability

The circuit is ad-DNNF... ... so probability computation iseasy!

•

¬ g

g⁰

P(g) :=1−P(g⁰)

•

∨ g

g⁰₁ g⁰₂

P(g) :=P(g⁰₁) +P(g⁰₂)

•

∧ g

g⁰₁ g⁰₂

P(g) :=P(g⁰₁)×P(g⁰₂)

(128)

Computing the circuit probability

The circuit is ad-DNNF... ... so probability computation iseasy!

•

¬ g

g⁰

P(g) :=1−P(g⁰)

•

∨ g

g⁰₁ g⁰₂

P(g) :=P(g⁰₁) +P(g⁰₂)

•

∧ g

g⁰₁ g⁰₂

P(g) :=P(g⁰₁)×P(g⁰₂)

Leveraging the structure of uncertain data