Leveraging the structure of uncertain data

(1)

Antoine Amarilli March 5, 2019

1/33

(2)

Example application: Subway routing

(3)

2/33

(4)

(5)

2/33

(6)

(Metro|RER)|(Bus|Tram)

(7)

13 6

23 14

Database

• (Hyper)graph

• Collection of ground facts

G(aa1, ab2), G(ab2, ac3), S(aa1, m4), S(ab2, rB), ...

3/33

(8)

Database theory and query evaluation

13 6

23 14

Query

?

+

Database

• (Hyper)graph

• Regular path

∀X(rm ∈ X ∧ ∀xy (Metro|RER)*

• Logic formula

(x ∈ X ∧ G(x, y) → y ∈ X)) → gn ∈ X

|(Bus|Tram)*

(9)

13 6

23 14

Query

?

+

Database

• (Hyper)graph

• Regular path

∀X(rm ∈ X ∧ ∀xy

i _Result

• TRUE/FALSE

↔ Model checking (Metro|RER)*

• Logic formula

(x ∈ X ∧ G(x, y) → y ∈ X)) → gn ∈ X

|(Bus|Tram)*

3/33

(10)

Probabilistic query evaluation

13 6

23 14

(11)

4/33

(12)

(13)

4/33

(14)

(15)

4/33

(16)

(17)

13 6

23 14

4/33

(18)

13 6

23 ^95% 14

5% 60

98%

2% 60

(19)

• (Hyper)graph

Probabilistic database

+ independent probabilities

13 6

23 ^95%14

5% 60

98%

2% 60

4/33

(20)

Query

?

+

• (Hyper)graph

• Regular path

• Logic formula

(x ∈ X ∧ G(x, y) → y ∈ X)) → gn ∈ X

|(Bus|Tram)*

Probabilistic database

13 6

23 ^95%14

5% 60

98%

2% 60

(21)

Query

?

+

• (Hyper)graph

• Regular path

• Logic formula

(x ∈ X ∧ G(x, y) → y ∈ X)) → gn ∈ X

|(Bus|Tram)*

Probabilistic database

Probabilistic i

Result

• Probability according to the input distribution

13 6

23 ^95%14

5% 60

98%

2% 60

proba to be on time: 98%

4/33

(22)

Computational complexity

•

Computing paths on a large graph:

→ Well-studied problem, efficient algorithms

→ #P-hardcomputational complexity in thedatabase

(23)

50%

90%

42%

37%

90%

83%

78%

72%

•

Computing paths on a largeprobabilisticgraph:

→ ???

5/33

(24)

50%

90%

42%

37%

90%

83%

78%

72%

•

→ Exponentialnumber of possibilities

(25)

50%

90%

42%

37%

90%

83%

78%

72%

•

→ Exponentialnumber of possibilities

5/33

(26)

Idea: use the structure of data

→ Shortest path:very easyon alarge tree

(27)

6/33

(28)

(29)

6/33

(30)

(31)

6/33

(32)

(33)

Leveraging the structure of uncertain data

Does query evaluation on probabilistic data havelower complexity when thestructureof the data isclose to a tree?

•

Existing tools fornon-probabilistic data:

• Tree automata, to evaluate queries on trees

• Treewidth, formalizes the notion of being “close to a tree”

• Courcelle’s theorem

•

^Introduce^{new tools}^and^results:

• Provenance circuitsof tree automata on uncertain trees

• Application toprobabilistic query evaluation

•

Other application:enumeration of query results

7/33

(34)

In this talk:

•

(35)

In this talk:

•

7/33

(36)

In this talk:

•

(37)

In this talk:

•

7/33

(38)

In this talk:

•

(39)

In this talk:

•

7/33

(40)

In this talk:

•

(41)

In this talk:

•

7/33

(42)

Table of contents

Introduction

Existing tools for non-probabilistic data

Provenance circuits and probabilistic query evaluation Application to enumeration

(43)

Query evaluation on words

Database: awordwwhere nodes have a color from an alphabet

?

inmonadic second-order logic(MSO) and a blue node?”

i

Result: TRUE/FALSE indicating if the wordwsatisfies the queryQ

Computational complexityas a function ofw (the queryQisfixed)

9/33

(44)

?

^Query^{Q: a}^sentence(yes/no question) inmonadic second-order logic(MSO)

“Is there both a pink and a blue node?”

i

(45)

?

i

(the queryQisfixed)

9/33

(46)

?

i

(47)

Monadic second-order logic (MSO)

•

^P ⁽^x⁾^{means “x}is blue”; alsoP (x),P (x)

•

^x^→^y^{means “x}is the predecessor ofy”

•

Propositional logic:formulas withAND∧,OR∨,NOT¬

• P (x)∧P (y)means“Nodexis pink and nodeyis blue”

•

First-order logic:addsexistential quantifier∃and universal quantifier∀

• ∃x y P (x)∧P (y)means“There is both a pink and a blue node”

•

Monadic second-order logic (MSO):addsquantifiers over sets

• ∃S∀x S(x)means“there is a setScontaining every elementx”

• Can expresstransitive closurex→^∗y, i.e.,“xis beforey”

• ∀x P (x)⇒ ∃y P (y)∧x→^∗y

means“There is a blue node after every pink node”

10/33

(48)

•

• ∀x P (x)⇒ ∃y P (y)∧x→^∗y

(49)

•

• ∀x P (x)⇒ ∃y P (y)∧x→^∗y

10/33

(50)

•

• ∀x P (x)⇒ ∃y P (y)∧x→^∗y

(51)

Word automata

Translate the queryQto adeterministic word automaton

Alphabet: w: Q:∃x y P (x)∧P (y)

•

^States:^{⊥,^B^,^P^,^>}

•

Final states:{>}

•

Initial function: ⊥ P B

•

Transitions(examples): ⊥

P P

>

> Theorem (Büchi, 1960)

MSOandword automatahave the sameexpressive poweron words Corollary

Query evaluation of MSO on words is inlinear time.

11/33

(52)

Word automata

Alphabet: w:

⊥ P P > >

Q:∃x y P (x)∧P (y)

•

^States:^{⊥,^B^,^P^,^>}

•

Final states:{>}

•

P P

>

(53)

Word automata

Alphabet: w: Q:∃x y P (x)∧P (y)

•

^States:^{⊥,^B^,^P^,^>}

•

Final states:{>}

•

P P

>

11/33

(54)

Word automata

Alphabet: w:

⊥ P P > >

Q:∃x y P (x)∧P (y)

•

^States:^{⊥,^B^,^P^,^>}

•

Final states:{>}

•

P P

>

(55)

Word automata

Alphabet: w:

⊥ Q:∃x y P (x)∧P (y)

•

^States:^{⊥,^B^,^P^,^>}

•

Final states:{>}

•

P P

>

11/33

(56)

Word automata

Alphabet: w:

⊥

P P > >

Q:∃x y P (x)∧P (y)

•

^States:^{⊥,^B^,^P^,^>}

•

Final states:{>}

•

P P

>

Theorem (Büchi, 1960)

(57)

Word automata

Alphabet: w:

⊥ P Q:∃x y P (x)∧P (y)

•

^States:^{⊥,^B^,^P^,^>}

•

Final states:{>}

•

P P

>

11/33

(58)

Word automata

Alphabet: w:

⊥ P P

> >

Q:∃x y P (x)∧P (y)

•

^States:^{⊥,^B^,^P^,^>}

•

Final states:{>}

•

P P

>

(59)

Word automata

Alphabet: w:

⊥ P P > Q:∃x y P (x)∧P (y)

•

^States:^{⊥,^B^,^P^,^>}

•

Final states:{>}

•

P P

>

11/33

(60)

Word automata

Alphabet: w:

⊥ P P > > Q:∃x y P (x)∧P (y)

•

^States:^{⊥,^B^,^P^,^>}

•

Final states:{>}

•

P P

>

(61)

Word automata

Alphabet: w:

⊥ P P > > Q:∃x y P (x)∧P (y)

•

^States:^{⊥,^B^,^P^,^>}

•

Final states:{>}

•

P P

>

MSOandword automatahave the sameexpressive poweron words

11/33

(62)

Word automata

Alphabet: w:

⊥ P P > > Q:∃x y P (x)∧P (y)

•

^States:^{⊥,^B^,^P^,^>}

•

Final states:{>}

•

P P

>

(63)

Query evaluation on trees

Database: atreeTwhere nodes have a color from an alphabet

?

^Query^{Q: a}^sentence^{in monadic}

second-order logic (MSO)

•P (x)means “xis blue”

•x→ymeans “xis the parent ofy”

∃x y P (x)∧P (y)

i

Result: TRUE/FALSE indicating if the treeTsatisfies the queryQ Computational complexityas a function ofT

(the queryQisfixed)

12/33

(64)

?

∃x y P (x)∧P (y)

i

(the queryQisfixed)

(65)

?

∃x y P (x)∧P (y)

i

Result: TRUE/FALSE indicating if the treeTsatisfies the queryQ

(the queryQisfixed)

12/33

(66)

?

∃x y P (x)∧P (y)

i

(the queryQisfixed)

(67)

Tree automata Tree alphabet:

•

Bottom-up deterministictree automaton

•

“Is there both a pink and a blue node?”

•

^States:^{⊥,^B^,^P^,^>}

•

Final states: {>}

•

Transitions(examples): P

P ⊥

> B P

⊥

⊥ Theorem [Thatcher and Wright, 1968]

MSOandtree automatahave the sameexpressive poweron trees Corollary

Query evaluation of MSO on trees is inlinear time.

13/33

(68)

Tree automata

Tree alphabet:

•

^States:^{⊥,^B^,^P^,^>}

•

Final states: {>}

•

P ⊥

> B P

⊥

(69)

Tree automata

Tree alphabet:

•

^States:^{⊥,^B^,^P^,^>}

•

Final states: {>}

•

P ⊥

> B P

⊥

13/33

(70)

Tree automata

Tree alphabet:

•

^States:^{⊥,^B^,^P^,^>}

•

Final states: {>}

•

P ⊥

> B P

⊥

(71)

Tree automata

Tree alphabet:

•

^States:^{⊥,^B^,^P^,^>}

•

Final states: {>}

•

P ⊥

> B P

⊥

13/33

(72)

B

⊥ P ⊥

•

^States:^{⊥,^B^,^P^,^>}

•

Final states: {>}

•

P ⊥

> B P

⊥

(73)

B

⊥ P ⊥

•

^States:^{⊥,^B^,^P^,^>}

•

Final states: {>}

•

Transitions(examples):

P P ⊥

>

B P

⊥

13/33

(74)

B

⊥ P

P ⊥

•

^States:^{⊥,^B^,^P^,^>}

•

Final states: {>}

•

P P ⊥

>

B P

⊥

Theorem [Thatcher and Wright, 1968]

(75)

>

B

⊥ P

P ⊥

•

^States:^{⊥,^B^,^P^,^>}

•

Final states: {>}

•

P P ⊥

>

B P

⊥

13/33

(76)

>

B

⊥ P

P ⊥

•

^States:^{⊥,^B^,^P^,^>}

•

Final states: {>}

•

P P ⊥

>

B P

⊥

MSOandtree automatahave the sameexpressive poweron trees

Corollary

(77)

Tree alphabet:

>

B

⊥ P

P ⊥

•

^States:^{⊥,^B^,^P^,^>}

•

Final states: {>}

•

P P ⊥

>

B P

⊥

Query evaluation of MSO on trees is inlinear time. ^13/33

(78)

?

∃x y P (x)∧P (y)

i

Result: TRUE/FALSE indicating ifT satisfies the queryQ Computational complexityas a function ofT

(the queryQisfixed)

(79)

Query evaluation on treelike data

Database: atreelike databaseT

???

?

QueryQ: asentencein monadic second-order logic (MSO)

(Metro|RER)^∗

|(Bus|Tram)^∗

i

Result: TRUE/FALSE indicating ifT satisfies the queryQ Computational complexityas a function ofT

(the queryQisfixed)

14/33

(80)

Treewidth

Treewidth by example:

•

^Treeshave treewidth1

•

^Cycleshave treewidth2

•

^k-cliques^and⁽^k⁻¹⁾^-gridshave treewidthk−1

→ Treelike: thetreewidthis bounded by aconstant

(81)

Treewidth

•

15/33

(82)

Treewidth

•

(83)

Treewidth

•

15/33

(84)

Treewidth

•

(85)

Treewidth

•

15/33

(86)

Treewidth

•

(87)

Treewidth

•

15/33

(88)

Treewidth

•

(89)

Treewidth

•

15/33

(90)

Treewidth

•

(91)

Treewidth

•

15/33

(92)

Treewidth

•

(93)

Treewidth

•

15/33

(94)

Treewidth

•

(95)

•

15/33

(96)

Courcelle’s theorem

(RER|metro)*

|(bus|tram)*

MSO query Treelike data

Theorem [Courcelle, 1990]

For any fixed Boolean MSO queryQandk∈N, given a databaseDof treewidth≤k,

we can compute inlinear timeinDwhetherDsatisfiesQ

(97)

Tree automaton (RER|metro)*

|(bus|tram)*

16/33

(98)

Tree automaton Tree encoding

(RER|metro)*

|(bus|tram)*

linear

(99)

(RER|metro)*

|(bus|tram)*

Query answer

TRUE

linear linear

16/33

(100)

(RER|metro)*

|(bus|tram)*

Query answer

TRUE

we can compute inlinear timeinDwhetherDsatisfiesQ ^16/33

(101)

Introduction

Existing tools for non-probabilistic data

Provenance circuits and probabilistic query evaluation Application to enumeration

17/33

(102)

Probabilistic query evaluation on treelike data

•

^Database^D^with^treewidth^≤^k

for some constantk

•

Probabilityof each fact ofD to be actually present in the data (independently from other facts)

?

(Metro|RER)^∗

|(Bus|Tram)^∗

i

Result:Probabilitythat the databaseDsatisfies queryQ Computational complexityas a function of thedatabaseD (the queryQisfixed)

(103)

•

for some constantk

• ?

(Metro|RER)^∗

|(Bus|Tram)^∗

i

Result:Probabilitythat the databaseDsatisfies queryQ Computational complexityas a function of thedatabaseD (the queryQisfixed)

18/33

(104)

•

for some constantk

• ?

(Metro|RER)^∗

|(Bus|Tram)^∗

i

Result: Probabilitythat the databaseDsatisfies queryQ

Computational complexityas a function of thedatabaseD (the queryQisfixed)

(105)

•

for some constantk

• ?

(Metro|RER)^∗

|(Bus|Tram)^∗

i

Result: Probabilitythat the databaseDsatisfies queryQ Computational complexityas a function of thedatabaseD (the queryQisfixed)

18/33

(106)

Roadmap

(RER|metro)*

|(bus|tram)*

linear

(107)

(RER|metro)*

|(bus|tram)*

Provenance circuit

19/33

(108)

Roadmap

Tree automaton Uncertain tree encoding

(RER|metro)*

|(bus|tram)*

MSO query Probabilistic treelike data

Provenance circuit

Probability

95%

linear

+probabilities

(109)

Uncertain trees

1

5

6 7 2

3 4

keep(1) ordiscard(0) node labels Valuation: {2,3,77→1, ∗ 7→0}

A:“Is there both a pink and a blue node?”

20/33

(110)

Uncertain trees

1

5

6 7 2

3 4

Avaluationof a tree decides whether to keep(1) ordiscard(0) node labels

Valuation: {2,3,77→1, ∗ 7→0}

(111)

Uncertain trees

1

5

6 7 2

3 4

Avaluationof a tree decides whether to keep(1) ordiscard(0) node labels Valuation: {2,3,77→1, ∗ 7→0}

20/33

(112)

Uncertain trees

1

5

6 7 2

3 4

Avaluationof a tree decides whether to keep(1) ordiscard(0) node labels Valuation: {27→1, ∗ 7→0}

(113)

Uncertain trees

1

5

6 7 2

3 4

Avaluationof a tree decides whether to keep(1) ordiscard(0) node labels Valuation: {2,77→1, ∗ 7→0}

20/33

(114)

Uncertain trees

1

5

6 7 2

3 4

(115)

1

5

6 7 2

3 4

Avaluationof a tree decides whether to keep(1) ordiscard(0) node labels Valuation: {2,3,77→1, ∗ 7→0}

The tree automatonAaccepts

20/33

(116)

Uncertain trees

1

5

6 7 2

3 4

Avaluationof a tree decides whether to keep(1) ordiscard(0) node labels Valuation: {27→1, ∗ 7→0}

The tree automatonArejects

(117)

1

5

6 7 2

3 4

The tree automatonAaccepts

20/33

(118)

Boolean circuit

∨

¬

x

∧

y

•

Directed acyclic graph ofgates

•

^Output^gate:

•

^Variable^gates: ^x

•

^Internal^gates: ^∨ ^∧ ^¬

•

Valuation: function from variables to{0,1} Example:ν ={x7→0, y7→1}... mapped to1

(119)

Boolean circuit

∨

¬

x

∧

y

•

^Output^gate:

•

^Variable^gates: ^x

•

21/33

(120)

Boolean circuit

∨

¬

x

∧

y

•

^Output^gate:

•

^Variable^gates: ^x

•

(121)

Boolean circuit

∨

¬

x

∧

y

•

^Output^gate:

•

^Variable^gates: ^x

•

21/33

(122)

Boolean circuit

∨

¬

x

∧

y

•

^Output^gate:

•

^Variable^gates: ^x

•

Valuation: function from variables to{0,1} Example:ν ={x7→0, y7→1}...

mapped to1

(123)

Boolean circuit

∨

¬

x

∧

y

•

^Output^gate:

•

^Variable^gates: ^x

•

Valuation: function from variables to{0,1} Example:ν ={x7→0, y7→1}...

21/33

Leveraging the structure of uncertain data

(Metro|RER)*|(Bus|Tram)*

Database

Query

?

+

Database

Query

?

+

Database

i Result

13 6

23 14

13 6

23 14

13 6

23 95% 14

5% 60

98%

2% 60

Probabilistic database

Query

?

+

Probabilistic database

Query

?

+

Probabilistic database

Probabilistic i

Result

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

?

i

?

i

?

i

?

i

•

•

•

•

•

•

•

•

•

(Metro|RER)|(Bus|Tram)

i _Result

23 ^95% 14