Uniform Reliability of Self-Join-Free Conjunctive Queries

(1)

Uniform Reliability of Self-Join-Free Conjunctive Queries

Antoine Amarilli¹, Benny Kimelfeld² March 11, 2021

1Télécom Paris

2Technion

(2)

Uncertain data and tuple-independent databases (TID)

•

We consider data in therelational modelon which we haveuncertainty

•

Simplest uncertainty model:tuple-independent databases(TID) Classes

Class Room Date

Databases 1 101 2021-03-26 80%

Databases 2 101 2021-04-02 70%

Lockdowns Date

2021-03-26 20%

2021-04-02 40%

•

^Semantics:

• Every tuple is annotated with aprobability

• We assume that all tuples areindependent

• A TID concisely represents aprobability distributionover thesubinstances

→ Warning:we only use the TID model to showtheoretical results:)

(3)

•

Databases 1 101 2021-03-26 80%

Databases 2 101 2021-04-02 70%

Lockdowns Date

2021-03-26 20%

2021-04-02 40%

•

^Semantics:

(4)

•

Databases 1 101 2021-03-26 80%

Databases 2 101 2021-04-02 70%

Lockdowns Date

2021-03-26 20%

2021-04-02 40%

•

^Semantics:

(5)

Probabilistic query evaluation (PQE)

•

We considerconjunctive queries(CQs), which we assume to beBoolean

• Q:∃c r dClasses(c,r,d)∧Lockdowns(d)

•

^Semantics^of^Qon a TID: compute theprobabilitythatQis true

•

Formally, problemPQE(Q)for a fixed CQQ:

• Input: a TIDI

• Output:the total probability of the subinstances ofIwhereQis true

•

We can always solvePQE(Q)by looking at all subinstances (exponential)

→ When can we achieve a better complexity?

(6)

•

• Input: a TIDI

•

(7)

•

• Input: a TIDI

•

(8)

•

• Input: a TIDI

•

(9)

•

• Input: a TIDI

•

(10)

Existing results

•

Complexity of PQE shown in [Dalvi and Suciu, 2007] forself-join-free CQs(SJFCQs)

• A CQ isself-join-freeif no relation symbol is repeated

•

Later extended tounions of conjunctive queries[Dalvi and Suciu, 2012]

In this work we stick to the result on SJFCQs: Theorem [Dalvi and Suciu, 2007]

LetQbe a SJFCQ. Then:

•

^Either^Q^ishierarchicalandPQE(Q)is inPTIME

•

^Or^Q^isnot hierarchicalandPQE(Q)is#P-hard What is this class ofhierarchical CQs?

(11)

•

In this work we stick to the result on SJFCQs:

Theorem [Dalvi and Suciu, 2007]

•

^Or^Q^isnot hierarchicalandPQE(Q)is#P-hard

What is this class ofhierarchical CQs?

(12)

•

In this work we stick to the result on SJFCQs:

Theorem [Dalvi and Suciu, 2007]

•

^Or^Q^isnot hierarchicalandPQE(Q)is#P-hard What is this class ofhierarchical CQs?

(13)

Hierarchical CQs

For a CQQ, writeatoms(x)for the set of atoms wherexappears

•

^{A CQ is}hierarchicalif for every variablesxandy

• Eitheratoms(x)andatoms(y)aredisjoint

• Or one isincludedin the other

•

^{A CQ is}non-hierarchicalif there are two variablesxandysuch that

• Some atom containsbothxandy

• Some atom containsxbut noty

• Some atom containsybut notx

→ Simplest example: theR-S-Tquery:Q1 :∃x y R(x),S(x,y),T(y)

•

^Intuitionfor arity-2 queries: the hierarchical CQs areunions of star-shaped patterns Exercise: Is our example CQ hierarchical?∃c r dClasses(c,r,d)∧Lockdowns(d)...Yes!

(14)

•

^Intuitionfor arity-2 queries: the hierarchical CQs areunions of star-shaped patterns Exercise: Is our example CQ hierarchical?∃c r dClasses(c,r,d)∧Lockdowns(d)...Yes!

(15)

•

^Intuitionfor arity-2 queries: the hierarchical CQs areunions of star-shaped patterns

Exercise: Is our example CQ hierarchical?∃c r dClasses(c,r,d)∧Lockdowns(d)...Yes!

(16)

•

^Intuitionfor arity-2 queries: the hierarchical CQs areunions of star-shaped patterns Exercise: Is our example CQ hierarchical?∃c r dClasses(c,r,d)∧Lockdowns(d)

...Yes!

(17)

•

^Intuitionfor arity-2 queries: the hierarchical CQs areunions of star-shaped patterns Exercise: Is our example CQ hierarchical?∃c r dClasses(c,r,d)∧Lockdowns(d)... Yes!

(18)

Uniform reliability

•

We study theuniform reliability(UR) problem, a simpler variant of PQE:

• Wefixa CQQ, and consider the problemUR(Q):

• Input: a database instanceIwithout probabilities

• Output:how manysubinstancesofIsatisfyQ

•

^Remark:^UR^is^PQEbut where all facts haveprobability 1/2

(up to amultiplicative factorof2^NforNthe number of facts of the TID)

•

^{Our goal:}What is the complexity of UR?

(19)

•

(20)

•

(21)

Our results

•

We only study the problem onSJFCQs(see future work)

•

^Forhierarchical SJFCQs, UR is tractable because PQE is tractable

•

^Fornon-hierarchical SJFCQs, the complexity of UR isunknown

• Thehardness proofof PQE (see later) crucially uses probabilities1/2and1 We settle the complexity of UR for SJFCQs by showing:

Theorem

LetQbe a non-hierarchical SJFCQ. ThenUR(Q)is#P-hard. Rest of the talk: proof sketch of this result

(22)

Our results

•

^Fornon-hierarchical SJFCQs, the complexity of UR isunknown

Theorem

(23)

Our results

•

^Fornon-hierarchical SJFCQs, the complexity of UR is unknown

• Thehardness proofof PQE (see later) crucially uses probabilities1/2and1

We settle the complexity of UR for SJFCQs by showing: Theorem

(24)

Our results

•

Theorem

LetQbe a non-hierarchical SJFCQ. ThenUR(Q)is#P-hard.

Rest of the talk: proof sketch of this result

(25)

Our results

•

Theorem

LetQbe a non-hierarchical SJFCQ. ThenUR(Q)is#P-hard.

Rest of the talk: proof sketch of this result

(26)

Reducing toR-S-T-type queries

•

^AnR-S-T-type queryis a non-hierarchical SJFCQ of the form:

R1(x), . . . ,Rr(x),S1(x,y), . . . ,Ss(x,y),T1(y), . . . ,Tt(y) for some integersr,s,t>0

•

^Lemma:for any non-hierarchical SJFCQQ, there is anR-S-T-type queryQ⁰ such that UR(Q⁰)reduces toUR(Q)

atoms(x) atoms(x)∩atoms(y) atoms(y)

r s t

•

So it suffices to show thatUR(Q⁰)is#P-hardfor theR-S-T-type queriesQ⁰

•

In this talk:we focus for simplicity onQ1 :∃x y R(x),S(x,y),T(y)

(27)

•

r s t

•

(28)

•

r s t

•

(29)

•

r s t

•

(30)

Hard problem: counting independent sets of bipartite graphs u₁

u₂ u3

v₁

v₂

•

Independent setof a bipartite graph: subset of its vertices that contains no edge

• Example:{u2,v1}

•

^{It is}#P-hard, given a bipartite graph, to count its independent sets This easily shows the#P-hardnessof PQE (but not UR!) forQ1:∃x yR(x),S(x,y),T(y):

R(u₁): 1/2 R(u2): 1/2 R(u₃): 1/2

T(v₁): 1/2

T(v₂): 1/2 S: 1

S: 1 S: 1

We will show how to reduce from counting independent sets toUR(Q1)

(31)

u₂ u3

v₁

v₂

•

• Example:{u2,v1}

•

^{It is}#P-hard, given a bipartite graph, to count its independent sets

This easily shows the#P-hardnessof PQE (but not UR!) forQ1:∃x yR(x),S(x,y),T(y): R(u₁): 1/2

R(u2): 1/2 R(u₃): 1/2

T(v₁): 1/2

T(v₂): 1/2 S: 1

S: 1 S: 1

(32)

u₂ u3

v₁

v₂

•

• Example:{u2,v1}

•

R(u₁): 1/2 R(u2): 1/2 R(u₃): 1/2

T(v₁): 1/2

T(v₂): 1/2 S: 1

S: 1 S: 1

(33)

u₂ u3

v₁

v₂

•

• Example:{u2,v1}

•

R(u₁): 1/2 R(u2): 1/2 R(u₃): 1/2

T(v₁): 1/2

T(v₂): 1/2 S: 1

S: 1 S: 1

(34)

Idea: parameterizing the count

u₁

u2

u₃

v₁

v₂

For a bipartite graph(U,V,E)and a subsetW⊆U∪V of vertices, we write:

•

^c(W)the number of edgescontainedinW

• Here,c(W) =1

•

^d(W)^(resp.,^d⁰^(W)) the number of edges having exactly their left (resp., right) endpoint inW

• Here,d(W) =2andd⁰(W) =1

•

^e(W)the number of edgesexcluded fromW

• Here,e(W) =1

•

Hard problem:counting independent setsX=|{W⊆U∪V |c(W) =0}|

•

Harder problem:computing all the values: X_c,d,d⁰_,e =

{W ⊆U∪V |c(W) =candd(W) =dandd⁰(W) =d⁰ ande(W) =e}

(35)

u₁

u2

u₃

v₁

v₂

•

• Here,c(W) =1

•

• Here,e(W) =1

•

(36)

u₁

u2

u₃

v₁

v₂

•

• Here,c(W) =1

•

• Here,e(W) =1

•

(37)

u₁

u2

u₃

v₁

v₂

•

• Here,c(W) =1

•

• Here,e(W) =1

•

(38)

u₁

u2

u₃

v₁

v₂

•

• Here,c(W) =1

•

• Here,e(W) =1

•

(39)

u₁

u2

u₃

v₁

v₂

•

• Here,c(W) =1

•

• Here,e(W) =1

•

Harder problem:computing all the values:

X_c,d,d⁰_,e =

(40)

Idea: coding to several copies

•

We want to design areduction:

• We reducefrom(we want): given a bipartite graphG, compute theX_c,d,d⁰_,e

• We reduceto(we have): given a database instanceD, computeUR(Q1)

•

^Idea:^code^G^{to a}^familyof instancesD_pindexedbyp>0

•

^{Fix a}^box^B^p^(a,^b)^{for index}^p^>⁰: an instance with two distinguished elements(a,b)

•

^Code^G^{for index}^p^>⁰to an instance by:

• putting anR-fact on eachU-vertex and aT-fact on eachV-vertex

• coding every edge(u,v)by acopy of the boxB_p(u,v) u₁

u2

u₃

v₁

v₂

R(u₁) R(u2) R(u₃)

T(v₁)

T(v₂) B_p

B_p B_p

B_p

Bp

(41)

•

u2

u₃

v₁

v₂

R(u₁) R(u2) R(u₃)

T(v₁)

T(v₂) B_p

B_p B_p

B_p

Bp

(42)

•

u2

u3

v₁

v₂

R(u₁) R(u₂) R(u3)

T(v₁)

T(v2) B_p

B_p B_p

Bp

(43)

Getting an equation system

Take the coding ofGfor indexp, and compute the numberN_pof subinstancesviolatingQ₁ R(u₁)

R(u₂) R(u₃)

T(v₁)

T(v₂) B_p

Bp

B_p B_p B_p

•

^{We have:}

N_p= X

W⊆V

N^W_p

= X

W⊆V

γ_p^c(W)δ^d(W)_p (δ_p⁰)^d⁰^(W)η_p^e(W) = X

c,d,d⁰,e

X_c,d,d⁰_,e×γ^c_pδ_p^d(δ_p⁰)^d⁰η_p^e

whereN^W_p is the number of subinstances violatingQ₁when fixing theR-facts and T-facts to be precisely onW

•

^Now^N^W^p only depends on:

• The numbersc(W),d(W),d⁰(W),e(W)of edgescontained,dangling, orexcludedfromW

• The numbersγ_p,δ_p,δ⁰_p,η_pof subinstances of the boxB_pthat violateQ₁ when fixingR-facts onaand/orT-facts onb

(44)

R(u₂) R(u₃)

T(v₁)

T(v₂) B_p

Bp

B_p B_p

B_p

•

^{We have:}

N_p= X

W⊆V

N^W_p

= X

W⊆V

c,d,d⁰,e

X_c,d,d⁰_,e×γ_p^cδ_p^d(δ_p⁰)^d⁰η_p^e

•

(45)

R(u₂) R(u₃)

T(v₁)

T(v₂) B_p

Bp

B_p B_p

B_p

•

^{We have:}

N_p= X

W⊆V

N^W_p

= X

W⊆V

c,d,d⁰,e

•

(46)

R(u₂) R(u₃)

T(v₁)

T(v₂) B_p

Bp

B_p B_p

B_p

•

^{We have:}

N_p= X

W⊆V

N^W_p

= X

W⊆V

c,d,d⁰,e

•

(47)

R(u₂) R(u₃)

T(v₁)

T(v₂) B_p

Bp

B_p B_p

B_p

•

^{We have:}

N_p= X

W⊆V

N^W_p

= X

W⊆V

c,d,d⁰,e

•

• The numbersγ_p,δ_p,δ⁰_p,η_pof subinstances of the boxB_pthat violateQ₁

when fixingR-facts onaand/orT-facts onb ^12/14

(48)

R(u₂) R(u₃)

T(v₁)

T(v₂) B_p

Bp

B_p B_p

B_p

•

^{We have:}

N_p= X

W⊆V

N^W_p = X

W⊆V

γ_p^c(W)δ^d(W)_p (δ⁰_p)^d⁰^(W)η_p^e(W)

= X

c,d,d⁰,e

•

(49)

R(u₂) R(u₃)

T(v₁)

T(v₂) B_p

Bp

B_p B_p

B_p

•

^{We have:}

N_p= X

W⊆V

N^W_p = X

W⊆V

γ_p^c(W)δ^d(W)_p (δ⁰_p)^d⁰^(W)η_p^e(W) = X

c,d,d⁰,e

X_c,d,d⁰_,e×γ_p^cδ_p^d(δ_p⁰)^d⁰η_p^e whereN^W_p is the number of subinstances violatingQ₁when fixing theR-facts and T-facts to be precisely onW

•

(50)

Equation system and conclusion

We have shown the equation:

N_p= X

c,d,d⁰,e

X_c,d,d⁰_,e×γ_p^cδ_p^d(δ_p⁰)^d⁰η^e_p

where:

•

^The^X^c,d,d⁰^,eare what wewant(to count independent sets)

•

^The^N^pare what wehave(by solvingUR(Q₁))

•

^The^γ^p^c^δ^p^d^(δ^p⁰⁾^d⁰^η^p^e^arecoefficientsof a matrixMdepending on our choice of boxBp

In other words we have:

~N=M~X

We can design a box whereMisinvertible, so we can recover~Xfrom~N, showing hardness

(51)

Equation system and conclusion

We have shown the equation:

N_p= X

c,d,d⁰,e

X_c,d,d⁰_,e×γ_p^cδ_p^d(δ_p⁰)^d⁰η^e_p

where:

•

^The^X^c,d,d⁰^,eare what wewant(to count independent sets)

•

^The^N^pare what wehave(by solvingUR(Q₁))

•

^The^γ^p^c^δ^p^d^(δ^p⁰⁾^d⁰^η^p^e^arecoefficientsof a matrixMdepending on our choice of boxBp

In other words we have:

~N=M~X

We can design a box whereMisinvertible, so we can recover~Xfrom~N, showing hardness