Playing spy games in Iris

Playing spy games in Iris

Paulo Em´ılio de Vilhena, Jacques-Henri Jourdan,Fran¸cois Pottier

November 11, 2019

Contents

Local generic solvers

Spying: implementation and specification of modulus Spying: verification of modulus

The conjunction rule Conclusion

Bibliography

Local generic solvers

A family of related algorithms for computing the least solutionof a system of recursive equations:

• Le Charlier and Van Hentenryck (1992).

• Vergauwen and Lewi (1994).

• Fecht and Seidl (1999) coin the term “local generic solver”.

• F. P. (2009) releasesFix and asks how toverifyit.

API of a solver

A solver computes the least fixed pointof a user-supplied monotone second-order function:

t y p e v a l u a t i o n = v a r i a b l e - > p r o p e r t y

val lfp : ( v a l u a t i o n - > v a l u a t i o n ) - > v a l u a t i o n

lfp eqs returns a functionphithat purports to be the least fixed point.

We are interested inon-demand, incremental, memoizingsolvers.

Nothing is computeduntil phiis applied to a variablev. Minimal work is then performed: the least fixed point is computed atv and at the variables that vdependsupon. It is memoized to avoid recomputation.

Dependencies are discovered at runtime via spying.

A challenge

F. P. (2009) offers the verification of a local generic solver as a challenge.

Why is it difficult?

A solver offers a pure API, yet uses mutable internal state:

• for memoization – use a lock and its invariant;

• for spying on the user-supplied functioneqs.

A challenge

F. P. (2009) offers the verification of a local generic solver as a challenge.

Why is it difficult?

A solver offers a pure API, yet uses mutable internal state:

• for memoization – use a lock and its invariant;

• for spying on the user-supplied functioneqs.

What we would like

In short, we want a modular specification in higher-order separation logic:

E is monotone ⇒ {eqs implements flipE}

lfp eqs

{get.get implementsµE}¯ µE¯ isthe optimal least fixed point of E.

Contents

Local generic solvers

Spying: implementation and specification of modulus

Spying: verification of modulus The conjunction rule

Conclusion Bibliography

The essence of spying

The essence of spying can be distilled in a single combinator, modulus, so named by Longley (1999).

val m o d u l u s :

(( ’ a - > ’ b ) - > ’ c ) - >

(( ’ a - > ’ b ) - > ’ c * ( ’ a l i s t ))

The call “modulus ff f” returns apairof

• the result of the call “ff f”, and

• the list of arguments with which ffhas queried fduring this call.

This is a complete list of points on which ff depends.

Implementation of modulus

Here is a simple-minded imperative implementation of modulus:

let m o d u l u s ff f = let xs = ref [] in let spy x =

(* R e c o r d a d e p e n d e n c y on x : *) xs := x :: ! xs ;

(* F o r w a r d the c a l l to f : *) f x

in

let c = ff spy in ( c , ! xs )

Longley (1999) gives this code and claims (without proof) that it has the desired denotational semantics in the setting of a pure λ-calculus.

Specification of modulus

What is a plausible specification ofmodulus?

{f implements φ∗ff implements F } modulus ff f

{(c,ws).dc =F(φ)e}

The postcondition means that c is the result of the call “ff f”...

and that cdoes not depend on the values taken by foutside of the listws.

“f implements φ” is sugar for the triple∀x.{true}f x {y.dy=φ(x)e}.

“ff implements F” means ∀f, φ.{f implementsφ}ff f {c.dc =F(φ)e}.

Specification of modulus

What is a plausible specification ofmodulus?

{f implements φ∗ff implements F } modulus ff f

{(c,ws).d∀φ⁰. φ⁰ =_ws φ⇒c =F(φ⁰)e}

The postcondition means that c is the result of the call “ff f”... and that cdoes not depend on the values taken by foutside of the listws.

“f implements φ” is sugar for the triple∀x.{true}f x {y.dy=φ(x)e}.

“ff implements F” means ∀f, φ.{f implementsφ}ff f {c.dc =F(φ)e}.

Contents

Local generic solvers

Spying: implementation and specification of modulus Spying: verification of modulus

The conjunction rule Conclusion

Bibliography

Why verifying modulus seems challenging

let m o d u l u s ff f = let xs = ref [] in let spy x =

xs := x :: ! xs ; f x in let c = ff spy in ( c , ! xs )

{f implementsφ∗ff implements F } modulus ff f

{(c,ws).d∀φ⁰. φ⁰=ws φ⇒c=F(φ⁰)e}

ff expects anapparently purefunction as an argument, so we mustprove

“spy implements φ⁰” forsomeφ⁰, and we will get c =F(φ⁰). However,

• Proving c =F(φ⁰) forone functionφ⁰ is not good enough. It seems as though as we need spy to implementallfunctionsφ⁰ at once.

• The set of functionsφ⁰ over which we would like to quantify isnot known in advance— it depends on ws, aresult of modulus.

• What invariant describesxs? Only in the enddoes it hold a complete listws of dependencies.

Ingredients of a solution

• We need spy to implement all functionsφ⁰ at once...

— Use aconjunction ruleto focus on one function φ⁰ at a time.

• The list wsis not known in advance...

— Use aprophecy variable to name this list ahead of time.

• What invariant describesxs?

— The elements currently recordedin !xs, concatenated with those thatwill be recordedin the future, form the list ws.

Ingredients of a solution

• We need spy to implement all functionsφ⁰ at once...

— Use aconjunction ruleto focus on one function φ⁰ at a time.

• The list wsis not known in advance...

— Use aprophecy variable to name this list ahead of time.

• What invariant describesxs?

— The elements currently recordedin !xs, concatenated with those thatwill be recordedin the future, form the list ws.

Ingredients of a solution

• We need spy to implement all functionsφ⁰ at once...

— Use aconjunction ruleto focus on one function φ⁰ at a time.

• The list wsis not known in advance...

— Use aprophecy variable to name this list ahead of time.

• What invariant describesxs?

— The elements currently recordedin !xs, concatenated with those thatwill be recordedin the future, form the list ws.

Ingredients of a solution

• We need spy to implement all functionsφ⁰ at once...

— Use aconjunction ruleto focus on one function φ⁰ at a time.

• The list wsis not known in advance...

— Use aprophecy variable to name this list ahead of time.

• What invariant describesxs?

— The elements currently recordedin !xs, concatenated with those thatwill be recordedin the future, form the list ws.

Past, present and future in Hoare Logic

In Hoare Logic and Separation Logic, assertions describe thecurrentstate.

• e.g., “at this point,!xs is the empty list[]”

The current state, possibly enriched with ghost state, reflects the past.

There is no way of talking about thefuture!

Enter prophecy variables (Abadi and Lamport 1988; Jung et al. 2020).

Past, present and future in Hoare Logic

In Hoare Logic and Separation Logic, assertions describe thecurrentstate.

• e.g., “at this point,!xs is the empty list[]”

The current state, possibly enriched with ghost state, reflects the past.

There is no way of talking about thefuture!

Enter prophecy variables (Abadi and Lamport 1988; Jung et al. 2020).

A prophecy variable primer

A ghost variablewith three operations: allocation, assignment, disposal.

The reasoning rules allow referring to the sequencexs of future writes.

Prophecy Allocation {true}

newProph() {p.∃xs.p will receive xs}

Prophecy Assignment {p will receive xs}

resolveProph p x

().∃xs⁰. dxs =x ::xs⁰e p will receive xs⁰

Prophecy Disposal {p will receive xs}

disposeProph p {().dxs = []e}

A weaker specification for modulus

Instead of establishing thisstrongspecification for modulus...

∀φ⁰.





{f implementsφ∗ff implements F } modulus ff f

{(c,ws).d∀φ⁰.φ⁰ =_ws φ⇒c =F(φ⁰)e}





...let us first establish a weakerspecification.

Then (later), use an infinitary conjunction ruleto argue (roughly) that the weaker spec implies the stronger one.

A weaker specification for modulus

Instead of establishing thisstrong specification formodulus...

∀φ⁰.





{f implementsφ∗ff implements F } modulus ff f

{(c,ws).d

∀φ⁰.

φ⁰ =_ws φ⇒c =F(φ⁰)e}





...let us first establish a weakerspecification.

Then (later), use an infinitary conjunction ruleto argue (roughly) that the weaker spec implies the stronger one.

Proof of modulus

Assume φ⁰ is given.

let m o d u l u s ff f =

let xs , p , lk = ref [] , n e w P r o p h () , n e w L o c k () in let spy x =

let y = f x in

w i t h L o c k lk (fun () - >

xs := x :: ! xs ; r e s o l v e P r o p h p x );

y in

let c = ff spy in

a c q u i r e L o c k lk ; d i s p o s e P r o p h p ; ( c , ! xs )

Step 1. Allocate a prophecy variable p.

Introduce the namews to stand for the list of future writesto p.

Proof of modulus

Assume φ⁰ is given.

let m o d u l u s ff f =

let xs , p , lk = ref [] , n e w P r o p h () , n e w L o c k () in let spy x =

let y = f x in

w i t h L o c k lk (fun () - >

xs := x :: ! xs ; r e s o l v e P r o p h p x );

y in

let c = ff spy in

a c q u i r e L o c k lk ; d i s p o s e P r o p h p ; ( c , ! xs )

Step 2. Allocate a lock lk, which owns xs andp. Its invariant is that the list ws of all writesto p can be split into two parts:

• thepast writes, the reverse of the current contents of xs;

• the remainingfuture writes top.

Proof of modulus

Assume φ⁰ is given.

let m o d u l u s ff f =

let xs , p , lk = ref [] , n e w P r o p h () , n e w L o c k () in let spy x =

let y = f x in

w i t h L o c k lk (fun () - >

xs := x :: ! xs ; r e s o l v e P r o p h p x );

y in

let c = ff spy in

a c q u i r e L o c k lk ; d i s p o s e P r o p h p ; ( c , ! xs )

Step 2. Allocate a lock lk, which owns xs andp. Its invariant is that the list ws of all writesto p can be split into two parts:

• thepast writes, the reverse of the current contents of xs;

• the remainingfuture writes top.

Movingx from one part to the other preserves the invariant.

Proof of modulus

Assume φ⁰ is given.

let m o d u l u s ff f =

let xs , p , lk = ref [] , n e w P r o p h () , n e w L o c k () in let spy x =

let y = f x in

w i t h L o c k lk (fun () - >

xs := x :: ! xs ; r e s o l v e P r o p h p x );

y in

let c = ff spy in

a c q u i r e L o c k lk ; d i s p o s e P r o p h p ; ( c , ! xs )

Because acquireLock exhales the invariant and disposeProph guarantees there are no more future writes, !xs on the last line yields ws (reversed).

Thus, the name ws in the postcondition ofmodulus and the namews introduced by newProph denotethe same set of points.

Proof of modulus

Assume φ⁰ is given.

let m o d u l u s ff f =

let xs , p , lk = ref [] , n e w P r o p h () , n e w L o c k () in let spy x =

let y = f x in

w i t h L o c k lk (fun () - >

xs := x :: ! xs ; r e s o l v e P r o p h p x );

y in

let c = ff spy in

a c q u i r e L o c k lk ; d i s p o s e P r o p h p ; ( c , ! xs )

Step 3. Reason by cases:

• Ifφ⁰=_ws φdoesnothold, then the postcondition ofmodulus istrue. Then, it suffices to prove thatmodulus is safe, which is not difficult.

• Ifφ⁰=ws φdoes hold, continue on to the next slides...

Proof of modulus

Assume φ⁰ is given. Assume φ⁰ =ws φholds.

let m o d u l u s ff f =

let xs , p , lk = ref [] , n e w P r o p h () , n e w L o c k () in let spy x =

let y = f x in

w i t h L o c k lk (fun () - >

xs := x :: ! xs ; r e s o l v e P r o p h p x );

y in

let c = ff spy in

a c q u i r e L o c k lk ; d i s p o s e P r o p h p ; ( c , ! xs )

Step 4. Prove that spy implementsφ⁰.

• We havey =φ(x). We wish to provey =φ⁰(x).

• Because φandφ⁰ coincide onws, the goal boils down tox∈ws.

• x ∈ws holds because we make it holdby writing x to p.

— “there, let me bend reality for you”

Proof of modulus

Assume φ⁰ is given. Assume φ⁰ =ws φholds.

let m o d u l u s ff f =

let xs , p , lk = ref [] , n e w P r o p h () , n e w L o c k () in let spy x =

let y = f x in

w i t h L o c k lk (fun () - >

xs := x :: ! xs ; r e s o l v e P r o p h p x );

y in

let c = ff spy in

a c q u i r e L o c k lk ; d i s p o s e P r o p h p ; ( c , ! xs )

Step 4. Prove that spy implementsφ⁰.

• We havey =φ(x). We wish to provey =φ⁰(x).

• Because φandφ⁰ coincide onws, the goal boils down tox∈ws.

• x ∈ws holds because we make it holdby writing x to p.

— “there, let me bend reality for you”

Proof of modulus

Assume φ⁰ is given. Assume φ⁰ =ws φholds.

let m o d u l u s ff f =

let xs , p , lk = ref [] , n e w P r o p h () , n e w L o c k () in let spy x =

let y = f x in

w i t h L o c k lk (fun () - >

xs := x :: ! xs ; r e s o l v e P r o p h p x );

y in

let c = ff spy in

a c q u i r e L o c k lk ; d i s p o s e P r o p h p ; ( c , ! xs )

Step 4. Prove that spy implementsφ⁰.

• We havey =φ(x). We wish to provey =φ⁰(x).

• Because φandφ⁰ coincide onws, the goal boils down tox∈ws.

• x ∈ws holds because we make it holdby writing x to p.

Proof of modulus

Assume φ⁰ is given. Assume φ⁰ =ws φholds.

let m o d u l u s ff f =

let xs , p , lk = ref [] , n e w P r o p h () , n e w L o c k () in let spy x =

let y = f x in

w i t h L o c k lk (fun () - >

xs := x :: ! xs ; r e s o l v e P r o p h p x );

y in

let c = ff spy in

a c q u i r e L o c k lk ; d i s p o s e P r o p h p ; ( c , ! xs )

Step 5. From “ff implements F” and “spy implementsφ⁰”, deduce that the call “ff spy” is permitted and that c =F(φ⁰) holds.

c =F(φ⁰) is the postcondition of modulus. We are done!

Contents

Local generic solvers

Spying: implementation and specification of modulus Spying: verification of modulus

The conjunction rule Conclusion

Bibliography

Motivation

Recall that, from this weakspecification ofmodulus...

∀φ⁰.





{f implementsφ∗ff implements F } modulus ff f

{(c,ws).d

∀φ⁰.

φ⁰ =_ws φ⇒c =F(φ⁰)e}





...we need to deduce thisstronger specification. This is where an infinitary conjunction ruleis needed.

Motivation

Recall that, from this weakspecification ofmodulus...

∀φ⁰.





{f implementsφ∗ff implements F } modulus ff f

{(c,ws).d∀φ⁰.φ⁰ =_ws φ⇒c =F(φ⁰)e}





...we need to deduce thisstronger specification.

This is where an infinitary conjunction ruleis needed.

An array of conjunction rules

Binary, Non-Dependent {P}e { .dQ₁e}

{P}e { .dQ₂e}

{P}e { .dQ₁∧Q₂e}

Binary, Dependent {P}e {y.dQ₁ye}

{P}e {y.dQ₂ye}

{P}e {y.dQ₁y∧Q₂ye}

Infinitary, Non-Dependent

∀x.{P}e { .dQ xe}

{P}e { .d∀x.Q xe}

Infinitary, Dependent

∀x.{P}e {y.dQ xye}

{P}e {y.d∀x.Q xye}

The non-dependent variants are sound.

The dependent variants may be sound (open question!).

We can derive an approximation that’s good enough for our purposes.

An unsound conjunction rule

All of the previous rules are restricted to purepostconditions.

An unrestricted conjunction rule is unsoundin the presence of ghost state.

Impure (Unsound!) {P}e { .Q₁} {P}e { .Q2} {P}e { .Q1∧Q2}

Open question!

Would this rule be sound if every ghost update was apparent in the code?

Proof outline — infinitary, non-dependent case

Hypothesis: ∀x.{P}e { .dQ xe}

Goal: {P}e { .d∀x.Q xe}

{P}

Case split: (∀x.Q x) ∨ (∃x.¬Q x)

{P ∗ d∀x.Q xe} e

{d∀x.Q xe}

{P ∗ d∃x.¬Q xe}

{∃x.P ∗ d¬Q xe} e

{∃x.dQ xe ∗ d¬Q xe}

{false}

{ d∀x.Q xe }

Proof outline — infinitary, non-dependent case

Hypothesis: ∀x.{P}e { .dQ xe}

Goal: {P}e { .d∀x.Q xe}

{P}

Case split: (∀x.Q x) ∨ (∃x.¬Q x)

{P ∗ d∀x.Q xe} e

{d∀x.Q xe}

{P ∗ d∃x.¬Q xe}

{∃x.P ∗ d¬Q xe} e

{∃x.dQ xe ∗ d¬Q xe}

{false}

{ d∀x.Q xe }

Proof outline — infinitary, non-dependent case

Hypothesis: ∀x.{P}e { .dQ xe}

Goal: {P}e { .d∀x.Q xe}

{P}

Case split: (∀x.Q x) ∨ (∃x.¬Q x)

{P ∗ d∀x.Q xe}

e {d∀x.Q xe}

{P ∗ d∃x.¬Q xe}

{∃x.P ∗ d¬Q xe} e

{∃x.dQ xe ∗ d¬Q xe}

{false}

{ d∀x.Q xe }

Proof outline — infinitary, non-dependent case

Hypothesis: ∀x.{P}e { .dQ xe}

Goal: {P}e { .d∀x.Q xe}

{P}

Case split: (∀x.Q x) ∨ (∃x.¬Q x)

{P ∗ d∀x.Q xe}

e {d∀x.Q xe}

{P ∗ d∃x.¬Q xe}

{∃x.P ∗ d¬Q xe} e

{∃x.dQ xe ∗ d¬Q xe}

{false}

{ d∀x.Q xe }

Proof outline — infinitary, non-dependent case

Hypothesis: ∀x.{P}e { .dQ xe}

Goal: {P}e { .d∀x.Q xe}

{P}

Case split: (∀x.Q x) ∨ (∃x.¬Q x)

{P ∗ d∀x.Q xe}

e {d∀x.Q xe}

{P ∗ d∃x.¬Q xe}

{∃x.P ∗ d¬Q xe}

e

{∃x.dQ xe ∗ d¬Q xe}

{false}

{ d∀x.Q xe }

Proof outline — infinitary, non-dependent case

Hypothesis: ∀x.{P}e { .dQ xe}

Goal: {P}e { .d∀x.Q xe}

{P}

Case split: (∀x.Q x) ∨ (∃x.¬Q x)

{P ∗ d∀x.Q xe}

e {d∀x.Q xe}

{P ∗ d∃x.¬Q xe}

{∃x.P ∗ d¬Q xe}

e

{∃x.dQ xe ∗ d¬Q xe}

{false}

{ d∀x.Q xe }

Proof outline — infinitary, non-dependent case

Hypothesis: ∀x.{P}e { .dQ xe}

Goal: {P}e { .d∀x.Q xe}

{P}

Case split: (∀x.Q x) ∨ (∃x.¬Q x)

{P ∗ d∀x.Q xe}

e {d∀x.Q xe}

{P ∗ d∃x.¬Q xe}

{∃x.P ∗ d¬Q xe}

e

{∃x.dQ xe ∗ d¬Q xe}

{false} { d∀x.Q xe }

The infinitary, dependent case

Same idea, but a prophecy variable must be used to namey ahead of time and allow the case split (∀x.Q xy)∨ ¬(∀x.Q xy).

Infinitary, Dependent

∀x.{P}e {y.dQ xye}

{P}e⁰ {y.d∀x.Q xye}

Because of this,e⁰ in the conclusion is a copy of e instrumented with newProph andresolveProph instructions. (Ouch.)

Contents

Local generic solvers

Spying: implementation and specification of modulus Spying: verification of modulus

The conjunction rule Conclusion

Bibliography

Contributions

• Extension of Iris’s prophecy API: disposeProph; typed prophecies.

• Proof of the conjunction rule.

• Specification and proof ofmodulus.

• Specification and proof of a slightly simplified version of Fix:

E is monotone ⇒ {eqs implements flipE}

lfp eqs

{get.get implements µE}¯ where ¯µE isthe optimal least fixed point of E.

Limitations

A few optimizations are missing, e.g.,

• Fixuses a more efficient representation of the dependency graph.

Caveats:

• Termination is not proved.

• Deadlock-freedom is not proved.

Wishes:

• Is there any way ofnotpolluting the code with operations on prophecy variables?

Take-home messages

Spying is another archetypical use of hidden state.

Prophecy variablesare fun,

and they can be useful not just in concurrent code, but also insequential code.

Contents

Local generic solvers

Spying: implementation and specification of modulus Spying: verification of modulus

The conjunction rule Conclusion

Bibliography

References I

Abadi, Martin and Leslie Lamport (1988). The Existence of Refinement Mappings. In:Logic in Computer Science (LICS), pp. 165–175.

Le Charlier, Baudouin and Pascal Van Hentenryck (1992). A Universal Top-Down Fixpoint Algorithm. Technical Report CS-92-25. Brown University.

Vergauwen, Bart and Johan Lewi (1994). Efficient Local Correctness Checking for Single and Alternating Boolean Equation Systems. In:

International Colloquium on Automata, Languages and Programming.

Vol. 820. Lecture Notes in Computer Science. Springer, pp. 304–315.

Fecht, Christian and Helmut Seidl (1999). A Faster Solver for General Systems of Equations. In:Science of Computer Programming 35.2–3, pp. 137–162.

References II

Longley, John (1999). When is a Functional Program Not a Functional Program?. In: International Conference on Functional Programming (ICFP), pp. 1–7.

Pottier, Fran¸cois (2009).Lazy Least Fixed Points in ML. Unpublished.

Jung, Ralf et al. (2020). The Future is Ours: Prophecy Variables in Separation Logic. In:Proceedings of the ACM on Programming Languages POPL.