Verifying a hash table and its iterators in higher-order separation logic

(1)

Verifying a hash table and its iterators in higher-order separation logic

François Pottier

CPP 2017

Paris, January 16, 2017

(2)

We want verified software...

(3)

Therefore, we need

VERIFIED

LIBRARIES.

(4)

The Vocal project is building a verified library of basic data structures and algorithms.

I

The code is in OCaml.

I

Verification can be done in higher-order separation logic :

I

Charguéraud’s CFML imports a view of the code into Coq ;

I

reasoning is carried out in Coq.

In this talk, I focus on one module : a hash table implementation.

(5)

Why verify a hash table implementation ?

I

a simple and useful data structure Why talk about it today ?

I

dynamically allocated ; mutable

I

equipped with two iteration mechanisms : fold, cascade

(6)

The data structure

First-order operations

Iteration via fold

Iteration via cascades

Conclusion

(7)

OCaml interface

An excerpt of HashTable.mli.

m o d u l e M a k e ( K : H a s h e d T y p e ) : s i g t y p e key = K . t

t y p e ’ a t (* C r e a t i o n . *)

v a l c r e a t e : int - > ’ a t v a l c o p y : ’ a t - > ’ a t (* I n s e r t i o n a n d r e m o v a l . *)

v a l add : ’ a t - > key - > ’ a - > u n i t v a l r e m o v e : ’ a t - > key - > u n i t (* L o o k u p . *)

v a l f i n d : ’ a t - > key - > ’ a o p t i o n v a l p o p u l a t i o n : ’ a t - > int

(* I t e r a t i o n . *)

v a l f o l d : ( key - > ’ a - > ’ b - > ’ b ) - >

’ a t - > ’ b - > ’ b v a l c a s c a d e : ’ a t - > ( key * ’ a ) c a s c a d e (* . . . m o r e o p e r a t i o n s , n o t s h o w n . *) e n d

(8)

OCaml interface

An excerpt of HashTable.mli.

t y p e ’ a t (* C r e a t i o n . *)

(* I t e r a t i o n . *)

v a l f o l d : ( key - > ’ a - > ’ b - > ’ b ) - >

First-order operations

(9)

OCaml interface

An excerpt of HashTable.mli.

t y p e ’ a t (* C r e a t i o n . *)

(* I t e r a t i o n . *)

v a l f o l d : ( key - > ’ a - > ’ b - > ’ b ) - >

Iteration

(producer in control)

(10)

OCaml interface

An excerpt of HashTable.mli.

t y p e ’ a t (* C r e a t i o n . *)

(* I t e r a t i o n . *)

v a l f o l d : ( key - > ’ a - > ’ b - > ’ b ) - >

Iteration

(consumer in control)

(11)

OCaml implementation

An excerpt of HashTable.ml.

m o d u l e M a k e ( K : H a s h e d T y p e ) = s t r u c t (* T y p e d e f i n i t i o n s . *)

t y p e key = K . t t y p e ’ a b u c k e t =

V o i d

| M o r e of key * ’ a * ’ a b u c k e t t y p e ’ a t a b l e = {

m u t a b l e d a t a : ’ a b u c k e t a r r a y ; m u t a b l e p o p u : int ;

i n i t : int ; }

t y p e ’ a t = ’ a t a b l e

(* O p e r a t i o n s : s e e f o l l o w i n g s l i d e s . . . *) e n d

(12)

OCaml implementation

An excerpt of HashTable.ml.

V o i d

i n i t : int ; }

t y p e ’ a t = ’ a t a b l e

A hash table is a record...

(13)

OCaml implementation

An excerpt of HashTable.ml.

V o i d

i n i t : int ; }

t y p e ’ a t = ’ a t a b l e

...whose data field is an

array of buckets...

(14)

OCaml implementation

An excerpt of HashTable.ml.

V o i d

i n i t : int ; }

t y p e ’ a t = ’ a t a b l e

...where a bucket is a list

of key-value pairs.

(15)

Separation logic invariant (in Coq)

An excerpt of HashTable_proof.v.

I m p l i c i t T y p e M : key - > l i s t A . D e f i n i t i o n h ˜ > TableInState M s :=

H e x i s t s d pop i n i t data , h ˜ > ‘{

d a t a := d ; p o p u := pop ; i n i t := i n i t } \*

d ˜ > Array data \*

\[ t a b l e _ i n v M i n i t d a t a ] \*

\[ p o p u l a t i o n M = pop ] \*

\[ s = ( d , d a t a ) ].

D e f i n i t i o n h ˜ > Table M :=

H e x i s t s s , h ˜ > TableInState M s.

We use s to demand / guarantee that certain operations are read-only.

(16)

Separation logic invariant (in Coq)

An excerpt of HashTable_proof.v.

\[ s = ( d , d a t a ) ].

A table represents a finite map of keys to (lists of) values.

We use s to demand / guarantee that certain operations are read-only.

(17)

Separation logic invariant (in Coq)

An excerpt of HashTable_proof.v.

\[ s = ( d , d a t a ) ].

This SL predicate asserts

“the table at address h encodes the dictionary M”.

We use s to demand / guarantee that certain operations are read-only.

(18)

Separation logic invariant (in Coq)

An excerpt of HashTable_proof.v.

\[ s = ( d , d a t a ) ].

This one names s the current concrete state

of the table.

We use s to demand / guarantee that certain operations are read-only.

(19)

Separation logic invariant (in Coq)

An excerpt of HashTable_proof.v.

\[ s = ( d , d a t a ) ].

There must be a record at address h...

We use s to demand / guarantee that certain operations are read-only.

(20)

Separation logic invariant (in Coq)

An excerpt of HashTable_proof.v.

\[ s = ( d , d a t a ) ].

...whose data field contains a pointer d...

We use s to demand / guarantee that certain operations are read-only.

(21)

Separation logic invariant (in Coq)

An excerpt of HashTable_proof.v.

\[ s = ( d , d a t a ) ].

...to an array.

We use s to demand / guarantee that certain operations are read-only.

(22)

Separation logic invariant (in Coq)

An excerpt of HashTable_proof.v.

\[ s = ( d , d a t a ) ].

The content of memory is related to M.

We use s to demand / guarantee that certain operations are read-only.

(23)

Separation logic invariant (in Coq)

An excerpt of HashTable_proof.v.

\[ s = ( d , d a t a ) ].

The address and content of the array are exposed

under the name s.

We use s to demand / guarantee that certain operations are read-only.

(24)

Separation logic invariant (in Coq)

An excerpt of HashTable_proof.v.

\[ s = ( d , d a t a ) ].

We hide s when we do not care about it.

We use s to demand / guarantee that certain operations are read-only.

(25)

The data structure

Iteration via fold

Conclusion

(26)

Specifying a first-order operation : insertion

The effect of add h k x is to add the key-value pair (k, x) to the dictionary.

This is stated as a Hoare triple :

T h e o r e m a d d _ s p e c : f o r a l l M h k x , app MK . add [ h k x ]

PRE ( h ˜ > Table M)

P O S T ( fun _ = > H e x i s t s M ’ , h ˜ > Table M ’ \*

\[ M ’ = add M k x ] \*

\[ l e a n M - > M k = nil - > l e a n M ’ ]).

(27)

Specifying a first-order operation : insertion

The effect of add h k x is to add the key-value pair (k, x) to the dictionary.

This is stated as a Hoare triple :

\[ M ’ = add M k x ] \*

The function call add h k x...

(28)

Specifying a first-order operation : insertion

The effect of add h k x is to add the key-value pair (k, x) to the dictionary.

This is stated as a Hoare triple :

\[ M ’ = add M k x ] \*

...requires a valid table...

(29)

Specifying a first-order operation : insertion

The effect of add h k x is to add the key-value pair (k, x) to the dictionary.

This is stated as a Hoare triple :

\[ M ’ = add M k x ] \*

...and produces a valid table...

(30)

Specifying a first-order operation : insertion

The effect of add h k x is to add the key-value pair (k, x) to the dictionary.

This is stated as a Hoare triple :

\[ M ’ = add M k x ] \*

...representing a dictionary

with one more key-value pair.

(31)

The data structure

Iteration via fold

Conclusion

(32)

Fold – for hash tables

l e t r e c f o l d _ a u x f b a c c u = m a t c h b w i t h

| V o i d - >

a c c u

| M o r e ( k , x , b ) - >

l e t a c c u = f k x a c c u in f o l d _ a u x f b a c c u

l e t f o l d f h a c c u = l e t d a t a = h . d a t a in l e t s t a t e = ref a c c u in

f o r i = 0 to A r r a y . l e n g t h d a t a - 1 do s t a t e := f o l d _ a u x f d a t a .( i ) ! s t a t e d o n e;

! s t a t e

Writing a specification for a fold raises some questions :

I

in what order does the consumer receive the key-value pairs ?

I

is the consumer allowed to access the table for reading ? for writing ?

(33)

Fold – for hash tables

| V o i d - >

a c c u

| M o r e ( k , x , b ) - >

! s t a t e

A loop over the data array...

Writing a specification for a fold raises some questions :

I

in what order does the consumer receive the key-value pairs ?

I

is the consumer allowed to access the table for reading ? for writing ?

(34)

Fold – for hash tables

| V o i d - >

a c c u

| M o r e ( k , x , b ) - >

! s t a t e

...a loop over a linked list...

Writing a specification for a fold raises some questions :

I

in what order does the consumer receive the key-value pairs ?

I

is the consumer allowed to access the table for reading ? for writing ?

(35)

Fold – for hash tables

| V o i d - >

a c c u

| M o r e ( k , x , b ) - >

! s t a t e

...a call to the consumer.

Writing a specification for a fold raises some questions :

I

in what order does the consumer receive the key-value pairs ?

I

is the consumer allowed to access the table for reading ? for writing ?

(36)

Fold – for hash tables

| V o i d - >

a c c u

| M o r e ( k , x , b ) - >

! s t a t e

Writing a specification for a fold raises some questions :

I

in what order does the consumer receive the key-value pairs ?

I

is the consumer allowed to access the table for reading ? for writing ?

(37)

Specifying an iteration order – in general

Really a matter of specifying which orders the consumer may observe.

The events that can be observed by a consumer are :

I

the production of one element ;

I

the end of the sequence (this event occurs at most once, and occurs last).

An observation can be defined as a sequence of events.

A set of observations can be described by two predicates (Filliâtre and Pereira) :

V a r i a b l e s p e r m i t t e d c o m p l e t e : l i s t A - > P r o p .

(38)

Specifying fold – in general

This is a higher-order specification : an implication between Hoare triples.

V a r i a b l e s p e r m i t t e d c o m p l e t e : l i s t A - > P r o p . V a r i a b l e I : l i s t A - > B - > h p r o p .

V a r i a b l e s S S ’ : C - > h p r o p . D e f i n i t i o n F o l d := f o r a l l f c ,

( f o r a l l x xs accu , p e r m i t t e d ( xs & x ) - >

c a l l f x a c c u

PRE ( S ’ c \* I xs a c c u )

P O S T ( fun a c c u = > S ’ c \* I ( xs & x ) a c c u ) ) - >

f o r a l l accu ,

app f o l d [ f c a c c u ]

PRE ( S c \* I nil a c c u ) P O S T ( fun a c c u = > H e x i s t s xs ,

S c \* I xs a c c u \*

\[ c o m p l e t e xs ]).

(39)

Specifying fold – in general

This is a higher-order specification : an implication between Hoare triples.

c a l l f x a c c u

f o r a l l accu ,

The spec is parameterized

over permitted and complete.

(40)

Specifying fold – in general

This is a higher-order specification : an implication between Hoare triples.

c a l l f x a c c u

f o r a l l accu ,

The consumer may assume every partial sequence she

observes is permitted.

(41)

Specifying fold – in general

This is a higher-order specification : an implication between Hoare triples.

c a l l f x a c c u

f o r a l l accu ,

After observing termination,

the consumer may deduce

the sequence is complete.

(42)

Specifying fold – in general

This is a higher-order specification : an implication between Hoare triples.

c a l l f x a c c u

f o r a l l accu ,

The spec is parameterized

over a loop invariant I.

(43)

Specifying fold – in general

This is a higher-order specification : an implication between Hoare triples.

c a l l f x a c c u

f o r a l l accu ,

The consumer

must preserve I.

(44)

Specifying fold – in general

This is a higher-order specification : an implication between Hoare triples.

c a l l f x a c c u

f o r a l l accu ,

The whole iteration is then

guaranteed to preserve I.

(45)

Specifying fold – in general

This is a higher-order specification : an implication between Hoare triples.

c a l l f x a c c u

f o r a l l accu ,

The spec is parameterized

over SL predicates S and S’.

(46)

Specifying fold – in general

This is a higher-order specification : an implication between Hoare triples.

c a l l f x a c c u

f o r a l l accu ,

The producer requires S

access to the collection.

(47)

Specifying fold – in general

This is a higher-order specification : an implication between Hoare triples.

c a l l f x a c c u

f o r a l l accu ,

The producer gets S’ access,

which may be weaker.

(48)

Specifying an iteration order – for hash tables

For hash tables, we give concrete definitions of permitted and complete :

D e f i n i t i o n p e r m i t t e d kxs :=

e x i s t s M ’ , r e m o v a l M kxs M ’.

D e f i n i t i o n c o m p l e t e kxs :=

r e m o v a l M kxs e m p t y .

where removal M kxs M’ means that from M one may remove the key-value-pair sequence kxs to obtain M’.

This specification is semi-deterministic :

I

two key-value pairs for different keys may be observed in any order ;

I

two key-value pairs for the same key must be observed most-recent-value-first.

(49)

Specifying fold – for hash tables

The specification of fold for hash tables is an instance of the general spec :

T h e o r e m f o l d _ s p e c _ r o : f o r a l l M s B I , F o l d MK . f o l d

(* C a l l i n g c o n v e n t i o n : *) ( fun f kx ( a c c u : B ) = >

app f [( fst kx ) ( snd kx ) a c c u ]) (* P e r m i t t e d / c o m p l e t e s e q u e n c e s : *) ( p e r m i t t e d M ) ( c o m p l e t e M ) I

(* f o l d r e q u i r e s & p r e s e r v e s t h i s : *) ( fun h = > h ˜ > TableInState M s)

(* f r e c e i v e s a n d m u s t p r e s e r v e t h i s : *) ( fun h = > h ˜ > TableInState M s ).

This spec allows read-only access to the table during iteration,

and guarantees that iteration itself is a read-only operation.

(50)

Specifying fold – for hash tables

The specification of fold for hash tables is an instance of the general spec :

The predicates permitted and complete for hash tables.

This spec allows read-only access to the table during iteration,

and guarantees that iteration itself is a read-only operation.

(51)

Specifying fold – for hash tables

The specification of fold for hash tables is an instance of the general spec :

fold guarantees that the table is not modified.

This spec allows read-only access to the table during iteration,

and guarantees that iteration itself is a read-only operation.

(52)

Specifying fold – for hash tables

The specification of fold for hash tables is an instance of the general spec :

The consumer cannot modify the table.

This spec allows read-only access to the table during iteration,

and guarantees that iteration itself is a read-only operation.

(53)

Specifying fold – for hash tables

If access to the table during iteration is not needed, a simpler spec can be given :

T h e o r e m f o l d _ s p e c :

f o r a l l M B I , F o l d MK . f o l d

( fun f kx ( a c c u : B ) = >

app f [( fst kx ) ( snd kx ) a c c u ]) ( p e r m i t t e d M ) ( c o m p l e t e M ) I

(* f o l d r e q u i r e s & p r e s e r v e s t h i s : *) ( fun h = > h ˜ > Table M)

(* f c a n n o t a c c e s s t h e t a b l e : *) ( fun h = > \ [ ] ) .

(54)

Specifying fold – for hash tables

If access to the table during iteration is not needed, a simpler spec can be given :

( fun f kx ( a c c u : B ) = >

fold does not guarantee that

the table is unchanged.

(55)

Specifying fold – for hash tables

If access to the table during iteration is not needed, a simpler spec can be given :

( fun f kx ( a c c u : B ) = >

The consumer gets no

access to the table.

(56)

The data structure

Iteration via fold

Conclusion

(57)

Iterators

An iterator is an on-demand producer of a sequence of elements.

(58)

Iterators

What should be the type of an iterator ?

p u b l i c i n t e r f a c e

I t e r a t o r < E > {

E n e x t ()

t h r o w s

N o S u c h E l e m e n t E x c e p t i o n ;

b o o l e a n

h a s N e x t ();

}

This interface :

I

requires the iterator to be mutable ;

I

is more complex than strictly necessary.

(59)

Iterators

What should be the type of an iterator ?

I t e r a t o r < E > {

E n e x t ()

t h r o w s

N o S u c h E l e m e n t E x c e p t i o n ;

b o o l e a n

h a s N e x t ();

}

This interface :

I

requires the iterator to be mutable ;

I

is more complex than strictly necessary.

(60)

Iterators

What should be the type of an iterator ?

I t e r a t o r < E > {

E n e x t ()

t h r o w s

N o S u c h E l e m e n t E x c e p t i o n ;

b o o l e a n

h a s N e x t ();

}

NOT GREAT

This interface :

I

requires the iterator to be mutable ;

I

is more complex than strictly necessary.

(61)

Cascades

A cascade, or delayed list, is perhaps the simplest possible form of iterator.

t y p e ’ a c a s c a d e = u n i t - > ’ a h e a d a n d ’ a h e a d =

| Nil

| C o n s of ’ a * ’ a c a s c a d e

This definition offers an abstract, consumer-oriented view. It does not reveal :

I

whether a cascade has mutable internal state, or is pure ;

I

whether elements are stored in memory, or computed on demand ;

I

whether elements are re-computed when re-demanded, or memoized.

(62)

Cascades

A cascade, or delayed list, is perhaps the simplest possible form of iterator.

| Nil

Computation occurs on demand...

This definition offers an abstract, consumer-oriented view. It does not reveal :

I

whether a cascade has mutable internal state, or is pure ;

I

whether elements are stored in memory, or computed on demand ;

I

whether elements are re-computed when re-demanded, or memoized.

(63)

Cascades

A cascade, or delayed list, is perhaps the simplest possible form of iterator.

| Nil

...yielding either end-of-sequence...

This definition offers an abstract, consumer-oriented view. It does not reveal :

I

whether a cascade has mutable internal state, or is pure ;

I

whether elements are stored in memory, or computed on demand ;

I

whether elements are re-computed when re-demanded, or memoized.

(64)

Cascades

A cascade, or delayed list, is perhaps the simplest possible form of iterator.

| Nil

...or an element and a tail.

This definition offers an abstract, consumer-oriented view. It does not reveal :

I

whether a cascade has mutable internal state, or is pure ;

I

whether elements are stored in memory, or computed on demand ;

I

whether elements are re-computed when re-demanded, or memoized.

(65)

Cascades

A cascade, or delayed list, is perhaps the simplest possible form of iterator.

| Nil

This definition offers an abstract, consumer-oriented view. It does not reveal :

I

whether a cascade has mutable internal state, or is pure ;

I

whether elements are stored in memory, or computed on demand ;

I

whether elements are re-computed when re-demanded, or memoized.

(66)

Cascades

A cascade, or delayed list, is perhaps the simplest possible form of iterator.

| Nil

This definition offers an abstract, consumer-oriented view. It does not reveal :

I

whether a cascade has mutable internal state, or is pure ;

I

whether elements are stored in memory, or computed on demand ;

I

whether elements are re-computed when re-demanded, or memoized.

Cascades are easy to build and use because they are

“just like lists”.

(67)

A cascade – for hash tables

Constructing a cascade is like constructing a list of all key-value pairs...

l e t r e c c a s c a d e _ a u x d a t a i b = m a t c h b w i t h

| M o r e ( k , x , b ) - >

C o n s ( ( k , x ) ,

f u n () - > c a s c a d e _ a u x d a t a i b )

| V o i d - >

l e t i = i + 1 in

if i < A r r a y . l e n g t h d a t a t h e n c a s c a d e _ a u x d a t a i d a t a .( i ) e l s e

Nil l e t c a s c a d e h =

l e t d a t a = h . d a t a in l e t b = d a t a . ( 0 ) in f u n () - >

c a s c a d e _ a u x d a t a 0 b

(68)

A cascade – for hash tables

Constructing a cascade is like constructing a list of all key-value pairs...

| M o r e ( k , x , b ) - >

C o n s ( ( k , x ) ,

| V o i d - >

l e t i = i + 1 in

...with a delay.

(69)

A cascade – for hash tables

Constructing a cascade is like constructing a list of all key-value pairs...

| M o r e ( k , x , b ) - >

C o n s ( ( k , x ) ,

| V o i d - >

l e t i = i + 1 in

The cascade must not be

used after the table is

modified!

(70)

Specifying a cascade – in general

A cascade is a function that returns an element and a cascade.

We use an impredicative encoding of this co-inductive specification.

V a r i a b l e I : h p r o p .

V a r i a b l e s p e r m i t t e d c o m p l e t e : l i s t A - > P r o p . D e f i n i t i o n c ˜ > Cascade xs :=

H e x i s t s S : l i s t A - > f u n c - > hprop , S xs c \*

\[ f o r a l l xs c , d u p l i c a b l e ( S xs c ) ] \*

\[ f o r a l l xs c , S xs c == > S xs c \* \[ p e r m i t t e d xs ]] \*

\[ f o r a l l xs c , app c [ tt ]

INV ( S xs c \* I ) P O S T ( fun o = >

m a t c h o w i t h

| Nil = > \[ c o m p l e t e xs ]

| C o n s x c = > S ( xs & x ) c end ) ].

(71)

Specifying a cascade – in general

A cascade is a function that returns an element and a cascade.

We use an impredicative encoding of this co-inductive specification.

m a t c h o w i t h

| Nil = > \[ c o m p l e t e xs ]

| C o n s x c = > S ( xs & x ) c end ) ].

The cascade has internal

invariant S...

(72)

Specifying a cascade – in general

A cascade is a function that returns an element and a cascade.

We use an impredicative encoding of this co-inductive specification.

m a t c h o w i t h

| Nil = > \[ c o m p l e t e xs ]

| C o n s x c = > S ( xs & x ) c end ) ].

...which must be duplicable.

A cascade is persistent.

(73)

Specifying a cascade – in general

A cascade is a function that returns an element and a cascade.

We use an impredicative encoding of this co-inductive specification.

m a t c h o w i t h

| Nil = > \[ c o m p l e t e xs ]

| C o n s x c = > S ( xs & x ) c end ) ].

Calling c requires validity

and produces

another valid cascade.

(74)

Specifying a cascade – in general

A cascade is a function that returns an element and a cascade.

We use an impredicative encoding of this co-inductive specification.

m a t c h o w i t h

| Nil = > \[ c o m p l e t e xs ]

| C o n s x c = > S ( xs & x ) c end ) ].

The spec is parameterized

over permitted and complete.

(75)

Specifying a cascade – in general

A cascade is a function that returns an element and a cascade.

We use an impredicative encoding of this co-inductive specification.

m a t c h o w i t h

| Nil = > \[ c o m p l e t e xs ]

| C o n s x c = > S ( xs & x ) c end ) ].

The consumer may assume

that the partial sequence

produced so far is permitted.

(76)

Specifying a cascade – in general

A cascade is a function that returns an element and a cascade.

We use an impredicative encoding of this co-inductive specification.

m a t c h o w i t h

| Nil = > \[ c o m p l e t e xs ]

| C o n s x c = > S ( xs & x ) c end ) ].

Upon termination, the consumer may deduce that

the sequence is complete.

(77)

Specifying a cascade – in general

A cascade is a function that returns an element and a cascade.

We use an impredicative encoding of this co-inductive specification.

m a t c h o w i t h

| Nil = > \[ c o m p l e t e xs ]

| C o n s x c = > S ( xs & x ) c end ) ].

The cascade has access to

an underlying data structure.

(78)

Specifying a cascade – for hash tables

T h e o r e m c a s c a d e _ s p e c : f o r a l l h M s , app MK . c a s c a d e [ h ]

INV ( h ˜ > TableInState M s) P O S T ( fun c = >

c ˜ > Cascade

( h ˜ > TableInState M s) ( p e r m i t t e d M ) ( c o m p l e t e M ) nil

).

“Concurrent modifications” are disallowed.

(79)

Specifying a cascade – for hash tables

c ˜ > Cascade

).

Same predicates permitted and complete as in fold.

“Concurrent modifications” are disallowed.

(80)

Specifying a cascade – for hash tables

c ˜ > Cascade

).

The cascade can be used only as long as the table

remains in state s.

“Concurrent modifications” are disallowed.

(81)

The data structure

Iteration via fold

Conclusion

(82)