Visitors Unchained Using visitors to traverse abstract syntax with binding François Pottier ICFP 2017, Oxford September 5, 2017

(1)

Visitors Unchained

Using visitors

to traverse abstract syntax with binding

François Pottier

ICFP 2017, Oxford

September 5, 2017

(2)

Visitors Unchained

Usingvisitors

to traverse abstract syntax with binding

François Pottier

ICFP 2017, Oxford September 5, 2017

OBJECTS

AT ICFP!?

(3)

Visitors Unchained

Using visitors

to traverseabstract syntax with binding

François Pottier

ICFP 2017, Oxford September 5, 2017

OBJECTS

AT ICFP!? BINDERS,

AGAIN!?

(4)

Boilerplate alert!

Manipulating abstract syntax with binding can be a chore.

Whenever one creates a new language, one must typically implement:

I substitution,α-equivalence,free names, – (all representations)

I opening/closing,shifting, – (some representations)

I convertingbetween representations...

This boilerplate code is known as nameplate (Cheney).

It is large, boring, error-prone.

It does not seem easily reusable, as it is datatype-specific.

(5)

Fighting boilerplate!

In this talk & paper:

A way of getting rid of nameplate, in OCaml, which

I

supports multiple representations of names and conversions between them,

I

supports complex binding constructs,

I

is modular and open-ended, that is, user-extensible,

I

relies on as little code generation as possible.

Based on a combination of auto-generated visitors and library code.

I

visitors, an OCaml syntax extension (released);

I

AlphaLib, an OCaml library (at a preliminary stage).

(6)

Wait! Isn’t this a solved problem already?

Several Haskell libraries address this problem: FreshLib, Unbound, Bound...

They exploit Haskell’s support for generic programming (SYB, RepLib, ...) Cool stuff can be done in Coq (Schäfer

et al., 2015) and

Agda (Allais

et al., 2017).

In the OCaml world:

I

C

α

ml (F.P., 2005), an ad hoc code generator; monolithic, inflexible.

I

Yallop ports SYB to MetaOCaml+implicits: next talk! The way of the future?

– this talk: making do with vanilla OCaml.

(7)

Visitors

(8)

Generating a “map” visitor, in a nutshell

Annotating a type definition with [@@deriving visitors { ... }]...

t y p e e x p r =

| E C o n s t of int

| E A d d of e x p r * e x p r

[ @ @ d e r i v i n g v i s i t o r s { v a r i e t y = " map " }]

... causes a visitor class to be auto-generated:

c l a s s v i r t u a l [ ’ s e l f ] map = o b j e c t ( s e l f : ’ s e l f ) i n h e r i t [ _ ] V i s i t o r s R u n t i m e . map

m e t h o d v i s i t _ E C o n s t env c0 =

l e t r0 = s e l f # v i s i t _ i n t env c0 in E C o n s t r0

m e t h o d v i s i t _ E A d d env c0 c1 =

l e t r0 = s e l f # v i s i t _ e x p r env c0 in l e t r1 = s e l f # v i s i t _ e x p r env c1 in E A d d ( r0 , r1 )

m e t h o d v i s i t _ e x p r env t h i s = m a t c h t h i s w i t h

| E C o n s t c0 - >

s e l f # v i s i t _ E C o n s t env c0

| E A d d ( c0 , c1 ) - >

s e l f # v i s i t _ E A d d env c0 c1 e n d

(9)

Generating a “map” visitor, in a nutshell

Annotating a type definition with [@@deriving visitors { ... }]...

t y p e e x p r =

... causes a visitor class to be auto-generated:

| E C o n s t c0 - >

| E A d d ( c0 , c1 ) - >

(10)

Generating a “map” visitor, in a nutshell

Annotating a type definition with [@@deriving visitors { ... }]...

t y p e e x p r =

... causes a visitor class to be auto-generated:

| E C o n s t c0 - >

| E A d d ( c0 , c1 ) - >

one method per

data type

(11)

Generating a “map” visitor, in a nutshell

Annotating a type definition with [@@deriving visitors { ... }]...

t y p e e x p r =

... causes a visitor class to be auto-generated:

| E C o n s t c0 - >

| E A d d ( c0 , c1 ) - >

one method per

data constructor

(12)

Generating a “map” visitor, in a nutshell

Annotating a type definition with [@@deriving visitors { ... }]...

t y p e e x p r =

... causes a visitor class to be auto-generated:

| E C o n s t c0 - >

| E A d d ( c0 , c1 ) - >

default behavior

is to rebuild a tree

(13)

Generating a “map” visitor, in a nutshell

Annotating a type definition with [@@deriving visitors { ... }]...

t y p e e x p r =

... causes a visitor class to be auto-generated:

| E C o n s t c0 - >

| E A d d ( c0 , c1 ) - >

an environment

is pushed down

(14)

Using a “map” visitor, in a nutshell

Inherit a visitor class and override one or more methods:

l e t o p t i m i z e : e x p r - > e x p r = l e t v = o b j e c t( s e l f )

i n h e r i t [ _ ] map

m e t h o d! v i s i t _ E A d d env e1 e2 =

m a t c h s e l f # v i s i t _ e x p r env e1 , s e l f # v i s i t _ e x p r env e2 w i t h

| E C o n s t 0 , e (* 0 + e = e *)

| e , E C o n s t 0 - > e (* e + 0 = e *)

| e1 , e2 - > E A d d ( e1 , e2 ) e n d in

v # v i s i t _ e x p r ()

No changes to this code are needed when more expression forms are added.

(15)

Visiting preexisting / parameterized types

Integers and lists can be visited, too.

t y p e e x p r =

| E A d d of e x p r l i s t

m e t h o d v i s i t _ E A d d env c0 =

l e t r0 = s e l f # v i s i t _ l i s t ( s e l f # v i s i t _ e x p r ) env c0 in E A d d r0

m e t h o d v i s i t _ e x p r env t h i s = ...

e n d

A visitor can be “taught” to traverse a data structure!

a preexisting

type

(16)

Visiting preexisting / parameterized types

Integers and lists can be visited, too.

t y p e e x p r =

e n d

A visitor can be “taught” to traverse a data structure!

visitor method

is inherited

(17)

Visiting preexisting / parameterized types

Integers and lists can be visited, too.

t y p e e x p r =

e n d

A visitor can be “taught” to traverse a data structure!

a preexisting

parameterized type

(18)

Visiting preexisting / parameterized types

Integers and lists can be visited, too.

t y p e e x p r =

e n d

A visitor can be “taught” to traverse a data structure!

inherited visitor method is

passed a visitor function

(19)

Visitors – summary

Although they follow fixed patterns, visitors are quite versatile.

They are customizable and composable.

More fun with visitors:

I visitors foropen data typesand their fixed points(link);

I visitors forhash-consed data structures(link);

I iteratorsout of visitors(link).

In the remainder of this talk:

I

Traversing abstract syntax with binding.

(20)

Visitors Unchained

(21)

Traversing syntax with binding

For modularity, it seems desirable to distinguish three concerns:

1. Describing a binding construct.

2. Describing an operation on terms.

I usually specific of onerepresentationof names and binders,

I sometimes specific oftwosuch representations, e.g.,conversions.

3. The end user should be insulated from this complexity.

– 1 & 2 are part of AlphaLib.

(22)

(23)

end user

operations specialist

binder maestro

(24)

The end user

(25)

The end user

The end user describes the structure of ASTs in a concise, declarative style.

For example, this is the syntax of the

λ

-calculus:

t y p e ( ’ bn , ’ fn ) t e r m =

| T V a r of ’ fn

| T L a m b d a of ( ’ bn , ( ’ bn , ’ fn ) t e r m ) abs

| T A p p of ( ’ bn , ’ fn ) t e r m * ( ’ bn , ’ fn ) t e r m [ @ @ d e r i v i n g v i s i t o r s

{ v a r i e t y = " map " ; a n c e s t o r s = [ " B i n d i n g F o r m s . map " ] }]

The type (’bn, ’term) abs represents an abstraction of one name in one term.

The end user gets a visitor for free.

He gets multiple representations of names:

t y p e r a w _ t e r m = ( string , s t r i n g ) t e r m t y p e n o m i n a l _ t e r m = ( A t o m . t , A t o m . t ) t e r m t y p e d e b r u i j n _ t e r m = ( unit , int ) t e r m

(26)

The end user

The end user describes the structure of ASTs in a concise, declarative style.

For example, this is the syntax of the

λ

-calculus:

t y p e ( ’ bn , ’ fn ) t e r m =

| T V a r of ’ fn

The type (’bn, ’term) abs represents an abstraction of one name in one term.

provided by the binder maestro

The end user gets a visitor for free.

He gets multiple representations of names:

(27)

The end user

The end user describes the structure of ASTs in a concise, declarative style.

For example, this is the syntax of the

λ

-calculus:

t y p e ( ’ bn , ’ fn ) t e r m =

| T V a r of ’ fn

The type (’bn, ’term) abs represents an abstraction of one name in one term.

The end user gets a visitor for free.

He gets multiple representations of names:

(28)

The end user

The end user describes the structure of ASTs in a concise, declarative style.

For example, this is the syntax of the

λ

-calculus:

t y p e ( ’ bn , ’ fn ) t e r m =

| T V a r of ’ fn

The type (’bn, ’term) abs represents an abstraction of one name in one term.

The end user gets a visitor for free.

He gets multiple representations of names:

(29)

The binder

maestro

(30)

The binder maestro

The maestro writes the module BindingForms. (Part of AlphaLib.) He defines binding constructs and teaches visitors how to traverse them.

In memory, an abstraction of one name in one term is just a pair:

t y p e ( ’ bn , ’ t e r m ) abs = ’ bn * ’ t e r m

Traversing it requires extending the environment — roughly like this:

c l a s s v i r t u a l [ ’ s e l f ] map = o b j e c t ( s e l f : ’ s e l f ) (* A v i s i t o r m e t h o d for the t y p e abs . *)

m e t h o d v i s i t _ a b s _ visit_ ’ t e r m env ( x1 , t1 ) =

l e t env , x2 = s e l f # e x t e n d env x1 in (* e x t e n d env w i t h x1 *)

l e t t2 = visit_ ’ t e r m env t1 in (* t h e n v i s i t t1 *)

( x2 , t2 ) e n d

There is a catch, though – what on earth should the method extend do?

(31)

The catch

The binder maestro:

I

does not know what operation is being performed,

I

does not know what representation(s) of names are in use,

I

therefore does not know the types of names and environments,

I

let alone how to extend the environment.

What he knows is where and with what names to extend the environment.

(32)

A deal

The binder maestro agrees on a deal with the operations specialist.

“I tell you when to extend the environment; you do the dirty work.”

The binder maestro calls a method which the operations specialist provides:

(* A h o o k t h a t d e f i n e s how to e x t e n d the e n v i r o n m e n t . *) m e t h o d p r i v a t e v i r t u a l e x t e n d : ’ env - > ’ bn1 - > ’ env * ’ bn2

This is a bare-bones API for describing binding constructs.

(33)

The operations

specialist

(34)

Implementing an operation

To implement one operation on terms, the specialist decides:

I

the types of names and environments,

I

how to extend the environment when entering the scope of a bound name,

I

what to do at a free name occurrence.

(35)

Example: converting raw terms to nominal terms

The specialist writes the module KitImport. (Also part of AlphaLib.)

t y p e env = A t o m . t S t r i n g M a p . t (* a map of s t r i n g s to a t o m s *) l e t e m p t y : env = S t r i n g M a p . e m p t y

e x c e p t i o n U n b o u n d of s t r i n g

c l a s s [ ’ s e l f ] map = o b j e c t ( _ : ’ s e l f )

(* At a binder , g e n e r a t e a f r e s h a t o m and e x t e n d the env . *) m e t h o d e x t e n d ( env : env ) ( x : s t r i n g ) : env * A t o m . t =

l e t a = A t o m . f r e s h x in

l e t env = S t r i n g M a p . add x a env in env , a

(* At a n a m e o c c u r r e n c e , l o o k up the e n v i r o n m e n t . *) m e t h o d visit_ ’ fn ( env : env ) ( x : s t r i n g ) : A t o m . t =

t r y S t r i n g M a p . f i n d x env

w i t h N o t _ f o u n d - > r a i s e ( U n b o u n d x ) e n d

(36)

Done? Almost.

(37)

Gluey business

The end user must work a little bit to glue everything together...

For each operation, the end user must write about 5 lines of glue code:

l e t i m p o r t _ t e r m ( t : r a w _ t e r m ) : n o m i n a l _ t e r m = (o b j e c t

i n h e r i t [ _ ] map (* g e n e r a t e d by v i s i t o r s *) i n h e r i t [ _ ] K i t I m p o r t . map (* p r o v i d e d by A l p h a L i b *) e n d) # v i s i t _ t e r m K i t I m p o r t . e m p t y t

As there are many operations, this is unpleasant.

Functors can help in simple cases, but are not flexible enough.

I use C-like macros, but this is ugly. Is there a better way?

(38)

Conclusion

(39)

Takeaway thoughts

Generated visitors allow a limited form of generic programming.

Visitor classes are partial, composable descriptions of operations.

Visitors can traverse abstract syntax with binding.

I

Syntax, binding forms, operations are described separately.

I

Syntax is described in a declarative style.

I

In the paper: towards a DSL for binding constructs.

(40)

Limitations

Not everything is perfect:

I

Three visitor classes are needed: map, iter, iter2.

I

The end user must use a C-like macro that I provide.

I

Some binding constructs cannot be implemented at all.

I

e.g., nonlinear patterns – (not representation-independent)

I

Some binding constructs are not easily supported in the high-level DSL.

I

e.g., Unbound’s

Rec

– (seems to require multiple subtraversals)

I

More practical experience is needed. (Guinea pigs Users welcome!)

(41)

Backup

(42)

Features of the visitors package

I

Several built-in varieties of visitors: iter, map, . . .

I

Arity two, too: iter2, map2, . . .

I

Generated visitor methods are monomorphic (in this talk),

I

and their types are inferred.

I

Visitor classes are nevertheless polymorphic.

I

Polymorphic visitor methods can be hand-written and inherited.

(43)

Support for parameterized data types

Visitors can traverse parameterized data types, too.

I

But: how does one traverse a subtree of type ’a?

Two approaches are supported:

I

declare a virtual visitor method visit_’a

I

pass a function visit_’a to every visitor method.

I

allows / requires methods to be polymorphic in ’a

I

more compositional

In this talk: a bit of both (details omitted...).

(44)

Predefined visitor methods

The class VisitorsRuntime.map offers this method:

c l a s s [ ’ s e l f ] map = o b j e c t ( s e l f ) (* One of m a n y p r e d e f i n e d m e t h o d s : *) m e t h o d p r i v a t e v i s i t _ l i s t : ’ env ’ a ’ b .

( ’ env - > ’ a - > ’ b ) - > ’ env - > ’ a l i s t - > ’ b l i s t

= f u n f env xs - >

m a t c h xs w i t h

| [] - >

[]

| x :: xs - >

l e t x = f env x in

x :: s e l f # v i s i t _ l i s t f env xs e n d

This method is polymorphic, so multiple instances of list are not a problem.

(45)

Visiting an abstraction

The class BindingForms.map offers the method visit_abs:

m e t h o d p r i v a t e v i s i t _ a b s : (* The method ’ s t y p e : *)

’ t e r m 1 ’ t e r m 2 . _ - >

( ’ env - > ’ t e r m 1 - > ’ t e r m 2 ) - >

’ env - > ( ’ bn1 , ’ t e r m 1 ) abs - > ( ’ bn2 , ’ t e r m 2 ) abs (* The method ’ s c o d e : *)

= f u n _ visit_ ’ t e r m env ( x1 , t1 ) - >

l e t env , x2 = s e l f # e x t e n d env x1 in l e t t2 = visit_ ’ t e r m env t1 in x2 , t2

(* A h o o k t h a t d e f i n e s how to e x t e n d the e n v i r o n m e n t . *) m e t h o d p r i v a t e v i r t u a l e x t e n d : ’ env - > ’ bn1 - > ’ env * ’ bn2 e n d

This method:

I

takes a visitor function for terms, an environment,

I

an abstraction, i.e., a pair of a name and a term, and

I

returns a pair of a transformed name and a transformed term.

(46)

Visiting an abstraction

The class BindingForms.map offers the method visit_abs:

’ t e r m 1 ’ t e r m 2 . _ - >

( ’ env - > ’ t e r m 1 - > ’ t e r m 2 ) - >

This method:

I

takes a visitor function for terms, an environment,

I

an abstraction, i.e., a pair of a name and a term, and

I

returns a pair of a transformed name and a transformed term.

(47)

Visiting an abstraction

The class BindingForms.map offers the method visit_abs:

’ t e r m 1 ’ t e r m 2 . _ - >

( ’ env - > ’ t e r m 1 - > ’ t e r m 2 ) - >

This method:

I

takes a visitor function for terms, an environment,

I

an abstraction, i.e., a pair of a name and a term, and

I

returns a pair of a transformed name and a transformed term.

(48)

Visiting an abstraction

The class BindingForms.map offers the method visit_abs:

’ t e r m 1 ’ t e r m 2 . _ - >

( ’ env - > ’ t e r m 1 - > ’ t e r m 2 ) - >

This method:

I

takes a visitor function for terms, an environment,

I

an abstraction, i.e., a pair of a name and a term, and

I

returns a pair of a transformed name and a transformed term.

(49)

Visiting an abstraction

The class BindingForms.map offers the method visit_abs:

’ t e r m 1 ’ t e r m 2 . _ - >

( ’ env - > ’ t e r m 1 - > ’ t e r m 2 ) - >

This method:

I

takes a visitor function for terms, an environment,

I

an abstraction, i.e., a pair of a name and a term, and

I

returns a pair of a transformed name and a transformed term.

(50)

Towards advanced

binding constructs

(51)

Defining new binding constructs

There are many binding constructs out there.

I

“let”, “let rec”, patterns, telescopes, ...

We have seen how to programmatically define a binding construct.

Can it be done in a more declarative manner?

(52)

A domain-specific language

Here is a little language of binding combinators:

t ::= . . .

sums, products, free occurrences of names, etc.

|

abstraction(p

)

a pattern, with embedded subterms

|

bind(p

,t)

— sugar for abstraction(p

×

inner(t))

p ::= . . .

sums, products, etc.

|

binder(x) a binding occurrence of a name

|

outer(t) an embedded term

|

rebind(p) a pattern in the scope of any bound names on the left

|

inner(t) — sugar for rebind(outer(t))

Inspired by C

α

ml (F.P., 2005) and Unbound (Weirich et al., 2011).

(53)

A domain-specific language

Here is a little language of binding combinators:

t ::= . . .

sums, products, free occurrences of names, etc.

|

abstraction(p

)

a pattern, with embedded subterms

|

bind(p

,t)

— sugar for abstraction(p

×

inner(t))

p ::= . . .

sums, products, etc.

|

binder(x) a binding occurrence of a name

|

outer(t) an embedded term

|

rebind(p) a pattern in the scope of any bound names on the left

|

inner(t) — sugar for rebind(outer(t))

Inspired by C

α

ml (F.P., 2005) and Unbound (Weirich et al., 2011).

(54)

A domain-specific language

Here is a little language of binding combinators:

t ::= . . .

sums, products, free occurrences of names, etc.

|

abstraction(p

)

a pattern, with embedded subterms

|

bind(p

,t)

— sugar for abstraction(p

×

inner(t))

p ::= . . .

sums, products, etc.

|

binder(x) a binding occurrence of a name

|

outer(t) an embedded term

|

rebind(p) a pattern in the scope of any bound names on the left

|

inner(t) — sugar for rebind(outer(t))

Inspired by C

α

ml (F.P., 2005) and Unbound (Weirich et al., 2011).

(55)

Example use: telescopes

A dependently-typed

λ

-calculus whose

Π

and

λ

forms involve a telescope:

# d e f i n e t e l e ( ’ bn , ’ fn ) t e l e

# d e f i n e t e r m ( ’ bn , ’ fn ) t e r m

(* The t y p e s t h a t f o l l o w are p a r a m e t r i c in ’ bn and ’ fn : *) t y p e t e l e =

| T e l e N i l

| T e l e C o n s of ’ bn b i n d e r * t e r m o u t e r * t e l e r e b i n d a n d t e r m =

| T V a r of ’ fn

| TPi of ( tele , t e r m ) b i n d

| T L a m of ( tele , t e r m ) b i n d

| T A p p of t e r m * t e r m l i s t [ @ @ d e r i v i n g v i s i t o r s {

v a r i e t y = " map " ;

a n c e s t o r s = [ " B i n d i n g C o m b i n a t o r s . map " ] }]

(56)

Implementation

These primitive constructs are just annotations:

t y p e ’ p a b s t r a c t i o n = ’ p t y p e ’ bn b i n d e r = ’ bn t y p e ’ t o u t e r = ’ t t y p e ’ p r e b i n d = ’ p

Their presence triggers calls to appropriate (hand-written) visit_ methods.

(57)

Implementation

While visiting a pattern, we keep track of:

I

the outer environment, which existed outside this pattern;

I

the current environment, extended with the bound names encountered so far.

Thus, while visiting a pattern, we use a richer type of contexts:

t y p e ’ env c o n t e x t = { o u t e r : ’ env ; c u r r e n t : ’ env ref }

— Not every visitor method need have the same type of environments!

With this in mind, the implementation of the visit_ methods is straightforward...

(58)

Implementation

This code takes place in a map visitor:

c l a s s v i r t u a l [ ’ s e l f ] map = o b j e c t ( s e l f : ’ s e l f )

m e t h o d p r i v a t e v i r t u a l e x t e n d : ’ env - > ’ bn1 - > ’ env * ’ bn2 (* The f o u r v i s i t o r m e t h o d s are i n s e r t e d h e r e ... *)

e n d

1. At the root of an abstraction, a fresh context is allocated:

m e t h o d p r i v a t e v i s i t _ a b s t r a c t i o n : ’ env ’ p1 ’ p2 . ( ’ env c o n t e x t - > ’ p1 - > ’ p2 ) - >

’ env - > ’ p1 a b s t r a c t i o n - > ’ p2 a b s t r a c t i o n

= f u n v i s i t _ p env p1 - >

v i s i t _ p { o u t e r = env ; c u r r e n t = ref env } p1

(59)

Implementation

2. When a bound name is met, the current environment is extended:

m e t h o d p r i v a t e v i s i t _ b i n d e r : _ - >

’ env c o n t e x t - > ’ bn1 b i n d e r - > ’ bn2 b i n d e r

= f u n visit_ ’ bn ctx x1 - >

l e t env = !( ctx . c u r r e n t ) in

l e t env , x2 = s e l f # e x t e n d env x1 in ctx . c u r r e n t := env ;

x2

(60)

Implementation

3. When a term that is not in the scope of the abstraction is found, it is visited in the outer environment.

m e t h o d p r i v a t e v i s i t _ o u t e r : ’ env ’ t1 ’ t2 . ( ’ env - > ’ t1 - > ’ t2 ) - >

’ env c o n t e x t - > ’ t1 o u t e r - > ’ t2 o u t e r

= f u n v i s i t _ t ctx t1 - >

v i s i t _ t ctx . o u t e r t1

(61)

Implementation

4. When a subpattern marked rebind is found,

the current environment is installed as the outer environment.

m e t h o d p r i v a t e v i s i t _ r e b i n d : ’ env ’ p1 ’ p2 . ( ’ env c o n t e x t - > ’ p1 - > ’ p2 ) - >

’ env c o n t e x t - > ’ p1 r e b i n d - > ’ p2 r e b i n d

= f u n v i s i t _ p ctx p1 - >

v i s i t _ p { ctx w i t h o u t e r = !( ctx . c u r r e n t ) } p1