Visitors Unchained Using

(1)

Visitors Unchained

Usingvisitors

to traverseabstract syntax with binding

François Pottier

WG 2.8, Edinburgh June 15, 2017

(2)

The problem

Manipulating abstract syntax with binding requires many standard operations:

I “opening”and“closing”binders (in the locally nameless representation);

I shifting(in de Bruijn’s representation);

I decidingα-equivalence;

I testing whether a nameoccursfree in a term;

I performing capture-avoidingsubstitution;

I convertinguser-supplied terms to the desired internal representation;

I etc., etc.

This requires a lot ofboilerplate, a.k.a.nameplate(Cheney).

(3)

Isn’t this a solved problem?

It may well be, depending on your programming language of choice.

Haskellhas

I FreshLib (Cheney, 2005),

I Unbound (Weirich, Yorgey, Sheard, 2011),

I Bound (Kmett, 2013?),

I maybe more?

These libraries may have bad performance, though (?).

OCamlhas... not much.

I Cαml (F.P., 2005) is monolithic, inflexible, and has performance issues, too.

(The problem also arises in theorem provers. Not studied here.)

(4)

Goal

I wish toscrap my nameplate, inOCaml, in a manner that is

I asmodular, open-ended,customizableas possible,

I while relying onas little code generationas possible,

I while (simultaneously!) supportingmultiple representationsof names,

I and supportingmultiple binding constructs, possibly user-defined.

It turns out that this can be done by exploiting the“visitor”design pattern.

(5)

Visitors

(6)

Installation & configuration

Installation:

o p a m u p d a t e

o p a m i n s t a l l v i s i t o r s

To configure ocamlbuild, add this in_tags:

t r u e : \

p a c k a g e ( v i s i t o r s . ppx ) , \ p a c k a g e ( v i s i t o r s . r u n t i m e ) To configure Merlin, add this in.merlin:

PKG v i s i t o r s . ppx PKG v i s i t o r s . r u n t i m e

(7)

An “iter” visitor

Annotating a type definition with[@@deriving visitors { ... }]...

t y p e e x p r =

| E C o n s t of int

| E A d d of e x p r * e x p r

[ @ @ d e r i v i n g v i s i t o r s { v a r i e t y = " i t e r " }]

c l a s s v i r t u a l [ ’ s e l f ] i t e r = o b j e c t ( s e l f : ’ s e l f ) i n h e r i t [ _ ] V i s i t o r s R u n t i m e . i t e r

m e t h o d v i s i t _ E C o n s t env c0 = l e t r0 = s e l f # v i s i t _ i n t env c0 in ()

m e t h o d v i s i t _ E A d d env c0 c1 = l e t r0 = s e l f # v i s i t _ e x p r env c0 in l e t r1 = s e l f # v i s i t _ e x p r env c1 in ()

m e t h o d v i s i t _ e x p r env t h i s = m a t c h t h i s w i t h

| E C o n s t c0 - >

s e l f # v i s i t _ E C o n s t env c0

| E A d d ( c0 , c1 ) - >

s e l f # v i s i t _ E A d d env c0 c1 e n d

... causes avisitor classto be auto-generated.

(8)

An “iter” visitor

t y p e e x p r =

| E C o n s t c0 - >

| E A d d ( c0 , c1 ) - >

(9)

An “iter” visitor

t y p e e x p r =

| E C o n s t c0 - >

| E A d d ( c0 , c1 ) - >

one method per data constructor

(10)

An “iter” visitor

t y p e e x p r =

| E C o n s t c0 - >

| E A d d ( c0 , c1 ) - >

one method per data type

(11)

An “iter” visitor

t y p e e x p r =

| E C o n s t c0 - >

| E A d d ( c0 , c1 ) - >

an environment is pushed down

(12)

An “iter” visitor

t y p e e x p r =

| E C o n s t c0 - >

| E A d d ( c0 , c1 ) - >

default behavior is to do nothing

(13)

An “iter” visitor

t y p e e x p r =

| E C o n s t c0 - >

| E A d d ( c0 , c1 ) - >

behavior at type intis inherited

(14)

A “map” visitor

There are several varieties of visitors:

t y p e e x p r =

[ @ @ d e r i v i n g v i s i t o r s { v a r i e t y = " map " }]

c l a s s v i r t u a l [ ’ s e l f ] map = o b j e c t ( s e l f : ’ s e l f ) i n h e r i t [ _ ] V i s i t o r s R u n t i m e . map

m e t h o d v i s i t _ E C o n s t env c0 = l e t r0 = s e l f # v i s i t _ i n t env c0 in E C o n s t r0

m e t h o d v i s i t _ E A d d env c0 c1 = l e t r0 = s e l f # v i s i t _ e x p r env c0 in l e t r1 = s e l f # v i s i t _ e x p r env c1 in E A d d ( r0 , r1 )

| E C o n s t c0 - >

| E A d d ( c0 , c1 ) - >

default behavior is to rebuild a tree

a “map” visitor is requested

(15)

Using a “map” visitor

Inherita visitor class andoverrideone or more methods:

l e t add e1 e2 = (* A s m a r t c o n s t r u c t o r . *) m a t c h e1 , e2 w i t h

| E C o n s t 0 , e

| e , E C o n s t 0 - > e

| _ , _ - > E A d d ( e1 , e2 ) l e t o p t i m i z e : e x p r - > e x p r =

l e t v = o b j e c t ( s e l f ) i n h e r i t [ _ ] map

m e t h o d! v i s i t _ E A d d env e1 e2 = add

( s e l f # v i s i t _ e x p r env e1 ) ( s e l f # v i s i t _ e x p r env e2 ) e n d in

v # v i s i t _ e x p r ()

This addition-optimization pass isunchangedif more expression forms are added.

(16)

What we have seen so far

I Several built-in varieties:iter,map, . . .

I Arity two, too:iter2,map2, . . .

I Generated visitor methods aremonomorphic(in this talk),

I and their types areinferred.

I Visitor classes are neverthelesspolymorphic.

I Polymorphicvisitor methods can be hand-written and inherited.

(17)

Support for parameterized data types

Visitors can traverse parameterized data types, too.

I But: how does one traverse a subtree of type’a?

Two approaches are supported:

I declare avirtual visitor methodvisit_’a

I pass afunctionvisit_’ato every visitor method.

I allows / requires methods to be polymorphic in’a

I more compositional

In this talk: a bit of both (details omitted...).

(18)

Visiting preexisting types

Lists can be visited, too.

t y p e e x p r =

| E A d d of e x p r l i s t

m e t h o d v i s i t _ E A d d env c0 =

l e t r0 = s e l f # v i s i t _ l i s t s e l f # v i s i t _ e x p r env c0 in ()

| E C o n s t c0 - >

| E A d d c0 - >

s e l f # v i s i t _ E A d d env c0 e n d

a preexisting parameterized type

(19)

Visiting preexisting types

t y p e e x p r =

| E C o n s t c0 - >

| E A d d c0 - >

visitor method is passed a visitor function

(20)

Visiting preexisting types

t y p e e x p r =

| E C o n s t c0 - >

| E A d d c0 - >

visitor method is inherited

(21)

Predefined visitor methods

The classVisitorsRuntime.mapoffers this method:

c l a s s [ ’ s e l f ] map = o b j e c t ( s e l f ) (* One of m a n y p r e d e f i n e d m e t h o d s : *) m e t h o d p r i v a t e v i s i t _ l i s t : ’ env ’ a ’ b .

( ’ env - > ’ a - > ’ b ) - > ’ env - > ’ a l i s t - > ’ b l i s t

= f u n f env xs - >

m a t c h xs w i t h

| [] - >

[]

| x :: xs - >

l e t x = f env x in

x :: s e l f # v i s i t _ l i s t f env xs e n d

This method ispolymorphic, so multiple instances oflistare not a problem.

(22)

Visitors – a summary

Although they follow fixed patterns, visitors are quiteversatile.

They are like higher-order functions, only morecustomizableandcomposable.

More fun with visitors:

I visitors foropen data typesand their fixed points(link);

I visitors forhash-consed data structures(link);

I iteratorsout of visitors(link).

In the remainder of this talk:

I Can we traverseabstract syntax with binding?

(23)

Visitors Unchained

(24)

Dealing with binding

Can a visitor traverseabstract syntax with bindingconstructs?

Can this be done in amodularway?

Exactly whichseparation of concernsshould one enforce?

I There are manybinding constructs,

I there are evencombinator languagesfor describing binding structure!

I and many commonoperationson terms,

I often specific of onerepresentationof names and binders,

I sometimes specific oftwosuch representations, e.g.,conversions.

I Can we insulate theend userfrom this complexity? We suggest distinguishingthreeprincipals...

(25)

Dealing with binding

(26)

Dealing with binding

(27)

Dealing with binding

I Can we insulate theend userfrom this complexity?

We suggest distinguishingthreeprincipals...

(28)

(29)

end user

(30)

end user

operations specialist

(31)

end user

operations specialist

binder maestro

(32)

The end user

(33)

Desiderata

The end user wishes:

I to describe the structure of ASTs in a concise anddeclarativestyle,

I not to be bothered with implementation details,

I possibly to have access toseveral representationsof names,

I to get access to a toolkit ofready-made operationson terms.

(34)

Example: abstract syntax of the λ -calculus

Let the type(’bn, ’term) absbe a synonym for’bn * ’term.

The end user defines his syntax as follows:

t y p e ( ’ bn , ’ fn ) t e r m =

| T V a r of ’ fn

| T L a m b d a of ( ’ bn , ( ’ bn , ’ fn ) t e r m ) abs

| T A p p of ( ’ bn , ’ fn ) t e r m * ( ’ bn , ’ fn ) t e r m [ @ @ d e r i v i n g v i s i t o r s { v a r i e t y = " map " ;

a n c e s t o r s = [ " B i n d i n g F o r m s . map " ] }]

t y p e r a w _ t e r m = ( string , s t r i n g ) t e r m t y p e n o m i n a l _ t e r m = ( A t o m . t , A t o m . t ) t e r m t y p e d e b r u i j n _ t e r m = ( unit , int ) t e r m He getsmultiple representationsof names.

I At least two are used in any single application. (Parsing. Printing.) He getsvisitorsfor free. The methodvisit_absis used at abstractions.

I iter,map,iter2needed in practice. Focusing onmapin this talk.

(35)

Example: abstract syntax of the λ -calculus

Let the type(’bn, ’term) absbe a synonym for’bn * ’term.

The end user defines his syntax as follows:

t y p e ( ’ bn , ’ fn ) t e r m =

| T V a r of ’ fn

| T L a m b d a of ( ’ bn , ( ’ bn , ’ fn ) t e r m ) abs

| T A p p of ( ’ bn , ’ fn ) t e r m * ( ’ bn , ’ fn ) t e r m [ @ @ d e r i v i n g v i s i t o r s { v a r i e t y = " map " ;

a n c e s t o r s = [ " B i n d i n g F o r m s . map " ] }]

t y p e r a w _ t e r m = ( string , s t r i n g ) t e r m t y p e n o m i n a l _ t e r m = ( A t o m . t , A t o m . t ) t e r m t y p e d e b r u i j n _ t e r m = ( unit , int ) t e r m He getsmultiple representationsof names.

I At least two are used in any single application. (Parsing. Printing.) He getsvisitorsfor free. The methodvisit_absis used at abstractions.

I iter,map,iter2needed in practice. Focusing onmapin this talk.

provided by the binder maestro

(36)

The binder

maestro

(37)

An easy job?

Implementingvisit_absis the task of our sophisticated binder maestro.

The key is toextend the environmentwhen entering the scope of a binder.

Easy?

Maybe — yet, the binder maestro:

I does not knowwhat operationis being performed,

I does not knowwhat representation(s)of names are in use,

I therefore does not know the types of names and environments,

I let alonehowto extend the environment.

What he knows iswhereandwith what namesto extend the environment.

(38)

An easy job?

Implementingvisit_absis the task of our sophisticated binder maestro.

The key is toextend the environmentwhen entering the scope of a binder.

Easy? Maybe — yet, the binder maestro:

I does not knowwhat operationis being performed,

I does not knowwhat representation(s)of names are in use,

I therefore does not know the types of names and environments,

I let alonehowto extend the environment.

What he knows iswhereandwith what namesto extend the environment.

(39)

A convention

The binder maestro agrees on adealwith the operations specialist.

“I tell you when to extend the environment; you do the dirty work.”

The binder maestrocallsa method which the operations specialistprovides:

(* A h o o k t h a t d e f i n e s how to e x t e n d the e n v i r o n m e n t . *) m e t h o d p r i v a t e v i r t u a l e x t e n d : ’ env - > ’ bn1 - > ’ env * ’ bn2 This is a bare-bonesAPIfor describing binding constructs.

(40)

Visiting an abstraction

The classBindingForms.mapoffers the methodvisit_abs:

c l a s s v i r t u a l [ ’ s e l f ] map = o b j e c t ( s e l f : ’ s e l f ) (* A v i s i t o r m e t h o d for the t y p e abs . *)

m e t h o d p r i v a t e v i s i t _ a b s : ’ t e r m 1 ’ t e r m 2 . _ - >

( ’ env - > ’ t e r m 1 - > ’ t e r m 2 ) - >

’ env - > ( ’ bn1 , ’ t e r m 1 ) abs - > ( ’ bn2 , ’ t e r m 2 ) abs

= f u n _ visit_ ’ t e r m env ( x1 , t1 ) - >

l e t env , x2 = s e l f # e x t e n d env x1 in l e t t2 = visit_ ’ t e r m env t1 in x2 , t2

(* A h o o k t h a t d e f i n e s how to e x t e n d the e n v i r o n m e n t . *) m e t h o d p r i v a t e v i r t u a l e x t e n d : ’ env - > ’ bn1 - > ’ env * ’ bn2 e n d

This method:

I takes avisitor functionfor terms, anenvironment,

I an abstraction, i.e., apairof a name and a term, and

I returns a pair of atransformed nameand atransformed term.

(41)

Visiting an abstraction

( ’ env - > ’ t e r m 1 - > ’ t e r m 2 ) - >

This method:

(42)

Visiting an abstraction

( ’ env - > ’ t e r m 1 - > ’ t e r m 2 ) - >

This method:

(43)

Visiting an abstraction

( ’ env - > ’ t e r m 1 - > ’ t e r m 2 ) - >

This method:

(44)

Visiting an abstraction

( ’ env - > ’ t e r m 1 - > ’ t e r m 2 ) - >

This method:

(45)

Visiting an abstraction

That’sall there isto single-name abstractions.

More binding constructs later on...

For now, let’s turn to the final participant.

(46)

The operations

specialist

(47)

A toolbox of operations

There are many operations on terms that the end user might wish for:

I testing terms forequalityup toα-equivalence,

I finding out which names arefreeorboundin a term,

I applying arenamingor asubstitutionto a term,

I convertinga term from one representation to another,

I (plus application-specific operations.)

(48)

Implementing an operation

To implement one operation, the specialist decides:

I thetypesof names and environments,

I what to doat afree nameoccurrence,

I how to extendthe environment when entering the scope of abound name.

(49)

Implementing import

As an example, let’s implementimport, which converts raw terms to nominal terms.

1. An import environment maps strings to atoms:

m o d u l e S t r i n g M a p = Map . M a k e ( S t r i n g ) t y p e env = A t o m . t S t r i n g M a p . t l e t e m p t y : env = S t r i n g M a p . e m p t y

(50)

Implementing import

2. When the scope ofxis entered,

the environment is extended with a mapping of the stringxto a fresh atoma.

l e t e x t e n d ( env : env ) ( x : s t r i n g ) : env * A t o m . t = l e t a = A t o m . f r e s h x in

l e t env = S t r i n g M a p . add x a env in env , a

(Anatomcarries a unique integer identity.)

This is true regardless of which binding constructs are traversed.

(51)

Implementing import

3. When an occurrence of the stringxis found,

the environment is looked up so as to find the corresponding atom.

e x c e p t i o n U n b o u n d of s t r i n g

l e t l o o k u p ( env : env ) ( x : s t r i n g ) : A t o m . t = t r y S t r i n g M a p . f i n d x env

w i t h N o t _ f o u n d - > r a i s e ( U n b o u n d x )

(52)

Implementing import

The previous instructions are grouped in a little class — a “kit”:

c l a s s [ ’ s e l f ] map = o b j e c t ( _ : ’ s e l f ) m e t h o d p r i v a t e e x t e n d = e x t e n d m e t h o d p r i v a t e visit_ ’ fn = l o o k u p e n d

This isKitImport.map.

That’s all there is to it...but...

(53)

Gluey business

The end user mustworka little bit toglueeverything together...

For each operation, the end user must write 5 lines of glue code:

l e t i m p o r t _ t e r m env t = (o b j e c t

i n h e r i t [ _ ] map (* g e n e r a t e d by v i s i t o r s *) i n h e r i t [ _ ] K i t I m p o r t . map (* p r o v i d e d by A l p h a L i b *) e n d) # v i s i t _ t e r m env t

For 15 operations, this hurts.

Functorscan help in simple cases, but are not flexible enough.

C-likemacroshelp, but are ugly. Is there a better way?

(54)

Towards advanced

binding constructs

(55)

Defining new binding constructs

There aremany binding constructsout there.

I “let”, “let rec”, patterns, telescopes, ...

We have seen how toprogrammaticallydefine a binding construct.

Can it be done in a moredeclarativemanner?

(56)

A domain-specific language

Here is a little language ofbinding combinators:

t ::= . . . sums, products, free occurrences of names, etc.

| abstraction(p) a pattern, with embedded subterms

| bind(p,t) — sugar for abstraction(p×inner(t))

p ::= . . . sums, products, etc.

| binder(x) a binding occurrence of a name

| outer(t) an embedded term

| rebind(p) a pattern in the scope of any bound names on the left

| inner(t) — sugar for rebind(outer(t))

Inspired by Cαml (F.P., 2005) and Unbound (Weirich et al., 2011).

(57)

A domain-specific language

(58)

A domain-specific language

(59)

Example use: telescopes

A dependently-typedλ-calculus whoseΠandλforms involve a telescope:

# d e f i n e t e l e ( ’ bn , ’ fn ) t e l e

# d e f i n e t e r m ( ’ bn , ’ fn ) t e r m

(* The t y p e s t h a t f o l l o w are p a r a m e t r i c in ’ bn and ’ fn : *) t y p e t e l e =

| T e l e N i l

| T e l e C o n s of ’ bn b i n d e r * t e r m o u t e r * t e l e r e b i n d a n d t e r m =

| T V a r of ’ fn

| TPi of ( tele , t e r m ) b i n d

| T L a m of ( tele , t e r m ) b i n d

| T A p p of t e r m * t e r m l i s t [ @ @ d e r i v i n g v i s i t o r s {

v a r i e t y = " map " ;

a n c e s t o r s = [ " B i n d i n g C o m b i n a t o r s . map " ] }]

(60)

Implementation

These primitive constructs are just annotations:

t y p e ’ p a b s t r a c t i o n = ’ p t y p e ’ bn b i n d e r = ’ bn t y p e ’ t o u t e r = ’ t t y p e ’ p r e b i n d = ’ p

Their presence triggers calls to appropriate (hand-written)visit_methods.

(61)

Implementation

While visiting a pattern, we keep track of:

I theouter environment, which existed outside this pattern;

I thecurrent environment, extended with the bound names encountered so far.

Thus, while visiting a pattern, we use a richer type ofcontexts:

t y p e ’ env c o n t e x t = { o u t e r : ’ env ; c u r r e n t : ’ env ref }

— Not every visitor method need have the same type of environments!

With this in mind, the implementation of thevisit_methods is straightforward...

(62)

Implementation

This code takes place in amapvisitor:

c l a s s v i r t u a l [ ’ s e l f ] map = o b j e c t ( s e l f : ’ s e l f )

m e t h o d p r i v a t e v i r t u a l e x t e n d : ’ env - > ’ bn1 - > ’ env * ’ bn2 (* The f o u r v i s i t o r m e t h o d s are i n s e r t e d h e r e ... *)

e n d

1. At the root of an abstraction,a fresh contextis allocated:

m e t h o d p r i v a t e v i s i t _ a b s t r a c t i o n : ’ env ’ p1 ’ p2 . ( ’ env c o n t e x t - > ’ p1 - > ’ p2 ) - >

’ env - > ’ p1 a b s t r a c t i o n - > ’ p2 a b s t r a c t i o n

= f u n v i s i t _ p env p1 - >

v i s i t _ p { o u t e r = env ; c u r r e n t = ref env } p1

(63)

Implementation

2. When a bound name is met, thecurrentenvironment isextended:

m e t h o d p r i v a t e v i s i t _ b i n d e r : _ - >

’ env c o n t e x t - > ’ bn1 b i n d e r - > ’ bn2 b i n d e r

= f u n visit_ ’ bn ctx x1 - >

l e t env = !( ctx . c u r r e n t ) in

l e t env , x2 = s e l f # e x t e n d env x1 in ctx . c u r r e n t := env ;

x2

(64)

Implementation

3. When a term that isnot in the scopeof the abstraction is found, it is visited in theouterenvironment.

m e t h o d p r i v a t e v i s i t _ o u t e r : ’ env ’ t1 ’ t2 . ( ’ env - > ’ t1 - > ’ t2 ) - >

’ env c o n t e x t - > ’ t1 o u t e r - > ’ t2 o u t e r

= f u n v i s i t _ t ctx t1 - >

v i s i t _ t ctx . o u t e r t1

(65)

Implementation

4. When a subpattern markedrebindis found,

thecurrentenvironment is installed as theouterenvironment.

m e t h o d p r i v a t e v i s i t _ r e b i n d : ’ env ’ p1 ’ p2 . ( ’ env c o n t e x t - > ’ p1 - > ’ p2 ) - >

’ env c o n t e x t - > ’ p1 r e b i n d - > ’ p2 r e b i n d

= f u n v i s i t _ p ctx p1 - >

v i s i t _ p { ctx w i t h o u t e r = !( ctx . c u r r e n t ) } p1 This affects the meaning ofouterinsiderebind.

(66)

Conclusion

(67)

Conclusion

Visitors arepowerful.

Visitor classes arepartial,composabledescriptions of operations.

Visitorscantraverseabstract syntax with binding.

I Syntax, binding forms, operations can beseparatelydescribed.

I Syntax and even binding forms can be described in adeclarativestyle.

I Open-ended, customizable approach.

Limitations:

I C-like macros are currently required.

I No proofs.

I Some operations may not fit the visitor framework;

I Some binding forms do not easily fit in the low-level framework or in the higher-level DSL, e.g., Unbound’sRec.