mm WM

mm WM

ORGANISATION M O N D I A L E D E LA SANTÉ

E X E C U T I V E B O A R D

彳

义

S i x t y - f i r s t S e s s i o n ^^HtCu-ff ^ '，.'、^ ‘ P r o v i s i o n a l a g e n d a i t e m 2 0 fj “ i ^

EB61/20

18 N o v e m b e r 1 9 7 7

INDEXED-

D E V E L O P M E N T O F H E A L T H P R O G R A M M E E V A L U A T I O N

R e p o r t of t h e P r o g r a m m e C o m m i t t e e of the E x e c u t i v e B o a r d

The P r o g r a m m e C o m m i t t e e w e l c o m e d the D i r e c t o r - G e n e r a l ' s p r o g r e s s r e p o r t o n s p r o g r e s s r e p o r t on the D e v e l o p m e n t c / w p / 6 (attached in A n n e x )， a n d of H e a l t h P r o g r a m m e E v a l u a t i o n c o n t a i n e d in d o c u m e n t EB61,

c o n s i d e r e d the d r a f t g u i d e l i n e s that w e r e a t t a c h e d to i t to b e t i m e l y a n d a d e q u a t e f o r test a p p l i c a t i o n .

P r o v i s i o n a l g u i d e l i n e s

2 . I t h a d b e e n r e c a l l e d that the p r i n c i p l e s for h e a l t h p r o g r a m m e e v a l u a t i o n h a d b e e n

a p p r o v e d b y the E x e c u t i v e B o a r d a t i t s f i f t y - s e v e n t h s e s s i o n i n J a n u a r y 1 9 7 6 . T h e p r i n c i p l e s had n o w b e e n c o n v e r t e d i n t o g u i d e l i n e s f o r a p r a c t i c a l e v a l u a t i o n p r o c e s s b y a t e a m in c l o s e c o l l a b o r a t i o n w i t h t h e r e g i o n s a n d r e l y i n g h e a v i l y o n n a t i o n a l a n d r e g i o n a l e x p e r i e n c e . 3 . T w o f e a t u r e s of the g u i d e l i n e s , i n p a r t i c u l a r , w e r e h i g h l i g h t e d . The f i r s t w a s that e v a l u a t i o n h a d to be a n i n t e g r a l p a r t of the h e a l t h d e v e l o p m e n t p r o c e s s , a p p l i e d iri

c o n j u n c t i o n w i t h the o t h e r parts； i t s h o u l d n o t b e a p p l i e d s e p a r a t e l y since e x p e r i e n c e h a d s h o w n that a t t e m p t s to e v a l u a t e in i s o l a t i o n h a d u s u a l l y f a i l e d to lead to i m p r o v e m e n t s in the p l a n n i n g a n d i m p l e m e n t a t i o n o f p r o g r a m m e s a n d i n t h e d e c i s i o n - m a k i n g p r o c e s s . T h e s e c o n d feature that w a s s t r e s s e d w a s the i m p o r t a n c e of e v a l u a t i o n of p r o g r a m m e r e l e v a n c e as w e l l a s the e f f e c t that a p r o g r a m m e h a d i n i m p r o v i n g h e a l t h s i t u a t i o n s . These a s p e c t s w e r e m o r e i m p o r t a n t than a s s e s s i n g the e f f i c i e n c y of i m p l e m e n t a t i o n a l o n e .

4 . The C o m m i t t e e felt that the d r a f t e v a l u a t i o n g u i d e l i n e s w o u l d be r e l a t i v e l y e a s y to a p p l y in v i e w of their s e n s i b l e , d o w n - t o - e a r t h a p p r o a c h a n d in v i e w o f the clear a n d c o n c i s e m a n n e r in w h i c h they h a d b e e n f o r m u l a t e d . The c l e a r d e s c r i p t i o n of such terms a s " e f f i c i e n c y " a n d

" e f f e c t i v e n e s s " , w h i c h h a d h i t h e r t o f r e q u e n t l y led to c o n f u s i o n , w o u l d f a c i l i t a t e u n d e r s t a n d i n g a n d the a p p l i c a t i o n of the e v a l u a t i o n p r o c e s s .

5 . The C o m m i t t e e , w h i l e a c c e p t i n g the p r o v i s i o n a l g u i d e l i n e s , s t r e s s e d t h a t t h e y h a d to b e u s e d in a f l e x i b l e m a n n e r a n d a p p l i e d s e l e c t i v e l y . T h e g u i d i n g p r i n c i p l e s f o r this s e l e c t i v i t y s h o u l d be the r e l e v a n c e a n d cost of a p p l y i n g the t o t a l p r o c e s s . F o r e x a m p l e , it m i g h t be

u s e f u l to a s s e s s the r e l e v a n c e o f some a c t i v i t i e s , s u c h as the p r o v i s i o n of c e r t a i n s u p p l i e s b y W H O to n a t i o n a l p r o g r a m m e s

； i t w o u l d , h o w e v e r , be i m p r a c t i c a l a n d t o o c o s t l y t o a s s e s s their

o v e r a l l i m p a c t . O n e v i t a l p r e r e q u i s i t e for the i m p l e m e n t a t i o n of the e v a l u a t i o n p r o c e s s , w h i c h i m p l i e d n o s e l e c t i v i t y

， w a s a p o s i t i v e state o f m i n d w i t h r e g a r d to e v a l u a t i o n a n d a

E v a l u a t i o n as p a r t o f the o v e r a l l h e a l t h d e v e l o p m e n t p r o c e s s

6 . A s an i n t e g r a l p a r t o f t h e h e a l t h d e v e l o p m e n t p r o c e s s e v a l u a t i o n s h o u l d h a v e s t r o n g links w i t h a l l o t h e r e l e m e n t s of t h a t p r o c e s s , such a s c o u n t r y h e a l t h p r o g r a m m i n g , m e d i u m - t e r m p r o g r a m m i n g , p r o g r a m m e b u d g e t i n g a n d i m p l e m e n t a t i o n , a n d the s u p p o r t i v e i n f o r m a t i o n s y s t e m . O n l y t h r o u g h the t o t a l i n t e g r a t i o n of the e v a l u a t i o n p r o c e s s i n t o t h e o v e r a l l h e a l t h m a n a g e m e n t , c o u l d m e a n i n g f u l e v a l u a t i o n take p l a c e . The u s e of e v a l u a t i o n in such a n i n t e g r a t e d m a n n e r c o u l d , in f a c t , b e u s e f u l to i d e n t i f y a n d i m p l e m e n t r e l e v a n t t e c h n i c a l c o o p e r a t i o n p r o g r a m m e s w h i c h w o u l d h a v e an i m p a c t a t c o m m u n i t y l e v e l .

EBÓl/20 page 2

I n v o l v e m e n t a n d r e s p o n s i b i l i t y

7 . The C o m m i t t e e l a i d e m p h a s i s on the f a c t t h a t e v a l u a t i o n s h o u l d b e a c o n t i n u o u s a n d d y n a m i c p r o c e s s b a s e d on b r o a d p a r t i c i p a t i o n of a l l t h o s e i n v o l v e d : a d m i n i s t r a t i v e , s c i e n t i f i c a n d t e c h n i c a l s t a f f , a t e a c h m a n a g e r i a l l e v e l in c o u n t r i e s a n d in W H O . The g u i d e l i n e s h a d

i n d i c a t e d the m a n a g e r i a l r e s p o n s i b i l i t i e s f o r e v a l u a t i o n at e a c h o p e r a t i o n a l l e v e l in c o u n t r i e s a n d in W H O . T h e s e r e s p o n s i b i l i t i e s s h o u l d lie w i t h the same p e o p l e w h o w e r e r e s p o n s i b l e for the o t h e r p a r t s of the h e a l t h d e v e l o p m e n t p r o c e s s . The g u i d e l i n e s h a d n o t a d e q u a t e l y c o n v e y e d the i m p o r t a n c e of the t e a m a p p r o a c h to e v a l u a t i o n i n the c o n t e x t of t h o s e r e s p o n s i b i l i t i e s . E v a l u a t i o n r e s u l t s , t r a d i t i o n a l l y a v a i l a b l e o n l y to h e a l t h a d m i n i s t r a t o r s a n d g o v e r n m e n t a l b o d i e s , s h o u l d be g i v e n w i d e d i s t r i b u t i o n so t h a t t h e y c o u l d b e u t i l i z e d b y the c o m m u n i t i e s or t a r g e t a u d i e n c e c o n c e r n e d , as w e l l a s b y h e a l t h w o r k e r s i n v o l v e d in the p r o g r a m m e d e l i v e r y . L i k e w i s e , j u s t as the s c i e n t i f i c m e t h o d s h o u l d be u s e d in the e v a l u a t i o n p r o c e s s , the r e s u l t s o f e v a l u a t i o n s h o u l d b e fed b a c k i n t o the s c i e n t i f i c p r o c e s s , a n d s h o u l d l e a d to the i d e n t i f i - c a t i o n of a r e a s for b i o m e d i c a l a n d h e a l t h s e r v i c e s r e s e a r c h .

C r i t e r i a a n d i n d i c a t o r s

8 . I n r e v i e w i n g the q u e s t i o n of c r i t e r i a a n d i n d i c a t o r s the C o m m i t t e e a g r e e d t h a t w h e r e v e r p o s s i b l e , t h e y s h o u l d b e q u a n t i f i a b l e . I t w a s , h o w e v e r , r e c o g n i z e d t h a t q u a l i t a t i v e c r i t e r i a a n d i n d i c a t o r s w e r e o f t e n v e r y u s e f u l for e v a l u a t i o n w h e n e v e r q u a n t i f i e d c r i t e r i a or i n d i c a t o r s w e r e n o t e a s i l y a v a i l a b l e . W h i l e the d r a f t g u i d e l i n e s p r o v i d e d g e n e r a l t y p e s of c r i t e r i a a n d i n d i c a t o r s , the a p p l i c a t i o n of the g u i d e l i n e s w o u l d h o p e f u l l y l e a d t o the g e n e r a t i o n of m o r e s p e c i f i c c r i t e r i a a n d i n d i c a t o r s for s p e c i f i c p r o g r a m m e s . G r e a t e r e m p h a s i s s h o u l d b e l a i d o n p o s i t i v e h e a l t h i n d i c a t o r s t h a n o n m o r t a l i t y a n d m o r b i d i t y i n d i c a t o r s . It s h o u l d n o t b e f o r g o t t e n t h a t c r i t e r i a a n d i n d i c a t o r s s h o u l d be c a p a b l e of p r o v i d i n g r e l e v a n t i n f o r m a t i o n for the d i f f e r e n t a s p e c t s of d e c i s i o n - m a k i n g i n r e g a r d to h e a l t h d e v e l o p m e n t . F o r e x a m p l e , i n d i c a t o r s r e l a t i n g to e c o n o m i c as w e l l as s o c i a l a s p e c t s h a d to b e t a k e n i n t o a c c o u n t . In a d d i t i o n , i n d i c a t o r s s h o u l d b e s u f f i c i e n t l y e a s y to a p p l y b y a l l i n v o l v e d in the e v a l u a t i o n p r o c e s s .

I m p a c t a n a l y s i s

9 . T h e i n t e r r e l a t i o n s h i p of h e a l t h a n d o t h e r s o c i a l a n d e c o n o m i c f a c t o r s i n d e v e l o p m e n t m a d e the a s s e s s m e n t o f the o v e r a l l i m p a c t of h e a l t h p r o g r a m m e s a l l the m o r e i m p o r t a n t . In this c o n t e x t , the n e e d w a s s t r e s s e d to a s s e s s a n y e f f e c t s of h e a l t h p r o g r a m m e s w h i c h h a d not b e e n a n t i c i p a t e d w h e n the o b j e c t i v e s of the p r o g r a m m e h a d b e e n f o r m u l a t e d . T h e s e u n a n t i c i p a t e d e f f e c t s c o u l d , of c o u r s e , h a v e e i t h e r p o s i t i v e or n e g a t i v e i m p l i c a t i o n s .

1 0 . I n c o n c l u s i o n , the P r o g r a m m e C o m m i t t e e :

1 0 • 1 R e c o m m e n d e d that the g u i d e l i n e s b e u s e d o n a p i l o t b a s i s in a n u m b e r of s e l e c t e d W H O p r o g r a m m e s a n d in a few c o u n t r i e s that m i g h t b e i n t e r e s t e d in a p p l y i n g t h e m to t h e i r p r o g r a m m e s；

10.2 L o o k e d f o r w a r d to a s p e e d y i m p l e m e n t a t i o n of this p i l o t a p p l i c a t i o n a n d a w a i t e d its r e s u l t s w i t h g r e a t i n t e r e s t ;

1 0 . 3 S t r e s s e d t h a t the e v a l u a t i o n p r o c e s s i t s e l f s h o u l d b e the s u b j e c t of e v a l u a t i o n f r o m t i m e to t i m e w i t h a v i e w to its improveraent a n d e n s u r i n g i t s p r o p e r a p p l i c a t i o n ;

1 0 . 4 R e a f f i r m e d i t s r e c o m m e n d a t i o n to the E x e c u t i v e B o a r d to p u b l i s h the G u i d e l i n e s for H e a l t h P r o g r a m m e E v a l u a t i o n j o i n t l y w i t h the G u i d e l i n e s for C o u n t r y H e a l t h P r o g r a m m i n g a n d M e d i u m - T e r m P r o g r a m m i n g .

W O R L D H E A L T H O R G A N I Z A T I O N

ORGANISATION M O N D I A L E D E L A SANTÉ _{A N N E X} EB61/20

E X E C U T I V E B O A R D E B6 1 / P C / W P/ 6

S i x t y - f i r s t S e s s i o n 12 O c t o b e r 1977

P r o g r a m m e C o m m i t t e e of the E x e c u t i v e B o a r d 31 O c t o b e r 一 4 N o v e m b e r 1977

D E V E L O P M E N T O F H E A L T H P R O G R A M M E E V A L U A T I O N P r o g r e s s r e p o r t b y the D i r e c t o r — G e n e r a l

The P r o g r a m m e C o m m i t t e e w a s r e q u e s t e d b y the E x e c u t i v e B o a r d i n r e s o l u t i o n E B 5 9 . R 2 7 in J a n u a r y 1977 : . . to b e c o m e a c t i v e l y i n v o l v e d in the d e v e l o p m e n t a n d p r a c t i c a l a p p l i c a t i o n of the O r g a n i z a t i o n ' s e v a l u a t i o n s y s t e m . • This d o c u m e n t c o n t a i n s i n f o r m a t i o n o n the p r o g r e s s m a d e i n d e v e l o p i n g h e a l t h p r o g r a m m e e v a l u a t i o n . Since the m a t t e r w a s last c o n s i d e r e d b y the E x e c u t i v e B o a r d in J a n u a r y 1 9 7 6 , P r o v i s i o n a l G u i d e l i n e s f o r H e a l t h P r o g r a m m e E v a l u a t i o n h a v e b e e n f o r m u l a t e d a n d a r e a p p e n d e d as a n a n n e x .

The p r e s e n t r e p o r t is p r e s e n t e d to the P r o g r a m m e C o m m i t t e e for its c o n s i d e r a t i o n a n d w i t h a v i e w to g u i d a n c e for f u t u r e a c t i o n .

1 . I n t r o d u c t i o n

1.1 In J a n u a r y 1977 the E x e c u t i v e B o a r d , a t i t s f i f t y - n i n t h s e s s i o n , w h e n r e v i e w i n g t h e S i x t h G e n e r a l P r o g r a m m e of W o r k , r e a l i z e d i n t e r a l i a the n e e d to d e f i n e c l e a r l y the m e c h a n i s m s for e v a l u a t i n g t h e i m p a c t of W H O¹ s p r o g r a m m e a t t h e c o u n t r y l e v e l . In r e s o l u t i o n E B 5 9 . R 2 7 , it r e q u e s t e d the P r o g r a m m e C o m m i t t e e of t h e E x e c u t i v e B o a r d : . . to b e c o m e a c t i v e l y

i n v o l v e d in the d e v e l o p m e n t a n d p r a c t i c a l a p p l i c a t i o n of the O r g a n i z a t i o n ' s e v a l u a t i o n s y s t e m , i n c l u d i n g the m e c h a n i s m for e v a l u a t i n g the i m p a c t o f W H O1 s p r o g r a m m e a t the c o u n t r y l e v e l

； to

； a n d to r e p o r t t h e r e o n to

。

2

。 P o l i c y b a s i s

2 . 1 In J a n u a r y 1 9 7 6 the D i r e c t o r - G e n e r a l p r e s e n t e d a r e p o r t o n the d e v e l o p m e n t of e v a l u a t i o n in WH〇1 to t h e f i f t y - s e v e n t h s e s s i o n o f t h e E x e c u t i v e B o a r d .

2.2 In this r e p o r t the D i r e c t o r - G e n e r a l s t r e s s e d the i m p o r t a n c e of p r o g r a m m e e v a l u a t i o n for i n p r o v i n g d e c i s i o n - m a k i n g a n d p r o g r a m m e f o r m u l a t i o n a n d d e l i v e r y , r e a l i z i n g that s u c h e v a l u a - t i o n p r e s e n t e d c o m p l e x p r o b l e m s for n a t i o n a l a n d i n t e r n a t i o n a l h e a l t h w o r k . If in the p a s t e m p h a s i s h a d b e e n laid m a i n l y on f a c t u a l a n a l y s i s of w h a t h a d h a p p e n e d , the r e p o r t n o w a d v o c a t e d that g r e a t e r a t t e n t i o n b e g i v e n to m e a s u r i n g the e x t e n t to w h i c h p r o g r a m m e o b j e c t i v e s w e r e b e i n g a t t a i n e d , a n d the e f f e c t a n d v a l u e o f w h a t h a d b e e n a c c o m p l i s h e d . The r e p o r t a l s o s t r e s s e d the n e e d f o r e v e r y o n e i n v o l v e d in the w o r k of t h e O r g a n i z a t i o n , a t w h a t e v e r o r g a n i z a - t i o n a l levelj to a s s u m e r e s p o n s i b i l i t y for e v a l u a t i o n w i t h i n h i s field of a c t i v i t i e s . T h e d i s c u s s i o n in the B o a r d led to the a d o p t i o n of a r e s o l u t i o n ( E B 5 7 . R 1 7 ) in w h i c h the D i r e c t o r - G e n e r a l w a s r e q u e s t e d to i m p l e m e n t the p r o p o s a l s m a d e in h i s r e p o r t a n d to i n f o r m the E x e c u t i v e

1 Ш 0 O f f i c i a l R e c o r d s , N o . 2 3 1 , 1 9 7 6 , P a r t I I , A p p e n d i x 7 .

EB61/PC/WP/6 page 2

Board and the World Health Assembly of experience gained and progress m a d e in the development of programme evaluation. The resolution also recommended that all Member States introduce the renewed approach to the evaluation of health programmes.

3 . Progress in the development of health programme evaluation

3.1 The Director-General informed the Programme Committee in November 1 9 7 61 that, following the adoption of resolution E B 5 7 . R 1 7 , he had set up a t headquarters a Team for the Development of Programme Evaluation. This interdisciplinary team w a s made responsible for developing the Organization1 s programme evaluation system under the auspices of the Headquarters Programme Committee and in close collaboration with focal points for evaluation in the regionse The Team designed first draft guidelines for evaluation. These w e r e based on the document

presented to the Executive Board (see section 2 above)； on recommendations m a d e in various numbers of the Technical Report Series ； on documents concerning evaluation processes in use in the various regions of the Organization； and on practical experience. The guidelines had been distributed to regional and headquarters staff for comments. The first draft w a s

confined to presenting the general principles of evaluation and to giving guidance on the process of evaluation in W H O in a step-by-step m a n n e r . The draft also indicated the linkages between evaluation and the planning and programming processes for h e a l t h , and also w i t h the information systems support required for proper evaluation.

3.2 Before the second draft w a s prepared, it was realized that, in view of the discussions at the Thirtieth World Health Assembly and the Programme Committee of the Executive B o a r d , the development of evaluation guidelines for use b y the countries themselves w a s a n urgent m a t t e r . Not only did this fall in line w i t h the recommendation in resolution EB57.R17 that all Member States introduce the new approach to evaluating health p r o g r a m m e s , b u t it also corresponded to the policy of concentrating on rendering services that would enable Member States to improve their health activities rather than on improving the Organization¹ s efficiency and effective- ness in an isolated m a n n e r . In revising the first draft guidelines, therefore, a whole new section was added on the process of evaluation of national health programmes b y national health personnel• An additional reason for formulating evaluation guidelines for national use w a s the increasing number of countries that w e r e introducing country health programming and were requesting a methodology that they could use for the evaluation of the health programmes initiated as a result of the country health programming process.

3.3 Close liaison is being maintained between the development of the health programme

evaluation process and that of the process for medium-term programming and programme budgeting.

For e x a m p l e , the guidelines for health programme evaluation will be applied to the evaluation of the Organization1 s medium-term progranimes as they evolve. For further d e t a i l s , see document EB6l/pc/wp/4, on medium-term programming.

3.4 L i k e w i s e , the guidelines w i l l be used for the evaluation of current programmes and

projects that form part of the O r g a n i z a t i o n1s programme budget. The Organization's information system, and in particular the reporting system, will b e used to provide information as a basis for e v a l u a t i o n , as w e l l as to record the evaluation results and transmit them to those

concerned. Further information w i l l be found in document EB6l/pc/wp/7, on the information system.

3.5 Close collaboration for the development of evaluation throughout the Uni ted Nations system has been maintained w i t h other agencies in the system, particularly under the aegis of the Joint Inspection Unit of the United Nations. This Unit has recently been given the

responsibility for better coordination of evaluation efforts throughout the United Nations ^ system. In this context it has prepared a report on evaluation in the United Nations system, in the preparation of which W H O w a s closely consulted. A n annex on the current u s e of

evaluation in WHO is included in the Joint Inspection U n i t1 s report. The entire report w i l l be presented to the sixty-first session of the Executive Board in January 1 9 7 8 .

1 W H O Official Records, N o . 2 3 8 , 1977, p . 2 3 6 . 2 “Document J I U / R E P / 7 7 . 1 .

E B 6 1 / P C / W P / 6 p a g e 3 3 . 6 T h e e v a l u a t i o n o f h e a l t h p r o g r a m m e s r e s u l t i n g f r o m c o u n t r y h e a l t h p r o g r a m m i n g , a s w e l l a s t h e e v a l u a t i o n o f t h e c o u n t r y h e a l t h p r o g r a m m i n g p r o c e s s i t s e l f , w e r e s t u d i e d a n d r e v i e w e d a t t h e i n t e r r e g i o n a l s e m i n a r o n c o u n t r y h e a l t h p r o g r a m m i n g h e l d i n N e w D e l h i i n F e b r u a r y 1977，

u n d e r t h e a u s p i c e s o f t h e R e g i o n a l O f f i c e f o r S o u t h - E a s t A s i a . T h e e v a l u a t i o n o f t h e c o u n t r y h e a l t h p r o g r a m m i n g p r o c e s s l e d t o t h e сone l u s i o n t h a t c o u n t r y h e a l t h p r o g r a m m i n g m u s t n o w m o v e f r o m t h e f e a s i b i l i t y p h a s e to t h e o p e r a t i o n a l p h a s e . I t m u s t n o w b e e v a l u a t e d i n t e r m s o f t h e d e s i r e d c h a n g e s i t w a s b r i n g i n g a b o u t in n a t i o n a l h e a l t h p o l i c i e s , p l a n s a n d p r o g r a m m e s - a n d e v e n t u a l l y i n t e r m s o f d e s i r e d c h a n g e s i n n a t i o n a l h e a l t h s i t u a t i o n s 一 r a t h e r t h a n i n t e r m s o f t h e n u m b e r o f c o u n t r i e s t h a t h a d i n i t i a t e d t h e p r o c e s s o r t h e i n c r e a s e d i n v e s t m e n t in h e a l t h r e s u l t i n g t h e r e f r o m . I t w a s a s y e t t o o e a r l y t o e v a l u a t e c o u n t r y h e a l t h p r o g r a m m i n g i n t h e s e t e r m s , s i n c e a n u m b e r of y e a r s m u s t e l a p s e f o r t h i s t o b e p o s s i b l e . N e v e r t h e l e s s i n d i c a t o r s , c r i t e r i a a n d q u e s t i o n s b a s e d o n d i s c r e t i o n h a d t o b e f o r m u l a t e d . F o r e x a m p l e , h o w m a n y

c o u n t r i e s w o u l d c o n t i n u e t h e p r o c e s s i n a p e r m a n e n t m a n n e r ? T o w h a t e x t e n t w o u l d p r i o r i t y p r o g r a m m e s b e i m p l e m e n t e d a n d h o w w o u l d t h i s i m p l e m e n t a t i o n b e r e f l e c t e d i n h e a l t h s e r v i c e s ? W o u l d w e l l - f o r m u l a t e d p r o g r a m m e s l e a d t o g r e a t e r i m p r o v e m e n t s i n t h e h e a l t h s i t u a t i o n t h a n p r e v i o u s h e a l t h p r a c t i c e s ? H o w w o u l d c o u n t r i e s d e a l w i t h p r o b l e m s t h a t h a d n o t q u a l i f i e d f o r p r i o r i t y a t t e n t i o n a n d y e t c o u l d n o t b e c o m p l e t e l y n e g l e c t e d ? I t w a s e m p h a s i z e d t h a t s e r i o u s e f f o r t s m u s t b e m a d e t o b u i l d a n e v a l u a t i o n e l e m e n t i n t o c o u n t r y h e a l t h p r o g r a m m i n g a t a l l o r g a n i z a t i o n a l l e v e l s , i t b e i n g c l e a r l y u n d e r s t o o d t h a t e v a l u a t i o n o f c o u n t r y h e a l t h p r o g r a m - m i n g w a s e s s e n t i a l l y a n a t i o n a l p r o c e s s . F u r t h e r d e t a i l s a r e to b e f o u n d i n d o c u m e n t EB61/PC/WP/8, o n c o u n t r y h e a l t h p r o g r a m m i n g .

3.7 T h e E x e c u t i v e B o a r d i t s e l f h a s b e e n a c t i v e i n a s s e s s i n g a n d e v a l u a t i n g W H Ofs a c t i v i t i e s a t t h e c o u n t r y l e v e l i n t h e c o n t e x t o f i t s o r g a n i z a t i o n a l s t u d y o n W H O ' S r o l e a t t h e c o u n t r y l e v e l , p a r t i c u l a r l y t h e r o l e o f t h e W H O r e p r e s e n t a t i v e s . T h i s a s s e s s m e n t t o o k t h e f o r m o f s t u d y v i s i t s t o 1 1 c o u n t r i e s i n f i v e r e g i o n s t o a l l o w a n o b j e c t i v e v i e w of t h e p r i n c i p a l p r o b l e m s o f t h e d i f f e r e n t c o u n t r i e s b a s e d o n d i r e c t o b s e r v a t i o n o f t h e i r s p e c i f i c s i t u a t i o n . To t h i s e n d , t h e m e m b e r s o f t h e g r o u p h e l d c o n s u l t a t i o n s w i t h t h e n a t i o n a l h e a l t h a u t h o r i t i e s a t t h e h i g h e s t l e v e l , w i t h r e p r e s e n t a t i v e s of o t h e r e c o n o m i c a n d s o c i a l s e c t o r s , a n d w i t h W H O

s t a f f f r o m t h e r e g i o n a l o f f i c e s , r e p r e s e n t a t i v e s of W H O or o f o t h e r U n i t e d N a t i o n s a g e n c i e s a n d f u n d s , a n d r e p r e s e n t a t i v e s o f o t h e r m u l t i l a t e r a l a n d b i l a t e r a l a i d a g e n c i e s . F o r t h i s a s s e s s - m e n t m e m b e r s o f t h e E x e c u t i v e B o a r d ' s W o r k i n g G r o u p u s e d a s e r i e s o f q u e s t i o n s a d a p t e d f r o m

c r i t e r i a i n W H O1 s f i r s t d r a f t G u i d e l i n e s f o r H e a l t h P r o g r a m m e E v a l u a t i o n , 4 . G u i d e l i n e s f o r H e a l t h P r o g r a m m e E v a l u a t i o n

4e1 T h e s e c o n d d r a f t o f t h e P r o v i s i o n a l G u i d e l i n e s f o r H e a l t h P r o g r a m m e E v a l u a t i o n i s a n n e x e d . T h e s e g u i d e l i n e s a r e b a s e d o n p r i n c i p l e s t h a t w e r e a p p r o v e d b y t h e E x e c u t i v e B o a r d i n 1 9 7 6 , w h e n i t a d o p t e d r e s o l u t i o n E B 5 7 . R 1 7 . T h e g e n e r a l p r i n c i p l e s f o r h e a l t h p r o g r a m m e e v a l u a t i o n a n d t h e p r o c e s s o f e v a l u a t i o n a r e p r e s e n t e d i n s o m e d e t a i l , s i n c e t h e y a p p l y b o t h t o e v a l u a t i o n in W H O a n d t o t h e e v a l u a t i o n o f n a t i o n a l h e a l t h p r o g r a m m e s a n d s e r v i c e s b y n a t i o n a l h e a l t h p e r s o n n e l . H o w e v e r , t h e s e p r i n c i p l e s h a d t o b e a d a p t e d t o t h e f r a m e w o r k o f e v a l u a t i o n i n W H O o n t h e o n e h a n d , a n d t o t h a t o f M e m b e r S t a t e s o n t h e o t h e re T h e g u i d e l i n e s s t r e s s t h e p u r p o s e o f e v a l u a t i o n a s a s y s t e m a t i c w a y o f l e a r n i n g f r o m e x p e r i e n c e a n d u s i n g t h e l e s s o n s

l e a r n e d to i m p r o v e c u r r e n t a c t i v i t i e s a n d p r o m o t e b e t t e r p l a n n i n g b y c a r e f u l s e l e c t i o n o f a l t e r n a t i v e s f o r f u t u r e a c t i o n

。

4 . 2 T h e g u i d e l i n e s o u t l i n e t h e c o n s t r a i n t s o f e v a l u a t i o n , a n d i n p a r t i c u l a r , t h e c o m p l e x c h a n g e s i n a h e a l t h s i t u a t i o n t h a t a r e o f t e n b r o u g h t a b o u t b y e l e m e n t s o u t s i d e t h e h e a l t h s e c t o r b e c a u s e of t h e i n t r i c a t e i n t e r r e l a t i o n s h i p s b e t w e e n t h e h e a l t h a n d t h e o t h e r s o c i a l a n d e c o n o m i c s e c t o r s . T h i s s i t u a t i o n o f t e n m a k e s i t n e c e s s a r y t o a p p l y q u a l i t a t i v e j u d g e m e n t , s u p p o r t e d w h e r e v e r p o s s i b l e b y r e l i a b l e , q u a n t i f i e d i n f o r m a t i o n . T h e g u i d e l i n e s a l s o e x p l a i n the p l a c e o f e v a l u a t i o n a s p a r t of a b r o a d e r p r o c e s s of h e a l t h d e v e l o p m e n t , r a n g i n g f r o m p o l i c y f o r m u l a t i o n , t h e t r a n s l a t i o n o f p o l i c i e s i n t o b r o a d p l a n s a n d t h e f o r m u l a t i o n of m o r e d e t a i l e d p r o g r a m m e s , t o t h e i m p l e m e n t a t i o n o f t h e s e p r o g r a m m e s .

4。3 T h e y d e s c r i b e t h e p r o c e s s o f e v a l u a t i o n , i n d i c a t i n g t h e t w o f o l d f r a m e w o r k , n a m e l y : (a) W H O1s p r o g r a m m e o f c o l l a b o r a t i o n w i t h c o u n t r i e s ； a n d ( b ) t h e c o u n t r i e s1 t o t a l h e a l t h p r o g r a m m e s . T h e y a l s o i n d i c a t e r e s p o n s i b i l i t i e s f o r e v a l u a t i o n a t a l l o p e r a t i o n a l l e v e l s w i t h i n c o u n t r i e s a n d w i t h i n W H O

； t h e y e x p l a i n t h e u s e of i n d i c a t o r s a n d c r i t e r i a f o r

E B 6 1 / P C / W P / 6 p a g e 4

e v a l u a t i o n a n d s t r e s s the i m p o r t a n c e of the s u p p o r t of v a l i d , r e l e v a n t a n d s e n s i t i v e i n f o r m a t i o n ； a n d t h e y s h o w h o w W H O¹ s i n f o r m a t i o n s y s t e m s s h o u l d b e u s e d for this p u r p o s e . The v a r i o u s c o m p o n e n t s of e v a l u a t i o n a r e l i s t e d , n a m e l y : the a s s e s s m e n t of r e l e v a n c e of p r o g r a m m e s , the p r o g r e s s b e i n g m a d e w i t h t h e i r d e l i v e r y , t h e e f f i c i e n c y w i t h w h i c h t h e y a r e b e i n g i m p l e m e n t e d , t h e i r e f f e c t i v e n e s s , a n d t h e i r o v e r a l l i m p a c t on h e a l t h a n d s o c i o e c o n o m i c d e v e l o p m e n t . T h e n the s t e p s in the e v a l u a t i o n p r o c e s s a r e l i s t e d , i t b e i n g s t r e s s e d t h a t t h e y are i n t e n d e d to b e f l e x i b l e a n d s h o u l d b e a d a p t e d to the c i r c u m s t a n c e s u n d e r w h i c h t h e y a r e to b e u s e d . D e t a i l s a r e t h e n g i v e n of h o w to f o l l o w t h e s e s t e p s for the p r o c e s s of e v a l u a t i o n i n W H O a n d for the p r o c e s s of n a t i o n a l h e a l t h p r o g r a m m e e v a l u a t i o n .

5 . F u t u r e a c t i o n

5 . 1 The s e c o n d d r a f t of the g u i d e l i n e s h a s b e e n p r e p a r e d for t e s t i n g i n M e m b e r S t a t e s a n d in W H O b e f o r e b e c o m i n g o p e r a t i o n a l . The t r y i n g o u t of the n a t i o n a l g u i d e l i n e s is f o r e s e e n i n a t

l e a s t one c o u n t r y i n e a c h r e g i o n in the n e a r f u t u r e , so t h a t a g r a d u a l i m p l e m e n t a t i o n of the r e n e w e d a p p r o a c h to e v a l u a t i o n c a n b e e n v i s a g e d for the n e x t b i e n n i u m . A t p r e s e n t , the p r o v i s i o n a l g u i d e l i n e s a r e b e i n g d i s c u s s e d a t r e g i o n a l o f f i c e s a n d a t m e e t i n g s of W H O r e p r e s e n t a t i v e s as w e l l as a t h e a d q u a r t e r s , w i t h a v i e w to i n t r o d u c i n g them p r o g r e s s i v e l y

t h r o u g h o u t the O r g a n i z a t i o n。 M e m b e r S t a t e s w i l l b e a p p r o a c h e d to a s c e r t a i n t h e i r i n t e r e s t in t e s t i n g the g u i d e l i n e s for the e v a l u a t i o n of their h e a l t h p r o g r a m m e s .

5.2 T h e r e v i e w of the g u i d e l i n e s b y the P r o g r a m m e C o m m i t t e e t h e r e f o r e c o m e s a t a c r u c i a l m o m e n t , s i n c e its c o m m e n t s a n d s u g g e s t i o n s w i l l h e l p to g u i d e the O r g a n i z a t i o n i n the p r o p e r c o u r s e of a c t i o n to b e f o l l o w e d . If the P r o g r a m m e C o m m i t t e e finds t h e s e g u i d e l i n e s (with p o s s i b l e m o d i f i c a t i o n s ) v a l i d and r e a d y for i m p l e m e n t i n g , i n i t i a l l y o n a n e x p e r i m e n t a l b a s i s , i t m a y w i s h to c o n s i d e r w a y s of p r o m o t i n g t h e i r u s e .

WORLD HEALTH ORGANIZATION ^{EB6l /pc/wp /6}

ORGANISATION MONDIALE DE LA SANTÉ

HPC/DPE/77.2

SECOND DRAFT:

PROVISIONAL GUIDELINES FOR H E A L T H PROGRAMME EVALUATION

CONTENTS

Page

1. INTRODUCTION 3 2 . GENERAL PRINCIPLES 4

2.1 Evaluation in the health development process • • • 4

2.1.1 Purpose of evaluation 4 2.1.2 Constraints of evaluation 4 2.1.3 Place of evaluation in the health development process 4

2.2 Process of evaluation 5 2.2.1 Framework for evaluation • • 5

2.2.2 Responsibilities for evaluation 5 2.2.3 Indicators and criteria for evaluation 7

2.2.4 Information support 8 2.2.5 Components of evaluation 9 2.2.6 STEPS to b e taken 10 3 . THE PROCESS OF EVALUATION IN W H O 10

3.1 Framework for evaluation 10 3.2 Responsibilities for evaluation 11

3.3 WHO information support • • • • • • 11 3 . 4 STEPS to

3 . 5 STEP 1:

3 . 6 STEP 2: Ensure . . . 1 4

3 . 7 STEP 3: . . . 1 5

3 . 8 STEP 4: Review . . . 1 5

3 . 9 STEP 5: V e r i f y . . . 1 6

3 . 10 STEP 6: Review . . . 1 7

3 . 11 STEP 7: Review . . . 1 7

Geneva J u l y 1977

The issue of this document does not constitute formal publication. It should not be reviewed, abstracted or quoted without the agreement of the World Health Organization. Authors alone are responsible for views expressed in signed articles.

Ce document ne constitue pas une publication.

Il ne doit faire l'objet d'aucun compte rendu ou résumé ni d'aucune citation sans l'autorisation de l'Organisation Mondiale de la Santé. Les opinions exprimées dans les articles signés n'engagent que leurs auteurs.

H P C / D P E / 7 7 . 2 p a g e 2

Page 3.12 STEP 8: Summarize m a i n features of evaluation

3.13 STEP 9: Draw conclusions and formulate proposals for future action 4 . THE PROCESS OF EVALUATION OF NATIONAL HEALTH PROGRAMMES AND SERVICES BY

NATIONAL HEALTH PERSONNEL

19 20

20 4

4 4 4 4 4 4 4 4 4 4 4 4 4 A N N E X 1 A N N E X 2 ANNEX 3 ANNEX 4

Framework for evaluation 20 Responsibilities for evaluation 21

National health information support 21

STEPS to be taken 21 Specify subject for evaluation 22

Ensure information support 22 Analyse adequacy of available information 24

STEP STEP STEP

STEP (Optional) Plan and implement information collection for special

studies 24 Review adequacy of problem definition 24

Verify programme relevance • 25 Review adequacy of programme or project formulation 26

Review p r o g r e s s , efficiency, effectiveness and impact 27

Summarize main features of evaluation 32 Draw conclusions and formulate proposals for future action 32

EB57.R17: Development of programme evaluation in WHO 33

Programmes and projects profiles 。 34 Revised outline of the country profile 40 .9 STEP 5:

.10 STEP 6:

.11 STEP 7:

.12 STEP 8:

.13 STEP 9:

.14 STEP 10

Outline of country programme profile and profile for services and

institutions 41

A N N E X 5: Example of a coçt analysis 43

H P C / D P E / 7 7 . 2 p a g e 3 10 I N T R O D U C T I O N

M e m b e r S t a t e s o f t h e W o r l d H e a l t h O r g a n i z a t i o n h a v e b e c o m e i n c r e a s i n g l y i n s i s t e n t t h a t h e a l t h p r o g r a m m e s s h o u l d b e r e l e v a n t to the m o s t i m p o r t a n t s o c i a l a n d h e a l t h n e e d s of t h e i r p o p u l a t i o n s . T h e i r d e s i r e t o i m p r o v e the h e a l t h a n d q u a l i t y o f l i f e o f t h e i r c i t i z e n s , i n s p i t e of s c a r c i t y o f r e s o u r c e s , m a k e s i t i m p o r t a n t t o e v a l u a t e h e a l t h p r o g r a m m e s in o r d e r t o e n s u r e t h a t t h e m o s t i s m a d e of t h e m .

M e m b e r S t a t e s h a v e r e q u e s t e d W H O to d e v e l o p g u i d e l i n e s for the e v a l u a t i o n o f i t s

p r o g r a m m e s t h a t w o u l d a l s o b e u s e f u l for t h e e v a l u a t i o n of n a t i o n a l h e a l t h p r o g r a m m e s . T h e p r o v i s i o n a l g u i d e l i n e s t h a t f o l l o w h a v e t h e r e f o r e b e e n p r e p a r e d for a l l t h o s e w h o a r e i n v o l v e d in t h e e v a l u a t i o n p r o c e s s , w h e t h e r in W H O o r i n c o u n t r i e s . T h e y a r e b a s e d on p o l i c i e s a n d p r i n c i p l e s w h i c h w e r e d i s c u s s e d a n d a p p r o v e d b y t h e E x e c u t i v e B o a r d o f t h e W o r l d H e a l t h O r g a n i z a t i o n . 1

E v a l u a t i o n i s b y n o m e a n s s o m e t h i n g n e w； i t s c o n c e p t , h o w e v e r , h a s u n d e r g o n e r e v i s i o n . I t s h o u l d n o l o n g e r b e u s e d a s a n a d h o c m e c h a n i c a l t o o l f o r m e a s u r i n g s u c c e s s o r f a i l u r e ； i t s h o u l d r a t h e r b e a c o n t i n u i n g p r o c e s s i n t e n d e d m a i n l y a t c o r r e c t i n g a n d i m p r o v i n g a c t i o n s a n d r e n d e r i n g h e a l t h p r o g r a m m e s m o r e r e l e v a n t a n d e f f e c t i v e . I n t h i s p e r s p e c t i v e e v a l u a t i o n c a l l s f o r a n a t t i t u d e o f m i n d t h a t i s o p e n to c o n s t r u c t i v e c r i t i c i s m . I t f u r t h e r r e q u i r e s w i l l i n g n e s s to c o m m u n i c a t e f r e e l y w i t h p r o f e s s i o n a l p e e r s a n d o t h e r p e r s o n s o r g r o u p s c o n c e r n e d a t v a r i o u s o p e r a t i o n a l l e v e l s of h e a l t h s y s t e m s a s w e l l a s o t h e r r e l a t e d s o c i a l a n d e c o n o m i c s y s t e m s , w h e t h e r n a t i o n a l or i n t e r n a t i o n a l#

E v a l u a t i o n r e q u i r e s s o u n d j u d g e m e n t ; t h a t i s , j u d g e m e n t in the s e n s e of c a r e f u l a s s e s s - m e n t a n d c r i t i c a l a p p r a i s a l o f g i v e n s i t u a t i o n s w h i c h s h o u l d l e a d t o t h e d r a w i n g o f s e n s i b l e c o n c l u s i o n s a n d t h e m a k i n g of u s e f u l p r o p o s a l s f o r f u t u r e a c t i o n , a n d n o t i n t h e j u d i c i a l s e n s e of p r o n o u n c i n g a s e n t e n c e . T h i s j u d g e m e n t h a s t o b e b a s e d o n r e l e v a n t a n d s e n s i t i v e i n f o r m a t i o n t h a t is r e a d i l y a v a i l a b l e , arid t h a t is m a d e a v a i l a b l e t o a l l t h o s e w h o n e e d i t

。

a p p e a r i n g i n t h e s e g u i d e l i n e s s h o u l d , t h e r e f o r e , b e r e g a r d e d a s a d e v i c e t o p r o v i d e a

s y s t e m a t i c b a s i s f o r e v a l u a t i o n . S u c h a b a s i s s h o u l d h e l p to r e d u c e t h e e f f o r t r e q u i r e d t o d e a l w i t h t h e m e c h a n i s m s f o r e v a l u a t i o n , t h u s f r e e i n g the m i n d to c o n c e n t r a t e o n t h e j u d g e m e n t r e q u i r e d .

E v e n t h o u g h a s t e p - b y - s t e p p r o c e s s i s s u g g e s t e d , i t s h o u l d b e n o t e d t h a t t h e g u i d e l i n e s - a s t h e w o r d i m p l i e s - a r e i n t e n d e d to p r o v i d e f l e x i b l e g u i d i n g p r i n c i p l e s

， a n d s h o u l d n o t b e

B e f o r e r e c o m m e n d i n g t h e s e g u i d e l i n e s f o r g e n e r a l u s e , i t i s i m p o r t a n t t h a t u s e r s b o t h i n W H O a n d i n c o u n t r i e s t r y t h e m o u t w i t h a v i e w t o i m p r o v i n g t h e m . I t is h o p e d t h a t t h e l e s s o n s

l e a r n e d f r o m t h e i r p r a c t i c a l a p p l i c a t i o n w i l l t h u s s h a r p e n t h e e v a l u a t i o n p r o c e s s c o n t a i n e d i n t h e m .

A d o u b t o f t e n r a i s e d w i t h r e s p e c t t o e v a l u a t i o n r e l a t e s t o i t s o w n b e n e f i t a s c o m p a r e d w i t h i t s c o s t . T h i s i s a v a l i d d o u b t t h a t e m p h a s i z e s the n e e d t o c o n s i d e r u n d e r t a k i n g e v a l u a t i o n a c t i v i t i e s i n t h e i r p r o p e r p e r s p e c t i v e i n t e r m s o f t h e t i m e a n d e n e r g y r e q u i r e d , a n d t h e r e f o r e t h e c o s t i n v o l v e d , i n r e l a t i o n to t h e i r p o t e n t i a l i n f l u e n c e i n l e a d i n g t o

i m p r o v e m e n t s i n the h e a l t h p r o g r a m m e s c o n c e r n e d . H o w e v e r , t h e m o r e e v a l u a t i o n is a c c e p t e d a s a c o n t i n u i n g p r o c e s s t h a t is p a r t o f the o v e r a l l p r o c e s s of d e v e l o p i n g a n d i m p l e m e n t i n g a n y h e a l t h p r o g r a m m e , t h e l e s s t h e q u e s t i o n of e x t r a c o s t w i l l a r i s e .

1 See r e s o l u t i o n E B 5 7eR 1 7 i n A n n e x !•

H P C / D P E / 7 7 . 2 p a g e 4

2 . G E N E R A L P R I N C I P L E S

2 . 1 E v a l u a t i o n in the h e a l t h d e v e l o p m e n t p r o c e s s 2 . 1 . 1 P u r p o s e of e v a l u a t i o n

E v a l u a t i o n is a s y s t e m a t i c w a y of l e a r n i n g f r o m e x p e r i e n c e arid u s i n g the l e s s o n s l e a r n e d t o i m p r o v e c u r r e n t a c t i v i t i e s a n d p r o m o t e b e t t e r p l a n n i n g b y c a r e f u l s e l e c t i o n of a l t e r n a t i v e s f o r f u t u r e a c t i o n。 T h i s i n v o l v e s a n a n a l y s i s of d i f f e r e n t p h a s e s of a p r o g r a m m e : its r e l e v a n c e , its f o r m u l a t i o n , its e f f i c i e n c y a n d e f f e c t i v e n e s s a n d its a c c e p t a n c e b y a l l p a r t i e s i n v o l v e d。

T h u s , t h e p u r p o s e of e v a l u a t i o n in the h e a l t h d e v e l o p m e n t p r o c e s s is to i m p r o v e h e a l t h p r o g r a m m e s a n d g u i d e t h e a l l o c a t i o n of h u m a n a n d f i n a n c i a l r e s o u r c e s in c u r r e n t a n d f u t u r e p r o g r a m m e s . I t s h o u l d n o t b e u s e d to j u s t i f y p a s t a c t i o n s or m e r e l y i d e n t i f y t h e i r i n a d e q u a - c i e s . I t is e s s e n t i a l to p e r c e i v e e v a l u a t i o n as a d e c i s i o n - o r i e n t e d t o o l³ a n d to link the e v a l u a t i o n p r o c e s s c l o s e l y w i t h d e c i s i o n - m a k i n g , w h e t h e r a t the o p e r a t i o n a l or the p o l i c y l e v e l . T h e v e r y p r o c e s s of c a r r y i n g o u t e v a l u a t i o n c a n b e j u s t as i m p o r t a n t as the c o n c l u - s i o n s d r a w n , s i n c e i n v o l v e m e n t in the p r o c e s s i t s e l f o f t e n i n d u c e s a b e t t e r u n d e r s t a n d i n g of the a c t i v i t i e s b e i n g e v a l u a t e d , a n d a m o r e c o n s t r u c ü i v e a p p r o e c h to t h e i r i m p l e m e n t a t i o n and f u t u r e a c t i o n r e q u i r e d .

2 . 1 . 2 C o n s t r a i n t s to e v a l u a t i o n

E v a l u a t i o n , d i f f i c u l t i n a n y f i e l d , p r e s e n t s p a r t i c u l a r p r o b l e m s in h e a l t h w o r k o w i n g to the v e r y n a t u r e of the a c t i v i t i e s , w h i c h o f t e n d o n o t l e n d t h e m s e l v e s e a s i l y to the m e a s u r e m e n t o f w h a t h a s b e e n a t t a i n e d a g a i n s t p r e d e t e r m i n e d , q u a n t i f i e d o b j e c t i v e s . I t is t h e r e f o r e o f t e n u n a v o i d a b l e t o a p p l y q u a l i t a t i v e j u d g e m e n t ^? s u p p o r t e d , w h e r e v e r p o s s i b l e , b y r e l i a b l e ,

q u a n t i f i e d i n f o r m a t i o n

。 A c c o u n t h a s to b e t a k e n of the i n t r i c a t e i n t e r r e l a t i o n s h i p s b e t w e e n

E v e n t h o u g h e v a l u a t i o n h a s b e e n a c c e p t e d , i n t h e o r y , as a v a l i d t o o l i n the o v e r a l l d e v e l o p m e n t p r o c e s s , t h e r e is s t i l l w i d e s p r e a d , i n - b u i l t r e s i s t a n c e to a c c e p t i n g i t a n d its r e s u l t s . A s e l f - d e f e n c e m e c h a n i s m m a k e s r e j e c t i o n e a s y , for it is n o t d i f f i c u l t to p r o v e t h a t a n e v a l u a t i o n p r o c e s s w a s n o t "scientific'¹ e n o u g h to p r o v i d e a " s o u n d " b a s i s for m a k i n g p r o g r a m m e d e c i s i o n s , or t o d i s c r e d i t the e v a l u a t i o n r e s u l t s b y c h a l l e n g i n g the v a l i d i t y of c e r t a i n c r i t e r i a u s e d .

O n e f u r t h e r l i m i t a t i o n for the e v a l u a t i o n of W H O1 s p r o g r a m m e lies i n t h e s p e c i f i c c a t a l y t i c r o l e of W H O i n c o u n t r i e s w h e r e i t s a c t i v i t i e s a r e a i m e d a t i n i t i a t i n g or c o l l a b o r a t i n g i n

n a t i o n a l p r o g r a m m e s to a t t a i n t h e i r s p e c i f i c o b j e c t i v e s . The a t t a i n m e n t of t h e s e o b j e c t i v e s o f t e n l i e s , to a g r e a t e x t e n t , o u t s i d e the i n f l u e n c e of W H O . I n the f i n a l a n a l y s i s h e a l t h d e v e l o p m e n t h a s to b e a s s e s s e d w i t h i n c o u n t r i e s , i r r e s p e c t i v e of w h e t h e r W H O c o n t r i b u t e d to s u c h d e v e l o p m e n t or n o t . The e f f e c t i v e n e s s of W H O c o l l a b o r a t i v e p r o g r a m m e s a t the c o u n t r y l e v e l c a n t h e r e f o r e o n l y b e e v a l u a t e d if the c o u n t r y is a l s o r e a d y to e v a l u a t e the e f f e c t i v e - n e s s of i t s p r o g r a m m e s .

2 . 1 . 3 P l a c e of e v a l u a t i o n in the h e a l t h d e v e l o p m e n t p r o c e s s

H e a l t h p r o g r a m m e e v a l u a t i o n is p a r t of a b r o a d e r p r o c e s s of h e a l t h d e v e l o p m e n t . This p r o c e s s c o n s i s t s of a n i n t e r l i n k i n g s e q u e n c e of e v e n t s r a n g i n g f r o m the g e n e r a l to the

p a r t i c u l a r . T h u s , h e a l t h p o l i c i e s h a v e to b e d e f i n e d . T h i s c o n s i s t s of s e l e c t i n g a c o u r s e of a c t i o n f r o m a m o n g a l t e r n a t i v e s i n o r d e r to g u i d e a n d d e t e r m i n e p r e s e n t a n d f u t u r e d e c i s i o n s . F o r e x a m p l e , a b r o a d h e a l t h p o l i c y m i g h t b e t o e n s u r e the e q u i t a b l e d i s t r i b u t i o n of h e a l t h r e s o u r c e s t h r o u g h o u t a c o u n t r y . A n i n t e r n a t i o n a l h e a l t h p o l i c y m i g h t b e to p r o m o t e c o o p e r a - t i o n a m o n g c o u n t r i e s so a s to d e v e l o p s e l f - r e l i a n c e in h e a l t h m a t t e r s .

H P C / D P E / 7 7 . 2 p a g e 5 To e n s u r e that the p o l i c i e s a r e t r a n s l a t e d i n t o a c t i o n , p l a n s a r e w o r k e d o u t . The p l a n s s h o u l d s p e c i f y o b j e c t i v e s . T h e a t t a i n m e n t of t h e s e o b j e c t i v e s w o u l d g i v e e f f e c t to the p o l i c y a d o p t e d . The p l a n n i n g p r o c e s s i m p l i e s s e l e c t i o n f r o m a m o n g a l t e r n a t i v e s , first of a l l

for s e l e c t i n g p r i o r i t y o b j e c t i v e s and t h e n for s e l e c t i n g the m e a n s for a t t a i n i n g t h e s e

o b j e c t i v e s . F o r e x a m p l e , to a t t a i n the o b j e c t i v e of r e d u c i n g i n f a n t m o r t a l i t y a b r o a d r a n g e of m e a n s is a v a i l a b l e , i n c l u d i n g e n s u r i n g safe d r i n k i n g - w a t e r , p r o m o t i n g b r e a s t - f e e d i n g , p r o v i d i n g m a t e r n a l a n d c h i l d h e a l t h s e r v i c e s and h e a l t h e d u c a t i o n , and e n c o u r a g i n g the a c t i v e p a r t i c i p a t i o n of m o t h e r s and f a t h e r s i n c h i l d c a r e .

For e a c h p r i o r i t y a r e a of c o n c e r n w i t h i n the p l a n , a m o r e d e t a i l e d p r o g r a m m e has to b e f o r m u l a t e d . For e x a m p l e , a m a t e r n a l a n d child h e a l t h p r o g r a m m e w i l l c o n s i s t of v a r i o u s i n s t i t u t i o n s , s e r v i c e s , a c t i v i t i e s , a n d p o s s i b l y d e v e l o p m e n t p r o j e c t s that h a v e to b e

e s t a b l i s h e d a n d m a d e o p e r a t i o n a l in o r d e r to a t t a i n the p r o g r a m m e¹ s o b j e c t i v e s . A p r o g r a m m e t h e r e f o r e c o n s i s t s of o b j e c t i v e s that h a v e b e e n d e t a i l e d in terms of q u a n t i t y and t i m e , the m e t h o d s to b e u s e d , the types and n u m b e r s of m a n p o w e r to b e e m p l o y e d , the p h y s i c a l f a c i l i t i e s , e q u i p m e n t a n d s u p p l i e s r e q u i r e d , the m o n e y n e e d e d for e a c h c o m p o n e n t o f t h e p r o g r a m m e a t

v a r i o u s p o i n t s of t i m e , a n d the s o u r c e s of the funds n e e d e d .

S p e c i f i c p l a n s of a c t i o n a r e r e q u i r e d for i m p l e m e n t i n g p r o g r a m m e s and for t r a c k i n g the v a r i o u s e v e n t s that g o i n t o the i m p l e m e n t a t i o n . S u c h t r a c k i n g , or m o n i t o r i n g , as it is c a l l e d ,

f a c i l i t a t e s d a y - t o - d a y r e g u l a t i o n or c o n t r o l of e v e n t s . The costs‘of p r o g r a m m e s h a v e to b e p r o p e r l y a c c o u n t e d for a l s o . L a s t b u t n o t l e a s t , p r o g r a m m e s n e e d to b e e v a l u a t e d to a s s e s s

their u s e f u l n e s s for s o l v i n g or r e d u c i n g h e a l t h p r o b l e m s a n d i m p r o v i n g h e a l t h s i t u a t i o n s . O n e w a y of c a r r y i n g o u t the a b o v e p r o c e s s that is b e i n g a p p l i e d b y g r o w i n g n u m b e r s of c o u n t r i e s is k n o w n as c o u n t r y h e a l t h p r o g r a m m i n g .

The d i a g r a m o n p a g e 6 i l l u s t r a t e s the p l a c e of e v a l u a t i o n in the h e a l t h d e v e l o p m e n t p r o c e s s .

2.2 P r o c e s s of e v a l u a t i o n 2 . 2 . 1 F r a m e w o r k for e v a l u a t i o n

The f r a m e w o r k for e v a l u a t i o n in t h e s e g u i d e l i n e s is t w o f o l d : (1) W H O 1 s p r o g r a m m e of c o l l a b o r a t i o n w i t h c o u n t r i e s

；

W i t h r e s p e c t to (1)， p r o g r a m m e a c t i v i t i e s take p l a c e m a i n l y w i t h i n s p e c i f i c c o u n t r i e s and a l s o a s i n t e r c o u n t r y c o l l a b o r a t i o n and a t the g l o b a l l e v e l . F u r t h e r d e t a i l s w i l l b e p r o v i d e d in s e c t i o n 39 "The p r o c e s s of e v a l u a t i o n in W H O " .

W i t h r e s p e c t to ( 2 ) , a l m o s t a l l c o u n t r i e s h a v e p l a n s to d e v e l o p t h e i r h e a l t h s y s t e m s , some h a v e s p e c i f i c h e a l t h p r o g r a m m e s , a n d a l l h a v e v a r i o u s t y p e s of h e a l t h i n s t i t u t i o n s a n d s e r v i c e s . The f r a m e w o r k for e v a l u a t i o n w i l l thus v a r y b y c o u n t r y . F u r t h e r d e t a i l s w i l l be p r o v i d e d in s e c t i o n 4， "The p r o c e s s of n a t i o n a l h e a l t h p r o g r a m m e e v a l u a t i o n " .

2 . 2 . 2 R e s p o n s i b i l i t i e s for e v a l u a t i o n

The p r i n c i p l e w h e r e b y e v a l u a t i o n s h o u l d take p l a c e a t a l l o p e r a t i o n a l levels of a h e a l t h s y s t e m i m p l i e s that i n d i v i d u a l s and g r o u p s r e s p o n s i b l e for the m a n a g e m e n t of s e r v i c e s ,

i n s t i t u t i o n s or p r o g r a m m e s f o r m i n g p a r t of the h e a l t h s y s t e m , a l s o c a r r y the r e s p o n s i b i l i t y for their e v a l u a t i o n . A t p e r i p h e r a l l e v e l s of a h e a l t h s y s t e m , p r o g r a m m e s , as w e l l as institu- tions for t h e i r d e l i v e r y , n o r m a l l y c o m p r i s e a l i m i t e d n u m b e r of s p e c i f i c a c t i v i t i e s . Those r e s p o n s i b l e for e v a l u a t i o n a t p e r i p h e r a l levels thus u s u a l l y h a v e f a i r l y w e l l - d e f i n e d a c t i v i - ties to e v a l u a t e in c o n j u n c t i o n w i t h o t h e r l o c a l , s o c i a l and e c o n o m i c a c t i v i t i e s . A t m o r e c e n t r a l l e v e l s , g r e a t e r v a r i e t i e s of m o r e s p e c i a l i z e d p r o g r a m m e a c t i v i t i e s and i n s t i t u t i o n s a r e u s u a l l y f o u n d . The c o m b i n a t i o n of t h e s e , their i n t e r r e l a t i o n s h i p s and their a g g r e g a t i o n

HPC/DPE/77.2 p a g e 6

D I A G R A M 1

E V A L U A T I O N IN R E L A T I O N T O T H E H E A L T H D E V E L O P M E N T P R O C E S S Health Development process Evaluation process

WHO 77533

H P C / D P E / 7 7 . 2 p a g e 7 w i t h a c t i v i t i e s a t p e r i p h e r a l l e v e l s a n d w i t h o t h e r s o c i a l a n d e c o n o m i c a c t i v i t i e s a t the same

l e v e l , a l l m a k e the f r a m e w o r k for e v a l u a t i o n m o r e c o m p l e x and add to the r e s p o n s i b i l i t i e s of those i n v o l v e d in the e v a l u a t i o n p r o c e s s .

A n i m p o r t a n t p a r t of the r e s p o n s i b i l i t y a t e a c h l e v e l is to e n s u r e that o t h e r i n d i v i d u a l s and g r o u p s c o n c e r n e d a t the s a m e level or a t o t h e r l e v e l s , w h e t h e r m o r e c e n t r a l l y or m o r e p e r i p h e r a l l y l o c a t e d , a r e in a p o s i t i o n to c o n t r i b u t e to the e v a l u a t i o n , a r e k e p t i n f o r m e d of its r e s u l t s and a r e r e q u e s t e d to take a p p r o p r i a t e a c t i o n . This is e s s e n t i a l in o r d e r to i n i t i a t e or m a i n t a i n the d i a l o g u e s a m o n g those c o n c e r n e d w h i c h a r e r e q u i r e d to b r i n g a b o u t i m p r o v e m e n t s .

F i n a l r e s p o n s i b i l i t y for the e v a l u a t i o n of the t o t a l h e a l t h s y s t e m , i n c l u d i n g p r o g r a m m e s , i n s t i t u t i o n s a n d s e r v i c e s , w i l l r e s t w i t h the c e n t r a l a u t h o r i t i e s . In c o u n t r i e s t h e s e m a y b e the C a b i n e t or the M i n i s t e r of H e a l t h , and i n a n o r g a n i z a t i o n such as W H O , the R e g i o n a l

D i r e c t o r , the D i r e c t o r - G e n e r a l a n d the W o r l d H e a l t h A s s e m b l y . For d e t a i l s of s p e c i f i c r e s p o n s i b i l i t i e s i n c o u n t r i e s , see s e c t i o n 4 . 2

； for W H O see s e c t i o n 3 . 2 .

2 . 2 . 3 I n d i c a t o r s a n d c r i t e r i a for e v a l u a t i o n

I n d i c a t o r s and c r i t e r i a a r e used as a i d s t h r o u g h o u t the e v a l u a t i o n p r o c e s s . T h e y a r e a l s o u s e d for p l a n n i n g a n d p r o g r a m m i n g . T h o s e u s e d for e v a l u a t i o n a r e s i m i l a r to the ones u s e d for p l a n n i n g a n d p r o g r a m m i n g , b u t s u i t a b l y a d a p t e d .

I n d i c a t o r s a r e v a r i a b l e s w h i c h h e l p to m e a s u r e c h a n g e s . C r i t e r i a a r e s t a n d a r d s a g a i n s t w h i c h a c t i o n s c a n b e c o m p a r e d .

(1) I n d i c a t o r s

I n d i c a t o r s c a n m e a s u r e c h a n g e d i r e c t l y or i n d i r e c t l y . F o r e x a m p l e , if the o b j e c t i v e of a p r o g r a m m e is to train a c e r t a i n n u m b e r of a u x i l i a r y h e a l t h p e r s o n n e l a n n u a l l y , a d i r e c t h e a l t h i n d i c a t o r for e v a l u a t i o n c o u l d b e the n u m b e r of s u c h p e r s o n n e l a c t u a l l y t r a i n e d e a c h y e a r . This is c a l l e d an o u t p u t i n d i c a t o r . If the s u b j e c t of e v a l u a t i o n is the r e s u l t of a p r o g r a m m e a i m e d a t i m p r o v i n g the l e v e l of h e a l t h of a c h i l d p o p u l a t i o n , i t m a y b e n e c e s s a r y to a s s e s s a n y i m p r o v e m e n t b y u s i n g s e v e r a l i n d i c a t o r s t h a t i n d i r e c t l y c o u l d m e a s u r e a c h a n g e in this l e v e l . S u c h i n d i c a t o r s c o u l d b e the n u t r i t i o n a l s t a t u s； w e i g h t in r e l a t i o n to h e i g h t ； a g e - s p e c i f i c m o r t a l i t y r a t e s ； d i s e a s e - s p e c i f i c m o r b i d i t y r a t e s ； l e a r n i n g c a p a c i t y , e t c .

I n d i c a t o r s h a v e to b e s e l e c t e d c a r e f u l l y to m a k e s u r e that t h e y a r e r e s p o n s i v e to c u r r e n t t r e n d s of d e v e l o p m e n t a n d that t h e y a r e u s a b l e for t h e a n a l y s i s of o n g o i n g a c t i v i t i e s . W h e n s e l e c t i n g i n d i c a t o r s , f u l l a c c o u n t has to b e taken of t h e e x t e n t to w h i c h t h e y a r e v a l i d ,

o b j e c t i v e , s e n s i t i v e and s p e c i f i c . V a l i d i t y i m p l i e s t h a t t h e i n d i c a t o r a c t u a l l y m e a s u r e s w h a t i t is s u p p o s e d to m e a s u r e . O b j e c t i v i t y i m p l i e s that e v e n if the i n d i c a t o r is u s e d b y d i f f e r e n t p e o p l e a t d i f f e r e n t times a n d u n d e r d i f f e r e n t c i r c u m s t a n c e s , the r e s u l t s w i l l be the s a m e .

S e n s i t i v i t y m e a n s that the i n d i c a t o r s h o u l d b e s e n s i t i v e to c h a n g e s in the s i t u a t i o n or p h e n o m e n o n c o n c e r n e d . H o w e v e r , i n d i c a t o r s c o u l d b e s e n s i t i v e to m o r e than o n e s i t u a t i o n or p h e n o m e n o n . S p e c i f i c i t y m e a n s that the i n d i c a t o r r e f l e c t s c h a n g e s o n l y in the s i t u a t i o n or p h e n o m e n o n c o n c e r n e d .

For e x a m p l e , the i n f a n t m o r t a l i t y r a t e is a s p e c i f i c , a l t h o u g h v e r y c r u d e , i n d i c a t o r of the l e v e l of h e a l t h of a c h i l d p o p u l a t i o n . I t is a d i r e c t m e a s u r e of d e a t h and o n l y a n i n d i r e c t m e a s u r e of h e a l t h , b u t w i t h i n this l i m i t a t i o n it is b o t h a v a l i d a n d a n o b j e c t i v e i n d i c a t o r . It is n o t s e n s i t i v e , h o w e v e r , i n r e l a t i o n to a n y p a r t i c u l a r h e a l t h a c t i o n , s i n c e its r e d u c t i o n c a n r e s u l t f r o m a large n u m b e r of f a c t o r s r e l a t e d to s o c i a l a n d e c o n o m i c d e v e l o p - m e n t , i n c l u d i n g h e a l t h d e v e l o p m e n t , a n d c a n r a r e l y b e a t t r i b u t e d to a n y o n e h e a l t h a c t i o n .

A n o t h e r i m p o r t a n t a t t r i b u t e of a n i n d i c a t o r is its a v a i l a b i l i t y , n a m e l y t h a t i t s h o u l d b e p o s s i b l e to o b t a i n the d a t a r e q u i r e d w i t h o u t u n d u e d i f f i c u l t y .

H P C / D P E / 7 7 . 2 p a g e 8

( 2 ) C r i t e r i a

C r i t e r i a m a y b e s o c i a l , t e c h n i c a l o r a d m i n i s t r a t i v e . T e c h n i c a l c r i t e r i a a r e n o r m a l l y h i g h l y s p e c i f i c t o p r o g r a m m e s . A n e x a m p l e of a s i m p l e s o c i a l c r i t e r i o n f o r a s s e s s i n g t h e q u a l i t y o f l i f e c o u l d b e t h e a v a i l a b i l i t y of s a f e d r i n k i n g - w a t e r . T h e t e c h n i c a l c r i t e r i o n t o g u a r a n t e e t h i s s a f e d r i n k i n g - w a t e r w o u l d h a v e t o b e a c e r t a i n t e c h n i c a l s t a n d a r d f o r the p u r e t y o f w a t e r . A n a d m i n i s t r a t i v e c r i t e r i o n t o g u a r a n t e e t h e c o n t i n u a t i o n o f t h e w a t e r s u p p l y c o u l d b e t h e e x i s t e n c e o f a c o m m u n i t y o r g a n i z a t i o n for t h e m a i n t e n a n c e of t h e s u p p l y .

T h e m a i n p u r p o s e of c r i t e r i a is to p r o v o k e t h o u g h t l e a d i n g to j u d g e m e n t . S u c h j u d g e m e n t c a n r a r e l y b e a r r i v e d a t o n l y t h r o u g h a d d i n g up n u m e r i c a l v a l u e s r e s u l t i n g f r o m c e r t a i n

q u a n t i f i e d e v a l u a t i o n c r i t e r i a .

N e v e r t h e l e s s , c r i t e r i a for t h e e v a l u a t i o n of h e a l t h p r o g r a m m e s s h o u l d b e q u a n t i f i e d w h e r e v e r p o s s i b l e , a l t h o u g h i n p r a c t i c e t h i s w i l l o f t e n n o t b e p o s s i b l e , p a r t i c u l a r l y w h e r e

s o c i a l or a d m i n i s t r a t i v e c r i t e r i a a r e c o n c e r n e d . I t is t h e r e f o r e o f t e n n e c e s s a r y t o r e s o r t t o a n e v a l u a t i o n b a s e d o n q u a l i t a t i v e r a t h e r t h a n q u a n t i t a t i v e a s s e s s m e n t .

( 3 ) P e r t i n e n t q u e s t i o n s

I t h a s to b e r e a l i z e d t h a t t h e r e w i l l b e h e a l t h a c t i v i t i e s f o r t h e e v a l u a t i o n of w h i c h n o s u i t a b l e i n d i c a t o r s a n d c r i t e r i a a r e a v a i l a b l e . I n t h e s e c a s e s , p e r t i n e n t q u e s t i o n s s h o u l d b e a s k e d c o n c e r n i n g t h e a c t i v i t y to b e e v a l u a t e d

。 A n s w e r s to t h e s e q u e s t i o n s w i l l h e l p t o

。 F o r

2 . 2 . 4 I n f o r m a t i o n s u p p o r t

E v a l u a t i o n h a s t o b e b a s e d o n v a l i d , r e l e v a n t a n d s e n s i t i v e i n f o r m a t i o n . T h e r e is o f t e n a n e x c e s s o f o n l y m a r g i n a l l y r e l e v a n t a n d s e n s i t i v e i n f o r m a t i o n。 O f t e n the m o s t n e e d e d i n f o r m a t i o n is n o t a v a i l a b l e a n d i t s c o l l e c t i o n c a n b e e x t r e m e l y c o s t l y . F o r the a b o v e r e a s o n s a n a p p r o a c h of h i g h s e l e c t i v i t y s h o u l d b e a d o p t e d a n d o n l y t h a t i n f o r m a t i o n s h o u l d b e c o l l e c t e d w h i c h h a s b e e n i d e n t i f i e d a s b e i n g r e a l l y c r u c i a l for t h e i s s u e b e i n g e v a l u a t e d .

T h e t y p e s of i n f o r m a t i o n r e q u i r e d m a y i n c l u d e p o l i t i c a l , s o c i a l , c u l t u r a l , e c o n o m i c a n d e n v i r o n m e n t a l f a c t o r s i n f l u e n c i n g t h e h e a l t h s i t u a t i o n a s w e l l a s m o r t a l i t y a n d m o r b i d i t y s t a t i s t i c s . A n y o f t h i s i n f o r m a t i o n m a y f o r m t h e b a s i s o f i n d i c a t o r s a n d c r i t e r i a f o r u s e i n t h e v a r i o u s s t e p s of t h e e v a l u a t i o n p r o c e s s . T h e i n f o r m a t i o n r e q u i r e d m a y a l s o c o n c e r n h e a l t h a n d r e l a t e d s o c i o e c o n o m i c p o l i c i e s , p l a n s a n d p r o g r a m m e s , a s w e l l a s t h e e x t e n t , s c o p e a n d u s e of h e a l t h s y s t e m s , s e r v i c e s a n d i n s t i t u t i o n s . T h u s , t h e i n f o r m a t i o n s o u g h t m a y b e h i s t o r i c a l , s o c i a l , p o l i t i c a l , e c o n o m i c , s c i e n t i f i c , t e c h n o l o g i c a l , d e m o g r a p h i c , e p i d e m i o l o g i c a l , o r g a n i z a - t i o n a l , l e g i s l a t i v e or r e l a t e d to i n v e n t o r i e s o f r e s o u r c e s .

I n f o r m a t i o n c o l l e c t i o n i s t h e l e a s t c o s t l y w h e n i t f o r m s a n i n t e g r a l p a r t o f a h e a l t h s y s t e m , p r o v i d e d t h e i n f o r m a t i o n is n e e d e d for m e a n i n g f u l p u r p o s e s s u c h a s h e a l t h p r o g r a m m e d e v e l o p m e n t or h e a l t h s e r v i c e m a n a g e m e n t .

I n W H O , a n i n f o r m a t i o n s y s t e m is i n t h e f i n a l s t a g e s o f d e v e l o p m e n t a n d w i l l b e c o m e p r o g r e s s i v e l y o p e r a t i o n a l s t a r t i n g f r o m 1 9 7 8 . F u r t h e r d e t a i l s of t h i s W H O i n f o r m a t i o n s y s t e m i n c l u d i n g i t s r e p o r t i n g c o m p o n e n t , i n s o f a r as i t a f f e c t s t h e e v a l u a t i o n p r o c e s s , a r e t o b e f o u n d i n s e c t i o n 3 . 3 b e l o w . N a t i o n a l h e a l t h i n f o r m a t i o n s y s t e m s v a r y w i d e l y b y c o u n t r y a n d i n m a n y c o u n t r i e s t h e r e is a n e e d t o e s t a b l i s h , r e i n f o r c e or r e s t r u c t u r e t h e s y s t e m t o p r o v i d e

s u p p o r t t o the h e a l t h d e v e l o p m e n t p r o c e s s , i n c l u d i n g e v a l u a t i o n . F u r t h e r d e t a i l s o f n a t i o n a l h e a l t h i n f o r m a t i o n s u p p o r t a r e t o b e f o u n d i n s e c t i o n 4 . 3 b e l o w .