(Instruments Used for Evaluation of

(1)

Appendix 9: Instruments l-4

(Instruments Used for Evaluation of

the Sustainability of Bench-Maji CDTI Project)

January-February 2006

filr)-

"

i1

(2)

_ lnstrument 1: national and project level

.-

NOTE:

.

ln

countries like Malawi, Uganda

and

Chad it

is

the

NOC

and her/

his

team that

is

actually implementing CDTI projects - while also

having

the national

role

of interacting with the NOTF

and

the Ministry of

Health.

This

instrument

collects information about everything these ' i _people

_do.

_All its indicators

are

therefore

relevant to them.

.

ln

countries like Nigeria, Cameroon and

DRC

these functions are shared

by

two

groups:

! " The project managers (State levelin Nigeria, p0ect levelin Cameroon and

DRC).

- A headquarters

^group

which oversees the several projects

as

they go

about

their

work.

Some indicators

in

this instrument

are

only relevant to

one of

these groups, not

both.

Others are

relevant

to

both.

The focus

of

this level's activities

in CDTI

The

main

function

of

this level

is

to develop,

revise

as necessary,

and

oversee implementation

of

CDTI

^policy.

This

level

also

^provides

supportto the

level

below

it:

.

Providing

targeted training,

HSAM

and monitoring/

supervision.

. Arranoino for

an

adeouate suoolv of

Mectizan.

Researcher:

^{Dr E}

Nnoruka

,

MrW Musolo, Mr

Hamus

Date: 25' January -

^15'n

February2006

Respondents: Zonal

Director

of

Health,

Zonal Oncho Cordinator, Zonal Director Public

^Health

u Diseases, NGDO Project Officer, ZonalAccountant, Capacity Building

Desk,

ZonalTransport

Desk,

Abbreviations/ acronyms

rJ CDD community directed distributor

CDTI community directed treatment with

ivermectin

r1 FLHF first line

health

facility

HSAM health education, sensitisation, advocacy,

mobilisation

-

^i.e.

activities which are aimed

at

getting allthe key

^players

to participate wholeheartedly

in

the

r \ ^programme

NGDO non-governmentaldevelopment organisation

.- NOC nationalonchocerciasis coordinator

NOCT nationalonchocerciasiscontrolteam

_

NOTF national onchocerciasis task force

Sustainability evaluation instrument no.1

-

national level

-

September 2004 version

(3)

fi!r{!":H kYffiffi*'#i,"

1.1

Check whether

there

is adequate

yearly planning for onchocerciasis control

in the health

service

at

this

level.

This indicator assesses whetherthe programme has become

integnted intothe

heafth seruice; whether management is accepting ownerchip of the programme; and whether the project is functioning

effectively in an

integrated

manner

Characteristics

of

the indicator

Sources

of information

a.

ln the overall written year plan of the health service at this level there should routinely be a section on onchocerciasis control. Note that this plan is usually not very detailed.

b.

There should also be a more detailed plan, specifically for onchocerciasis control, which makes provision for all key elements of onchocerciasis control: Mec{izan supply; targeted training ; targeted HSAM _;targeted monitoring/ supervision.

(Note that in the case of a c,ountry which contains many projects, the NOTF plan will be less detailed).

c.

The plan varies from year to year, showing that it is targeted to the soecific needs of each vear.

.

Examination of:

*

Written plans: yearly, quarterly, monthly etc.

*

Minutes of planning meetings.

.

lnterview with senior health service staff at this level:

+

Project director, her/ his team, her/ his superiors.

r

NOC, her/ his team, her/ his srrneriors

Describe the present situation:

There are yearly written activity plan for onchocerciasis control (200212003

-

2005/2006) containing all CDTI activities at the project office. Plans do not vary from year to year and is not targeted to specific needs. lnputs are obtained from mainly the Woreda Health

offices.

Similarly at the Zonal Health office there were Bugdet plans ,which lumped Onchocerciasis under Vector control for similar periods for Bench Maji health office.

These

were not detailed and did not reflect key elements of Onchocerciasis control. Plans were essentially the same without targeting areas of specific needs.

ls there a practice of integration?

Plans did not reflect any form of integration, despite the fact that vehicles and office equipment are utilized in an integrated manner.

lf planning and implementation of CDTI is not part of the overall year plan:

.

Why is this?

Planning for CDTI activities was not taken seriously from inception of the project because it had been perceived as an NGDO project.

.

Which steps are being taken to improve the situation?

Analysis

.

When writing the report you have to summarise:

*

The evidence about how well this indicator is being achieved.

i

Reasons for poor performance (if any); steps being taken to improve it; and how this is likely to affect sustainabilitv.

.

Your overall iudqement is this indicator for sustainabilitv beino achieved?

Fully I ^Highly I Moderately I Sfghty I

^{Not at}

all I

Not applicable

a

-

^national^level

-

(4)

1.2

Check whether

all partners

(government, UN agencles, NGDOs etc.) are

meaningfully involved

in the

overall

This indicator assesses whether the programme

is functioning

effectively

-

if each partner is clear about its role, this is good for sustainability.

Gharacteristics

of

the indicator

Sources

of information

a.

All partners should contribute to the routine planning of a project.

b.

Partners should be clear about their own roles, and those

ofthe

other partners.

Examination of

*

Plans: yearly, quarterly, monthly etc.

*

Minutes of NOTF meetings.

lnterview with senior staff at this level:

' ^P0ect

directors and their superiors.

'

NOC and persons in MoH superior to her/ him.

.

NOTF members.

.

Other stakeholders: relevant UN aqencies. NGDOs Describe the present situation:

NGDO partner plans together with zonal offie and Woredas yearly. Some partners particularly Zonal is not yet clear of its role in regards to CDTI activities in Onchocerciasis control. There were no minutes, no lf partners are not meaningfully involved in planning:

.

Why is this?

Poor understanding and insufficient awareness of CDTI Strategy.

. \Mich

steps are being taken to improve the situation? Planning to embark on HSAM activity to the Policy makers at the zone.

Analvsis

*

Your overall iudqement: is this indicator for sustainabilitv beinq achieved?

Fully I Highly I ilodffaildy I ^Slightly I

^{Not at}

all _I

Not applicable

-

national level

-

^September2004 version

(5)

Ihis

assesses whether the programme is ^',and that management has begun to APOG

fundino

is

withdrawn.

take

ownerchip

of it and can mobilise the resources

it

needs.

Characteristics

of

the indicator

Sources

of lnformation

a.

Members of NOTF as well as project managers have made plans for this period, which will enhance

programme sustainability. This planning should include:

identifying resource gaps; strategies to cut expenditure;

and strategies to find dependable sources of resources.

b.

There should be written evidence that such planning has taken place.

c.

There should be evidence that the plans are being successfullv imolemented.

.

Examination of the written sustainable plans.

.

* P0ect

directors and their superiors.

.

NOC and persons in MoH superior to her/ him.

*

NOTF members.

*

Senior NGDO staff.

FI

There are

no

Sustainabihty plans yet at Zonal prqect office nor at the Zonal Health office. Zonal Director for Health has no specific plans for counterpaft funding from Government, since the amount allocated for health is not

much

Feels Government is poor therefore

the

suppofting partners should continue funding CDTI. Cunently NGDO paftner has no Post APOC plans yet but believes they will continue with

technical supporl for Bench

Maji

CDTI propcts.

lf there has been little or no planning for sustainability:

.

Why is this?

Despite being informed by NGDO partner, Zone project office has not yet deemed it necessary yet.

.

None

Analysis

.

I

*

Reasons for poor performance (if any); steps being taken to improve it; and how this is likely to affect sustai nabilitv.

.

Your

overalljudgement

is this indicator for sustainability being achieved?

Fully I ^Highly I Moderately I ^Slightly I ltbtdC lNotapplicable

-

^national^level

-

(6)

This indicator assesses whether the programme is functioning

Characteristics

of the

indicator

Sources

of informatlon

a.

Staff should provide

o

A written work plan

o

Plan shows how implementation of activities in an integrated manner will be achieved

b.

Staff combines two or more tasks on a single trip:

*

Monitoring / supervision for CDTI.

*

Training for CDTI.

*

HSAM.

*

Fetching records.

*

Delivering Mectizan.

b.

Staff combines CDTI activities with those of other programmes, where this is relevant (e.9. supervision, lrainino. HSAM. Mectizan deliverv).

.

Examination of documents: trip authorisations, log books, trip reports etc.

.

lnterviews with:

*

Staff from this level (managers, administrators, drivers etc.).

*

Staff from the next level below.

There were no written plans for CDTI lntegration with other PHC activities at the Zonal level. All tasks carried out by staff are purely Oncho activities just like other health programmes carry on with their own, however equipment and transport are utilized in an integrated manner. Training, mectizan deliveries or returns of reports are solely done on its own.There are no trip

repo(

of integrated activities within the field.

Describe the situation the year before:

Same as above

lf integration between support activities is poor:

.

Why is this?

'Because we are poor in keeping note we have it in our heads says the Zonal oncho Coordinator.

.

Analysis

.

*

Reasons for poor performance (if any); steps being taken to improve it; and how this is likely to affect sustainability.

Examine the trend in the integration of activities

-

is it becoming more common?

Your overalljudgement: is this indicator for sustainability being achieved?

Fully I ^Higtrty I Moderately I ^Sfghry |

^{Not at}

all _I

Not applicable Sustainability evaluation instrument no.1

-

national level

-

^September^2OO4^version

(7)

I

ffi{,MMm }Wir_'}*#,i

3.1

Check whether NOCT/ NOTF/

prolect leadee

are

exercising effective

leadership

for the

at

their

level.

This indicator assesses whether this leadership is functioning

etfectively,

and has taken ownerchip

of

the programme.

Characteristics

of

the lndicator

Sources

of information

a.

The leaders are fully aware of the progress, successes and problems of the projecU projects for which they are responsible.

b.

The leadership is up to date with its reports and targeted activities.

c.

The leadership delegates responsibilities appropriately, to colleagues at this and lower levels.

d.

The leadership has a sound and collegialworking relationship with junior colleagues, and keeps them fully informed.

e.

NOTF meets regularly and its members are fully aware of current issues in the CDTI programme.

.

lnspection

ot

'

Year plans.

.

Reports.

'

NOTF minutes.

.

*

APOC HQ managers.

'

^Staff^atproject level.

'

NOC, her/ his superiors in MoH.

'

NOTF members.

'

Senior NGDO staff.

*

DislricJ/ I GA staff

There is aZonal Focal person in charge of the

project.

He is aware of the project to some extent and is able to handle some problems in the field,while the Zonal Health Dept is not fully

aware

of its

responsibility to its CDTI project.

Leadership is not up to date with reports and targeted activities. No specific training or

monitoring/supervisory

reports

are available however there was a 20031 2004 perfomance report for the

project.

Leadership

of

Bench Maji has been subject to a lot of politics and so has accounted for poor documentation of reports.

Leadership delegates responsibilities to his colleagues at this level particularly during monitoring and supervision. NOTF/Regional/NGDO partner guide the zonal level from time to time.

There are specific NGDO quarterly reports for 2003,

lf leadership at this level is not effective:

.

Why is this?

Leadership

alZonal

level has not fully understood it role in CDTI strategy for Bench Maji.

.

NGDO has spoken to them on the issue of leadership and has been monrtoring them qurte closdly.

.

*

_Reasonsfor poor performance (if any); steps being taken to improve it; and how this is likely to affecl Your overall judgement: is this indicator

Susteinability evaluation instrument no.1

-

national level

-

(8)

4.1

Check whether data and records

concerning

CDTI

activities

are being

effectively

handled

this

level.

effectively.

Characteristics

of

the indicator

Sources

of information

a.

lmportant records are readily available at this level:

treatment summary records, inventories of equipment, financial records, periodic reports (e.9. annual technical reports).

b.

The records are of good quality: the content is clear and seems accurate/ trustworthv.

.

Examination of the documents mentioned, and their source

documents (e.9. districU LGA reports).

.

lnterviews with:

*

Managers and staff at this level.

'

Staff at the districU LGA level.

-

national level

-

!/

There were available treatment summary reports(2003-2005), inventories of equipment, proposal requests for release of funds for CDTI activities for (2003-2005),NGDO

annual

plan reports. Content

of

reports are clear

Only reports from the zone comprising mainly of treatment reports generated from the Woreda levels are sent to direct

to

^NOTF/MOHat Addis. There were no finacial records and nor annual periodic

repofts.

lf records are not being handled properly:

.

Why is this?

Unaware of the importance of report writing.

.

Analysis

a When writing the report you have to summarise:

*

Reasons for poor performance (if any); steps being taken to improve it; and how this is likely to affect suslainabilitv.

.

Your overall judgement: is this indicator for sustainability being achieved?

Fully I Hbhry | Moderately I ^Slightly |

^ruot^at

att _I

Not appticabte

(9)

efficiently.

Characteristlcs

of

the indicator

Sources

of information

a.

Staff members at this level should routinely only

supervise the level immediately below them. Staff should nof supervise the FLHF or community levels. 'Spot checks' may however be done from time to time.

b.

Staff members at this level should have empowered staff members at the level below them to supervise activities at lheir own level, as well as levels furlher down.

.

Examination of:

*

Supervisory checklists, plans and reports.

*

Visitor's books at all the levels below this one.

*

Trip authorisations.

.

lnterviews with:

*

_{Staff at}_this_level.

*

Staff at levels below this one.

Routine

monitoing

and supervision has been carried out yearly by the Zonal project _officestaff.This was carried out to all levels( Woreda, FLHF and communftes)in the guidance

of

NGDO partner.

Staff at the zone remain in the Woredas for monitoring and supervision, but move around with the Woreda staff, FLHF officer, and CDDs within the kebeles until after distribution.

They monitor training, census update and the distribution within communilies. Occassionally spot checks are carried out to the inaccessible areas by the woreda staff.

Describe the situation the previous year:

Same as above.

lf staff members are not being used appropriately for monitoring/ supervision:

.

Why is this?

.

Analysis

.

*

Reasons for poor performance (if any); steps being taken to improve it; and how this is likely to affecl suslainabilitv.

.

Fully I ^Highly I Moderately | Sftlily I

^{Not at}

all _I

Not applicable

-

^national^level

-

^September²⁰⁰⁴^version

(10)

manner.

Gharacteristics

of

the indicator

Sources

of infomation

a.

One routine supervision visit per year should be done to the next level (project or districU LGA). lf more visits are carried out the need should be clearly indicated.

b.

Resources for supervision (human, transport etc.) should be efficiently used:

*

Using as few staff members as possible.

*

Planning trips to cut down on distance travelled.

*

Not spending unnecessarily many nights out etc.

c.

Suoervision visits should be thorouoh. usino a checklist.

.

Examination of:

*

Supervisory checklists, plans and reports.

*

Visitor's books at the level immediately below this one.

.

lnterviews with:

*

Staff at this level.

*

Staff at the level.

immediatelv below this one.

There are several monitoring and supervisory visits conducted over a period of 4-6 weeks yearly to all levels by the Zone during distribution. All supervisory and monitoring visits are routine visits not targeted to improving needy aspects. Four zonal project staff in addition to NGDO are involved with supervisory visits and move into the field with all transport available to the project.

Supervision is not integrated. Checklisfs are used. lt is routtnely done and there are no

reports

All superwsory vistts were dependent on APOC funds.

Same as previous years ^.

lf monitoring/ supervision is not being done efficiently:

.

Why is this?

No previous understandng of exact Zonal role

Unaware of the need to target monitoring and supervision according to needs.

.

Nothing yet.

Analvsis

*

.

Examine the trend in monitoring/ supervision activity: is it becoming more efficient?

Your overall

judgement

Fully I Highly I Moderately I Slgttfy I

^{Not at}

all I

Not applicable

-

national level

-

(11)

4.4 Check whether there is a

routine process

of management

of problems

and successes,

which

are Indlcated by the

monitoring

system.

This indicator assesses whether the programme is running

efficiently

and effectively, and whether management is beginning to accept ownerchip of the programme.

Characteristics

of

the indicator

Sources

of information

a.

As soon as problems are identified as a result

of

supervision visits, or from coverage data (i.e. areas with low coverage) the appropriate manager should dealwith them.

b.

Such problems should usually be passed on to the appropriate managers at the next level below to deal with, with the necessary support

-

thus empowering these persons.

c.

Successes should be noted and reported, and appropriate feedback given.

d.

There should be evidence of action taken based on recommendations in the reports of previous monitorino exercises.

.

Examination of the following documents:

*

Plans: yearly, quarterly, monthly etc.

*

Minutes of staff/ planning meetings.

*

Reports of previous monitoring exercises.

*

Letters of commendation.

*

Letters with information and feedback.

.

lnterviews with:

*

Staff at this level.

*

_{Staff at}the levels below this one.

When a problem

such

as refusals and low coverage

are

identified, measures are taken by the Zonal Oncho project Co-ordinator to tackle this in the

feld

and this is even extended to lower levels if he

s

called upon. There is a reward system in place whereby NGDO rewards

peiormance

by giving a Cerlificate. Cerlificates were seen

Verbal reports are the main system of giving feedback reports.

lf the system of managing problems/ sucoesses is weak:

.

Why is this?

.

Analysis

*

.

Your overall

judgement

Fully I ^Highly I ^moOrd*y I ^Slightly I

^{Not at}

all _I

Not applicable

-

^national^level

-

(12)

ll

&..Lltl

⁴

4.1

Check whether

sufficient

Mectlzan

is

belng ordered,

storcd

and

distributed within the

at

this

level.

This indicator assesses whetherthe programme isfunctioning

efficienily,

ifsprocesses are simple, and it is becoming more integrated into the govemment system.

Gharacteristics

of

the indicator

Sources

of lnformation

The Mectizan supply should be controlled within a government system.

This does not have to be the system routinely used for the supply of other drugs.

The system should be effective, uncomplicated and efficient.

This system should use dependable, sustainable resources for its operation. lt is desirable that these resources should be supplied by the government.

The system should supply sufficient Mectizan for the needs of all the oroiects concerned. in qood time.

.

Examination of all Mectizan ordering and stock control documentation at this level.

.

lnterviews with staff at this level

(government, NGDO etc.).

Zone picks mectizan from Addis once information is passed down by the Region from NOTF( once mectizan is cleared by WHO). Mectizan is then stored in the Zonal Drug store from where it is dispensed to the FLHF. Mectizan is thus controlled within the government system. The system is effective,

uncomplicated and efficient. APOC provides

funds

for collection and distribution to the communities.

Mectizan documentation were available however there were no stock cards.

Zonal office sends treatmenUpopulation data to Regional Health Offlce who then place orders through NOTF with the result that excessive amounts of Mectizan are ordered yearly. Zonal Oncho coordinator has no logical way of how Mectizan orders are being made.

Eg in 2003 1 _{, 1}09,000 tablets of mectizan were received , 342,555 tablets were used, 1 029 wasted, thereby giving a balance of 765,416 tablets as left overs. ln 2005, 1183,275 were received, 890,469 were utilized, 2455 were wasted leaving a balance of 290,351.

Mectizan is recieved in good time for instance Mectizan for 2006 is already available at the zonal drug store.

Describe the situation the year before that:

Same as above.

.

Why is this?

.

Analysis

*

_Reasonsfor poor performance (if any); steps being taken to improve it; and how this is likely to affect sustainabilitv.

.

What has been the trend in Mectizan supply?

.

Your overall

judgement

Fully I Hhhly I Moderately I ^Slightly I

^{Not at}

all I

-

national level

-

(13)

ffi,s's

This indicator assesses whether the programme is

Characteristics

of the

indicator

Sources

of information

a.

Staff at this level should routinely only train staff at the level immediately below it.

b.

Staff members at the national level should have empowered the level immediately it to conduct training activities at the levels below it again (i.e. districU LGA or FLHF)

independentlv.

.

Examination of training materials,

planv

programmes, reports:

*

At this level.

*

At the levels below this one.

.

lnterviews with:

*

_{Staff at}this level (the trainers).

*

Staff at the very next level below (the trainees).

*

Staff at

ihe

FLHF level.

Findi

Staff at this level routinely train Woreda staff,FLHF and CDDs. Training manual for CDDs was available but there were no training reports available.

Describe the situation the year before:

Same as above.

lf staff members are not being used efficiently as trainers:

.

Why is this?

.

Analysis

.

*

.

Examine the trend in the way in which staff are being used as trainers:

.

Fully I ^Highly I Moderately I Slglily I

^{Not at}

all _I

Not applicable

-

national level

-

(14)

efficienily

and in

an

integrated manner.

Characteristics

of

the indlcator

Sources

of information

a.

There should be an objective need for each episode of training. This means there should be evidence that staff to be trained lack knowledge and skills to perform the job, _and the training should then focus on this deficiency only.

Repeat training of already skilled staff should not happen.

b.

Resources for training (human, transport etc.) should be efficiently used:

r

Using as few staff members as possible.

*

Using as little time as possible (without sacrificing quality)

*

Choosino the most epst-effeclive site etc.

.

Examination of training materials, plans/ programmes, reports:

*

At this level.

*

At the levels below this one.

.

lnterviews with:

*

Staff at this level (the trainers).

*

Staff at the very next level below (the trainees).

*

_{Staff at}the FLHF level.

Training is routine and not targeted to need. Neither was it imntegrated. I person trains 10 people per training session at the Health center. Training is for 2 days for CDDS, 1 day for Health workers and 1

day for retraining of CDDs. APOC funds were utilized for training Describe the situation the year before:

Same as above

training is not being carried out in an efficient and integrated manner:

.

Why is this?

.

Analysis

I \A/hen writing the report you have to summarise:

*

.

Examine the trend in training activities

-

is it becoming more efficient and integrated?

.

Fully I ^Highly I Moderately I Shn0y I

^{Not at}

all I

Not appticabte

-

national level

-

(15)

ai

manner.

This indicator assesses whether the programme is

efficiently

and

effectively,

and whether managers are taking

ownership

of the programme.

Characteristics

of

the indicator

Sources

of information

a.

Staff members identify situations where decision makers lack information abouU commitment to CDTI, and undertake activities to inform and persuade these persons.

b.

HSAM activities are properly planned. They are only carried out where there is an objective need for them, and not as a matter of routine.

c.

Such activities should only be carried out at the national level, and at times at the level immediately below (but only when staff at that level asks for help).

d.

There is evidence that these HSAM activities have been effective and have led to action.

Examination of HSAM plans/

programmes and reports.

lnterviews with:

*

Staff (programme and management) at this level.

t

Civil authorities at this level.

t

Staff and civil authorities at the next level down.

Describe the present situation (in relation to efficiency and outcome):

There has been no evidence

of

HSAM activity alZonal level. No HSAM activity reports/nor future plans are available despite the fact that proposals for these activity were made for2003-2005.

Describe the situation the year before (in relation to efficiency and outcome):

Same as above.

lf HSAM activities are not being carried out efficiently:

.

Why is this?

No reason given for this.

.

Nothing

Analysis

*

Reasons for poor performance (if any); steps being taken to improve it; and how this is likely to affec{ sustai nabilitv.

.

Examine the trend in HSAM activities

-

is it becoming more efiicient and effective?

.

Your overall

judgement

Fully I ^Highly I Moderately I ^Slightty I HotCd lNotappticabte

-

national level

-

(16)

?ir'r'i,Ifffiffi j+\,*q$SffiWeq'],'"{{tr,W 7.1

Check whether

appropriate amounts

are budgeted

for

planned

onchocerciasis control

activities

at

this

level

efficienily.

Gharacteristics

of

the indlcator

Sources

of lnformation

The costs for each onchocerciasis control activity in the year plan at this level should be clearly spelt out in a budget.

There is evidence of a cost reduction/ containment strategy (e.9.

targeted training, HMAS and monitoring/ supervision; training conducted at the next level below etc).

Projecl managers should have a clear estimate of the funds that will be available to them for onchocerciasis control in the coming year, and should be able to

justiff

this belief.

The total amount budgeted for in the year plan should fallwithin this estimated income.

Examination of the budget documents.

lnterviews with the health service managers at this level.

The budget and estimated income:

.

For this year:

There are no clearly spelt out budget estimates for CDTI activity for this year in the zone.

Estimated incomes are reflected below:

There are no evidence of cost reduction /containment strategy. Training is not targeted, HSAM not carried

out

and M& S is carried out for very long time while in the field. Zonal coordinator is unaware of funds allocated for CDTI activities and has no budget proposal for various CDTI activities for the coming year.

For the previous year:

Same as above.

lf budgeting has been inappropriate:

.

Why is this?

.

Analysis

.

*

Reasons for poor performance (if any); steps being taken to improve it; and how this is likely to affec{ suslainabilitu.

.

Examine the trend in the budgeted amount and the expected income:

.

Fully I ^Highly I Moderately I ^Slightly I

^{Not et}

aI I

Not applicable

-

national level

-

(17)

7.2

Check whether the

government is budgeting

and

dlsburslng sufflclent amounts for onchocerciasis control

yearly. and in qood tlme.

indicator assesses whefher the programme is becoming

integnted,

and whetherthe govemment is accepting ownership of the programme and can mobilise the resources it needs.

Characteristics

of

the indicator

Sources of

lnformation

a.

The relative budgetary contributions of the govemment and other partners to onchocerciasis control should be clearly spelt out.

b.

The amount that the government has budgeted in one or more specific onchcerciasis control budget lines (e.9. current and capital) should be increasing yearly, as a proportion of total expenses. By the end of Year 5 of APOC funding the bulk of onchocerciasis control expenses at this level should be met from government funds; by the end of Year 3 at least half of ^it.

c.

The amounts actually disbursed from such budget lines should be increasing yearly, as a proportion of total expenses. (Note that actual disbursement is more important than budgeting, and is a real sion of oolitical commitment).

.

Examination of:

*

Budget documents (government, NGDO etc.)

t

Records of disbursement and expenditure (ledgers, orders, approvals for expenditure etc.)

.

lnterviews with managers at this level (government, NGDO etc.).

The budget and disbursements:

.

For this year

Government has not yet fulfilled its own role in regards to counterpart funding of the CDTI programme The relative budgetary proposed commitments (from letters of agreement) are as follows:

Year APOC

^NGDO

Partner

Government(Regional/Zonal)

2002t3 $283,446 $81,830

^$78,895

2004 $170,440 $73,872

^$106,564

2005 $142,184 $69,143

$120,176

Bench Maji currently in its 3'd year of CDTI programme implementations is still heavily dependent on APOC funds and government has just been providing salaries and office space.

A zonal account ( as requested by APOC) was

only

opened this month January

2006.

Actual disbursement for NGDO partner were seen.

.

For the previous year:

Same as above, there were no government releases apart from salaries.

.

For the year before that:

Same as above, there were no government releases apaft from salaries.

lf the government proportion of expenditure is not increasing proportionately:

.

Why is this?

According to one of the Zonal officals interviewed' we are poor and so external donors should continue funding this programme to help our people'.

.

VVhich steps are being taken to improve the situation?

No steps are being taken yet.

Analysis

*

.

Examine the trend in government budgeting and disbursements:

.

Fully I ^Highly I Moderately | ^S{eh$y I

^{Not at}

all _I

-

^national^level

-

(18)

7.3

Check whether

in

case of a

deficit

between estimated

costs

and the

amount provided

by

the is

belno made

to

meet it.

This indicator assesses whether the govemment is able to mobilise the resources

it

needs, as well as its commitment to

ownership.

Characteristics

of

the indicator

Sources of

information

a.

Project management at this level should be aware of the shortfall, if one exists, and of its size.

b.

Project management should have specific and realistic plans to bridge the shortfall.

c.

lf it is planned that non-government sources of funding are to be used afier APOC funding ends, wriften commitment for this should have been obtained at the highest level in these donor organisations. Projects in Year 3 of APOC funding should also be well on the way to achieving such commitment.

.

lnspection

ot

*

The budget documents.

(government, NGDO etc.).

*

Records of expenditure (ledgers, orders, approvals for expenditure etc.).

*

Letters of agreement.

.

Zonal Management are unaware of any existing shortfalls, nor of its magnitude. There are no plans to bridge the shortfall. There were no records of budget documents nor expenditures made. Currently because of poor retirement APOC funds have not yet been released for the current year and NGDO had to wade in to help the project.

Zonal Management are unaware of any existing shortfalls, nor of its magnitude. There are no plans to bridge the shortfall.

There

were no records of budget documents nor expenditures made.

lfffi

.

Why is this?

Becaue they are unaware

of

source of funding for trheir CDTI projects

.

Analvsis

.

*

Reasons for poor performance (if any); steps being taken to improve it; and how this is likely to affeci sustainabilitv.

.

Examine the trend in shortfall and how it is to be supplemented:

.

Your overall

judgement

Fully I ^Highly I Moderately I Slgttty I

^{Not at}

all _I

Not applicable

-

national level

-

(19)

7.4

Check whether

funds disbursed for onchocerciasis control from

the budget at

thls

leyel are This indicator assesses whether the programme is functioning

efficiently.

Characteristics

of

the indicator Sources

of

lnformation

a.

The budget holder should be using a control system with the following elements:

*

Approval of each item of expenditure.

*

Allocation of expenditure against specific budget headings.

*

Regular calculation of residual amounts under budget headings.

b.

All the funds released vearlv should be sDent as budoeted.

.

lnspection of:

*

The budget documents (government, NGDO etc.).

*

Financial control records (ledgers, orders, approvals for expenditure etc.).

.

lnterviews with managers at this level (oovernment. NGDO etc.).

Requests proposals for CDTI activities were available but actual approvals of proposals made for the various CDTI activities each year were not available. There were no finacial disbursments reflecting yearly releases, expenditures or regular calculations of residual amounts seen at the zonal level.

However there were queries issued to different woreda health workers requesting retirement of funds given to them for 200312004.

.

Approval of expenditure:

.

Allocation of expenditure:

.

Regular insight into budget line balances:

lf the funds are not being well managed:

. \

/hy is this?

.

Analvsis

i

*

.

Your overall

judgement

Fully I ^Highly I Moderately J ^Slightly I

^iaot

it.f I

Not applicable

-

national level

-

(20)

&J"'}.'milm :r.3gffiffiF$*!

8.1

^{Check if}adequate and

appropriate transport

and

other

material resources are available

for

CDTI

activities

at

this

level.

This indicator assesses whether the programme is functioning effectively, and whether it is able to mobilise the resources

it

needs.

Characteristics

of the

indicator

Sources

of information

a.

There are adequate numbers of functional vehicles available for necessary CDTI activities.

b.

The vehicles are appropriate for the purpose they are intended to fulfil

-

tough but not luxurious.

c.

There is sufficient office equipment available, in working order: computers, printers, photocopiers

-

also stationery and materials for training and HSAM.

d.

The running costs for these vehicles and equipment are met from deoendable. sustainable souroes.

.

lnspection

ot

*

Each vehicle in the pool, each piece of

equipment

its source; its functional status.

*

Training materials and stationery stocks.

.

Describe the availability/ suitability/ functionality of the present vehicles, equipment and materials, considering the work still to be done in the coming 5-10 years:

Vehicle is in fairly good order but not suitable for the job. Equipments are available and suitable for the job for the next coming 2-3 years. Will definitely require new vehicle and equipment for the post Post APOC

era.

IEC material and training manuals

are

currently grossly inadequate. Running

& maintenance of the vehicles are bv APOC.

Toyota double Cabin Still working but unsuitable b/c it uses petrol which is scarce in Bench Maji.( Diesel Double cabin should be most aooriooriate next

5 still working, 3 non functioning; suitable for the

ls workinq, suitable & enouoh for the iob..

ls workinq. suitable & enouqh for the iob.

ls workinq, suitable & enough for the i

ls workinq. suitable & enouqh for the ls workinq, suitable & enouqh for the iob.

ls workino, suitable & enouqh for the

Non funclional from onset because of fluctuation in

'

APOC, MoH, NGDO, other (specify) ** ls it working? ls there enough of it for the job? ls it

lf transport, equipment and materials are inadequate and/or funded from sources which are not dependable:

Why is this?

All have been supplied by APOC apart from the IEC materials.

.

-

national level

-

(21)

t

*

Reasons for poor performance (if any); steps being taken to improve it; and how this is likely to

.

Your overall iudgement: is this indicator for sustainability being achieved?

-

national level

-

(22)

8.2

Check if

transport

and

other

material

resources in

use at

this

Ievel are adequately and maintained.

This indicator assesses whether the programme is functioning effectively and

efficiently

Characteristics

of

the indicator

Sources of

information

a.

There is a routine maintenance schedule for each vehicle, which is adhered to and recorded. This includes weekly driver maintenance, scheduled garage servicing, and replacement of worn tyres.

b.

Equipment such as photocopiers and generators is regularly maintained according to a schedule, and this is recorded.

c.

Staff members have ways of coping when vehicles break down or are not available, so that CDTI activities are not disrupted.

d.

The costs for vehicle and equipment maintenance and repair are met ftom dependable/ sustainable sources.

e.

Reoairs to vehicles and eouioment are raoidlv and efficientlv done.

.

lnspection

ot

*

Vehicle and equipment maintenance

schedules.

*

Vehicle and equipment maintenance records.

.

lnterviews with managers at this level (government, NGDO etc.), drivers.

Routine maintenance schedules were not available, nor were log books, but there was a system by which vehicle movement were known. There were no schedule for mainatenanc of vehicles and other equipments.

Vehicles are used in an integrated manner and so staff have a way of coping.

APOC is responsible for repair and maintenance of vehicle and equipment, but records of such repairs are not documented.

Repairs to vehicle and equipment are not efficiently nor rapidly done for rnstance the 3 broken down motor bikes and

fax

equipment have not yet been repaired because, Zonal co-ordinator has not yet deemed it necessary. Nor has he put up a request or proposed budget for its repair.

lf the vehicles and equipment are not being well maintained, and/ or funded from sources which are not dependable:

.

Why is this?

APOC has been the sole

source

of funding.

.

Nothing yet.

.

*

.

Your overall

judgement

-

national level

-

(23)

Gharacterlstics

of

the indicator

^Sources

of lnformation

a.

Transport is used at this level, and to undertake suppoft activities at the next level. lt should not be used for CDTI implementation activities at lower levels.

b.

The use of transport is properly controlled:

r

Trips made for CDTI purposes should be properly authorised in writing by the relevant official.

r

_Eachtrip undertaken for CDTI purposes should be recorded in a log book.

*

Trip authorities and log book entries are regularly reconciled, and aciion laken if there are discreoancies.

.

lnspection of vehicle control documents:

*

Copies of trip authorities (also noting destination and purpose).

*

Log books.

.

Transport is used in an integrated fashion, however in most cases it gets to the lower levels(FlHF and Kebeles) during monitoring/ supervision and distribution

There are no trip documents,log books nor relevant official document to show vehicle movement.

lf the transport is not being well managed:

.

Why is this?

.

Analvsis

.

*

Reasons for poor performancc (if any); steps being taken to improve it; and how this is likely to affect sustai nabilitv.

.

Your overall

judgement

Fully I ^Highly I Moderately I ^Slightly I ^NotetC lNotapplicable

-

national level

-

(24)

8.4

^{Check if}there are

appropriate

and

realistic

plans

for the

replacement of

transport

and

other material

resources, when APOC

support

comes

to

an end.

This indicator assesses whether the programme managers are taking ownership of the programme, and are able to find resources for it.

Characteristics

of

the indicator

Sources

of information

a.

Management should know that replacements will be needed before the end of the programme, and have specific, realistic plans to meet the need at that time.

b.

lt should be planned that the government will:

*

Provide replacements for vehicles and equipment.

+

Maintain existing vehicles and equipment.

*

Provide stationery and materials for training and HSAM.

c.

lf it is planned that replacement will be from non-government sources, written commitment for this should have been obtained at the highest level in these donor organisations (end of Year 5), or negotiations should have started (end of Year 3).

.

lnspection of lefters of agreement.

.

lnterviews with:

*

Programme managers at this level (government, NGDO etc.).

*

High-ranking Ministry officials and other decision makers at this level.

There are no realistic

plans

yet for replacement of vehicle, and equipment when APOC comes to an end.

lf the plans for replacing vehicles, equipment and materials are unsatisfactory:

.

Why is this?

Unaware of the need for such replacement plans feels donors should carry on.

.

Nothing yet.

Analysis

.

*

Reasons for poor performancc (if any); steps being taken to improve it; and how this is likely to affect sustainabilitv.

Fully I ^Highly I Moderately I ^Slightly I

^{Not at}

all I

Not applicable

-

national level

-

(25)

ffi *{ffi ffi

^;

^t' l.ffi _ffi,ffi ^,ffi $'r$,,*e&iYl

9.1

Check whether

staff

at

this

level is

skilled,

knowledgeable and

committed,

regarding

the

of CDTI

in its

area

of

This indicator assesses whether the programme has been able to develop sufficient resources for itseff.

Characterlstics

of

the indlcator

Sources

of lnformatlon

The number of staff members in the government health service at this level should be appropriate to the task in hand: not too many or too few.

Team members should have enough knowledge and skill to undertake all the key CDTI activities themselves, without help:

*

_Planning

*

Report writing

*

Training and HSAM

*

Monitoring/supervision

There should be evidence that the team is commifted to the success of the programme (from the evidence of the partners, as well as workers at the next level below: from wriften reoorts and timetables).

*

Data management

*

Computer skills

*

Mectizan ordering/ distribution

lnspection of:

*

Staff files.

*

Training reports and timetables.

lnterviews with

*

Managers and other staff at this level (government, NGDO etc.).

*

Staff at the next level below.

lf the staff at this level lack skills and commitment:

.

Why is this?

Staff are very committed.

.

Number of

pergg4gylq

show that they are committed to their work, and perform it well :

No. of persons qualified in this area

.

*

.

Your overall

judgement

Not applicable

-

national level

-

(26)

9.2

Check whether

staff

at

this

level is stable, and

whether provision

is made

for passing

on CDTI

skllls when

a

trained

person moves away.

This indicator assesses whether the programme has been are able to maintain ifs resources.

Characteristics

of

the Indicator

Sources

of information

a.

Staff at this level should remain in one post for at least five years.

b.

There should be immediate orientation (in CDTI) of new, unskilled project staff members.

c.

During their training new professional health workers in the country (doctors, nurses etc.) are provided with up-to-date knowledge and skills in the field of CDTI.

.

lnspection of staff files.

.

The table in 9.1.

.

lnterviews with:

*

Managers and other staff at this level.

*

The person in the Ministry responsible for human resouroe develonmanl

Staff are relatively stable and stay in a post for a period of upwards of 5-15

years.

New staff are rarely oriented/ updated on the CDTI program.

Describe the situation two years ago:

Staff

are relatively stable and stay in a post for a period of upwards of 5-15

years. Staff

on ground then were oriented and exhibited reasonable knowledge of the CDTI approach.

Describe the situation when APOC funding started being given:

During inception of CDTI strategy, initial core staff were oriented on CDTI strategy and exhibited reasonable of its activity.

lf the staff is not stable, and new staff is not being trained:

.

Why is this?

.

Analvsis

I \A/hen writing the report you have to summarise:

*

Reasons for poor performance (if any); steps being taken to improve it; and how this is likely to affec{ sustai nability.

.

What is the trend in the number and quality of staff?

.

Your overall judgement: _isthis indicator for sustainability being achieved?

Fully I ^Highly I lloOcracly I ^Slightly I

^{Not at}

all _I

Not applicable

-

national level

-

(27)

{$i.;-t, tis+sx,ll$lgs,g-qdrJffit}H'iFsr+"# It;lr;*,$iw&

10.1

Check from records whether

all projects

in the

country/state/province

have a

satisfactory

rate

is

effective

-

^if^therates are poor the projects are clearly struggling and /ess sustain able.

NOTE:

The denominator in the calculation of GCR is the total number of projects in the

country/state/provi nce

Sources

of information

Characteristics

of

the indicator

a.

The GCR for all the projects in the

country/state/province should approach 1 00%

b.

This GCR rate should be stable or increasing.

.

lnspection

ot

*

Distribution reports and statistics at national/ project level, for the past 3 years.

*

National REMO records.

.

lnterviews with staff at:

t

This level.

*

The next level below.

The GCR in the country/ state/province

.

At the last distribution:

100%

.

The year before:

100o/o

.

The year before that:

100o/o

lf the

GcR

is poor,

.

Why is this?

.

Which steps are being taken to improve the situation? (if such steps are already being taken that is good for sustainability)

Analysis

When writing the report you _haveto summarise lhe reasons for poor performance (if any); steps beino taken

lo

imorove it: and how this is likelv to affec't sustainabilitv.

.

What is the trend in geographical coverage?

Your overall iudoement: is this indicator of sustainabilitv beino achieved?

Futy

({tCRfor0rcoors1t

pdcerrbSt0tSclmnhD

^Highly(rate is 90- 95% for country's

nroieclsl

Moderately (rate is 80-

8qo/^\

Slightly ^(rate isTO-79%l

Negligibly (rate is <70%)

-

national level

-

(28)

I therapeutic

coverage rate

(TCR).

I

is

effective

-

^if^therates are poor the projects are clearly struggling, and /ess sustainable.

NOTE:

.

The of in the

Characteristics

of

the indicator

Sources

of information

All projects in the country/state/province

should have a TCR of 65% or higher.

These rates should be stable or increasing.

a.

b.

lnspection of:

*

Distribution reports and statistics at national/ project level, for the past 3 years.

'

National REMO records.

lnterviews with staff at:

*

This level.

*

The next level below.

The TCR in the country/state/province:

.

At the last distribution:

2005

84.9o/o

.

The year before:

2004

63.8%

There was this dip b/c funds were not released ontime for CDTI activities in one woreda

.

The year before that:

2003

75.1o/o lf the TCR is poor:

r

\A/hy is this?

.

Analysis

When writing the report you have to summarise the reasons for poor performance (if any); steps beinq taken to improve it: and how this is likelv to affect sustainabilitv.

What is the trend in therapeutic coverage?

Your overall iudoement: is this indicator of sustainabilitv beino achieved?

Frrlu Glr.

stlIp

^TCR

brtr

cou@r p{cctr

^{b :EOlt}^-Cebb or

lncmdnol

Highly (the average TCR for the country's projects is

60-64%)

Moderately (the TCR is 55-59%)

Slightly (the TCR is

50-540/")

Negligibly (the TCR is<5Ool")

-

national level

-

(Instruments Used for Evaluation of

Appendix 9: Instruments l-4