Background
Key performance indicator data is used to calculate the performance envelope award
To promote accountability, the Alberta post-secondary education sector, working in concert with the Department of Advanced Education and Career Development, has
developed a set of key performance indicators (KPIs).
Beginning in 1997-98, certain KPIs formed the basis for the permanent, annual distribution of approximately $15 million from the Department’s performance funding envelope.
In order to obtain assurance that institutional systems used to generate key performance indicators were functioning appropriately, the Department requested my staff to perform specified audit procedures on the data submitted by
institutions for the 1998-99 KPI performance envelope funding.
The sector is breaking new ground and is encouraged to continue
The KPI reporting process and performance based funding for the sector is new. The sector is breaking new ground in linking funding to performance. It has established KPIs to improve performance by rewarding success. The
Department has created a sector-wide committee to deal with reporting issues that arise. I strongly encourage the sector to continue its endeavors to enhance the relevancy of
performance measurement.
The process is not fully developed, and opportunities for improvement exist
Since the KPI reporting process is new, the underlying reporting principles and practices and related systems are not yet fully developed. Many of the related systems require manual processes and are not integrated with the institutions’
management information systems. There is an opportunity for the sector to improve the reliability of the information that is provided, and to improve the efficiency of the process, through automation, development and implementation of appropriate control processes and through integration with existing management information systems.
Credit full load equivalent students, graduate employment rate and graduate satisfaction KPIs are three of the key
measures of an institution’s performance and also provide the majority of the weighting in the calculation of the
performance envelope award. The following
recommendation addresses issues regarding these KPIs.
KPI Reliability Recommendation No. 11
It is recommended that the Department of Learning work with the public post-secondary education institutions to improve the reliability of KPIs for credit full load equivalent student, graduate employment rate and graduate satisfaction.
Administration of surveys and accuracy and reliability of survey data could be improved
Graduate employment rate and graduate satisfaction are calculated from survey data. At many institutions, controls over data collection procedures need to be established and data collected needs to be reconciled to registrar records and survey populations. In some cases, staff conducting surveys are not adequately trained and are not sufficiently
independent. Furthermore, at some institutions, data
compilation and input is not double-checked. Consequently, some survey results are unreliable and a number of
compilations, processing and inputting errors remain undetected.
Accuracy of full load equivalent data can be improved through reconciliations between institutional and Departmental systems
For the credit full load equivalent student KPI, some institutions transfer student information manually from the registration system to computer files that are uploaded onto the Department’s Common Information System (CIS). Many institutions do not have procedures to reconcile CIS
information with institutional records, including transcripts and program calendar information. Furthermore, when such procedures are in place, the results are not reviewed.
Consequently, transcription errors occur and compilation, processing and input errors remain undetected.
KPI information can be more reliable with proper segregation of duties, proper documentation and use of reconciliations
To help ensure it has correct information upon which to base the performance awards, the Department should work with the institutions to develop a method that will provide staff with training on appropriate internal control processes. Such processes would include proper segregation of duties, proper control of survey forms, independent verification,
reconciliations, and proper review and approval of key performance indicator data.
The Department and institutions could also reduce the risk of error and improve efficiency by exploring ways to facilitate an automated transfer of FLE data from institutional
registration systems.
Business plans In my 1997-98 annual report (page 37), I recommended that the Department, working with public post-secondary
education institutions, develop strategies to ensure that institution business plans contain the planning information necessary to meet the needs of the institutions and the Department.
Revised guidelines have resulted in significant improvements in the institutional business plans
I am pleased to report that the revised Business Plan and Annual Report Guidelines issued by the Department in 1997-98 and the discussions held between the Department and the institutions have resulted in significant
improvements in the institutional business plans. In general, the 1998-99 three-year business plans are providing more complete planning information regarding enrollment projections, facility needs and funding strategies.
Additionally, initiatives described in the narrative portion of the business plans are more consistently reflected in the associated financial plans.
Deferred Maintenance Recommendation No. 12
It is recommended that the Department of Learning and the public post-secondary education institutions continue to improve the system to manage the infrastructure by evaluating the overall progress made towards addressing the critical health and safety risks relating to deferred maintenance.
Last year, my staff recommended that the Department and the public post-secondary education institutions improve the system to manage the infrastructure by evaluating the risks relating to unfunded deferred maintenance. I am pleased to report the Department has agreed with the recommendation.
Sufficient information is not available to determine the overall progress made towards addressing the critical health and safety risks relating to the deferred maintenance
The institutions are responsible for capital asset
maintenance. Since 1997-98, a total of $70 million has been granted through the Infrastructure Renewal Envelope to help institutions address the $362 million backlog of deferred maintenance identified through a Departmental study in 1997. To date the institutions have submitted Infrastructure Renewal Funding plans and a first series of summary accountability reports listing the initiatives undertaken with the first allotment of Infrastructure funding. A second round of summary reports is expected in the fall. Neither the plans nor the reports, however, indicate which projects address critical health and safety concerns. Such concerns include
for example, elevator and structural safety and fire
suppression systems. Therefore, although the Department conducts annual project site visits, sufficient information is not available to determine the overall progress made towards addressing the critical health and safety risks relating to deferred maintenance.
My staff will continue to monitor progress in managing the sector’s unfunded deferred maintenance
The institutions are expected to complete their Infrastructure grant projects within three years. At the end of this period, the Department is considering conducting another
comprehensive facility study to reassess the level of deferred maintenance within the sector and to evaluate the progress made with the $105 million Infrastructure grant program.
Future funding strategies to address the remaining deferred maintenance and the accumulating ongoing operational maintenance are also currently under review. My staff will continue to monitor the departmental and institutional progress in managing the sector’s unfunded deferred maintenance.
Long Term Capital Planning Recommendation No. 13
It is recommended that the Department of Learning, working with the public post-secondary education institutions, continue to develop a long-range capital planning system for Ministry infrastructure.
In my 1997-98 annual report (page 41), I recommended that the Department, working with the public post-secondary education institutions, develop a long-range capital planning system for post-secondary institutional infrastructure. The Department has agreed with the recommendation and has begun to take action.
The Department is currently working with all departments through a government-wide Capital Investment Planning Committee to develop a capital overview and a capital management system
The Department is currently working with all departments through a government-wide Capital Investment Planning Committee to develop a capital overview and establish the basic elements of a capital management system that will serve as an outline for each department. One of the primary objectives of this initiative is for each department to prepare a long-term strategic capital plan by March 2000. To begin the process, the Department conducted a review of the condition and the estimated replacement cost for each facility within the sector. The Department has also completed a study of the student capacity available in the system given the current capital structure.
A significant amount of detail remains to be collected from the institutions
I have repeated this recommendation as I believe that a long-range capital planning system for Ministry
infrastructure is critical to the long-term success of the sector and a significant amount of detail remains to be collected from the institutions. To meet the requirements of the government-wide initiative, the institutions must prioritize and submit their long-term capital needs based on enrollment forecasts and proposed sources of funding to meet these requirements. The capital needs projections should also consider the impact of future changes to delivery methods.
Once the information has been gathered, the Department must prepare a listing of prioritized projects within the sector and thereby determine the sector’s overall capital needs.
Such information will also assist the Department to justify the sector needs at a government-wide level.