MHAwards Final Project Evaluations Report ENG

(1)

MHCC NATIONAL MENTAL HEALTH AWARDS PROJECT EVALUATIONS

Final Report Prepared for the Mental Health Commission of Canada

March 1, 2013

Prepared By:

Ference Weicker & Company 550-475 West Georgia Street Vancouver, BC V6B 4M9 (604) 688-2424

(2)

Introduction

1 Purpose of the Evaluations 1 Method of Study

1 Structure of the Report

Description of the Projects and Highlights 2 Description of the Projects

3 Highlights from the Evaluation Findings

Project Evaluation Reports

Capital District Mental Health - Consumer, Family and Provider Collaboration in Treatment, Program and Policy Planning, and Evaluation

Mental Health Works

Canadian Coalition for Seniors’ Mental Health - National Guidelines for Seniors' Mental Health: From Evidence to Implementation

The Strongest Families Institute

Schizophrenia Society of Canada - Understanding the link between cannabis and psychosis: An awareness strategy

(3)

FERENCE WEICKER & COMPANY

I. INTRODUCTION A. PURPOSE OF THE EVALUATIONS

The objective of this study was to conduct evaluations of initiatives recognized as part of the 5^th Anniversary National Mental Health Awards of the Mental Health Commission of Canada (MHCC). Initiatives that were recognized at the Awards were provided with an evaluation opportunity as part of their Awards package. Five initiatives chose to participate in the evaluation process, including:

 “Consumer, Family and Provider Collaboration in Treatment, Program and Policy Planning, and Evaluation” of the Capital District Mental Health Program in Nova Scotia;

 “Mental Health Works” of the Canadian Mental Health Association-Ontario Division;

 “National Guidelines for Seniors' Mental Health: From Evidence to Implementation” of the Canadian Coalition for Seniors’ Mental Health;

 “Strongest Families Institute” in Nova Scotia; and

 “Understanding the link between cannabis and psychosis: An awareness strategy for youth” of the Schizophrenia Society of Canada.

The overall purpose of each evaluation was to determine the initiative’s impacts and success, to highlight its innovative approach and design, and to support the initiative’s organization and the MHCC in sharing best practices related to the initiative.

B. METHOD OF STUDY

The methodology undertaken to conduct the evaluation involved a number of steps, including:

 Conducting a preliminary review of documents (e.g., Awards application forms);

 Undertaking project launch meetings with each initiative’s project manager;

 Developing an evaluation methodology report including a list of evaluation issues and questions,¹ and an interview guide, with additional tailored questions for each initiative;

 Conducting a detailed review of relevant documents that provide information on the initiatives’

objectives, activities, performance, achievements and other impacts (e.g., program and policy documents, evaluation reports, and other literature);²

 Undertaking interviews with 5 to 8 stakeholders per initiative (31 in total) including project managers, and other stakeholders involved in the design and delivery of the initiatives (e.g., service providers, policy makers, researchers, family members, consumers, youth, among others);

 Undertaking an analysis of all the quantitative and qualitative data obtained from the different lines of evidence and summarizing the data in response to each evaluation question; and

 Preparing draft evaluation reports for each initiative which summarize all the major findings and conclusions;

 Submitting the draft evaluation reports to each project manager to review and provide feedback; and

 Combining each evaluation report into this document.

C. STRUCTURE OF THE REPORT

This document contains two chapters, including this introduction (Chapter I). Chapter II provides a description of the projects and highlights from the evaluation findings. The full project evaluation reports for each initiative are included in the appendices.

1 The evaluation questions are presented in Appendix I of each evaluation report.

2 The list of documents reviewed is presented in Appendix II of each evaluation report.

(4)

II. PROFILE OF THE PROJECTS AND HIGHLIGHTS

This chapter provides a brief description of the five projects and highlights from the evaluation findings.

A. DESCRIPTION OF THE PROJECTS

The following table provides a description of the projects that were evaluated and the MHCC Award categories in which they were recognized. The projects included initiatives to improve collaboration in mental health care (i.e., among consumers, family members and providers), workplace mental health, seniors’ mental health care, access to mental health services for families, and to improve awareness and understanding of the link between cannabis use and psychosis from the perspective of youth.

Description of the Projects

Project Description Award Category

Consumer, Family and Provider Collaboration in Treatment, Program and

Policy Planning, and Evaluation

(Capital District Mental Health Program, Nova Scotia)

The Consumer, Family and Provider Collaboration Initiative supports consumers, families and providers to effectively work together to support recovery and ultimately improve the mental health care system as a whole. Within this model, consumers, families and providers are all part of treatment planning, program and policy planning, implementation, and evaluation which leads to improved outcomes for all.

Community Capacity (Winner)

Mental Health Works (Canadian Mental Health Association-Ontario Division)

Mental Health Works is a national initiative of CMHA-ON and is dedicated to advancing the field of workplace mental health through skills enhancement training, awareness education and stigma reduction efforts.

Workplace Initiative (Winner)

National Guidelines for Seniors' Mental Health: From

Evidence to Implementation (Canadian Coalition for Seniors’ Mental Health)

In 2005, CCSMH was funded by the Public Health Agency of Canada to create Canada’s first interdisciplinary evidence- based guidelines in seniors’ mental health. The four guidelines focused on key mental health issues (depression, delirium, mental health issues in long-term care homes, and suicide prevention). Since their release in 2006, the guidelines have been widely disseminated, piloted across seven sites, and translated into booklets, toolkits, and quick reference pocket cards.

Partnership (Honourable

Mention)

Strongest Families Institute

The Strongest Families Institute is a not for profit organization based in Halifax, Nova Scotia. Strongest Families is a proven, evidence-based distance mental health intervention program designed to help families in the comfort and privacy of their home at convenient times.

Social Innovation (Winner)

Understanding the link between cannabis and psychosis: An awareness

strategy for youth (Schizophrenia Society of

Canada)

The Cannabis and Psychosis Awareness Project is a national initiative of the Schizophrenia Society of Canada. Its aim was to increase awareness and understanding of the relationship between cannabis use and psychosis from the perspective of youth.

Research (Winner)

(5)

B. HIGHLIGHTS FROM THE EVALUTION FINDINGS

There were a number of similar themes across the five projects that were evaluated, particularly with respect to best practices and innovative approaches in their design and delivery. Key themes included:

 Partnerships

 The Seniors’ Guidelines Initiative had a strong framework for collaboration and partnership. Key team members were involved from the beginning and others were invited to participate.

Members were engaged, committed, and leaders in their field. Central to the success of the collaboration was the planning and structure that was established (e.g., outlining roles, responsibilities, deadlines, and expectations).

 The delivery model for Mental Health Works is based on partnerships and licensing the training content to other delivery organizations, which allows for greater expansion and reach of the Mental Health Works training content. Mental Health Works engages partners in marketing, development, service delivery, and product sponsorship.

 The Strongest Families Institute delivers its services in other regions of Canada and the world through partnerships with regional health services organizations. For example, the Strongest Families Institute coaches based in Halifax provide telephone coaching services to families in Calgary, Alberta, British Columbia, Thunder Bay, and the Region of Peel, Ontario.

 Participatory approaches

 The Cannabis and Psychosis Awareness Project took a participatory action research approach.

This approach allowed youth researchers to facilitate an open and non-judgmental discussion with other youth about their experience with cannabis and psychosis, contributing to rich and new data. Youth identified the key messages and created arts-based knowledge exchange products. In addition, youth gained new skills and confidence helping them cope with their own recovery.

 The Consumer, Family, Provider Initiative also promoted participatory approaches. The initiative encouraged collaboration among consumers, family members and providers at various levels including at policy levels through participation in committees and working groups and in the development and roll out of collaboration tools, and day-to-day through comment cards, family and patient surveys, and one-on-one interactions between consumers, families, and providers.

 Innovative delivery and dissemination methods

 The Strongest Families Institute uses telephone and other technology to deliver distance mental health treatment services to parents of children with mild to moderate mental health issues. The technology and materials (i.e., telephone coaching services, DVDs, booklets and website) facilitate the delivery of the services without requiring an in-person meeting. Well-trained and highly monitored non-professional coaches help parents to learn and adopt skills over the phone. This creates a barrier free environment, which saves parents time and money, and saves costs to the formal system.

 The Cannabis and Psychosis Awareness Project used innovative approaches in the dissemination of its findings. The youth developed arts-based messaging pertaining to their experiences with Cannabis and Psychosis, including poems, raps, videos and personal stories.

(6)

The messaging worked both to benefit youth in the creation of art (e.g., therapeutic and healing effects), to develop a new knowledge base from youths’ perspectives, and to engage target audiences in its dissemination (e.g., impact on practice change, uptake, and awareness).

 Evidence-based approaches

 Mental Health Works developed its training based on best practices in both the content and delivery of its training. It also engages partners and participants in providing feedback on the delivery of the training, and up-dates the training accordingly.

 The Consumer, Family, Provider Initiative uses evidence-based training to train service providers in family work. It was noted that having input and guidance from experts in this area helped to get buy-in from service providers.

 The Strongest Families Institute based its programming on research and evidence, and has extensive ongoing evaluation, research, and quality assurance mechanisms to track progress and monitor effectiveness.

The evaluations also found that projects advance the Mental Health Strategy for Canada in different ways. The table below provides some examples of how the projects advance the Strategic Directions and Priorities of the Strategy.³

How the Projects Advance the Goals of the Mental Health Strategy for Canada

Project How the Project Advances the Goals of the Strategy

Consumer, Family and Provider Collaboration in Treatment, Program and

Policy Planning, and Evaluation (Capital District Mental Health Program, Nova

Scotia)

 Increases the capacity of families and caregivers to promote the mental health of infants, children, and youth, prevent mental illness and suicide wherever possible, and intervene early when problems first emerge (Priority 1.2) by involving them more in the care of their loved one;

 Actively involves people living with mental health problems and illnesses and their families in making decisions about service systems (Priority 2.2) and expands their leadership role in setting mental health policy (Priority 6.4) by involving consumers and family members in committees and working groups;

and

 Upholds the rights of people living with mental health problems and illnesses (Priority 2.3) by ensuring that information is shared in a manner that abides by federal and provincial confidentiality laws.

Mental Health Works (Canadian Mental Health

Association-Ontario Division)

 Promotes mental health in workplaces (Strategic Direction 1);

 Works to create mentally healthy workplaces (Priority 1.3) by providing training in workplace mental health to managers, employees, and other workplace representatives; and

 Contributes to increased awareness and reduced stigma with respect to mental health issues.

3 MHCC. 2012. Mental Health Strategy for Canada: Changing Directions, Changing Lives. Accessed November 22, 2012 from:

http://strategy.mentalhealthcommission.ca/download/

(7)

National Guidelines for Seniors' Mental Health:

From Evidence to Implementation (Canadian

Coalition for Seniors’

Mental Health)

 Increases awareness about how to promote mental health and prevent mental illness and suicide, and reduce stigma (Strategic Direction 1), through the development of guidelines on seniors’ mental health;

 Increases the capacity of older adults, families, care settings, and communities to promote mental health later in life, prevent mental illness and suicide wherever possible, and intervene early when problems first emerge (Priority 1.4); and

 Improves mental health data collection, research and knowledge exchange on best practices across Canada (Priority 6.2), since the guidelines represent the first collection of best practices and evidence in specific areas of seniors’

mental health (assessment and treatment of delirium, depression, suicide risk and prevention, and mental health issues in long term care homes).

Strongest Families Institute

 Increases awareness and capacity of families to promote mental health of children and youth, and to intervene early when problems first emerge

(Priorities 1.1 and 1.2), by providing families with tools and telephone coaching services; and

 Increases the availability and coordination of mental health services in the community, by increasing the access and timeliness of services, particularly for families in rural areas and remote communities (Priority 3.2 and 4.3).

Understanding the link between cannabis and psychosis: An awareness

strategy for youth (Schizophrenia Society of

Canada)

 Increased awareness about how to promote mental health, prevent mental illness and suicide, and reduce stigma (Priority 1.1), by involving youth with psychosis in the process and disseminating results;

 Helped to foster the recovery and well-being of the youth researchers and participants living with psychosis that were involved (Strategic Direction 2); and

 Developed a new knowledge base by obtaining youth perspectives on cannabis use and psychosis, thereby improving mental health data collection, research and knowledge exchange within Canada (Priority 6.2).

(8)

PROJECT EVALUATION REPORTS

(9)

MHCC NATIONAL MENTAL HEALTH AWARDS PROJECT EVALUATION:

Capital District Mental Health - Consumer, Family and Provider Collaboration in Treatment, Program and Policy Planning, and Evaluation

Final Report Prepared for the Capital District Mental Health Program & the Mental Health Commission of Canada

March 1, 2013

Prepared By:

(10)

Introduction

1 Purpose of the Evaluation 1 Method of Study

Profile of the Consumer, Family, Provider Collaboration Initiative 2 Objectives

2 Activities 4 Target Groups

5 Organizational Structure

Summary of the Major Findings 6 Impacts & Success

8 Initiative Design

9 Knowledge Exchange & Evaluation

Conclusions 11 Conclusions

Appendices A – 1 Evaluation Questions

A - 2 List of Documents Reviewed

(11)

I. INTRODUCTION A. PURPOSE OF THE EVALUATION

The objective of this study was to conduct an evaluation of the “Consumer, Family and Provider Collaboration in Treatment, Program and Policy Planning, and Evaluation” initiative (Consumer, Family, Provider Collaboration Initiative) of the Capital District Mental Health Program in Nova Scotia as part of the 5^th Anniversary National Mental Health Awards of the Mental Health Commission of Canada (MHCC).

The Consumer, Family and Provider Collaboration Initiative supports consumers, families and providers to effectively work together to support recovery and ultimately improve the mental health care system as a whole.

Within this model, consumers, families and providers are all part of treatment planning, program and policy planning, implementation, and evaluation which leads to improved outcomes for all.⁴ The initiative was selected as a winner in the Community Capacity category of the Awards and as part of its Awards package it was given an evaluation opportunity.

The overall purpose of the evaluation is to determine the initiative’s impacts and success, to highlight its innovative approach and design, and to support the Capital District Mental Health Program and the MHCC in sharing best practices related to the initiative.

B. METHOD OF STUDY

The methodology undertaken to conduct the evaluation involved a number of steps, including:

 Conducting a preliminary review of documents (e.g., Awards application form)

 Undertaking a project launch meeting with the initiative’s Program Manager;

 Developing an evaluation methodology report including a list of evaluation issues and questions,⁵ and an interview guide;

 Conducting a detailed review of relevant documents that provide information on the initiative’s objectives, activities, performance, achievements and other impacts (e.g., Quality Program Annual Report);⁶

 Undertaking interviews with 6 key stakeholders including 1 program manager, 1 community partner, 1 family member, 1 consumer/family member and 2 service providers/clinicians;

 Undertaking an analysis of all the quantitative and qualitative data obtained from the different lines of evidence and summarizing the data in response to each evaluation question; and

 Preparing this evaluation report which summarizes all the major findings and conclusions.

C. STRUCTURE OF THE REPORT

This document contains four chapters, including this introduction (Chapter I). Chapter II provides a profile of the Consumer, Family,Provider Collaboration Initiative. Chapter III describes the major findings of the evaluation with respect to the initiative’s impacts and success to date, design, and knowledge exchange and evaluation activities. Chapter IV presents the major conclusions arising from the evaluation.

4 MHCC. 2012. National Mental Health Awards Application Form - Capital District Mental Health Program, Nova Scotia for “Consumer, Family and Provider Collaboration in Treatment, Program and Policy Planning, and Evaluation,” p. 4.

5 The evaluation questions are presented in Appendix I.

6 The list of documents reviewed is presented in Appendix II.

(12)

II. PROFILE OF THE CONSUMER, FAMILY, PROVIDER COLLABORATION INITIATIVE

This chapter provides an overview of the Consumer, Family, and Provider Collaboration Initiative in terms of its objectives, activities, target groups, and organizational structure.

A. OBJECTIVES

A key identified priority of the Capital District Mental Health Program is to ensure that persons who have lived experience with mental illness and members of the circle of support participate in the design, delivery and evaluation of mental health services. Within its Quality Improvement Program, the Capital District Mental Health Program has four performance domains and three outcome domains, as described below:

Performance domains:

1. Access - To provide people with easy and timely access to appropriate mental health care. Mental health services are based on the demographic needs of our community.

2. Consumer, Family and Provider Collaboration - To collaborate with consumers and families in the planning and delivery of care.

3. Continuity of Care - To provide continuous and integrated care across services.

4. Safety - To provide a safe environment of care for consumers, families, and staff.

Outcome domains:

5. Service Effectiveness - The Mental Health Program offers effective services as measured by outcomes.

6. Consumer and Family Satisfaction - Consumers and families experience satisfaction and positive outcomes with services.

7. Staff Satisfaction - Staff like their work and workplace and are satisfied with service they provide.

They have input into planning.⁷

Stakeholders that were interviewed as part of this evaluation were asked to provide their opinion on what they saw as the primary objectives of the initiative. All stakeholders (6 stakeholders) agreed that the objective of the initiative was to improve communication and collaboration among consumers, family members, and health care providers with the view to improving mental and physical health outcomes for consumers, experiences of family members, and understanding of providers with respect to consumer and family member experiences.

Other specific outcomes mentioned included reduced lengths of stay in the hospital, fewer hospital admissions, and fewer relapses. Stakeholders also explained that the purpose of the initiative was to develop tools, protocols, and training to facilitate collaboration (e.g., information sharing guidelines, the participation of consumers and family members in committees, etc.) so that the involvement of consumers and family members in the process of care was more than just a “token” involvement.

B. ACTIVITIES

A number of activities have been implemented to support the Consumer, Family, and Provider Collaboration Initiative. The main activities and their progress to date are described below.⁸

7 Capital Mental Health Program. 2011. Quality Improvement Program, p. 2.

(13)

1. Imbed the concept of collaboration within the Mission, Vision, and Guiding Principles of the Mental Health Program. The Mission statement includes family and individuals living with mental health problems as partners in the delivery process: “Together with families, communities and providers, we work with individuals living with mental health problems and mental illnesses, helping them achieve recovery through treatment and support.”⁹ Also, family collaboration is listed as a guiding principle in terms of service delivery: “We include and value families, and have a duty of care towards them.”¹⁰ Specific references to family are made in 7 out of 15 Guiding Principles.

2. Create and implement guidelines regarding information sharing. A workgroup composed of consumers, family, and provider representatives in consultation with Capital Health Legal Services was initiated to develop guidelines for disclosure of information to families. The results of this work are the Information Sharing Guidelines.¹¹ The guidelines describe information sharing within a triangle of care (e.g., the person living with mental illness, their circle of support, and the providers of mental health services). One of the key components of the guidelines is the role of providers in ensuring patients are aware of their right to confidentiality. Also central to the guidelines is the role of providers to engage patients in discussions about the advantages and disadvantages of sharing specific information with individuals within their circle of support. The guidelines also include examples and cases for reflection.¹²

Training has been conducted to roll out the implementation of the guidelines. A “train the trainer”

approach was adopted with support from the Meriden Family Programme for approximately forty participants who would act as champions on information sharing. The sessions were provided by groups of three including a consumer, a family member and a provider.¹³ As of December 2012, 382 staff and 49 non-staff had completed training on the information sharing guidelines.

3. Develop and implement recommendations that recognize and support the education needs of the CFP groups. In December 2010, the Mental Health Program established a Consumer, Family, Provider Collaboration Group, which was asked to identify the education needs for these three groups with respect to enhancing collaboration and to formulate recommendations. The group relied upon evidence/findings from: the experiences of individuals, consumers, family members, providers, and organizations that work with these groups; opinions offered in focus groups conducted by the Mental Health Program; and review of selected literature, documents, and web-based information. A total of 107 recommendations were identified through this work and are being addressed by key program-wide quality initiatives, as well as by the Quality Teams which have identified areas for improvement relevant to their service areas.¹⁴

4. Meet the education needs of staff in the collaboration work. A strategy was developed to implement the Behavioural Family Therapy model and approach. The Mental Health Program obtained funding from the Mental Health Foundation of Nova Scotia in order to implement training in the model through a partnership with the UK-based Meriden Family Programme.¹⁵ Meriden is a National Health Service Programme which specializes in training and organizational development in family-sensitive, evidence-based mental health services. A cascade system of training is delivered whereby people are trained in Behavioural Family Therapy and are then provided with supervision to work with families. A number of therapists are then trained as trainers and supervisors. These trainers provide in-service

9 Capital District Mental Health Program. 2010. Vision, Mission & Guiding Principles, p. 1.

10 Ibid.

11 Capital District Mental Health Quality Program. 2012. Annual Report 2011/2012, p. 10.

12 CDHA Mental Health Program. 2011. Information Sharing within the Triangle of Care Guidelines.

13 Capital District Mental Health Quality Program. 2012. Annual Report 2011/2012, p. 10-11.

14 Ibid, p. 12.

15 Meriden Family Programme Website: http://www.meridenfamilyprogramme.com/

(14)

courses in their own localities to multi-disciplinary professional groups, individuals and family members.¹⁶

Implementation of the first phase began in June 2012 with 25 providers from across the Mental Health Program being trained in Family Work (Behavioural Family Therapy). In November 2012, 26 additional staff were trained in Family Work. In addition, in September 2012, 18 staff and 8 family members were trained in Family Education and Support.¹⁷

5. Complete an Evaluation that measures the activities, impacts and outcomes of the collaboration work. The mandate of the Feedback and Experience Action Team (FEAT) is to develop mechanisms for consumers and families to provide valuable feedback to the Mental Health Program and to make recommendations based on that feedback. A set of four satisfaction surveys, developed by FEAT, were launched in the spring of 2012 including outpatient/inpatient satisfaction surveys for patients as well as family and friends. The surveys were developed over the course of eighteen months by a group of consumers, family members, and mental health staff using focus group feedback, input from other available surveys and their own experience. The surveys cover a wide range of topics and have a particular focus on collaboration. In addition, comment cards and boxes were made available in all service areas in 2010/2011.¹⁸

6. Implement a tool that supports a common understanding of CFP Collaboration. The Quality Steering Committee developed a tool that supports a common understanding of CFP Collaboration and this is used as the basis of the collaboration work. This has been adopted by the Mental Health Program.¹⁹ In the Triangle of Care, the knowledge, values and beliefs, spiritual and cultural backgrounds of the person and their circle of support are incorporated into care planning and decision making. This special relationship is illustrated in a diagram at the following link:

http://ourhealthyminds.com/LinkClick.aspx?fileticket=QsR0BA4XjkI%3d&tabid=59.

7. Implement and evaluate a process to orient, place, support, and recognize consumer and family involvement within quality committees, teams, and councils. In partnership with the Healthy Minds Cooperative, consumers and family members have been placed on key committees across the Mental Health Program, including the Leadership Council, Quality Council and each of the eight Quality Teams. The Healthy Minds Cooperative is an independent non-profit organization and long-time partner of Capital District Mental Health. Healthy Minds provides training, recruitment, and support for families and consumers to participate in committees. The consumers and families participating on these committees provide insight and support to decision making processes related to the design, implementation and evaluation of Mental Health Services.²⁰ As of December 2012, there were 21 consumer positions and 26 family spots on committees and working groups.

C. TARGET GROUPS

The target groups for the initiative are consumers of mental health services (patients/persons living with mental illness), family members and caregivers of persons living with mental illness, providers of mental health services (e.g., clinicians, psychiatrists, therapists, etc.), and community partner organizations (e.g., the Healthy Minds Cooperative).

16 Capital Health Mental Health Program Person and Family Centred Collaborative Care Funding Proposal Submitted to the Nova Scotia Mental Health Foundation January 10, 2012, pp. 2-3.

17 Capital District Mental Health Program. 2012. Report to Quality Council November 26, 2012.

(15)

D. ORGANIZATIONAL STRUCTURE

The Capital Health Addictions and Mental Health Program, together with the Dalhousie Department of Psychiatry, provide crisis, emergency, inpatient and community mental health support in the Capital District of Nova Scotia. The majority of the services and supports are provided at three facilities (Abbie J. Lane, East Coast Forensic Hospital, and the Nova Scotia Hospital). Many, however, are located in the community.²¹

The services provided by the Capital Health Addictions and Mental Health Program include Acute Services, Mental Health (Short Stay Unit), Community Mental Health Services, Community Outreach Assessment and Support Team – COAST, Connections Clubhouse, Dual Diagnosis Program Mental Illness/Developmental Disability Eating Disorders Program, Emergency Assessment Services, Mental Health Day Treatment, Mental Health Case Management Services, Mental Health Mobile Crisis Team, New Beginnings Clubhouse, and Seniors Services and the Sleep Disorders Program and the recently added Addictions Program.²²

The Mental Health Quality Program was formed as the result of the 2005 Healthy Minds Initiative strategic plan which adopted quality evaluation and improvement as a strategic direction. The Mental Health Quality Improvement Program Framework, which includes the Quality Council, Quality Teams, and Quality Domains was implemented and has since been the foundation of quality work in Mental Health.

 Leadership Council: The Leadership Council is responsible for overseeing the Quality Council and, through the Quality Council, the evaluation and quality improvement strategy of the Mental Health Program. The Leadership Council reviews and acts on the reports and recommendations of the Quality Council to ensure quality improvement is undertaken as appropriate.

 Quality Council: The Quality Council is appointed by the Leadership Council. It oversees evaluation and quality improvement in the Mental Health Program including ongoing implementation and evolution of the evaluation and quality improvement framework.

 Quality Teams: The Quality Teams are appointed by the Leadership Council. Each represents one or more related services. They monitor the performance of their constituent services and take action to improve performance where needed. They report to the Quality Council on a regular basis.

As shown in the figure below, an accountability loop ensures sustained quality evaluation and improvement.

Accountability includes annual reporting of evaluations of the Quality Teams, major councils and committees, including the Leadership Council, to the Quality Council which reports annually on its evaluation to the Leadership Council.²³

Accountability Loop

21 CDMH Program Website. Accessed November 20, 2012 from: http://www.cdha.nshealth.ca/mental-health-program.

22 Ibid.

23 Capital Mental Health Program. 2011. Quality Improvement Program 2011, pp. 1-3.

Services Quality Teams Quality Council Leadership

Council C O N S U M E R S – F A M I L I E S – C O M M U N I T Y – P R O V I D E R S – C D H A

(16)

III. SUMMARY OF THE MAJOR FINDINGS

This chapter describes the summary of the major findings including with respect to impacts and success, project design, and knowledge exchange and evaluation.

A. IMPACTS & SUCCESS

1. The Consumer, Family, Provider Collaboration Initiative has been successful in empowering consumers and families, changing understanding and attitudes, and rolling-out specific activities, however it will take time to see changes to the full system and culture.

Stakeholders provided a rating of 4/5, when asked how successful the Consumer, Family, Provider Collaboration Initiative has been in achieving its objectives on a scale from 1 to 5 where 1 is not at all successful, 3 is somewhat successful, and 5 is very successful.

Many stakeholders (4 stakeholders) explained that there has been increased and more meaningful involvement of family members and consumers. Stakeholders explained that the initiative has empowered consumers and family members to have their voices heard and to see their ideas put into practice through a number of mechanisms (e.g., involvement in committees, development of policies, comment cards, focus groups, satisfaction surveys, etc.). Stakeholders also reported that the relationships and level of trust between families, consumers and providers has improved and that it is less often that a family member will feel excluded from the provision of care.

Stakeholders also noted that there have been changes in the attitudes and perceptions of providers, consumers and family members. It was noted that providers sitting on committees listen with great interest to family members and consumers since they recognize the value of the opinions, experiences and skills offered by family members and consumers. For example, it was noted that providers have a better understanding of the social determinants that affect consumers’ mental health (e.g., housing, poverty, etc.) and the challenges they face. One provider noted that working with family members and consumers has changed their own personal practice and it has had a huge impact on their work. Also, it was reported that consumers and family members have a better understanding of the challenges providers face, which has enhanced relationships and the overall tone of conversations.

Specific activities were identified by stakeholders as having a successful roll-out. For example the consumer, family, provider collaboration recommendations and the patient and family satisfaction questionnaires highlight key issues which are important for families and consumers. Also the family work training and roll-out has resulted in 34 families being offered family work and 21 families being involved in family work across five service areas (i.e., Cole Harbour, Mood Disorders, Simpson Landing, Connections Dartmouth, and ECFH).²⁴ And as noted earlier, as of December 2012, 382 staff and 49 non-staff had completed training on the information sharing guidelines.

Some stakeholders (2) noted that changes to the full system and culture will take time as more staff are trained and once feedback mechanisms are in place.

24 Capital District Mental Health Program. 2012. Report to Quality Council November 26, 2012, p. 4.

(17)

2. Areas where the initiative has been less successful include reluctance from some providers and consumers to change behaviours due to fears over confidentiality or stigma, and challenges in generating awareness about the initiative among consumers and families.

Several stakeholders (5) noted that, although it is a minority, some providers and consumers are reluctant to adopt collaborative attitudes and behaviours. Providers are reluctant due to fear with respect to legal consequences and confidentiality (e.g., fear of litigation) and mixed messages from superiors in the past. Stakeholders also noted that other organizations within Capital District Mental Health (e.g., emergency services) are not necessarily as collaborative as the Mental Health Program and this is at odds with what the initiative is trying to accomplish. It was also noted that some consumers are reluctant to share information with family members or providers due to fear of stigma associated with their mental illness.

Another challenge identified was the difficulty in generating awareness about the collaboration initiative among consumers and family members that are not already engaged in committees. It was noted that if consumers and family members are aware of the efforts being made to increase collaboration then this also might help to ease their concerns and encourage their involvement in care.

3. Factors that contributed to the success of the project included the support and buy-in for the initiative at all levels, and the time and funding that was allocated for specific activities.

Stakeholders (4) noted that a major contributing factor to the success of the initiative was the support and buy-in for the initiative at all levels. Key to the success was that the Mental Health Program leadership made collaboration a priority. Quality committees were formed in each service area and collaboration was imbedded in policies and guiding principles. Stakeholders also explained there was support and buy-in among leaders within the service provider group and willingness of consumers and family members to share their views. Another key group in the process was community partners such as the Healthy Minds Cooperative which helped to support the involvement of consumers and families and liaise with providers.

Stakeholders also noted that the allocation of resources to the initiative was important. To date, there have been no additional paid positions created to do this work. Although,` time and funding was dedicated for provider training, supervision, and family work. This way the initiative was integrated as part of providers’ day-to-day work and not seen as an “add-on.” Also consumers and family members were provided with a nominal compensation, and had expenses covered (e.g., printing and transportation) so they could participate in committees.

4. Factors that constrained the success of the initiative included resistance among some providers and consumers with respect to information sharing, leadership changes within the Mental Health Program, and ongoing struggles for timely access to information and/or care.

Several factors constrained or slowed the success of the initiative. As noted earlier, a few providers and consumers resisted change due to worries about confidentiality and stigma. It was also noted that turnover among leadership within the Mental Health Program constrained the success since new managers and leaders had to learn about the collaboration policies. Another factor was that some consumers and families continue to struggle with getting timely access to information and care, which limits the level of collaboration that can occur day-to-day.

(18)

B. INITIATIVE DESIGN

1. The initiative demonstrated a number of best or promising practices including collaboration among consumers, family members and providers at policy and practical levels, and using evidence-based training (e.g., Meriden) to implement collaborative practices among providers.

All stakeholders (6) noted that a best practice of the initiative was that collaboration was imbedded at the policy and practical levels. Consumers, family members, and providers collaborated at the policy and project level through the participation in committees and working groups. And, at the practical level, collaboration was supported through the use of tools (e.g., information sharing guidelines, and feedback surveys, etc.).

Stakeholders (4) also identified the Meriden training in family work as a best practice. They explained that the model had been proven to have worked in other areas and was recognized as a best practice in the literature. Stakeholders also noted that having input and guidance from Meriden helped to get buy-in from providers.

2. Central to overcoming challenges related to confidentiality and shifting the culture was the development of the information sharing guidelines and widespread training on the guidelines.

It was noted by stakeholders (4) that the development of the information sharing guidelines contributed to addressing challenges around confidentiality. The guidelines were developed over two years and in collaboration with Meriden, a consumer, family and provider working group, and a legal team. It was identified that the legislation was being interpreted too strictly and that this was preventing collaboration among consumers, families, and providers. The guidelines that were developed follow the federal and provincial laws but also allow for collaboration. Widespread training was conducted to raise awareness and promote implementation of the guidelines.

The guidelines encourage providers to have conversations with patients about information sharing, e.g., what information is ok to share, who is important in the patient’s life, etc. This way providers are able to share some information while still maintaining the confidentiality rights of their patient. Also, providers can check in at different times with the patient. The guidelines give providers the permission and freedom to share information with consumers and families, and to help family members to feel safe providing information about their loved one.

3. With respect to implementing other similar initiatives, stakeholders noted that it was important to have processes in place to ensure that collaboration remains imbedded, and to recognize that it is a long term initiative (a cultural shift), among other recommendations.

Stakeholders made a number of recommendations with respect to the implementation of other similar initiatives in the future including:

 Have processes to ensure collaboration remains imbedded and mechanisms to track how often providers collaborate with consumers and families (3);

 Recognize that it is a long term initiative (a cultural shift) and establish timelines that allow stakeholders to embrace it rather than see it as a burden (2);

 Ensure all stakeholders are involved from the beginning, their roles are clarified, and that the initiative has support from leadership (2);

 Include a budget for implementation of some aspects (e.g., Meriden training had additional funding from the Mental Health Foundation of Nova Scotia);

(19)

 Provide family members and consumers with support and an orientation on participating in committees (e.g., as provided by the Healthy Minds Cooperative) so that they are comfortable expressing their opinions in these settings; and

 Ensure the right people are recruited to participate in committees and lead dissemination on the initiative (e.g. people that volunteer to participate, are open-minded, and passionate about the initiative).

4. The initiative advances the Mental Health Strategy for Canada by actively involving people living with mental health problems and illnesses and their families in making decisions about service systems, and by upholding the rights of people living with mental health problems and illness.

The work of the Consumer, Family, and Provider Initiative advances a number of priorities of the Mental Health Strategy for Canada. The initiative increases the capacity of families and caregivers to promote the mental health of infants, children, and youth, prevent mental illness and suicide wherever possible, and intervene early when problems first emerge (Priority 1.2) by involving them more in the care of their loved one. The initiative also more actively involves people living with mental health problems and illnesses and their families in making decisions about service systems (priority 2.2) and expands their leadership role in setting mental health policy (Priority 6.4) by involving consumers and family members in committees and working groups. The initiative also upholds the rights of people living with mental health problems and illnesses (Priority 2.3) by ensuring that information is shared in a manner that abides by federal and provincial confidentiality laws.²⁵

C. KNOWLEDGE EXCHANGE & EVALUATION

1. Various stakeholders have been involved in the design and implementation of the initiative, including consumers, family members and providers as well as community partners.

Stakeholders (6) highlighted that consumers, family members, providers and some community organizations have been involved in the Mental Health Program Quality Team committees, sub- committees and focus groups. Imbedded within the Quality Framework, the Consumer, Family, Provider Collaboration Initiative reaches all 32 services and their connected service providers. The Quality Council, each Quality Team and working group has representation from consumers, families and providers. It was noted that these mechanisms promote accountability and transparency since consumers and family members are able to review policy documents and leaders are very forthcoming with program information. Feedback and input is also provided by consumers and family members through comment cards and patient and family surveys that were developed by the Feedback Experience Action Team. Providers communicate with their managers to provide feedback as well.

Other active key stakeholders include a number of community partners such as the Schizophrenia Society, CMHA, and the Healthy Minds Cooperative, who are also actively involved in committee and council structures. In addition, the Healthy Minds Cooperative has a key role in training, placing and supporting consumers and families to sit on various committees and councils throughout the program.

25 MHCC. 2012. Mental Health Strategy for Canada: Changing Directions, Changing Lives. Accessed November 22, 2012 from:

(20)

2. Information generated by the project has been shared to some extent through different events and meetings and there are opportunities to expand the reach of the knowledge exchange through partnerships or events.

Information about the project has been shared externally through the Capital Health Addictions and Mental Health Program’s annual Academic Education Day and through the provincial Mental Health Directors meetings. It was noted that information has been shared with the community through large- scale public forums (over 600 attendees), bringing together policy makers, providers, and community members. There have been initial consultations with two other health districts regarding the initiative and a presentation was delivered when receiving the Meaningful Involvement of Consumer Award from the Department of Health and Wellness. It has also been shared through the Capital Health and Mental Health Program's Newsletters. A presentation was made at the Capital Health Quality Summit in 2012, Bronze Quality Award Winner. It is anticipated that there will be further provincial and possibly national presentations and consultations as the work is implemented and evaluated.²⁶

Stakeholders recommended that knowledge exchange and awareness-raising about the initiative continue and that more opportunities should be created for sharing of experiences and approaches. It was noted that there could be more sharing of documentation that has been developed (e.g., through the Mental Health Program website or community partner organizations). Stakeholders also suggested creative ways to share information such as having town halls should be explored.

3. There are some mechanisms in place to monitor and evaluate the progress of the collaboration initiative, however a clear strategy for their implementation will help to facilitate their uptake.

Stakeholders described different mechanisms that are in place to track and monitor the initiative’s progress. For example, there are forms to track the number of families who have been engaged as part of family work. Also the Feedback and Experience Action Team will provide reports on the feedback that is received from patient and family comment cards and surveys. Outcome evaluations are being conducted of the Meriden training. Also, the Mental Health Program is preparing for Accreditation Canada’s accreditation of the area hospitals and health programs in 2013, which will require direct feedback from consumers and family members.

In terms of challenges, stakeholders identified that it is difficult to conduct the evaluation activities due to lack of time and resources to assist in the evaluation. Stakeholders noted that it needs to be determined who is responsible for what tasks with respect to evaluation. It is difficult for providers to collect and interpret data that can be used for tracking since they have many other competing demands. Another challenge with respect to evaluation and accountability is the size organization (e.g., over 700 employees), and that it will take time to shift the procedures and culture organization-wide.

Stakeholders recommended that some resources are dedicated to evaluation activities or it could be conducted in collaboration with a local university. It was suggested that independent researchers could assist with evaluation activities, such as a PhD student working towards a dissertation. Another suggestion was to use peer support in assisting to put the surveys online through a website link.

Stakeholders noted that it will be important to have good outcome measurement tools for the mental health population and that the Mental Health Program could look at best practices for outcome measures and indicators from other districts (e.g., reduced relapse rates, reduced admissions for clients, and reduced lengths of stay).

26 Ibid, pp. 6-8.

(21)

IV. CONCLUSIONS

1. The Consumer, Family, Provider Collaboration Initiative has been successful in empowering consumers and families, changing understanding and attitudes, and rolling-out specific activities. Factors that contributed to the success of the project included the support and buy-in for the initiative at all levels, and the time and funding that was allocated. However, in order to see broader impacts to the full system and culture, it will take time as more staff are trained and more stakeholders are made aware of the initiative.

2. An area that has constrained or slowed the success of the initiative is the reluctance from some providers to change behaviours due to fears over confidentiality, and consumers to reveal information due to stigma, although the information sharing guidelines are helping to alleviate concerns. Other factors that constrained the success of the initiative included leadership changes within the Mental Health Program and ongoing struggles for timely access to information and/or care.

3. The initiative demonstrated a number of best or promising practices including:

 Collaboration among consumers, family members and providers at policy levels through participation in committees and working groups and in the development and roll out of tools (e.g., information sharing guidelines), and day-to-day through comment cards, family and patient surveys, and one-on-one interactions between consumers, families, and providers.

 Using evidence-based training such as the Meriden training in family work. The model has been proven to have worked in other areas and is recognized as a best practice in the literature.

Having input and guidance from Meriden consultants helped to get buy-in from providers.

 Overcoming issues related to confidentiality with the development and roll-out of the information sharing guidelines. The guidelines emphasize that having conversations about information sharing can help providers to share information in a way that respects the patient’s rights and promotes their health and safety.

4. The Consumer, Family, Provider Initiative advances the Mental Health Strategy for Canada by actively involving people living with mental health problems and illnesses and their families in making decisions about service systems, and by upholding the rights of people living with mental health problems and illness.

5. There are some mechanisms in place to monitor and evaluate the progress of the collaboration initiative (e.g., comment cards, feedback surveys, etc.), however a clear strategy for their implementation will help to facilitate their uptake. Information generated by the project has been shared to some extent through different events and meetings and there are opportunities to expand the reach of the knowledge exchange through partnerships or events.

6. With respect to implementing other similar initiatives, stakeholders noted that it was important to have processes in place to ensure collaboration remains imbedded, and to recognize that it is a long term initiative (a cultural shift), among other recommendations.

(22)

APPENDIX I: EVALUATION QUESTIONS

The evaluation questions, presented in the following table, guided the data collection and analysis components of the evaluation. Data was gathered on each evaluation question through multiple lines of evidence. The data collection instruments were developed based on these questions and modified to suit the relevance of the information source or stakeholder group.

Evaluation Questions IMPACTS & SUCCESS

1. What are the intended impacts and objectives of the project? In what manner and to what extent have they been achieved?

2. What other impacts, whether positive or negative, have been generated by the project?

3. What factors contributed to the success of the project? What factors have served to constrain the success of the project?

PROJECT DESIGN

4. What are the best or promising practices that have been developed as part of the project?

5. What lessons have been learned with respect to the implementation of this project that can facilitate the implementation of other similar initiatives in the future?

6. In what manner and to what extent does the project advance the priorities of the Mental Health Strategy for Canada?

KNOWLEDGE EXCHANGE & EVALUATION

7. How have different stakeholders been involved in the design and implementation of the project?

8. How has knowledge generated by the project been shared with stakeholders? What are some ways in which these activities could be enhanced?

9. What mechanisms are in place to monitor and evaluate project impacts?

What are some ways in which these activities could be enhanced?

(23)

APPENDIX II: LIST OF DOCUMENTS REVIEWED

Capital District Mental Health Program. 2010. Vision, Mission & Guiding Principles.

Capital District Mental Health Program. 2012. Report to Quality Council November 26, 2012.

Capital District Mental Health Quality Program. 2012. Annual Report 2011/2012.

Capital Health Mental Health Program Person and Family Centred Collaborative Care Funding Proposal Submitted to the Nova Scotia Mental Health Foundation January 10, 2012.

Capital Mental Health Program. 2011. Quality Improvement Program.

CDHA Mental Health Program. 2011. Information Sharing within the Triangle of Care Guidelines.

CDMH Program Website. http://www.cdha.nshealth.ca/mental-health-program.

Meriden Family Programme Website. http://www.meridenfamilyprogramme.com/

MHCC. 2012. Mental Health Strategy for Canada: Changing Directions, Changing Lives.

MHCC. 2012. National Mental Health Awards Application Form - Capital District Mental Health Program, Nova Scotia for “Consumer, Family and Provider Collaboration in Treatment, Program and Policy Planning, and Evaluation.”

(24)

MHCC NATIONAL MENTAL HEALTH AWARDS

PROJECT EVALUATION: MENTAL HEALTH WORKS

Final Report Prepared for the Canadian Mental Health Association-Ontario Division &

the Mental Health Commission of Canada March 1, 2013

Prepared By:

(25)

Introduction

1 Purpose of the Evaluation 1 Method of Study

Profile of Mental Health Works 2 Objectives

2 Activities 3 Target Groups

3 Organizational Structure

Summary of the Major Findings 6 Impacts & Success

8 Initiative Design

10 Knowledge Exchange & Evaluation

Conclusions 12 Conclusions

Appendices A – 1 Evaluation Questions A – 2 List of Documents Reviewed