Accreditation systems are based on a number of principles and purposes that vary across jurisdictions. Decision making about accreditation governance suffers from a paucity of evidence. This paper evaluates the pros and cons of continuous quality improvement (CQI) within educational institutions that have traditionally been accredited based on episodic evaluation by external reviewers.
A naturalistic utility-focused evaluation was performed. Seven criteria, each relevant to government oversight, were used to evaluate the pros and cons of the use of CQI in three medical school accreditation systems across the continuum of medical education. The authors, all involved in the governance of accreditation, iteratively discussed CQI in their medical education contexts in light of the seven criteria until consensus was reached about general patterns.
Because institutional CQI makes use of early warning systems, it may enhance the reflective function of accreditation. In the three medical accreditation systems examined, external accreditors lacked the ability to respond quickly to local events or societal developments. There is a potential role for CQI in safeguarding the public interest. Moreover, the central governance structure of accreditation may benefit from decentralized CQI. However, CQI has weaknesses with respect to impartiality, independence, and public accountability, as well as with the ability to balance expectations with capacity.
CQI, as evaluated with the seven criteria of oversight, has pros and cons. Its use still depends on the balance between the expected positive effects—especially increased reflection and faster response to important issues—versus the potential impediments. A toxic culture that affects impartiality and independence, as well as the need to invest in bureaucratic systems may make in impractical for some institutions to undertake CQI.
The design [1], as well as the value of accreditation systems [2, 3], are topics that require additional research. In this study we reflect on the virtues of continuous quality improvement (CQI) in medical education.
There is an implicit belief that the accreditation of medical education programs ensures the quality of the education these programs provide and the graduates they produce [4,5,6]. The importance of the accreditation of medical education programs has been further heightened by a recent announcement by the Educational Commission for Foreign Medical Graduates. The announcement states that, by the year 2023, the only candidates who can be certified, and therefore considered for residency education in the United States, will be graduates of medical schools that have been accredited by organizations approved by the World Federation of Medical Education as working to appropriately formulated and applied standards [7]. Despite the respect that accreditation is accorded, little has been written about whether the accreditation process actually achieves its intended outcomes. In many jurisdictions, accreditation of medical education programs has traditionally been based on episodic evaluation [8]. In general terms, an episodic evaluation is an external review by an accreditation authority that includes on-site visits and is repeated at specified intervals [8, 9]. More recently, the focus of accreditation has been shifting from episodic evaluation to CQI [10,11,12,13]. CQI, which originated in industry as an approach to quality control for services or products [14,15,16,17], is a never-ending cycle of organizational improvement that uses quality indicators [18].
This paper, one of a series arising from the first World Summit on Accreditation Outcomes in Medicine, explores the role of CQI in systems traditionally based on episodic evaluation. As explained before, the evidence in literature for the benefits of a CQI system in medical education is insufficient, although CQI has been introduced in several jurisdictions. The result is a new bureaucratic instrument that is being used without sufficient empirical evidence. As such, it is important to investigate the pros and cons of CQI in medical education. In this utility-focused evaluation [19] we use a set of quality criteria for governance oversight to shed more light on these pros and cons.
We chose to perform a naturalistic utility-focused evaluation [19]. Three of the authors (LP, CC, and FS) were responsible accreditation officers in their distinct fields of medical education who wished to add to the evidence and inform decision making around investments in CQI. The fourth author (NA) is a certified lawyer, medical doctor and PhD student researching accreditation systems. At the time of this evaluation, there were no conflicts concerning CQI that might have influence their opinions.
We chose to use a set of quality criteria for government oversight that did not originate from the field of medical education, because a firm theoretical background in the accreditation literature for medical education was not available.
Our author team was assigned to the roles of both descriptor of the accreditation cases and evaluator of CQI based on the criteria. The authors discussed and evaluated each system using the criteria and searched for patterns with respect to the quality criteria. During these discussions, detailed questions about each system were answered by the author responsible for that system. This process was repeated iteratively until the authors reached consensus for each criterion.
In the next section, the accreditation systems are briefly introduced to give the reader a context for our discussion about the pros and cons of CQI. A comprehensive description of the systems may be found on the websites of the accreditation authorities (afmc.ca; cacms-cafmc.ca; lcme.org; www.knmg.nl; www.royalcollege.ca) or obtained from the authors. LP, CC and FS provided detailed descriptions of the systems—both written and oral.
In North America, two authorities accredit medical degree-granting programs: the Committee on Accreditation of Canadian Medical Schools (CACMS) for medical schools in Canada, and the Liaison Committee on Medical Education (LCME) for medical schools in the United States and Canada. The CACMS and the LCME are external bodies with respect to the programs they accredit. The accreditation cycle includes an on-site visit by a peer-review team; the interval between one full survey and the next is 8 years. The internal review process for the full survey requires approximately 2 years and involves a major expenditure of human and financial resources. In 2009, the Association of Faculties of Medicine of Canada (AFMC) approved a mandatory interim review process (IRP) for Canadian medical schools that includes a review of CACMS accreditation standards at the mid-point of the formal accreditation cycle, along with CQI to achieve and maintain compliance with standards throughout the cycle [8]. The AFMC IRP is currently independent of the CACMS, and the effectiveness of the process is the subject of ongoing review. The LCME, with the endorsement of CACMS, has introduced an accreditation standard requiring schools to have a CQI process to monitor compliance with accreditation standards; this came into effect during the 2015–16 academic year [5]. Research is needed to determine the efficacy of these mandatory requirements for CQI.
In the Dutch system there are two important external authorities for the accreditation of residency training: the CGS (Legislative College for Accreditation of Residency Training) and the RGS (Specialist Registration Committee). These are bodies of the Royal Dutch Medical Association (KNMG). The CGS is responsible for setting regulations (training requirements) for accreditation, while the RGS’s role is to monitor compliance with these regulations. The medical specialties in teaching hospitals are visited every 5 years by a survey team commissioned by the RGS. There are no specific mandatory internal quality management systems regarding residency training in teaching hospitals; however, sufficient instruments and systems are made available to conduct internal quality control. The postgraduate medical education programs are required to perform some form of CQI.
There are two basic types of continuous professional development (CPD) accreditation systems: provider-based accreditation and activity-based accreditation. In activity-based CPD accreditation systems, providers submit evidence to the accreditation organization about whether or not an activity they have developed complies with established administrative, educational, and ethical standards. If the accreditation authority agrees that the standards have been met, the activity is approved. Activity-based accreditation is used, for example, by the College of Family Physicians of Canada. Provider-based CPD accreditation systems recognize organizations as institutional providers of continuing education on the basis of their ability to demonstrate adherence to a set of standards. The accreditation process frequently includes an on-site visit in which a peer-review team has the opportunity to discuss specific standards with, among others, the CPD organization’s leadership, administrative staff, and participants. Once an organization is approved, all activities it develops during its accreditation cycle are automatically approved for credit. A provider-based accreditation system is used by the Royal College of Physicians and Surgeons of Canada (RCPSC) to focus on national and provincial CPD organizations, including national specialty societies, while the Committee on Accreditation of Continuing Medical Education (CACME) focuses on the university offices of continuing medical education (CME) in the faculties of medicine. These two provider accreditation systems will be used to illustrate the process by which continuous accreditation is expressed and promoted.
Accreditation is actually a method of oversight. Oversight functions of government and accreditation processes in medicine may struggle with similar difficulties in achieving quality improvement [20, 21], even though different stakeholders are involved. The Netherlands Scientific Council for Government Policy (WRR) identified seven criteria as the basis for societally responsive oversight [22] (Table 1). Given their general applicability, we used these criteria to evaluate the pros and cons of CQI in undergraduate medical education in the United States and Canada; postgraduate medical education in the Netherlands; and CPD in Canada.
Table 1 Summary of the seven criteria and associated tasksThe results are summarized in Table 2 and reflect the authors’ evaluation of the pros and cons of CQI at the institutional level.