FROM GOOD TO GREAT: How CARB-TCMPA Created a More Secure, Defensible Exam Program


| Case Studies | Share:   | Print this Article

OVERVIEW

The Canadian Alliance of Regulatory Bodies of Traditional Chinese Medicine Practitioners and Acupuncturists (CARB-TCMPA) is the national forum and voice of provincial regulatory authorities. The founding members comprise the regulatory bodies of Traditional Chinese Medicine Practitioners and Acupuncturists (TCM/A) of Alberta, British Columbia, Newfoundland and Labrador, Ontario, and Quebec.  

CARB-TCMPA administers three exams—the Pan-Canadian Entry-Level Examination for TCM Practitioners, for Acupuncturists, and for TCM Herbalists. These exams assess the occupational competencies of entry-level TCM professionals and determine whether candidates have the skills needed to provide safe and ethical health care. 

TIMELINE

CARB-TCMPA SNAPSHOT

Represents the 5 regulated jurisdictions in Canada


Delivers 3 different exams 2 times per year


Assesses 400-500 candidates per year


PRODUCTS PROVIDED

SERVICES USED

Blueprinting

Item Writing

Test Registration

Test Administration

Item Analysis

Exam Scoring & Marking

Psychometric Consulting

TIMELINE

CARB-TCMPA SNAPSHOT

Represents the 5 regulated jurisdictions in Canada


Delivers 3 different exams 2 times per year


Assesses 400-500 candidates per year


PRODUCTS PROVIDED

SERVICES USED

Blueprinting

Item Writing

Test Registration

Test Administration

Item Analysis

Exam Scoring & Marking

Psychometric Consulting

THE GOALS

CARB-TCMPA had already decided to transition from paper-based testing (PBT) to computer-based testing (CBT) by the time it began working with Yardstick. To do so, CARB-TCMPA prioritized four ambitious goals:

 

Strengthen the development processes

Prior to the Yardstick partnership, CARB-TCMPA candidates needed to clear two hurdles before becoming credentialed: a multiple-choice (MC) component and a clinical component with scenario-driven short-answer (SA) questions. These parts were scored separately and treated as individual exams.

Ensure exam defensibility

Making sure the program was defensible and complied with standards for high-stakes exams was very important to CARB-TCMPA, which placed extra emphasis on mitigating exam exposure.

Provide a better candidate experience

Because CARB-TCMPA conducted only PBT originally, its candidates had to take exams at brick-and-mortar testing centers. While some candidates worked better at testing centers, other candidates preferred the convenience of remote testing.

Streamline the scoring process

Prior to the 2020 administration, scorers were required to meet in person because CARB-TCMPA delivered only paper-based exams. Since TCM/A has been practiced for thousands of years, scorers varied in their philosophical approaches—leading to challenges in the manual scoring process.

THE SOLUTION

DEVELOPING MORE INNOVATIVE ITEMS

To make the test development process easier, Yardstick transitioned CARB-TCMPA to the Measure Exam Delivery Software—an intuitive solution for every stage of the exam process. Dan Garcia, Executive Director of CARB-TCMPA, says, “We’ve moved to Measure, which has been very helpful. It’s definitely a big step forward.”

The first task involved expanding the competency profile for TCM/A professionals. CARB-TCMPA and Yardstick knew this process would be crucial for the success of its candidates, helping ensure they had the knowledge and skills necessary to effectively perform TCM/A care.

Next, Yardstick helped CARB-TCMPA use Measure to create blueprints from scratch. Yardstick’s psychometricians then coached the CARB-TCMPA team on writing innovative items—including multiple select, reorder, and case-based MC—that assess higher-level cognitive processes.

With a brand-new item bank, CARB-TCMPA zeroed in on exam design. The updated design? Candidates must take both the MC and the clinical component, with each comprising 50% of the total score.

ADDING LAYERS OF DEFENSIBILITY

For CARB-TCMPA, having a more secure and psychometrically sound program meant fortifying the development processes. It wasn’t enough to produce new item types that would test higher cognitive levels. The overall process had to be more defensible. So, Yardstick got to work:

  • Creating committees of SMEs
  • Facilitating item review sessions
  • Validating item keys

CARB-TCMPA avoided issues related to exam exposure by strengthening the entire development process. The items were not only more complex—which reduced the likelihood of a candidate being able to memorize and later share them with other candidates—but also reviewed more rigorously.

Feeling confident in its program’s defensibility, CARB-TCMPA turned its attention to delivery.

BOOSTING THE CANDIDATE EXPERIENCE

CARB-TCMPA notified its educators and candidates in 2019 that paper-based exams would be offered for the last time in January 2020. This gave educators a chance to update their programs. It also made sure candidates could develop the skills required for the new competency profile.

Then COVID-19 hit.

Now CARB-TCMPA had to not only deliver a computer-based exam for the first time but also find an online proctoring partner. So, it turned to ProctorU—Yardstick’s sister company and a global leader in online testing. The transition to remote proctoring was relatively straightforward since CARB-TCMPA was already leveraging Measure’s platform.

The resulting move to CBT and online proctoring was a big success for CARB-TCMPA. It tangibly improved the program’s security and offered candidates a more personalized experience.

The adoption and use of online proctoring has been a game changer for us. It’s more cost effective and arguably more secure than test centers if you look at the proctor-to-candidate ratio and the ability to record your sessions.

Dan Garcia, Executive Director of CARB-TCMPA

STANDARDIZING SCORING

CARB-TCMPA eliminated the scoring variability when it switched from SA items to more innovative item types. Since the new items required less scorer judgment, there were fewer debates about which philosophical approach was right.

THE OVERALL RESULT

In the end, Yardstick helped CARB-TCMPA surpass its goals. Chris Beauchamp, Yardstick’s VP of Psychometrics, claims their efforts “made a good program even better.” Now CARB-TCMPA has peace of mind knowing its program is more defensible and secure.

Our item discrimination and our difficult values are the strongest that this exam program has ever seen.

Dan Garcia, Executive Director of CARB-TCMPA

THE TAKEAWAY

With the support of Yardstick, CARB-TCMPA turned its ideas into reality. It didn’t happen overnight. In fact, the process involved a lot of time and energy. But the reward? A secure and defensible program that helps move TCM/A professionals forward.