How COPR Improved Exam Quality While Increasing Efficiencies with Automated Item Generation


| Case Studies | Share:   | Print this Article

OVERVIEW

The Canadian Organization of Paramedic Regulators (COPR) promotes and advances best practices in paramedicine nationwide. Founded in 2009, COPR is comprised of self-regulating colleges and government regulators. With the help of nine provinces, COPR develops the entry-to-practice exams for the paramedic profession and administers them in seven provinces. COPR also conducts assessments of internationally educated paramedic professionals for all Canadian jurisdictions. The organization administers the following exams each year:

  • Primary Care Paramedic (PCP) Entry to Practice Examination
  • Advanced Care Paramedic (ACP) Entry to Practice Examination
  • Emergency Medical Responder (EMR) Entry to Practice Examination

COPR SNAPSHOT

Delivers exams for 3 different practice levels 4 times per year


Assesses 1,200+ candidates per year

PRODUCTS PROVIDED

  • Itematic Automated Item Generation Software
  • ProctorU Online Proctoring Platform
  • Measure Exam Delivery Software

SERVICES USED

Blueprinting

Item Writing

Test Registration

Test Administration

Item Analysis

Exam Scoring & Marking

Psychometric Consulting

THE CHALLENGE

In 2010, paramedicine was considered a noncompliant industry in Canada for labor mobility. To help move its candidates forward, COPR needed to develop pan-Canadian exams for paramedic professionals. 

COPR had three goals for its new exams:

  1. Ensure the exams are valid, reliable, and fair
  2. Standardize the licensing process across the paramedicine industry
  3. Develop a robust item bank as quickly as possible

The process of item development is notorious for being time- and resource-intensive, resulting in one of the costliest components of the testing process. To develop pan-Canadian exams with enough content to comprehensively assess its candidates, COPR needed to move quickly, deliberately, and cost effectively.

THE SOLUTION

COPR ultimately put its trust in Yardstick. And before long, Yardstick set its sights on supplementing the invaluable work of COPR’s subject matter experts (SMEs). To do so, Yardstick and COPR worked closely on piloting Itematic—an automated item generation (AIG) tool. Unlike other AIG tools, Itematic allows SMEs to specify how many items they want to generate and then creates only the most unique subset. After that, a Yardstick psychometrician can seamlessly import approved items into an existing item bank. The goal of using Itematic was not to eliminate the need for COPR’s SMEs but to give them an extra tool to save time, increase productivity, and enjoy the item-writing process.

To get started, COPR SMEs designed the structure of new items using cognitive models that define the knowledge and skills required of candidates. Next, the SMEs created item models, specifying which item features could be changed. After that, Itematic took over. Its dynamic algorithms matched the cognitive models to the item models, rapidly generating dozens of unique items. Because Itematic slashed the most time-consuming part of the development process, COPR SMEs were able to spend more time reviewing the items to make sure they were valid, reliable, and fair. 

Traditional Item Development VS. Itematic Development Process

COPR’s SMEs develop items individually.

COPR’s SMEs create templates so algorithms can develop many unique items.

It costs an average of $500-$1,000 for COPR to develop a single multiple-choice item.

COPR’s per-item cost is decreased by 30%-50% while maintaining quality.

A COPR SME can create 3–5 high-quality items per day.

The SME’s item-writing output can be ramped up significantly based on item-banking needs and the quality of the template.

THE RESULTS

With the help of Yardstick, COPR became the first Canadian organization to use AIG methodology. By using Itematic, COPR eased the work of its SMEs and invested in dynamic technology that worked alongside its most valuable resources, boosting overall efficiency and filling the shallow areas of its existing item pool. “It’s another tool in the toolbox,” says Tammy Leach, the examination manager at COPR. AIG tools have come along way over the past few years, but there’s a lot of work that still needs to be done. While Itematic is a resource for SMEs during the development process, it can’t automate the whole process. It can only add to the value created by human expertise and decision-making.

33%

Decrease in item development costs

20%

Expansion of COPR’s approved item bank by Itematic

.72.72

Comparable average difficulty between Itematic questions and non-Itematic questions


 

THE TAKEAWAY

When it comes to test development and delivery, having the right partner is a vital first step. Says Leach, “[Yardstick has been] a true partner in defining the exam process and implementing solutions that address [COPR’s] unique situation.” With Yardstick, COPR felt confident taking a risk with AIG—inevitably reaping the reward of a higher-quality exam, while saving time and money.

We are very pleased with the quality of the exam and the rigor behind the development, maintenance, and monitoring, particularly in the psychometric area. I think that Yardstick’s strength is its psychometrics.

— Tammy Leach, Examination Manager,
Canadian Organization of Paramedic Regulators