There are different exam question formats that can be used for credentialing examinations. The most common format is multiple-choice questions. However, some credentialing examinations also use constructed-response questions to assess knowledge, skills, and abilities in their credentialing process. Constructed-responses questions require exam candidates to provide a written answer in the form of a few words to a few sentences.
Constructed-response questions cannot be marked automatically the same way multiple-choice questions can. In order to assist in the marking of constructed-response questions, Yardstick Assessment Strategies (YAS) has developed an innovative tool called RedPen™.
RedPen™ is an extension of Yardstick Measure™ for the marking of constructed-response questions by markers who are trained to score exam questions using a defined scoring rubric. The question and the scoring rubrics are created, stored, and administered through Yardstick Measure™, and once the exam has been administered, markers connect to RedPen™ to access and score constructed-response questions online.
RedPen™ was initially built for the marking of constructed-response for the Certified Financial Planner(CFP®) examination developed by FP Canada. FP Canada is the certification body for the nearly 17,000 CFP professionals in Canada. The idea behind RedPen™ was to help FP Canada build efficiencies in the scoring of the CFP exam, removing the need to export exam content and mark constructed-responses using PDF documents and complex score recording systems. Instead, RedPen™ created a flow that allows a fully online experience for teams of markers. It allowed FP Canada to streamline the marking process and allow for more exams to be scored within a defined period of time while allowing for better tracking of marker performance and a timely feedback in addressing scoring discrepancies.
How does RedPen™ work?
RedPen™ allows the assignment of constructed-response cases or specific questions to markers, or teams of markers, for scoring using a fully developed scoring rubric. Markers, after having been trained on the content to mark, are able to navigate from one candidate to the other as they complete the marking through clicking on radio buttons.
RedPen™ can be configured to allow for multiple markers to mark the same candidate response, and flag discrepancies in the scores provided for review and reassessment by a team leader, or arbitrator. The arbitrator is requested to provide a score only for questions or elements of a question that have been flagged with a score discrepancy. When accessing the flagged questions, the arbitrator is able to view what score was provided from each marker and confirm the final score to be provided. This allows for as accurate of a score as possible to be provided to the candidate, allowing for discrepancies to be reviewed and assessed to ensure consistency in scoring.
How is Redpen being used at FP Canada?
FP Canada has been using RedPen™ for the scoring of constructed-responses on the CFP exam since 2013 and has marked the responses of over 4,000 candidates using this tool.
FP Canada also brought RedPen™ use to an entirely new level when they implemented it as part of the CFP Exam Practice Exam. This practice exam is composed of a combination of multiple-choice questions and constructed-response questions, aligned with the CFP exam blueprint parameters. The objective of the practice exam is to help candidates familiarize themselves with the exam content and structure, as well as the exam platform they will write their exam on. One of the significant advantages of the CFP Exam Practice Exam is that it allows candidates to access RedPen™ to self-score their answers to the practice exam constructed-response questions. By accessing RedPen™, they can see the structure of the scoring keys and gain a better understanding of how the exam is scored and how the marks are allocated. Once they are done with their self-scoring using RedPen™, candidates received a diagnostic report on how they performed based on each of the competency areas and information on their answers and their final score, with scoring keys and justifications for both the multiple-choice and constructed-response questions.
The CFP Exam Practice Exam has been a valuable tool to date for candidates and the feedback received has been very positive. Eighty percent of candidates who purchase the practice exam report that they felt “better prepared to write the CFP exam” as a result of completing the practice exam. Candidates appreciate the better visibility they get, through RedPen™ scoring, on how marks are allocated by markers, using the same tool as the markers when scoring the CFP exam.
Closing thoughts
RedPen™ has proven to be a valuable tool that specifically helped FP Canada be more efficient in scoring constructed-response questions and making the process as easy as possible for their markers. In addition, it allowed for the creation of a practice tool that brought a lot of value to candidates in preparing for taking the CFP exam, by allowing them access to the scoring tool used in real life to score their constructed response answers. This is one of the most innovative application for the use of RedPen™ that could be of value to any credentialing program using constructed-response question type for their exam.