The next instalment in Chief Scientific Officer Dr. Greg Sadesky’s series on virtual and remote proctoring.
It’s time to explore the potential for remote proctoring in the high-stakes testing industry.
In this latest instalment in the blog series on virtual proctoring and its appropriateness for use in high-stakes testing (Part 1), I spoke with Mickie Rops. Mickie is an ANSI assessor, who recently recommended accreditation for a program that uses remote proctoring to administer their exams.
I recently interviewed Isabelle Gonthier, an NCCA assessor and psychometrician, and she expressed significant reservations about remote proctoring for high-stakes testing. An alternative accreditation to the NCCA for high-stakes programs is ISO 17024, administered by ANSI.
Given the potential impact that accreditation of remote proctored programs could have for the testing industry, I wanted to know how Mickie and ANSI came to their decision, despite the judgment of other testing professionals and organizations, including the NCCA, that remote proctoring is not yet meeting the standard required for high-stakes. I found her answers fascinating, illuminating, and potentially disruptive for our industry.
Greg Sadesky: Thank-you for agreeing to provide your perspective on this issue! Tell me a bit about yourself and your experience.
Mickie Rops: Well, I’ve been in the certification business for about 25 years; first as a Program Director for a certification body, then started my own consulting practice just over 20 years ago. I’ve worked with certifying bodies, designing and improving their programs. And, I’ve been an assessor for ANSI’s personnel certification accreditation program for almost 15 years.
GS: As you know, I want to focus today on your work as an assessor. You’ve evaluated different types of programs, those with different kinds of “modes” of administration, including virtual proctoring, correct?
MR: Yes, as an ANSI assessor, I’ve assessed certifying bodies that use testing agency proctors and physical testing centres, their own network of proctors and physical testing centres, and those that use remote or virtual proctoring.
GS: And you’ve recently been part of the assessor team that reviewed and recommended accreditation for one such remote or virtual proctoring program.
MR: Yes. Well, technically, it was a program that was already accredited using computer-based testing at a testing agency and at their own network of physical testing centres, and they successfully applied for continued accreditation adding remote proctoring as another test delivery mode.
GS: Wow. I think that’s kind of a landmark decision in our industry. Up until now, were there many programs using remote proctoring that were denied accreditation?
MR: Hmm. I’m not aware of any that were reviewed and declined accreditation. I believe it was more that certifying bodies were reticent to apply using remote proctoring since it hadn’t been approved yet.
GS: That surprises me! I guess I would’ve expected that this acceptance would have only come after a number of unsuccessful attempts by others… isn’t the industry generally reluctant to accept remote proctoring as a legitimate administration medium for high-stakes?
MR: The industry overall does seem reluctant to accept remote proctored testing for certification. But just because it’s new doesn’t mean it’s inferior. That does seem to be a common perception. But ANSI assessors don’t assess based on perceptions; we look at evidence… As an assessor, I see new approaches all the time. Just this week I saw a certifying body use digital badges in place of (not in addition to) a paper or PDF certificate. It caused me pause because 17024 requires a certificate to be “signed or authorized by a responsible member of the certifying body” and we are used to seeing a signature on a certificate. There are no signatures on digital badges. Is that a non-conformity then? I don’t think so. We go back to the intent of the standard and see if “authorized by” can be shown in another way. (Hint: yes it can.) So, just because we are used to seeing something done one way doesn’t mean it has to be. In my view, that something is new may be a yellow-caution flag, but it’s not a red-stop flag. If the certifying body can provide evidence that what they are doing meets the intent of the standard, then they can become accredited.
GS: So, as an assessor, you don’t have any specific statements or guidelines that deal with virtual proctoring?
MR: There is one clause of ISO 17024 that directly speaks to proctor requirements; it’s clause 7.4.3 and it requires a proctor (invigilator) “to be present.” So, from that, we know that the “record and review later” method of remote test delivery (or physical testing, for that matter) would not meet the standard. Beyond that, the requirements are the same for any type of test delivery: a certifying body must provide evidence that its test delivery is secure, that fraudulent exam practices are prevented and that test delivery procedures are consistent with each administration.
GS: That’s interesting because when I talked to Isabelle Gonthier, a commissioner with the NCCA, she felt that additional research needed to be conducted to address particular gaps with remote proctoring, specifically regarding cheating. I take it you don’t entirely share this view?
MR: This is important research for the field, but not so much for purposes of accreditation, at least through ANSI. We look at each case individually, always. Perhaps it’s surprising, but we assessors assign plenty of non-conformities to certifying bodies using traditional methodologies that are grounded in research! Every situation is unique. Not all methodologies work in all situations, and not all good methodologies are deployed appropriately.
Plus, new approaches shouldn’t be held to a higher standard than to what traditional approaches are held. Is physical proctoring really the gold standard to which remote proctoring should aspire? I don’t think so. As we all know, cheating does happen in that environment. And remote proctoring actually has some security advantages over physical proctoring. For example, the risk of collusion between the proctor and the candidate is virtually zero for remote proctoring because a proctor is usually randomly assigned when the candidate signs on. In contrast, the risk of collusion is higher when a candidate can befriend the proctor (or front office staff) who work daily at a global physical testing site.
GS: So, remote proctoring opens up some specific vulnerabilities, but closes others.
MR: Right. And my job as an assessor is not to pass judgment of what method is being used; it’s to determine whether the requirements of ISO 17024 are being met. And again, there is no gold standard for test delivery that I compare against. The only standard that matters is ISO 17024 since that is the standard upon which ANSI accreditation is based.
GS: Okay, so it’s been great to get your perspective on all this! Thank-you so much for taking the time to talk today! I really appreciate your view on this important topic for our industry.
MR: My pleasure!
Virtual Proctoring and the Future of Assessment, with Dr. Greg Sadesky Series
Part 1 | Part 2 | Part 3 | Part 4