Stage 1.  Defining competencies and creating specifications

Whether we’re conducting a job analysis, building a competency profile, or defining a curriculum, our testing experts will consult with subject matter experts, practitioners, and published materials to strategically identify the knowledge and skills that are required for safe,  competent and ethical performance in a given subject area.

We use this knowledge to design an examination “blueprint” (also known as a table of specifications), which identifies the specific knowledge and skills that will be assessed on your examination; and how heavily each area should be weighted on the examination to accurately reflect the breakdown of knowledge and skills required for competent practice.

Our specialized services include:

• Competency profiling and job task analysis
• Surveys and consultation
• Blueprint development

Stage 2.  Item development

If you want to accurately measure someone’s qualifications, having the right number and right type of questions on your examination is essential. Filling an examination with accurate, valid, well-written questions is careful work, requiring meticulous review and revision.

Luckily, our testing experts are available to help you with this, and will coach you through the item development process. This includes training your subject matter experts on how to write effective examination questions; evaluating your draft items to ensure they are fair, clear, and conform to item writing best practices; and project managing the item development process to ensure that the items being written and approved for your examination conform to the target specifications in your examination blueprint.

Our specialized services include:

• Item writing workshops and guides
• Professional item development
• Item development project management
• Item translation services

Stage 3.  Assembling examination form(s)

An effective examination must be carefully designed. It needs to have the right number of questions to effectively assess the subject area; it needs the right breakdown of items to be reflective of the examination blueprint; and it needs to include questions that assess candidates using the right distribution of cognitive levels, question formats, and target populations in order to be reflective of the topic area being assessed.

This may seem like a daunting and exacting process, but our testing experts will work with you to design examination forms that fit these specifications. To do this, YAS draws questions from the item bank that we help you create in order to make sure that any examination forms fit the unique constraints of your examination. We can even automate this process for you using specialized software.

Our specialized services include:

• Automated test assembly (ATA)
• Facilitation of the examination validation process
• Practice test development

Stage 4.  Test administration

Once the examination forms are assembled, candidates finally have the opportunity to demonstrate what they know and can do. To make sure this process runs smoothly, our psychometricians work with our examination administration team to ensure that your examination is administered in a way that securely handles sensitive information, consistent with the stakes (or seriousness) of your examination.

Our specialized services include:

• Paper to computer-based transition
• Collusion detection
• Exam delivery in test centres
• Remote proctoring services

Stage 5.  Item and test analysis

After the examination is administered, our testing experts will review the results to ensure that both the exam, and each item that comprises it, work as they should. In doing so, our testing experts can evaluate the overall reliability of your examination; and identify any items that are too easy, too hard, or fail to discriminate between competent and not-yet-competent candidates. Based on these post-administration findings, we will consult with you to decide whether to retain, revise, or remove questions for future assessments, in order to maximize the effectiveness of your assessment moving forward.

Our specialized services include:

• Classical Test Theory (CTT) item analysis
• Item Response Theory (IRT) item analysis
• Differential Item Functioning (DIF) analysis

Stage 6.  Scoring and reporting

After candidates finish writing your examination, it is important to ensure that their results are calculated efficiently and accurately; and reported back to them as soon as possible. At YAS, our testing experts will work with you to get your examination scored; and will create customized reporting solutions that break down candidate scores based on specific competencies, regions, or other categories of interest.

If you use different examination forms, our testing experts can also statistically equate candidate scores across the different form. This ensures that – even if candidates receive different items – their overall scores are comparable, improving the fairness of the assessment process.

Our specialized services include:

• Redundant scoring using different forms of software to ensure accurate calculations
• Statistical equating between examination forms
• Customized candidate, school, and national results reporting

Stage 7.  Standard setting

Credentialing examinations and other “criterion-referenced” assessments determine a candidate’s success based on whether or not their score on the examination meets or exceeds a set passing mark (or “cut score”). This cut score serves as the dividing line between candidates who are “competent” versus candidates who are “not-yet-competent”.

This is not a decision to be made lightly. A valid, fair, and legally defensible examination requires a well-informed cut score. At YAS, our testing experts will work with your subject matter experts and stakeholders to determine a passing mark for your examination that is informed by the knowledge and skills that your examination is meant to assess, as well as the difficulty of the examination and items that comprise it.

Our specialized services include:

• Consultation and training on various standard setting methods (e.g., Angoff, Ebel, Bookmark, Borderline)
• Facilitation of standard setting sessions with subject matter experts