CEdMA Blog

Certification Exam Item Writing Best Practices

Posted by Joe Cannata on 1/7/2018

By Joe CannataCertification Director at Kinaxis

How you construct your exam items says a lot about your certification program.

Your goal is to produce items that measure what you intend them to measure, yet also be professionally written. If the items on your exams contain grammatical or spelling errors, it gives the impression of a lack of quality, and the candidate wonders how serious the certification program is overall.

In addition, your items directly reflect upon your organization and your company. Websites, documentation, product help files and training classes, any assets that are outwardly facing, need to look and show properly. They must give the reader confidence in the creating entity. If I “writed” this blog with poor “grammer”, spelling, and “tpyos,” you would stop reading here; my credibility would be gone and this would be a waste of pixels.

Since certification exams are a “product,” they deserve the same writing governance as all other assets your company puts forth. Anyone that leads exam development as a living has their individual ideas about what is “good and proper;” opinions vary. However, there are basic rules that all programs should follow when constructing exam items, regardless of the industry or application.

This blog describes the more common exam writing rules, obtained from my experience networking with many exam workshop leaders across several companies, and represents the rules I have my SMEs follow with the exams we develop at Kinaxis.

In no particular order, these are best practices to follow for your exam items, with my comments in italics:

  • Ensure each item is technically correct and accurate, with no room for debate.
    • Try to find ANY possible way someone could argue for an answer other than the one you deem correct.

  • Support each item with a customer-facing reference document
    • To be fair to the candidate, you should test on content that could be researched or learned in a training class.

  • Write meaningful, job-related content; stick to the blueprint.

  • Items should be congruent to the objectives in your exam blueprint.

  • Do not use jargon, slang or cultural references and also avoid using internal company jargon.

  • There should be no trick questions.
    • You are testing someone’s skills or knowledge, not sending them on a mental amusement park ride.

  • Be a minimalist, remove extra wording where possible. (Remember your English as a second language (ESL) candidates)

  • Do not teach with the exam; ask questions and don’t tell facts about your product(s).
    • This happens often, an item writer chooses to show what they know instead of trying to find what the candidate knows.
  • Ensure your stem or scenario does not give a clue to the answer.

  • Keep items independent of one another; the delivery order is typically random.

  • Include all information required to answer the question.
    • The candidate should answer the question based on the information provided and make no assumptions.

  • Avoid using compound answers.
    • An example of this would be “red and blue” or “menus and radio buttons.” If this is a “choose two” question for example, a candidate could argue that by choosing “red and blue” they chose two answers.

  • Focus on one concept per item.

  • Avoid “not” questions or negative phrasing in the scenario or stem.
    • This is like tossing out a mental hand grenade or a speed bump at the candidate. Instead of asking, “Which feature is not supported?” turn it around and ask “Which three features are supported?”

  • Do not use words like “always,” “never," “best” (unless “best” is a documented “best practice”).
    • These are dead giveaways that an option is not correct. There are few instances when you can guarantee that an “always” or “never” condition will not have an exception. It’s not worth the time to try and cover all bases.

  • Never use “all of the above”, “none of the above, “only B & C”.
    • This is just being lazy, or an item writer not being able to create some distractors. Also, make it a “choose two” question if there are two possible answers.

  • Eliminate the phrase “Which of the following…”
    • Oh, if I only received a penny for each time I’ve seen this, I would be very rich. I’ve seen this used with single answer multiple choice questions, and it just adds extra words. On a multiple answer question, you should tell them how many correct answers are required.
  • Double cue multiple answer items.
    • For a multiple answer item ask, “Which three file types…? (Choose three.)”
  • For multiple choice questions you use one of these variations: 1 out of 4, 2 out of 4, or 3 out of 5.
    • I have seen 6 out of 14, which was totally ridiculous. While candidates may not like multiple answer questions, especially if there is no part credit, sometimes they are necessary, and they do raise the bar.

  • Make your distractors plausible and parallel, and do not make up things that are not part of your product or software.
    • You do not want the candidate to have better than a 25% chance of guessing an item correctly. Also, fabricating commands, part names or features is weak. If there are not enough plausible distractors available, then abandon the idea for that item.

  • Avoid asking trivia questions.
    • Here I mean that a technical professional knows how to find information in a manual or use a help file. Ask them about something they should know, and should not have to research. Yes, sometimes a command or some aspect of your product should be known, but do not go into trivial things such as command options, or have them memorize a GUI. It would be better to show a graphic of the GUI and ask them to identify some aspect, or use point and click.

You may not agree with some of these suggestions, and that is fine. As a program leader decide for yourself how your exams will be presented, and which style guidelines you will follow. My goal is to present industry best practices and to maybe to stir up some debate. Please feel free to leave a comment at the bottom.

Joe Cannata

Certification Director at Kinaxis

Board of Directors Membership Trustee, CEdMA

Topics: certification

Welcome to the CEdMA Blog

Our blog is built to explore new insights of software training. We invite you to comment, ask questions, and engage with us. 

 

Subscribe to Email Updates

Recent Posts