Microsoft Certification Myths ANSWERED with Microsoft's Liberty Munson

There are so many Microsoft Certification Myths out there! So, I asked Liberty Munson from Microsoft to come back on my channel to answer your top certification myths!

Looking for more content on a Microsoft Certification Exam, Microsoft 365, and the Power Platform?

Watch my 50+ courses on Pluralsight

Video Summary

Here are the key points from the video:

• Exam Myths Debunked: Liberty Munson, Director of Psychometrics at Microsoft, debunks common myths about Microsoft certification exams, clarifying misconceptions about question difficulty, scoring, and the purpose of pre-exam surveys.
• Question Scoring: Not all questions are worth the same points. Points are awarded based on the number of correct actions or answers within a question, making the scoring more nuanced.
• Adaptive Testing: Microsoft no longer uses computer adaptive testing due to the complexity and rapid changes in cloud technologies, ensuring a consistent experience across exams.
• Open Book Exams: The term “open book” might be misleading. It actually refers to having access to online documentation during the exam, not unlimited resources or time.
• Certification Development: Microsoft certifications are developed based on market needs and job roles, not just marketing pushes. A governance council evaluates the value and demand before creating new certifications.

For more information, read the transcript blog below, or watch the video above!

Video Transcript

Taking an exam is never an easy task, whether it’s at school, at work, or a professional certification such as a Microsoft certification. As a Microsoft Certified Professional for over 15 years now, I’ve heard many rumours, tips, tricks, and even conspiracy theories around the world of certification. Some of them are pretty realistic, while others are a bit more far-fetched. Today, we are here to separate myth from reality.

We have an expert here today to help us debunk the most common certification myths that we could find. We have Liberty Munson, Director of Psychometrics at Microsoft, here with us. Hi Liberty, thank you so much for joining us today.

Thank you for having me back. I’m looking forward to busting some myths with you. This will be fun.

Well Liberty, since this is the second time you’re on the channel and the last video about open book certifications had over 5,000 viewers to date and it keeps growing every day, and of course, you’re famous for everybody following anything related to certification, so instead of a normal introduction, can you please tell us why you and your role as a Director of Psychometrics at Microsoft make you an expert on this topic?

It is my job to ensure that whatever tool or process we use to validate skills is valid and reliable. That means are we asking the right questions, are we asking them in the right way, and when we start talking about performance kinds of stuff, are we designing the tasks the right way, are we scoring them the right way? My job is really to ensure that whatever the outcome is from the certification process, it has meaning, it has value, it’s fair, and it’s reliable. Every quality certification program or any type of credentialing program should have a psychometrician actively involved because we’re here to make sure that the experience is the right experience to ensure that you’re truly qualified and have truly earned whatever that outcome is—in this case, a certification that means you have demonstrated the skills that you need to be successful in a job role that uses Microsoft Technology Solutions. So, psychometricians rule!

And you’ve been in this role for 15 years if I remember correctly, right?

I am going to be at Microsoft for almost 17 years in January. I’ll be here 17 years, so just longer than you’ve been an MCT.

Wow, awesome. Okay, let’s start with our myths. The first one is, you know, at the beginning of each exam, for those of you watching, you have it on the screen, and we get some questions about our expertise on the topic in general. So if I pass an Azure Administrator exam, I’ll get some questions that will ask me, “Hey Vlad, for how long have you been an Azure admin?” And then for each one of the exam objectives, it will ask me, “Okay Vlad, how do you feel about, let’s say, managing virtual machines? Are you an expert or have you never done this before?” Now, the word on the street or on the web, I guess, is that the answers you give to those questions will affect the questions on the exam. So if I say I’m an expert in everything and I have 10 years of experience in Azure, the questions I’ll get on the exam will be more difficult. Now Liberty, is this true? Can it even be true with the way that the questions are chosen? Can you shed some light on that?

I have heard this myth before and I’m so glad to be here to bust it because it is absolutely not true. Those questions have no bearing on the questions you’ll see on the exam. That survey is there so that I can actually evaluate the cut score to make sure it’s set in the right place. So if you don’t tell me the truth, I may end up setting the cut score too high as a result. So do answer those questions honestly because it’s one of the many ways that I check to make sure that we’re setting the cut score—or I guess I should say passing score—in the right place. It’s very much an art and a science on how we get there because Microsoft doesn’t set it arbitrarily. It’s a conversation that we have with subject matter experts who help us understand if you think about all the items in the item pool, are they overly difficult for the target audience, are they too easy for the target audience, and they kind of help us balance where the cut score should be given the difficulty of the item pool. Although we scale the exam to 700—that’s the cut score across all of our exams—that doesn’t mean 70%. Sometimes it does, but that’s kind of an accident that happens. It can vary and it does vary because sometimes our item pools end up being a little more difficult than we intended them to be for the target audience and sometimes they’re a little bit easier than we intended them to be for the target audience. So we leverage where we set the cut score to make sure that it’s a fair evaluation of the skills given to who we are targeting the certification for. Those questions after we launch an exam help me go back and make sure that we have the cut score in the right place because over time people get more familiar with our technologies and we have to make certain adjustments to the cut score to make sure it’s still targeting the right audience we intend to earn the certification.

So if everybody says they’ve never done Azure before and they scored in the 800-900s because in fact they have years of experience, that will make the exam more difficult for everybody?

It will increase the cut score, but the difficulty of the exam stays the same. You’re just less likely to pass.

Wow, that’s something to think about whenever you get those questions. Do not lie because it can affect everybody else in the future. Awesome. Our second myth and this one is also on exam questions, is that Microsoft purposely inserts fake questions just to confuse candidates and make them fail exams. This way, Microsoft can make more money.

I haven’t heard this one, but this is a good one. No, all of my questions on the exam have a purpose. Some of them aren’t scored because we do have to pilot test some of the items, much like a beta. Once an exam goes live, we are constantly writing new items, and so we’ll put items on the exam that aren’t part of your final score so we can collect the psychometric data we need to evaluate them. But every question has a purpose. Every question is aligned to an objective. One of the things I will say is I do get feedback from people saying that questions are too tricky, and I take that quite seriously because tricky is bad. We never want to measure something that is kind of a niche case, a corner case that only experts would know the answer to. So if I see people saying it’s tricky, I always go back to my subject matter experts and I ask them, “Is it tricky or is it difficult?” There’s a difference. It’s okay for a question to be difficult because it’s something more people with more expertise would potentially be able to answer. It is not okay for a question to be tricky because it’s something that is so niche or a corner case that very few people would know it. My advice related to this one is I’m not trying to trick you. I’m not putting fake questions on the exam. If you think that, you have probably read more into the question than you should have. So take a step back, say, “Liberty wouldn’t do this to me,” reread the question, and think. Just take what’s in the question and the logical assumptions you would make about what we tell you in the question apply those logical assumptions and answer the question. Don’t say, “Oh, Microsoft’s trying to trick me,” and then try to figure it out from there because you’re going to get it wrong because I’m not trying to trick you.

And I guess if you find that a question is more tricky than difficult, make sure to mark it in the comments whenever you have the opportunity after the exam to give comments to the team. Some of them go to you, and some of them go to other teams at Microsoft, but make sure you mark it in there because I’m sure if 80% of the people give the same feedback, that question will end up for review at some point.

Yes, absolutely. Great feedback.

Awesome. Okay, our third myth is that all of the questions in a Microsoft exam are worth the same amount of points. Now, this one is interesting because it depends on how you define a question and it depends on how you define a point. We’re going to get into a little bit of psychometric conversation here, but originally, when I first started in this role many, many moons ago, all of our questions were worth one point regardless of how many actions you had to take. So if you answered a multiple-choice select one, it was one point. If you answered a multiple-choice select three, it was one point. If you did a build list where you had to move five objects, it was one point. About four or five—oh, it may be longer than that; I’ve been here a long time—let’s just say a while back, it occurred to me that maybe that wasn’t the best way to get the right kind of psychometric information I needed about questions because what I was seeing was people were skipping questions that required them to spend more time because it was only worth one point. So at that point, I decided that I was going to change the way we thought about scoring questions. Every action you take in a question, you get a point. The only exception is the build list because those are super complicated in the way we have to score them. So every build list question is still only worth a point. But if you do a multiple-choice select three, you get a point for every correct answer that you select. So you can get zero points, one, two, or three points depending on how many of those answers you select. If you do a drag-and-drop, then you have to drop four sources into four targets.