Are your users really using Microsoft 365 Copilot—or are the built-in reports just telling you they aren’t?
In this interview from the @PowerPlatConf, I’m joined by Microsoft MVP Loryan Strant to walk through why the native Copilot analytics (M365 Admin Center, Copilot dashboard, Viva Insights, etc.) don’t show the full picture—and how his open-source Copilot usage dashboard fills the gaps.
You’ll see how his solution:
✅ Pulls real Copilot interaction data via the Graph API (including apps like Forms, Planner, Stream, SharePoint, and more!
✅ Runs nightly, so you can actually see the impact of trainings and pilots
✅ Lets you combine usage data with survey/feedback data for real adoption insights
✅ Respects privacy, while still giving departments, managers, and execs the data they need
You can find all the resources mentioned here.
Want to learn from more pros? Learn from some of the top talent around the world at M365 Community Conference, and get $150 off with discount code VLAD150! Have questions? Check out my Microsoft Community Conference FAQ!
Watch my 100+ courses on Pluralsight
Video Summary
- Out-of-the-box Copilot dashboards aren’t enough – They only show usage in core apps like Word and Excel, missing apps such as Forms, Planner, and SharePoint—so they don’t give the full adoption picture.
- Custom solutions give deeper insights – Laurian built an open-source solution using Graph API to track Copilot sessions and interactions across all apps, updated nightly, with filtering by department or role for better insights.
- Data tells a bigger story – Usage data alone can be misleading. Combining it with survey feedback on productivity and satisfaction helps organizations understand real value, not just raw numbers.
- Privacy and flexibility matter – The solution avoids storing sensitive prompt data while still allowing for customization. It also enables leaderboards and “laggard boards” to identify low users for coaching, not just license removal.
- Community-driven innovation – Laurian encourages the community to download V2 from GitHub, customize it, and share improvements. Future versions aim to add richer visuals, trend analysis, and even ROI calculations, like time saved converted into dollar value.
For more information, read the transcript blog below, or watch the video above!
Transcript
We all know that Microsoft has been pushing Copilot heavily in recent years, and many companies have started purchasing it, running pilots, but before rolling it out to the whole company, there’s one big pain point: are my users actually using it? So this is why today I’m joined by Microsoft MVP Loryan Strant. Thank you so much for being here with me.
Thanks for the time and for having me on. And we’re at the Power Platform Community Conference. You just had a session that was all about monitoring Copilot, figuring out if people actually use it. And you told me you can just do everything with dashboards out of the box, right? They’re amazing.
They are, but they’re not enough. There aren’t enough. And I didn’t detect the sarcasm clearly enough there. They do a job, but they don’t do a good enough job. And for me, where they fall is that they don’t list all of the apps where Copilot exists. And if you’re not listing all the apps, then it’s not giving an accurate picture. They focus on the obvious ones like Office—Word, Excel, Teams, PowerPoint, Chat—but they ignore Forms, Planner, Stream, SharePoint, all these things. So if you’re using it in those other apps but not the core Office apps, then the out-of-the-box functionality is going to say that you’re not using it much, which means someone will say, “I’ll take away your license, thanks.” But I’m using it, and I am productive.
And I agree with you, what we have out of the box—and I was just joking—most enterprises are like, “That’s a good starting point, but that’s not enough for me to actually decide if I take your license away.”
Yeah, absolutely.
But you found a better way.
Yeah. And I think it’s important to kind of understand what the options are with the different reporting. So I do this comparison with clients, and I’ve also got an extensive write-up on my site about it. But effectively, we’ve got the native stuff. So you’ve got the M365 Admin Center, but that’s limited because there’s certain reporting there, but also it shows the date of last use, which I don’t care about. I want to know how much. So it has some of it, but a lot of times, the Admin Center only shows the last 28 days. It’s starting sometimes in some areas to go further back. We’ve got the Copilot dashboard—that’s good, it’s got some pretty graphs, you know, the things execs need. And then you’ve got Viva Insights, which has your Copilot adoption and a few other Copilot-esque reports, but all of them only show a subset of the data.
So when we’ve got custom solutions, we can customize the Viva Insights report, but I still have the same limitation that the data is simply not there. Or when we look at Graph API, which is what I’ve done, you would think, obviously, go to the usage reports of Graph API, but no, that’s the same as the M365 Admin Center, so not helpful. Or you know, Copilot interactions is a dedicated endpoint in there that I created. And then there are third-party solutions which are basically using the same as what I’ve created. And I’m really good friends with the people at Tiger and Point, so I actually promote theirs, but I basically built my own because I just like to touch the paint to be sure it’s dry.
So, I just needed to kind of prove that, hey, there’s more there. And I think that for me, it’s you need to have the full picture to actually make proper business decisions, not some of the picture. Because if we said to someone, “Oh, we can show some of the Copilot usage. Is that good enough?” They’ll be like, “No.” So why do we have it? So yeah, I wanted to build something better.
Awesome. And what you’ve built, you told me you actually open-sourced.
Yes, absolutely.
Right. So, can’t wait to see it.
Yes, we can. So this is an example of not the actual solution but a version of one that I’ve built, where the first thing—and what I was saying in the session—is the solution that I’ve built, take it and customize it. One of the things is not to just put Copilot data in there. Add your own data from surveys so that you can start seeing that if Copilot usage is going down, well, you usually think that’s not good. But if you add other data in there like “We are finding it more valuable, it’s making business faster, it’s making my day easier,” that’s going up, then this data about the actual usage is not as important. So you need both of those to tell a true story. So the comparison of the two solutions is, you know, there’s the out-of-the-box ones, and then there’s the custom, which, as we can see, is like a lot more apps. So this is actually in a fellow MVP’s tenant that I kind of—I wouldn’t say I built it in her tenant, I probably finessed it in her tenant because I’d already built it two times over. One was using the audit log, which I don’t recommend to anybody to do because firstly there’s too much data that the solution will get access to, and I don’t like it having that because I have to get every activity you did, find the Copilot ones. I can’t filter and query just the Copilot ones. I have to get everything. So it’s heavier to run, but also, there’s too much data there.
So yeah, this V2 one that I’ve got here is showing all of the apps that I can find. I can go as far back as I can. So when you implement it, I don’t just want it from today; I want to go back and get all the data as well. So that’s what this does. It’s got accurate license counts because the other ones are influenced by other factors, whether you’ve got Viva Insights licenses and things. But when we look at what the actual data is, I can see a lot more apps in there. I can see their history, whether they’re growing. But also, I can filter down by different departments, and I can actually filter by whether I want individual users. Now people would say straight away, like privacy—I don’t want to see Vlad has been doing this—and you know what, I completely respect that. The thing i,s the solution needs the data to be able to look at Vlad’s department, Vlad’s location. So whether or not you choose to allow that data to be shown is up to you, but I still need the data in there because what I would envision is organizations would customize this and have the departmental level view or a manager level view or an exec view.
We pull this kind of information in. We’ve got a lot more because I’m pulling other fields from Entra ID. We can sort and filter by those kinds of things. But also, we can start creating leaderboards. And I think what’s important with the leaderboards is that the version that exists out there for the community to take doesn’t have something that I built in the V1 and will bring back in V3, which is the laggard board—show me your lowest users. Okay. Now, it’s because a lot of people will think, oh, the lowest users, you know, the intent is to take their licenses away. And look, that actually may be a valid thing. I see it more as an opportunity to go, okay, who are the lowest users of Copilot? Let’s buddy them up with somebody in their organization, in their team, in their role who is using it really highly, and let’s see if we can actually transfer some knowledge and skills and help them. So I see more of an opportunity to grow the usage as opposed to, you know, take their license away. I think having lots of data is better than having not enough data.
I do have some questions for you. So you said you’re now using data from the audit log.
Correct. Where do you get the data from?
So there is a Copilot endpoint in the Graph API called Copilot. It only works for licensed users, unfortunately. So I don’t get to see the non-licensed Copilot usage. And in there, there’s a call you make called AI Get AI Enterprise Interaction History. And in that interaction history, there is a lot of data, which again I remove a lot of the data because I don’t want it, but yeah, so that dedicated one allows me to query—I basically pass it this licensed user ID, give me their interactions within this date range.
Okay.
And that’s how the solution runs on a nightly basis, whereas the other ones are generally up to a two-day lag to get data. This runs nightly, so I can see, hey, we ran a training session, and now I can see a spike today. So that’s what I wanted is also more current and relevant.
Well, so you know what causes it. Like, there’s a spike, why?
That’s right.
We need to know when the spike is and why it’s happening.
Exactly. Because it’s all about having, you know, keep hearing the term data-driven insights. Cool. Get the data, connect it to other data, and then we have the insight.
And the other question I had for you, you mentioned that you can only get it for licensed users, correct? And right now, I mean, we just talked about it on how the Copilot button is everywhere, whether you like it or not, licensed or not, you have the BIS chat. Well, M365 Copilot chat is not connected to your work data. That would be web chat, which is what it’s called behind the scenes.
Web chat.
Okay. Because you call it BIS chat, no, not you. And it’s referred to as BIS chat, but BIS chat actually is talking with your business data.
Yeah.
So, it’s when you’re looking at the worker web tab. So if you’re a licensed user, you get Worker Web. If you’re unlicensed, you just get web, which is web chat versus BIS chat. And I didn’t know the technical terms. So I learned something. I learned even more stuff right now.
Okay. So, web chat, if I use web chat in Office, can I see that?
Yes.
So I can show you an example of what the data looks like. Whenever we do a query to see what the interactions are, we actually get two. So the first one is the user prompt. And in this user prompt, we can see a lot of rich information in there around basically the ID. We can also see the session ID and the interaction ID because sessions and interactions are different, and then different usage metrics. We can see where they were prompting it from. So were they in an app or were they in chat, as in the chat interface? If they were prompting it from an app, we can also see where the content was that was located. So whether it was OneDrive or SharePoint. If it’s in Teams, we can see if it’s in a chat channel or a meeting. And as you can see in the screenshot, we can also see the actual prompt itself. Now this is relevant that I’ll get to in a sec. I don’t want to see that prompt—for me, that’s a privacy issue. If someone wants to see it from an audit log perspective for security, go do that. My solution is not for that purpose. But if somebody ever wants to adapt it, they could absolutely just go and modify the workflow, and where I have a select action in the workflow that says these specific things, add it back in so that you get that. So yeah, we can see whether there was an attachment related to it. So a whole bunch of context there. The other one we get is the AI response. So I discard that one entirely because otherwise I end up with you, Vlad, were in a Word document and you asked it a question, and it responded. That’s two interactions.
Okay.
So, but that’s not actually how many times you used it. So, because of that, I actually discard the response entirely.
Okay.
Because then I get the how many times did you use it, not how many times was there an interaction.
And just to take a step back, because you explained this to me yesterday, the difference between a session and an interaction. So I’m going to try to explain i,t and you tell me if I do a good job.
Okay.
So every time you open up, let’s say Word or PowerPoint, and you open up the Copilot on the right side, you open up a new session and then every time you ask Copilot, “Hey, change this paragraph, do this, add a joke in here,” every time you write to it is one interaction.
Yep.
But as long as that Word document is open, we’re still in that same session.
Yes.
If I close the Word document and then reopen it in an hour, that creates a new session.
Yes.
Right.
Capture it perfectly.
Awesome. So, it’s important to know the difference because sometimes you look at the sessions and some people think that’s how many prompts they did.
Yeah. And the other thing as well is everybody works differently, right? So the way that I work with, for example, just using the M365 Copilot chat as opposed to in an app is when I’m asking it about a thing, and then I move on to the next thing, I will create a new conversation window because to me it’s a different context, different things. I don’t want it to be done. Whereas other users might not. They may just keep going with it. So that to me is two sessions, one had a few interactions, one had a lot. The thing is, though, what does sessions versus interactions mean? That’s subjective because what I genuinely expect will happen over time is that we’ll see sessions and interactions. We might see sessions going u,p but interactions going down. And you’re like, well, hang on, which one’s the good metric? I don’t know. And it’s very subjective based on the app itself, but also, what it could actually be is that they’re getting better at prompting. So instead of “I went in and I said create a PowerPoint slide that’s got this information and then add a banner and then put a picture and do this,” and before I know it, I’ve got 10 prompts just to create one slide. Instead, as I get better as a user, I would go in and say “Create a slide that is talking about this, that has a banner, that’s got this, that’s got a picture of a person showing this, that this is this, this” and instead of going 10 prompts, I’ve now got two, maybe three.
So interaction isn’t always a bad thing.
That’s right. And that’s why that slide that I showed before, about having the qualitative data about people saying, “It is saving me time, I am getting value out of it,” is critical to basically offset that so you can tell a clearer picture. Now, we’re going back to the data that’s shown here; there’s a lot of information that’s shown here that I don’t store. What I actually store is a very much cut-down version of it. Now, without getting into the details of the behind-the-scenes stuff, it’s still quite messy, and I store it as JSON blobs in a database table because I pick it up in Power BI and then do things with the JSON blob. But here, though, you can see I’ve got a file location field. I’ve got a Teams location field, but also the app class is, you know, IPM.Skypeyp.teamsmessage.compile.outlook. What does that mean? Outlook. So that’s where, in Power BI, I clean all of this up and then group it by different things, which gives me that was two interactions in one session. I’m not recording the response at all. I’m not recording the prompt. I am purely recording the context, the location of it, and yeah, time, date, and basically just data points. So that’s it.
Awesome. Man, Skype is going to be there forever, I think.
I honestly don’t know why that’s even in there now.
Probably using some of the old APIs from Skype and things like that.
I do have one question for you. So I see OneDrive here for an example, and you said you told me you can see SharePoint as well.
Yes.
Can I see what site exactly it references, or what sites?
You absolutely can. I don’t store that because again, from a privacy perspective, but that’s why the workflow will basically summarize that where it says if it sees hyphen-my.sharepoint.com, SharePoint.com OneDrive, if it sees SharePoint.com without the hyphen-m,y then sorry that yeah so OneDrive or SharePoint and that’s all stored. Now that’s all I’m storing. If someone wanted to modify the workflow and change that and store the specific site URL or those kinds of things, they can actually.
I’m just—I’m wow, I’m mind blown right now. So I’m thinking in theory, if let’s say I saved everything, I could create a map of every user and what SharePoint site they accessed.
Yeah.
So I can also say, “Hey, I want this SharePoint site. Who accessed it with AI?”
Well, that’s the thing.
With Copilot, I could if I wanted to.
You absolutely could. And that is actually part of the reason why I do record whether it’s SharePoint or OneDrive. And also with Teams, whether it’s a chat, channel, or meeting, is to be honest actually not for Copilot’s benefit. It’s actually for a broader adoption benefit because if I see that most people are interacting with files in OneDrive, then that raises a different alarm bell, which is that our people probably aren’t putting files in the right place. Or if they’re interacting with chats and meetings but not much channel, then again, maybe they’re not using the channel functionality in general as opposed to just in Copilot. But I mean, I’ve got other solutions where I pull all that other data out. And what I would love is for people to then connect the dots and again, a clearer picture, more context, you know, so that every company will have unique needs. So that’s the cool thing. You can decide what you build with this.
Absolutely. And the thing i,s what I’m trying to work on is so what I’m showing here, this screenshot, these bits of data, these are actually from the V3 that I’m working on.
Okay.
Where because currently V2 doesn’t have these because there was less data in the API.
Okay.
From that Copilot interaction method. Now it’s there. So now I’m going great, I’ve got more information I can put in. Let’s put it in there. And then I’m also trying with the V3 version, which is actually to put in more data about the sessions and interactions to show that. So I show you an example of what I’m trying to get to. It’s rough. So I show you an example of what I’m trying to get to. It’s rough, but I love it. So this is an example of V3. Now it’s not really telling us much, but the thing is what I’m showing, though, is sessions and interactions together because I intend to show that maybe I can see more sessions, fewer interactions or more interactions, fewer sessions. And I think this is where people have to then look at how the users are doing things and again use that survey data to go, okay, maybe people are finding that—I don’t know—that maybe using Copilot in Excel is actually not good. You know, so it’s a case of not good or maybe they don’t know how to use it, or maybe they need to have the expectation set as to what it can and can’t be done. So that’s why I’m just starting to play with that. I thought I’d put some spider radials in because I had a client go crazy for them. So I thought I’d put that in. Trying to put some averages in there as well. And the other one as well is with the solution. This is where I’m again putting sessions and interactions together and then trying to draw a trend line because, again, now this is off a demo environment of five users. So really hard to—and we tried hard to simulate usage. But yeah, this is showing all those different apps. And what I want to see is, as I said, maybe I can see the trend line for, you know, if I look at like OneNote, right, it’s very slowly going down, but you know, the usage is actually potentially so high. So I’m not a Power BI expert, so this has a lot of work to be done, and that’s why I put it out to the community, because hey, make it better, share it with me, everybody else can benefit from it. But yeah, and also those logos pull from my Microsoft cloud logos repo. I see that SharePoint and Teams have been updated. Outlook is updated.
All updated because—and even that Forms one, I updated the logo on it yesterday, and I hit refresh on my report, and it updated. That is cool. So yeah, anybody who deploys the V3, if, for example, the Whiteboard logo gets the same treatment as the Stream logo, the next day after I’ve updated it, they will get it.
I don’t know how I feel about the ones that didn’t get the new logo. Are they?
I feel like, you know, Microsoft has favorites.
Oh, absolutely. The fact that Forms actually got a new logo, I was surprised about because yeah, they’ve left them behind previously. So yeah, but that’s the thing is I thought that could make it richer. And the thing is, people don’t have to do it in the same layout I’ve got because, as I said, I’m not a Power BI expert. Like I was playing with another layout here where I’m trying to try, you know, the two bottom rows, those are actually the same data but different graph types. Yeah. Because, you know, this is where I encourage people in the community to help out with this because, hey, if you’re a Power BI visuals expert, great. Help me build better visuals because, you know, there’s an MVP buddy of mine in Melbourne, Greg Nash, who, a few times, has criticized my overuse of pie charts. And the thing is, I’m not a BI person. It’s a means to an end for me. So, you know, if there are better charts, let’s use them. Yeah. I’m just doing this because that gives me the data that I want.
You should ask Copilot. Copilot, here’s my data. Give me the charts.
Actually, I use Copilot a lot with Power BI stuff where I say, I’ve got this and this,s and I want to do this, and it will say Go create a measure for this. Here’s the code. There are other things that I’m trying to do beyond V3, where I’m trying to add in some additional functionality. So, I want a standalone report for execs. So, they don’t see all of the other pages. They just see the big ones with the big graphs and the things going up and the satisfaction rating. So, like some of the ones that I showed before, I want a manager level one so they can see their team, how their team is performing. But also,o I want an individual one, like how do I stack up against other people in my team without seeing the details of their names, obviously or other people in my country or other people with the same job title as me. And I think that’s the thing is with all this data, it’s completely up to the person who’s implementing it if they want to provide that. But that’s what I want to do. And yeah, I want to also get more trends and things and data science stuff because I just don’t know. There’s only so much I can do when I’m figuring out what actually means behind the scenes, trying to figure out what the Graph API response is, because there’s a bunch that aren’t in V3 yet, because I don’t know what they mean. Okay. As in the word, for example, when I pretty up the name, one of them is Viva Goals. And I look at that—it’s gone. Exactly. So, how am I getting Viva Goals data? There’s—I’ve seen Copilot Search has appeared, which is now the new search interface. There was another one that was simply Viva Copilot. I think it might be Copilot for Sales, because that was originally Viva Sales. So I’m like, god, this is what goes into this. I’ve figured it out, and I guess none of it is documented. Not really. No, those are not documented. Some of them are, but that’s where I basically bash my head against the wall, try to figure it out and make it so that when you open it up, it says Whiteboard or Engage or yeah or like BIS chat, web chat—it doesn’t say that. It says chat, work chat, web. That’s what I’m doing.
Yeah. So this is exciting.
It’s rewarding but frustrating that I have to do it, but it’s rewarding when it helps. So with Lindsay, my co-presenter that I did this session with today, this flat out resulted in them buying an extra 2,000 licenses over their 10,000. So that’s the thing is for me if anybody from Microsoft—and I said in the session hey if you want to have an issue with the fact that I’m saying potentially, you know, not positive things about the tools and don’t they have lots of value, lots of merit—but this is resulting in actual license purchases then hey and you know what if Microsoft wants to take it and incorporate it I’d be happy, please. So yeah, that is awesome. So I’m super excited. Thanks.
How can somebody get started today?
So, there is the V2 that they can already download, which has been on my GitHub since February, when I first did a series of blog posts, because what I actually have is a dedicated Copilot usage landing page that has four different blog posts of me ranting but explaining things. Which, to be quite frank, I’m sure people will just open it up, Copilot, and Edge and whatever, and just go summarize, cut out Loryan’s rantings. But yeah, it talks about the challenges with the Viva Engage data, the challenges with and the different methods. But fundamentally, there is one that says how to build your own. And there is another one that says how to implement mine with step-by-step instructions. I have had people reach out to me who are not technical at all, who are the adoption lead for a major global airline, like one of the top five, who has had basically very little support in terms of her team’s availability, and she has deployed this herself.
Wow. Because I have provided the files, there is a zip file that is the solution for all the workflows, all the tables, everything. There is a Power BI template, so effectively that blog post you follow the steps to provision and import the solution, run these things in order, then you open up the Power BI file, it’ll say what’s your host address, which you get from the instructions, you put it in and then it sits and refreshes, and you have data. Wow. You hit publish and set a refresh cycle; you’re done. So all you need is really Dataverse capacity and a Power BI Pro license to consume the report from a requirements—and I know that’s probably in the blog—that is in the reading part. So yeah, effectively all people need is a Power Automate Premium license.
You know, you could use an existing service account. I would suggest not using a dedicated one. It’s not exactly a high cost because you don’t want to have throttling from other workflows doing things. Similar to the Power BI Pro license, more so not so much from a throttling perspective, but if you, Vlad, do this and you publish the report, and then you leave the organization, the refresh will fail. So put it under a service account. Yeah, I recommend that people also create a dedicated Power Platform environment to put it in, also from a security perspective and lock it down, put the report into a specific workspace. But that’s it. Yeah, 29 bucks US a month. That’s amazing, and you get this solution.
That is awesome. And for everybody, the link will be in the description below to the series of blog posts on the solution as well. This way, you can quickly get to it. Thank you very much. Thank you. I will need to try it. I’m a bit, you know, my demo environment is me and five users, a bit like you, but I want to play with it. I want to play with the API. I want to see what all the data you can do. Probably, my reports are not going to be as good as yours. Mine are not fantastic. So, you know, don’t beat yourself up. One of the questions I got in the session was around whether I use the prompt data that people have typed in, like basically get some insights as to what they’re prompting, which is an amazing idea.
And sorry, now I’m sitting thinking of building a different solution. You’re giving me ideas on stuff. Well, because I don’t record the prompt data, but I could create a different solution that records just the prompt data in some secure vault and then runs like Azure OpenAI at it and then basically summarizes what people are asking for, where they are doing it, that kind of stuff further. Every meeting summary, you assign it a dollar value, and then you can actually get a dollar amount on your Power BI report for every 30 minutes of meeting that you summarize. You save 20 minutes, add a dollar value to that, and then you can actually calculate how much money Copilot saved you. That’s a brilliant idea. That just came to my head now that you mentioned that. That’s fantastic. No, that’s—and this is why I put it out to the community, because I didn’t want to just do a blog post and go here’s how you do it. I want to say no, here’s mine. Use this as a step up, then go build your own, and please share it back with me so that we can add, and you know, and others can benefit from it. So it was good in today’s session that there were like hundreds of people there, like snap snap snap snap snap, like take it please let’s make this better together. So yeah, that is awesome
I definitely have homework now. I’ll add it to the list, but thank you so much for your time. This has been very, very valuable for everybody watching. You’ll have Loryan’s socials blog everything in the description. Please connect with him,m he always shares a bunch of awesome and useful stuff as well for everybody watching.
Thank you so much for your time. If you’re looking for more news from the Power Platform Community Conference, you’ll have one appear on the screen right now. And if you enjoyed the video, please like it and subscribe to the channel. It means a lot to me. And see you all in the next one.
