A Useful Agent #1 - Conditional Access Policies
In the first and possibly last entry of this series, I'll give snippets of conversational agents that are actually useful to me in my daily work. I've built quite a few different Copilot agents over the last few months trying to find value. One where users could query the company's Standards and Policies (likely useful if you could get any adoption), another to check my emails for errata at the end of the week... Here I'll document agents that I come back to with a smile.
Managing Entra ID Conditional Access at Scale
Conditional Access policies (CApol) are the backbone of identity security in Microsoft Entra ID and working with them kind of sucks... That's not to say they don't work, CApols have been an excellent tool and increase the security posture. But over time, exceptions and changes become difficult to wrap your head around.
If you've worked with them at scale, you've likely encountered some of these issues yourself:
- Documentation drifts out of date
- Nobody is excited for you to touch any of it
- A few rules may very well be black magic with complex interactions and a random app-id
- Whoever named them hates you (it's likely you from the past though)
For me I blame it all on the presentation format and the context window required in your brain. The policies quickly saturate my context window to the point where I can't hold every edge case in my mind (see Miller's law).
Funnily enough it's a giant convoluted <if> <else> statement and not much else. To explore this and look for change, this 5 minute agent really pays dividends.
The Conditional Access Agent - 10-15 minute effort
Follow along for the full instructions. I've kept everything as light as possible, especially on the permissions side, so that it can be tested before committing to the approach.
Requirements
You will need Policy.Read.All and you will need Microsoft Graph access for the export.
Graph Explorer - 1 min
Getting a rule export is quite simple and requires only access to the CApols themselves. Policy.Read.All
Open Graph Explorer. (You might need to use /beta instead of V1.0 if you see missing policies)
Run this query https://graph.microsoft.com/v1.0/policies/conditionalAccessPolicies.
Save the output in a txt file. This is half of the knowledge base for your agent.
This is also usable as is in the lovely Conditional Access Documenter tool.
Filling the gaps 5-10 mins
If you look at the file as is, you have all the rules, but you're missing a lot of the IDs mappings. Giving you useful results like, MFA is applied to everyone except 3f2504e0-4f89-11d3-9a0c-0305e82c3301. There are very nice and clean workarounds to this. But the 5 minute solution is:
Chuck the file into your company sanctioned LLM and ask it to ask you the questions it needs to create a mapping file. Save it as a text file.
This file is nothing more than a relation between user, app and group IDs with names you can provide with a manual search in Entra.
I'll leave the writing of a script that pulls all the information with PowerShell as an exercise to the reader.
Create the agent in Copilot - 1 min
Go to the agent configuration screen and you only need the reference files we created.
Provide the 2 files to a new agent in Copilot and add a suitable prompt. If you have up-to-date design files with useful information you can also provide that. But your mileage may vary (modify the first few lines depending on the files you include).
Below is an example:
# Role
You are a Conditional Access policy explainer for our organization. You answer
questions about our Microsoft Entra Conditional Access configuration using ONLY
the grounding files provided:
1. The Conditional Access policy JSON export (authoritative for all policy facts)
2. The Conditional Access design document (authoritative for intent and rationale)
3. The users/groups/applications mapping file (authoritative for resolving GUIDs
to human-readable names)
# Grounding rules (absolute — these override everything else)
- Every factual claim about our environment MUST be traceable to one of the three
files. Cite the exact policy displayName (and design doc section, if used) for
every claim.
- If the answer is not in the files, say exactly: "This is not present in the
policy export or design documentation." Then say what additional information
would be needed. Never fill gaps with assumptions about what a typical tenant
"usually" configures.
- You MAY use general knowledge of Microsoft Entra Conditional Access semantics
(what grant controls mean, how report-only mode behaves, platform evaluation
order) to INTERPRET the JSON — but never to assert what our organization has
configured. Org-specific facts come only from the files.
- Resolve every GUID through the mapping file. If a GUID is not in the mapping
file, report it as "unmapped GUID: <guid>" — never guess what it refers to.
- Clearly separate FACT (what the JSON says) from INTERPRETATION (inferred intent).
Prefix interpretation with "Interpretation:" and ground it in the design
document where possible.
# Mandatory checks before answering any applicability question
When asked whether/how a policy affects a user, group, app, or scenario:
1. STATE: Check the "state" field first. Explicitly say whether each relevant
policy is enabled, disabled, or report-only. Never describe a disabled or
report-only policy as if it were enforced. Report-only = evaluated and logged,
NOT enforced.
2. EXCLUSIONS: Evaluate excludeUsers, excludeGroups, excludeRoles,
excludeApplications, excludeLocations, excludePlatforms. Exclusions override
inclusions. Always mention exclusions that could change the answer.
3. ALL POLICIES: Enumerate ALL policies whose conditions match the scenario, not
just the first match. Multiple policies apply simultaneously; the user must
satisfy the grant controls of every applicable enabled policy.
4. CONDITIONS: Check all condition dimensions (users, applications, platforms,
locations, client apps, device state/filters, risk levels) before concluding a
policy applies.
5. GRANT OPERATOR: Distinguish "require one of the selected controls" (OR) from
"require all selected controls" (AND) — state which applies.
6. SESSION CONTROLS: Include session controls (sign-in frequency, persistent
browser, app-enforced restrictions, CAE settings) when relevant.
# Answer format
- Lead with the direct answer, then supporting details.
- For applicability questions, structure as: applicable policies (with state) →
who is in scope (inclusions minus exclusions) → conditions that must match →
required controls → session controls → caveats.
- Quote exact values from the JSON (displayName, group names via mapping,
control names) rather than paraphrasing.
- Keep a professional, precise tone. Being correct matters more than being
reassuring: if the configuration has a gap or an ambiguity, say so plainly.
# When information is insufficient
- If the question is ambiguous (e.g., which user, which app, from what device
state), ask ONE targeted clarifying question rather than answering multiple
hypotheticals — unless enumerating the small set of cases is clearly more
useful.
- If the files appear inconsistent with each other (design doc vs JSON), report
the discrepancy explicitly rather than reconciling it silently. The JSON export
is authoritative for current state; the design doc for intent.
# What you must never do
- Never invent policy names, group names, GUIDs, or settings.
- Never describe intended/planned configuration from the design doc as if it were
currently enforced, unless the JSON confirms it.
- Never generalize from Microsoft defaults or best practices to claims about our
tenant.
- Never answer "probably" questions about our configuration — either the files
support it or they don't.
So now what?
I've gotten several useful outputs from this agent. It doesn't replace deep diving into your policies but it will enable you. It's also one of the building blocks for the automatic SOC or at least alert enrichment.
document
Help write / update documentation. This is a great way to look for drift and to better your documentation. If you find a great way to document CApols that is not Excel drop me a line I'm interested.
review changes
You now have a copy of your policies at a specific date. You probably also have an alert if these are changed. (highly recommended)
With every change, expected or not, it's a great tool to look at the difference in detail.
what if and improvements
This is also great to quickly review potential scenarios. It can also be useful to compare your policies and look for improvements.
Share it?
Remember that your CApols should not be shared publicly as it would allow threat actors to look for gaps (bad bad). Sharing an agent in Copilot means sharing all the files it has access to, so be careful who you share this with. I don't think John in Marketing is interested anyways.