Responsible AI operating review
Everything you have learned comes together in one place: designing a real, bounded use of AI and being able to defend it. This module walks you through that design step by step, then helps you attack your own plan before anyone else can. The result is a pilot you understand well enough to stand behind.
By the end, you will be able to
- Define a mission problem and describe how it is handled today without AI.
- Map the people, data, authority, and possible harms in your pilot.
- Apply a risk level with matching controls, human review, appeal, and stop conditions.
- Choose honest measures and write a pilot plan you can defend.
Lesson 1
Start with the mission problem and the current baseline
A good pilot begins with a real problem, not a tool. And you cannot judge improvement until you know how the work is done today.
Pick one specific problem in your work where AI might help. Not a broad ambition, a concrete task with a clear edge. The narrower it is, the better you will be able to design and defend it.
Then describe how that task is handled right now, without any AI. Who does it, how long it takes, and where it gets stuck. This is your baseline, and it is the honest thing you will compare against later.
Vague: use AI to help with our programs. Bounded: draft first-pass replies to routine intake questions, which one staff member currently writes by hand, taking a hypothetical two hours a day and often delaying families who are waiting.
Note
If you cannot describe the current, non-AI way clearly, you are not ready to add AI to it yet. Understanding the task as it stands is the foundation for everything that follows.
Try it
Write your mission problem in one sentence, then describe today's baseline in two or three more. If either feels fuzzy, tighten it before moving on.
RememberStart with a narrow, real problem and an honest picture of how it is done today. That is what your pilot improves on.
Lesson 2
Map the people, data, authority, and possible harms
Before you design controls, you have to see clearly who and what your pilot touches, and how it could go wrong.
Now map the ground your pilot stands on. This is the same thinking from the foundation modules, applied to your own case. Take each part in turn and write down what is true for your task.
- Affected people: who is touched by this, including the people you serve and your own staff.
- Data: what information the task uses, and how sensitive it is.
- Authority: which rules apply, from law and contracts to your own organization's policy.
- Dependencies: what tools, vendors, or systems the pilot relies on.
- Possible harms: what could go wrong for a real person if the AI is wrong or misused.
Important
Do not rush the possible-harms step. The failures from earlier, a fabricated fact, a biased result, a private detail exposed, are exactly what you are looking for here. Naming a harm now is how you design to prevent it.
Try it
For your own pilot, write one sentence under each of the five headings. The harms line is the one to sit with longest.
RememberMap the affected people, the data, the authority, the dependencies, and the harms before you design anything.
Lesson 3
Set the risk level, controls, and stop conditions
The map from the last lesson tells you how much care this pilot needs. Now you turn that into concrete protections.
Use what you mapped to place your pilot on a simple scale of care. This is the same tiering idea from the governance module, compressed to a few levels for a single pilot. A low-stakes task that only shapes language needs light handling. A task that touches a person's outcome, like eligibility or a service, needs the strongest protections and may not be a fit for AI at all.
- Risk level: from your harm map, name whether this is routine, sensitive but manageable with controls, or off-limits.
- Controls: decide what safeguards match that level, like synthetic data, verification checks, and approved tools only.
- Human review: name the person with the authority, information, and time to actually change the outcome.
- Appeal: give affected people a real way to ask why and to challenge the result.
- Stop conditions: decide in advance what would make you pause or end the pilot.
Try it
Write your pilot's stop conditions first, before its controls. If you cannot name what would make you stop, you do not yet understand the risk well enough.
RememberMatch the level of care to the harm you mapped, keep a real human owner and an appeal path, and set your stop conditions in advance.
Lesson 4
Choose honest measures
A pilot you cannot measure is a pilot you cannot judge. The measures you pick decide whether you will learn the truth.
Decide how you will know whether this pilot is actually working, and be honest about more than speed. A tool can save time and still cause harm, so your measures have to look at both the benefit and the cost.
- Mission value: is the actual mission problem better, not just faster?
- Quality: is the work good enough, and how much fixing does it need?
- Accessibility: does it work for everyone it touches, including people who need another path?
- Equity: does it perform evenly across the different people it affects?
- Cost and adoption: what does it cost, and are people actually using it well?
- Incidents: are near misses and problems being noticed and reported?
Important
Beware of measuring only output volume. More drafts produced is not the same as more mission delivered. If your only measure is speed, you will miss the harms that matter most.
Try it
Pick the two measures that matter most for your pilot, and make sure at least one of them is about quality or harm, not just speed.
RememberMeasure benefit and cost honestly, including quality, equity, and harm, so the pilot tells you the truth about itself.
Lesson 5
Pressure-test your pilot and decide
Before you would ever run a pilot, attack it yourself. The plan that survives your own hard questions is one you can stand behind.
Now play the toughest, fairest critic of your own plan. This is where the whole academy pays off. Walk through your pilot as if you were the person most affected by it, and as if you were a careful board member deciding whether to approve it.
- As an affected person: what would worry you, and could you find out a decision was made and challenge it?
- As a careful reviewer: where is the data handling weak, and is the human owner real or just a rubber stamp?
- As a skeptic: what is the strongest reason this pilot should not proceed as written?
Then make an honest call. Your options are to proceed, to revise the plan first, to pause and gather more information, or to decide this is not a fit for AI. All four are legitimate. Choosing not to proceed, when the risk is real, is a sign of good judgment, not a failure.
Try it
Argue the strongest case against your own pilot for two minutes, out loud. Then write your decision and the one condition that would have to be true for you to feel good about it.
RememberAttack your own plan from every side, protect data, authority, escalation, and affected people, and then make an honest go, revise, pause, or stop call.
What you leave with
Board-ready 30/60/90-day responsible-AI pilot charter
A short, defensible charter for one bounded pilot. It names the mission problem and baseline, the people and data and harms, and the risk level with its controls and stop conditions. It ends with honest measures and your decision. It is the single document that shows you can put the whole method to work on real organizational work.
This is just for you. It saves on this device only, and nothing is scored.