Measuring AI by Outcomes Instead of Output
AI makes output nearly free, so counts of emails, reports and tasks rise fast while results stay put. How a small firm sets one outcome number before it switches an AI tool on.
Most firms measure what's easy to count instead of what matters, and AI makes that habit more expensive. Output is easy to measure. A team can count tasks done, features shipped and hours logged, and those numbers give a clear picture of activity.
Activity looks like progress, and that's the trouble. Outcomes are harder to measure, because they ask whether anything actually improved for the business or the customer. When output takes the place of outcome as the main measure, a team gets better at moving and no better at getting results. Nobody decides this on purpose. It happens because the output number is ready every Friday and the outcome number takes work to find.
What changes when AI arrives
AI makes output close to free. A team with an AI helper can write three times the emails, produce a report in an hour instead of a day and turn one meeting into five pages of notes. If the firm measures output, every one of those counts goes up in the first month, and the AI looks like a great success.
Take a hypothetical sales team of four at a small distribution firm. Before AI, they sent about 40 follow-up emails a month after trade enquiries. With an AI helper drafting them, they send 120, and the report for the owner shows the number tripled. The number of enquiries that turned into a first order stayed at 5 a month, before and after.
So the output tripled and the outcome didn't move. The firm counted the wrong thing, which means it can't yet tell whether the tool helped, hurt or did nothing at all. Some of those extra emails went to people who were never going to buy, and some regular customers could have noticed that the messages started to sound alike.
Define success as a change in state
The operators who get the most from their effort define success as a change in state, not as more activity. A change in state is something about the business that's different afterwards and stays different. It answers the question "what's better now?" instead of "how much did we do?"
A few examples of the difference, all hypothetical:
- Output: 200 customer queries answered by the AI helper this month. Outcome: the share of queries answered on the same day went from 50 percent to 80 percent.
- Output: 30 reports produced. Outcome: the owner's Monday review takes 1 hour instead of 3, because the numbers arrive already checked.
- Output: 120 follow-up emails sent. Outcome: enquiries that become a first order went from 5 a month to 8.
Each outcome names a number that existed before the tool arrived, and that's the useful habit. Before a firm switches on an AI tool for a piece of work, someone writes down the outcome it's meant to change and what that number is today. A month later, the firm checks the same number. If it moved, the tool is earning its place, and if it didn't, the firm has learned something cheaply that the output count would never have shown.
Why this is harder than it sounds
The output number is always easier to get. It sits in the tool's own dashboard, it rises quickly and it looks good in a meeting. The outcome number usually needs someone to pull data from two places and wait a few weeks for it to settle. Teams under pressure report what's ready, so a firm that wants outcome numbers has to ask for them by name.
There's also a quieter reason. An outcome number can go the wrong way, and an output number almost never does. Measuring outcomes means accepting that some of the work won't show up as results, and that some of it should stop.
The rule I hold myself to is short. If the result doesn't change, the work didn't count, however much of it there was.
If your leadership team wants to set outcome measures before its first AI rollout, ask about the workshop here.