croissantHow to measure the value of AI automation in a small business
Evaluate time, review effort, errors, and operational outcomes instead of relying on impressive-looking activity counts.
An automation creates value when it improves the business process after accounting for its costs and the work people still do. Counting messages processed or drafts created can be useful, but those counts do not prove that time was saved or that customers received a better answer.
Record a baseline before changing the process
Sample the current workflow. Note volume, handling time, waiting time, corrections, and the people involved. Use a representative period instead of the easiest example. If the process is already changing, record those changes so you do not attribute every difference to the agent.
Include the human work that remains
Reviewing output, correcting mistakes, and maintaining instructions are real costs. A useful estimate subtracts this effort from any handling time removed. Also account for software charges and implementation work. Treat an early calculation as a planning estimate until you have observed the workflow in use.
A simple calculation to make explicit
For planning, estimate the handling time removed across the observed volume, then subtract review, correction, and maintenance time. Keep financial costs separate until you have an agreed way to value staff time. Do not convert every minute saved into cash savings automatically: the business benefit may be extra capacity or more consistent follow-through. Record the assumptions beside the calculation and replace them with observed figures as the pilot runs.
Connect the measures to a decision
Choose a small set of metrics that tells you whether to expand, revise, or stop. For enquiries, consider unanswered cases and useful next steps alongside speed. For reporting, consider factual corrections and whether the report supports a decision. Avoid turning an internal dashboard into a claim about results you have not demonstrated.
A reality check before you hit run
Anthropic recommends evaluations that make changes in agent behavior measurable. Pair quality checks with your own handling-time and cost baseline. [1]
NIST’s framework connects AI design with use and evaluation. A launch-day demo is only one checkpoint. [2]
Implementation checklist
- Write down the current process and its costs.
- Measure review and correction effort after launch.
- Keep estimates separate from observed results.
- Use the numbers to make an explicit continue-or-change decision.
The sources behind the sauce.
Primary research and official documentation behind the cited points. The workflow examples and checklists are Croissant’s own guidance.