Before you start an AI trial, write the sentence you want to be able to say when it ends. “We tried it and people liked it” will not help you decide much. “It prepared usable job briefs, handled incomplete requests correctly and reduced the team’s effort” gives you something to test.
You have already chosen the workflow. The next question is whether this particular setup performs it well enough to keep. A trial needs a defined finish, examples that resemble the real work and a standard agreed before anyone sees the results.
Two weeks can support an initial decision about a bounded workflow, even if that decision is that you need more evidence.
Write the agreement before the first test
Imagine an equipment-service company that wants AI to turn incoming repair requests into internal job briefs. The brief should identify the customer, site, equipment, reported fault and missing information.
The company prepares this trial agreement:
| Decision | Agreement for this example |
|---|---|
| Start and finish | First Monday through the second Friday |
| Work being tested | Prepare internal briefs from repair requests and their attachments |
| Required result | A reviewer can understand the request without reconstructing the entire email thread |
| Critical errors | Wrong customer, wrong site, wrong equipment or invented facts |
| Trial boundary | Draft briefs only; the existing team handles all customer communication and scheduling |
| Decision at the end | Continue within scope, revise and retest, stop, or record insufficient evidence |
The boundary prevents enthusiasm from expanding the test halfway through. If the team discovers another useful application, it can record the idea without adding it to this trial.
Build a small test set with uncomfortable cases
Choose examples from work the company actually receives. Here, the evaluation set contains twenty previously completed requests: twelve routine requests, four with missing information, two with later amendments and two that duplicate earlier requests. Use separate examples for initial setup so the evaluation does not merely repeat what the team already tuned the instructions to answer.
Before testing, an experienced employee records the expected brief or correct handling for each case. An incomplete request should produce a brief that names the missing information, and a duplicate should be flagged rather than turned into an extra job. An amendment should change the relevant detail without erasing information that is still valid.
Those expectations give the reviewer an answer to compare with. Without them, a polished summary can receive a generous score simply because nobody remembers the difficult detail in the original request.
Keep the set manageable, but do not make it artificially easy. These twenty cases support an initial decision; they do not prove how often errors will occur. A rare type of request can still be tested with a known past example.
Define a pass in terms of work you can use
For this company, the owner and service manager agree on the following marks. They are business choices, not universal AI benchmarks.
| Measure | Pass mark for the twenty evaluation cases |
|---|---|
| Critical factual accuracy | No wrong customer, site or equipment, and no invented facts |
| Difficult cases | All eight incomplete, amended or duplicate cases handled as expected |
| Usability | At least sixteen briefs need no substantive rewrite |
| Total human effort | At least sixty minutes less human effort than the current process on the same twenty requests, including preparation, review and corrections |
Define “substantive rewrite” before scoring. Here it means changing the meaning, putting back important details the brief left out or rebuilding its structure, but not adjusting a heading or fixing punctuation. A brief that correctly reports missing information can still be usable.
The time threshold should reflect what would make the workflow worth maintaining. This company might choose a different threshold if it receives five requests a week instead of fifty. When you add up the effort, count the checking too, and record setup time separately.
Quality and effort are separate conditions. Impressive speed does not cancel a critical error, and accurate briefs do not automatically justify extra work.
Use the second week to challenge the result
In the first week, configure the workflow with the setup examples, then run and score the evaluation set. Save the instructions and settings used so the team knows which version produced the results.
During the second week, prepare briefs alongside the normal process for new requests. The existing process continues to determine the actual job. Look for differences the historical examples missed: unfamiliar attachments, forwarded threads, unclear equipment names or a request split across several messages.
Do not quietly change the instructions and combine the improved results with the earlier scores. Record the change, rerun the affected evaluation cases and report the versions separately. Otherwise the final result describes a setup that never existed.
End with a decision, even if the answer is “not yet”
When the trial ends, choose one of four outcomes:
- Continue if the workflow meets the agreed marks and the live examples reveal no unresolved critical problem. Stay within the tested boundary: passing a drafting trial does not grant permission to schedule work or contact customers.
- Revise and retest if a fixable issue blocks the pass. Write down the change and set a short retest date.
- Stop if the workflow fails the quality or effort requirement repeatedly. Spend no more time on this version.
- Record insufficient evidence if too little relevant work arrived. Specify what additional cases are needed.
Put the decision meeting on the calendar when the trial starts. Bring the scored examples, total human effort and unresolved exceptions. Those three things make it possible to choose, rather than allowing an experiment to become permanent because nobody formally ended it.
