Auralis pilot scorecard for testing support requests on your own data

A pilot only tells you something if you agreed on what to measure before it started. Otherwise, stakeholders may read the results differently after the pilot. This scorecard covers the workflows a copier dealer’s service desk may hand to an AI help desk: meter reads, toner and supply orders, service calls, dispatch logging, and the exceptions that go to a person.

Use a representative sample of your own requests. A sales call may rely on a vendor’s strongest recording. Your meter reads, your contract limits and your dispatch records are what show whether the system fits your business.

Agree the pass lines first

Each row needs a pass line, set with the people who run the desk and the people who run e-automate. Set the target around your volume, your branches and your current manual workload. Document the pass line before the pilot, so it is not adjusted after the results are known.

The scorecard

| Area | What to record | Where to check it | Your pass line | |---|---|---|---| | Meter reads | Reads posted without a rekey, reads held for review, and the reason each held read was held | Meter entries in e-automate, compared with e-automate’s meter volume check | | | Toner and supply orders | Orders placed without a person keying them, and any supply past the contract item limit | Contract item limits in e-automate | | | Service calls | Calls logged with the category describing the nature of the problem | Service call records in e-automate | | | Dispatch logging | Minutes from first contact to a logged call | Dispatch records and timestamps in your system | | | Exceptions | Share of exceptions that reached a person with a reason attached, and time to that person’s response | Exception records you keep, one per held item | | | Customer wait | Minutes from first contact to a confirmed answer | Your contact records, with timestamps | | | Audit trail | For each automated action: the source, the check it passed and the threshold it sat inside | Require this record for every automated action | Yes or no | | Access | Only the approved objects are reachable, and access can be revoked | The integration’s approved scopes and revocation controls, reviewed with IT | Yes or no |

How to score it

1. Sample across the whole pilot, not the three best days. Include the day after a holiday and the day a branch was short-handed.
2. Trace every automated action back to its record in e-automate. If you can’t find the record, the row fails.
3. For every held item, check that a person closed it and that the reason is written down.
4. Run the same measurements for the period before the pilot to establish a documented baseline.

Include the failure path

Put failure-path behavior in the scorecard. NIST’s playbook recommends assessing whether a system can “fail safely, particularly if made to operate beyond its knowledge limits.” Auralis product material describes the intended routing: a meter read inside the variance you set posts on its own, while a read outside it stops and goes to a person with the reason attached (internal). Write that as a test requirement, and check it on the held reads in your sample.

The same guidance says that documenting test sets and metrics makes a check repeatable. Keep a one-page record of how you tested, so someone on your team can run the same check again next quarter and get a comparable answer.

Put IT on the list early

Ask IT to review the access row before the pilot starts. Auralis product material describes the connection this way (internal): an ECI partner integration via the ESN API, running under a service account you create, scoped to the objects you approve: contracts, meters, supplies, service calls, invoices and parts. You grant each object and revoke it the same way, without a call to us.

Listen before you score

Ask for a recording of a real service call handled from start to finish. Score the recordings on their own criteria, such as accuracy, whether the call went to a person when it should have, clarity, and how the caller was treated. Report qualitative call scores separately from volume metrics because the two measure different outcomes.

What one dealer saw

One dealer we work with had about 80% of meter reads automated within two weeks of go-live, and customer wait time dropped from a few hours to a couple of minutes. That happened on the same team, right after the dealer absorbed two acquisitions. Set pass lines that reflect your operation before reviewing the results.

Book a 30-minute walkthrough.

Sources

• NIST, AI Risk Management Framework, framework released January 26, 2023; page checked October 23, 2026.
• NIST, AI RMF Playbook: Measure, undated; checked October 23, 2026.
• ECI, e-automate online help (version 250): Meter Volume Checks, Entering Supply Item Limits, Creating Service Calls.
• Auralis product information (internal).
• Auralis customer results (internal, anonymized).