Why We Stopped Measuring Support by Response Time
A fast reply that doesn't answer the question isn't actually fast. Here's the metric we replaced response time with, and why it changed how AI Agent is built.
Quick Summary
Every merchant I talked to in MerchantDesk's first year asked the same question: how fast does it reply? I understood why — response time is easy to measure and easy to compare. But it's also easy to game, and gaming it doesn't help the shopper waiting on an answer.
A fast acknowledgment isn't a fast answer
Response time only measures how quickly something arrives in the chat window, not whether that something actually resolves the shopper's question. A canned "thanks for reaching out, someone will get back to you shortly" replies in under a second and does nothing for the shopper who wanted to know if a jacket runs small.
We built AI Agent around a different question: can this conversation end here, with the shopper's actual question answered, without anyone waiting on a human? That's a harder bar to clear than replying fast, but it's the one that actually matters to the person on the other end.
Steps between a question and an answer
Illustrative exampleWhat we measure instead
Internally, the number we care about is how many conversations resolve without ever reaching a human — not because we want to avoid people, but because every one of those is a shopper who got their answer immediately instead of waiting. The conversations that do need a person get flagged and escalated with full context, not silence.
Response time still matters as a floor — nobody wants to wait minutes for even an acknowledgment — but it stopped being the number we optimized for the moment we realized it said nothing about whether the shopper actually got helped.
- AI Agent answers directly from your store's real policies, catalog, and order data — not a canned response
- Escalations to your team carry the full conversation, not a cold "hi, what's this about"
- Draft Reply AI drafts a first-pass answer for the conversations a person does need to send, so the team isn't starting from a blank reply either
Why this matters more at 11pm than at 11am
Response-time targets are usually built around business hours, because that's when a human team is watching the queue. A shopper asking a question at 11pm doesn't care what your team's hours are — they close the tab and, more often than not, don't come back. AI Agent doesn't have hours.
Takeaway
Fast doesn't help if it isn't right. We'd rather AI Agent take the time to check your actual order data and answer correctly than fire back a placeholder in under a second.
Was this article helpful?
See AI Agent live on a real storefront.
Start Free TrialFrequently asked questions
No — AI Agent still replies immediately in most cases. The difference is what happens in that reply: an actual answer pulled from your store's real data, not just an acknowledgment.
Written by
Aqib Shahzad
Founder, MerchantDesk AI
Founder of MerchantDesk, leading product on AI Agent, Shop with AI, and Order Tracking. Writes about how the AI actually works under the hood, not just what it promises.
View all articles by Aqib
You’ve seen how it works. Try it on your own store.
Put this playbook to work in an afternoon — free for 7 days.
