Why WhatsApp order bots confirm orders that were never placed
The worst message an order bot can send is not a wrong price. It's this:
Order #1187 confirmed. Delivery tomorrow.
…when the customer never said yes.
We hit this four times while building Shopap — in August twice, late August once, and again in September after we'd "fixed" it. Each time the fix was a wording change to the prompt. Each time it held for a while and then didn't. That pattern is the lesson.
How it happens
A customer sends an order. The agent lists items and a total and asks "Shall I confirm?" So far, fine. Then the customer sends something that isn't a yes:
- "aur 2 packet biscuit bhi" — an addition, not a confirmation
- "ok" — an acknowledgement, not a confirmation
- "rate kya hai?" — a question
- a voice note that transcribes to nothing useful
A language model reading that thread is under pressure to make progress. "ok" looks like consent. An addition reads like enthusiasm. And the model has just been asked to produce the next message in a conversation whose obvious next message is a confirmation. So it writes one.
Why prompt wording doesn't fix it
You can write "only confirm on an explicit yes" into the system prompt. We did. It reduces the frequency and doesn't eliminate it, because the decision is still being made by the thing that's under pressure to make progress.
The fix isn't a better sentence. It's taking the decision away from the model.
What we do now
The model proposes; the system decides.
The order lives as a draft with a state. Only a separate confirmation step — a deterministic check, not a generation — can move it to confirmed. That check looks at the customer's latest message alone and asks one question: is this an unambiguous yes to this draft? "haan confirm", "yes", "confirm karo" pass. "ok", "aur 2 packet", a question or silence do not; they route to the draft-editing path instead.
If the check can't decide, it asks. One short question — "Confirm the order above?" — costs a second. A phantom order costs a customer.
And because the state lives in the system, a confirmation can also require a human approval step per business. Commercial writes wait for a person until the owner turns that off.
The general rule
Anywhere a model's output would trigger a commercial action — an order, a booking, a payment — the action needs a gate that isn't the model. Prompts steer behaviour; they don't guarantee it. Guarantees come from state machines.
We run an evaluation set of the messages that fooled us. Every prompt or model change runs against it before it ships. The set grows every time a customer finds a new way to not say yes.