Two releases landed on November 5 that read like the same study run from opposite ends. Visa tested what AI shopping agents can do. FIS asked what people will let them do. The answers line up uncomfortably well.
Visa’s new Agentic Commerce Benchmark put 22 models from 12 labs through 15 shopping tasks, things like buying a TV, a laptop or a personal care appliance, with each model getting three tries at each scenario. The catch is that the customer’s requirements aren’t handed over up front. The agent has to draw them out of a conversation, then make the right call, which can be a purchase or a decision not to buy. A run only passes if both parts are right.
Overall pass rate: 57.2%. So 42.8% of runs failed.
The spread is wide. The best models passed 82.2% of their runs and the worst 28.9%. According to Fortune’s reporting, agents from Google and Meta tied at the top around 82%, Claude came third at about 76% and OpenAI sixth at 64%. Price helped, but not as much as you’d hope: the five most expensive models per run passed 67.1%, the five cheapest 50.2%. Consistency was worse than the headline. A model cleared all three attempts at the same scenario only 34.2% of the time.
The failures still spent money
This is the number that matters for anyone in payments. In 54% of the failed runs the agent bought something anyway. It either picked a product that didn’t meet what the customer had said, or it checked out before it had asked enough questions to know. Run the arithmetic and that’s roughly one in four of all runs ending in a purchase the customer probably didn’t want.
For a card network, a wrong purchase isn’t a UX problem. It’s a dispute, a chargeback, a refund request and an argument over who authorized what. Visa’s own Trust Index found that 72% of US consumers have used an AI assistant, yet only 23% trust one to make payments for them. The benchmark gives that gap a measurement.
What consumers will actually allow
FIS, working with Kantar, surveyed US adults from September 16 to 30 for its 2026 Consumer Pulse Survey. The headline finding: 57% are comfortable with an AI assistant flagging or freezing a suspicious charge without asking first, while 50% want to approve every transaction an AI makes. Defense yes, spending no.
The conditions people set for trusting an AI with money are specific. Asked for their top three, 58% picked a refund guarantee and 56% picked access to a human who can review and override the decision. Among those not ready to hand money management to AI, 70% said they want to stay in control. And the generational gap is enormous: 61% of millennials say they’re ready to let an assistant run day-to-day money management, against 3% of adults 70 and older.
Scams sit underneath all of it. 35% of respondents have been targeted by an AI impersonation scam or know someone who has, and 22% were targeted personally. Only 47% are confident their bank would return money lost to a deepfake.
Read them together
The FIS refund-guarantee answer and the Visa wrong-purchase rate describe the same risk. People want somebody else to carry the cost when an agent gets it wrong, and the benchmark says agents get it wrong often enough that someone will have to. That’s why the near-term winners in agentic commerce are likely to be the rails that can reverse a transaction cleanly, not the models that complete one.
FIS put its own conclusion plainly: banks should start with protection, keep the customer in control and prove accountability before moving to autonomous spending. Visa says the benchmark isn’t a verdict that agents aren’t ready, only that reliability is the open problem. Anyone can try the same test on Visa’s Payments Frontier site.
Fraud alerts first. Checkout later.