Two releases landed on November 5 that read like the same study run from opposite ends. Visa tested what AI shopping agents can do. FIS asked what people will let them do. The answers line up uncomfortably well. Visa's new Agentic Commerce Benchmark put 22 models from 12 labs through 15 shopping tasks, things like buying a TV, a laptop or a personal care appliance, with each … [Read more...] about Visa Benchmark Finds AI Shopping Agents Fail 42.8% of Runs as FIS Survey Shows Consumers Want AI Guarding Money First
benchmarks
OpenAI Quietly Revised Its GPT-6 Astra Evaluation Metrics After Launch
Emily Forlini reports in Fortune that OpenAI has quietly changed how it evaluates GPT-6 Astra, in ways that appear to flatter the model, and that it has continued revising other metrics since launch. Changing a measurement after you have seen the result it produces is the oldest problem in empirical work. It is not automatically dishonest. Evaluation suites do get saturated, … [Read more...] about OpenAI Quietly Revised Its GPT-6 Astra Evaluation Metrics After Launch