Why AI Shopping Agents Just Got a Brutal New Report Card

Measuring AI used to be simple. A few years ago, when AI was just a chatbot, you could test it with questions: write an essay, solve a math problem, generate code. The result was a score, and the score told you how smart the model was. That was the old world.

Now AI has stepped out of the chat window. It's an agent that can call tools, browse web pages, handle files, and execute tasks across systems. And that changes everything about how we should test it.

We see the consumer side: AI on your phone, adding items to a shopping cart, placing an order. But behind the scenes, on the merchant side, AI is doing far more complex work. That's where things get messy.

Enter RealReplicaBench, a new benchmark designed to test AI models in realistic e-commerce environments. The results are brutal. Across 107 real business tasks, none of the 13 models evaluated—including GPT, Claude, GLM, Qwen, DeepSeek, and (predictably last) Gemini—managed to pass. The highest score was 56.1 out of 100.

That's a failing grade for everyone.

Why Traditional Benchmarks Don't Cut It

Traditional benchmarks work like a multiple-choice exam. You get points for correct answers, partial credit for showing your work. If you don't know the last math problem, you might write down a formula and get a pity point.

But when AI enters a business workflow, partial answers aren't enough. A merchant doesn't need a thoughtful essay about how to find a supplier. They need a supplier that's been vetted, a purchase order that's been sent, and a calendar entry that's been created—all linked together so the next step can happen.

RealReplicaBench's philosophy is simple: if a task isn't complete, it's zero. There's no partial credit. The technical lead explained, "Even if a task is 80% done, if the remaining 20% requires the user to handle it manually—and that 20% might be the most critical part—then to the user, it's not a usable deliverable."

Real Work, Real Environment, Real Verification

Building a benchmark that tests real work is hard. You can't just write a text prompt and expect a text answer. Real tasks involve changing page states, conflicting information, and mismatched fields across systems.

RealReplicaBench tries to replicate that complexity. It doesn't just provide questions; it recreates front-end UIs, browser operations, CLI, API/MCP access, file systems, and backend states. The agent faces an actual business environment, not a piece of paper.

One task, for example, asks the agent to sift through about 300 high-noise emails to reconstruct a real procurement request, then select a supplier, tag the email, draft a reply, and create a Kickoff calendar event. That's not testing whether the model can summarize emails. It's testing whether it can extract the actual business fact from noise and turn it into action.

Another task is even more involved: the agent must turn 5,383 customs records into a cross-system procurement control tower. It has to aggregate data by category and supplier, filter the top three suppliers based on procurement policy, then create a dashboard in Google Workspace, an evidence folder in Box, and project tasks in Jira—all referencing the same suppliers and IDs.

This matters because in real work, state changes. The agent must keep track of context and adjust accordingly. If it writes a wrong dynamic ID, the handoff fails, and the audit trail breaks.

The Verifier Reads the Environment, Not the Agent

Another key design choice: RealReplicaBench doesn't trust the agent's self-report. It has a verifier that directly reads the final environment state.

For instance, in a logistics task, the agent must enumerate viable routes for a shipment from China to the US, factoring in ocean freight, trucking, final-mile delivery, insurance, customs clearance, bond, and platform fees. It must exclude any route exceeding 30 days or with invalid port connections, then complete the ocean booking and shipment verification. The final check isn't a plausible route suggestion; it's the actual Shipment ID that was generated.

This shifts the definition of "done" from the model's thought process to the environment itself. Listing status, booking confirmations, calendar events, file relationships, dynamic IDs—these are the evidence that work was actually completed.

Built on Real Business Data

Why can Accio Work's team design such a rigorous benchmark? Because it's grounded in real business needs. The 107 tasks weren't invented by researchers; they came from approximately 1.6 million complete conversations, 200,000 business execution trajectories, and 2,000 high-value workflows. These were distilled into 107 tasks that are both reproducible and verifiable.

The team behind RealReplicaBench is from Accio Work, Alibaba's AI agent focused on e-commerce. Their goal is to help merchants manage multiple storefronts from a single AI workbench, automating operations and analysis to boost efficiency.

The benchmark reflects their understanding of what an agent should be. If you think an agent is just a smarter Q&A system, you test answer quality. If you think it's about getting work done, you test environment, tools, state, verification, and failure attribution.

What This Means for the Future of AI Testing

RealReplicaBench isn't just a one-time release. The team plans to keep adding new real tasks, updating the evaluation environment, and using the benchmark for model evaluation, training optimization, and model routing. They want to match the right model to the right task, balancing effectiveness and cost.

For merchants, this benchmark helps them choose the right AI product. For the broader tech community, it highlights an emerging truth: as AI enters business workflows, the ability to build the harness—the environment, tools, and verification—is becoming as important as the model itself.

Users won't pay for an agent that "almost" gets the job done. They'll pay for a system that takes over the work and hands back a finished result.

We used to ask how much an AI knows. Then we asked if it can reason. Now, in the agent era, the question is shifting to whether it can actually finish the job. RealReplicaBench turns that simple question into a repeatable, verifiable, and actionable standard. The benchmark tests models, but in the end, it tests how deeply a team understands what it means for AI to do real work.