If you sell on Amazon and quit AI after it made something up, you quit a version of it that no longer exists.
I’m at Amazon Unboxed in San Francisco, talking with the owner of an Amazon agency about how I run my business with AI. He tells me he tried AI in 2025. It made things up, said them with total confidence, and he stopped using it.
I’ve heard some version of that from several sellers and agency owners. It’s a reasonable conclusion. If a new analyst invented an ACoS figure in a client report, he wouldn’t hand them the next one.
But I don’t argue with him. I show him something instead.
My YouTube channel. The presenter is my AI avatar. AI generated the video. AI generated the B-roll. AI did the editing.
Then I tell him the part that matters.
The workflow behind that video didn’t exist three months earlier.
Not a year. Three months.
His verdict wasn’t wrong. It was dated. He judged a snapshot, and the thing in the snapshot has been replaced several times since.
Plenty of sellers and agencies quit the same way
I went looking for how common this is. No study I found measures exactly “tried AI, got a bad answer, quit,” and none looks at Amazon sellers specifically. But the gap between trying AI and actually using it is large, and it shows up everywhere.
In a February 2026 Pew survey, about half of US adults said they use AI chatbots. Only about a quarter use one daily. A Menlo Ventures survey from April 2025 found the same shape: 61% of US adults had used AI in the previous six months, but fewer than one in five used it every day.
Small businesses, the closest group to most Amazon sellers, went the other direction for a while. In a NEXT survey of 1,500 small business owners, the share using AI fell from 42% in 2024 to 28% in 2025. They pointed to cost and complexity rather than accuracy, but the pattern is the same one: try it, decide it isn’t worth it, stop.
Andrej Karpathy, one of the founders of OpenAI, put it this way in April 2026: “I think a lot of people tried the free tier of ChatGPT somewhere last year and allowed it to inform their views on AI a little too much.”
Why one bad answer is enough
There’s a name for this. Researchers at Wharton called it algorithm aversion. In a 2015 study, people lost confidence in an algorithm faster than in a human who made the same mistake. They walked away from the algorithm even after watching it outperform the human.
A human’s mistake reads as a bad day. A machine’s mistake reads as a permanent defect.
AI makes that worse because its mistakes are well dressed. In a Purdue study, 52% of ChatGPT’s answers to programming questions contained incorrect information, and participants missed the errors 39% of the time because the answers sounded so sure of themselves.
On Amazon, a confident wrong answer has a price. A made-up fee total makes a losing SKU look profitable. An invented ACoS goes into a client report. Catch one of those and it doesn’t feel like an error. It feels like being lied to. That’s the moment sellers and agencies quit.
Why the verdict goes stale
The machine doesn’t stay the same machine.
METR, a research group that measures what AI agents can actually do, found that the length of tasks agents can complete on their own doubled about every seven months from 2019 to 2025. A verdict that’s a year old is almost two doublings out of date.
For a seller, that’s the difference between an AI that writes a decent listing bullet and one that can work through a question like “which of my SKUs lost money last week after ads, fees and COGS?” For an agency, it’s the difference between a writing assistant and something that can run the same check across every client account.
Newer isn’t the whole fix
Here’s the honest part: AI still makes things up.
Newer doesn’t automatically mean more truthful, either. When OpenAI released its o3 model in April 2025, its own testing found o3 made up answers on one benchmark about twice as often as the model before it.
So “try it again, it’s better” is only half the answer. The other half is what the AI is working from.
Ask a chat window for your TACoS or your profit on a SKU and it has nothing to go on. It never saw your Seller Central account. It can tell you it doesn’t know, or it can guess. The guess is the hallucination that made people quit.
I don’t use AI that way. My AI works from my actual data and files. It runs under a standing rule that it doesn’t make a claim it can’t back up with data. On anything that matters, a second model checks the first one’s work.
That doesn’t make errors disappear. It changes where they get caught: before they reach me, instead of after they reach a client.
Before you write AI off for good
- Put a date on your verdict. “AI made things up when I tried it in 2025” is a fact. “AI makes things up” is a conclusion that expires.
- Test it on a real business question, not a quiz. Ethan Mollick at Wharton suggests about ten hours of use before you know what it can and can’t do. BCG found that people with five or more hours of training are far more likely to become regular users.
- Give it your numbers. An AI answering from memory has to guess about your account. An AI working from your actual orders, fees and ad data has something to check its answers against.
- Make it show its work. Require the source behind every number that matters: which report, which date range, which SKUs. When it can’t produce one, you’ve caught the error yourself.
Why we built the Seller Labs MCP Server
That third step is the one most sellers can’t do on their own, so we built it.
The Seller Labs MCP Server connects AI tools like Claude to a seller’s own Amazon data. It runs on Data Hub: each seller’s data lands in their own dedicated database of more than 50 tables, covering orders, settlements, fees, ads, inventory, returns and reimbursements. Every SKU gets one profit row per day, with sales, refunds, FBA and referral fees, storage, Sponsored Products spend and your COGS, across more than two years of history.
Next to the data sits a map written for the AI: which table answers which question, how to calculate each metric correctly, and a set of tested queries. So when you ask which keywords are spending money on products that lose money, the AI doesn’t improvise the math. It follows a path that has already been worked out, and it answers from your numbers.
Agencies add each client’s seller account to the same connection, so one question can run across every brand they manage.
The answer still comes from an AI. But it comes from a query against your own Amazon data, which means you can check it. That’s the difference between an AI that guesses about your business and one that knows it.
He wasn’t wrong about the AI he tried. He was judging it by a photograph.
If you quit AI after it made something up, you quit a version of it that no longer exists.
