If you have ever tried this, you already know the screen.
You ask Claude to pull the title and bullet points for one of your ASINs. Something you own. Something a shopper sees in one click. You get back a red Failed to fetch, a fallback web search that finds nothing useful, and an honest shrug:
“I couldn’t get it. Amazon’s robots.txt blocks automated fetching, so the page request was refused outright, and a search on the ASIN turned up nothing but unrelated junk matching fragments of the string. I have no reliable data on what B0D1XD1ZV3 is, and I’m not going to guess at a title or invent bullets.”
Most sellers read that and assume the model is having a bad day. It is not. It is policy, written down, and published at a URL you can open yourself.
The policy is public, and it names names
Amazon publishes a file at amazon.com/robots.txt that tells automated visitors where they may go. Near the top it sets general rules. Then it starts naming names.
When we read that file on September 15, 2026, 98 individually named AI crawlers and agents each carried the same instruction: Disallow: /. Not “stay out of checkout.” Not “skip the wish lists.” The whole domain, for those agents.
That is Amazon’s call to make, and it is a reasonable one. The point here is not that the rule is wrong. It is that the rule exists, it is explicit, and it has a consequence most sellers have not thought through.
The list reads like a directory of every assistant your team might use:
- ClaudeBot, Claude-User, Claude-SearchBot
- GPTBot, ChatGPT-User, OAI-SearchBot
- Gemini-Deep-Research, GoogleAgent-Shopping, Google-NotebookLM
- Copilot, CopilotNative, Perplexity-User, GrokBot, DeepSeekBot
GoogleAgent-Shopping is the one worth sitting with. An agent built to understand shopping, told to stay out of the largest store on the internet.
Well-behaved AI tools honor that file. So when your assistant refuses your listing, it is not failing. It is obeying. The better the tool, the more reliably it obeys.
Why this is your problem, not Amazon’s
You can see your listing. Your customer can see your listing. The thing you increasingly ask to do your analysis cannot.
And the listing is the easy case. The questions that actually move money live on the search results page, which is not a document sitting at a URL at all. It is assembled live, per shopper, per location.
Who ranks above me for my main keyword. Which of those positions are paid and which are earned. Who is bidding on my term this week who was not there last week. What are they charging, and how much are they selling.
Every one of those is a five minute task for a human with a browser, and an impossible task for an assistant that is not allowed through the door.
So the work stays manual. Not because it is hard, but because the tool that should do it cannot see the page.
The answer is a door that is meant to be opened
The fix is not a cleverer scraper or a user-agent pretending to be a person. That is a race you lose slowly, and it puts the compliance problem on you.
The fix is to stop pointing your agent at a page it was never meant to parse, and give it a data source built for the job instead.
Ask RankGenius about a listing and it hands your agent the product as structured data rather than a page to parse: title, brand, bullets, rating and review count, the attribute table, the main image, and per-aspect review sentiment. For a representative consumer-electronics ASIN, that came back as ten bullets, a 4.6 rating across 47,020 ratings, and eight scored review aspects.
Ask it about a keyword and you get the part no assistant can reach at all. Searching “airpods pro 2” against the US marketplace, pinned to zip 30303, returns the first page as data: 16 organic positions in rank order and 3 sponsored placements, each flagged so you can tell an ad from an earned position. Every row carries the competitor’s ASIN, title, product image, price and list price, rating, review count, and an estimate of recent purchase volume.
That last part is the difference between knowing your rank and understanding your market. Organic position tells you how you are doing. Ad placement tells you what your competitors are willing to pay to get in front of your customer. You need both to decide where the next dollar goes, and pinning a zip keeps the numbers comparable run over run.
More importantly, it all arrives as JSON rather than a wall of HTML, which is the form an agent can actually reason over. Your assistant stops apologizing and starts working.
rankgeniusapp.com
