Being Found in AI Engines

The crawl layer most brands forget

Every major AI engine has one or more dedicated crawlers reading the public web. They are not the same as Googlebot, and your rules need to explicitly allow them. A default WordPress or Shopify install does not automatically welcome them — and many brands unknowingly block access with well-intentioned security plugins, aggressive rate-limiting, or default Cloudflare settings.

The AI crawlers you should know

Crawler	Belongs to	What it feeds
GPTBot	OpenAI	Future ChatGPT training data
ChatGPT-User	OpenAI	Live browsing inside ChatGPT
OAI-SearchBot	OpenAI	SearchGPT / search feature
ClaudeBot	Anthropic	Future Claude training data
Claude-Web	Anthropic	Live retrieval inside Claude
Google-Extended	Google	Gemini training (separate from Googlebot)
PerplexityBot	Perplexity	Perplexity search and answers
Applebot-Extended	Apple	Apple Intelligence models
CCBot	Common Crawl	Shared training data for many models

The conservative default for a brand serious about GEO is to allow all of the above. If you want to retain control, allow the live retrieval crawlers (ChatGPT-User, Claude-Web, PerplexityBot) and decide about the training crawlers based on IP considerations — but understand that blocking training means you are invisible to the "memory" layer of AI answers long-term.

Rendering, where content goes to die

Most AI crawlers are weaker than Googlebot at JavaScript rendering. If your core content is rendered client-side only, much of it is invisible.

• Above-the-fold content must be in the HTML source. View-source your page — if the main headline, lede and primary claims are not there, you have a problem.
• Lazy-loaded content below the fold is usually fine for humans but risky for AI. Where possible, server-render.
• Tabs, accordions and modals that populate only on click are often invisible. Either server-render the content into the DOM or provide a no-JS fallback.
• SPAs without SSR are the highest-risk pattern. If you are on Next.js, Remix, Nuxt or similar, make sure you are using server rendering or static generation for key routes.

Structured data, speaking the AI language

Schema.org structured data is how you explicitly tell an AI what your content is. Without it, the AI has to guess; with it, you dramatically increase the odds of being correctly understood and cited.

The five schema types every brand should have

1. Organization — name, logo, URL, social profiles, contact, description. The entity anchor.
2. Website — canonical site info and, ideally, a SearchAction.
3. Product (if you sell physical or digital products) — name, description, price, availability, brand, SKU, GTIN, images, reviews, ratings.
4. FAQPage — on every major collection, category, service or article page.
5. Article or BlogPosting — on every editorial page, with author, datePublished, dateModified and headline.

Schema mistakes we see constantly

• Schema exists in one template but is broken or missing on key pages.
• FAQ schema with generic or keyword-stuffed questions that don't match any actual customer question.
• Product schema missing GTIN, brand or availability — Google ignores feeds with this gap.
• sameAs properties pointing to dead or incorrect social profiles.
• Schema nested inside JavaScript that the crawler never executes.

Page speed, mobile and Core Web Vitals

Slow pages get partial retrieval. A page that takes 8 seconds to render may only give the AI the first 1,000 characters before the crawler's soft timeout — everything below the fold is effectively invisible. Performance is a GEO signal, not just a UX one.

• Time to first byte under 500ms for most pages.
• Largest Contentful Paint under 2.5s on 4G mobile.
• No render-blocking scripts above the fold.
• Mobile-first rendering. The mobile HTML is what most crawlers read.

Sitemaps, canonicals and internal linking

• XML sitemap submitted in Search Console, covering all pages you actually want indexed.
• Canonical tags that match your intended URL structure — no self-referential loops, no duplicates.
• Internal linking that gives AI crawlers breadcrumb-style context about how pages relate.
• No orphan pages — if nothing links to a page, don't expect AI to find it.

Rendering for different AI engines

Different AI engines have different quirks. A few that matter in practice:

• ChatGPT aggressively prefers clean HTML, short paragraphs and clearly delimited sections.
• Gemini leans heavily on Google's own indexing — so your Google SEO hygiene is Gemini hygiene.
• Claude is strong at reasoning over long passages but still benefits from explicit structure and headers.
• Perplexity explicitly cites sources and is therefore unusually sensitive to schema, freshness and perceived trust.

A technical starter checklist

1. Robots.txt allows GPTBot, ClaudeBot, PerplexityBot, Google-Extended.
2. Organization, Product and FAQPage schema validate cleanly.
3. Above-the-fold HTML contains your headline, lede and core claim server-rendered.
4. Core Web Vitals pass on your top 10 pages.
5. Sitemap submitted and all priority URLs indexed.
6. Internal linking is logical and every key page has at least three internal links in.
7. No important content hidden behind JavaScript-only tabs or accordions.
8. Canonicals resolve cleanly, no redirect chains longer than 1 hop.

Every one of these is a pass/fail inside our audit. The technical section alone routinely catches 8–15 fixable issues on otherwise well-run sites.