Technical GEO tips: how to make your site readable to AI

Google's page on AI features and your website says there are "no additional requirements to appear in AI Overviews or AI Mode." To be shown as a supporting link, a page must be indexed and eligible to appear in Google Search with a snippet.

That makes your usual technical SEO the first layer of technical GEO. Pages need to be crawlable, indexable and allowed to show a snippet. Google lists the controls that limit what appears from your pages in Search, nosnippet, data-nosnippet, max-snippet and noindex, so check that none of them sit on pages you want cited. Our guide to technical SEO covers the basics.

Each AI company runs more than one crawler, and each one does a different job.

OpenAI. Its crawler documentation says OAI-SearchBot and GPTBot are independent settings. You can allow OAI-SearchBot so your site appears in ChatGPT search results while disallowing GPTBot to opt out of training. ChatGPT-User handles certain actions a person starts in ChatGPT, and OpenAI notes that because people start those visits, robots.txt rules may not apply.

Perplexity. Its crawlers page describes PerplexityBot as the agent that surfaces and links websites in Perplexity's search results, and says it is not used to crawl content for AI foundation models.

Google. Google's list of common crawlers describes Google-Extended as a robots.txt token that controls whether content Google crawls may be used to train future Gemini models and for grounding in Gemini Apps. Google says Google-Extended does not affect a site's inclusion in Google Search and is not a ranking signal.

Anthropic. Its help article on crawling describes ClaudeBot, which collects web content that could contribute to model training, and Claude-SearchBot, which analyzes online content to improve the relevance and accuracy of search responses. Anthropic says disabling Claude-SearchBot may reduce your site's visibility and accuracy in user search results.

Check two places. First, read yourdomain.com/robots.txt for rules that name these crawlers. Second, ask whoever runs your hosting or security settings whether a bot protection feature blocks them, because a firewall can block a crawler that robots.txt allows.

Vercel studied how AI crawlers fetch pages across its network. Its study, published in December 2024, found that "none of the major AI crawlers currently render JavaScript." That may have changed since, so test your own pages. The study names OpenAI's crawlers, Anthropic's ClaudeBot, Meta-ExternalAgent, ByteDance's Bytespider and PerplexityBot. It also notes that Google's Gemini uses Googlebot's infrastructure, which can render JavaScript.

If your services, prices or locations only appear after scripts run in the browser, some AI crawlers may see an empty page. Make sure the text that matters is in the HTML your server sends. Google's own AI features page makes a related point and lists making sure important content is available in textual form among its best practices.

To see what Google's crawler gets, run a page through the URL Inspection tool in Search Console. It can show a rendered version of the page and the page code Google loaded. For a rough idea of what a crawler that skips JavaScript sees, view the page source in your browser. If your main message is missing there, those crawlers will miss it too.

Google's AI features page also lists making sure your structured data matches the visible text on the page. It adds that there is no special schema.org markup you need to appear in AI features. Use structured data to describe what is already on the page, such as your organization, your articles and your FAQs, and keep it accurate when the page changes.

The same Google page says you don't need to create new machine readable files, AI text files or markup to appear in its AI features. That is worth knowing before anyone sells you a special file as the fix. Our post on llms.txt explains what that proposed file is and where it fits.

Google lists checking that your Merchant Center and Business Profile information is up to date among its AI features best practices. For local businesses, that puts an accurate Google Business Profile on your technical GEO checklist.

Technical fixes are only worth doing if you can see the results. Watch Search Console for indexing and snippet issues, and set up an AI channel in your analytics so you can see visits from assistants. Then check what the assistants actually say about you, which our guide to tracking your brand in AI search walks through.

The pattern across all of these technical GEO tips is the same. AI systems can only cite pages they can fetch and read, which is what good technical SEO has asked for all along.