SEO is still powerful but we are seeing more users find us through chatgpt; AEO solves for the new dynamic. Here’s why being the answer matters now, and the eight things we shipped in our codebase to get there.
AEO (Answer Engine Optimization) for Large Language Models (LLMs) is the new-age SEO (Search Engine Optimization), and it needs to be part of your digital marketing strategy. It helps a brand’s website get selected and cited as a trusted source in AI-generated answers on platforms like ChatGPT, Perplexity, Gemini, Claude, DeepSeek and Google’s own AI Overviews.
As search behaviour shifts toward complex, conversational questions, AEO is about giving direct, concise and structured answers. The goal is to be the answer, not just to rank somewhere on the search engine results page (SERP). To “be the answer” here means succinctly answering the question any person using an LLM asks in your niche.
When we noticed that people had changed how they search for and find answers, and that it looked very different from how we traditionally searched, we knew it was time to do something about it.
Let me be clear about one thing upfront: AEO does not replace SEO. It sits on top of it. Even Google says so. In its May 2026 guide, Optimizing your website for generative AI features on Google Search, Google writes: “From Google Search’s perspective, optimizing for generative AI search is optimizing for the search experience, and thus still SEO.” I agree with the foundation part. What I don’t agree with is that Google is the only answer engine you should optimise for, ChatGPT, Claude and Perplexity each have their own crawlers, indexes and preferences.
This is what we noticed and prompted the shift.
People are moving their questions to AI. OpenAI reported 900 million weekly ChatGPT users in February 2026, and at DevDay on September 29, 2026 the figure shown was 1.2 billion weekly users. Google’s AI Overviews hit 2 billion monthly users back in July 2025, and by Google I/O 2026 Google was reporting over 2.5 billion monthly users for AI Overviews and over 1 billion for AI Mode.
Fewer clicks are leaving the results page. Even before AI answers went mainstream, SparkToro and Datos found that 58.5% of US Google searches and 59.7% of EU searches ended with zero clicks in 2024. AI summaries push that further:
- Pew Research Center tracked the real browsing of 900 US adults across 68,879 Google searches in March 2025. Users clicked a traditional result in 8% of visits when an AI summary appeared, versus 15% when it didn’t, and sessions ended right there in 26% of visits with a summary versus 16% without one. Pew found users clicked a link inside the summary itself in just 1% of visits to pages that had one. (Google disputed the methodology.)
- Ahrefs found that an AI Overview reduces the click-through rate of the number one result by 58% (December 2025 data, 300,000 keywords).
- Seer Interactive saw organic CTR on informational queries with AI Overviews drop 61%, from 1.76% to 0.61%, across 3,119 queries from 42 organizations. The silver lining: brands that were cited inside the AI Overview earned noticeably more clicks than brands that weren’t.
Analysts saw this early on. In February 2024, Gartner predicted that traditional search engine volume would drop 25% by 2026 as AI chatbots became “substitute answer engines.” To be fair, that was/is a prediction, and Google search hasn’t fallen off a cliff yet.
AI traffic is small, but it converts. Conductor’s study of 13,770 domains put AI referrals at about 1.08% of sessions on average, with ChatGPT sending 87.4% of them. Small, yes. But in a Seer Interactive case study, ChatGPT referrals converted at 15.9% versus 1.76% for Google organic, and Similarweb’s ecommerce data showed 11.4% for AI referrals versus 5.3% for organic. Results vary a lot by industry. The pattern is that someone who arrives after an AI has already recommended you is usually further along in deciding to buy.
AEO, GEO and where SEO fits
You’ll see two acronyms used almost interchangeably:
- AEO (Answer Engine Optimization): shaping your content and site so AI assistants pick you as the answer and cite you.
- GEO (Generative Engine Optimization): a term formalized in the paper GEO: Generative Engine Optimization by Aggarwal et al. (Princeton, Georgia Tech, the Allen Institute for AI and IIT Delhi), first published in November 2023 and presented at KDD 2024.
That paper is worth a read. The researchers built a benchmark (GEO-bench) of diverse queries and tested content tweaks against generative engines. They report that GEO methods “can boost visibility by up to 40%” in generative engine responses. The strongest methods were the ones a good writer would use anyway: citing sources, adding quotations from credible people and adding statistics. Keyword stuffing did not help. They also found that what works varies by domain, so there’s no single trick.
That finding is basically why this article is full of links and numbers. I’m walking the talk.
Layer one was SEO, layer two is AEO
The first layer was SEO, and I wrote about it in How We Rebuilt Intelli’s SEO And What You Can Learn From It. The next layer was making sure our content was written in a way and a format that’s easy for LLMs to understand. Then we built tools so that delivering that content was easy and repeatable for the team.
The other half of the AEO layer is technical: the things we changed in our product to make Intelli easier for LLMs to read, recognise and cite. Here’s the deep dive.
What we shipped
1. We published an llms.txt file
File: public/llms.txt
llms.txt is a proposal from Jeremy Howard of Answer.AI (September 2024). It’s a plain-Markdown file at the root of your site that gives a language model a clean, curated summary and a map of your most important pages, so it doesn’t have to dig through navigation menus, cookie banners and JavaScript to understand who you are.
Ours is a 111-line brief written for the model, not the human. It contains:
- A one-line entity definition (> Intelli is an AI-native Customer Engagement software platform by Intelli Holdings Inc...). This is the "blockquote summary" pattern the spec recommends.
- Structured entity facts: legal name, founding year (2024), HQ (Nairobi, Kenya), operations in Accra (Ghana), certifications (Meta Business Technology Provider), contact and social profiles.
- “What problems does Intelli solve?”, answered in a numbered, easy-to-extract format.
- Each core product with its canonical URL and a “Key fact” line.
- Pricing as explicit numbers ($20-$55/mo for Whatsapp plans).
- A FAQ block where every answer ends with a Source: URL. This is the key move. It hands the model a ready-made citation so it can attribute the claim to a specific page.
- A “last updated” date to signal freshness.
Now the honest part. llms.txt is not the “robots.txt for LLMs”. It doesn’t control access at all; robots.txt does that. It’s also not yet a proven ranking or citation lever:
- Google’s AI optimization guide says you don’t need llms.txt to appear in AI Overviews or AI Mode, and John Mueller has compared it to the old keywords meta tag, calling its impact “purely speculative for now.”
- SE Ranking’s analysis of nearly 300,000 domains found just 10.13% had an llms.txt file, and removing that variable from its citation-prediction model actually made the model more accurate. In other words, having the file told the model nothing about who gets cited.
- On the other side, Profound reported that crawlers from OpenAI and Microsoft do fetch llms.txt and llms-full.txt, and AI coding agents and browser agents increasingly read it.
So why did we keep it? Because it took an afternoon, it costs nothing to host, and it forced us to write one canonical, sourced description of Intelli. That description now feeds our homepage copy, our schema and our FAQs. Think of llms.txt as a great internal discipline that doesn’t hurt to have.
2. We explicitly invited AI crawlers in robots.txt
Files: public/robots.txt and app/robots.ts
AEO starts with access. If a crawler can’t read your pages, nothing else in this article matters.
Our first version named and explicitly allowed seven answer-engine user agents: GPTBot, ChatGPT-User, Google-Extended, PerplexityBot, anthropic-ai, ClaudeBot and Amazonbot. Each one got an explicit Allow: for /, /blog/, /llms.txt and /feed.xml, while private app routes (/api/, /auth/, /dashboard/, /onboarding/) stayed disallowed. The file ends with a Sitemap: directive.
When I checked each provider’s documentation for this article, I found that we had the right idea but the wrong list. AI companies now split their bots by job: one for training, one for search indexing and one for user-triggered fetches. The search bots are the ones that get you cited.
- OpenAI (docs): GPTBot is for training. OAI-SearchBot powers ChatGPT search, and OpenAI states that "sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers." ChatGPT-User handles user-initiated visits, and OpenAI notes robots.txt rules may not apply to it. We were missing OAI-SearchBot, the most important one for ChatGPT citations.
- Anthropic (docs): ClaudeBot is for training, Claude-SearchBot improves Claude's search results and Claude-User fetches pages when a user asks Claude a question. Anthropic says blocking the last two "may reduce your site's visibility." anthropic-ai is a legacy token; current Anthropic bots don't use it.
- Google: Google-Extended is a control token, not a separate crawler. It decides whether your content can be used for Gemini model training and grounding. Google says it "does not impact a site's inclusion or ranking in Google Search." AI Overviews and AI Mode are built from the regular Search index, so Googlebot is what gets you into them.
- Perplexity (docs): PerplexityBot surfaces and links sites in Perplexity's results and "is not used to crawl content for AI foundation models." Perplexity-User fetches pages for user questions and "generally ignores robots.txt rules."
- Apple (docs): Applebot-Extended lets you opt out of Apple's foundation model training. It doesn't crawl anything itself, and it doesn't affect Siri or Spotlight search visibility.
- Amazon (docs): Amazonbot "may be used to train Amazon AI models", and Amazon now also documents Amzn-SearchBot for search experiences and Amzn-User for user-triggered fetches.
Here’s the shape of our updated file:
# Answer-engine search and user-fetch bots (these get you cited)
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Amzn-SearchBot
Allow: /
Disallow: /api/
Disallow: /auth/
Disallow: /dashboard/
Disallow: /onboarding/
# Training and grounding tokens (a business decision; we allow them)
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Amazonbot
Allow: /
Disallow: /api/
Disallow: /auth/
Disallow: /dashboard/
Disallow: /onboarding/
Sitemap: https://intelliconcierge.com/sitemap.xml
Two things I learned the hard way:
- Silence is not the same as blocking. If your robots.txt doesn’t mention a bot, that bot is allowed by default. Naming them explicitly matters most when you have a broad Disallow rule.
- Your CDN can overrule your robots.txt. On July 1, 2025, Cloudflare started blocking AI crawlers by default for new domains and launched “Pay Per Crawl”, which lets publishers charge AI companies per request. Cloudflare sits in front of roughly a fifth of the web. That’s a reasonable choice for publishers protecting their content. But if you’re a SaaS company that wants to be recommended, check your CDN and firewall bot settings (Cloudflare, Vercel or whatever you use). Your robots.txt can say “welcome” while the edge says “access denied.”
Whether to allow training bots is a business decision. For a company like ours, which sells to businesses that are asking AI “what’s the best WhatsApp Business API platform in Africa?”, being in the training data is a plus for us, an advantage we would benefit from. A media agency or publisher whose content is the product might reasonably decide differently. To each their own.
3. We layered structured data (JSON-LD / Schema.org)
File: components/seo/JsonLd.tsx
This is a reusable component library that injects machine-readable Schema.org markup. We use six schema types, each placed where it earns its keep:
You can view the content of our schema here: https://share.gemini.google/TihJGllOTHSu
Organization and WebSite are injected globally so every page inherits the same entity context. The sameAs and knowsAbout arrays help a machine connect the dots: this brand = this company = these topics.
What structured data does and doesn’t do in 2026:
- Google narrowed FAQ rich results to government and health sites in August 2023 and dropped HowTo rich results the following month. In May 2026, Google retired FAQ rich results entirely. So FAQPage and HowTo markup won’t get you those expandable snippets in Google any more.
- Google’s AI optimization guide also says structured data isn’t required for its AI features and that there’s no special “AI schema.”
- I haven’t found solid public evidence that JSON-LD directly increases how often ChatGPT or Perplexity cite you. Most of what’s out there is correlation or vendor opinion.
So why keep it? Because the markup itself is still valid. It gives every crawler an unambiguous, machine-readable statement of facts (who we are, what we cost, when a post was published), and our pricing and organization data feed Google’s normal rich results and knowledge systems. Structured data is how you turn prose into declared facts. I just wouldn’t sell it as an AEO cheat code.
4. We built a machine-readable RSS feed
File: app/feed.xml/route.ts
There’s a live RSS 2.0 feed at /feed.xml, referenced in both robots.txt and llms.txt, that combines our CMS blog with my Medium feed. It gives crawlers a structured, dated, continuously updated content stream: a freshness and discovery channel beyond the sitemap.
One caution if you mix in syndicated posts like we do: make sure each article has one canonical home, so engines don’t see two competing copies of the same piece.
5. We generate a dynamic sitemap with priorities
File: app/sitemap.ts
This isn’t a static dump. It’s a generated sitemap that assigns deliberate priorities (1.0 for the homepage and core product pages, 0.7 for blog and comparison pages, lower for legal pages and the changelog). It pulls in blog posts dynamically from our CMS and Medium feeds, so new content is discoverable automatically.
A small honesty note: Google’s sitemap documentation states plainly that “Google ignores <priority> and <changefreq> values,” but it does use <lastmod> when it's consistently accurate. The value here is the automatic discovery and accurate <lastmod> dates, not the priority numbers. We keep the priorities mostly as documentation for ourselves.
6. We wrote answer-shaped content, for how users ask questions.
- Comparison pages (/compare/intelli-vs-wati, /compare/intelli-vs-respond-io), built to win "X vs Y" conversational queries, which is exactly how people phrase questions in Perplexity and ChatGPT.
- Listicle blog posts like /blog/ai-features-organizations ("7 Essential AI Features..."), with question-style H2 headings and short 40-80 word answers directly underneath. Engines can lift that format cleanly into a cited answer.
- A reusable FAQ component (components/seo/AeoFaqSection.tsx) that renders human-friendly accordions, while the paired FAQPageJsonLd provides the machine-readable version.
The research partly backs this up, with one caveat. Seer Interactive’s analysis of more than 2 million ChatGPT citations found listicles made up 16.6% of them. The same study saw listicle citations fall 30% between December 2025 and January 2026, so format alone won’t carry you. Meanwhile, the GEO paper showed that adding sources, quotes and statistics measurably improves visibility. Write the answer first, back it with evidence, then add the context.
7. We kept metadata disciplined and set canonical URLs across ~50 pages
- A root metadata block in the layout: title template, description, keywords, Open Graph, Twitter card and an RSS alternate link.
- Per-page metadata / generateMetadata on 30+ pages, including dynamic per-article OG tags (type: "article", publishedTime, a per-post image).
- A single getCanonicalUrl() helper (lib/metadata.ts) used everywhere, so every page declares one authoritative URL. This prevents the duplicate-content confusion that dilutes an entity's authority.
8. We looked after performance and crawl health (the unglamorous foundation)
- Next.js image optimization with a wide allowlist of CDNs (next.config.mjs), plus Vercel Speed Insights to track Core Web Vitals.
- Security headers (HSTS, CSP, nosniff). I don’t have solid proof that answer engines score security headers directly. What they do is keep the site reliable, safe and well maintained, and Google’s guidance consistently rewards sites that give people a good page experience.
- Don’t forget Bing. Microsoft says Bing powers AI tools including ChatGPT, and Copilot is built on Bing’s index. IndexNow lets you ping participating search engines the moment a URL is added, updated or deleted. In February 2026, Bing wrote that IndexNow “helps ensure that AI systems reference the most current version of a page when generating answers.” If you’re using react, vite or Next.js, wiring IndexNow into your publish flow takes very little code.
The part your codebase can’t do: get talked about
Everything above makes you readable. Getting chosen depends a lot on what the rest of the internet says about you.
- Mentions beat links. Ahrefs studied 75,000 brands and found that branded web mentions correlated with AI Overview visibility at 0.664, compared with 0.218 for backlinks. A follow-up found YouTube mentions were the strongest signal of all, at around 0.737. Correlation isn’t causation, but the direction is clear.
- Each engine has favourite sources. Profound analysed 680 million citations and found that Wikipedia makes up 7.8% of ChatGPT’s citations, while Reddit leads for Perplexity (6.6%) and Google AI Overviews (2.2%). Peec AI’s analysis of 30 million sources put Reddit, YouTube and LinkedIn at the top across engines.
For us, that means: honest answers in Reddit threads where people compare WhatsApp Business API providers, short product walkthroughs on YouTube, reviews on software directories, a complete Meta Business Partner listing and consistent company details everywhere our sameAs links point. Your schema says who you are; third parties confirm it.
Why this matters even more if you’re building in Africa
According to DataReportal’s July 2025 Global Statshot, 42.1% of Kenyan internet users aged 16 and over had used ChatGPT in the previous month. That was the highest rate of any country DataReportal tracked, ahead of South Africa (15.3%), Egypt (9.8%) and Nigeria (8.2%). Kenya was also third in the world for ChatGPT website traffic, behind only the US and India.
That’s our home market. When a business owner in Nairobi asks an AI assistant which WhatsApp Business API provider to use, we want Intelli in that answer, described accurately. There’s also an opportunity: much of the content AI engines have learned from about “customer engagement” was written for US and European buyers. African companies that publish clear, sourced, locally relevant answers have less competition for those questions.
How we measure it
You can’t improve what you don’t measure. Our setup at Intelli:
- GA4 AI traffic channel. Create a custom channel group (or an exploration filter) that matches referrers with a regex like chatgpt\.com|chat\.openai\.com|perplexity\.ai|gemini\.google\.com|claude\.ai|copilot\.microsoft\.com. Expect undercounting: some AI visits arrive without a referrer and show up as "direct."
- Google Search Console. Google now provides a Generative AI performance report for how your content performs in AI features in Search and Discover.
- Bing Webmaster Tools. Its AI Performance report shows how your content appears in Copilot and Bing’s AI summaries, including the “grounding queries” the AI used.
- Visibility tools. Profound, Microsoft Clarity, the Semrush AI Toolkit and Ahrefs Brand Radar track how often you’re mentioned or cited for a set of prompts across engines.
- Manual prompt testing. Every month, we run the same 20 prompts our customers would ask (“best WhatsApp Business API provider in Kenya”, “Intelli vs Wati”, “how much does WhatsApp API cost”) in ChatGPT, Perplexity, Gemini and Claude. We log whether Intelli appears, what it says about us and which URL it cites. It’s low-tech, but it’s the closest thing to seeing what your customers see.
tldr;
Here’s the full list of steps in case you didn’t read all the way through:
- Grant access → name the AI search and user-fetch bots (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User) in robots.txt, decide on training bots, and check your CDN isn’t blocking them.
- Speak the model’s language → publish an llms.txt with a tight entity summary and sourced FAQs (a cheap bet, not a guarantee).
- Declare your facts → add Organization, FAQPage, Product/SoftwareApplication and BlogPosting JSON-LD.
- Shape content as answers → comparison pages, question headings, 40–80 word answers, FAQs, plus real sources, quotes and statistics.
- Stay discoverable and fresh → dynamic sitemap with accurate lastmod dates, RSS feed, canonical URLs and IndexNow for Bing.
- Be a trustworthy source → fast pages, clean metadata, security headers and a consistent entity identity across your sameAs links.
- Get talked about → earn mentions on Instagram, LinkedIn, Reddit, YouTube, review sites and partner directories.
- Measure it → GA4 AI channel, Search Console, Bing AI Performance and a monthly prompt test.
If this was helpful, you can find more of my writing on Substack. I write about building in Africa, AI, startups and the things nobody tells you about shipping product. If you happen to run a B2C business currently overwhelmed with automating your support, marketing and customer engagement, visit https://intelliconcierge.com and sign up. If you are a developer looking to embed messaging in your app/product visit https://intelliconcierge.dev. Cheers.