Search Engine Optimization (SEO) was, and still is, one of the main ways for websites to acquire traffic. The principle was relatively simple: produce relevant content, make it easier for search engines to index it, and improve its ranking in search results. I've been doing this for more than 10 years. Time flies...

But this model is evolving with the integration of artificial intelligence into search engines. Google now provides AI-generated answers through AI Overviews and AI Mode. Meanwhile, ChatGPT and Perplexity allow users to perform conversational searches and summarize information from different sources.

I'm seeing the consequences of this shift on my own websites 🙁 Despite regularly working on SEO and publishing new content, some pages are generating fewer visits than before. Why visit a website when you can now get an answer directly through an AI interface, without necessarily clicking on the websites that provided the information?

This is the context in which GEO (Generative Engine Optimization) emerged. It brings together practices aimed at improving a website's visibility in AI-generated answers.

But behind this new marketing term, there are some interesting technical developments: new content retrieval mechanisms, specialized crawlers, semantic search, Retrieval-Augmented Generation (RAG), and new methods for measuring visibility.

Let's look at what really differentiates SEO from GEO from a technical perspective, and see what technical adjustments are worth considering.

SEO and GEO: Two different objectives

SEO (Search Engine Optimization) consists of optimizing a website to improve its visibility in traditional search engine results.

GEO (Generative Engine Optimization), on the other hand, aims to increase the likelihood that a website's content will be used, mentioned, or cited in answers generated by artificial intelligence systems.

The difference is therefore not just about how content is written, but also about the desired outcome.

With SEO, the main goal is to rank a URL for a given query. With GEO, the goal is to make sure that information available on a website is selected as a source when an answer is generated.

CriterionSEOGEO
ObjectiveRank a page in search resultsGet content included in an AI-generated answer
DiscoveryCrawling and link explorationCrawling, indexing, and source retrieval
SearchMatching queries with documentsLexical and semantic search
ProcessingPage analysis and rankingDocument retrieval followed by generation
ResultURL in a search results pageCitation, mention, or link in an answer
MetricsRankings, impressions, CTR, clicksCitations, mentions, and referral traffic

However, we should avoid an overly simplistic distinction. AI search engines don't all work in the same way, and some rely directly on traditional search engine infrastructure. Anyway, it's not that simple.

How do AI search engines retrieve information?

To understand the differences between SEO and GEO, we need to look at how the search systems used by language models work. A Large Language Model (LLM) has knowledge acquired during its training. However, this knowledge is not necessarily up to date and doesn't always allow it to answer questions about recent information.

To overcome this limitation, some systems use RAG (Retrieval-Augmented Generation).

Understanding how RAG works

The principle behind RAG is to retrieve relevant documents before asking the LLM to generate an answer.

A simplified pipeline (the goal isn't to build one, but simply to understand the principle) can be represented as follows:

User question
        |
        v
Query analysis
        |
        v
Search for relevant documents
        |
        v
Content retrieval
        |
        v
Information selection
        |
        v
Context construction
        |
        v
Answer generation by the LLM
        |
        v
Answer with possible citations

Document retrieval can rely on different techniques.

Lexical search: the system searches for documents containing terms that match the query. BM25 is an example of an algorithm used to rank documents according to their textual relevance.

Vector search: texts are transformed into numerical representations called embeddings. These vectors make it possible to find content with semantic similarities, even when the words used are different.

Hybrid search: both approaches can be combined to benefit from their respective strengths.

For example, a search for "how to check the structure of a JSON document" could retrieve an article entitled "Validate a JSON file with a schema". The terms are not identical, but both pieces of content cover a similar topic.

It's important to point out that these mechanisms describe a general RAG architecture. Commercial search engines can use much more complex infrastructures, with multiple indexes, ranking models, and selection stages.

Query Fan-Out: Breaking down a question into multiple searches

Another interesting development, which I wasn't familiar with before writing this article, is Query Fan-Out. Instead of treating a complex question as a single query, an AI search engine can break it down into several more specific searches.

Let's imagine the following question:

"What are the best websites for working with JSON data?"

The search engine could perform several complementary searches:

  • Websites offering an online JSON validator.
  • JSON editors accessible from a browser.
  • Websites for comparing two JSON documents.
  • Online tools for formatting and minifying JSON.

The results can then be grouped together to build a more complete answer.

Google specifically documents the use of this technique in AI Overviews and AI Mode. This means that a page could potentially be relevant to a sub-question, even if it doesn't directly target the original query!

This development makes targeting a single main keyword less representative of how certain searches are processed. It also gives us an idea of some of the differences GEO introduces compared to SEO.

AI Crawlers: Understanding the new user-agents

From a webmaster's perspective, one of the first observable differences concerns the bots that visit a website. Historically, we mainly monitored Googlebot and sometimes Bingbot (depending on your country). Now, several platforms have their own crawlers.

These bots don't necessarily all have the same purpose.

The different OpenAI bots

OpenAI distinguishes between three main User-Agents:

User-AgentPurpose
GPTBotCrawling content that may be used to train models
OAI-SearchBotCrawling for ChatGPT search features
ChatGPT-UserAccess to pages triggered by certain user actions

This distinction is important: allowing a model to be trained on your content and allowing that content to be used in search results are two different decisions.

For example, you can allow OAI-SearchBot while blocking GPTBot.

Here is a corresponding robots.txt configuration:

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: ChatGPT-User
Allow: /

This configuration is an example of an access policy, not a guarantee of ranking or citation.

There are also bots associated with other platforms. It's therefore worth checking their official documentation to understand their purposes and User-Agents.

Analyzing crawler visits in HTTP logs

One practical way to monitor AI bot activity is to analyze your web server's access logs. With Nginx, the following information can be particularly useful:

  • Client IP address.
  • Requested URL.
  • HTTP method.
  • Response code.
  • Response time.
  • User-Agent.

For example, a custom log format can be defined as follows:

log_format crawler '$remote_addr - $time_local '
      '"$request" $status $body_bytes_sent '
      '"$http_referer" "$http_user_agent" '
      '$request_time';

You can then search for requests associated with the different bots.

Command-line example:

grep -Ei 'GPTBot|OAI-SearchBot|ChatGPT-User|Googlebot|PerplexityBot' \
    /var/log/nginx/access.log

To get an initial statistical overview:

grep -Ei 'GPTBot|OAI-SearchBot|ChatGPT-User|PerplexityBot' \
    /var/log/nginx/access.log \
    | awk -F'"' '{print $6}' \
    | sort \
    | uniq -c \
    | sort -nr

This allows you to identify the most active User-Agents.

However, be careful: a User-Agent can be easily spoofed. For reliable analysis, you should also verify IP addresses using the ranges published by the relevant providers or their official verification methods.

Finally, the presence of a crawler in your logs doesn't prove that the retrieved content will be used in an AI-generated answer. It simply indicates that a resource was requested.

JavaScript rendering and content accessibility

Another technical topic concerns page rendering. Some websites rely heavily on JavaScript to generate their content. In an application built with React, Vue, or Angular, the server may sometimes return an almost empty HTML document before JavaScript actually builds the page.

Example of an initial HTML response:

<!DOCTYPE html>
<html>
<head>
    <title>JSON validator</title>
</head>
<body>
    <div id="app"></div>
    <script src="/assets/app.js"></script>
</body>
</html>

The main content only appears after JavaScript has been executed.

Googlebot has a rendering system based on Chromium and can execute JavaScript to analyze generated content. However, you shouldn't assume that all AI crawlers have the same capabilities. Some may simply retrieve the HTML returned by the server.

It's therefore preferable to provide the main content directly in the HTTP response whenever possible.

Several approaches make this possible:

  • SSR (Server-Side Rendering): the server generates the page's HTML before sending it to the browser.
  • SSG (Static Site Generation): pages are generated in advance during the build process.
  • Prerendering: a pre-generated HTML version is provided for the relevant pages.

These techniques are not specific to GEO. They also improve performance, accessibility, and compatibility with crawling tools.

To check the content actually sent by the server, you can use curl:

curl -L https://example.com/article

You can then examine the returned HTML and check that important information is present without executing JavaScript.

You can also compare this response with the DOM obtained after rendering the page in a browser.

JSON-LD and structured data: What’s their value for GEO?

Structured data makes it possible to describe page information in a format that can be understood by systems supporting the vocabulary being used.

Schema.org provides several entity types, including:

  • Article.
  • Organization.
  • Product.
  • SoftwareApplication.
  • FAQPage.
  • HowTo.

The JSON-LD format is particularly convenient because it allows this information to be declared in a block separate from the HTML content.

Example:

<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "Article",
  "headline": "SEO vs GEO: Technical differences",
  "description": "Understanding how AI search engines retrieve website content.",
  "author": {
    "@type": "Organization",
    "name": "Jsoning"
  },
  "publisher": {
    "@type": "Organization",
    "name": "Jsoning"
  }
}
</script>

This data makes it possible to explicitly describe the nature of content and certain relationships between entities.

However, we need to distinguish between two things:

  1. A system's ability to understand structured data.
  2. The actual selection of a page as a source in a generated answer.

Using JSON-LD absolutely does not guarantee that a page will be cited by ChatGPT, Perplexity, or Google AI Overviews.

Google also points out that no specific Schema.org markup is required to appear in its generative features. Personally, I stopped using JSON-LD a few years ago.

Structured data remains useful, however, for systems that support it, particularly rich results in traditional search engines.

Robots.txt, HTTP directives, and indexing control

GEO also raises an important question: how can you control the use of your content by different AI systems? We need to distinguish between several mechanisms.

Robots.txt

The robots.txt file allows you to communicate crawling rules to bots.

You can define different rules depending on the User-Agent, as we saw earlier.

However, this is not a security mechanism: a bot can ignore these directives.

Meta robots and X-Robots-tag

Robots directives allow you to control certain aspects of indexing and content display in search engines that support them.

For example:

<meta name="robots" content="noindex, nofollow">

You can also send directives through HTTP headers:

X-Robots-Tag: noindex

Other directives exist, such as nosnippet, max-snippet, and data-nosnippet, which can be used to control snippets displayed in certain search results.

With Google, AI features integrated into Search use existing control mechanisms for Googlebot. Therefore, simply blocking Google-Extended is not enough to prevent Google from crawling a page intended for its search results.

Should You Block AI Crawlers?

The answer depends on the website owner's strategy. A publisher may want to prevent their content from being used to train models while still wanting to appear in AI search engine answers.

Conversely, a website may prefer to restrict certain automated access to reduce server resource consumption or protect its content.

You should therefore examine the purpose of each bot separately before defining your access policy.

Still, it's not necessarily easy to put all this in place.

Measuring SEO and GEO performance: Which metrics should you use?

Performance analysis is another area where the two approaches differ.

For SEO, the tools are relatively mature. Google Search Console provides impressions, clicks, CTR, and average position, among other metrics.

For GEO, measurement is more complex.

Tracking citations in AI-generated answers

One approach is to build a set of queries representative of your business and test them regularly across different platforms.

For a website specializing in development tools, for example, you could track:

  • What are the best online JSON tools?
  • How can you compare two JSON objects?
  • Which tools can validate a JSON file?
  • How can you generate JSONL data?

But how do you collect this information?

There are several possibilities:

  • Manually: regularly enter queries into ChatGPT, Perplexity, Gemini, or other assistants, then record the websites mentioned and any sources cited.
  • With specialized tools: some AI search visibility platforms allow you to configure queries and automatically monitor mentions, citations, and competitors.
  • Through automation: use the APIs offered by some platforms to send a list of questions, retrieve the answers, and analyze any URLs they contain. However, this approach requires you to develop your own data collection and analysis system.

Identifying traffic from AI assistants

Another approach is to examine traffic sources in your audience analytics tools. In particular, you can monitor referrals from domains such as:

  • chatgpt.com
  • perplexity.ai
  • copilot.microsoft.com

This allows you to measure some of the visits generated by assistants. However, this method doesn't account for users who discover a website in an AI-generated answer and visit it later through direct access or another source.

GEO: Are the New Practices Really Different from SEO?

After examining the technical mechanisms, one question remains: do we need to completely rethink how we develop and optimize a website? Not necessarily.

Google states that SEO fundamentals still apply to its generative features. In particular, a page must be accessible, indexable, and eligible to appear in search results.

There is no special markup that guarantees visibility in AI Overviews or AI Mode. The same applies to some popular practices in the GEO field.

For example, the llms.txt file is a proposal intended to provide AI systems with a structured overview of a website's resources. It may be useful to some tools, but Google says it doesn't use it to improve visibility in Search. So we'll have to see how adoption develops.

Optimizing for Google AI Overviews therefore doesn't guarantee better visibility in ChatGPT or Perplexity. Ouch.

Conclusion

SEO and GEO don't share the same objectives, but they have a large part of their technical foundations in common.

While SEO aims to improve rankings in search results, GEO seeks to make content usable and citable by artificial intelligence systems.

For developers and webmasters, this means looking at how these systems discover pages, retrieve content, and use it. HTTP log analysis, crawler management, JavaScript rendering, and structured data all remain important topics.

GEO probably isn't a complete break from SEO, but rather a new dimension of web visibility.

On my websites, AI assistants currently bring me 20 times less traffic than search engines... I'm not sure it's worth spending time on GEO for everyone... At least not for now.

SEO vs GEO: What are the technical differences in the age of AI search engines?
Popular posts