What exactly does this index measure?

The share of .fr sites that name an AI crawler in their robots.txt and forbid it the root, for ten crawlers followed one by one, and the share that names them at all, whether to turn them away or to welcome them.

Where does the panel come from?

From the same random draw as the WordPress patch-lag index, in Afnic's open data file. The seed and the file's date are enough to rebuild the same panel.

Does a site that blocks every crawler count?

It is recorded, and counted apart. An instruction that closes the root to every crawler is about crawling in general, search engines included; the index counts the sites that name an AI crawler.

AI crawler access to French websites

What .fr sites answer the AI crawlers in their robots.txt, agent by agent.

As of the 03/09/2026 reading, 2.2% of the .fr sites whose policy is settled name at least one AI crawler and turn it away.

Turn away at least one AI crawler

2.2 %

Name at least one AI crawler

3.0 %

Publish a robots.txt

85.3 %

Agent by agent

Each AI crawler has its own group in a robots.txt: a site answers OpenAI, Google and Common Crawl separately, and that is what the index follows.

Crawler	Operated by	Sites turning it away
CCBot	Common Crawl	2.0 %
GPTBot	OpenAI	2.0 %
Bytespider	ByteDance	1.8 %
ClaudeBot	Anthropic	1.8 %
Google-Extended	Google	1.8 %
Meta-ExternalAgent	Meta	1.7 %
Applebot-Extended	Apple	1.6 %
ChatGPT-User	OpenAI	0.3 %
PerplexityBot	Perplexity	0.1 %
OAI-SearchBot	OpenAI	0.1 %

Separately, 3.1% of sites close the root to every crawler at once, search engines included. That instruction is about crawling in general, and it is counted apart from the figures above.

Week after week

See the figures as a table

Date	Turn away at least one AI crawler
2026-09-03	2.2 %

Method

The panel: The same panel as the patch-lag index: 15,000 domains drawn at random from Afnic's open data file, which lists the 4,590,553 active .fr domains. The draw orders every domain by the sha256 fingerprint of its name, and the published seed is enough to rebuild it.
The cadence: The robots.txt is read every Monday, in the pass that already opens each panel domain. The panel stays the same from one week to the next: a move in the series is the parc changing its mind.
The reading: A crawler counts as turned away when its own group forbids the site's root. A wildcard group that forbids everything is recorded apart, since it is about crawling in general.
The base: The figures cover the sites whose policy is settled: those that serve a robots.txt, and those that answer they have none, which allows everything. That is 98% of the panel's sites that serve a page.
The agents followed: 10 AI crawlers, from the model that trains to the engine that answers. The list is frozen with the index version, since it decides what "at least one" counts.
The API: GET https://api.stileex.xyz/v1/series/fr/ai-crawler-blocked-share/latest

Measured on the same panel, in the same weekly pass: WordPress patch lag in France