Operated by Google
Google-ExtendedAbout this crawler
Google is Google's model training crawler. Collects page content that may be used to train or improve a model. Nobody is waiting on the other end, and nothing is cited back at the time of the visit.
A request from Google usually means the page content is being collected and may be used to train or improve a model, and nobody is reading it right now.
User agent
Match on the token, not the whole header. Operators change version numbers and the surrounding text freely, so a substring match on the token is the only stable rule.
Mozilla/5.0 (compatible; Google-Extended/1.0; +https://developers.google.com/search/docs/crawling-indexing/google-extended)
Typical shape. The exact header varies by operator and version.
Blocking it in robots.txt
User-agent: Google-Extended Disallow: /
robots.txt is a request, not enforcement. Well behaved crawlers honour it; nothing stops one that does not.
Verifying a request really came from Google
Scrapers routinely borrow a well known crawler's user agent to slip past rate limits. Checking takes three pieces.
1. User agent
What the request claims to be. It is a plain header that anyone can set, so on its own it proves nothing.
2. Request IP
The address the request actually arrived from. Your server sees it directly, and it cannot be set by the caller the way a header can.
3. Range check
The operator publishes no fixed IP list. Instead, the address a request came from is looked up in reverse DNS, the resulting hostname has to end in one of the operator’s domains, and that hostname has to resolve forward to the same address.
Approved hostname endings: googlebot.comgoogle.com.
This operator publishes no fixed address blocks, so there is nothing to list here. The reverse DNS check above is the documented method.
More from Google
Cloud Vertex Bot
CloudVertexBotModel training
Gemini
Gemini-Deep-ResearchAI answers
Google Agent
Google-AgentAutonomous agents
Google Cloud Vertex Bot
Google-CloudVertexBotModel training
Google Firebase AI
Google-FirebaseAutonomous agents
Gemini CLI
Google-Gemini-CLIAutonomous agents
Google Inspection Tool
Google-InspectionToolAI search indexing
NotebookLM
Google-NotebookLMAI answers
Project Mariner
GoogleAgent-MarinerAutonomous agents
Google URL Context
GoogleAgent-URLContextAI answers
GooglebotAI search indexing
GoogleOtherModel training
GoogleOther Image
GoogleOther-ImageModel training
GoogleOther Video
GoogleOther-VideoModel training
Is Google allowed on your site?
Zenovay reads your robots.txt and llms.txt and shows, crawler by crawler, which AI products you currently allow and which you block. It also tracks the AI assistants that send you real visitors, and ties those visits to revenue.
Start free