Google quietly added a brand new bot to their crawler documentation that crawls on behalf of business shoppers of their Vertex AI product. The documentation says that the brand new crawler might crawl websites on the website proprietor’s request.
Vertex AI Brokers
The brand new crawler, referred to as Google-CloudVertexBot, crawls web sites content material for Vertex AI shoppers, which is totally different from the opposite bots listed within the Search Central documentation which might be tied to Google Search or promoting.
The official Google Cloud documentation presents the next info:
“In Vertex AI Agent Builder, there are numerous varieties of information shops. An information retailer can include just one kind of information.”
It goes on to record six forms of knowledge, one in every of which is public web site knowledge. On crawling, the documentation says that there are two sorts of web site crawling:
- Primary web site indexing
- Superior web site indexing
Documentation
The documentation explains web site knowledge:
“An information retailer with web site knowledge makes use of knowledge listed from public web sites. You possibly can present a set of domains and arrange search or suggestions over knowledge crawled from the domains. This knowledge contains textual content and pictures tagged with metadata.”
The outline of Primary web site indexing doesn’t say something about website proprietor verification however somebody from Google contacted me and suggested that the fundamental web site indexing simply makes use of of slice of what’s already crawled by Google.
Superior web site indexing, which makes use of the brand new Google-CloudVertexBot, requires area verification and there are indexing quotas. It seems that the brand new crawler isn’t crawling public web sites however fairly it crawls on the “website homeowners’ request” so it could be that it gained’t come crawling public websites.
The Changelog notation for this new crawler says this:.
Right here’s what the changelog says:
“Introducing the Google-CloudVertexBot crawler
What: Added Google-CloudVertexBot to the record of Google crawlers, a brand new crawler that crawls websites on the location homeowners’ request when constructing Vertex AI Brokers.
Why: The brand new crawler was launched to assist website homeowners determine the brand new crawler visitors.”
New Google Crawler
The brand new crawler is named Google-CloudVertexBot.
That is the brand new info on it:
“Google-CloudVertexBot crawls websites on the location homeowners’ request when constructing Vertex AI Brokers.
Person agent tokens
- Google-CloudVertexBot
- Googlebot”
Person agent substring
Google-CloudVertexBot
Google-CloudVertexBot
The documentation signifies that the brand new crawler doesn’t index public websites and the changelog signifies that it was added in order that website homeowners can determine visitors from the brand new crawler. Do you have to block the brand new crawler with a robots.txt simply in case? It seems to not be needed so as to add it to the robots.txt as a result of it solely crawls by website proprietor’s request.
Learn Google’s new documentation:
Google-CloudVertexBot
Featured Picture by Shutterstock/ShotPrime Studio
