Google’s John Mueller responded to a query on LinkedIn to debate using an unsupported noindex directive on the robots.txt of his personal private web site. He defined the professionals and cons of search engine assist for the directive and provided insights into Google’s inner discussions about supporting it.
John Mueller’s Robots.txt
Mueller’s robots.txt has been a subject of dialog for the previous week due to the overall weirdness of the odd and non-standard directives he used inside it.
It was nearly inevitable that Mueller’s robots.txt was scrutinized and went viral within the search advertising and marketing neighborhood.
Noindex Directive
Every thing that’s in a robots.txt is known as a directive. A directive is a request to an internet crawler that it’s obligated to obey (if it obeys robots.txt directives).
There are requirements for tips on how to write a robots.txt directive and something that doesn’t conform to these these requirements is prone to be ignored. A non-standard directive in Mueller’s robots.txt caught the attention of somebody who determined to publish a query about it to John Mueller through LinkedIn, to know if Google supported the non-standard directive.
It’s a superb query as a result of it’s straightforward to imagine that if a Googler is utilizing it then possibly Google helps it.
The non-standard directive was noindex. Noindex is part of the meta robots commonplace however not the robots.txt commonplace. Mueller had not only one occasion of the noindex directive, he had 5,506 noindex directives.
The search engine marketing specialist who requested the query, Mahek Giri, wrote:
“In John Mueller’s robots.txt file,
there’s an uncommon command:
“noindex:”
This command isn’t a part of the usual robots.txt format,
So do you assume it’s going to have any affect on how search engine indexes his pages?
John Mueller curious to learn about noindex: in robots.txt”
Why Noindex Directive In Robots.txt Is Unsupported By Google
Google’s John Mueller answered that it was unsupported.
Mueller answered:
“That is an unsupported directive, it doesn’t do something.”
Mueller then went on to clarify that Google had at one time thought-about supporting the noindex directive from inside the robots.txt as a result of it could present a approach for publishers to dam Google from each crawling and indexing content material on the identical time.
Proper now it’s potential to dam crawling in robots.txt or to dam indexing with the meta robots noindex directive. However you possibly can’t block indexing with the meta robots directive and block crawling within the robots.txt on the identical time as a result of a block on the crawl will stop the crawler from “seeing” the meta robots directive.
Mueller defined why Google determined to not transfer forward with the concept of honoring the noindex directive inside the robots.txt.
He wrote:
“There have been many discussions about whether or not it needs to be supported as a part of the robots.txt commonplace. The thought behind it was that it could be good to dam each crawling and indexing on the identical time. With robots.txt, you possibly can block crawling, or you possibly can block indexing (with a robots meta tag, in the event you enable crawling). The thought was that you can have a “noindex” in robots.txt too, and block each.
Sadly, as a result of many individuals copy & paste robots.txt recordsdata with out them intimately (few folks look so far as you probably did!), it could be very, very straightforward for somebody to take away crucial elements of a web site by chance. And so, it was determined that this shouldn’t be a supported director, or part of the robots.txt commonplace… most likely over 10 years in the past at this level.”
Why Was That Noindex In Mueller’s Robots.txt
Mueller made clear that it’s unlikely that Google would assist that tag and that this was confirmed about ten years in the past. The revelation about these inner discussions is fascinating however it’s additionally deepens the sense of weirdness about Mueller’s robots.txt.
See additionally: 8 Frequent Robots.txt Points And How To Repair Them
Featured Picture by Shutterstock/Kues
