.Google.com's John Mueller answered a concern concerning why Google marks webpages that are refused coming from creeping by robots.txt and why the it is actually safe to neglect the related Look Console documents about those crawls.Crawler Traffic To Inquiry Specification URLs.The individual talking to the question recorded that bots were actually making web links to non-existent concern criterion URLs (? q= xyz) to webpages with noindex meta tags that are actually likewise blocked in robots.txt. What cued the inquiry is actually that Google.com is actually crawling the hyperlinks to those web pages, acquiring blocked out by robots.txt (without noticing a noindex robots meta tag) then acquiring shown up in Google.com Search Console as "Indexed, though obstructed through robots.txt.".The individual talked to the following concern:." Yet here's the big concern: why would certainly Google.com mark pages when they can not even observe the content? What is actually the advantage because?".Google.com's John Mueller confirmed that if they can not crawl the webpage they can not see the noindex meta tag. He also produces a fascinating reference of the web site: hunt operator, recommending to overlook the outcomes because the "typical" customers will not view those end results.He composed:." Yes, you're correct: if our team can't creep the webpage, our team can not observe the noindex. That mentioned, if our experts can't creep the webpages, after that there is actually certainly not a whole lot for us to mark. Thus while you might view a few of those pages with a targeted web site:- question, the common individual will not see them, so I definitely would not bother it. Noindex is additionally fine (without robots.txt disallow), it only implies the Links will certainly end up being crept (and also wind up in the Browse Console report for crawled/not indexed-- neither of these conditions induce concerns to the remainder of the internet site). The fundamental part is that you do not make all of them crawlable + indexable.".Takeaways:.1. Mueller's solution verifies the limits in operation the Web site: search accelerated hunt driver for analysis causes. One of those explanations is actually because it's certainly not connected to the routine hunt index, it's a separate point entirely.Google.com's John Mueller commented on the website hunt operator in 2021:." The quick solution is actually that a site: inquiry is not suggested to become total, nor utilized for diagnostics functions.A site query is a details sort of search that limits the outcomes to a particular internet site. It's basically simply words site, a colon, and afterwards the web site's domain.This question restricts the end results to a certain website. It's not indicated to become a thorough selection of all the webpages coming from that website.".2. Noindex tag without utilizing a robots.txt is actually fine for these kinds of scenarios where a bot is actually linking to non-existent web pages that are actually acquiring uncovered through Googlebot.3. Links along with the noindex tag are going to produce a "crawled/not listed" item in Look Console and also those will not have a negative result on the rest of the site.Read the question as well as address on LinkedIn:.Why will Google mark webpages when they can not also find the information?Featured Graphic through Shutterstock/Krakenimages. com.