Fetch or paste robots.txt rules and test whether a URL is allowed for a crawler.
Robots.txt controls crawling, not indexing. A blocked URL can still appear in search if Google discovers it elsewhere. Use noindex for index control and robots.txt for crawl control.