Google Explains Robots.txt: Four Supported Fields and the Limits of AI Crawler Rules
Google's new robots.txt explainer confirms that an invented instruction may be ignored. Here is how to separate crawl rules, Search visibility controls, and Google-Extended's AI use preferences.

























