检测内容
读取 robots.txt，了解网站对爬虫的访问规则。
用途说明
将检测到的公开配置与网站预期设置核对，可帮助定位访问、证书和抓取问题。结果反映检测当时的响应，网络环境或网站防护可能影响部分项目。
抓取规则
读取 robots.txt，了解网站对爬虫的访问规则。
技术术语
- Search Engine Crawling
- robots.txt can ask search engines to stay away from some or all of a site.
- AI Crawlers
- Bots that fetch pages to train AI models or answer questions, like GPTBot and ClaudeBot. Sites can ask them to stay away in robots.txt, though not all listen.
- llms.txt
- A proposed file at /llms.txt that gives AI tools a short, plain summary of a site and links to its key pages.