Show HN: Which big sites block AI training crawlers but let AI search in
Which of 945 well-known websites block GPTBot, ClaudeBot, PerplexityBot and other AI crawlers in robots.txt, by crawler and by kind of site. 25.9% block at least one.
Which of 945 well-known websites block GPTBot, ClaudeBot, PerplexityBot and other AI crawlers in robots.txt, by crawler and by kind of site. 25.9% block at least one.