AI · Data · 2026-10-27 13:18
A scan of robots.txt files across 945 popular websites found that 69 sites block AI training crawlers while still allowing all AI search crawlers, revealing a deliberate split between training and search access.
Auto-generated summary for reference only. RSS