Network & IT · Security · 2026-09-09 04:13:00
On 7 September 2026 the author fetched robots.txt for the top 5,000 Tranco sites; 2,771 served one. 535 block GPTBot, and 238 of those also block OAI-SearchBot, while 297 block GPTBot alone. 296 sites block at least one AI search crawler yet still allow Googlebot, including facebook, instagram, twitter, amazon, and major news publishers. Big publishers often do this deliberately for licensing; smaller businesses almost always unintentionally. Roughly one in five sites that allow AI crawlers still offer nothing to read, and llms.txt adoption is 7.5%.
Source: ai-visibility.lastminutedealshq.com
Auto-generated summary for reference only. RSS