Undercover AI Agents · watching one file

theguardian.com
robots.txt, over time.

Currently blocks 9 of the agents on the register: Claude-SearchBot, Claude-User, ClaudeBot, PerplexityBot, Meta-ExternalAgent, Bytespider, Amazonbot, CCBot, Applebot. Last read Sat, 12 Sep 2026 08:23 UTC; watched since Fri, 04 Sep 2026 17:01 UTC.

Follow this page as a feed: /watch/theguardian.com/feed.xml — works in any feed reader, Slack’s /feed, or an automation. Or get it by email.

Changes

1. No change recorded yet. The first read was Fri, 04 Sep 2026 17:01 UTC; each read since has found the same policy.

Read the file as it stands · Check another domain