I just blocked the following ASN(s) from being able to hit twtxt.net
or mills.io
:
16509 - AMAZON-02
32934 - FACEBOOK
Why? Because the Claude Bot web crawler from facebookexternalhit and Metaโs facebookexternalhit web crawler are both behaving badly for pages that have no cache headers. Not sure if this is malicious, an oversight, a bug or me just being stupid and not ensuring every web resource or page had appropriate Cache
headers? ๐ค In any case, until I hear back from at least facebookexternalhit (whom Iโve reached out to), these ASN(s) will remain entirely blocked.
That is the entirety of Amazon Web Services and Facebook.