:twiz: (npub1ygr…cg6z) At no point did I actually think to look at their robots.txt… Honestly this is a good ledger of ‘known bad crawlers’ lol.
I found out years ago that (at that time anyways) they had a bunch of regular expressions saved off somewhere to match known spammy/URL shortner URLs so I grabbed those to use to filter my guestbook spam lol