Raphael Lullis on Nostr: robots.txt is no contract, no one should expect to be protected from being crawled ...

robots.txt is no contract, no one should expect to be protected from being crawled just because they placed some hints indicating what user agents are welcome.

Relying on consent and general good behavior is just bad digital hygiene. I'd rather people just acknowledged that and make it a basic principle: do not post anything on the open web unless you are sure you don't mind it being available to any present or future malicious/adversarial entity.