Codeberg.org on Nostr: There is rate limiting in place, and there is robots.txt. If a crawler peaks Google, ...

There is rate limiting in place, and there is robots.txt. If a crawler peaks Google, Microsoft and the other big players, it typically means they do more than they are allowed to.

You don't need to ask questions in this case.