#AskFedi
Okay ... I am working on developing a counter-culture #RobotsTxt (robots.txt) file that excludes the major search engine crawlers ('cuz fuck them), but allows for assorted hobbyist, experimental, small-web-focused and other non-corporate crawlers ... I'm looking at search engines like https://wiby.me/ and https://search.marginalia.nu/ and https://stract.com/ and the like.
In the interests of not re-inventing the wheel, does anyone know of any good examples, or resources along these lines?
Thanks in advance
Eric the Cerise /
npub17a…y06g6
2024-05-08 13:10:07
Author Public Key
npub17azv0qt0crmghsm2d09xpe9yvn97ge7x3f6jpu79e0nfa50m3rfq4y06g6Published at
2024-05-08 13:10:07Event JSON
{
"id": "a78ccc074294990dbe8e80670f9e3e80111d459e25f57faad4732729dae96704",
"pubkey": "f744c7816fc0f68bc36a6bca60e4a464cbe467c68a7520f3c5cbe69ed1fb88d2",
"created_at": 1715173807,
"kind": 1,
"tags": [
[
"t",
"askfedi"
],
[
"t",
"robotstxt"
],
[
"proxy",
"https://kolektiva.social/users/ErictheCerise/statuses/112405630665660016",
"activitypub"
]
],
"content": "#AskFedi \n\nOkay ... I am working on developing a counter-culture #RobotsTxt (robots.txt) file that excludes the major search engine crawlers ('cuz fuck them), but allows for assorted hobbyist, experimental, small-web-focused and other non-corporate crawlers ... I'm looking at search engines like https://wiby.me/ and https://search.marginalia.nu/ and https://stract.com/ and the like.\n\nIn the interests of not re-inventing the wheel, does anyone know of any good examples, or resources along these lines?\n\nThanks in advance",
"sig": "b7afa5e6c7a387907100d3e506f92b07a20a2a88017843a03ddcfe42514ad50fdf4491846161077e1a1d1dd78b0d2f5d6c1e0ad083df35265ae80c3e1498db35"
}