I asked Perplexity if it was ignoring robots.txt instructions when crawling the web, and the response was quite detailed -- and revealing.
It basically confessed.
https://www.perplexity.ai/page/Does-Perplexity-Ignore-v4ZEwKKCRme6wOo6PvEuzg
Dan Gillmor /
npub1qy…yayrq
2024-06-22 07:20:23
Author Public Key
npub1qyhnuy0ac92x497z0ytx95dwd2c6p5lv8l36uv0gv8jeel0pu0ks9yayrqPublished at
2024-06-22 07:20:23Event JSON
{
"id": "3f90a27767e2176db5c439b05e101afd6e142c50dd708d872be50b95a2db3394",
"pubkey": "012f3e11fdc1546a97c2791662d1ae6ab1a0d3ec3fe3ae31e861e59cfde1e3ed",
"created_at": 1719040823,
"kind": 1,
"tags": [
[
"proxy",
"https://mastodon.social/users/dangillmor/statuses/112659059409050551",
"activitypub"
]
],
"content": "I asked Perplexity if it was ignoring robots.txt instructions when crawling the web, and the response was quite detailed -- and revealing.\n\nIt basically confessed. \n\nhttps://www.perplexity.ai/page/Does-Perplexity-Ignore-v4ZEwKKCRme6wOo6PvEuzg",
"sig": "0e5cb0d9241c451694a57a9d421b015aa5b89729ec871920efd91df57c64edaa2a973338c9986238ebae4b5477c2103b4010fcacdecfbe0164c5ad0cb47e14c4"
}