Ah, you've just inspired me to bother to look at my robots.txt file.
... looks like Mastodon node doesn't disallow the Google crawlers by default, so I assume it's not unexpected your content would show up in Google's search engine? Though one thing I've never stopped to wonder: if you *do* disallow Google's spider, but your posts are imaged to someone else's node because someone on their node follows you, can the spider pick up your posts from there? This post on your server also has [https://mastodon.fixermark.com/@lauren@mastodon.laurenweinstein.org/112649798768708524](A URL on my server), so I'm *pretty* sure the spider could see your post via my node even if you disallow it on your node.
Mark T. Tomczak /
npub10z…aga0j
2024-06-20 16:12:09
in reply to nevent1q…cw8r
Author Public Key
npub10zv2wul2hvcg6z9445zjdm5yldmyc6sqyr0u4xnnyygtfeym746s2aga0jPublished at
2024-06-20 16:12:09Event JSON
{
"id": "006b6b8e561b95d2cf01ba5041b2ba23c790a6cd563a7dd61efc31e518a267e7",
"pubkey": "7898a773eabb308d08b5ad0526ee84fb764c6a0020dfca9a732110b4e49bf575",
"created_at": 1718899929,
"kind": 1,
"tags": [
[
"p",
"f8347ec37d9f23959ba62dd8391fbcade87a7c071205c96f890432e5c047a6ed"
],
[
"proxy",
"https://mastodon.fixermark.com/@mark/112649825778737348",
"web"
],
[
"e",
"7a4a209c22696ff6ab07a96474c26b86987ffd863c5d46aaf03ad6e300d2016a",
"",
"root"
],
[
"proxy",
"https://mastodon.fixermark.com/users/mark/statuses/112649825778737348",
"activitypub"
],
[
"L",
"pink.momostr"
],
[
"l",
"pink.momostr.activitypub:https://mastodon.fixermark.com/users/mark/statuses/112649825778737348",
"pink.momostr"
]
],
"content": "Ah, you've just inspired me to bother to look at my robots.txt file.\n\n... looks like Mastodon node doesn't disallow the Google crawlers by default, so I assume it's not unexpected your content would show up in Google's search engine? Though one thing I've never stopped to wonder: if you *do* disallow Google's spider, but your posts are imaged to someone else's node because someone on their node follows you, can the spider pick up your posts from there? This post on your server also has [https://mastodon.fixermark.com/@lauren@mastodon.laurenweinstein.org/112649798768708524](A URL on my server), so I'm *pretty* sure the spider could see your post via my node even if you disallow it on your node.",
"sig": "bf10798272f2310be95c0bb4511ec68adf87f7b0f399f2969aacc85a35ede2707e7e6fdbc47712fc94ca7269b598c9a4c6e1de20825f0d80036541d185fdb4dd"
}