Things should be getting back to normal now.
Something changed somewhere (maybe in the last update and I just never noticed, or maybe the default container ENV changed, not sure), and the result was that all temporary files were trying to be written to an incorrect directory (/opt/mastodon instead of /opt/mastodon/tmp).
The container filesystem is read-only which made those writes fail. They then entered the retry queue which backed up ~1.2m items.
I've triaged and fixed the root cause.
Matt Hamilton [Maryland] /
npub1sa…p27a4
2023-10-05 07:29:57
Author Public Key
npub1sat3yl2hv2df5xmuqj75gvzfmvyxs5x36vaur9jwvjzejtx7y2hskp27a4Published at
2023-10-05 07:29:57Event JSON
{
"id": "4e031250f6af98d59a77899840ad2eb356fa350a395e56b2ab927149645245ca",
"pubkey": "8757127d57629a9a1b7c04bd443049db086850d1d33bc1964e6485992cde22af",
"created_at": 1696490997,
"kind": 1,
"tags": [
[
"proxy",
"https://noagendasocial.com/users/eriner/statuses/111181234025718353",
"activitypub"
]
],
"content": "Things should be getting back to normal now.\n\nSomething changed somewhere (maybe in the last update and I just never noticed, or maybe the default container ENV changed, not sure), and the result was that all temporary files were trying to be written to an incorrect directory (/opt/mastodon instead of /opt/mastodon/tmp).\n\nThe container filesystem is read-only which made those writes fail. They then entered the retry queue which backed up ~1.2m items.\n\nI've triaged and fixed the root cause.",
"sig": "dfbaf07f6b58e024af0a0b258812889faef405202a1311ed01632de0f68d96101fa23d38c253a42e70aeb50bd7ec6882d4450cf01824ac34544559f828ccb158"
}