Reiterating this from Telegram so people can see it.
--
The site went down last night because I added a new database to the cluster which I was hoping to set up quietly without any downtime. Unfortunately, it caused other databases to crash, which I'm having trouble recovering, because when multiple nodes go down restarting everything is a massive pain in the ass.
The new node is going to replace this more complicated setup. Over the last month, I've found more sure footing for our physical hardware I have not been able to put into use for over 6 months now. This transfer is to put our hardware back in use so I can stop juggling servers on multiple continents.
We will still retain redundancy so that if the Kiwi Farms is ever disconnected from the Internet again, there will not be extended downtime. However, the overall topology will be simplified and (hopefully) random crashes like this will stop happening so frequently.
TL;DR: We're basically right back to where we started in August and the last thing I have to do is get Ol' Faithful running again, which unfortunately means some downtime.
Josh /
npub1ul…dhtel
2023-04-15 09:55:01
Author Public Key
npub1ulv30q7h4j5tj4zl0rverqmqu89x8q0s5s7ef4vkupvkss5p73as2dhtelPublished at
2023-04-15 09:55:01Event JSON
{
"id": "767196d53cf635c71a09cd709fa455f49ccca4d1cbae260b46b6073a521e4132",
"pubkey": "e7d91783d7aca8b9545f78d9918360e1ca6381f0a43d94d596e059684281f47b",
"created_at": 1681552501,
"kind": 1,
"tags": [
[
"mostr",
"https://kiwifarms.cc/objects/8f384431-bd73-45eb-aad5-a8c0fd5dfbdc"
]
],
"content": "Reiterating this from Telegram so people can see it.\n\n--\n\nThe site went down last night because I added a new database to the cluster which I was hoping to set up quietly without any downtime. Unfortunately, it caused other databases to crash, which I'm having trouble recovering, because when multiple nodes go down restarting everything is a massive pain in the ass.\n\nThe new node is going to replace this more complicated setup. Over the last month, I've found more sure footing for our physical hardware I have not been able to put into use for over 6 months now. This transfer is to put our hardware back in use so I can stop juggling servers on multiple continents.\n\nWe will still retain redundancy so that if the Kiwi Farms is ever disconnected from the Internet again, there will not be extended downtime. However, the overall topology will be simplified and (hopefully) random crashes like this will stop happening so frequently.\n\nTL;DR: We're basically right back to where we started in August and the last thing I have to do is get Ol' Faithful running again, which unfortunately means some downtime.",
"sig": "fa5943930f2db611eb5d968d81189b4f8328f8b8c281a5f70f6a2fbaa451c8ffa47297b7dd31ea269883b58995da4807405c27dca604c435757d0798ca30cc6a"
}