Went from taking 8 hours to import 50 million rows to 45 minutes. Wild. And instead of constantly hitting my disk at 300MB/s the entire time its now ~16MB/s with spikes here and there. Lesson learned: INSERT as many rows as you can rather than a single row insert at a time. Very happy more experienced folks told me I was doing it the slow way, they weren't kidding!
If I parallelize this I bet I can knock it down to 10 minutes or so.
zyd /
npub184…7w6ek
2024-06-06 18:59:01
Author Public Key
npub184ejnrlc3p2whr4xpkxk72slwu9pfvtfg0zzv3wvk6963gtz2wnqw7w6ekPublished at
2024-06-06 18:59:01Event JSON
{
"id": "fc6236deeac351dffc9d09e34963959c54b91355d70ad0c8eb7e11b2df044c27",
"pubkey": "3d73298ff88854eb8ea60d8d6f2a1f770a14b16943c42645ccb68ba8a16253a6",
"created_at": 1717700341,
"kind": 1,
"tags": [
[
"proxy",
"https://emacs.ch/users/zyd/statuses/112571209576338522",
"activitypub"
]
],
"content": "Went from taking 8 hours to import 50 million rows to 45 minutes. Wild. And instead of constantly hitting my disk at 300MB/s the entire time its now ~16MB/s with spikes here and there. Lesson learned: INSERT as many rows as you can rather than a single row insert at a time. Very happy more experienced folks told me I was doing it the slow way, they weren't kidding!\n\nIf I parallelize this I bet I can knock it down to 10 minutes or so.",
"sig": "ccd9bf9b8eed1854e48d18136bc06e86c7c9b3eff13d35330e94e8a1f8181b7fe74c3d1e8285bc938aab42ef7b883b95e672c6e1228ab89fbac26bcefdeeaa4d"
}