jb55 on Nostr: Traditionally when using databases you have to serialize data in and out. With ...
Traditionally when using databases you have to serialize data in and out. With nostrdb your in-memory data structures are the same as the disk representation.
It uses virtual memory to map the entire database (up to 32gb). When accessing data the operating system only reads parts of this file into memory when accessing data.
The end result is you get the speed of an in-memory data structure but without having to allocate 32gb of memory to read it all (bad and impossible)
Since nostrdb uses lmdb, it arranges the data into an efficient search tree so that the most common accessed data is always hot in the cache. The end result is that the code is much simpler and faster.
The next step is to replace the in memory note cache with nostrdb as well.
This will put damus apps in a league of their own, thanks for your patience as I hook all this up 🙏
Profiles in nostrdb (#1520)
This adds profiles to nostrdb and replaces our in-memory and ProfileDatabase cache with it
- Remove in-memory Profiles caches, nostrdb is as fast as an in-memory cache
- Remove ProfileDatabase and just use nostrdb directly
## TODO
- [ ] cache lnurl properly
- [ ] add database versions for migrations
nostrdb-profiles, 5 commits
https://github.com/damus-io/damus/pull/1520
Published at
2023-08-28 15:33:49Event JSON
{
"id": "67c53732048df9d0419ee8ee453d9866dc99c954f659e7016ce0e016f6c6f3f8",
"pubkey": "32e1827635450ebb3c5a7d12c1f8e7b2b514439ac10a67eef3d9fd9c5c68e245",
"created_at": 1693236829,
"kind": 1,
"tags": [
[
"q",
"94e1786481c3ba505b3417a8ef34811c63d5d7f8258e9bcc94d53d8e106489b3"
],
[
"p",
"edf16b1dd61eab353a83af470cc13557029bff6827b4cb9b7fc9bdb632a2b8e6"
]
],
"content": "Traditionally when using databases you have to serialize data in and out. With nostrdb your in-memory data structures are the same as the disk representation.\n\nIt uses virtual memory to map the entire database (up to 32gb). When accessing data the operating system only reads parts of this file into memory when accessing data.\n\nThe end result is you get the speed of an in-memory data structure but without having to allocate 32gb of memory to read it all (bad and impossible)\n\nSince nostrdb uses lmdb, it arranges the data into an efficient search tree so that the most common accessed data is always hot in the cache. The end result is that the code is much simpler and faster.\n\nThe next step is to replace the in memory note cache with nostrdb as well.\n\nThis will put damus apps in a league of their own, thanks for your patience as I hook all this up 🙏 nostr:note1jnshseypcwa9qke5z75w7dypr33at4lcyk8fhny5657cuyry3xesqhufqv",
"sig": "efbc5b1f85b3fe9a868e4eba6e1ee585999674bef91f4ae09bfd12ba78f9507471d92367e08f090047ac2734c275ced8230a02a5f26fb66042c5ab62da9b7209"
}
