Doug Hoyte

Public Key

npub1yxprsscnjw2e6myxz73mmzvnqw5kvzd5ffjya9ecjypc5l0gvgksh8qud4

NIP-05 Address

doug@hoytech.com

Profile Code

nprofile1qqszrq3cgvfe89vadjrp0gaa3xfs82txpx6y5ezwjuufzqu20h5xytgppemhxue69uhkummn9ekx7mp0qythwumn8ghj7cmp9eex2mrp09skymr99ehhyee0tx2hss

Publishing to

nos.lol

ca.relayable.org

relay.damus.io

Author Public Key

npub1yxprsscnjw2e6myxz73mmzvnqw5kvzd5ffjya9ecjypc5l0gvgksh8qud4

Show more details

Last Notes

2024-09-24 18:45:11

nostr has no global source of truth, and that is a good thing
Out of interest, I follow the progress of a lot of other projects similar to nostr, and a couple links surfaced today:
BlueSky has a big "firehose" connection that streams all updates (new posts, reactions, etc) to subscribers. Unsurprisingly, this is difficult to process except on beefy servers with lots of bandwidth. So, one proposed solution is to strip out all that pesky cryptography (signatures, merkle tree data, etc): https://jazco.dev/2024/09/24/jetstream/
And over on Farcaster, keeping their hubs in sync is too difficult, so they want to make all posts globally sequenced, like a blockchain. The details are still being worked out, but I think it's safe to assume there will be a privileged global sequencer who decides on this ordering (and possibly which posts are included at all): https://github.com/farcasterxyz/protocol/discussions/193
In my opinion, both of these issues are symptoms of an underlying errant philosophy. These projects both want there to be a global source of truth: A single place you can go to guarantee you're seeing all the posts on a thread, from a particular user, etc. On BlueSky that is https://bluesky.app and on Farcaster that is https://warpcast.com .
Advocates of each of these projects of course would dispute this, pointing out that you could always self-host, or somehow avoid depending on their semi-official infrastructure, but the truth is that if you're not on bluesky.app or warpcast.com, you don't exist, and nobody cares that you don't exist.
nostr has eschewed the concept of global source of truth. You can't necessarily be sure you are seeing everything. Conversations may sometimes get fragmented, posts may disappear, and there may be the occasional bout of confusion and chaos. There is no official or semi-official nostr website, app, or relay, and this is a good thing. It means we are actually building a decentralised protocol, not just acting out decentralisation theatre, or pretending we'll get there eventually and that the ends justify the means.
Back when computers were primitive and professional data-centres didn't exist, it was impossible to build mega-apps like Twitter. Protocols had to be decentralised by default -- there was simply no other way. We can learn a lot by looking back to protocols of yesteryear, like Usenet and IRC, and still-popular protocols like email and HTTP. None of these assume global sources of truth, and they are stronger and better for it, as is nostr.

2024-09-13 12:29:17

- reply

https://github.com/nostr-protocol/nips/pull/1494

2024-09-13 05:54:31

I just tagged strfry 1.0.0. Here are some of the highlights:
* negentropy protocol 1: This is the result of a lot of R&D on different syncing protocols, trying to find the best fit for nostr. I'm pretty excited about the result. Negentropy sync has now been allocated NIP 77.
* Better error messages for users and operators.
* Docs have been updated and refreshed.
* Lots of optimisations: Better CPU/memory usage, smaller DBs.
Export/import has been sped up a lot: 10x faster or more. This should help reduce the pain of DB upgrades (which is required for this release). Instructions on upgrading are available here:
https://github.com/hoytech/strfry?tab=readme-ov-file#db-upgrade
Thanks to everyone who has helped develop/debug/test strfry over the past 2 years, and for all the kind words and encouragement. The nostr community rocks!
We've got a few things in the pipeline for strfry:
* strfry proxy: This will be a new feature for the router that enables intelligent reverse proxying for the nostr protocol. This will help scale up mega-sized relays by allowing the storage and processing workload to be split across multiple independent machines. Various partitioning schemes will be supported depending on performance and redundancy requirements. The front-end router instances will perform multiple concurrent nostr queries to the backend relays, and merge their results into a single stream for the original client.
* As well as scaling up, reverse proxying can also help scale down. By dynamically incorporating relay list settings (NIP-65), nostr queries can be satisfied by proxying requests to external relays on behalf of a client and merging the results together along with any matching cached local events. Negentropy will be used where possible to avoid wasting bandwidth on duplicate events.
* Archival mode: Currently strfry stores all events fully indexed in its main DB, along with their full JSON representations (optionally zstd dictionary compressed). For old events that are queried infrequently, space usage can be reduced considerably. As well as deindexing, we are planning on taking advantage of columnar storage, aggregation of reaction events, and other tricks. This will play nicely with strfry proxy, and events can gradually migrate to the archival relays.
* Last but not least, our website https://oddbean.com is going to get some love. Custom algorithms, search, bugfixes, better relay coverage, and more!

2024-09-01 18:09:29

Exporting and importing events into a new strfry instance (which you need to do when the DB version changes) takes too long. Here's a feature I just added that speeds this up a lot:
https://github.com/hoytech/strfry/blob/next/docs/fried.md
Going forward, there is a release-0.9 branch. I'm going to tag one more release on that branch soon (after back-porting a couple fixes). It will have strfry export --fried but not import. I'm planning on this being the last release of the 0.9 series.
I'm working on a 1.0 release in the next branch. I just did a big refactor of the DB format that I've wanted to do for some time. I also removed prefix matching on id/pubkey (this was removed from NIP-01) and fixed a bunch of bugs. This release will also have the latest negentropy protocol version and BTree code.

2023-12-07 05:05:27

Hi all! I have just pushed a major update to the negentropy project. It implements protocol version 1 (previous was version 0).
The protocol has been simplified: ID size is no longer configurable, there are no more continuation ranges, and the output can be constructed in the same input-scan pass.
There is a new fingerprint function, based on an incremental hash. This allows fingerprints to be pre-computed and stored in a tree structure. The C++ implementation includes a B+Tree implementation that allows fingerprints to be computed without collecting IDs from the entire DB.
I have written a comprehensive article that goes over the theory of RBSR and the negentropy implementation:
https://logperiodic.com/rbsr.html
Comments are appreciated!
Finally, I have integrated the new version of negentropy and the B+Tree with strfry. It's in a development branch and not quite ready for production, but my testing indicates this will be a massive improvement for full-DB syncs, especially on relays with very large DBs. In-sync or nearly in-sync relays should sync almost instantly with negligible resource usage. Relays will also use the B+Tree for filters that contain only until and/or since, meaning that date-range full-DB syncs are also efficient. Syncing arbitrary filters works as before (but I have begun work to make these more efficient as well).
Unfortunately, this is a breaking change for the negentropy protocol (this really should be the last one!) and will also require a new DB version.
I'm going to take this opportunity to make a few more breaking DB changes, and plan to release strfry 1.0 after a beta testing period.

2023-11-20 19:38:35

- reply

This is a bit challenging, but I will look into it. Another possibility is we could try to optimise the import flow. Right now it's single-threaded and there are a bunch of other things that could be tuned also.

2023-11-06 18:18:12

- reply

Hah, nice rant! Yeah sorry, my code is a bit unconventional. I'm happy to offer you a full refund. :)
FWIW there are 2 `Dockerfile`s in the repo: One for alpine and one for Ubuntu. If you get it working on another platform, I'd be happy to merge a pull request.

2023-11-06 17:14:57

- reply

Thanks for letting me know, glad you enjoyed it!

2023-09-29 13:19:39

- reply

FWIW I agree with you. Right now the front page is really boring. I have some ideas and I'm working on it!

2023-09-28 15:48:55

- reply

It's based on kind 7 "reactions", roughly. It gets adjusted by the community's algorithm, and not everyone's reactions count (otherwise it would be trivially gameable by making a bunch of random pubkeys). I'll have more info on this soon, once community algorithms are ready.

2023-09-28 13:10:00

- reply

Yes, it's actually embedded into strfry executable as an "app". WIP code is in the web branch of the strfry repo: https://github.com/hoytech/strfry/tree/web/src/apps/web

2023-09-28 06:30:11

- reply

Yeah it's not great right now, sorry. Work in progress, stay tuned :)

2023-09-28 06:28:11

- reply

Thanks all! I wasn't quite ready to launch this thing, but I really appreciate the feedback, cheers!
I was having some stability issues which was why the site was down for a while earlier today. It should be fixed now.

2023-09-28 06:23:03

- reply

Hi! I wasn't really ready for a launch yet but I'm glad people are enjoying it :)
Actually it's already open-source, it's in a branch of the strfry repo: https://github.com/hoytech/strfry/tree/web/src/apps/web

2023-09-13 00:10:43

- reply

Yes, I have seen nostrdb. I think it's a great idea, and that it will be a big benefit for native clients. Let me know if I can help in any way!
I was thinking about it a bit and maybe flatbuffers is a bit overkill for the indexed nostr events in strfry, and I'm working on a branch that uses a custom event representation similar to nostrdb. I usually use flatbuffers by default because the generated code is convenient and it's easy to evolve the schema in a backwards compatible way. Also it comes with some nice utilities for debugging and securely parsing untrusted data (not needed here).
BUT the core nostr event layout is pretty much fixed and saving some space here is important not least because it will let us pack more records into the page cache. I'll keep you posted on how that goes. Cheers!

2023-09-12 20:28:50

- reply

Awesome! I made a bit of progress on connecting the test harness, but I found an area where the implementations differ. I'm going to keep looking into it, but I wrote up what I've found so far here:
https://github.com/yukibtc/rust-negentropy/issues/1

2023-09-12 07:13:09

- reply

600ms seems too long for that set size. I will take a look at the performance when I have my Rust harness working. Let's make sure it's working the same way as the C++/JS impls first, and then we can see if there any speed-ups.
One optimisation I did in the C++ was to ensure that the XORing is always done word-wise, instead of byte-for-byte. In fact, with attention to the alignment, I got the compiler to emit PXOR SIMD instructions to do multiple words in a single instruction.
That's very cool about the UniFFI bindings, I'm looking forward to it!

2023-09-12 06:59:13

- reply

I only have a very rough understanding of how minisketch works, but from what I know it is the opposite.
Minisketch is very good with small set sizes -- so good that it approaches the information-theoretic bounds of what is possible, bandwidth-wise. However, look at the first graph on the minisketch site: https://raw.githubusercontent.com/sipa/minisketch/master/doc/plot_capacity.png
Here you can see that the CPU time required is exponential in its "capacity" parameter (note the Y axis is log-scale), and will become impractical in the low thousands. The text hints at this too: "For the largest sizes currently of interest to the authors, such as a set of capacity 4096 with 1024 differences".
OTOH, negentropy works predictably well into the millions or tens of millions of events. By pre-computing fingerprints as described in Aljoscha Meyer's paper, this will scale to arbitrarily large sizes.
Some other downsides that I see with minisketch are that you need to choose an appropriate capacity before you begin the reconcilliation, otherwise it will fail (?). How do you choose that? Also, minisketch's performance depends on the field-size, which I believe is basically how large the IDs being reconciled can be. The largest example shown is 64 bits, which is uncomfortably small if trying to avoid collisions. negentropy by default uses 128 bits, but can scale this arbitrarily high with only linear overhead in bandwidth and CPU.
By contrast, negentropy requires almost no tuning. The default parameters I have selected seem to work well for both small and large set sizes, and with small and large symmetric differences between the two sides.
I am in the process of adding negentropy support to gensync, which is a general benchmarking framework for set reconcilliation protocols. I'll post any updates about my progress here: https://github.com/nislab/gensync/issues/9

2023-09-11 20:24:08

- reply

This is awesome, thank you!
I'm trying to create a harness for this so we can run it against the test-suite used by the C++ and JS implementations. For example, here's the C++ harness:
https://github.com/hoytech/negentropy/blob/master/test/cpp/harness.cpp
This is as far as I got:
https://gist.github.com/hoytech/96e65fc488fc39c2e782830f38727391
This is pretty much the first Rust code I've ever written, sorry about the bad quality. The problem I'm running into is that the reconcile() and reconcile_with_ids() methods seem to return String. If I understand rightly, Rust strings must be utf-8 encoded right?
In the current implementations, the reconcile messages must be in an 8-bit clean medium (or hex encoded). Same with the IDs (since they are typically hash function outputs). In C++ I use std::string because this allows any u8 values (and doesn't care anywhere about utf-8). In JS I allow Uint8Array values *or* hex-encoded strings.
Can we make the rust interface standardise on something 8-bit clean? Vec<u8> or some such? Or it's very possible I'm misinterpreting the code here. Any pointers are very much appreciated.

2023-09-08 22:04:43

I just landed a new feature in strfry: "strfry router"
Docs: https://github.com/hoytech/strfry/blob/master/docs/router.md
I kind of think of this like "nginx for nostr". It lets you manage many up/down/both streams to remote relays using a single process and config file, and you can edit that config file without interrupting existing streams (mostly).
You can also use nostr filters and/or plugins to filter the traffic in both up and down directions, independently.

2023-07-14 16:47:36

- reply

You're welcome, I'm glad you like it!

2023-07-14 16:47:15

- reply

Hehe thank you!

2023-07-13 12:10:04

Hey all! I'm going to be doing a talk on nostr and the architecture of the strfry relay. It's next week at the CppNorth conference in Toronto, Canada:
https://cppnorth.ca/speaker-doug-hoyte.html
If anyone's around, it'll be a pretty fun conference! You can enter NOSTR in the checkout to get a discount on the ticket.

2023-06-09 19:03:00

- reply

Hi! Yeah, sorry, the sync support has been broken for a little while. However, this is now fixed in the `next` branch and very soon there will be a 1.0.0 release with this included.

2023-06-09 19:00:06

- reply

Hi!

2023-05-08 16:19:54

- reply

Got it, makes sense. I think if you make sure the event is written before updating the DB then that would be safe, as you describe. It still might be worthwhile to check if there is any performance difference between the parallel file and storing the validated event JSON in sled DB along with the indices. If it works like LMDB then retrieving it will be functionally identical (an offset into an mmap), and an event write will require only one fsync, not two. It also might save you a lot of trouble later on, when it comes to backups and periodic rewrites you mention.
About the indices: No, I don't think it's a dumb algorithm. It's actually a really difficult problem, to figure out which index or indices to use in a scan. strfry has all the indices you mention, also composite with created_at. Additionally, there is a (pubkey,kind,created_at) index, which I think is useful because I believe queries like "get contact list of user X" are quite common.
strfry currently never joins two indices together or does a set intersection as you describe -- its algorithm is even dumber. It only ever uses 1 index. If the query can be fully satisfied by the index then it doesn't need to load the flatbuffers (most queries are like this). But if it does need to load the flatbuffers this is when it's important that accessing them and applying filters are fast. Optimisations for this code path are also beneficial for the live events filtering (ie post EOSE).
Your approach sounds like it might be quite effective. The only thing I'd watch out for is if your IO patterns are excessively random. The nice thing about working on one index at once is that the access is largely contiguous data. Even several contiguous streams is probably OK, but some queries have like hundreds of pubkeys in them (for instance).

2023-05-07 15:45:51

- reply

In strfry there's currently no way to do read-time filtering, but I'm going to put some thought in how to do this flexibly (see my sibling reply). Your ruleset description will be quite helpful as a solid use-case - thanks!
Also, that's very cool with your new relay design, I'm looking forward to seeing it.
About using a parallel mmap'ed file to store the event data, I have some experience with this approach and frankly I would not do that except as a last resort.
The biggest downside IMO is that you won't be guaranteeing transactional consistency between the event data and the indices. For instance, what if a write update to the sled DB succeeds, but writing the event to the mmap fails? This could happen when you're in low-diskspace conditions, or maybe there's an application bug or server failure and you crash at a bad time. In this case, you'd have a corrupted DB, because there would be pointers into invalid offsets in the mmap. Similarly, deleting an event and recovering the space will become much harder without transactional consistency.
OTOH, one really great thing you can do is directly `pwritev()` from the mmap to the socket. I have had success with this in the past, because the data doesn't even need to be copied to userspace and userspace page-tables don't need updating. In fact, with `sendfile()` and good enough network card drivers, it can actually be copied directly from the kernel's page cache to the network card's memory.
Although possible, I don't actually do this in strfry for a bunch of reasons. First of all, nostr events are usually too small to really make this worthwhile. Also, I don't want to deal with the complexity of long-running transactions in the case of a slow-reading client. Lastly, this usually isn't possible anyway because of websocket compression and strfry's feature where events can be stored zstd compressed on disk.
Anyway, on balance I think it's preferable to store the event data in the same DB as the indices. With LMDB you get pretty much all the benefits (and downsides) of a mmap() anyway. I don't know about sled, I haven't looked into that before.
About serialisation, I quickly looked at speedy and from what I can tell it seems like it's pretty simple and concatenates the struct's fields directly in-order (?). That may work pretty well in some cases, but let me elaborate a bit on my choice of flatbuffers.
flatbuffers have an offsets table (like a "vtable" if you're familiar with the C++ terminology). This lets you directly access a single field from the struct without having to touch the rest of the struct. For instance, if you wanted to extract a single string from a large object, you look up its offset in the vtable and directly access that memory. This has the advantage that none of the rest of the memory of the struct needs to be touched. If you're using a mmap like LMDB, then some of the record may not even need to be paged in.
Typically you will never fully "deserialise" the record - that doesn't even really make sense in flatbuffers because there is no deserialised format (or rather, the encoded format is the same as the "deserialised" one). This means that getting some information out of a flatbuffer does not require any heap memory allocations. Additionally, small fixed-size fields are typically stored together (even if there is some long string "in between" them), so loading one will often as a side-effect cause the others to be paged in/loaded as well, and live in the same CPU cache line.
I may be wrong, but speedy looks more like protobufs where you have to traverse and decode the entire struct, and allocate memory for each field before you can access a single field in the record. In the case where you wanted to read just a couple fields (pretty common for a DB record), this could be pretty wasteful. Again, the data required for indexing nostr events is fairly small so this may not be a big deal -- my choice of default is from experience with much larger records.
One more thing: flatbuffers is quite interesting (and I think unique?) in that it allows you to separate the error checking from the access/deserialisation. For example, when you deserialise you typically assume this input is untrusted, so you'll be doing bounds-checking, field validation, etc. However, in the case where the records are purely created and consumed internally, these checks are unnecessary. This is one reason flatbuffers is so effective for the use-case of database records.
Sorry for this big wall of text!

2023-05-07 14:47:37

- reply

Got it! I will try to think of the best way to do read-level authorisation, which I think is the building block we need for this. I'm thinking something like a "virtual filter" that is applied to all queries and has the ability to remove disallowed events from the output. I think this would be integrated at the DB querying level, so that (ie) you don't get fewer results than your limit, as might happen if it was done at a post-processing stage.

2023-05-05 17:41:38

- reply

Hey guys, sorry for the delay!
Just to clarify: You'd like it so a particular user (authenticated in some way, maybe NIP-42?) can only access a subset of the keys in the DB?
I think that would be possible (assuming authentication is implemented of course). As Mike sas, it wouldn't be quite as easy as with SQL, where you could add a WHERE clause to all the required queries. However, I think you could do it in a way that wouldn't affect performance drastically.
If the sets of events for each user are mostly disjoint (not too many duplicated), then you could possibly run a strfry instance for each user. A running instance has a relatively low overhead, especially if you scale down the various thread counts.

2023-02-17 22:53:17

- reply

Actually strfry *does* compress relay->client messages by default, using websocket per-message compression with sliding window (if supported by client).
This is configurable with the relay.compression.enabled and relay.compression.slidingWindow parameters in the beta branch.
The strfry logs also include the compression negotation parameters on client connect, and the total compression ratios on disconnect (up and down).

2023-01-12 15:21:13

- reply

There is a "websocket" thread that is async since it handles multiplexing all the different websocket connections. However, it delegates most CPU or I/O bound operations to various thread pools. There are more details in the Architecture section of the README: https://github.com/hoytech/strfry#architecture

2023-01-10 18:22:03

- reply

Thanks! I'm looking forward to actually testing strfry on production traffic. :)
I think the merkle-tree syncing could really help with distributing data amongst relays too, I'm going to write more on this shortly.

2023-01-09 21:00:05

Hey all! Here's a new nostr relay implementation I'm working on: https://github.com/hoytech/strfry
It's still in beta/dev, but pretty close to production ready. Most interesting feature is a merkle-tree based set reconciliation protocol for syncing messages between servers.