Doug Hoyte

Public Key

npub1yxprsscnjw2e6myxz73mmzvnqw5kvzd5ffjya9ecjypc5l0gvgksh8qud4

NIP-05 Address

doug@hoytech.com

Profile Code

nprofile1qqszrq3cgvfe89vadjrp0gaa3xfs82txpx6y5ezwjuufzqu20h5xytgppemhxue69uhkummn9ekx7mp0qy2hwumn8ghj7un9d3shjtnyv9kh2uewd9hj7gcjh9t

Publishing to

nos.lol

relay.damus.io

Author Public Key

npub1yxprsscnjw2e6myxz73mmzvnqw5kvzd5ffjya9ecjypc5l0gvgksh8qud4

Show more details

Last Notes

2025-01-03 18:37:15

- reply

Thank you! LMK if you have any feature requests!
I'm going to be rolling out support for custom feeds soon and a few other neat things, like support for long-form notes.

2024-12-29 05:45:05

- reply

I'm in, please bridge me. You are doing great work Alex, thank you!

2024-12-24 17:49:57

- reply

Here's the video of it in action, it's awesome!
https://www.youtube.com/watch?v=rN7HYXmgxzk

2024-12-24 15:32:23

- reply

It depends on what you want to print. For me, I did FDM printing (the regular "melt plastic filament using a motorised hot end") for years and was never totally happy with the results or the process. Calibrating and maintaining the machines is a ton of work (some more than others of course). There's always a compromise between speed and quality, and even with the slowest/best settings you're always going to have visible layer lines and other small defects.
Last year I bought a resin printer (Elegoo Saturn 3 Ultra) and this has completely changed the hobby for me:
* Almost no maintenance. Once you've got it setup it pretty much just works. There are some exceptions naturally, but in general they need a lot less babysitting than FDM machines.
* Quality is incredible. The detail can be absolutely stunning -- way beyond what even the best FDMs are capable of. I've printed stuff for people and they literally did not believe me that I printed them ('you just ordered this on Amazon bro, this is clearly injection molded').
* Much better options for materials. There's somewhat more variety of resins than there are filaments, plus you can mix them: Want a red tinted plastic with a bit of flex to it that glows in the dark? No problem, just needs some experimentation with the proportions.
* Speed. If you're just printing one thing then FDM is probably faster (though the Saturn 3 Ultra can actually go quite fast). BUT resin printers really shine when you're printing lots of things at the same time. Once you have your model perfected, suppose you want to print 10 copies. With an FDM printer that's roughly 10x the time. For resin, assuming you can fit all of them on the bed, it takes the same time as printing 1, since each layer has a constant exposure time.
There are some downsides to resin though:
* You need to buy and store extra devices, like washing and curing stations.
* My garage is now filled with dangerous chemicals and I had to install a ventilation system to expel toxic fumes (how toxic they are is actually debatable, but I'm playing it safe).
* I have to periodically drive to a sketchy Chinese warehouse to buy 99% pure isopropyl alcohol in bulk. And getting rid of the waste is complicated/annoying. Look into your options for chemical disposal. Don't even bother trying the water washable resins, you'll just go alcohol eventually.
Anyway, I had fun with FDM for years but at this point I'm much much happier with resin printing. Everyone's situation and goals are going to be different, I just wanted to give you my perspective. You'll have a blast with whatever you choose. Good luck!

2024-12-23 18:18:47

- reply

This is one thing I really wish you could do in C++. noexcept is... not this.

2024-12-23 18:16:24

Should I block AI web crawlers on Oddbean?
On oddbean.com I see a *lot* of web crawling traffic from AI bots like GPTBot hoovering up nostr notes presumably for training purposes. I guess it's probably one of the easiest nostr sites to crawl since everything is rendered as plain HTML and they don't need to execute JS code to query relays.
To avoid wasting bandwidth I decided to use the following method to soft-block them (honour-system robots.txt): https://coryd.dev/posts/2024/go-ahead-and-block-ai-web-crawlers
You could argue they're just wasting my resources and won't bring any visitors or benefit the nostr community in any way. On the other hand, I guess they can/will access this data in some other way, and maybe the world-at-large gets some modicum of benefit from better AI models (?).
Thoughts? #asknostr

2024-12-22 17:50:18

- reply

> sig-chains - you know if you're missing content in PZP
You could do the same on nostr if anybody actually cared about this. Just include a "prev" tag with the ID of the previous message in the chain you're building. You could also have a "last" tag or something to indicate the chain is complete. Clients could render it as "message 3/9" or whatever, like people do manually on Twitter.
In fact, PZP's replication appears to *rely* on these hash chains, since it uses hash graph replication: https://codeberg.org/pzp/pzp-sync
I did some analysis of this sync method and if you tried to use it for non-linked data you can easily cause it to degrade into a worst-case behaviour: https://github.com/hoytech/automerge-poison
IMO RBSR/negentropy is better, one of the reasons being you can sync arbitrary collections of (unlinked) data, for example all kind 0 notes, or all notes with a certain hash tag.

2024-12-20 19:44:48

- reply

Yep looks like the same approach to me.

2024-12-20 19:18:47

- reply

> It might help you find a better alternative by searching for his recommendation for C/C++ projects needing to do this
Thanks, I will look into this. Go has much better Unicode support than C++, which basically doesn't have it at all, or rather, you need to pull in a library to do even basic things. This is why I'm doing the hack I mentioned above: I don't want to add a dependency on a library like ICU (also it is very efficient).
OTOH, Perl has outstanding Unicode support. If you don't care about byte length, then you can simply pick out the first 100 grapheme clusters with a regexp like this: /^\X{0,100}/. This will handle the segmentation according the the TR-29 rules.
Although you're probably right for this use-case, in my experience byte-length limits are quite common and you have to deal with them somehow, ideally without causing weird artifacts. For example in nostr you have byte-size limits on note length, tag values, etc. Another tricky aspect is that theoretically grapheme clusters are unbounded in length. So a single "character" could take up gigabytes of encoded space -- worth keeping in mind due to the DoS risk.

2024-12-20 18:13:07

- reply

Is there an easy way to truncate to a max byte-length using the Go standard library? This is the best answer I could find (after only 1 minute of searching though):
https://stackoverflow.com/questions/46415894/golang-truncate-strings-with-special-characters-without-corrupting-data/76502408#76502408
Note that as the answer says, this doesn't understand grapheme clusters, meaning that café can become cafe (depending on normalisation form). Also it involves another pass over the data which, if already known to be UTF-8, is redundant.
This 3rd party package looks to be the rough equivalent of my Perl module: https://github.com/rivo/uniseg

2024-12-20 07:32:55

Truncating text is complicated.
Today I spent some time fixing some bugs on oddbean.com that I've been putting off for a while. Most just involved some uninteresting grunt work, but there's one that is a huge rabbit hole and, if you've never thought about it before you may be surprised at how deep it goes.
On Oddbean, we only show the first ~100 characters of a nostr note and then cut it off ("truncate" it). This is all well and good, except some titles got an unexpected weird character at the end:
Nostr Advent Calendar 2024 の 11 日目の記事を書きました。 2024年のNostrリレ�…
Now, I'm no expert on Japanese script but I'm pretty sure that diamond question mark character is not supposed to be there. What gives?
The answer is that almost all text on the web is encoded in UTF-8, which is a multi-byte Unicode encoding. That means that these Japanese characters actually take up 3 bytes, unlike Latin letters which take up 1. Oddbean was taking the first 100 bytes and cutting it off there. Unfortunately, that left an incomplete UTF-8 encoded code point which the browser replaces with a special replacement character (U+FFFD, the diamond question mark).
OK, easy fix right? Just do substr() on the code-points (not the UTF-8 encoding). Sure, but that is quite inefficient, requiring a pass over the data. Fortunately there is a more efficient way to fix this that relies on the fact that UTF-8 is a self-synchronising code, meaning you can always find nearest code point boundaries no matter where in the string you jump to. So that is what I did:
https://github.com/hoytech/strfry/blob/863edcff17834af5f51654b546e34de965382756/src/apps/web/WebUtils.h#L120-L127
Problem solved right? Well, that depends on your definition of "solved". Notice above I've been referring to "code points" instead of characters? In many languages such as English we can pretty much get away with considering these the same. However in other scripts this is not the case.
Sometimes what we think of as a character can actually require multiple code-points. For example, the character 'â' can be represented as 'a' followed by a special ' ̂' combining character. Most common characters such as â *also* have dedicated code-points, and which representation is used depends on the Unicode Normal Form. You may also have seen country flags represented by two composite characters, or emoji alterations such as skin tone -- it's the same principle. Cutting in between such characters will cause truncation artifacts.
So rather than "character" (which is an imprecise notion), Unicode refers to Extended Grapheme Clusters, which correspond as closely as possible with what we think of as individual atoms of text. You can read more than you ever wanted to know about this here: https://www.unicode.org/reports/tr29/#Grapheme_Cluster_Boundaries
Note that many langauges need special consideration when cutting on graphemes (or indeed words, lines etc). Especially Korean Hangul script is interesting, having been designed rather than evolved like most writing systems -- in fact it's quite elegant!
So my hack for Oddbean doesn't do all this fancy grapheme truncation, and that's because I know if I tried I would end up in a seriously deep rabbit hole. I know because I have and I did! 10 years ago I published the following Perl module: https://metacpan.org/pod/Unicode::Truncate
I'm pretty proud of this yak shave, because of the implementation. I was able to adapt the regular expressions from Unicode TR29, compose them with a UTF-8 regular expression, and compile it all with the Ragel state machine compiler ( https://www.colm.net/open-source/ragel/ ). As a result, it can both validate UTF-8 and (correctly!) truncate in a single-pass.
If you want (a lot) more Unicode trivia, I also made a presentation on this topic: https://hoytech.github.io/truncate-presentation/

2024-12-16 04:28:57

- reply

Yeah this was a pretty interesting article. Reminds me how deep the hash-table rabbit hole can go!

2024-12-11 03:40:46

- reply

If I had more time to play around with radios I'd snap this up.
25W output, built-in FT-8, open source, reasonable price.

2024-09-25 18:33:33

- reply

Interesting perspective. We could debate what decentralised means, but I doubt we'd ever be able to find a universal definition -- it is too much of a spectrum. If your definition of decentralised is that there are no servers at all, then I guess you'd think only purely P2P protocols are, and not even nostr would qualify. From a total purity perspective, probably anything using DNS would be disqualified too.
My view on these protocols is as follows:
Usenet has essentially the same model as nostr. Yes, there are servers (relays), but people are free to choose which ones they use. They can post their messages to any of them, and those messages may get propagated to other servers. Each server can have its own message acceptance/forwarding policies, and choose which other servers to connect with.
IRC is also a decentralised network (the R stands for relay). An IRC network consists of many different servers relaying messages. Each server agrees roughly with the rules of the wider network, but is generally free to administer its server as it sees fit (including user bans, preventing relaying certain channels, etc). Sometimes server operators disagree, and this results in them leaving the network and establishing their own. That's why there are many different IRC networks, EFnet, IRCnet, DALnet, etc.
HTTP and email are both decentralised in the sense that you don't need to get anybody's permission to connect to the network, and there are no single points of failure.

2024-09-25 18:15:49

- reply

> is there a historical business model, or models, that we could mimic?
Good question! Usenet and IRC were often hosted by ISPs, which users paid for indirectly with their internet subscriptions. People also paid for access to Usenet. DejaNews was a popular archive/provider service before it was bought by Google. On IRC people pay for service providers to keep them constantly connected (bouncers) and provide vanity DNS names. Also, people pay to run bots like Eggdrop to manage/moderate their channels.
For email and HTTP there are the obvious hosting and other service providers, but I suppose the biggest value they support is the things built on top of them, which is maybe fitting for a general-purpose protocol.

2024-09-24 18:45:11

nostr has no global source of truth, and that is a good thing
Out of interest, I follow the progress of a lot of other projects similar to nostr, and a couple links surfaced today:
BlueSky has a big "firehose" connection that streams all updates (new posts, reactions, etc) to subscribers. Unsurprisingly, this is difficult to process except on beefy servers with lots of bandwidth. So, one proposed solution is to strip out all that pesky cryptography (signatures, merkle tree data, etc): https://jazco.dev/2024/09/24/jetstream/
And over on Farcaster, keeping their hubs in sync is too difficult, so they want to make all posts globally sequenced, like a blockchain. The details are still being worked out, but I think it's safe to assume there will be a privileged global sequencer who decides on this ordering (and possibly which posts are included at all): https://github.com/farcasterxyz/protocol/discussions/193
In my opinion, both of these issues are symptoms of an underlying errant philosophy. These projects both want there to be a global source of truth: A single place you can go to guarantee you're seeing all the posts on a thread, from a particular user, etc. On BlueSky that is https://bluesky.app and on Farcaster that is https://warpcast.com .
Advocates of each of these projects of course would dispute this, pointing out that you could always self-host, or somehow avoid depending on their semi-official infrastructure, but the truth is that if you're not on bluesky.app or warpcast.com, you don't exist, and nobody cares that you don't exist.
nostr has eschewed the concept of global source of truth. You can't necessarily be sure you are seeing everything. Conversations may sometimes get fragmented, posts may disappear, and there may be the occasional bout of confusion and chaos. There is no official or semi-official nostr website, app, or relay, and this is a good thing. It means we are actually building a decentralised protocol, not just acting out decentralisation theatre, or pretending we'll get there eventually and that the ends justify the means.
Back when computers were primitive and professional data-centres didn't exist, it was impossible to build mega-apps like Twitter. Protocols had to be decentralised by default -- there was simply no other way. We can learn a lot by looking back to protocols of yesteryear, like Usenet and IRC, and still-popular protocols like email and HTTP. None of these assume global sources of truth, and they are stronger and better for it, as is nostr.

2024-09-14 13:33:45

- reply

LMK if there's any bug on the strfry side and I will look into it!

2024-09-14 13:31:59

- reply

I haven't heard about any, please drop by the telegram channel and let us know if you notice any!

2024-09-14 00:22:34

- reply

The answer to this is surprisingly complicated.
TLS can optionally support compression which would most likely have universally worked for all wss:// connections. However, this was disabled in OpenSSL and other TLS libraries because of a critical information leakage that arises when secret and non-secret information are combined in the same compression context: https://blog.qualys.com/product-tech/2012/09/14/crime-information-leakage-attack-against-ssltls
HTTP-level compression does not apply to websockets (since its framing replaces/upgrades the HTTP framing) so instead compression is specified by the websocket RFCs. It is optional, so not all clients support this.
Websocket compression happens per message, and can use an empty window for each message, or can have a "sliding compression" window where messages are effectively compressed with previous messages. Some implementations will support both of those modes, some only one, and some neither. Even if an implementation supports compression, it may choose not to use it, and/or may use it only for particular messages (and not others). Furthermore, in the websocket compression handshake, bi-directional window sizes need to be negotiated and sometimes windows cannot be negotiated in one or both directions.
Almost all browser websocket clients support full compression with sliding windows in both directions, and so does strfry. The sliding window has a relatively large memory overhead per connection, so it can optionally be disabled. The compression ratios can be seen in the strfry logs.
Although strfry <> browser connections are almost always compressed both ways, different clients and relays have different levels of support and often can't negotiate optimal compression.

2024-09-13 22:30:29

- reply

Hehe in fact my original attempt at sync in strfry was a protocol called "yesstr" using my Quadrable library: https://github.com/hoytech/quadrable
It used binary flatbuffers over websockets. It caused a lot of problems since binary websocket messages require a distinct message type from text ones. Since nothing else in nostr uses these, of client libraries were having trouble integrating with it. Using a structured binary format like CBOR or flatbuffers also means clients would have to pull in a (probably heavy) serialisation library.
The nice thing about the current approach is that any existing nostr client already must support websocket text messages containing JSON containing hex data.

2024-09-13 21:23:19

- reply

> If you’re sending binary, base64 only inflates by 33% compared to hex strings’ 100%.
On purely random data, after compression hex is usually only ~10% bigger than base64. For example:
$ head -c 1000000 /dev/urandom > rand
$ alias hex='od -A n -t x1 | sed "s/ *//g"'
$ cat rand |hex|zstd -c|wc -c
1086970
$ cat rand |base64|zstd -c|wc -c
1018226
$ wcalc 1086970/1018226
= 1.06751
So only ~7% bigger in this case. When data is not purely random, hex often compresses *better* than base64. This is because hex preserves patterns on byte boundaries but base64 does not. For example, look at these two strings post-base64:
$ echo 'hello world' | base64
aGVsbG8gd29ybGQK
$ echo ' hello world' | base64
IGhlbGxvIHdvcmxkCg==
They have nothing in common. Compare to the hex encoded versions:
$ echo 'hello world' | hex
68656c6c6f20776f726c640a
$ echo ' hello world' | hex
2068656c6c6f20776f726c640a
The pattern is preserved, it is just shifted by 2 characters. This means that if "hello world" appears multiple times in the input, there may be two different patterns for it in Base64, but only one in hex (meaning hex effectively has a 2x larger compression dictionary).
Since negentropy is mostly (but not entirely) random data like hashes and fingerprints, it's probably a wash. However, hex is typically faster to encode/decode and furthermore is used for almost all other fields in the nostr protocol, so on the whole seems like the best choice.
> Personally, I’d prefer to see the message format specified explicitly as debuggable JSON, if feasible.
This is theoretically possible, but it would be very difficult to interpret/debug it anyway, and it would add a lot of bandwidth/CPU overhead.

2024-09-13 20:47:52

- reply

Probably our telegram channel is best currently: https://t.me/strfry_users
Hopefully some day we'll migrate fully to nostr!

2024-09-13 20:05:18

- reply

Good question! Right now the implementation is pretty simple, but I think will work well in most cases: It always splits a range into 16 equal-sized ranges unless the range has 32 or fewer IDs, in which case it just sends the whole list of IDs. I think there is probably some low-hanging fruit on tuning that threshold.
The nice thing is that nothing in the protocol needs to change to tune this. In fact, the protocol theoretically supports dynamic adjustment of those parameters to target a particular point in the latency/bandwidth tradeoff space.
There are also some other reasons you might want to customise the range selection, which I described here: https://logperiodic.com/rbsr.html#range-choice

2024-09-13 19:49:16

- reply

No, both of those versions use DB version 2, so their DBs are compatible.

2024-09-13 19:36:02

- reply

The best way is to first upgrade to 0.9.7, which is the latest release in the 0.9 series. This has the "--fried" option for export (but not import).
After you have the fried export, upgrade to 1.0.0 and import with "--fried" too.
Sorry for the trouble, but it will save a *lot* of time versus normal import!

2024-09-13 19:33:28

- reply

NIP-77 I believe is still a draft/work-in-progress, so I don't know the final form yet.
I think fiatjaf's intent was just to standardise the nostr integration, rather than negentropy as a whole. I guess this is similar to how JSON/SHA256/secp256k/etc are not themselves specified in NIPs, but instead external specifications are referred to. That said, the negentropy spec is relatively straightforward and there are already at least 5 implementations (and I know of some more in development too): https://github.com/hoytech/negentropy/tree/master?tab=readme-ov-file#definitions
> It also doesn’t seem to cover the actual sending of events in either direction (after the sets have been reconciled).
True, but by design this is out-of-scope of the negentropy protocol. Negentropy just performs a set reconciliation so the client knows which IDs it has and/or needs. It is then up to it to decide what to do. Currently `strfry sync` will post events it has with regular EVENT messages, and/or download events it needs with regular REQ messages.
However, there are a lot of other possibilities. If a client is only interested in uploading (or downloading), it may only do one of those actions. In fact it may just store those IDs and fetch them later, or through some other relay or mechanism. Alternatively, if a client doesn't actually care about the event contents and is only trying to compute an aggregate "reaction count" across relays, it may never actually download the events, and purely use negentropy for de-duplication purposes.

2024-09-13 19:24:01

- reply

Good question! Appending at the end (or near the end) is actually the best-case scenario in my B-tree implementation, because it leaves the right-most leaf fully packed, unlike a classic B-tree which keeps it at half-capacity. More info here: https://github.com/hoytech/negentropy/blob/master/cpp/negentropy/storage/btree/core.h#L15-L31
Making multiple updates to the B-tree in the same transaction is also highly optimised: The modified nodes are edited in-place in memory, and only committed to the DB at the end of the transaction. So on average, inserting 40 or fewer records at the right edge of the tree in one transaction will require only 1 DB read and 1 DB write (in addition to the metadata page).

2024-09-13 19:17:46

- reply

I appreciate you letting me know -- I'm glad you like it!

2024-09-13 19:17:08

- reply

Oh no, I missed this one, thanks for linking! This is a really good write-up and there are a lot of similarities with what I'm designing for strfry proxy. Like some of the comments, I'm not sure if partitioning on ID will be optimal in all cases. I can also imagine variations on pubkey and created_at.
Some variant of consistent hashing will also be necessary I think, for failure recovery, rebalancing, changing the number of backend relays, etc.

2024-09-13 12:29:17

- reply

https://github.com/nostr-protocol/nips/pull/1494

2024-09-13 05:54:31

I just tagged strfry 1.0.0. Here are some of the highlights:
* negentropy protocol 1: This is the result of a lot of R&D on different syncing protocols, trying to find the best fit for nostr. I'm pretty excited about the result. Negentropy sync has now been allocated NIP 77.
* Better error messages for users and operators.
* Docs have been updated and refreshed.
* Lots of optimisations: Better CPU/memory usage, smaller DBs.
Export/import has been sped up a lot: 10x faster or more. This should help reduce the pain of DB upgrades (which is required for this release). Instructions on upgrading are available here:
https://github.com/hoytech/strfry?tab=readme-ov-file#db-upgrade
Thanks to everyone who has helped develop/debug/test strfry over the past 2 years, and for all the kind words and encouragement. The nostr community rocks!
We've got a few things in the pipeline for strfry:
* strfry proxy: This will be a new feature for the router that enables intelligent reverse proxying for the nostr protocol. This will help scale up mega-sized relays by allowing the storage and processing workload to be split across multiple independent machines. Various partitioning schemes will be supported depending on performance and redundancy requirements. The front-end router instances will perform multiple concurrent nostr queries to the backend relays, and merge their results into a single stream for the original client.
* As well as scaling up, reverse proxying can also help scale down. By dynamically incorporating relay list settings (NIP-65), nostr queries can be satisfied by proxying requests to external relays on behalf of a client and merging the results together along with any matching cached local events. Negentropy will be used where possible to avoid wasting bandwidth on duplicate events.
* Archival mode: Currently strfry stores all events fully indexed in its main DB, along with their full JSON representations (optionally zstd dictionary compressed). For old events that are queried infrequently, space usage can be reduced considerably. As well as deindexing, we are planning on taking advantage of columnar storage, aggregation of reaction events, and other tricks. This will play nicely with strfry proxy, and events can gradually migrate to the archival relays.
* Last but not least, our website https://oddbean.com is going to get some love. Custom algorithms, search, bugfixes, better relay coverage, and more!

2024-09-12 19:22:01

- reply

It waits forever, so plugins should ensure they do any timing out on their side. I have a debugging script that can be used to detect a stalled/missing response. More info in this issue: https://github.com/hoytech/strfry/issues/112
Currently strfry always waits until it gets a response about the previous event before sending another request, so the order is not applicable. Eventually I might make the implementation asynchronous in which case yes, mixed order would be acceptable. This is why the event ID must be included in a plugin reply even though technically it's not needed today. I clarified this in the docs: https://github.com/hoytech/strfry/commit/d2d8dc7572e63d74caf1810673981add47c9f328

2024-09-09 21:17:21

- reply

Thanks, I'll give it a try!
I should mention that the existing tests and the differential fuzzing described in the strfry README pass on the 1.0 series, naturally.

2024-09-06 23:41:50

- reply

Glad to hear! Thanks for letting me know, I appreciate it!

2024-09-06 19:05:05

- reply

Heh yeah that is a pretty cool use-case actually.
But this feature was removed from NIP-01 and strfry tries to follow the spec as close as possible, so take it up with the nostr CEO. ;)

2024-09-06 03:47:57

- reply

Thanks!
This change shouldn't make much difference in import's behaviour. Because import pretty much constantly has a writer lock, relay *write* operations will be blocked/delayed (and may timeout), but *read* operations should continue as normal.
We could make import release its lock for a few milliseconds periodically so that write operations can go through. I'll look into doing that.

2024-09-05 22:17:26

I just tagged 2 strfry releases: 0.9.7 and 1.0.0-beta1
1.0.0-beta1 is a candidate release of strfry 1.0.0 -- help is needed testing!
The internal strfry DB version has been increased to 3, which means that you will need to rebuild your DBs to use this new version.
0.9.7 has some bugfixes and changes that accumulated in master, and has a "strfry export --fried" feature that can be used to create DB exports that can be rapidly imported by 1.0.0 series releases.
The full changelogs are available here: https://github.com/hoytech/strfry/blob/1f34794945cf380035822278c46070a9923129f3/CHANGES
Thank you to everyone who contributed and helps testing! If you need help or run into any issues, reply on nostr or stop by our telegram channel.

2024-09-01 18:32:26

- reply

Yes, I saw -- it means a lot, especially coming from jb55. Thanks!

2024-09-01 18:09:29

Exporting and importing events into a new strfry instance (which you need to do when the DB version changes) takes too long. Here's a feature I just added that speeds this up a lot:
https://github.com/hoytech/strfry/blob/next/docs/fried.md
Going forward, there is a release-0.9 branch. I'm going to tag one more release on that branch soon (after back-porting a couple fixes). It will have strfry export --fried but not import. I'm planning on this being the last release of the 0.9 series.
I'm working on a 1.0 release in the next branch. I just did a big refactor of the DB format that I've wanted to do for some time. I also removed prefix matching on id/pubkey (this was removed from NIP-01) and fixed a bunch of bugs. This release will also have the latest negentropy protocol version and BTree code.

2023-12-07 05:05:27

Hi all! I have just pushed a major update to the negentropy project. It implements protocol version 1 (previous was version 0).
The protocol has been simplified: ID size is no longer configurable, there are no more continuation ranges, and the output can be constructed in the same input-scan pass.
There is a new fingerprint function, based on an incremental hash. This allows fingerprints to be pre-computed and stored in a tree structure. The C++ implementation includes a B+Tree implementation that allows fingerprints to be computed without collecting IDs from the entire DB.
I have written a comprehensive article that goes over the theory of RBSR and the negentropy implementation:
https://logperiodic.com/rbsr.html
Comments are appreciated!
Finally, I have integrated the new version of negentropy and the B+Tree with strfry. It's in a development branch and not quite ready for production, but my testing indicates this will be a massive improvement for full-DB syncs, especially on relays with very large DBs. In-sync or nearly in-sync relays should sync almost instantly with negligible resource usage. Relays will also use the B+Tree for filters that contain only until and/or since, meaning that date-range full-DB syncs are also efficient. Syncing arbitrary filters works as before (but I have begun work to make these more efficient as well).
Unfortunately, this is a breaking change for the negentropy protocol (this really should be the last one!) and will also require a new DB version.
I'm going to take this opportunity to make a few more breaking DB changes, and plan to release strfry 1.0 after a beta testing period.

2023-11-20 19:38:35

- reply

This is a bit challenging, but I will look into it. Another possibility is we could try to optimise the import flow. Right now it's single-threaded and there are a bunch of other things that could be tuned also.

2023-11-06 18:18:12

- reply

Hah, nice rant! Yeah sorry, my code is a bit unconventional. I'm happy to offer you a full refund. :)
FWIW there are 2 `Dockerfile`s in the repo: One for alpine and one for Ubuntu. If you get it working on another platform, I'd be happy to merge a pull request.

2023-11-06 17:14:57

- reply

Thanks for letting me know, glad you enjoyed it!

2023-09-29 13:19:39

- reply

FWIW I agree with you. Right now the front page is really boring. I have some ideas and I'm working on it!

2023-09-28 15:48:55

- reply

It's based on kind 7 "reactions", roughly. It gets adjusted by the community's algorithm, and not everyone's reactions count (otherwise it would be trivially gameable by making a bunch of random pubkeys). I'll have more info on this soon, once community algorithms are ready.

2023-09-28 13:10:00

- reply

Yes, it's actually embedded into strfry executable as an "app". WIP code is in the web branch of the strfry repo: https://github.com/hoytech/strfry/tree/web/src/apps/web

2023-09-28 06:30:11

- reply

Yeah it's not great right now, sorry. Work in progress, stay tuned :)

2023-09-28 06:28:11

- reply

Thanks all! I wasn't quite ready to launch this thing, but I really appreciate the feedback, cheers!
I was having some stability issues which was why the site was down for a while earlier today. It should be fixed now.

2023-09-28 06:25:18

- reply

Thank you!!

2023-09-28 06:23:03

- reply

Hi! I wasn't really ready for a launch yet but I'm glad people are enjoying it :)
Actually it's already open-source, it's in a branch of the strfry repo: https://github.com/hoytech/strfry/tree/web/src/apps/web

2023-09-13 00:10:43

- reply

Yes, I have seen nostrdb. I think it's a great idea, and that it will be a big benefit for native clients. Let me know if I can help in any way!
I was thinking about it a bit and maybe flatbuffers is a bit overkill for the indexed nostr events in strfry, and I'm working on a branch that uses a custom event representation similar to nostrdb. I usually use flatbuffers by default because the generated code is convenient and it's easy to evolve the schema in a backwards compatible way. Also it comes with some nice utilities for debugging and securely parsing untrusted data (not needed here).
BUT the core nostr event layout is pretty much fixed and saving some space here is important not least because it will let us pack more records into the page cache. I'll keep you posted on how that goes. Cheers!

2023-09-12 20:28:50

- reply

Awesome! I made a bit of progress on connecting the test harness, but I found an area where the implementations differ. I'm going to keep looking into it, but I wrote up what I've found so far here:
https://github.com/yukibtc/rust-negentropy/issues/1

2023-09-12 07:13:09

- reply

600ms seems too long for that set size. I will take a look at the performance when I have my Rust harness working. Let's make sure it's working the same way as the C++/JS impls first, and then we can see if there any speed-ups.
One optimisation I did in the C++ was to ensure that the XORing is always done word-wise, instead of byte-for-byte. In fact, with attention to the alignment, I got the compiler to emit PXOR SIMD instructions to do multiple words in a single instruction.
That's very cool about the UniFFI bindings, I'm looking forward to it!

2023-09-12 06:59:13

- reply

I only have a very rough understanding of how minisketch works, but from what I know it is the opposite.
Minisketch is very good with small set sizes -- so good that it approaches the information-theoretic bounds of what is possible, bandwidth-wise. However, look at the first graph on the minisketch site: https://raw.githubusercontent.com/sipa/minisketch/master/doc/plot_capacity.png
Here you can see that the CPU time required is exponential in its "capacity" parameter (note the Y axis is log-scale), and will become impractical in the low thousands. The text hints at this too: "For the largest sizes currently of interest to the authors, such as a set of capacity 4096 with 1024 differences".
OTOH, negentropy works predictably well into the millions or tens of millions of events. By pre-computing fingerprints as described in Aljoscha Meyer's paper, this will scale to arbitrarily large sizes.
Some other downsides that I see with minisketch are that you need to choose an appropriate capacity before you begin the reconcilliation, otherwise it will fail (?). How do you choose that? Also, minisketch's performance depends on the field-size, which I believe is basically how large the IDs being reconciled can be. The largest example shown is 64 bits, which is uncomfortably small if trying to avoid collisions. negentropy by default uses 128 bits, but can scale this arbitrarily high with only linear overhead in bandwidth and CPU.
By contrast, negentropy requires almost no tuning. The default parameters I have selected seem to work well for both small and large set sizes, and with small and large symmetric differences between the two sides.
I am in the process of adding negentropy support to gensync, which is a general benchmarking framework for set reconcilliation protocols. I'll post any updates about my progress here: https://github.com/nislab/gensync/issues/9

2023-09-11 20:24:08

- reply

This is awesome, thank you!
I'm trying to create a harness for this so we can run it against the test-suite used by the C++ and JS implementations. For example, here's the C++ harness:
https://github.com/hoytech/negentropy/blob/master/test/cpp/harness.cpp
This is as far as I got:
https://gist.github.com/hoytech/96e65fc488fc39c2e782830f38727391
This is pretty much the first Rust code I've ever written, sorry about the bad quality. The problem I'm running into is that the reconcile() and reconcile_with_ids() methods seem to return String. If I understand rightly, Rust strings must be utf-8 encoded right?
In the current implementations, the reconcile messages must be in an 8-bit clean medium (or hex encoded). Same with the IDs (since they are typically hash function outputs). In C++ I use std::string because this allows any u8 values (and doesn't care anywhere about utf-8). In JS I allow Uint8Array values *or* hex-encoded strings.
Can we make the rust interface standardise on something 8-bit clean? Vec<u8> or some such? Or it's very possible I'm misinterpreting the code here. Any pointers are very much appreciated.

2023-09-08 22:04:43

I just landed a new feature in strfry: "strfry router"
Docs: https://github.com/hoytech/strfry/blob/master/docs/router.md
I kind of think of this like "nginx for nostr". It lets you manage many up/down/both streams to remote relays using a single process and config file, and you can edit that config file without interrupting existing streams (mostly).
You can also use nostr filters and/or plugins to filter the traffic in both up and down directions, independently.

2023-07-14 16:47:36

- reply

You're welcome, I'm glad you like it!

2023-07-14 16:47:15

- reply

Hehe thank you!

2023-07-13 12:10:04

Hey all! I'm going to be doing a talk on nostr and the architecture of the strfry relay. It's next week at the CppNorth conference in Toronto, Canada:
https://cppnorth.ca/speaker-doug-hoyte.html
If anyone's around, it'll be a pretty fun conference! You can enter NOSTR in the checkout to get a discount on the ticket.

2023-06-09 19:03:00

- reply

Hi! Yeah, sorry, the sync support has been broken for a little while. However, this is now fixed in the `next` branch and very soon there will be a 1.0.0 release with this included.

2023-06-09 19:00:06

- reply

Hi!

2023-05-08 16:19:54

- reply

Got it, makes sense. I think if you make sure the event is written before updating the DB then that would be safe, as you describe. It still might be worthwhile to check if there is any performance difference between the parallel file and storing the validated event JSON in sled DB along with the indices. If it works like LMDB then retrieving it will be functionally identical (an offset into an mmap), and an event write will require only one fsync, not two. It also might save you a lot of trouble later on, when it comes to backups and periodic rewrites you mention.
About the indices: No, I don't think it's a dumb algorithm. It's actually a really difficult problem, to figure out which index or indices to use in a scan. strfry has all the indices you mention, also composite with created_at. Additionally, there is a (pubkey,kind,created_at) index, which I think is useful because I believe queries like "get contact list of user X" are quite common.
strfry currently never joins two indices together or does a set intersection as you describe -- its algorithm is even dumber. It only ever uses 1 index. If the query can be fully satisfied by the index then it doesn't need to load the flatbuffers (most queries are like this). But if it does need to load the flatbuffers this is when it's important that accessing them and applying filters are fast. Optimisations for this code path are also beneficial for the live events filtering (ie post EOSE).
Your approach sounds like it might be quite effective. The only thing I'd watch out for is if your IO patterns are excessively random. The nice thing about working on one index at once is that the access is largely contiguous data. Even several contiguous streams is probably OK, but some queries have like hundreds of pubkeys in them (for instance).

2023-05-07 15:45:51

- reply

In strfry there's currently no way to do read-time filtering, but I'm going to put some thought in how to do this flexibly (see my sibling reply). Your ruleset description will be quite helpful as a solid use-case - thanks!
Also, that's very cool with your new relay design, I'm looking forward to seeing it.
About using a parallel mmap'ed file to store the event data, I have some experience with this approach and frankly I would not do that except as a last resort.
The biggest downside IMO is that you won't be guaranteeing transactional consistency between the event data and the indices. For instance, what if a write update to the sled DB succeeds, but writing the event to the mmap fails? This could happen when you're in low-diskspace conditions, or maybe there's an application bug or server failure and you crash at a bad time. In this case, you'd have a corrupted DB, because there would be pointers into invalid offsets in the mmap. Similarly, deleting an event and recovering the space will become much harder without transactional consistency.
OTOH, one really great thing you can do is directly `pwritev()` from the mmap to the socket. I have had success with this in the past, because the data doesn't even need to be copied to userspace and userspace page-tables don't need updating. In fact, with `sendfile()` and good enough network card drivers, it can actually be copied directly from the kernel's page cache to the network card's memory.
Although possible, I don't actually do this in strfry for a bunch of reasons. First of all, nostr events are usually too small to really make this worthwhile. Also, I don't want to deal with the complexity of long-running transactions in the case of a slow-reading client. Lastly, this usually isn't possible anyway because of websocket compression and strfry's feature where events can be stored zstd compressed on disk.
Anyway, on balance I think it's preferable to store the event data in the same DB as the indices. With LMDB you get pretty much all the benefits (and downsides) of a mmap() anyway. I don't know about sled, I haven't looked into that before.
About serialisation, I quickly looked at speedy and from what I can tell it seems like it's pretty simple and concatenates the struct's fields directly in-order (?). That may work pretty well in some cases, but let me elaborate a bit on my choice of flatbuffers.
flatbuffers have an offsets table (like a "vtable" if you're familiar with the C++ terminology). This lets you directly access a single field from the struct without having to touch the rest of the struct. For instance, if you wanted to extract a single string from a large object, you look up its offset in the vtable and directly access that memory. This has the advantage that none of the rest of the memory of the struct needs to be touched. If you're using a mmap like LMDB, then some of the record may not even need to be paged in.
Typically you will never fully "deserialise" the record - that doesn't even really make sense in flatbuffers because there is no deserialised format (or rather, the encoded format is the same as the "deserialised" one). This means that getting some information out of a flatbuffer does not require any heap memory allocations. Additionally, small fixed-size fields are typically stored together (even if there is some long string "in between" them), so loading one will often as a side-effect cause the others to be paged in/loaded as well, and live in the same CPU cache line.
I may be wrong, but speedy looks more like protobufs where you have to traverse and decode the entire struct, and allocate memory for each field before you can access a single field in the record. In the case where you wanted to read just a couple fields (pretty common for a DB record), this could be pretty wasteful. Again, the data required for indexing nostr events is fairly small so this may not be a big deal -- my choice of default is from experience with much larger records.
One more thing: flatbuffers is quite interesting (and I think unique?) in that it allows you to separate the error checking from the access/deserialisation. For example, when you deserialise you typically assume this input is untrusted, so you'll be doing bounds-checking, field validation, etc. However, in the case where the records are purely created and consumed internally, these checks are unnecessary. This is one reason flatbuffers is so effective for the use-case of database records.
Sorry for this big wall of text!

2023-05-07 14:47:37

- reply

Got it! I will try to think of the best way to do read-level authorisation, which I think is the building block we need for this. I'm thinking something like a "virtual filter" that is applied to all queries and has the ability to remove disallowed events from the output. I think this would be integrated at the DB querying level, so that (ie) you don't get fewer results than your limit, as might happen if it was done at a post-processing stage.

2023-05-05 17:41:38

- reply

Hey guys, sorry for the delay!
Just to clarify: You'd like it so a particular user (authenticated in some way, maybe NIP-42?) can only access a subset of the keys in the DB?
I think that would be possible (assuming authentication is implemented of course). As Mike sas, it wouldn't be quite as easy as with SQL, where you could add a WHERE clause to all the required queries. However, I think you could do it in a way that wouldn't affect performance drastically.
If the sets of events for each user are mostly disjoint (not too many duplicated), then you could possibly run a strfry instance for each user. A running instance has a relatively low overhead, especially if you scale down the various thread counts.

2023-02-17 22:53:17

- reply

Actually strfry *does* compress relay->client messages by default, using websocket per-message compression with sliding window (if supported by client).
This is configurable with the relay.compression.enabled and relay.compression.slidingWindow parameters in the beta branch.
The strfry logs also include the compression negotation parameters on client connect, and the total compression ratios on disconnect (up and down).

2023-01-23 17:24:26

- reply

It's a cool idea! I wonder what the best thing to shard by would be. I could imagine other topologies where you shard on the last bits of author pubkey, or root event ID of a thread.

2023-01-12 15:21:13

- reply

There is a "websocket" thread that is async since it handles multiplexing all the different websocket connections. However, it delegates most CPU or I/O bound operations to various thread pools. There are more details in the Architecture section of the README: https://github.com/hoytech/strfry#architecture

2023-01-11 01:02:08

- reply

Thanks!
"yesstr" is currently only a basic flatbuffers wrapper around the sync protocol from the Quadrable library. Quadrable itself is used for a few niche things currently, but I believe it will be a perfect fit for nostr.
strfry always has a plan for any possible query, even if it's not optimal. Worst-case scenario is it falls back to a full scan of an index. However: 1) all indices are clustered by timestamp, so since/until can be used to reduce the query execution time, and 2) scanning an index is actually really fast (millions of rows per second, depending on what proportion is in the page cache and how fast the disk is). This is similar to what any SQL DB will do too, in the case it can't find an appropriate index.
There are also some basic limits for queries that can optionally be configured, such as maximum value for "limit" in a query, maximum query execution time, a limit on the time in the past for records that can be queried, etc. I'm planning on watching the performance with real-world queries and figuring out what the best configuration ends up being.

2023-01-10 22:31:11

- reply

I'm planning on it, yes! It will be at relay.strfry.net :)

2023-01-10 18:22:03

- reply

Thanks! I'm looking forward to actually testing strfry on production traffic. :)
I think the merkle-tree syncing could really help with distributing data amongst relays too, I'm going to write more on this shortly.

2023-01-10 18:02:58

- reply

Hi guys! Sorry still figuring out how to use the clients properly. :)
I'm using astral, with default-configured relays, FWIW..

2023-01-09 21:00:05

Hey all! Here's a new nostr relay implementation I'm working on: https://github.com/hoytech/strfry
It's still in beta/dev, but pretty close to production ready. Most interesting feature is a merkle-tree based set reconciliation protocol for syncing messages between servers.