/ Squiggs is on Nostr

"88b50b3fb240ab6620447e6acb7b5198ce3b1627281e3a65a3a078e8678ae9b9163f12d54f5433b41cecf261da4fec372060aee0140ee9a12fad66f09f84a586"

Last Notes

2025-01-08 10:21:32

- reply

🍊

2024-10-31 07:49:18

- reply

Great work! Is this going to be a regular activity at events?
I picked a couple of nodes after Nostriga when sightseeing from the roof of the
Latvian Academy of Sciences.

2024-08-09 11:25:50

- reply

# Semantic Ultra-compression for Low Bandwidth Communication
## Concept Overview
In any communication system, there is work required to transmit information. This work is distributed among three main components:
1. Sender
2. Transmission System
3. Receiver
Traditional high-bandwidth systems place most of the workload on maintaining the transmission infrastructure, with minimal effort required from senders and receivers. However, in very low-bandwidth scenarios (e.g., mesh radio networks), this approach becomes impracticable.
The Semantic Ultra-compression project aims to shift the workload from the transmission system to the senders and receivers, enabling effective communication over extremely limited bandwidth channels.
## Key Components
1. Shared Reference Dictionary
2. Encoding Process
3. Transmission
4. Decoding Process
## Process Flow
1. Pre-setup:
- All participants download a shared large language reference dictionary and the encoding/decoding application.
- This represents a significant upfront investment of time and resources by all users.
2. Encoding (Sender's work):
- Sender types a message
- App finds closest semantic matches in the reference dictionary
- Sender manually reviews and selects best combinations to convey intended meaning
- This process may involve multiple iterations and careful consideration by the sender
- References are hashed into a Merkle Tree
3. Transmission:
- Only the top Merkle Tree hash is transmitted
- This minimises the work required by the low-bandwidth transmission system
4. Decoding (Receiver's work):
- Receiver's app receives the Merkle Tree hash
- App performs intensive computational work to reconstruct the message:
* It systematically tries combinations from the shared reference dictionary
* Each combination is hashed and compared to the received Merkle Tree hash
* This process continues until an exact match is found
- The receiver may need to wait a significant amount of time for this process to complete
- Once a match is found, the app reconstructs the exact message approved by the sender
- The receiver reviews the decoded message to understand the sender's intended meaning
This flow emphasises that both sender and receiver invest significant time and computational resources in the communication process, offsetting the limitations of the low-bandwidth transmission system. The decoding process, while computationally intensive, results in an exact reconstruction of the sender's approved message, ensuring fidelity of communication.
## Benefits
1. Enables communication over extremely low-bandwidth networks
2. Preserves exact semantic meaning of messages as approved by the sender
3. Utilises participant resources instead of transmission infrastructure
4. Ensures perfect reconstruction of the original message, eliminating ambiguity
## Challenges
1. Requires significant pre-setup (dictionary download)
2. Encoding process requires manual effort from the sender
3. Decoding process is computationally intensive and potentially time-consuming
4. Balancing compression ratio with semantic accuracy during the encoding phase
5. Ensuring the shared dictionary is comprehensive enough for various communication needs
## Optimization Strategies
While the basic conceptual model demonstrates the core idea of shifting work from the transmission system to senders and receivers, it presents extreme inefficiencies, particularly in the decoding process. Here are some strategies to address these inefficiencies:
### Partner Merkle Tree Hash
The primary optimization involves the sender's app generating a partner Merkle Tree hash that accompanies the main message hash. This partner hash is designed to be low-work to resolve and contains information to drastically reduce the permutations needed to decode the main message hash.
#### Concept:
1. Divide the shared dictionary into multiple sections (e.g., 1024 sections).
2. These sections could be organised:
- Randomly (to distribute common phrases)
- By semantic association (grouping related concepts)
- By frequency of use (to optimise for common communications)
3. The encoding process tracks which dictionary sections were used.
4. A partner Merkle Tree is created from this section usage information.
5. Both the main message hash and the partner hash are transmitted.
#### Decoding Process:
1. The receiver's app first decodes the partner hash (low-work process).
2. This reveals which dictionary sections were used in encoding.
3. The app then only needs to check permutations within these identified sections to decode the main message hash.
#### Benefits:
- Drastically reduces the number of permutations to check during decoding.
- Maintains the low-bandwidth transmission requirement.
- Preserves the security and privacy aspects of the hashing system.
### Additional Optimization Ideas:
1. **Adaptive Dictionary Sections**: Dynamically adjust section sizes based on usage patterns to further optimise common communications.
2. **Layered Hashing**: Use multiple layers of partner hashes for extremely large dictionaries, each layer providing more specific guidance for the decoding process.
3. **Semantic Tagging**: Include broad semantic categories in the partner hash to help guide the decoding process towards relevant dictionary sections more quickly.
4. **Frequency Hints**: Incorporate usage frequency data in the partner hash to prioritise checking of more commonly used phrases or words.
5. **Error Correction**: Implement a simple error correction system in the partner hash to allow for some transmission errors without completely failing the decoding process.
## Potential Applications
1. NOSTR protocol over LoRa radio
2. Disaster communication scenarios
3. Censorship-resistant messaging systems
This concept leverages the motivation of specific user groups (e.g., NOSTR and mesh radio network users) who are willing to invest more effort in the communication process to overcome bandwidth limitations. The significant work required by both senders and receivers makes this approach suitable only for scenarios where traditional high-bandwidth communication is unavailable or undesirable. However, it guarantees exact reconstruction of the sender's intended message, making it reliable for critical communications in challenging environments.

2024-03-02 18:50:24

Just noting an idea that dropped into my head as I experiment with Meshtastic over LoRa, which I'm (almost certainly naively) thinking could allow "compressed" message size such that consumption of relay storage is dramatically reduced and transmission could be supported even on low-bandwidth decentralised mesh networks over LoRa radio...
- Train a custom, single-purpose LLM for a type of Dictionary-based Redundancy Compression (DbRC), such that messages can be losslessly described by the minimal-size format of reference(s) to the model (base best-match reference plus modifier references to refine it to an exact match)
- Clients and Relays can download a copy of the DbRC LLM (existing practicality of local voice recognition data files on mobile makes me think this could be too)
Each message is optionally passed as the reference(s) to the shared DbRC LLM
- Clients/relays without a local copy of the LLM can decompress (and compress, if this would ever be worthwhile?) via a service (assuming they have sufficient bandwidth)
I know basically nothing about compression approaches or custom LLMs, so is this even practicable?
Has the challenge of Nostr over low-bandwidth mesh networks been solved already?

2024-01-13 20:19:17

- reply

Was your sump pump overwhelmed too quickly to move precious items?
Or no sump pump at all?
https://jenkinsrestorations.com/the-basics-and-the-benefits-of-a-sump-pump
Really sorry to hear about this. Thanks for everything you do.

2023-11-02 07:46:56

@npub1sn0…jdv9 🫂

2023-10-31 01:52:19

- reply

Signs up saying no Halloween events on Shibuya streets. Last night there were some access restrictions.

2023-10-30 13:14:11

- reply

I'm back at my hotel (phone died). It looks like the building and sign in Leo's photo

2023-10-30 12:39:16

- reply

I think there's more than one Belle Salle on Tokyo. I'm at the Sumitomo Fudosan Shibuya First Tower Belle Salle now, and can't see anything matching your photo - have I got the right Belle Salle?

2023-10-27 14:02:07

- reply

Some Kato track in there? https://www.katomodels.com/
I'm flying over the Black Sea on my way to Nostrasia right now - hoping to check out some of the local model railway culture while I'm there.

2023-08-05 08:56:35

- reply

"Deletion-compliance relay crawlers" sending then deleting notes from random nsecs could do a reasonable job of flagging relays that didn't comply (whether they claim to or not).
Couldn't detect "logical deletion" this way, though. So likelihood of a copy somewhere is high.

2023-07-16 10:08:33

- reply

I signed up a while back but no acknowledgement or confirmation, so I can't book flights or accommodation as I've no way of knowing if I have a spot reserved.

2023-07-16 10:03:33

- reply

Same for me. Can't book flights and accommodation until I get confirmation

2023-04-20 20:28:20

- reply

... using self-hosted, decentralised AI