Brandon Rohrer

Robots and data and robots learning from data. Writing a book: How to Train Your Robot https://tyr.fyi 
#Robots
#Robotics
 #ReinforcementLearning 
#MachineLearning
#DataScience

[book] How to Train Your Robot: https://tyr.fyi
[blog] End to End Machine Learning: https://BrandonRohrer.com

Public Key

npub1jh4qsxnz0nhyfefjsfvcdmxxvgfe6p5vf0dvh6pq4r6ytwwxcp4sl9eag0

NIP-05 Address

brohrer_at_recsys.social@momostr.pink

Profile Code

nprofile1qqsft6sgrf38emjyu5egykvxanrxyyuaq6xyhkktaqs23az9h8rvq6c2k5ds6

Author Public Key

npub1jh4qsxnz0nhyfefjsfvcdmxxvgfe6p5vf0dvh6pq4r6ytwwxcp4sl9eag0

Show more details

Last Notes

2024-08-28 00:42:05

- reply

is this what they mean when they talk about the IKEA effect?

2024-08-23 18:44:25

- reply

I adore robots and I approve of this message

2024-08-18 12:34:12

The more I code the less I feel like I need to apologize for print debugging

2024-08-15 19:39:54

If a construction company started advertising they were “chainsaw-centric” that would sound funny, right?

2024-07-10 01:53:53

- reply

oof. You said it. It breaks me.

2024-07-10 01:18:40

I always thought that they would come back and give you a medal or a gift certificate or something once events prove that you were right all along, but it turns out they never do.

2024-07-09 14:41:32

- reply

high fives!

2024-07-09 14:34:48

- reply

💙

2024-07-09 14:34:15

I miss living in a town with functional public transportation
https://cdn.masto.host/recsyssocial/media_attachments/files/112/757/024/318/063/935/original/db7aa5e9d458d4a2.jpeg

2024-07-09 13:05:08

- reply

sounds like a radically new therapy modality to me. I’m in.

2024-07-09 13:03:26

I’ve learned the hard way that “bring your whole self to work” is a minefield for the Neurospicy so it’s safer to channel all of my quirk into Mastodon. Lucky you!

2024-07-09 11:48:39

- reply

Absolutely. Every new level of performance is a Super Mario Power-Up to the heart.

2024-07-09 10:46:26

- reply

Runtime optimization is my favorite video game

2024-07-09 10:45:29

My favorite bug:
code runs too slow
OPTIMIZATION TIME!!!
* airhorns *
* mariachi grito as they strike up their interpretation of “who let the dogs out?” *

2024-07-09 10:42:15

You are the type of person who creates a no-followers alt just to scream into the void, then you might really like journaling.
https://mastodon.social/@Daojoan/112752628110240930

2024-07-08 14:12:05

I bring order to my physical environment precisely *because* my mental environment is so chaotic. My organized desk is not the sign of an organized mind.

2024-07-08 01:01:03

This is some potent wisdom. Good stuff. https://mastodon.social/@Daojoan/112747895034335324

2024-07-07 01:13:41

- reply

I think that’s one of many forms it can take, But I feel like the spirit your characterization is accurate. I’ve been struggling to think of a planner as something that is independent of the model that it references, and I haven’t been successful. I’m sitting down to write a model class with a planning method.

2024-07-06 20:28:29

- reply

i’m coming around to this. It’s requiring a mental shift.

2024-07-06 20:20:43

- reply

it’s some function that takes in a history of states, actions, and rewards and uses it to choose the next action. It can do this by building an explicit model of its environment and simulating several steps into the future, so it is often called a planner.

2024-07-06 20:06:40

- reply

In architecture diagrams, they are often drawn as separate boxes, but as I go to implement a handful of use cases, I’m having trouble making that abstraction.
Speculative musings welcome
#ReinforcementLearning

2024-07-06 20:04:42

A fun part of working on a #ReinforcementLearning workbench is that I get to think about how to connect different kinds of agents to different kinds of worlds – representation, interfaces, abstraction.
Something I’m stumbling on is representing models and planners.
Is there such a thing as a planner distinct from a model? Or is planning just something a model does?
In object-oriented programming terms, would a planner be a separate class from a model? Or would it be a method in a model class?

2024-07-06 10:38:37

- reply

Here's the demo code, in case you feel like trying out your own tweaks:
https://codeberg.org/brohrer/myrtle/src/branch/main/src/myrtle/demos/pendulum.py

2024-07-06 10:11:40

- reply

This is unusually good performance for Q-learning. I juiced it up with my own home rolled curiosity-driven experimentation approach. I haven’t written it up yet, but the code is here.
#ReinforcementLearning
https://codeberg.org/brohrer/myrtle/src/branch/main/src/myrtle/agents/q_learning_curiosity.py

2024-07-06 10:07:12

- reply

Each time step it gets a reward based on the height of the pendulum is— ranging from zero if it’s at the bottom to two if it’s at the top.
By the time it reaches a thousand episodes, it’s performing near optimally, with an average reward of 1.96, which includes spinning up from the bottom.
That represents 1 million times steps of learning at four times steps per second-about three days at 1X speed.

2024-07-06 10:01:41

- reply

After 100 episodes this fake little #robot is really doing OK. It’s spends the majority of its time dancing around the upward position.
https://cdn.masto.host/recsyssocial/media_attachments/files/112/738/964/837/506/466/original/da4a4a88a72b7a32.mp4

2024-07-06 09:58:42

- reply

Watching an algorithm flail gets old after a few minutes. Since this is only a simulation for now, it can be speeded up. Here it is at 32X.
Every 1000 times steps a new episode kicks off, and the pendulum resets pointed downward.
https://cdn.masto.host/recsyssocial/media_attachments/files/112/738/953/515/632/504/original/38c1497ae82b8cf7.mp4

2024-07-06 09:47:44

- reply

this seems like bit that works in audio but not text, like the “you say tomato, I say tomato” show tune.

2024-07-06 09:46:21

Side project update on Myrtle, a real time reinforcement learning workbench.
I have a classic algorithm (Q-Learning) learning a classic task (getting a simulated pendulum to balance pointing upward).
When it’s just starting out, it’s just a flailing pendulum, depicted in this console animation.
https://cdn.masto.host/recsyssocial/media_attachments/files/112/738/905/064/268/128/original/bc14d9465426e5f4.mp4

2024-07-05 15:15:50

- reply

this is why they don’t let me make the rules.

2024-07-05 10:36:51

- reply

ow this one hurts a little

2024-07-05 01:23:36

Rule Of Three for avoiding over-engineering
(not standard and recommended by no one but me):
Don’t implement a fix for something until the third time it has bit you.
Save type hints for after your third type-related bug.
Don’t optimize your code until the third time it has times out.
Don’t create that tracking dashboard until the third time you’ve wished for it.
Don’t automate that deployment step until the third time it’s been forgotten.

2024-07-04 12:51:29

I feel like “microlomaniac” is a missed opportunity

2024-07-04 12:26:06

- reply

oh I reeeeeally hope they do

2024-07-04 12:14:20

- reply

they rejected my proposal of “Fiddle-Around, Find-Out merging”

2024-07-04 12:11:34

- reply

OK, now we are seriously shopping for one. It’s hard to tell from the photo – is it plastic or wood construction?

2024-07-04 01:02:34

- reply

True. Nice can.

2024-07-04 01:01:04

- reply

oh my god, it’s ENORMOUS. I bet it grinds hard.

2024-07-03 20:08:46

- reply

well now that is absolutely on my list of things to try this weekend

2024-07-03 20:06:13

Cybertruck just drove by. It really captures the upscale trash can aesthetic.

2024-07-03 13:18:13

- reply

it’s math!

2024-07-03 13:11:50

OH: She’s the answer to “what if a red flag could waive red flags?”

2024-07-03 12:51:13

If you are going to the Boston Pops concert on the Esplanade tomorrow night, it’s not too early to stake your claim on the grass. You wouldn’t be the first.

2024-07-03 11:26:53

The most direct path is rarely the one that feels the fastest in the moment.
The corollary to this is that “moving fast” usually means taking a less direct path and, ironically, taking longer to reach the goal.

2024-07-03 09:44:00

- reply

I love this. I never realized that what really sells the illusion is that Fred Astaire already moves like he’s floating through space. It makes those room axis tilts invisible to my eye in the original.

2024-07-02 12:39:23

- reply

Yesterday Me is a dick

2024-07-01 23:30:24

- reply

yes! My own entry into the competition is raising cosine similarity to an exponent, in order to make it more selective.
Sharpened Cosine Similarity https://github.com/brohrer/sharpened-cosine-similarity

2024-07-01 21:21:04

- reply

Such a classic. A blunt treatment of gradient descent.

2024-07-01 19:16:57

This tracks
https://mastodon.social/@olafurw/112712605181780855

2024-07-01 13:06:43

Life hack: When I think of something else that needs doing I put it on a list for Tomorrow Me to handle because screw that guy

2024-06-29 19:32:04

I heard @npub1wkr…yvj6 ‘s phrase “bullshit industrial complex” and now it’s living rent free in my head

2024-06-28 10:05:26

- reply

What glorious witchcraft is this?

2024-06-27 08:21:08

“And for the first time in years, when I look in the mirror, I can recognize the person looking back.”
https://mastodon.social/@Daojoan/112684896387654901

2024-06-25 16:19:27

- reply

I once joked with a data scientist in charge of defining a company’s top level metrics that she had more power than the CEO. In retrospect, it was no joke.

2024-06-25 15:16:32

- reply

This is the open secret of reinforcement learning. Sure, there are methods that can optimize against arbitrary reward functions, but the process of choosing a reward function to get the behavior you want is the darkest of arts.

2024-06-25 14:52:22

This is from a “position paper” (read “intellectually hand-wavy speculation”) by none other than Yann LeCun. Is it possible that biologically-inspired architectures are making their way from the fringe to the main stream?
https://openreview.net/pdf?id=BZ5a1r-kVsf
https://cdn.masto.host/recsyssocial/media_attachments/files/112/677/823/196/931/134/original/82cc972d3a3c0ec0.jpeg

2024-06-25 13:31:22

- reply

One interesting case study here is the human brain moving the musculoskeletal system. The ability to predict the state of the system a short time into the future appears to be a critical component of its robust performance.
6/6

2024-06-25 13:31:10

- reply

In practice this will introduce a small delay of several hardware steps, resulting in some noise in the agent. I don’t know of any way to avoid this. This is an important feature of real time #ReinforcementLearning that gets lost in simulations. Any real time controller or physical hardware has to deal with this delay.

2024-06-25 13:30:47

- reply

A better way to work around this is to take advantage of the fact that robots typical have a fast internal clock, and to have them try to read a new action at every step (or few steps) of this hardware clock, rather than at every agent step. This will ensure that as soon as the new action is available, it will get incorporated into the world.
4/

2024-06-25 13:30:26

- reply

One workaround would be to have the world sit idle for a short period to enable agent to run itself, but this would require that timing in the world to be tuned to the needs of the agent, which is dependent on the algorithm, the hardware, and other things that the world should not have to know about.
3/

2024-06-25 13:30:04

- reply

The instant it reports its sensor readings to the agent, they are already obsolete. The world has already moved on in its machinations.
If the world moves slowly compared to the speed at which the agent computes, this mismatch can be mostly ignored. But if computations are intensive, as with some vision applications, or compute power is limited, as with many mobile robots, then this delay can become considerable. In practical applications it can render modern algorithms infeasible.
2/

2024-06-25 13:29:36

A thing I learned about synchronization in real-time reinforcement learning
Synchronization between agents in worlds gets trickier when the world is operating in real time, as with robots and other physical systems. In non real-time RL, the world can wait for the agent to plan its next action. It is a turn-taking scenario where the agent waits for the world to make its next move and vice versa. But for physical hardware, the world doesn't wait for the agent to compute.
1/

2024-06-25 07:35:19

- reply

this describes my zero-to-one creating process too, and I even use the same terminology in my head, but I don’t know whether it’s standard or traditional at all.
I start top-down when the ideas are still nebulous, do more bottom-up one they become crisp. There’s lots of alternating between the two.

2024-06-24 19:56:58

- reply

you get it!

2024-06-24 19:09:27

- reply

I tried to explain how much I love this stuff, but as it comes out of my mouth, it sounds weird even to me. You love what you love.

2024-06-24 19:08:12

Chasing down a bug.
1. Is it a dumb off-by-one error? No.
2. Is it a subtle order of operations error? No.
3. Is it a slippery inter-process communication flow glitch? Not even that.
4. Is it a fundamental misunderstanding of the real time simulation reinforcement learning workbench I’m trying to build? Yes!
1. Bug diagnosed.
2. Head scratched.
3. Understanding deepened.
4. New approach developed.
5. Fix put in place.

2024-06-24 07:46:03

- reply

👏

2024-06-23 19:19:49

- reply

utterly inspiring

2024-06-23 17:51:18

I didn’t know Baldur’s Gate was based on a true story
https://cdn.masto.host/recsyssocial/media_attachments/files/112/667/202/097/350/561/original/8031833c5844005f.jpeg

2024-06-21 07:43:49

- reply

high five on SMWS—they have a perfect malt for every occasion. This one definitely screams NEW GIG.

2024-06-20 19:52:03

- reply

oh Adrianna I’m so sorry. We will remember her and the joy her pictures brought us.

2024-06-17 20:38:58

Hey #mosstadon there is a lot of moss in Iceland. In case you didn’t know.
https://cdn.masto.host/recsyssocial/media_attachments/files/112/633/887/563/466/380/original/9b31d6845b5e3b1b.jpeg

2024-06-16 20:56:32

When you read a blog post on a new architecture, but missed a key concept during implementation
https://cdn.masto.host/recsyssocial/media_attachments/files/112/628/294/472/320/502/original/25283f6bdc479ac2.jpeg

2024-06-16 17:29:57

I found the kid holding the pot of grant funding you’ve been looking for
Seljalandsfoss #Iceland
https://cdn.masto.host/recsyssocial/media_attachments/files/112/627/481/935/862/447/original/c50583c3f9871297.jpeg

2024-06-15 15:16:03

- reply

I read “can get up to 17.5 inches in length”

2024-06-15 13:27:15

- reply

I think you broke my metaphor

2024-06-15 13:00:15

- reply

you are not the hero we deserve, but you are the hero we need

2024-06-15 12:46:03

- reply

it’s telling that the business calls it a sprint

2024-06-15 12:45:18

- reply

I can walk a lot further than I can run

2024-06-15 12:44:42

My life got a lot better when I had this realization:
- if I want to look like I’m getting somewhere fast, sprint.
- if I want to feel like I am getting somewhere fast, take it at a gentle run.
- if I actually want to get somewhere fast, take it at a steady walk.
- if I want to get there even faster, sit down for a minute and find a shorter route, or choose a closer destination.

2024-06-15 01:21:01

- reply

I believe it’s a gravy salad

2024-06-14 11:34:04

Wow. I was on the edge of my seat the whole time. https://infosec.exchange/@jonathancare/112604951721836743

2024-06-13 16:18:46

- reply

eyeball and nostril count checks out

2024-06-13 16:11:24

It’s fun that Google lists me as an internet personality, but I didn’t sign up for this new look
https://cdn.masto.host/recsyssocial/media_attachments/files/112/610/186/064/602/686/original/90a4ce19b0907781.png
https://cdn.masto.host/recsyssocial/media_attachments/files/112/610/186/392/048/580/original/452de7b270e53d0b.png

2024-06-13 13:57:33

- reply

https://cdn.masto.host/recsyssocial/media_attachments/files/112/609/659/591/235/419/original/ea77f8be1af41325.jpeg

2024-06-13 12:36:57

- reply

Two days ago I wished Roger a happy Father’s Day. Apple, did you just cleverly infer that Roger is my father? And assume that I would need to check in with him after going for a walk on a sunny Thursday morning?

2024-06-13 12:36:12

Apple AI just volunteered to write me a text to check in with Roger when I reach my destination. My destination is home, 10 minutes away. Rogers is my father-in-law. I’m a grown man with grown children. Roger lives on the other side of the country And doesn’t know or care that I’m going for a walk this morning.

2024-06-12 12:24:46

Going to popularize working end to end projects with a small collection of lightweight tools as being a “short stack engineer”

2024-06-11 21:49:28

- reply

I like how you think

2024-06-11 21:49:12

- reply

sounds like a Tuesday!

2024-06-11 21:16:58

- reply

What cocktail pairs with anger?

2024-06-11 21:10:09

You know how some days you just feel mad? Today I’m mad.

2024-06-10 23:52:41

- reply

because of course it is. Made my day.

2024-06-10 21:46:19

- reply

oh wow. They just leaned right into it. Good for them.

2024-06-10 21:40:07

- reply

tagline “bring back the spice”

2024-06-10 11:53:17

I don’t reshare LI posts lightly but @npub14rj…8dnu goes for the jugular in this one. Thank you sir.
https://www.linkedin.com/posts/johnpcutler_me-ok-you-have-about-150-projects-in-progress-activity-7202362414325981184-nFMw?utm_source=share&utm_medium=member_ios

2024-06-09 21:27:02

- reply

🔥

2024-06-08 23:23:42

- reply

How do I get it?
pip install myrtle
How do I run the demo?
python3
>>>import myrtle.demo
>>>myrtle.demo.run_demo()
Where do I learn more?
The README at
https://codeberg.org/brohrer/myrtle

2024-06-08 23:22:59

- reply

Why write a new RL library?
This is a minor departure from the classical formulation of RL, but a single, consistent reward channel is an assumption baked into all existing RL libraries.
Why is it only for Linux?
Myrtle uses multiple processes, in order to get the most out of your local hardware. Forking behavior in Python’s multiprocessing library is OS dependent, and has quirks under MacOS and Windows that I haven’t worked out yet.

2024-06-08 23:22:36

Personal project update: Myrtle is a workbench for running reinforcement learning algorithms against real-time environments (including physical robots). Version 0.0.5 is out and has a snappy demo.
What’s different about Myrtle?
It allows for multiple, intermittent reward channels. This is to allow for ongoing human feedback, but not to read anything into it when the human gets distracted or steps out for a coffee.
https://cdn.masto.host/recsyssocial/media_attachments/files/112/583/533/184/204/595/original/20bbd6a47a7104d3.png