Itamar Turner-Trauring

Articles on Python performance (with a focus on data science / scientific computing) and Docker packaging at https://pythonspeed.com (click through and sign up for my newsletter for weekly articles in your inbox!). Building a performance and memory profiler for Python data science and scientific computing at https://sciagraph.com.

Public Key

npub1l3hcw0upjp59wljh5l5e70uzter9kez5t4d6vvl657w4u685alfs52qxlu

NIP-05 Address

itamarst_at_hachyderm.io@mostr.pub

Profile Code

nprofile1qqs0cmu887qeq6zh0et606vl87p9u3jmv3296kaxx0a20827dr6wl5cpz4mhxue69uhhyetvv9ujumt0wd68ytnsw43qelm75j

Publishing to

relay.mostr.pub

Author Public Key

npub1l3hcw0upjp59wljh5l5e70uzter9kez5t4d6vvl657w4u685alfs52qxlu

Show more details

Last Notes

2024-09-16 16:38:23

- reply

@npub1pfe…e2dm You can review https://github.com/twisted/twisted/pull/12312 while you wait!

2024-09-16 13:36:02

Writing a Rust extension for Python is surprisingly simple: the Maturin packaging tool makes it really easy to get started, as it initializes a full working project template for you, and PyO3 gives you access to the full power of Python from Rust.
I wrote an article demonstrating a simple example, and then how I optimize it to run even faster.
https://pythonspeed.com/articles/intro-rust-python-extensions/
#python #rust

2024-09-13 20:47:54

Python 3.8 is end of life in October. It's time to upgrade!
(14% of PyPI downloads are from Python 3.8. Lots of people are still using it...)
https://pythonspeed.com/articles/stop-using-python-3.8/
#python

2024-08-06 21:48:15

- reply

@npub1pfe…e2dm There's a "Filter this post" on the toot-specific "..." menu in my server, at least.

2024-07-12 16:07:25

For those not aware: you should not be using Eventlet (and probably not Gevent either). The fundamental approach of monkey-patching the Python standard library is untenable without massive resources, resources that don't exist.
Eventlet has a migration path off via asyncio integration, not sure about Gevent.
#python

2024-07-11 00:30:16

I am about to run cmake send help

2024-07-08 14:45:07

Oh, cool, Python exceptions now have as of 3.11 an `add_note()` function that lets you add extra notes to an exception. They get displayed with the traceback.
https://docs.python.org/3/library/exceptions.html#BaseException.add_note
#python

2024-06-28 14:33:19

PyO3 0.22 is out, so if you have Rust-based packages you can now start releasing Python 3.13 wheels.
#python

2024-06-20 19:03:38

Working on upgrading some open source Python packages to support 3.13, and it's kinda neat how much Rust support is becoming a blocker for progress, because it means Rust is being used in key packages.
(Upcoming PyO3 0.22 will add 3.13 support, so this blocker will hopefully be gone in a short while.)

2024-06-12 15:55:17

I wrote a mini-benchmark to test my obviously-won't-slow-things-down bug fix.
It slowed things down by 100×...

2024-05-30 15:23:49

Another example of why you want to optimize your code before switching to parallel execution: an image processing algorithm where I get a 15× speedup by replacing generic APIs with specialized datastructures that allow reducing repetitive execution.
https://pythonspeed.com/articles/lets-optimize-median-local-threshold/
#python

2024-05-07 21:58:36

The hardest problem in computer science is linking (the thing that happens after compilation and breaks everything, not the thing websites do).

2024-04-23 19:36:19

TFW the code is doing something ridiculously inefficient but you can't fix it because it's an accidentally public API in a library.

2024-04-17 23:03:52

Anyone have suggestions for a tech conference in the US that is:
1. Suitable for both frontend and backend web developers, and
2. In a state anyone from the US can feel comfortable and safe traveling to?
#python #javascript #programming

2024-03-20 22:53:24

At some point I added a book about ISO 9001 to my wishlist and I don't even know what that is and I will probably be sad if I look it up.

2024-03-19 19:13:39

The hardest problem in computer science is remembering this device.

2024-03-15 18:38:03

I'm half-sick so not at my best, and still, got a random widely used open source library to be meaningfully faster than before without trying very hard.
I strongly believe that many open source libraries could run 2×-10× faster if someone just spent the time optimizing them.
And if you want to pay my consulting rates, that someone could be me!

2024-03-09 13:34:19

There has always been quite varied Jewish opposition to Zionism, from early 20th-century socialists (https://en.wikipedia.org/wiki/General_Jewish_Labour_Bund) to modern-day fundamentalist (https://en.wikipedia.org/wiki/Satmar) and secular Jews (https://en.wikipedia.org/wiki/Tony_Judt).
As someone Jewish who grew up in Israel, and has family there, I am very skeptical of people who claim all anti-Zionism is antisemitic. There's usually another post nearby explaining why it's totally OK for Israel to kill 10,000 children.

2024-02-26 18:36:05

AI! It can help you with cooking! (If you're not an American, the punchline to the joke is that fluid ounces are a measure of volume.)
https://media.hachyderm.io/media_attachments/files/111/999/223/018/703/790/original/3f1b0670e6d5e4b9.png

2024-02-24 13:27:29

I've been getting ads from IBM for modernizing COBOL codebases with AI and now I'm concerned all the money in my bank is going to disappear.

2024-02-16 14:15:58

Google Gemini is still confabulating APIs that don't exist (e.g. claiming `numba.profile` is a thing).
Whenever I complain about this people say "the web also has wrong results". But someone has to write that wrong result on a page somewhere for it to show up.
This in contrast is vastly worse than a search engine: it's not summarizing a page that happens to be wrong, it's creating _new_ nonsense.
Humans are really good at generating nonsense, I don't want help from computers.

2024-02-13 18:01:47

If you want to use GPUs from Python, most libraries are NVIDIA-only. That means running your software on Macs, or in CI, is going to be difficult. But portable GPU programming is possible; in this article I talk about one option, the wgpu-py library.
https://pythonspeed.com/articles/gpu-without-cuda/
#python

2024-01-26 20:22:17

Update: I now have a working Numba profiler prototype. Technically, "anything you can get a stack trace out of with gdb" profiler.
10ms sampling resolution seems like it'd work on most computers. The naive "launch gdb for every sample" gives me 700ms resolution, so this is a lot better.
Next is UX. I'm hoping I can do more than just full process profiling (Jupyter support would be very nice) but even that is useful.
#python

2024-01-11 19:19:41

Writing about software performance is so fun...
Me: I'll write a little example to demonstrate this particular effect, where a small dataset will run faster than a large dataset.
Computer: I think you mean the smaller dataset will be slower?
Me: No?
Computer: Yes.
Me: What.
Computer: HAHA

2024-01-09 15:55:52

NumPy 2 is coming out in couple months! And it's a little backwards incompatible, which means any applications that depend on it (directly or indirectly) might break.
I wrote an article showing how to prevent breakage in the short term, and how to automatically upgrade in the long term.
https://pythonspeed.com/articles/numpy-2/
#Python #NumPy

2024-01-02 22:55:36

- reply

@npub1pfe…e2dm I assume it's like, a trademark violation or something?

2023-12-25 21:58:05

- reply

@npub1pfe…e2dm It has preset profiles, doesn't it? Right above the individual settings, IIRC. So you just choose the Low profile and move on.

2023-12-18 19:26:44

- reply

@npub1pfe…e2dm Right, but also e.g. copying mega-org design patterns like microservices in small orgs.

2023-12-12 18:03:35

https://github.com/nschloe/perfplot is a neat library for generating comparative performance graphs for different functions across a range of inputs.
Unfortunately it's GPL-3 which means I can't use it in some contexts. Anyone know of an alternative for #Python?
So far I've found https://github.com/MSeifert04/simple_benchmark

2023-12-06 17:49:52

Still trivially easy to make Google Bard confabulate utter bullshit.
Prompt: "what crimes were committed by cambridge ma city council members" (correct answer: last one I can find was 1994).
What Bard says: "The City Council is responsible for awarding contracts for services and goods. There have been allegations of favoritism or impropriety in the procurement process."
The City Council is _not_ responsible for awarding contracts. At all. The answer is just plausible-sounding nonsense.

2023-11-21 17:50:08

- reply

@npub1pfe…e2dm @npub143y…3g53 Can't wait https://www.youtube.com/watch?v=f7sjvD89Mek

2023-11-21 17:28:29

- reply

@npub1pfe…e2dm @npub143y…3g53 Why this one though? It's not like we lack more realistic apocalypses.

2023-11-20 16:03:04

- reply

@npub1mjj…h2pr https://pypi.org/project/line-profiler/ is probably the easiest way to get started. It has some limitations which you'll start noticing after using it for a bit, but it's a good starting point.

2023-11-20 15:38:01

- reply

@npub1mjj…h2pr For speed, the first answer is almost always another question: do you have evidence that this is a bottleneck? If it's 1% of your program runtime, making it twice as fast won't make much of a difference...
If you have evidence that this part of the code is a bottleneck, easiest thing to do is just measure the two versions and compare. E.g. with %timeit magic in IPython/Jupyter: https://ipython.readthedocs.io/en/stable/interactive/magics.html#magic-timeit

2023-11-10 03:11:32

Thinking about it some more, the fundamental flaw with CoPilot is that the goal is to make it faster and easier to generate code.
As a programmer, you don't want to generate code.
You want to solve problems, and the less code involved the better. I don't want to tools that will generate boilerplate, I want less boilerplate.
https://github.blog/2022-09-07-research-quantifying-github-copilots-impact-on-developer-productivity-and-happiness/ shows the problem. "Less mental effort on repetitive tasks", sure, but you should be trying to _prevent_ the repetition, not do more of it.

2023-11-09 21:37:01

The year I worked at Google was the Year Of Social, and everyone's OKRs involved integrating with Google+ somehow, and if you did that you got a large bonus.
Anyway getting strong vibes like that when GitHub's new font family's landing page starts randomly talking about Copilot half-way.

2023-10-16 16:14:56

- reply

@npub1pfe…e2dm Weirdly you get this on the left too. "The market is why housing is so expensive, so market solutions can't fix it", without acknowledging how utterly dependent the outcomes are on massive government regulation (zoning, building code). Regulations that could be changed!

2023-10-16 16:05:27

- reply

@npub1pfe…e2dm More seriously I am reminded I need to read https://en.wikipedia.org/wiki/The_Great_Transformation_(book):
"The Great Transformation argues that markets cannot solely be understood through economic theory. Rather, markets are embedded in social and political logics, which makes it necessary for economic analysts to take into account politics when trying to understand the economy."

2023-10-16 16:04:20

- reply

@npub1pfe…e2dm
https://media.hachyderm.io/media_attachments/files/111/245/538/146/315/862/original/228a71d329926027.png

2023-10-14 23:11:48

- reply

2/ In the past month, 26 billion packages were downloaded from PyPI. Most of that with pip.
I recently landed a PR in pip that cut memory usage dramatically, especially for large downloads. (If the next release of pip breaks, sorry, my bad 😟 ). Technically most of it was in a third party library pip depended on, whose maintainer barely had time to do anything.
There's no blame here, it's just people don't have time. I only did because I was paid to fix this issue.
So—

2023-10-14 23:08:25

1/ What can you, a software engineer, do to fight #ClimateChange? My initial answer continues to be politics (https://codewithoutrules.com/2019/09/10/software-developers-climage-change/) but I now have a secondary answer.
There's a lot of software out there and it's ludicrously inefficient. Not because people don't care, but because no one bothered to look, or the cost is spread across millions of people, or because no one has time (open source! it's fun.)
Let me give you some examples—

2023-07-18 15:27:30

New major release of Cython means a bunch of #Python packages using Cython that didn't pin their build dependencies are now broken. And if there's no wheels, now that's your problem 😭