Tom_Drummond

I’m the Melbourne Connect Chair of Digital Innovation for Society at the University of Melbourne.

Public Key

npub1g4ss3v8573z4aus0ytsq3ewah9cphk5p48ffnfz5hp8htsc9ndjqphtq97

NIP-05 Address

Tom_Drummond@mathstodon-xyz.mostr.pub

Profile Code

nprofile1qqsy2cggkr60g3277g8j9cqguhwmjuqmm2q6n55e532tsnm4cvzekeq5ggrjz

Author Public Key

npub1g4ss3v8573z4aus0ytsq3ewah9cphk5p48ffnfz5hp8htsc9ndjqphtq97

Show more details

Last Notes

2025-04-28 05:45:45

- reply

@nprofile…59k6 @nprofile…4wdl OK - I think you win the argument with "enjoinment". This kind of thing is an example of language evolution perhaps...
There's one example that annoys me here in Aus; the word "verse" has acquired a new sense as a verb meaning "to oppose" or "to play against" (in sports). So for example it would be common to hear "Collingwood versed Essendon on ANZAC day" (which was last Friday). I think this is a backformation that started with hearing "versus" as "verses" and interpreting it as 3rd person present of "to verse".
Of course "to verse" already has a meaning as a verb as those of us well versed in the English language are aware!

2025-04-23 01:54:42

- reply

@nprofile…59k6 "confabulation" is one that got me; its sense nearly flips as you cross the Atlantic.

2025-04-23 01:51:05

- reply

@nprofile…59k6 Well - your legal dictionary clearly specifies it as a contronym in the first sentence!
I think I was sensing that the two share a commonality in that they both constitute instructions. In these examples, "from" is used in its preventing form and I'm supposing that it can be interpreted as "to not" and so I'm moving the sense inversion from enjoin into from (but maybe that's a stretch too far...?).
Language is slippery..! (and no, I don't mean it can keep your feet warm :)

2025-04-23 00:40:17

- reply

@nprofile…59k6 Hmm - I think in both cases the word indicates an instruction or direction. Sometimes it's to take an action and sometimes to not take one - but I think that distinction comes from the subsequent preposition (to => act; from => don't act)...?

2025-04-15 03:29:04

- reply

@nprofile…59k6 Another way I have used to get at the exponential map when I'm teaching is the series \( \exp(x) = \lim_{n\to\infty} \left( I+\tfrac{x}{n} \right)^n\). We're still working embedded in GL(n), but the intuition is that if \( x \) is in the tangent space at \( I \) then \( I+\tfrac{x}{n} \) converges onto the group manifold as \( n\to\infty \). Then raising this group element to some power gives us a group element back by closure.
I admit this is still much less elegant than defining in terms of a derivation but it seems less like cheating to me... whereas the Taylor series approach does feel like pulling a rabbit from a hat.

2025-04-10 06:16:13

- reply

@nprofile…59k6 I think my first was the Palm Vx - very pretty design!

2025-04-10 02:44:01

- reply

@nprofile…59k6 I'm guessing you are the one with the palm pilot...?

2025-04-10 02:42:52

- reply

@nprofile…fa4k I'll take the compliment; thank you!
I think it's a case where I intuitively realised two things were connected but never got round to putting the effort to think about it carefully enough to get the correspondence correct.
Interestingly, a case where first order harmonics do appear is in plasmons in gold nanoparticles, where the principal modes can be thought of as the electron cloud oscillating from side to side relative to the mass of the nanoparticle.
IIRC, these are usually stimulated by laser, but are 10 (possibly more...?) times smaller than the optical wavelength, so can be used to create large local field strengths and interrogate matter on a sub-wavelength scale. Or to make pretty glass... https://en.wikipedia.org/wiki/Lycurgus_Cup

2025-04-09 23:01:33

- reply

@nprofile…fa4k
Right, now that I can count properly (6 dof in symmetric matrices = 1+5), I realise that first order series represents a shift in the centre while second order gives the ellipsoid shapes.
Thanks for fixing up my thinking!

2025-04-09 22:58:00

- reply

@nprofile…fa4k Really? - only 3 degrees of freedom in symmetric 3x3 (not 5 as one would expect for d orbital)…?
I always thought of p orbitals as deviations from a sphere - so if p1 = x then points on the sphere with positive x move proportionately in the positive x direction, while those with negative x move in the negative x direction to create an ellipsoid, a bit like tides.

2025-04-09 22:44:36

- reply

@nprofile…fa4k Not incidentally, this is equivalent to expressing the shape in terms of 0 and first order spherical harmonics which also describe s and p orbitals, while higher order harmonics express f orbital, etc. Spherical harmonics arise as a basis for solutions to the wave equation on a sphere - and can be expressed as homogeneous harmonic polynomials of increasing order evaluated on the unit sphere - behaving like a fourier series (indeed spherical harmonics on S1 are exactly a fourier series).

2025-04-09 22:27:16

- reply

@nprofile…fa4k It’s usually parameterized as the set of symmetric 3x3 matrices. The ellipsoid is the set of points that satisfy x^T M x = 1. One can create a homogeneous version with 4x4 matrices that also express the centre of the centre of the ellispoid.

2025-03-20 01:44:01

- reply

@nprofile…59k6 @nprofile…fa4k @nprofile…hszg Since I follow all three of you, I can also follow this sub-thread. I have been feeling your collective pain (both past and present) and wondering what I can possibly add beyond platitudinous condolences and noting that a large fraction of the planet is reeling from this one.
The question is what we can do about the future - in particular the talk that @nprofile…fa4k has to give on this topic. I'm suspect that it is extremely difficult to thrive as an academic in an oppressive regime and a huge number of academics are still in denial or anger about this. Some have moved on to bargaining - trying to work out what compromises they will have to make to continue to thrive - thus the talk. Personally, I'm caught in depression but I think the best advice when people are ready to accept it and disruptive as it surely is, will be to move to another country (as you are). I suspect that academicjobs.com or jobs.ac.uk (and similar) are going to be heavily visited.
Apologies for the 5-stages framing. It's probably over simplified as a model - but seemed useful in this circumstance.
I count myself unbelievably lucky to be in Australia, but of course nowhere is immune...

2025-02-16 21:09:41

- reply

@nprofile…59k6 I just read that there’s also a 0.3% chance it will hit the moon. Not sure if this mitigate the 2.3% at all though…

2024-11-19 02:53:22

- reply

@nprofile…59k6 @nprofile…4xpp
Ah - I think I understand your point now.
If in practice tokens only ever pay attention to recent tokens, then full attention matrices are very inefficient. I believe some implementations have done lower band diagonal implementations (ie rolling context windows) to address this - but it would eliminate the occasional long distance reference that a full-fat transformer would permit - maybe this matters…?

2024-11-18 01:54:07

- reply

@nprofile…59k6 As I understand it, masking is used to obtain causality so each token only attends to the tokens preceding it (and itself).
Each token emits a query and a key and the code computes all pairwise dot products between queries and keys (using matmul) to create the attention matrix. This is then 'masked' by subtracting a large number from all the anti-causal logits in the matrix (everything above the leading diagonal if rows correspond to queries).
So the first token attends only to itself, the second attends to the first and itself - and so on. In general this uses half of the attention matrix. So the early tokens are being very wasteful, those in the middle are a little bit wasteful and those near the end are barely wasteful at all.
But maybe I have misunderstood your question...?

2024-11-17 22:08:00

- reply

@nprofile…59k6 I used a similar trick in my code to compute Hausdorff dimension; this computes all pairwise distances using the matrix mult tensorcores and so only needs the upper triangular part of the result - which again is never materialized in RAM - but is used immediately for the dimension calculation.

2024-11-17 22:04:32

- reply

@nprofile…59k6 - it’s only a factor of 2. You could probably write upper triangular only matrix mults to get it back if you cared.
There’s also ‘fast’ transformers implementations that never materialize the attention matrix in RAM - they compute tiles and immediately use these to compute contributions to the final returned value vectors (I think this is done by keeping track of the logsumexp of the contributions as they go) - these methods might also make the factor of 2 saving for causual attention.

2024-11-16 02:04:15

- reply

@nprofile…59k6 This brought back a memory…
I once managed to create a file whose name was backspace (^H) - by typing a bunch of stuff in vi thinking I was insert mode not command mode. It only showed up in a multicolumn ls where in one row two of the columns were shifted left by one character.
It took me several months to figure out how to get rid of it. I tried the obvious thing of moving all other files out of the directory and then using ‘rm -rf’ on the directory which caused rm to crash :(.
Eventually I worked out how to hex edit the file associated with the directory inode to change the filename to ‘A’ (65) - and then it was easy to delete...

2024-11-12 21:52:20

- reply

@npub1c9m…9xjd I love it; alcohol as regulariser!

2024-10-22 22:16:31

- reply

@npub1c9m…9xjd Also my unimelb user name when I try to log in via SSO. Surely there are some obvious contexts (urls, usernames) where predictive/corrective text shouldn’t apply..? They already worked out not to apply it to password fields for example.

2024-10-15 05:19:28

- reply

@npub1knz…c73p Ah I see - because at each layer you can make a discrete choice so \( 2^{\aleph_0} \) in total. I wonder how much play there is in a lattice once constructed…?

2024-10-15 05:13:53

- reply

@npub1c9m…9xjd A colleague did some work on this using burrows-wheeler. Paper is here: https://direct.mit.edu/tacl/article/doi/10.1162/tacl_a_00112/43376/Fast-Small-and-Exact-Infinite-order-Language

2024-10-15 04:58:45

- reply

@npub1knz…c73p Not only are there uncountably many ways to achieve kissing number = 12 in 3D - there’s so much extra space there that you can slide the 12 balls around and swap the positions of any pair without breaking contact.

2024-09-13 04:39:43

- reply

@npub1c9m…9xjd Scissors, paper, rock, lizard, Spock (surely...?)

2024-07-17 00:03:27

- reply

@npub1c9m…9xjd It has always annoyed me that Zipf's law doesn't converge... and so can only apply to finite sets... (in which case the law surely fails to describe the rarest elements well)

2024-05-10 04:16:10

- reply

@npub1c9m…9xjd How does it work? - does it use the mic on your smartphone to understand what you hear - or is it self-contained? (apart from the sound!)

2024-03-25 06:46:38

- reply

@npub1knz…c73p blood orange marmalade is always a good option :)