**RT @karpathy:**
Oops haven't tweeted too much recently; I'm mostly watching with interest the open source LLM ecosystem experiencing early signs of a cambrian explosion. Roughly speaking the story as of now:
1\. Pretraining LLM base models remains very expensive. Think: supercomputer + months.
2\. But finetuning LLMs is turning out to be very cheap and effective due to recent PEFT (parameter efficient training) techniques that work surprisingly well, e.g. LoRA / LLaMA-Adapter, and other awesome work, e.g. low precision as in bitsandbytes library. Think: few GPUs + day, even for very large models.
3\. Therefore, the cambrian explosion, which requires wide reach and a lot of experimentation, is quite tractable due to (2), but only conditioned on (1).
4\. The de facto OG release of (1) was Facebook's sorry Meta's LLaMA release - a very well executed high quality series of models from 7B all the way to 65B, trained nice and long, correctly ignoring the "Chinchilla trap". But LLaMA weights are research-only, been locked down behind forms, but have also awkwardly leaked all over the place... it's a bit messy.
5\. In absence of an available and permissive (1), (2) cannot fully proceed. So there are a number of efforts on (1), under the banner "LLaMA but actually open", with e.g. current models from @togethercompute (https://nitter.moomoo.me/togethercompute), @MosaicML (https://nitter.moomoo.me/MosaicML) ~matching the performance of the smallest (7B) LLaMA model, and @AiEleuther (https://nitter.moomoo.me/AiEleuther) , @StabilityAI (https://nitter.moomoo.me/StabilityAI) nearby.
For now, things are moving along (e.g. see the 10 chat finetuned models released last ~week, and projects like llama.cpp and friends) but a bit awkwardly due to LLaMA weights being open but not really but still. And most interestingly, a lot of questions of intuition remain to be resolved, e.g. especially around how well finetuned model work in practice, even at smaller scales.
https://nitter.moomoo.me/karpathy/status/1654892810590650376#m
Brian Armstrong 🛡️ / @brian_armstrong (RSS Feed) /
npub1jj…6h3zc
2023-05-06 16:56:07
Author Public Key
npub1jj9575693w997wqncd3daxp8alpxs9uamu9k03xtrymtaxxfzfkst6h3zcSeen on
wss://nos.lolPublished at
2023-05-06 16:56:07Event JSON
{
"id": "50a5d9277aeb186c3d2729a6ee777d40d1cb3621f72f7109a8fa407ce7308384",
"pubkey": "948b4f53458b8a5f3813c362de9827efc268179ddf0b67c4cb1936be98c9126d",
"created_at": 1683392167,
"kind": 1,
"tags": [],
"content": "**RT @karpathy:**\n\nOops haven't tweeted too much recently; I'm mostly watching with interest the open source LLM ecosystem experiencing early signs of a cambrian explosion. Roughly speaking the story as of now:\n\n1\\. Pretraining LLM base models remains very expensive. Think: supercomputer + months.\n\n2\\. But finetuning LLMs is turning out to be very cheap and effective due to recent PEFT (parameter efficient training) techniques that work surprisingly well, e.g. LoRA / LLaMA-Adapter, and other awesome work, e.g. low precision as in bitsandbytes library. Think: few GPUs + day, even for very large models.\n\n3\\. Therefore, the cambrian explosion, which requires wide reach and a lot of experimentation, is quite tractable due to (2), but only conditioned on (1).\n\n4\\. The de facto OG release of (1) was Facebook's sorry Meta's LLaMA release - a very well executed high quality series of models from 7B all the way to 65B, trained nice and long, correctly ignoring the \"Chinchilla trap\". But LLaMA weights are research-only, been locked down behind forms, but have also awkwardly leaked all over the place... it's a bit messy.\n\n5\\. In absence of an available and permissive (1), (2) cannot fully proceed. So there are a number of efforts on (1), under the banner \"LLaMA but actually open\", with e.g. current models from @togethercompute (https://nitter.moomoo.me/togethercompute), @MosaicML (https://nitter.moomoo.me/MosaicML) ~matching the performance of the smallest (7B) LLaMA model, and @AiEleuther (https://nitter.moomoo.me/AiEleuther) , @StabilityAI (https://nitter.moomoo.me/StabilityAI) nearby.\n\nFor now, things are moving along (e.g. see the 10 chat finetuned models released last ~week, and projects like llama.cpp and friends) but a bit awkwardly due to LLaMA weights being open but not really but still. And most interestingly, a lot of questions of intuition remain to be resolved, e.g. especially around how well finetuned model work in practice, even at smaller scales.\n\nhttps://nitter.moomoo.me/karpathy/status/1654892810590650376#m",
"sig": "7a7d9551536b3faa65d25262b5cc1cb7ed123a16ad8ce50afc3cf27b0b13b0cd287bea0b8caf862668ea21e5f06eec45e27457ee8f2eb183527a548e7086a41a"
}