Jamie on Nostr: I'd previously been using the #LocalAI Vulkan container for #LLM inference. Recently ...
I'd previously been using the #LocalAI Vulkan container for #LLM inference. Recently I got #ROCm working on my RX 5600 XT thanks to #Debian and it's 2x to 4x faster. Cool. Today I compiled #llamacpp Vulkan server from source and inference runs at the same speed as ROCm. I wonder what's going on here...
Published at
2024-12-22 12:25:28Event JSON
{
"id": "49fffb2e76bdbe3966743c5285f564724f665fe8537da88b87dde0e83b7431f4",
"pubkey": "c26652ca20d6b13926606a33999782d412ab4c3b68fdbda131245c3c997d2070",
"created_at": 1734870328,
"kind": 1,
"tags": [
[
"t",
"localai"
],
[
"t",
"llm"
],
[
"t",
"rocm"
],
[
"t",
"debian"
],
[
"t",
"llamacpp"
],
[
"proxy",
"https://fosstodon.org/users/suprjami/statuses/113696461873522764",
"activitypub"
]
],
"content": "I'd previously been using the #LocalAI Vulkan container for #LLM inference. Recently I got #ROCm working on my RX 5600 XT thanks to #Debian and it's 2x to 4x faster. Cool. Today I compiled #llamacpp Vulkan server from source and inference runs at the same speed as ROCm. I wonder what's going on here...",
"sig": "633e8e0c952b72ef67337617c5c695a61e664410be8c44ec5a8be88c1a706054249796ae24e8f8ae9cf2c2a614364e280e8555c63f802b2a985949f4a672c440"
}
