You can use it as chat with “ollama run [modelname]” but it also has a systemd service and runs as a REST api and you can build anything with it you’d build with the cloud api’s,
CodeCompanion.nvim is a good example that provides functionality similar to Copilot in neovim locally.
One gotcha with ollama is parameter size, it downloads the Q4_0 quantization of a model by default when you don’t ask for any, and today it’s generally the sentiment that there are better quantizations at the same size, and that for many of the small models Q4_0 quantization renders them useless. A good middle ground value is Q6_K, you can figure out how to pick particular ones from the ollama website’s model index.
Models to try in the size that fits you are llama3.2, gemma2, mistral-nemo, and qwen2.5
npub1q9…l5y4h
2024-10-20 09:57:35
in reply to nevent1q…q2x3
Author Public Key
npub1q94px43ftt4e6wve2ckypytlqsaktp7gksx5m4fuxgtxhdhusdxs3l5y4hPublished at
2024-10-20 09:57:35Event JSON
{
"id": "6c235684c4873c4c796e15c2780467b981d3d35d89189dac4ee9051cea1fc232",
"pubkey": "016a1356295aeb9d3999562c40917f043b6587c8b40d4dd53c32166bb6fc834d",
"created_at": 1729418255,
"kind": 1,
"tags": [
[
"e",
"8b829f2070d51be917cdc1e8c2616b51d07d4f43bd19afb1622e7c8899428a3f",
"",
"root"
],
[
"e",
"073c536743f5401de56ee5dcc387c1dd7535383bcdb48bc6b47ecf0c88f98360",
"",
"reply"
],
[
"p",
"675b84fe75e216ab947c7438ee519ca7775376ddf05dadfba6278bd012e1d728"
],
[
"p",
"c7eda660a6bc8270530e82b4a7712acdea2e31dc0a56f8dc955ac009efd97c86"
]
],
"content": "You can use it as chat with “ollama run [modelname]” but it also has a systemd service and runs as a REST api and you can build anything with it you’d build with the cloud api’s,\n\n CodeCompanion.nvim is a good example that provides functionality similar to Copilot in neovim locally.\n\nOne gotcha with ollama is parameter size, it downloads the Q4_0 quantization of a model by default when you don’t ask for any, and today it’s generally the sentiment that there are better quantizations at the same size, and that for many of the small models Q4_0 quantization renders them useless. A good middle ground value is Q6_K, you can figure out how to pick particular ones from the ollama website’s model index.\n\nModels to try in the size that fits you are llama3.2, gemma2, mistral-nemo, and qwen2.5",
"sig": "0bac6121522cf01c838ff4892fb4c6a23f111daecfa8386994b2686465ffe79a8eaf1ea32ecd1ebc982f8959d0e8ff2211fbf21ed079f36ef57f30d428bfc61b"
}