{"id":7,"date":"2026-09-28T22:43:48","date_gmt":"2026-09-28T20:43:48","guid":{"rendered":"https:\/\/linuxundich.de\/en\/?p=7"},"modified":"2026-10-03T16:58:37","modified_gmt":"2026-10-03T14:58:37","slug":"llmfit-which-ai-models-run-on-my-machine","status":"publish","type":"post","link":"https:\/\/linuxundich.de\/en\/gnu-linux\/llmfit-which-ai-models-run-on-my-machine\/","title":{"rendered":"llmfit: Which AI models run on my machine?"},"content":{"rendered":"\n

Anyone who wants to try local AI will eventually run into a pretty mundane question: which model actually runs on my machine? The selection of open-source LLMs is huge by now, and the candidates differ quite a bit in size, quantization, and memory requirements. If you go digging through model cards and gigabytes of GGUF files, that’s a whole evening gone.<\/p>\n\n\n\n

llmfit<\/a> takes this research off your hands. The open-source tool reads out your hardware and compares it against a database of language models. What you get is a pretty concrete answer to what can actually run sensibly on your own machine.<\/p>\n\n\n\n

Which models fit my hardware?<\/h2>\n\n\n\n

After launching, llmfit first detects your hardware: CPU, RAM, graphics card and available VRAM. It then compares those resources against the models it supports and sorts the list by a score.<\/p>\n\n\n\n

\"\"\/\n\t\t\t\n\t\t\t\t\n\t\t\t<\/svg>\n\t\t<\/button>
Terminal window showing llmfit’s model list on a laptop with a GeForce GTX 1050, sorted by score, with columns for parameters, tokens per second, quantization and memory requirements.<\/figcaption><\/figure>\n\n\n\n

This is useful mainly because raw model size isn’t the only thing that counts. A model can theoretically fit into the available memory and still run painfully slowly in practice. So llmfit estimates how well a model suits your hardware and what speed and memory requirements you can expect.<\/p>\n\n\n\n

Just how much this differs from machine to machine shows up with two devices from my stash. On the laptop with a dedicated GTX 1050, the 4 GB of VRAM is a hard limit. On a computer with Iris Xe graphics, on the other hand, the CPU and the GPU share the same memory, and suddenly almost everything says \u201cPerfect\u201d.<\/p>\n\n\n\n

\"\"\/\n\t\t\t\n\t\t\t\t\n\t\t\t<\/svg>\n\t\t<\/button>
llmfit’s model list on a system with integrated Intel Iris Xe graphics, where almost all entries in the Fit column are rated Perfect.<\/figcaption><\/figure>\n\n\n\n

The program also takes different quantizations into account. With locally run models, you quickly run into terms like Q4, Q5 or Q8. Put simply, this reduces the precision of the model weights to save memory and computing power. Depending on the level, you have to accept some loss in quality in return.<\/p>\n\n\n\n

This matters most for GGUF models, which are commonly used for local inference. llmfit factors the different quantization levels into its rating, so it can also compare models that come in several variants.<\/p>\n\n\n\n

If you’ve been using tools like Inxi<\/a> on Linux to get an overview of your hardware, you already know the principle. llmfit goes one step further, though, and puts the collected data directly in relation to possible AI models.<\/p>\n\n\n\n

Installation on Arch Linux<\/h2>\n\n\n\n

Update September 29, 2026:<\/strong> The wheel of time keeps turning, and on Arch it famously turns a little faster. When I started writing this article, llmfit was only available for Arch as an AUR package. Because of the turbulence and attacks on the AUR, I try to avoid it these days. In the meantime, the llmfit package has made it into the official Arch repositories, so all you need to install it is: sudo pacman -S llmfit<\/code><\/p>\n\n\n\n

On Arch Linux, uv is available directly from the official repositories. That makes the installation pleasantly unspectacular:<\/p>\n\n\n\n

sudo pacman -S uv<\/code><\/pre>\n\n\n\n
After that, you can install llmfit as a tool with uv<\/code>:<\/p>\n\n\n\n
uv tool install -U llmfit<\/code><\/pre>\n\n\n\n
Now llmfit<\/code> is available right away as a command.<\/p>\n\n\n\n
If you just want to try the program first, you can use uvx<\/code> instead. That runs llmfit without installing it permanently as a tool:<\/p>\n\n\n\n
uvx llmfit<\/code><\/pre>\n\n\n\n
So on Arch Linux, you don’t need a separate installer for uv.<\/p>\n\n\n\n
Installation on Ubuntu<\/h2>\n\n\n\n
On Ubuntu, the route is a bit different. Depending on your Ubuntu version and the repositories you use, you can’t simply install uv with apt<\/code>. The installer provided by the uv developers is therefore a straightforward option.<\/p>\n\n\n\n
With curl<\/code> it works like this:<\/p>\n\n\n\n
curl -LsSf https:\/\/astral.sh\/uv\/install.sh | sh<\/code><\/pre>\n\n\n\n
Alternatively, you can download and run the installation script with wget<\/code>:<\/p>\n\n\n\n
wget -qO- https:\/\/astral.sh\/uv\/install.sh | sh<\/code><\/pre>\n\n\n\n
After that, first check whether uv<\/code> is available:<\/p>\n\n\n\n
uv --version<\/code><\/pre>\n\n\n\n
From here on, installing llmfit works exactly like on Arch Linux:<\/p>\n\n\n\n
uv tool install -U llmfit<\/code><\/pre>\n\n\n\n
If you just want to try it out without a permanent installation, uvx is available on Ubuntu as well:<\/p>\n\n\n\n
uvx llmfit<\/code><\/pre>\n\n\n\n
If you’d rather not use uv<\/code>, you can download the prebuilt binaries from the llmfit GitHub Releases instead.<\/p>\n\n\n\n
Getting rid of llmfit again<\/h2>\n\n\n\n
If the tool doesn’t convince you after all, the way back is short, but on Arch there’s a catch. The tool itself goes away with<\/p>\n\n\n\n
uv tool uninstall llmfit<\/code><\/pre>\n\n\n\n
and this tells you whether anything is left over:<\/p>\n\n\n\n
uv tool list<\/code><\/pre>\n\n\n\n
If you only tried llmfit with uvx, you never installed anything permanently in the first place. The downloaded packages still sit in uv’s cache, though. You empty it with<\/p>\n\n\n\n
uv cache clean<\/code><\/pre>\n\n\n\n
Now for the catch: if you installed uv through the package manager, you’re tempted to simply type<\/p>\n\n\n\n
sudo pacman -Rns uv<\/code><\/pre>\n\n\n\n
But pacman only removes its own package. What uv itself created in your home directory is unknown to the package manager, and without uv you can no longer run uv tool uninstall<\/code> either. So uninstall the tool first and uv<\/code> second, not the other way around.<\/p>\n\n\n\n
These commands show you where the leftovers are:<\/p>\n\n\n\n
uv tool dir\nuv cache dir<\/code><\/pre>\n\n\n\n
Usually that’s ~\/.local\/share\/uv\/tools<\/code> and ~\/.cache\/uv<\/code>. If uv is already gone and llmfit is still around, the only option is to clean up by hand:<\/p>\n\n\n\n
rm -r ~\/.local\/share\/uv<\/code><\/pre>\n\n\n\n
Be careful here: this also wipes out all tools installed with uv and the Python versions managed by uv, not just llmfit.<\/p>\n\n\n\n
On Ubuntu, where uv comes from the install script, there is no uv self uninstall<\/code>. There, after the uv tool uninstall<\/code>, you remove the two launcher binaries yourself:<\/p>\n\n\n\n
rm ~\/.local\/bin\/uv ~\/.local\/bin\/uvx<\/code><\/pre>\n\n\n\n
The TUI is more than a simple model list<\/h2>\n\n\n\n
The terminal interface is one of the more interesting parts of llmfit to me. Instead of just printing a list, it lets you move through the results, search, and pull up more information on each model.<\/p>\n\n\n\n
Use \/<\/code> to search for a specific model, the arrow keys or j<\/code> and k<\/code> to navigate, and h<\/code> opens the help. With almost 10,000 entries in the list, the search is no luxury.<\/p>\n\n\n\n
\"\"\/\n\t\t\t\n\t\t\t\t\n\t\t\t<\/svg>\n\t\t<\/button>
llmfit in the terminal on TUXEDO OS, the model list shows values between 2.9 and 65.7 in the tok\/s column, with the search running at the bottom.<\/figcaption><\/figure>\n\n\n\n
The built-in benchmark features are interesting, too. b<\/code> brings up community benchmarks, while I<\/code> starts an inference benchmark on your own hardware. That slowly turns a pure estimate into a measurement with real numbers.<\/p>\n\n\n\n
It reminds me a bit of classic system tools. If you use Mission Center<\/a> on GNOME, for example, you get CPU, RAM, GPU and other resources presented graphically. llmfit, on the other hand, looks specifically at the question of what you can do with those resources in terms of local AI.<\/p>\n\n\n\n
Estimates instead of marketing promises<\/h2>\n\n\n\n
Of course, you shouldn’t mistake llmfit’s numbers for a real benchmark. A theoretical calculation can only estimate how fast a certain model might run on a certain piece of hardware.<\/p>\n\n\n\n
The hardware simulation is quite nice in this context. You can tweak RAM, VRAM, and core count just to see how the rating shifts. If you’re toying with the idea of buying more memory or a new graphics card, you at least get a ballpark figure.<\/p>\n\n\n\n
\"\"\/\n\t\t\t\n\t\t\t\t\n\t\t\t<\/svg>\n\t\t<\/button>
llmfit with the Hardware Simulation dialog open, where you can change RAM, VRAM, and the number of CPU cores.<\/figcaption><\/figure>\n\n\n\n
That’s exactly why the benchmarks that are now built in are interesting. llmfit can download a model, start it via a supported runtime provider, and then measure the actual speed on your own hardware. The results are stored locally.<\/p>\n\n\n\n
If you like, you can then share your measurements with the community via<\/p>\n\n\n\n
llmfit bench --share<\/code><\/pre>\n\n\n\n
The results are submitted as a GitHub pull request. This way, a database of real measurements slowly builds up, instead of just theoretical calculations.<\/p>\n\n\n\n
With local AI in particular, I find this approach sensible. After all, the statement \u201cfits into VRAM\u201d doesn’t tell you much about whether you can actually work with the model in a reasonable way.<\/p>\n\n\n\n
Also usable as a command-line tool<\/h2>\n\n\n\n
If a TUI isn’t your thing, you don’t have to use llmfit interactively. The program works entirely from the command line, too.<\/p>\n\n\n\n
For example,<\/p>\n\n\n\n
llmfit recommend<\/code><\/pre>\n\n\n\n
gives you a recommendation for your hardware. With<\/p>\n\n\n\n
llmfit recommend --json<\/code><\/pre>\n\n\n\n
you can output the results as JSON. That’s obviously a lot more interesting if you want to build llmfit into your own scripts or other tools.<\/p>\n\n\n\n
You can also start a web interface or an API:<\/p>\n\n\n\n
llmfit serve --host 0.0.0.0 --port 8787<\/code><\/pre>\n\n\n\n
There is also a ready-made image for containers:<\/p>\n\n\n\n
ghcr.io\/alexsjones\/llmfit<\/code><\/pre>\n\n\n\n
That means llmfit runs on a server or in an existing Docker setup, too. If you keep your containers up to date regularly, you may already know the drill from Dockcheck<\/a>.<\/p>\n\n\n\n
And what do you run the model with?<\/h2>\n\n\n\n
One distinction you should keep in mind: llmfit is not an AI runtime itself. The program helps you choose and evaluate a model, while other tools do the actual inference.<\/p>\n\n\n\n
\"\"\/\n\t\t\t\n\t\t\t\t\n\t\t\t<\/svg>\n\t\t<\/button>
Detail view in llmfit for DeepSeek-R1-Distill-Qwen-7B with a score breakdown, memory requirements per quantization level, and a GGUF download hint from bartowski.<\/figcaption><\/figure>\n\n\n\n
Supported tools include Ollama, llama.cpp, MLX, Docker Model Runner, and LM Studio. That means llmfit can, for example, benchmark a model through an existing Ollama or llama.cpp stack.<\/p>\n\n\n\n
On Linux, Ollama<\/a> is certainly one of the easier ways to get local models up and running. If you want more control over the inference itself, you quickly end up with llama.cpp<\/a>. The two projects take somewhat different approaches, but both combine well with llmfit’s hardware check.<\/p>\n\n\n\n
Interesting for local AI<\/h2>\n\n\n\n
To me, llmfit closes a gap between \u201cI’d like to try local AI\u201d and \u201cI first have to work my way through the hardware requirements of dozens of models.\u201d That’s far more helpful than a generic list of the \u201cbest\u201d local models. In the end, what matters isn’t which model scores well on some test system, but which one runs properly on your hardware for your use case.<\/p>\n\n\n\n
Of course, the estimates don’t replace your own testing. That’s exactly what the built-in benchmarks are for, and they’re what I want to tackle next: my own measurements on different machines, and then a comparison of how well llmfit’s predictions match reality.<\/p>\n\n\n\n
The tool should be especially interesting for developers. If you want to use a local AI assistant for programming, you first find out which models fit your hardware, and then experiment with the speeds you can actually reach.<\/p>\n\n\n\n
You’ll find more information, the current documentation, and the source code on GitHub<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"
Want to run a local AI but not sure which model really fits your hardware? llmfit reads out your specs, scores thousands of options, and estimates speed and VRAM requirements. This open-source tool takes the guesswork out and gives you concrete recommendations. In short: no more agonizing over quantization, you just know what runs.<\/p>\n","protected":false},"author":2,"featured_media":8,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"lui_source_id":47801,"lui_source_hash":"06857770d0924aeb2d8c4199a6f7464eac1949af9285f2d54015b39bd02783d4","lui_source_translated":"2026-10-03","lui_source_reviewed":true,"lui_via_url":"","lui_via_label":"","lui_source_url":"","lui_source_label":"","footnotes":""},"categories":[2],"tags":[],"class_list":["post-7","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-gnu-linux"],"_links":{"self":[{"href":"https:\/\/linuxundich.de\/en\/wp-json\/wp\/v2\/posts\/7","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/linuxundich.de\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/linuxundich.de\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/linuxundich.de\/en\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/linuxundich.de\/en\/wp-json\/wp\/v2\/comments?post=7"}],"version-history":[{"count":1,"href":"https:\/\/linuxundich.de\/en\/wp-json\/wp\/v2\/posts\/7\/revisions"}],"predecessor-version":[{"id":9,"href":"https:\/\/linuxundich.de\/en\/wp-json\/wp\/v2\/posts\/7\/revisions\/9"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/linuxundich.de\/en\/wp-json\/wp\/v2\/media\/8"}],"wp:attachment":[{"href":"https:\/\/linuxundich.de\/en\/wp-json\/wp\/v2\/media?parent=7"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/linuxundich.de\/en\/wp-json\/wp\/v2\/categories?post=7"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/linuxundich.de\/en\/wp-json\/wp\/v2\/tags?post=7"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}