We Built Our Own ChatGPT style inference on a single desktop GPU
If you've ever pasted a chunk of code into a public AI chatbot to fix an error, you know how useful it is. But have you ever stopped to think about where that code actually goes?
That was one of the questions the team at Whileone kept coming back to. The honest answer is: to a server somewhere, owned by someone else. For a cookie recipe that's fine. For the code we write for clients, the team felt it wasn't good enough.
So, we decided to run its own AI model, on its own hardware, inside the office. I was lucky enough to work on this project and see it go from an idea to something my colleagues use every day. This is my attempt to explain what we did, why we did it, and the things that went wrong along the way (there were a few), in a way that makes sense even if you're not an AI engineer.
Why would a company run its own AI?
The big companies already do this. Banks, hospitals and large tech firms run their own large language models (LLMs) on their own infrastructure, mostly because they don't want their data leaving the building. The question at WhileOne was whether a mid-sized engineering company could do the same, without a data centre or a huge budget.
When the team listed the reasons, they were pretty simple:
Privacy - Prompts and answers stay on our own network. If an engineer pastes a client's config file to debug it, that file never leaves the office.
Control - We decide which model runs, how it behaves and who can use it.
Cost - No per-token bill that grows every month. The hardware is paid for once.
Freedom. Open models get better every few months, and we can switch whenever a better one comes out.
There was one more reason I really liked: the company wanted to understand how this technology works under the hood, not just use it through someone else's website. That meant a lot of learning for me.
Choosing a model
An LLM is basically a giant pile of numbers called parameters, which the model learned by reading huge amounts of text. More parameters usually mean a smarter model, but it also needs more memory on the graphics card (GPU) that runs it.
Whileone deliberately started small. One desktop server, one NVIDIA RTX 4060 with 8 GB of memory. That's closer to a gaming card than a data centre card, so the really big models were out straight away. A 27-billion-parameter model, for example, needs a card with around 48 GB.
After comparing a few options, the team went with Qwen3.5-9B, an open model from Alibaba's Qwen team. It's released under the Apache 2.0 licence, so it can be used commercially, and for a 9-billion-parameter model it does surprisingly well on reasoning and coding benchmarks. On paper it even beats some open models that are many times bigger.
Fitting it onto an 8 GB card
This was the first real task. In its normal form the model needs about 18 GB of memory. We had 8.
The answer is something called quantization. The easiest way I can explain it: it's like saving a photo as a smaller JPEG. Each number is stored with less precision (4 bits instead of 16), and the model shrinks to about 5.3 GB. A tiny bit of quality is lost, but in everyday use the difference is hard to notice.
Two more settings helped. While the model is answering, it keeps a short-term memory of each conversation, called the KV cache, and the team stores that at 8-bit to halve its size. We also turned on flash attention, a more memory-efficient way of doing the model's main calculation. Together, that left enough room for many conversations at the same time.
What went wrong
The first problem showed up as soon as more than one person used it. The team had started with a popular, very easy-to-use tool for running models, and with one user it was great. Then we ran a load test that simulated lots of colleagues asking questions at once. With 24 simulated users, people were waiting almost 49 seconds before the first word of an answer appeared. Nobody is going to use a chatbot like that.
It turned out the tool deliberately handles only one request at a time for this particular model family. So, we switched the serving engine to llama.cpp, which uses something called continuous batching. The way we understood it: instead of a cook finishing one order before starting the next, it cooks several orders on the same stove at once. With 12 people asking at the same time, the wait went from about 24 seconds to about 1 second. That was a good day in the office.
The second problem was sneakier. We wanted developers to be able to use the model from their terminal like Claude Code, an AI coding agent. What none of us expected was how much text a coding agent sends with every request: around 16,000 to 18,000 tokens of instructions before you've even asked anything. Our limit was 32,000 tokens, so the agent kept running out of room and summarising its own memory. In one test it reported that it had fixed a bug, but when we opened the file, nothing had changed. That was a bit alarming.
We also had to patch the model's chat template (basically the formatting rules for conversations), because it didn't like the way the agent arranges its messages. In the end the team raised the limit to 48,000 tokens per request and accepted serving 12 people at once instead of 16. After that, the agent found and fixed a test bug in about 20 seconds, and this time the file actually changed.
The third problem was the simplest and the most annoying. One morning the assistant just wouldn't start. Another service on the same machine had loaded its own model onto the GPU and taken most of the memory, and on an 8 GB card there's no room for two. The fix was easy once we found it, but the lesson stuck: give the model its own GPU, and keep an eye on it.
What we ended up with
Everything runs as containers on that one server. The main pieces:
llama.cpp serving the model, with the whole model on the GPU.
12 requests handled at the same time, sharing a pool of 96,000 tokens of conversation memory, with up to 48,000 tokens in and 16,000 out per request.
Sampling set to temperature 0.6, top-p 0.95 and top-k 20, which is what the Qwen team recommends for accurate coding answers.
A "thinking" mode where the model reasons before it answers. People can switch it off when they just want a quick reply.
Around that, the team added a ChatGPT-style web interface (Open WebUI), where colleagues sign up and an admin approves them. There's also an API gateway (LiteLLM) that gives each person their own key and keeps track of how many tokens they use, and a small open-source monitoring dashboard called Beszel that shows the GPU, memory and disk at a glance.
How my colleagues actually use it
This is the part I enjoy most. Someone gets a confusing error, pastes the stack trace into the web chat and gets an explanation in a few seconds. Developers type one command in their terminal and let the coding agent read through the project and fix things. People use it to summarise long documents before meetings. One of our older internal tools needed to keep talking to the model the way it always had, so the team wrote a small compatibility layer, and it carried on working without a single code change.
So far 35 colleagues have been onboarded. A single user gets around 44 tokens per second, which is faster than I can read. And the number the whole team likes most: zero prompts have left our network.
Being honest about the limits
A 9B model is not going to replace the biggest AI models out there. It's great for everyday coding help, explanations, scripts and drafting. For long, complicated, multi-step tasks, a bigger model does noticeably better. The obvious next step for WhileOne is a 48 GB GPU, which would let us run a 27-billion-parameter model and serve 30 or more people at once.
Where WhileOne is taking this
Building this in-house showed the company something. A lot of businesses want AI, but they're not comfortable sending their data to someone else's cloud. So WhileOne is going to offer this as a service: helping companies pick the right model and hardware, set it up, secure it and keep it running, using everything we learned here, mistakes included.
Big companies run their own AI. After seeing this project up close, I don't see why everyone else shouldn't.
If your company has tried this, or is thinking about it, I'd love to hear how it went or what's stopping you. Drop a comment or send me a message.
Comments