Self-hosted AI assistant

Architecture of the assistant: Telegram in, an async Python core with routing and memory, and local models for chat, vision and speech

What it is

This is the assistant I talk to from my phone. It lives on a PC at home, answers through Telegram, and everything — the language model, image understanding, speech recognition and synthesis, memory — runs on that machine. There is no cloud API behind it and no monthly bill. It also doubles as the remote control for that PC: service status, disk usage, running processes, downloads, reboots.

Why local

Three reasons, in order of importance:

Privacy. It knows my calendar, my notes and my home network. That does not belong on someone else’s server.
Cost. A 3B-parameter model on a consumer GPU answers in a couple of seconds and costs nothing per message. That changes how freely you use it.
Learning. Running models yourself teaches you what a hosted API hides: context limits, VRAM, latency, when a small model is enough and when it is not. The duplicate-detection pipeline I built at work on a local LLM came straight out of what I learned here.

How it works

Core: async Python. Every model call is awaited so a slow answer never blocks a voice note or a status command.
Chat: qwen2.5 served by Ollama, kept resident in memory (keep_alive) so the first token does not pay a load penalty. The prompt is built from a system profile, the last conversation turns and a language tag.
Language: real detection with lingua (langdetect confused “Hola” with Turkish), short messages inherit the language of the previous turn, and an /undo command fixes a wrong guess.
Vision: photos go to moondream, a small local vision model, for descriptions and simple questions about an image.
Voice: Vosk transcribes Spanish and English offline; Piper synthesises the reply. Voice notes are routed by intent — “task: buy bread” creates a task, “search …” hits the web, everything else is chat.
Memory: a background extractor reads each conversation and proposes facts about the user; confirmed facts are stored in SQLite and injected into later prompts. The history migrated from a JSON file to SQLite with a verification script that compared every count.
Tools: DuckDuckGo search, tasks and reminders, file search, flight lookup, service management and a /status command that reports VRAM, loaded models and GPU temperature.
Operations: two systemd services with a watchdog, structured logging, timeouts on every model call, nightly backups of the database.

What I learned

Small models need tight prompts. A 3B model follows a short, explicit system prompt far better than a long one; structure beats verbosity.
Rules first, model second. Intent routing by keywords is faster and more predictable than asking the model what the user meant. The LLM handles what the rules cannot.
Verify migrations like data work. Moving memory from JSON to SQLite got the same treatment as a data migration at work: counts compared on both sides before switching over.
Contamination is real. A few English answers to Spanish messages ended up in the history and biased later replies. Now the history is purged of mixed-language pairs and the language tag is explicit.

Status

In daily use. Next: replace keyword routing with a small classifier, and a proper evaluation set for answers so model upgrades can be measured instead of felt.