A private AI assistant that runs with no GPU and no internet
We gave a government agriculture portal an AI assistant that answers farmers' questions from official scheme documents. It runs entirely on the portal's own locked-down server.
Farmers needed clear answers about government schemes: who qualifies, what share they pay, which documents to submit. Those answers were buried in long PDFs full of tables and abbreviations. The usual AI toolkit wasn't an option, because the server had no GPU, no Python, no Docker and no internet access after setup.
We built the whole assistant in Node.js. A compact open-source language model runs locally on the CPU. Search combines meaning-based matching with exact keyword matching, so scheme acronyms aren't missed. A reranking step then picks the most relevant passages, and strict prompting keeps the model answering only from the documents. All of it fits inside the existing app, starts automatically on reboot, and uses a fraction of the server's memory.
Node.js · llama.cpp · Qwen2.5-3B · Qdrant · BGE embeddings & reranker · MiniSearch (BM25) · Angular
Read the engineering write-up ↗ by Sidhant Tiwari