On-premise AI Assistant — Legal Platform

From February to September 2026, I worked as a freelance AI Engineer for a legal case management platform (confidential client), within a team of 5: a tech lead, 2 developers and 2 project managers. My engagement: design the platform's AI assistant and put it into production. Constraints: • 100% on-premise deployment: no data leaves the client's infrastructure • CPU-only server (16 cores, no GPU) • Integration into an existing Symfony back end • Sensitive legal data: access control and security as top priorities

The assistant relies on a FastAPI gateway integrated into the platform's Symfony back end. Tech stack: • LLM: Ollama with Qwen models, chosen after a benchmark under CPU constraints • Hybrid search: Qdrant (vector) + Elasticsearch (lexical) • Asynchronous processing: Celery + Redis • SSE streaming responses • Monitoring: Prometheus / Grafana Quality and security: • Prompt injection guardrails • Role-based access control • Circuit breaker to isolate downstream service failures

Each question first goes through a deterministic pre-router, then through LLM routing when needed, to specialized agents: • Document agent — analysis of case documents • Statistics agent — multi-tenant indicators, isolated per platform customer • Assistant agent — general questions about the platform Document agent: • Asynchronous pipeline with recovery after failures • Structural chunking that follows the documents' organization • Map-reduce synthesis into typed outputs • Every piece of information is sourced to the page

Technical choices were validated through measurement rather than intuition. Evaluation approach: • Golden set of 30 questions representative of real usage • Benchmark of 3 LLMs to choose the model Results: • The deterministic pre-router picks the right agent in 85% of cases, versus 52% with the LLM alone • The golden set score went from 77% to 93% • More than 1,000 tests, 87% code coverage