> git log --author="abdel" --oneline

Open Source: RAGnarok-AI & Contributions

Open source products and community contributions. Because the best code is the one you share.

> ls ~/products/opensource

RAGnarok-AI

productionAGPL-3.0v1.11

Local-first RAG evaluation framework with a CI/CD GitHub Action. 20 adapters, Medical Mode, ~24K queries/sec.

153Python 3.10+

View on GitHub Documentation GitHub Action Try on Colab

Evaluating a RAG system shouldn't require sending your data to the cloud. RAGnarok-AI is a local-first evaluation framework that measures your RAG pipelines and AI agents quality — entirely on your machine. LLM-as-Judge with Prometheus 2, cost tracking, checkpointing, and native Jupyter integration.

"RAG evaluation shouldn't depend on cloud APIs. I built RAGnarok-AI so every developer can measure their RAG quality — on their machine, with no compromises on privacy."

Installation

$ pip install ragnarok-ai

pypi.org/project/ragnarok-ai

Features

Retrieval metrics (Precision@K, Recall, MRR, NDCG), generation metrics (faithfulness, relevance, hallucination), agent evaluation (tool-use, trajectory). Multi-provider cost tracking, plugin system, rich HTML visualization in Jupyter. Compatible with Ollama, OpenAI, Anthropic, vLLM.

Architecture

Modular architecture: Core (types, protocols), Evaluators (metrics), Adapters (LLMs, vector stores, frameworks), Reporters (JSON, HTML, CLI). Supports LangChain, LangGraph, LlamaIndex, DSPy. Built-in caching, async batching, checkpointing.

Version History

v0.1Foundation — Retrieval metrics, CLI, reporters

v0.2Generation Metrics — Faithfulness, relevance, hallucination

v0.3Test Generation — Synthetic questions, multi-hop

v0.4Framework Adapters — LangChain, observability

v0.5Performance — Async, parallelization, concurrency

v0.6Cloud & Local — OpenAI, Anthropic, vLLM, FAISS

v0.7More Frameworks — LlamaIndex, DSPy

v0.8Benchmarking — Comparison, regression

v0.9Agent Evaluation — Tool-use, trajectory

v1.0Production Ready — PyPI, documentation

v1.1Jupyter Integration — Rich HTML visualization

v1.2Cost Tracking — Multi-provider cost monitoring

v1.3Plugin System — Extensible architecture, Prometheus 2

v1.4GitHub Action — Native CI/CD, Medical Mode, 9 adapters, ~24K q/s

v1.5Enterprise Deployment — Kubernetes Helm chart, air-gapped deployment

v1.6Production Monitoring — MonitorClient, SQLite storage, Prometheus export

v1.7Alerting — Threshold rules, Webhook and Slack alerts

v1.8Medical Disambiguation — SLM-based abbreviation disambiguation

v1.9Production Intelligence — Drift detection, A/B testing, root-cause diagnosis

v1.10Trustworthy Evaluation — Production-mined testsets, judge calibration

v1.11Developer Experience — Evaluate your pipeline from the CLI, live streaming, results browserlatest

Comparison

	RAGnarok-AI	Giskard	RAGAS
100% Offline	✓	~	✗
Evaluation Resume	✓	✗	✗
Speed	✓	✗	✓
Built-in CLI	✓	✗	✗
GitHub Action CI/CD	✓	✗	✗
Medical Mode	✓	✗	✗

Features

adapters

350+

medical abbreviations

100%

local-first

CI/CD

GitHub Action

Performance

(Apple M2 16GB)

Metric	Value
Retrieval metrics	~24,000 q/s
LLM-as-Judge	~25s / eval
Peak RAM (5000q)	0.17 MB

What's Next

RAGnarok-AI is a living project, constantly evolving. Here's what's coming:

Monitoring dashboardMulti-agent metricsMLflow integration

RAGLLMGitHub ActionCI/CDMedical ModeOllamaLangChainHaystack

See more on GitHub

github.com/2501Pr0ject