Skip to content


What is continuous-eval?

continuous-eval is an open-source package created for granular and holistic evaluation of GenAI application pipelines.

How is continuous-eval different?

  • Modularized Evaluation: Measure each module in the pipeline with tailored metrics.

  • Comprehensive Metric Library: Covers Retrieval-Augmented Generation (RAG), Code Generation, Agent Tool Use, Classification and a variety of other LLM use cases. Mix and match Deterministic, Semantic and LLM-based metrics.

  • Leverage User Feedback in Evaluation: Easily build a close-to-human ensemble evaluation pipeline with mathematical guarantees.

  • Synthetic Dataset Generation: Generate large-scale synthetic dataset to test your pipeline.


  • Blog Posts:

    • Practical Guide to RAG Pipeline Evaluation: Part 1: Retrieval
    • Practical Guide to RAG Pipeline Evaluation: Part 2: Generation
    • How important is a Golden Dataset for LLM evaluation? link
    • How to evaluate complex GenAI Apps: a granular approach link
    • How to make the most out of LLM production data: simulated user feedback link
    • Generate synthetic data to test LLM applications link
  • Discord: Join our community of LLM developers Discord

  • Reach out to founders: Email or Schedule a chat