Core Concepts

Here are the entities that we use to organize in Relari's platform:

• Organization
  The top-level entity in Relari. It is a container for projects, datasets, experiments, prompts, and metrics.

  • Project
    A project is a container for datasets, experiments, prompts, and metrics. It is a way to organize your work in Relari.

    • Dataset
      A dataset is a collection of examples that includes the Input and (optionally) Expected Outputs of an LLM system.

      • User-Provided Datasets
        User-uploaded datasets for evaluation and optimization.

      • Synthetic Datasets
        Synthetically generated datasets that contains the Inputs and Expected Outputs.

    • Experiment (Evaluation)
      An experiment measures the performance of a LLM system (or part of an LLM system) on a dataset.

    • Prompt Optimization
      A prompt optimization run leverages a Dataset and Metrics to automatically improve the prompt from an initial user-provided version.

    • Runtime Monitor
      Runtime monitors track the performance of a LLM system in production.

  • Metrics
    Metrics are used to evaluate and/or optimize the performance of a LLM system.

    • Standard Metrics
      Standard off-the-shelf metrics that covers a variety of LLM use cases.

    • Custom Metrics
      Custom metrics that capture use-case specific rubric and user preferences. Can be data optimized.