Auto Prompt Optimization

If you have a good golden dataset of inputs and their expected outputs, you can use it to automatically optimize your prompts to capture nuanced requirements. This process can save you many hours of manual trial and error.

Here we will walk through the steps to optimize your prompts:

Step 1. Navigate to Prompt Optimization Tab

First, navigate to the prompt optimization tab and click "Optimize New Prompt". This will open the interface for setting up your prompt optimization.

Step 2. Select a Dataset, initial prompt and parameters

In this step, you'll need to provide key information to start the optimization process.

Dataset

Select the dataset that contains your golden inputs and expected outputs. This dataset will be used to evaluate the effectiveness of your prompts.

Metric

Choose the metric that will be used to measure the performance of your prompts. Currently you can only select one metric to optimize your prompt for. However you can uses different metrics in multiple runs to optimize different aspects of the prompt through.

Initial Prompt

Provide the initial system prompt and user prompt. This serves as the starting point for optimization. The system prompt sets the context, while the user prompt should include all the variables. Note that the system prompt will be evolved to include all the instructions, where the user prompt will not be changed since it only provides the input variables.

tip

The task description and user prompt description is critical to help the optimizer understand the task and the inputs.

Step 3. View Optimization Results

The result will contain different prompt versions that were attempted by the optimizer and their corresponding performance. The new prompts will include the system prompt and the optimized few shot examples. The user prompt (input variables) remains fixed.