LLM-based Context Precision
Definition
Context Precision is used to measure information density.
Example Usage
Required data items: question
, retrieved_context
from continuous_eval.metrics.retrieval import LLMBasedContextPrecisionfrom continuous_eval.llm_factory import LLMFactory
datum = { "question": "What is the capital of France?", "retrieved_context": [ "Paris is the capital of France and also the largest city in the country.", "Lyon is a major city in France.", ],}
metric = LLMBasedContextPrecision(LLMFactory("gpt-4-1106-preview"), log_relevance_by_context=True)print(metric(**datum))
Note: optional variable log_relevance_by_context
outputs LLM_based_context_relevance_by_context
- the LLM judgement of relevance to answer the question per context retrieved.
Sample Output
{ 'LLM_based_context_precision': 0.5, 'LLM_based_context_average_precision': 1.0, 'LLM_based_context_relevance_by_context': [True, False]}