DeBERTa Answer Scores
Definitions
DeBERTa scores measure semantic relationship between the Generated Answer and the Ground Truth Answers in three categories:

Entailment: the Generated Answer IMPLIES a Ground Truth Answer.

Contradiction: the Generated Answer CONTRADICTS a Ground Truth Answer.

Neutral: the Generated Answer and the Ground Truth Answer have neutral logical relationship.
This metric leverages the NLI DeBERTa v3 model to calculate the scores. This DeBERTa model (Decodingenhanced BERT with Disentangled Attention) is a finetuned version specifically designed to measure the above relationships.
The scores output the probability of the model’s prediction of each class (between 0 and 1). Because we are mostly interested in finding out if entailment or contradiction relationships, our scores only output those two.
Example Usage
Required data items: answer
, ground_truths
Example Output
Default: reverse = False
The above scores suggests that the model is highly confident that the Generate Answer implies at least one of the Ground Truth Answers, and that it unlikely contradicts with any of them.
Default: reverse = True
The above scores suggests that the model is highly confident that the Generate Answer implies at least one of the Ground Truth Answers, and that it unlikely contradicts with any of them.