Skip to content

Code String Match

Definitions

Code String Match measures how close the generated code string is to the ground truth code string.

It outputs both the binary exact match score and the fuzzy match score in the range of (0.0 - 1.0).


Example Usage

Required data items: answer, ground_truth_answers

from continuous_eval.metrics.code import CodeStringMatch
datum = {
"answer": "def function(x, y):\n return x + y",
"ground_truth_answers": [
"def foo(x, y):\n return x * y",
"def foo(x, y):\n return x + y",
],
},
metric = CodeStringMatch()
print(metric(**datum))

Example Output

{
"Exact_Match_Score": 0,
"Fuzzy_Match_Score": 0.89
}