Python AST Similarity

Definitions

Python AST Similarity compares the structure of two Python programs (generated code string vs. ground truth code string) by analyzing their Abstract Syntax Trees (ASTs). It evaluates how similar these programs are by matching nodes in the trees, considering both the types of statements and their organization. The comparison can involve reordering certain parts for a deeper match and uses a scoring system to quantify similarity.

Note

The metric depends on syntactically correct Python scripts to produce the Abstract Syntax Trees (ASTs). If the scripts contain syntax errors and cannot be parsed, the metric will yield a score of -1.0.

Example Usage

Required data items: answer, ground_truth_answers

from continuous_eval.metrics.code.python import PythonASTSimilarity

datum = {
    "answer": "def function(x, y):\n  return x + y",
    "ground_truth_answers": [
        "def foo(x, y):\n  return x * y",
        "def foo(x, y):\n  return x + y",
    ],
}

metric = PythonASTSimilarity()
print(metric(**datum))

Example Output

{
    "Python_AST_Similarity": 1.0
}