Skip to content

Python AST Similarity

Definitions

Python AST Similarity compares the structure of two Python programs (generated code string vs. ground truth code string) by analyzing their Abstract Syntax Trees (ASTs). It evaluates how similar these programs are by matching nodes in the trees, considering both the types of statements and their organization. The comparison can involve reordering certain parts for a deeper match and uses a scoring system to quantify similarity.



Example Usage

Required data items: answer, ground_truth_answers

from continuous_eval.metrics.code import PythonASTSimilarity
datum = {
"answer": "def function(x, y):\n return x + y",
"ground_truth_answers": [
"def foo(x, y):\n return x * y",
"def foo(x, y):\n return x + y",
],
}
metric = PythonASTSimilarity()
print(metric(**datum))

Example Output

{
"Python_AST_Similarity": 1.0
}