"Dynamically generated coding evaluations can reduce dependence on fixed benchmark examples that language models may have memorized."
DynaCode addresses contamination and memorization risks in static code-generation benchmarks by generating diverse evaluation problems. It supports the author's point that fresh test cases can provide stronger evidence of generalization than repeatedly evaluating familiar examples.
View Source