OpenAI wants to retire the leading AI coding benchmark—and the reasons reveal a deeper problem with how the whole industry measures itself.

CATArena (Code Agent Tournament Arena) is an open-ended environment where LLMs write executable code agents to battle each other and then learn from each other. CATArena is an engineering-level ...

Abstract: One of the key challenges in code translation is converting code from one language to another while maintaining functionality, design accuracy, and performance. Converting mobile ...

Use the vitals package with ellmer to evaluate and compare the accuracy of LLMs, including writing evals to test local models ...