A coding model should be evaluated on tasks that resemble your repository's work. Sorting three numbers or producing a plausible explanation does not establish that the model can safely modify authentication, migrations or payment state.
Build a small benchmark from real work
Select resolved issues or representative tasks with a known expected behavior. Remove private credentials and unnecessary customer data. Give each model the same allowed context and tools. Keep acceptance tests out of the prompt when they are intended to reveal whether the solution genuinely handles edge cases.
| Task | Acceptance evidence | Review concern |
|---|---|---|
| Bug fix | Original failure and regression cases pass | Unrelated behavior changed |
| API change | Contract and error handling remain correct | Invented endpoint or SDK method |
| Database change | Migration and rollback verified | Historical data corruption |
| UI change | User can complete the task | Visual success hides a broken action |
Validate behavior rather than confidence
For input validation, decide what the input means before checking it. A string containing digits is still a string; rejecting it with “must be a string” confuses a type requirement with a content rule. Similar ambiguity in a model-generated patch can silently reject valid business data.
GitHub responsible use of Copilot code completion identifies risks of incorrect or fabricated code suggestions and the need for human review. Tests should include meaningful boundaries, not just repeat the implementation. Check the generated dependencies against their actual documentation and version.
Measure the whole contribution
Record successful tasks, time to reviewed acceptance, manual fixes and regressions. A faster first response can still create more review work. Repeat important tasks where sampling variability could change the result, while keeping the evaluation budget bounded.
How should selection affect daily use?
Use evaluation results to assign tasks and controls, then recheck after model or tooling changes. Keep an engineer accountable for the final diff. For AI integration, scope a repository-specific pilot instead of a universal “best coding model” claim.