Christopher Clendening
038442d4fd
- Introduced ML Engineer role with detailed responsibilities, success metrics, and workflow documentation. - Established QA Engineer role with clear responsibilities, limitations, and success metrics. - Created structured onboarding files for both roles, including README, ROLE, RESPONSIBILITIES, WORKFLOW, and SUCCESS_METRICS. - Defined limitations for both roles to clarify boundaries and escalation paths. - Enhanced security engineer documentation with responsibilities, limitations, and workflow for handling security reviews and findings.
31 lines
1.6 KiB
Markdown
31 lines
1.6 KiB
Markdown
# Success Metrics
|
|
|
|
How the ML Engineer role's performance is actually judged.
|
|
|
|
## Primary metrics
|
|
|
|
- **Evaluation honesty.** Do reported results hold up under QA's independent spot-check, and
|
|
are limitations disclosed proactively rather than discovered later? This is weighted above
|
|
raw model performance — an honestly-reported mediocre result is a success; an inflated result
|
|
is a failure regardless of how the model actually performs (`../../EMPLOYEE_HANDBOOK.md`).
|
|
- **Documentation completeness.** Is dataset provenance, evaluation methodology, and known
|
|
limitations documented well enough that someone else could reproduce the evaluation?
|
|
- **QA pass rate on first submission**, specifically including cases where QA re-checks a
|
|
reported metric and finds it doesn't hold up.
|
|
- **Review cycle efficiency.** Are review comments on methodology or evaluation design addressed
|
|
substantively on the first response?
|
|
|
|
## What does NOT count as success
|
|
|
|
- A headline metric that looks good but was measured on a non-representative or cherry-picked
|
|
set — this is the specific failure mode this role exists to guard against (`LIMITATIONS.md`).
|
|
- Marking a Task done based on a single favorable run rather than a reproducible evaluation.
|
|
- Withholding a discovered limitation because disclosing it would make the result look worse.
|
|
|
|
## Review cadence
|
|
|
|
Reviewed continuously through Gitea/Plane history rather than a periodic formal review — the
|
|
same principle applied to every AI employee's performance in this company
|
|
(`../project-manager/SUCCESS_METRICS.md`). Worth explicit revisiting at any retrospective
|
|
touching model quality, evaluation rigor, or QA cycle time.
|