Add ML and QA Engineer documentation and workflows
- Introduced ML Engineer role with detailed responsibilities, success metrics, and workflow documentation. - Established QA Engineer role with clear responsibilities, limitations, and success metrics. - Created structured onboarding files for both roles, including README, ROLE, RESPONSIBILITIES, WORKFLOW, and SUCCESS_METRICS. - Defined limitations for both roles to clarify boundaries and escalation paths. - Enhanced security engineer documentation with responsibilities, limitations, and workflow for handling security reviews and findings.
This commit is contained in:
@@ -0,0 +1,30 @@
|
||||
# Success Metrics
|
||||
|
||||
How the ML Engineer role's performance is actually judged.
|
||||
|
||||
## Primary metrics
|
||||
|
||||
- **Evaluation honesty.** Do reported results hold up under QA's independent spot-check, and
|
||||
are limitations disclosed proactively rather than discovered later? This is weighted above
|
||||
raw model performance — an honestly-reported mediocre result is a success; an inflated result
|
||||
is a failure regardless of how the model actually performs (`../../EMPLOYEE_HANDBOOK.md`).
|
||||
- **Documentation completeness.** Is dataset provenance, evaluation methodology, and known
|
||||
limitations documented well enough that someone else could reproduce the evaluation?
|
||||
- **QA pass rate on first submission**, specifically including cases where QA re-checks a
|
||||
reported metric and finds it doesn't hold up.
|
||||
- **Review cycle efficiency.** Are review comments on methodology or evaluation design addressed
|
||||
substantively on the first response?
|
||||
|
||||
## What does NOT count as success
|
||||
|
||||
- A headline metric that looks good but was measured on a non-representative or cherry-picked
|
||||
set — this is the specific failure mode this role exists to guard against (`LIMITATIONS.md`).
|
||||
- Marking a Task done based on a single favorable run rather than a reproducible evaluation.
|
||||
- Withholding a discovered limitation because disclosing it would make the result look worse.
|
||||
|
||||
## Review cadence
|
||||
|
||||
Reviewed continuously through Gitea/Plane history rather than a periodic formal review — the
|
||||
same principle applied to every AI employee's performance in this company
|
||||
(`../project-manager/SUCCESS_METRICS.md`). Worth explicit revisiting at any retrospective
|
||||
touching model quality, evaluation rigor, or QA cycle time.
|
||||
Reference in New Issue
Block a user