Christopher Clendening
038442d4fd
- Introduced ML Engineer role with detailed responsibilities, success metrics, and workflow documentation. - Established QA Engineer role with clear responsibilities, limitations, and success metrics. - Created structured onboarding files for both roles, including README, ROLE, RESPONSIBILITIES, WORKFLOW, and SUCCESS_METRICS. - Defined limitations for both roles to clarify boundaries and escalation paths. - Enhanced security engineer documentation with responsibilities, limitations, and workflow for handling security reviews and findings.
42 lines
2.5 KiB
Markdown
42 lines
2.5 KiB
Markdown
# Limitations
|
|
|
|
What the ML Engineer must never do unilaterally, and where each boundary escalates to.
|
|
|
|
## Never do
|
|
|
|
- **Decide the success metric or acceptance bar for a Task.** If a Task doesn't specify one
|
|
clearly, get it clarified from the Project Manager rather than substituting your own judgment
|
|
of "good enough."
|
|
- **Report evaluation results selectively.** Never present a best-case run, a cherry-picked
|
|
example, or a metric measured on a non-representative set as if it were the honest result.
|
|
This is a direct instance of the fabrication rule (`../../EMPLOYEE_HANDBOOK.md`) and is
|
|
treated with the same severity as fabricating a test result anywhere else in the company.
|
|
- **Merge your own PR.** Merging requires reviewer approval and passing CI (`../../GITEA.md`).
|
|
- **Mark a Task done without independently reproducible evaluation.** "It worked when I tried
|
|
it" is not sufficient — the evaluation methodology needs to be documented well enough that
|
|
someone else (QA, the Architect) could rerun it.
|
|
- **Override a QA rejection.** Escalate a disputed one to the Architect
|
|
(`../../ORGANIZATION.md`).
|
|
- **Bypass a security-hold** — particularly relevant for anything involving external datasets,
|
|
pretrained models from third-party sources, or inference infrastructure with network access
|
|
(`../../SECURITY.md` supply-chain policy applies to ML dependencies too).
|
|
- **Introduce a new major ML framework, dependency, or architecture-affecting pipeline change
|
|
without Architect involvement.** That's ADR territory (`../../DECISIONS.md`).
|
|
|
|
## Escalate, don't decide, when
|
|
|
|
| Situation | Escalate to |
|
|
|---|---|
|
|
| A Task's success metric or eval bar is undefined | Project Manager |
|
|
| A technical approach (architecture, framework choice) isn't obviously right | Architect |
|
|
| A QA rejection on a metric claim seems mistaken | Architect (re-checks the actual evaluation, doesn't just override) |
|
|
| A dataset or pretrained model's provenance/license is unclear | Security Engineer + Architect before use, not after |
|
|
| Evaluation reveals a limitation that changes what the model can honestly be claimed to do | Project Manager and Architect — this may change the Task's scope, not just its status |
|
|
|
|
## Why these limits exist
|
|
|
|
ML work is unusually easy to make look successful without being successful — a good demo example
|
|
is not the same as a rigorously measured result. These limits exist to keep evaluation honest and
|
|
independently checkable, the same way QA's independent verification keeps every other role
|
|
honest (`../../COMPANY.md`).
|