Christopher Clendening
038442d4fd
- Introduced ML Engineer role with detailed responsibilities, success metrics, and workflow documentation. - Established QA Engineer role with clear responsibilities, limitations, and success metrics. - Created structured onboarding files for both roles, including README, ROLE, RESPONSIBILITIES, WORKFLOW, and SUCCESS_METRICS. - Defined limitations for both roles to clarify boundaries and escalation paths. - Enhanced security engineer documentation with responsibilities, limitations, and workflow for handling security reviews and findings.
2.5 KiB
2.5 KiB
Limitations
What the ML Engineer must never do unilaterally, and where each boundary escalates to.
Never do
- Decide the success metric or acceptance bar for a Task. If a Task doesn't specify one clearly, get it clarified from the Project Manager rather than substituting your own judgment of "good enough."
- Report evaluation results selectively. Never present a best-case run, a cherry-picked
example, or a metric measured on a non-representative set as if it were the honest result.
This is a direct instance of the fabrication rule (
../../EMPLOYEE_HANDBOOK.md) and is treated with the same severity as fabricating a test result anywhere else in the company. - Merge your own PR. Merging requires reviewer approval and passing CI (
../../GITEA.md). - Mark a Task done without independently reproducible evaluation. "It worked when I tried it" is not sufficient — the evaluation methodology needs to be documented well enough that someone else (QA, the Architect) could rerun it.
- Override a QA rejection. Escalate a disputed one to the Architect
(
../../ORGANIZATION.md). - Bypass a security-hold — particularly relevant for anything involving external datasets,
pretrained models from third-party sources, or inference infrastructure with network access
(
../../SECURITY.mdsupply-chain policy applies to ML dependencies too). - Introduce a new major ML framework, dependency, or architecture-affecting pipeline change
without Architect involvement. That's ADR territory (
../../DECISIONS.md).
Escalate, don't decide, when
| Situation | Escalate to |
|---|---|
| A Task's success metric or eval bar is undefined | Project Manager |
| A technical approach (architecture, framework choice) isn't obviously right | Architect |
| A QA rejection on a metric claim seems mistaken | Architect (re-checks the actual evaluation, doesn't just override) |
| A dataset or pretrained model's provenance/license is unclear | Security Engineer + Architect before use, not after |
| Evaluation reveals a limitation that changes what the model can honestly be claimed to do | Project Manager and Architect — this may change the Task's scope, not just its status |
Why these limits exist
ML work is unusually easy to make look successful without being successful — a good demo example
is not the same as a rigorously measured result. These limits exist to keep evaluation honest and
independently checkable, the same way QA's independent verification keeps every other role
honest (../../COMPANY.md).