Multi-model Coding Agent Harness — 選型與設計卡

When to use

• You want Claude Code / Codex / Cursor-class UX but must swap models without rewriting the workflow.
• You are escaping vendor lock-in (switch cost, capability gaps, price swings).
• You need a decision memo before buying seats or wiring API keys.

Problem framing

Whole-repo lock-in (one vendor owns tools + memory + UI) vs per-task routing (same tool/MCP/permission/memory layer; pluggable providers underneath).

Public signal (abstracted): builders wish for a harness “as good as Claude/Codex/Cursor” that still swaps all models — some already hack ugly scripts to do it.

Target outcomes

• One tool / MCP / permission / memory surface; adapters for Claude / Codex / Cursor-class / open weights.
• Observable: cost, latency, fail rate, human-takeover rate.
• Acceptance: same task can change provider without rewriting prompts or tool defs; failures degrade with a reason log.

Score harnesses (1–5)

Architecture checklist

• Unified interface: tools, MCP, fs/shell permissions, memory/transcript
• Provider adapter contract: input/output, streaming, abort
• Routing surface: task type, file type, budget, SLA
• Audit log + spend cap + max turns
• Golden task suite (3 issues) for bake-offs

Routing rule templates

Example YAML:

routes:
  - when: { task: triage }
    model: cheap
  - when: { file_glob: "**/*auth*" }
    model: strong_local
  - when: { task: hard_refactor }
    model: strong
fallback: strong
on_fail: degrade_to_cheap_and_log

Also support per-file, per-PR, per-task-file tables with the same columns.

Migration playbook

1. Inventory current harness (tools, secrets, memory).
2. Extract adapter boundary; keep UX.
3. Dual-run primary vs backup on golden suite.
4. Cut traffic gradually; keep escape hatch 2 weeks.

Recommended shapes

1. Vendor-first + escape hatch — lowest friction.
2. Router harness — best with tests / issue bots.
3. Dual-lane — planner (strong) + applier (cheap) if patches are auto-gradable.

Anti-patterns

• Secrets hardcoded per provider
• No fallback / no audit trail
• Abstraction that kills observability
• Pay-to-play “universal harness” before hello-world
• Impersonating a human on ToS-sensitive surfaces

Bake-off script

1. Pick 3 real issues (easy → hard).
2. Run each harness; record time-to-PR, interventions, $ estimate.
3. Re-run hardest on backup model only.
4. Keep winner + backup for 2 weeks; document failures.

Output template

Job:
Candidates + scores (6 axes):
Routing mode:
Winner + backup:
Spend / seats:
Risks:
Next experiment (≤1 week):

Out of scope

Full harness implementation; enterprise contract negotiation; paid signup without operator approval.