Prompt Ops & LLM Evals
Golden datasets, evals, and CI gates so prompt changes ship without regressions.
The eval and observability layer your AI features need: golden datasets, rubric scoring, Braintrust/Langfuse, prompt registry, CI regression gates.
Problems I solve
- Prompt changes silently regress quality
- No way to compare model versions safely
- AI cost per customer is a mystery
- Support tickets from AI hallucinations
What you get
- Golden datasets and rubric-based scoring
- Braintrust / Langfuse instrumentation
- Prompt + model version registry
- CI gate that blocks regressions before deploy
Use cases
Eval harness bootstrap
Stand up evals + golden sets for 3–5 core AI features in one engagement.
Prompt registry + CI gate
Every prompt / model change runs against evals before merge.
Per-tenant cost dashboards
Attribute LLM cost + quality per customer, feature, and model.
Online eval + alerting
Sampled production traces scored continuously with regression alerts.
Examples I've shipped
SaaS eval stack
Outcome — Blocked 4 silent regressions in the first month.
Cost attribution
Outcome — Found and fixed a $9k/month unit-economics leak.
How I work
- 1
Audit
Inventory prompts, models, and current failure modes.
- 2
Golden sets
Build 100–500 case datasets per feature.
- 3
Instrument
Braintrust / Langfuse in prod + CI.
- 4
Gate
CI check that must pass before any prompt / model change.
Deliverables
- Eval harness + golden datasets
- Prompt + model registry
- CI gate integration
- Cost & quality dashboards
Benefits
- Ship AI changes without fear
- Predictable unit economics
- Regressions caught before users
Frequently asked questions
Braintrust or Langfuse?+
Braintrust for eval-first teams; Langfuse if you want self-hosted + observability. I set up whichever fits your infra.
How big should a golden set be?+
Start with 100 cases covering common + edge scenarios. Grow as you find new failure modes in production.