LLM Product Development
Ship AI features inside your SaaS — with evals, guardrails, and cost control.
Design and ship user-facing AI features — copilots, generators, semantic search — with eval harnesses, safety, and cost dashboards.
Problems I solve
- AI feature demoed well but broke in prod
- Costs unpredictable, no per-customer attribution
- No way to tell if a prompt change made things worse
- Safety incidents from ungrounded outputs
What you get
- Feature scoping tied to a measurable eval
- Prompt + model versioning with A/B evals
- Streaming UX with cost + latency SLOs
- Content safety, PII redaction, and audit logs
Use cases
In-app AI copilot
Chat-with-your-data copilot inside a SaaS with tool calls scoped to the tenant.
Structured output generators
Generate emails, contracts, or reports with schema-validated outputs.
Semantic search + summarize
Hybrid search across docs with grounded summaries and citations.
AI classifier at scale
Fine-tuned small model + LLM fallback for cost-efficient classification.
Examples I've shipped
SaaS analytics copilot
Outcome — 70% of dashboard questions answered without writing SQL.
Contract clause generator
Outcome — Generates review-ready clauses with typed outputs.
Case study
Challenge
AI drafting feature had 22% hallucination rate on citations and cost was untracked per customer.
Approach
- Introduced a 300-case golden eval + CI gate
- Added retrieval + citation-required prompting
- Per-tenant cost tracking with hard caps
- Model routing: Haiku for drafts, Sonnet for finals
Results
How I work
- 1
Scoping + eval design
Define success metric, build a small golden dataset first.
- 2
Prototype
Ship a thin end-to-end slice against real inputs.
- 3
Harden
Guardrails, retries, structured outputs, cost caps.
- 4
Ship + observe
Streaming UX, telemetry, per-tenant cost.
Deliverables
- Production AI feature integrated in your app
- Eval harness + CI gate
- Cost & latency dashboards
- Prompt + model version registry
Benefits
- AI features that survive real users
- Predictable unit economics
- Confidence to iterate prompts
Frequently asked questions
Do you work inside our existing codebase?+
Yes — TypeScript/Python most commonly. I can PR into your repo or ship a standalone service you consume.
How do you handle evals?+
A golden dataset per feature + Braintrust/Langfuse for online eval, gated in CI before any prompt or model change.