Evals for forward-deployed engineers: proving an AI agent works at a customer site

Short answer: a forward-deployed engineer proves a customer's AI agent works by agreeing pass/fail criteria with the customer first, building an eval set from the customer's own data, scoring a baseline, and rerunning that set plus a shared regression set on every change until the agreed bar is met. Then shadow-run on live inputs and hand over online evals and a review queue. Updated 11 Nov 2026.

By Evals Review editorial team · Published 11 Nov 2026 · Updated 11 Nov 2026

Context: forward-deployed engineer openings grew 4.6x in a year, to about 1,450 in October 2026, in Lenny Rachitsky's State of the Tech Job Market 2026 (TrueUp data). How we checked: role descriptions quote public job postings read on 11 Nov 2026 (postings change or close); tool facts come from vendor pricing pages read the same day. Docs review, not a hands-on test. How we review.

The customer-site eval workflow

  1. Agree the scoring criteria with the customer before you build. Write the pass/fail criteria and the go-live bar with the customer's process owner, and get them signed off. One FDE posting lists agreeing "scope, access owners, scoring criteria, dependencies" before delivery begins. [Sainapse job posting] Without that, "is it working?" becomes an opinion. Writing criteria: how to write AI evals.
  2. Build a per-customer eval set from their own data. Sample real tickets, calls, documents or transcripts from the customer's systems, with their permission and with personal data handled under their rules. Postings describe exactly this: building "per-customer evals against their own data" to prove accuracy and catch regressions, [Aron] and running a loop of "golden transcripts, rubric judges, regression suite". [Clutch job posting]
  3. Keep a shared core set and a customer set. A core regression set covers behavior every deployment needs; each customer set covers their workflow, vocabulary and edge cases. Run both on every change, so a fix for one customer can't silently break another.
  4. Baseline before you change anything. Score the out-of-the-box agent on the customer set first. The gap between baseline and go-live bar is your delivery plan, and the before/after numbers are what the customer's sponsor will ask for.
  5. Iterate with eval-driven development. Change one thing at a time (prompt, tool description, retrieval source, model), rerun both sets and compare per example. Keep a held-out slice of customer cases you don't tune on. [LangChain blog] The full loop: eval-driven development.
  6. Shadow-run before go-live. Run the agent beside the customer's current process (shadow or copilot mode) and score its outputs on live inputs before it acts on its own; one posting names "shadow or copilot operation" and a controlled ramp as part of the role. [Sainapse job posting]
  7. Hand over online evals and a review queue. Before you leave, set up sampled online scoring on production traces and an annotation queue the customer's experts can work through, and route confirmed failures back into the customer set. Then feed recurring patterns back to your product team.

Tool facts that matter at a customer site

Three questions decide the tool at most customer sites: can the data stay where the customer needs it, can each customer get a separate project, and can their experts review without a seat fee for each one.

Hosting, separation and reviewer access (vendor pages read 11 Nov 2026)
ToolWhere data can livePer-customer separationReviewer seats
LangfuseFree to self-host (open source); Cloud in US, EU or Japan, HIPAA region on Enterprise [Langfuse pricing]Unlimited projects on every plan [Langfuse pricing]2 users on Hobby, unlimited from Core ($29/month) [Langfuse pricing]
Comet OpikOpen source self-hosted is free; Free and Pro Cloud in the US; flexible deployments on Enterprise [Comet Opik pricing]Not verified on 11 Nov 2026Up to 10 members free, 50 on Pro ($19/month) [Comet Opik pricing]
LangSmithCloud in US or EU; self-hosted and hybrid on Enterprise [LangChain pricing page]Custom workspaces on Enterprise [LangChain pricing page]1 seat free; $39 per seat on Plus [LangChain pricing page]
BraintrustHosted; on-prem or hosted deployment on Enterprise [Braintrust pricing]Unlimited projects on Starter [Braintrust pricing]Unlimited users on Starter ($0) [Braintrust pricing]
Arize AXData region US, EU or CA on all plans; self-hosted on Enterprise [Arize AX pricing]1 space on Free, 2 on Pro [Arize AX pricing]Not verified on 11 Nov 2026
Confident AIDedicated on-prem and custom data residency on Enterprise [Confident AI pricing]1 project free, 5 on Starter, unlimited on Team [Confident AI pricing]2 seats free; unlimited from Starter ($200/month) [Confident AI pricing]

Self-hosting trade-offs: Opik vs Langfuse and Arize Phoenix vs Langfuse. If the customer's PMs will own the golden set after handover, see LLM evaluation tools for product managers.

Evals for AI product teams: the full series

FAQ

What evals does a forward-deployed engineer run?

A per-customer eval set built from the customer's own data, scored against criteria the customer agreed to, plus a shared regression set that every deployment must pass. After go-live, sampled online evals on production traffic and a human review queue for the customer's experts.

How do I prove an AI agent works for a customer?

Agree pass/fail criteria and a go-live bar with the customer first, score the agent's baseline on their data, show the before and after on the same set, and shadow-run it on live inputs before it acts alone.

Should I self-host the eval tool at a customer site?

If the customer's data can't leave their environment, yes, or pick a vendor region that matches their rules. Langfuse and Comet Opik can be self-hosted from open source; LangSmith, Braintrust and Arize offer self-hosted or on-prem options on Enterprise plans.

How fast are forward-deployed engineer roles growing?

In Lenny Rachitsky's November 2026 report on TrueUp data, forward-deployed engineer openings grew 4.6x in a year, to about 1,450 open roles in October 2026.

Sources