How to evaluate Gemini 4 Carbon

By Evals Review editorial team · Published 11 Nov 2026 · Sources read 11 Nov 2026 · Next price and tool-support check 1 Dec 2026

Short answer: run your own fixed test set against gemini-4-carbon at each thinking level (low, medium and high), and score quality, cost and latency side by side. Then add five targeted checks: retrieval across the 1M-token window, long decodes, the thinking-level trade-off, multi-step tool use and Office or OpenDocument files.

How we checked: facts come from Google's announcement, the Gemini API pricing and model docs and Google Cloud's developer guide, all read on 11 Nov 2026. Tool support comes from each vendor's docs or GitHub. We have not run Carbon ourselves. The introductory price ends on 30 Nov 2026, so cost figures change on 1 Dec. How we review.

TL;DR

  • Google's own advice: run your evals at all three thinking levels, then pick the lowest one that meets your quality bar. [Google docs]
  • Thinking tokens are billed as output at $10 per 1M (until 30 Nov), so the thinking level you choose is a large part of your cost. [pricing]
  • Carbon ignores temperature, so an eval at temperature 0 is not reproducible. Run each case at least three times. [Google Cloud]
  • Check your harness first: frequency_penalty, presence_penalty, thinking_budget or minimal return HTTP 400. [Google Cloud]

Gemini 4 Carbon facts for eval planning

Checked against Google's pages on 11 Nov 2026
Model idgemini-4-carbon (stable) [model page]
Released10 Nov 2026, on the Gemini API paid tier and Gemini for business [Google]
Context window1,048,576 tokens, shared by input, thinking and output [model page]
Max output65,536 tokens per synchronous response by default; up to 1,048,576 minus your input with long decode [model page]
Thinking levelslow, medium (default) and high; thinking_budget and minimal return HTTP 400 [Google docs] [Google Cloud]
Inputs (Gemini API)Text, image, video, audio and PDF [model page]
Office and OpenDocument filesDocumented on Google Cloud's Agent Platform: .docx, .dotx, .doc, .pptx, .ppt, .xlsx, .xls, .odt, .odp, .ods, inline or from Cloud Storage [Google Cloud]. The Gemini API model page we read lists only text, image, video, audio and PDF [model page]
Price$2 per 1M input and $10 per 1M output tokens through 30 Nov 2026; $4 and $20 from 1 Dec 2026 [pricing]
Thinking billingOutput price includes thinking tokens; every token the model processes is billed [pricing]
Free tierNot available on the free tier [pricing]
CachingImplicit caching only, $0.10 per 1M cached tokens until 30 Nov ($0.20 from 1 Dec); explicit caching not supported [pricing]
Sampling controlstemperature, top_p and top_k are ignored; frequency_penalty, presence_penalty and candidate_count above 1 return HTTP 400 [Google Cloud]

Google positions Carbon for complex, multi-step work and says it uses more tokens than Gemini 4 Argon to explore and verify. [Google] Both models have the same per-token price, and Google says Carbon spends more thinking tokens per turn than Argon, which raises per-turn token use and latency. [Google docs] Before you switch, run your eval against Argon too.

Before you start: fix your harness

  1. Remove rejected parameters. Drop frequency_penalty, presence_penalty, candidate_count above 1, thinking_budget and minimal. Each returns HTTP 400. [Google Cloud]
  2. Remove prefilled model turns. Chat histories can't end with a model turn, and prefilled model turns aren't supported. [Google Cloud]
  3. Plan for repeats. Sampling controls are ignored, so budget for at least three runs per case and report the spread, not one score. [Google Cloud]
  4. Log usage per call. Record input, cached, output and thinking tokens. On long decodes, input counts grow with each continuation segment, so sum the segments to get billed tokens. [Google Cloud]
  5. Check quotas before a big run. Grounding with Google Search has a default cap of 1,500 requests per day per project on the Gemini Developer API. [pricing] The pages we read don't list Carbon's token or request rate limits, so check your project's limits in AI Studio and start with a small batch.
  6. Keep a baseline. Run the same test set on your current model and on Gemini 4 Argon, so you judge Carbon on your data rather than on launch benchmarks. Keeping one fixed test set and scoring every model or prompt change against it is eval-driven development.

What changes versus Gemini 4 Argon and Gemini 3.1 Pro

Eval-relevant differences, from Google's docs (11 Nov 2026)
Gemini 4 CarbonGemini 4 ArgonGemini 3.1 Pro Preview
Price per 1M tokens (input / output)$2 / $10 to 30 Nov; $4 / $20 from 1 DecSame as Carbon$2 / $12 up to 200k-token prompts; $4 / $18 above
Token limitsOne shared 1,048,576-token windowSame shared 1M windowSeparate 1M input and 64K output budgets
Reasoning controlthinking_level low/medium/high; thinking_budget errorsSame levels, default mediumlow/medium/high, default high
Tokens per turnMore thinking per turn than ArgonFewer tokens, faster responsesNot compared by Google

Sources: pricing page, [pricing] long-context and thinking docs, [Google docs] [Google docs] what's new, [Google docs] and Google Cloud's migration notes. [Google Cloud] Because Carbon and Argon cost the same per token, the cost gap between them comes from tokens used per task, so measure that on your own test set.

Gemini API (AI Studio) or Google Cloud? On Google Cloud's Gemini Enterprise Agent Platform, Carbon runs in the global, us and eu locations, is limited to verified enterprise customers in select regions, and needs the v1beta1 API for long decode. [Google Cloud] Google says Agent Platform prices may differ from the Gemini API prices on this page. [pricing] Run your eval on the surface you will ship on.

Launch leaderboards are not your eval. Third-party boards such as Artificial Analysis report Carbon at its high thinking level. [Artificial Analysis] If you plan to ship at medium or low, those numbers don't describe your setup. For one benchmark we already track, see Humanity's Last Exam.

What to test, with an eval plan for each

1. Long-context retrieval across the 1M window

The 1,048,576-token window is shared with thinking and output. If input plus max_output_tokens goes over the limit, the API cuts the output limit to what is left. [Google docs]

  • Test set: take a real document set (contracts, logs, a codebase) and plant 20–50 facts whose answers you know, at the start, middle and end of the context.
  • Sizes: run at about 100k, 500k and 900k input tokens. A near-full prompt leaves little room for thinking and the answer.
  • Score: exact-match or rubric accuracy by position and size, plus multi-hop questions that need two planted facts.
  • Watch: answers that come back empty or cut short at large inputs. That is usually an output-room problem, not a retrieval failure.

2. Long decodes

A synchronous response stops at 65,536 output tokens by default, and thinking counts toward that cap. Long decode continues in segments up to 1,048,576 tokens minus your input. [model page] [Google Cloud] Google lists "evaluation harnesses that stop at the output limit" as a case for long decode, because a hard prompt can spend a whole segment on thinking. [Google Cloud]

  • Test set: 10–20 tasks that need long output, such as a full module rewrite, a long proof or a large report.
  • Run twice: once at the default cap and once with long decode on.
  • Score: the share of answers cut off at the cap (status incomplete or MAX_TOKENS), task success, total billed tokens and wall-clock time.
  • Watch: Google warns that long decode can run for several minutes, and that it gives the model more room to reason without making answers longer. [Google Cloud]

3. Thinking levels: the cost and quality trade-off

Google recommends low for classification, summarisation and extraction, medium for general coding, tool use and document Q&A, and high for multi-file engineering, formal proofs and architecture. [Google Cloud]

  • Test set: your fixed test set, unchanged, run at low, medium and high.
  • Score: quality, output plus thinking tokens per case, cost per case and time to first token.
  • Decide: pick the lowest level whose quality is within your tolerance of the best level. Re-check per task type, because one level rarely fits all of them.
  • Watch: to save tokens, lower thinking_level rather than setting a small max_output_tokens. A small cap can return truncated or empty output while still billing the thinking. [Google docs]

4. Agentic tool use

Carbon supports function calling, code execution, computer use, URL context, file search and grounding with Google Search. [model page] Its tool loop has strict rules: each function response must match the id, name and order of the preceding call, and Google recommends running tool turns one after another. [Google Cloud]

  • Test set: 20–50 multi-step tasks against mocked tools with known correct end states, including a few where a tool returns an error.
  • Score: task success from the end state, wrong or extra tool calls, recovery after a tool error, number of turns and tokens per task.
  • Watch: format errors your harness causes, such as mismatched function response ids. Count them apart from model failures. Grounded calls also bill tool instructions and declarations as tokens. [pricing]

5. Office and OpenDocument file inputs

Google Cloud's guide says Carbon accepts Word, PowerPoint and Excel files plus their OpenDocument versions, inline or from Cloud Storage. [Google Cloud] The Gemini API model page lists text, image, video, audio and PDF inputs, so confirm the file type works on your surface before you build a test set. [model page]

  • Test set: 20–30 real files of each type you use, with questions about tables, charts, speaker notes and formulas.
  • Compare: the native file against the same content as PDF and as extracted text, to see whether native input is worth it.
  • Score: answer accuracy, numbers read correctly from cells and charts, and input tokens per file.

Which eval tools support Gemini 4 Carbon on day one

We checked each tool's own docs or GitHub on 11 Nov 2026. "Generic" means the docs say you can pass any model id. It does not mean the vendor named Carbon.

Gemini 4 Carbon support in eval tools we review (11 Nov 2026)
ToolStatusWhat we found
Inspect AIWorks with an arbitrary model id; named entry merged, not yet releasedUse --model google/gemini-4-carbon. A model-info entry for Carbon (1M context, 65,536 output, default thinking medium) was merged to main on 10 Nov. [GitHub] The newest release when we checked, 0.3.289, came out before that merge. [GitHub] The PR also notes that unlisted Gemini names already run, and that Inspect does not request long decode, so outputs stop at 65,536 tokens. [GitHub]
BraintrustIn the open-source proxy model catalogA merged 10 Nov change adds gemini-4-carbon to the braintrust-proxy catalog for Google, Vertex and OpenRouter endpoints at $2/$10 per 1M tokens. [GitHub] We did not check when that reaches the hosted Braintrust app.
DeepEvalGeneric: any Gemini model nameThe docs say you can use any Gemini model by passing its name, for example to GeminiModel or a metric's model argument. [DeepEval] Carbon is not named in the docs we read. Carbon ignores temperature, so a DeepEval temperature setting has no effect on it. [Google Cloud]
LangfuseTracing yes; built-in Carbon price not yet listedLangfuse's default price list on main had gemini-4-argon but no gemini-4-carbon entry when we read it. [GitHub] Until it does, add a custom model definition or ingest the cost from the API response. [Langfuse]
PromptfooNot verifiedThe Google provider uses ids of the form google:<model>, with examples such as google:gemini-3.8-flash. [Promptfoo] The page we read does not mention Carbon or say that any model id is accepted, so we have not listed it as supported.
LangSmithNot verifiedWe found no LangSmith note on Carbon. A pull request adding Carbon profiles to the langchain-google integration was still open when we checked. [GitHub]
Comet OpikNot verifiedWe found no Carbon mention in Opik's GitHub issues or pull requests on 11 Nov 2026.

Gateway route: LiteLLM, which we don't review, announced day 0 support as gemini/gemini-4-carbon and maps reasoning_effort to Carbon's thinking level. [LiteLLM] Tools that call models through LiteLLM can use that id, but check the tool's own docs first.

To choose between the tools, see LLM evaluation tools compared and Langfuse vs LangSmith. If you are a product manager signing off on a model switch, start with the best LLM evaluation tools for AI product managers.

Cost notes, with a worked example

Prices from the Gemini Developer API pricing page: $2 per 1M input and $10 per 1M output tokens (thinking included) through 30 Nov 2026, then $4 and $20 from 1 Dec 2026. Cached input costs $0.10 per 1M ($0.20 from 1 Dec). [pricing] The token counts below are an example plan, not measurements.

Example: 200 cases × 3 thinking levels × 3 repeats = 1,800 calls; 20,000 input tokens per call; 4,000 output and thinking tokens per call on average
ItemTokensUntil 30 NovFrom 1 Dec
Input36M$72$144
Output incl. thinking7.2M$72$144
Total43.2M$144$288
  • Long-context runs are input-heavy. Ten questions over the same 900,000-token context is 9M input tokens: $18 before 1 Dec, $36 after. If the shared prefix is served from the implicit cache, those tokens bill at $0.10 per 1M, so about $0.90. Implicit caching starts at shared prefixes of 4,096 tokens, so put the document first and the question last. [Google Cloud] Cache hits are automatic, not promised, so budget for the uncached figure.
  • Thinking levels change the output line. Output costs five times as much per token as input. If high doubles output tokens per call, the output line for that level doubles too. Measure tokens per level on 20 cases before running everything.
  • Batch. The model page lists Batch API support, and the paid tier lists a 50% cost reduction for Batch. [model page] [pricing] Carbon's own pricing table shows standard rates only, and we found no Carbon-specific batch rate on Google's pages, so confirm the batch price on your bill before you count on half price.
  • Timing. Runs before 1 Dec cost half as much as the same runs after. Re-baseline your cost per case when the price changes.

What can go stale: the introductory price ends on 30 Nov 2026; tool support in the table above changes as vendors release updates; and the model id is gemini-4-carbon (stable) as of 11 Nov 2026. Pin the id in your config so a rename shows up as an error, not a silent switch. We will re-check prices and tool support on 1 Dec 2026.

FAQ

How do I evaluate Gemini 4 Carbon?

Run your own fixed test set against gemini-4-carbon at all three thinking levels (low, medium, high), score quality, cost and latency for each, and add targeted tests for long-context retrieval, long decodes, multi-step tool use and the file types you use. Google's own guidance is to pick the lowest thinking level that meets your quality bar.

How much does Gemini 4 Carbon cost?

On the Gemini Developer API it costs $2 per 1M input tokens and $10 per 1M output tokens through 30 November 2026, then $4 and $20 from 1 December 2026. Output includes thinking tokens. There is no free tier for this model.

What is Gemini 4 Carbon's context window?

1,048,576 tokens, shared between input, thinking and output. A synchronous response returns up to 65,536 output tokens by default; long decode extends output up to 1,048,576 tokens minus your input.

Which thinking levels does Gemini 4 Carbon support?

low, medium and high, with medium as the default. Setting thinking_budget or the minimal level returns an HTTP 400 error.

Can Gemini 4 Carbon read Word, Excel and PowerPoint files?

Google Cloud's Agent Platform guide says it accepts Microsoft Office (.docx, .dotx, .doc, .pptx, .ppt, .xlsx, .xls) and OpenDocument (.odt, .odp, .ods) files. The Gemini API model page we read lists text, image, video, audio and PDF inputs only, so check the surface you use.

Does temperature 0 make Gemini 4 Carbon deterministic for evals?

No. Google's guide says Carbon ignores temperature, top_p and top_k, so you cannot pin sampling. Run each test case more than once and report the spread.

Should I evaluate Gemini 4 Carbon or Gemini 4 Argon?

Both. They cost the same per token and share the 1M window, but Google says Carbon spends more thinking tokens per turn, so it is slower and uses more tokens per task. Run the same test set on both and compare quality against cost per task.

Which eval tools support Gemini 4 Carbon?

On 11 November 2026: Inspect AI runs it via google/gemini-4-carbon, with a named entry merged but not yet released; the open-source Braintrust proxy catalog lists it; DeepEval accepts any Gemini model name; Langfuse traces it but had no built-in Carbon price yet. We could not verify Promptfoo, LangSmith or Opik.

Related on Evals Review

Sources

The Google AI Studio post links to the developer guide as an X article. We used Google's docs pages for every fact quoted here.