# Practitioner mentions of LLM eval / observability tools: methodology

Public page: [/data/practitioner-mentions-llm-eval-tools](/data/practitioner-mentions-llm-eval-tools) (CSV download on that page). Collected 2 Nov 2026 (IST) by Evals Review.

**Dataset version:** v1.1 (v1.0 had 285 kept mentions; v1.1 removes 3 non-tool Braintrust matches, see below). **Pull date:** 2026-11-02 (HN Algolia pull finished 2 Nov 2026, ~17:50 IST, per the collection log). Both fields are on every row of `summary.csv` and `thread-counts.jsonl`. **Source: Hacker News only.**

## Files
- `summary.csv`: one row per tool (aggregated counts only).
- `thread-counts.jsonl`: one row per (tool, thread) with the thread URL and counts. No comment text, no usernames (hashed or otherwise), no comment IDs.
- Comment-level working rows (comment ID, permalink, stance, short snippet, author hash used only for dedupe/self-promo) are kept in a private working folder outside the repo and are not published.

## Sources
- Hacker News via the official Algolia search API (`https://hn.algolia.com/api/v1/search_by_date`, `tags=comment`), no login, ~0.5 s between requests.
- Reddit: planned via public `search.json` endpoints (r/LocalLLaMA, r/MachineLearning, r/LangChain, r/LLMDevs, r/OpenAI) with a descriptive User-Agent and 2.5 s spacing. **Every request returned HTTP 403 "Blocked" on 2 Nov 2026.** We did not work around the block (no login, no API key, no scraping), so **this dataset contains no Reddit data**.

## Date range
Query window: 2 Nov 2024 00:00 UTC to 2 Nov 2026 00:00 UTC (24 months). Earliest kept mention 3 Nov 2024, latest 28 Oct 2026.

## Query terms (HN, each paged up to 1,000 hits)
promptfoo, deepeval, ragas, langfuse, langsmith, braintrust, arize phoenix, arize, helicone, trulens, inspect_ai, inspect ai, opik, lm-evaluation-harness, lm-eval-harness, openai evals, weave wandb, galileo llm, how do you evaluate llm, llm evaluation tools, llm observability, llm evals.

## How a mention is counted
- A comment counts once per tool if its text matches that tool's pattern (e.g. `langfuse`, `deep ?eval`, `lm[-_ ]eval(uation)?[-_ ]harness|lm_eval`, `inspect[ _-]?ai`; ambiguous names need context: Galileo must appear as Galileo AI / rungalileo / galileo.ai, Ragas near eval/RAG words, Weave near W&B/wandb, Phoenix near Arize/tracing/eval words).
- Dedupe: by comment ID across all queries (a comment returned by several queries counts once per tool). `distinct_authors` uses an internal author hash (private file only); `distinct_threads` uses the HN story ID.
- Stance (keyword rules on a ~240-character window around the tool name): **complaint** if it contains words such as broken, buggy, slow, expensive, clunky, ditched, stopped using; else **recommend** if first-hand use or advice ("we use", "I switched to", "recommend", "check out", "happy with", "big fan", "favourite"); else **neutral** (passing mention, list of tools, link, question). Only "recommend" rows should be read as recommendations; most mentions are neutral.

## Exclusions
- **Self-promotion:** if an author discloses an affiliation in a comment mentioning a tool ("I work at", "founder", "maintainer here", "disclaimer", "we built", "our product", "Hi HN", "we've launched"), all of that author's comments about that tool are excluded.
- **Vendor launch threads:** comments in a story whose title names the tool and reads like a launch (Show HN, Launch HN, "introducing", release/version) are excluded.
- **Job threads:** "Who is hiring" / "Who wants to be hired" / freelancer threads (résumé skill lists, job ads) are excluded: 99 rows.
- **Off-topic name matches** (Galileo the astronomer, Indian classical ragas): 11 rows.
- Totals: 505 raw tool-mention rows, 395 after job/off-topic filters, 110 excluded as self-promo or launch thread, 285 kept by the automatic rules; **282 kept in v1.1** after removing 3 non-tool Braintrust matches.

## Sample sizes
- 7,985 distinct HN comments returned by the queries (the "scanned" set); 282 kept tool mentions across 199 tool-thread pairs (v1.1); 0 Reddit.
- Precision spot-check: two random samples of 40 kept rows, read by hand.
  - Sample 1 (before the job-thread / ambiguous-name filters were added): 3/40 matched the wrong thing (Galileo x2, ragas x1); 9/40 were résumé or hiring posts; 4 recommendations were labelled neutral; 1 launch post slipped through. These findings drove the extra filters.
  - Sample 2 (final rules): tool identity correct 40/40; stance correct 37/40 (one recommendation labelled neutral, one recommend and one complaint mislabelled); 2/40 were undisclosed vendor replies that the self-promo rule missed.

## Limits and biases
- **HN only.** Reddit blocked unauthenticated access, so the data reflects HN's audience (startup/infra-heavy, US-leaning, open-source-friendly).
- **Keyword stance is crude.** It under-counts recommendations (many real endorsements read as neutral) and can attach a sentiment word to the wrong tool when several tools are named together. Treat recommend/complaint splits as rough.
- **Self-promo detection only catches disclosed affiliations.** Undisclosed vendor staff remain (seen in the spot-check).
- **Name ambiguity and popularity:** generic names (Braintrust, Weave, Phoenix, Inspect, Galileo) are under-counted by strict patterns; well-known names are over-represented because people list them in passing.
- **Algolia limits:** 1,000 hits max per query; very common terms may be truncated. News events (acquisitions, launches) create mention spikes unrelated to use.
- Counts are mentions, not market share or quality.

## Read this first: caveats
- **Mentions are not market share.** A count is how often a tool's name came up in HN comments, not how many teams use it, pay for it, or like it.
- **HN leans toward open-source and self-hosted tools.** Its audience is startup/infra-heavy and open-source-friendly, so tools like Langfuse and promptfoo are likely over-represented versus closed SaaS tools.
- **Filter rules** (detail above): one count per comment per tool; deduped by comment ID; job/hiring threads and off-topic name matches removed; disclosed self-promotion and vendor launch threads (title names the tool + Show HN / Launch HN / introducing / release) removed. 7,985 comments scanned, 505 raw tool mentions, 285 kept by the automatic rules, 282 after the manual Braintrust check (v1.1).
- **QA:** on a hand-checked random sample of 40 kept rows, tool identity was correct 40/40 and stance was correct 37/40. **Stance under-counts real recommendations**: many genuine endorsements are labelled neutral, so the `recommend` column is a floor.
- **Zero is a real result.** TruLens and Galileo were in the search terms (`trulens`, `galileo llm`) and had 0 kept mentions in the window; they are kept as explicit 0 rows. (Galileo name matches that were about the astronomer were removed as off-topic.)

## Thread concentration
A single news thread can swing the counts: the "ClickHouse acquires Langfuse" thread alone holds 25 of Langfuse's 111 mentions.

**Sort order (v1.1):** `summary.csv` is sorted by `mentions_by_unique_thread` (number of distinct HN threads) first, with raw comment `mentions` as the secondary column and tiebreak. Thread count is the primary measure because one busy thread can inflate comment counts. Every row also has `top_thread_share_pct` (share of mentions from the single busiest thread), `mentions_excluding_top_thread` and `threads_excluding_top_thread`.

`mentions` counts comments; `mentions_by_unique_thread` counts distinct HN stories. `top3_threads_mentions` / `top3_threads_share_pct` show how much of a tool's count comes from its three busiest threads, and `max_mentions_single_thread` the busiest one.

| Tool | Mentions (comments) | Unique threads | Top 3 threads | Share | Top threads |
|---|---|---|---|---|---|
| Langfuse | 111 | 60 | 35 | 31.5% | [ClickHouse acquires Langfuse](https://news.ycombinator.com/item?id=46656552) (25); [LLM Observability in the Wild – Why OpenTelemetry Should Be the Standard](https://news.ycombinator.com/item?id=45398467) (6); [Doubling Down on Open Source](https://news.ycombinator.com/item?id=44180954) (4) |
| promptfoo | 45 | 38 | 9 | 20.0% | [Ask HN: How are people doing AI evals these days?](https://news.ycombinator.com/item?id=47319587) (4); [About AI Evals](https://news.ycombinator.com/item?id=44430117) (3); [Bypass DeepSeek censorship by speaking in hex](https://news.ycombinator.com/item?id=42891042) (2) |
| Braintrust (v1.1) | 41 | 34 | 8 | 19.5% | [Ask HN: How are people doing AI evals these days?](https://news.ycombinator.com/item?id=47319587) (4); [Show HN: Kelet – Root Cause Analysis agent for your LLM apps](https://news.ycombinator.com/item?id=47767606) (2); [Launch HN: Langfuse (YC W23) – OSS Tracing and Workflows to Improve LLM Apps](https://news.ycombinator.com/item?id=42441258) (2) |
| LangSmith | 38 | 30 | 8 | 21.1% | [RAG Logger: An Open-Source Alternative to LangSmith](https://news.ycombinator.com/item?id=42485113) (4); [Ask HN: What are you working on? (September 2026)](https://news.ycombinator.com/item?id=49686380) (2); [Open SWE by LangChain](https://news.ycombinator.com/item?id=44837601) (2) |

**Langfuse finding:** 35 of 111 mentions (31.5%) come from 3 threads, and 25 (22.5%) from one: the ClickHouse acquisition news thread. That is a news spike, not a launch or Show HN thread (tool launch threads were already excluded). Without that one thread Langfuse has 86 mentions across 59 threads, still the most-mentioned tool. None of the top 3 threads for the four tools is the tool's own launch/Show HN thread; one Braintrust top thread is a Show HN for a different product, where it was named as a comparison. 

## Manual Braintrust check (v1.1)
"Braintrust" is also the name of a talent network / hiring marketplace. Two doubtful threads were checked by hand against the private comment rows and the HN thread context:
- [Job-seekers are dodging AI interviewers](https://news.ycombinator.com/item?id=44783155): the thread discusses a Fortune article quoting the CEO and founder of Braintrust, "a company that distributes AI interviewers". All 3 kept Braintrust comments in this thread refer to that talent-network company (its CEO, its AI interviewer, a "Trainbust" joke), not the eval/observability tool. **All 3 removed.** (Another 9 Braintrust comments in the thread had already been excluded by the automatic rules; they also refer to the talent network.)
- [Show HN: Kelet](https://news.ycombinator.com/item?id=47767606): both kept comments name Braintrust alongside LangSmith/Langfuse/LangChain as trace and telemetry platforms, i.e. the eval/observability tool. **Both kept.**

Braintrust corrected: 44 → **41 comments**, 35 → **34 threads**, 42 → 39 distinct authors, neutral 39 → 36; top-3 threads 8 of 41 (19.5%), top thread 4 (9.8%). The thread row for 44783155 was removed from `thread-counts.jsonl`. Totals: 285 → 282 kept mentions, 200 → 199 tool-thread pairs. Other Braintrust matches were not all hand-read; the 40-row QA sample found no further talent-network errors.

## Privacy
Only aggregates and HN thread links are published. No comment text, usernames or author hashes. No Reddit data exists in this version.