Safety & factuality

DailySafe

by Xuankun Rong, Mang Ye et al. (Wuhan University, Ant Group and collaborators)

What is DailySafe? DailySafe is a safety & factuality eval made by Xuankun Rong, Mang Ye et al. (Wuhan University, Ant Group and collaborators). Checks whether a multimodal assistant notices when an ordinary, legitimate task turns risky partway through a conversation. Each of the 195 English cases is a multi-turn chat with images (692 user turns and 475 images in total, across 10 everyday domains and 42 task categories) where the warning signs only appear gradually, in earlier turns or in the pictures, so no single request looks harmful on its own. Answers are judged against case-specific safety notes for two things: whether the model handled the risk (Safety Rate) and whether it still helped with the real goal (Helpfulness Rate). The repository holds the reference runner and scoring code; the authors report wide gaps between the 16 models they tested, with small open-weight vision models scoring lowest (arXiv:2611.26298).

Page last updated 2026-10-18 · Evals Review

Editor note

Checks whether a multimodal assistant notices when an ordinary, legitimate task turns risky partway through a conversation. Each of the 195 English cases is a multi-turn chat with images (692 user turns and 475 images in total, across 10 everyday domains and 42 task categories) where the warning signs only appear gradually, in earlier turns or in the pictures, so no single request looks harmful on its own. Answers are judged against case-specific safety notes for two things: whether the model handled the risk (Safety Rate) and whether it still helped with the real goal (Helpfulness Rate). The repository holds the reference runner and scoring code; the authors report wide gaps between the 16 models they tested, with small open-weight vision models scoring lowest (arXiv:2611.26298).

What it tests
Context-dependent safety in multi-turn, multi-image everyday assistance, scored for both safety and helpfulness.
Original source
github.com/XuankunRong/DailySafe
Last updated
arXiv:2611.26298; repo last pushed 2026-11-20 (GitHub API, read 21 Nov 2026)
License
MIT (code, GitHub API, read 21 Nov 2026)

Facts come only from the linked source (checked 15–16 Oct 2026). Credit belongs to the creators. We don't score or rank evals: how we review.

Reviews

No reviews yet.

Reviews are written by people who sign in with GitHub, one per account per eval. We can't confirm that each reviewer used the eval; we remove reviews that break our rules, never just for being negative, and we never add, buy or edit reviews. Averages, when shown, come only from these reviews.

Sign in with GitHub to review

Related safety & factuality evals

  • HarmBench — Robustness to jailbreaks and harmful requests.
  • TruthfulQA — Truthfulness versus imitating human misconceptions.
  • StrongREJECT — Real effectiveness of jailbreak attacks.
  • SimpleQA — Short-form factuality and calibration.

All Safety & factuality evals

Discussion

No one has discussed DailySafe yet. Have you run it? Share what it caught, what it missed or how you set it up.

Sign in with GitHub to comment

Be specific and kind. No promotion or affiliate links. Community rules · All discussions