DailySafe
by Xuankun Rong, Mang Ye et al. (Wuhan University, Ant Group and collaborators)
What is DailySafe? DailySafe is a safety & factuality eval made by Xuankun Rong, Mang Ye et al. (Wuhan University, Ant Group and collaborators). Checks whether a multimodal assistant notices when an ordinary, legitimate task turns risky partway through a conversation. Each of the 195 English cases is a multi-turn chat with images (692 user turns and 475 images in total, across 10 everyday domains and 42 task categories) where the warning signs only appear gradually, in earlier turns or in the pictures, so no single request looks harmful on its own. Answers are judged against case-specific safety notes for two things: whether the model handled the risk (Safety Rate) and whether it still helped with the real goal (Helpfulness Rate). The repository holds the reference runner and scoring code; the authors report wide gaps between the 16 models they tested, with small open-weight vision models scoring lowest (arXiv:2611.26298).
Editor note
Checks whether a multimodal assistant notices when an ordinary, legitimate task turns risky partway through a conversation. Each of the 195 English cases is a multi-turn chat with images (692 user turns and 475 images in total, across 10 everyday domains and 42 task categories) where the warning signs only appear gradually, in earlier turns or in the pictures, so no single request looks harmful on its own. Answers are judged against case-specific safety notes for two things: whether the model handled the risk (Safety Rate) and whether it still helped with the real goal (Helpfulness Rate). The repository holds the reference runner and scoring code; the authors report wide gaps between the 16 models they tested, with small open-weight vision models scoring lowest (arXiv:2611.26298).
- What it tests
- Context-dependent safety in multi-turn, multi-image everyday assistance, scored for both safety and helpfulness.
- Original source
- github.com/XuankunRong/DailySafe
- Last updated
- arXiv:2611.26298; repo last pushed 2026-11-20 (GitHub API, read 21 Nov 2026)
- License
- MIT (code, GitHub API, read 21 Nov 2026)
Related safety & factuality evals
- HarmBench — Robustness to jailbreaks and harmful requests.
- TruthfulQA — Truthfulness versus imitating human misconceptions.
- StrongREJECT — Real effectiveness of jailbreak attacks.
- SimpleQA — Short-form factuality and calibration.
Discussion
No one has discussed DailySafe yet. Have you run it? Share what it caught, what it missed or how you set it up.