Gemini 2.5 Deep Think: Promising Reasoning, Mixed Safety Signals
Google’s latest model, Gemini 2.5 Deep Think, introduces a new reasoning mode designed for deeper, more deliberate responses. But how does it perform on safety and instruction-following?
The model card (Aug 1, page 6) reveals some important insights:
📊 Evaluation Results (vs. Gemini 2.5 Pro)
Text-to-Text Safety: ↓ 16.3%
Less effective at filtering unsafe content.
Multilingual Safety: ↓ 1.0%
Slight drop in safety across languages.
Image-to-Text Safety: ↑ 2.1% (non-egregious)
Slight improvement in handling image-based prompts.
Tone (Refusal Objectivity): ↑ 16.3%
More appropriate and objective refusals.
Instruction Following: ↓ 9.9%
More frequent over-refusals of benign queries.
Gemini 2.5 Deep Think shows improved tone and image safety, but also a tendency to over-refuse benign queries and a drop in text-based safety filtering.
My Takeaway: While Deep Think shows promise in reasoning and tone, its safety profile still needs refinement.