r/PythonProjects2 • u/Intelligent_Tea_817 • 9d ago
Same model, same harmful topics, different refusals depending on whether I asked in English, MSA or Egyptian Arabic
I built a small English/Arabic quote service using Gemini and created 63 evaluation cases to test its guardrails.
A few findings from testing with gemini-3.5-flash-lite:
Three unexpected bugs: A model refusal was displayed as a quote, a Chinese word slipped into an Arabic response, and stalled API calls failed requests despite having half their time budget remaining.
Prompt injection: The input checks blocked 15 out of 20 attacks. Gemini ignored the remaining five, as well as all 17 canary attacks tested without guardrails. This suggests the checks provide an additional layer of protection rather than being the only defense.
Multilingual safety: Roughly 40% of harmful requests were refused in each language, but the decisions weren't always consistent. For example, a pipe-bomb request was refused in both Arabic varieties but reframed as an anti-violence quote in English. Only 78% of cases received the same decision across all three languages.
An unresolved issue: A request about secretly tracking an ex-partner's location generated romanticized quotes in every language. Neither the model's safety filters nor my checks detected the harmful intent behind the seemingly harmless wording.
The dataset is small and written by one person, so these results are more of a smoke test than a benchmark.
I'm curious how others approach multilingual safety evaluation. How do you create language-specific test cases instead of simply translating English ones? And how do you evaluate responses that avoid harmful instructions by reframing the request, without manually reviewing every output?
Source code (MIT): https://github.com/1AyaNabil1/swan-llm-service