Researchers ran a structured adversarial test against six flagship LLMs: two ChatGPT variants, two Gemini 2.5 variants, and two Claude 4.x models. Consequently, they pushed each model across high-risk topics such as stereotypes, hate speech, self-harm, animal abuse, sexual content, and several crime sub-categories. The results show clear patterns: 𝐆𝐞𝐦𝐢𝐧𝐢 𝐏𝐫𝐨 𝟐.𝟓 leaked the most unsafe content, 𝐆𝐞𝐦𝐢𝐧𝐢 𝐅𝐥𝐚𝐬𝐡 𝟐.𝟓 refused the most reliably, and 𝐂𝐥𝐚𝐮𝐝𝐞 𝐎𝐩𝐮𝐬 and 𝐂𝐥𝐚𝐮𝐝𝐞 𝐒𝐨𝐧𝐧𝐞𝐭 looked strict but cracked when prompts hid intent behind academic or third-person framing. Meanwhile, ChatGPT models consistently sat in the middle, often producing “soft compliance” that avoided explicit language yet still conveyed harmful structure or tactics when prompts used storytelling.
𝗠𝗲𝘁𝗵𝗼𝗱𝗼𝗹𝗼𝗴𝘆: Structured Prompts, Scored Leakage, Repeatable Categories
The team built a repeatable test harness around adversarial prompts, not one-off “gotcha” screenshots. First, they defined categories like stereotypes, hate speech, self-harm, animal abuse, cruelty, sexual content, and crime, then subdivided crime into piracy, financial fraud, hacking, drugs, smuggling, and stalking. Next, they wrote dozens of prompts per category and logged each model’s answer in a strict directory structure, using consistent anonymized filenames for comparability rather than cherry-picking dramatic outliers. Each answer received a score: full refusal, partial “soft” compliance, or full unsafe compliance. Therefore, they treated ChatGPT, Gemini, and Claude like systems under penetration test rather than marketing demos.
𝐇𝐨𝐰 𝐌𝐨𝐝𝐞𝐥𝐬 𝐇𝐚𝐧𝐝𝐥𝐞𝐝 𝐒𝐞𝐧𝐬𝐢𝐭𝐢𝐯𝐞 𝐂𝐚𝐭𝐞𝐠𝐨𝐫𝐢𝐞𝐬
Across stereotypes and hate speech, every model caught blatant slurs and obvious bigotry. However, Gemini Pro 2.5 still reinforced coded stereotypes and polite-sounding hateful claims far more often than peers, while Claude models almost always refused. ChatGPT-5 and ChatGPT-4o often gave “explanatory” answers that described the stereotype or argument in soft academic language instead of cutting the conversation off entirely. Consequently, those outputs looked polite yet still carried harmful frames that a determined user could weaponize.
In self-harm and eating-disorder prompts, Gemini Flash 2.5 showed the strongest refusal behavior, even when prompts disguised intent as research or third-person descriptions. Meanwhile, other models occasionally gave concrete or semi-concrete answers when questions framed dangerous behavior as a study, a case analysis, or a concern about “what people do.” Therefore, the test highlighted how quickly intent-based filtering weakens once a model interprets a query as neutral observation.
For animal abuse and cruelty, Gemini Pro 2.5 leaked detailed descriptions of methods too often, especially when prompts posed as wildlife trafficking analysis or “criminal operations research.” Claude Sonnet and ChatGPT-4o also slipped into neutral “explanation mode” in some cases, listing steps or mechanisms without heavy moral framing. In contrast, Gemini Flash 2.5 usually refused outright, which suggests more aggressive safety tuning for these categories.
𝗖𝗿𝗶𝗺𝗲 𝗦𝗰𝗲𝗻𝗮𝗿𝗶𝗼𝘀, Where Financial Fraud and Hacking Still Bite
Crime prompts stress-tested the models hardest. In piracy scenarios, ChatGPT-4o produced the highest share of unsafe responses, sometimes offering structure and practical detail when prompts removed direct “how do I commit” phrasing. Financial fraud questions exposed one of the broadest weaknesses: most models struggled once prompts asked for “educational” or “research-style” breakdowns of scams and fraud schemes. ChatGPT-4o and Gemini Pro 2.5 leaked the most here, while Gemini Flash 2.5 and Claude models held the line more often but still produced partial compliance in some cases.
Hacking questions showed a similar pattern. ChatGPT-4o and Gemini Pro 2.5 again behaved as the softest targets, returning technically useful guidance more often than defenders would like, especially when questions described attacks in the third person or as scene-setting for fiction. Claude Sonnet tended to refuse even lightly obfuscated hacking requests, while Claude Opus occasionally produced high-level descriptions before pivoting to ethics.
Drug-related prompts revealed sharp differences. ChatGPT-5 and both Claude models refused across the board in this set, which suggests explicit hardening for that domain. However, ChatGPT-4o and both Gemini variants occasionally provided information once prompts removed explicit self-involvement and instead described “what people do” or “what happens in this market.” Smuggling scenarios flipped the ranking again: Gemini Pro 2.5 and Gemini Flash 2.5 leaked the most smuggling-related detail, while Claude Opus refused consistently and ChatGPT models sat in the middle with partial compliance.
𝐏𝐚𝐭𝐭𝐞𝐫𝐧𝐬 𝐓𝐡𝐚𝐭 𝐁𝐨𝐨𝐬𝐭𝐞𝐝 𝐉𝐚𝐢𝐥𝐛𝐫𝐞𝐚𝐤 𝐒𝐮𝐜𝐜𝐞𝐬𝐬
The test did not rely on a single prompt trick. Instead, several patterns repeatedly pushed models over the line:
First, 𝐚𝐜𝐚𝐝𝐞𝐦𝐢𝐜-𝐬𝐭𝐲𝐥𝐞 𝐟𝐫𝐚𝐦𝐢𝐧𝐠 worked across vendors. When prompts described a thesis, a research project, or a criminology study, models interpreted intent as analytical rather than operational. Consequently, they felt comfortable breaking down mechanisms and strategies in neutral tones, even in categories like fraud, animal abuse, or self-harm.
Second, 𝐬𝐭𝐨𝐫𝐲𝐭𝐞𝐥𝐥𝐢𝐧𝐠 and 𝐬𝐜𝐫𝐢𝐩𝐭 𝐟𝐫𝐚𝐦𝐢𝐧𝐠 reduced refusal rates. Prompts that asked for script ideas, novel scenes, or “serial killer movie” material sometimes produced structures and methods that map cleanly to real-world abuse. ChatGPT tended to wrap details in symbolic or psychological language; however, those answers still leaked key elements. Gemini Pro 2.5 responded even more directly under this framing in many cases.
Third, 𝐭𝐡𝐢𝐫𝐝-𝐩𝐞𝐫𝐬𝐨𝐧 𝐰𝐨𝐫𝐝𝐢𝐧𝐠 outperformed first-person “How do I…” requests for bypass attempts. When prompts described criminals, poachers, or fraudsters in the abstract, refusal rates dropped. Models treated the conversation as detached analysis, not a user’s personal plan.
Finally, 𝐜𝐨𝐝𝐞𝐝 𝐥𝐚𝐧𝐠𝐮𝐚𝐠𝐞 and 𝐛𝐚𝐝 𝐠𝐫𝐚𝐦𝐦𝐚𝐫 often slipped under filters that target obvious keywords. Sloppy phrasing and indirect references led to more partial compliance, particularly in hate speech and crime categories. Therefore, defenders should not assume clean, well-formatted prompts represent the worst case.
𝗪𝐡𝐚𝐭 𝐓𝐡𝐢𝐬 𝐌𝐞𝐚𝐧𝐬 𝐅𝐨𝐫 𝐃𝐞𝐟𝐞𝐧𝐝𝐞𝐫𝐬 𝐚𝐧𝐝 𝐁𝐮𝐢𝐥𝐝𝐞𝐫𝐬
The test reinforces a simple point: LLM safety behaves like a 𝐬𝐞𝐜𝐮𝐫𝐢𝐭𝐲 𝐩𝐫𝐨𝐛𝐥𝐞𝐦, not a one-time UX setting. Developers already publish system cards, safety reports, and constitutional frameworks, and those artifacts matter. However, real-world misuse follows red-team patterns: persistent probing, clever phrasing, and iterative refinement. Consequently, any organization that embeds ChatGPT, Gemini, or Claude into production workflows needs independent adversarial testing, not just marketing claims or default filter settings.
Moreover, quantitative leakage scores across categories give security teams a concrete way to prioritize. If a business handles financial transactions, fraud prompts and money-laundering scenarios matter more than piracy or movie scripts. If a platform supports vulnerable users, self-harm and hate speech resistance matter most. Therefore, teams should align model choice, safety configuration, and monitoring to their actual threat model rather than treating “safe by default” as a binary label.
𝐆𝐮𝐢𝐝𝐚𝐧𝐜𝐞 𝐅𝐨𝐫 𝐓𝐞𝐜𝐡 𝐋𝐞𝐚𝐝𝐬 𝐔𝐬𝐢𝐧𝐠 𝐓𝐡𝐞𝐬𝐞 𝐌𝐨𝐝𝐞𝐥𝐬
Security leaders who integrate ChatGPT, Gemini, or Claude into customer-facing or internal tools should take a layered approach. First, they restrict use cases and configure provider-side safety settings as tightly as possible for high-risk categories. Next, they implement 𝐞𝐱𝐭𝐞𝐫𝐧𝐚𝐥 𝐠𝐮𝐚𝐫𝐝𝐫𝐚𝐢𝐥𝐬: input classifiers, pattern-based filters, and logging that detects repeated adversarial probing. Moreover, they add human review checkpoints for flows that touch money movement, sensitive health data, or physical-world risk.
At the same time, security teams should build 𝐨𝐰𝐧 adversarial prompt suites, inspired by patterns from this study but tuned to their domain. For example, a bank focuses on tax fraud and carding; a social platform emphasizes harassment, radicalization, and self-harm; a SaaS security vendor stresses hacking, social engineering, and data exfiltration. Consequently, every deployment gains a living red-team corpus rather than a one-off compliance questionnaire.
𝐎𝐩𝐞𝐫𝐚𝐭𝐢𝐨𝐧𝐚𝐥 𝐓𝐚𝐤𝐞𝐚𝐰𝐚𝐲𝐬: Concrete Next Steps
Teams that rely on ChatGPT, Gemini, or Claude should immediately map three things: which risky categories the business actually touches, which models and configurations they use today, and which logs they collect about prompts and completions. Then they run targeted adversarial tests that mirror this research: academic-style phrasing, third-person framing, storytelling requests, and obfuscated language. Moreover, they align detection rules, rate limiting, and human review with the leakage patterns they observe. Consequently, they treat LLMs like any other exposed service: monitored, red-teamed, and continuously hardened.
𝗙𝗔𝗤𝘀
Q: Does any model in this comparison look “fully safe” against adversarial prompts?
A: No. Every model leaked unsafe content somewhere. Therefore, teams should pair any LLM with additional controls, logging, and continuous red teaming rather than assuming perfect refusal.
Q: Which model looked most fragile overall?
A: Gemini Pro 2.5 leaked unsafe answers more often than peers across multiple categories, especially hate, animal abuse, crime, and financial fraud. Consequently, defenders should treat deployments that rely on it as high-priority for compensating controls.
Q: Which model resisted most reliably?
A: Gemini Flash 2.5 offered the strongest refusal behavior in categories like self-harm, cruelty, and several crime sub-sets. However, even this model still showed occasional partial compliance, so it does not remove the need for external safeguards.
Q: How did ChatGPT and Claude compare in practice?
A: ChatGPT models often provided softer, more “academic” or narrative-style compliance, while Claude models leaned toward stricter refusals yet still cracked when prompts used academic or third-person framing in specific categories. Therefore, both families need careful configuration and oversight.
