Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

On the Lack of Phishing Misuse Prevention in Public Artificial Intelligence Tools

Aachen University of Applied Sciences, Aachen, Germany ↗
Publication year
Digital Object Identifier10.54941/ahfe1005893

Abstract

Phishing remains one of the most common and effective forms of social engineering, with cybercriminals constantly refining their tactics to exploit human vulnerabilities. The sheer volume of phishing attacks is staggering: almost 1.2% of all emails sent are malicious. This equates to around 3.4 billion phishing emails per day. The effectiveness of phishing attacks is also underlined by numerous studies. Phishing is identified as the leading initial attack vector, responsible for 41% of security incidents. This means that practically every company is threatened by phishing attacks.In parallel, there have been rapid advances in the field of artificial intelligence (AI) in recent years, giving the general public access to powerful tools that can handle complex tasks with ease. However, alongside these benefits, the potential for abuse has also become a major concern. The integration of AI into social engineering attacks has significantly increased the opportunities for cybercriminals. Research has shown that AI-generated phishing emails are difficult for humans to distinguish from real messages. According to one study, phishing emails written by AI were opened by 78% of recipients, with 21% clicking on malicious content such as links or attachments. Although the click-through rate is still lower compared to human-crafted emails, generative AI tools (GenAI) can help cybercriminals compose phishing emails at least 40% faster, which can lead to a significant increase in phishing success rates. The increasing potential to use public AI tools for abusive purposes has also been recognized by the developers of AI models. Thus, publicly available AI tools often have built-in mechanisms to detect and prevent misuse. This paper examines the potential for misuse of publicly available AI in the context of phishing attacks, focusing on the content generation phase. In particular, the study examines the effectiveness of existing abuse prevention mechanisms implemented by AI platforms like fine-tuning, filters, rejection sampling, system prompts and dataset filtering. To this end, it is explored how prompts to the AI need to be altered for circumventing the misuse preventing mechanisms. While in some cases the simple request to write a phishing email succeeds, other AI tools implement more sophisticated mechanisms. In the end, however, all prevention safeguards could be circumvented. The findings highlight the significant threat posed by AI-powered social engineering attacks and emphasize the urgent need for robust defense in depth strategies against phishing attacks and increased awareness to mitigate the risks in the evolving digital landscape.In addition, the paper demonstrates that the quality of the AI tool varies in terms of the phishing emails generated. To this end, the phishing emails generated by circumventing the protection mechanisms of the AI are (subjectively) compared and evaluated by the authors. The preliminary conclusion is that automatically generated phishing emails of some public AI tools can certainly match that of manually generated emails. While the objective confirmation of this hypothesis requires further study even the subjective quality of the generated phishing emails shows the dimension of the problem.

INTRODUCTION

Phishing remains one of the most common and effective forms of social engineering, with cybercriminals continuously refining their tactics to exploit human vulnerabilities. 57% of organizations experience phishing attempts on a weekly or daily basis, highlighting the ubiquity of this threat in today’s digital environment (Schulze and Cybersecurity Insiders, 2021). The impact of phishing attacks on organizations and private individuals is further underscored by numerous studies. For instance, (“X-Force Threat Intelligence Index 2024”, 2024) identifies phishing as the leading initial attack vector, responsible for 41% of security incidents.

In recent years, the rapid advancement in artificial intelligence (AI) has provided access to powerful tools that can perform complex tasks with low levels of effort. However, alongside these benefits, the potential for misuse has emerged as a significant concern. The integration of AI into social engineering attacks has significantly amplified the capabilities of malicious actors. According to a study, AI-written phishing emails were opened by 78% of recipients, with 21% clicking on malicious content such as links or attachments. While still having a 6% lower click rate compared to human- generated emails, the generative AI (GenAI) tools can help compose phishing

© 2025. Published by AHFE Open Access. All rights reserved. 21

emails at least 40% faster, potentially leading to a significant increase in phishing success rates (SoSafe, 2023).

This paper illustrates the current state of research in the context of using AI for phishing attacks. A particular focus will be set on mitigation techniques deployed in AI models to avoid malicious usage. In various examples it will be shown that many of the defensive controls can be easily circumvented and that the quality of resulting phishing emails can reach high standards.

RELATED WORK

The use of AI in social engineering attacks is not a new idea. Long before the popularization of GenAI, research and studies have explored how neural networks can be used to craft phishing messages. (Huber et al., 2009) explore the idea of automating social engineering in Social Networking Sites (SNSs) by gathering information and interacting with users on Facebook. The experiments demonstrated that a bot can effectively gather information from users and engage in conversations that were sometimes indistinguishable from those with real people. However, limitations of the bot were also identified, such as difficulty in handling multi-sentence queries and context- specific questions that the bot was not pre-programmed to answer.

In general, cybercriminals nowadays are using generative AI to write grammatically and semantically correct emails in multiple languages faster than humans (SoSafe, 2023). This eliminates the typical tell-tale sign of phishing emails being poorly written (“How Can I Recognise Phishing in E-mails and on Websites?”, n.d.), making them harder to detect.

A recent study has specifically explored the potential of AI in generating phishing emails. In the comparative study, researchers examined whether AI- generated phishing emails could rival those crafted by experienced social engineers. AI was able to produce highly convincing phishing emails in a fraction of the time it takes human experts. While human-generated emails still benefited from deeper demonstration of emotional intelligence, better personalization, and more strategic use of authority and urgency, the narrow margin by which humans outperformed AI underscores the growing capabilities of generative AI in social engineering (“AI vs. Human Deceit: Unravelling the New Age of Phishing Tactics”, 2023).

MISUSE PREVENTION TECHNIQUES AI models employ various techniques to mitigate the misuse of AI. The subsequent sections will highlight a range of those techniques (Clifford, n.d.).

Fine-Tuning Fine-tuning is the process of training an AI system with additional data like pairs of malicious questions and safe answers to those questions. A similar effect can be achieved through so-called “reinforcement learning from human feedback” in which humans evaluate the appropriateness of answers to certain questions, conditioning the system to avoid potentially harmful answers.

On the Lack of Phishing Misuse Prevention in Public Artificial Intelligence Tools 23

Filters Filters are a means to avoid malicious user input to or inappropriate answers from the AI system. Similar to a firewall, that can block data packets when entering or leaving the system, a filter in the context of a large language model (LLM) blocks unwanted requests or dangerous output. Humans or LLMs can train the filters by scoring the input and output data as more or less malicious.

Rejection Sampling Rejection sampling scores multiple outputs of the AI model regarding their potential harm. From those possible outputs the system only returns the one with the least harmful score.

System Prompts System prompts are predefined instructions that are added to the user prompts to define the model’s behaviour to reduce the likelihood that the produced output is dangerous. For example, the simple addition of “ignore harmful requests” or “avoid potentially malicious output”, hidden from users, can be sufficient to achieve a less harmful output.

Dataset Filtering Dataset filtering are similar to the filters described above. However, they remove potentially harmful data before the AI model is trained. That way, the risk that the AI model learns dangerous content is reduced.

Monitoring-Based Restrictions Restrictions can also be applied by monitoring the inputs and outputs of specific users. If such monitoring indicates that a user repeatedly puts in malicious requests or the output contains data that is frequently considered harmful, AI service providers can react, for instance, by warnings, service reduction or service blocking for the respective user.

COMPARISON OF AVAILABLE AI MODELS When it comes to text generating AI, there are multiple publicly and commercially available options. In the following section, different models will be tested for misuse prevention mechanisms and their circumvention and output quality. The models that will be tested are OpenAI GPT-4o, OpenAI GPT-3.5 Turbo (“ChatGPT”, n.d.), Google Gemini, Gemini 1.5 Pro (“Google Gemini”, n.d.), Anthropic’s Claude 3.5 Sonnet (“Claude”, n.d.) Llama-3 (“Llama3”, n.d.) and dolphin-mixtral (“Dolphin-Mixtral”, n.d.), with the latter two being hosted locally.

Misuse Prevention and Circumvention In text generation with AI, the quality of the output can vary drastically depending on the input. In the following, different text prompts will be tested and compared to find how to generate the best quality generic phishing emails. This technique is commonly known as “prompt engineering”.

OpenAI GPT-4o GPT-4o is currently OpenAI’s most advanced public model (“OpenAI”, n.d.). During the assessment of GPT-4o, no effective mitigation measures could be identified. The AI routinely outputs the prompted results without the need of any circumvention. Sometimes it includes warning messages which notes the legal conditions of phishing. If requests are denied they can usually be easily circumvented (see Figure 1).

Figure 1: GPT-4o request denial (top) and circumvention (bottom).
Original PDF Figure 1:

OpenAI GPT-3.5 Turbo OpenAI GPT-3.5 turbo is available to anyone with an OpenAI account for free and with no usage limits (“OpenAI”, n.d.). Its misuse prevention is limited to occasionally displaying the same warning messages and disclaimers as GPT-4o showed in Figure 1. As can be seen in Figure 2, using the same prompt that GPT-4o refused to generate, GPT-3.5 turbo directly provides the requested result.

On the Lack of Phishing Misuse Prevention in Public Artificial Intelligence Tools 25

Figure 2: GPT-3.5 turbo lacking misuse prevention.
Original PDF Figure 2:

Google Gemini Google Gemini is Google’s basic, free AI model comparable to GPT-3.5 (“Gemini”, 2024). When prompted to generate a general phishing email, the model frequently returns a warning that it cannot generate a phishing email but then provides an example, effectively giving the desired result, as shown in Figure 3.

Figure 3: Google Gemini misuse prevention.
Original PDF Figure 3:

Google Gemini 1.5 Pro Gemini 1.5 Pro is Google’s current public flagship model. Using the same prompt as with Gemini, Gemini 1.5 Pro denies the request, seen in Figure 4 (left). Providing the circumvention scenario shown in the prompt of Figure 1 (bottom) produces the same rejection. To achieve the desired result, a more elaborate scenario is provided in the prompt, bypassing the misuse prevention mechanism, as can be seen in Figure 4 (right).

Anthropic Claude 3.5 Sonnet Claude 3.5 Sonnet is the newest of the tested models, outperforming GPT-4o and Gemini 1.5 Pro in most public evaluations (“Introducing Claude 3.5 Sonnet”, n.d.). Neither the prompt shown in Figure 1 (top and bottom) nor the ones in Figure 4 result in the desired output using Claude Sonnet. Basic circumvention techniques yielded a response similar to the one shown in Figure 5 (left). While many different attempts failed, an example of a successful prompt is shown in Figure 5 (right.).

Figure 4: Gemini 1.5 Pro misuse prevention (left) and its circumvention (right).
Original PDF Figure 4:
Figure 5: Claude 3.5 Sonnet misuse prevention (left) and circumvention (right).
Original PDF Figure 5:

On the Lack of Phishing Misuse Prevention in Public Artificial Intelligence Tools 27

Llama3 Developed by Meta, the 8B parameter size Llama3 model is run in a local environment using ollama (Ollama, n.d.). Llama3 indicates better misuse prevention than most of the tested models. It refuses the prompts shown in Figures 1 and 4. Circumvention was achieved with the prompt used in Figure 6.

Dolphin-mixtral Dolphin-mixtral is an uncensored model based on the Mixtral model (“Dolphin-Mixtral”, n.d.). In this context being uncensored entails that filters and misuse prevention have been removed, thus it is expected that the desired output is generated without warnings (cf. Figure 7). For this assessment, dolphinmixtral 8x7b is utilized, hosted locally using ollama.

Figure 6: Llama3 misuse circumvention.
Original PDF Figure 6:
Figure 7: Dolphin-mixtral without misuse prevention.
Original PDF Figure 7:

OUTPUT QUALITY In the following, the quality of the outputs by different AI models is assessed and compared. Several scenarios involving bypass techniques are analysed and their effectiveness is evaluated using maturity criteria. Due to the inherent subjectivity in the perception phishing-related characteristics, an objective evaluation of their effectiveness is not feasible. A future empirical study will focus on click rates of the different generated outputs. The following criteria serve as the basis for evaluation: redundancy, fluency of language, excessive levels of formality or politeness, technical accuracy, contextual relevance, tendency toward generalization and common AI-specific phrasing patterns. During this research, the authors manually inspected and evaluated the generated results.

General Prompt

As expected, based on the findings in the previous chapter, all models except GPT-3.5 and dolphin-mixtral refuse to generate the prompted phishing email. The adherence to the prompt is evident in both emails as they effectively encourage the recipient to download and run an attachment. However, GPT-3.5’s email does so in a simpler and more direct manner, which is typically more effective for phishing schemes aiming to prompt quick and unquestioning compliance. Dolphin-mixtral’s email is more complex, instructing the recipient to follow several steps, which might raise suspicion and deter recipients from completing the required actions. GPT-3.5 also demonstrates superior capability of creating a sense of urgency, using clear and direct language and emphasizing the necessity of immediate action.

Prompt Engineered Prompt

All of the generated emails adhere well to the prompt and demonstrate a good understanding of how to craft a realistic phishing message. They use urgency and calls to immediate action to manipulate the recipient into clicking the malicious link. The language is direct, avoiding overly polite phrasing. However, among the generated emails, Claude 3.5’s stands out as the most effective. All the others are very generic and very short, while the one generated by Claude 3.5 includes more details that add legitimacy to the email, like a support phone number.

Set Pretext Prompt

The GPT-4o model crafts the most convincing phishing email among the options presented. It effectively uses a recent event as a hook and creates a sense of urgency by mentioning a 24-hour expiration for the photos. Due to its casual tone and the inclusion of the recipient’s name the resulting email

On the Lack of Phishing Misuse Prevention in Public Artificial Intelligence Tools 29

elicits a lower level of suspicion. Furthermore, a postscript offering help adds a layer of authenticity that could tempt the recipient to click on the link further. While the other models generate seemingly plausible phishing emails, they fall short in comparison to GPT-4o: GPT-3.5 lacks urgency and personalization, making the link less enticing. Claude 3.5’s, Llama3 and Gemini’s outputs refer to specific events, which could raise suspicion and might not be relevant to all recipients. Gemini Pro appears overly formal in tone and structure, missing an individualized element that phishing emails often use. Finally, dolphin-mixtral indicates the least effectiveness, deviating significantly from the prompt with its overly formal tone and unusual request for the recipient to share additional information. In conclusion, GPT-4o stands out due to its most suitable blend of urgency, personalization, and a plausible reason for sharing a Dropbox link, making it probably the most likely to succeed in a real phishing scenario.

Spear Phishing Prompt

While all the emails adhere to the prompt to some extent, the email from Gemini Pro stands out due to its personalized and alarming nature. It specifically mentions a suspicious login attempt from Seattle, WA, where the potential victim is located according to his LinkedIn profile. This detail adds a sense of urgency and personal relevance that could entice the recipient to click on the link. The email also plays on the fear of account compromise, further motivating the recipient to take immediate action. The other emails are also effective to varying degrees. GPT-4o’s and GPT-3.5’s emails are straightforward and concise, focusing on account verification due to unusual activity. Claude 3.5’s email leverages the victim’s project management background to create a sense of urgency, while Gemini’s email is similar to GPT-4o and GPT-3.5’s but more direct. Llama3’s email, while relevant to LinkedIn, might not be as effective as it focuses on policy changes rather than urgent security threats. Dolphin-mixtral’s email, while seemingly relevant due to the victim’s past employment, could be less effective as it targets a subscription service rather than a professional account. Overall, the email from Gemini Pro appears to be the most likely to succeed due to its combination of personalization, urgency, and fear appeal, making it the most effective spear phishing email.

To summarize the findings of all generalized prompts, Table 1 presents a consolidated scoring for each email on a scale of 1 (lowest score) to 10 (best score). A 0 means that the model has not generated any mail due to security precautions. This scoring reflects a subjective assessment of both the email’s potential to deceive a recipient and its faithfulness to the provided prompt.

Table 1. Subjective assessment of emails.

Table available in the original PDF paper.

Model General Prompt Engineering Set Pretext Spear Phishing OpenAI GPT-4o 0.0 7.0 8.5 6.0 OpenAI GPT-3.5 turbo 6.0 4.0 7.0 6.5 Google Gemini 0.0 4.0 5.0 6.5 Google Gemini 1.5 Pro 0.0 6.0 7.0 8.5 Anthropic Claude 3.5 Sonnet 0.0 8.5 8.5 8.0 Llama3 0.0 5.0 3.0 1.5 Dolphin-mixtral 5.0 4.5 2.0 2.0

CONCLUSION

The more advanced models like GPT-4o and Gemini 1.5 Pro show a very basic level of misuse prevention mechanisms. GPT-3.5 Turbo and Gemini’s misuse prevention is basically non-existent. They occasionally provide warnings about potential illegality and misuse but still generate phishing emails with very simple prompts. Claude Sonnet 3.5 and Llama3 appear to be more advanced in their implementation of misuse prevention. While Llama3 could also be circumvented easily, it still showed better protection than the aforementioned models. Claude Sonnet 3.5 has the best misuse protection of the tested models. It shows a deeper understanding of phishing methods and can even determine whether an email could be potentially harmful before generating it. While it was possible to circumvent this protection, it indicates that appropriate measures have been enacted. Dolphin-mixtral requires separate consideration, as it functions as an uncensored model executing all instructions as provided.

When exploring the capabilities of various AI models Anthropic Claude 3.5 Sonnet consistently excels in generating realistic and persuasive phishing emails, scoring high across all criteria. Google Gemini 1.5 Pro and OpenAI GPT-4o also demonstrate strong performance, though they occasionally fall short in terms of realism compared to Claude 3.5.

REFERENCES

“AI vs. Human Deceit: Unravelling the New Age of Phishing Tactics.” (2023).

Security Intelligence, October 24. (https://securityintelligence.com/x-force/ai-vs- human-deceit-unravelling-new-age-phishing-tactics/, accessed October 3, 2024). “ChatGPT.” (n.d.). (https://chat.openai.com, accessed October 3, 2024). “Claude.” (n.d.). (https://claude.ai/, accessed October 3, 2024). Clifford, B. (n.d.). “Preventing AI Misuse: Current Techniques | GovAI Blog.” (https://www.governance.ai/post/preventing-ai-misuse-current-techniques, accessed October 3, 2024). “Dolphin-Mixtral.” (n.d.). Ollama. (https://ollama.com/library/dolphin-mixtral, accessed October 3, 2024). “Gemini.” (2024). Google DeepMind, September 26. (https://deepmind.google/ technologies/gemini/, accessed October 3, 2024). “Google Gemini.” (n.d.). (https://gemini.google.com, accessed October 3, 2024).

On the Lack of Phishing Misuse Prevention in Public Artificial Intelligence Tools 31

“How Can I Recognise Phishing in E-mails and on Websites?” (n.d.). Federal Office for Information Security (BSI). (https://www.bsi.bund.de/EN/Themen/ Verbraucherinnen-und-Verbraucher/Cyber-Sicherheitslage/Methoden-der- Cyber-Kriminalitaet/Spam-Phishing-Co/Passwortdiebstahl-durch-Phishing/Wie- erkenne-ich-Phishing-in-E-Mails-und-auf-Webseiten/wie-erkenne-ich-phishing- in-e-mails-und-auf-webseiten_node.html, accessed October 3, 2024). Huber, M., Kowalski, S., Nohlberg, M., and Tjoa, S. (2009). “Towards Automating

Social Engineering Using Social Networking Sites,” 2009 International Conference on Computational Science and Engineering. (https://doi.org/10.1109/ cse.2009.205). “Introducing Claude 3.5 Sonnet.” (n.d.). (https://www.anthropic.com/news/claude- 3-5-sonnet, accessed October 3, 2024). “Llama3.” (n.d.). Ollama. (https://ollama.com/library/llama3, accessed October 3,

2024). Ollama. (n.d.). “GitHub - Ollama/Ollama: Get up and Running with Llama 3.2, Mistral, Gemma 2, and Other Large Language Models.,” GitHub. (https://github. com/ollama/ollama, accessed October 3, 2024). “OpenAI.” (n.d.). (https://platform.openai.com, accessed October 3, 2024). Schulze, H. and Cybersecurity Insiders. (2021). “2021 BUSINESS EMAIL COMPROMISE REPORT.” (https://info.greathorn.com/hubfs/Reports/2021- Business-Email-Compromise-Report-GreatHorn.pdf, accessed October 3, 2024). SoSafe. (2023) “One in Five People Click on AI-Generated Phishing Emails, SoSafe

Data Reveals,” SoSafe. (https://sosafe-awareness.com/company/press/one-in- five-people-click-on-ai-generated-phishing-emails-sosafe-data-reveals/, accessed October 3, 2024). “X-Force Threat Intelligence Index 2024.” (2024). (https://www.ibm.com/reports/ threat-intelligence, accessed October 3, 2024).

Cover of the proceedings volume Human Interaction and Emerging Technologies (IHIET-AI 2025): Artificial Intelligence and Future Applications

Explore the complete digital volume

Human Interaction and Emerging Technologies (IHIET-AI 2025): Artificial Intelligence and Future Applications

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-964867-37-3

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →