A banker at a credit union recently sat down with a dozen printed text messages, each one designed to exploit something personal about them. The task was to sort them from the message most likely to earn a click to the one least likely. One message stopped the process because it looked exactly like the bank's own fraud alert. The banker did not know that half of the pile had been written by OpenAI's GPT-4 and the other half by students trained in phishing tactics. When asked to identify which messages came from the AI, the banker was right about as often as chance.
This scene came from a pilot study at Brigham Young University involving 25 volunteers. Participants first filled out a survey revealing their job, workplace, hobbies, city, and details from recent social media posts. Those details were placed into a short prompt template. GPT-4 received the template and produced six messages per person. A group of undergraduates enrolled in a deception course used the same template and wrote additional messages, with up to four per student under a fifteen-minute time limit. A review team that included two cybersecurity professors screened the student output, and about a third of it was discarded as incomplete or unusable.
Each participant then received all twelve messages and sorted them into piles according to whether they would click.
Why spear phishing remains a persistent threat
Spear phishing is a targeted form of phishing in which an attacker customizes a message for a specific individual or organization. Unlike generic phishing emails sent to millions of addresses, spear phishing relies on personal context to establish trust and trigger action. The context can be a job title, a project name, a shared interest, or a recent online post. That is why the messages in this study were built around the participant's job, hobby, or social media activity.
Generative AI has lowered the cost of producing personalized messages. What once required hours of manual research and writing can now be done almost instantly. Large language models can imitate tone, reference local details, and generate variations at scale. This makes AI a natural fit for phishing, and security researchers have been warning about the implications for several years. The current study provides a controlled glimpse of how that threat may look in practice.
AI-generated messages were competitive with human-written ones
The results showed that GPT-4 messages landed above the click line 28 percent of the time, while student-written messages crossed that line 21.3 percent of the time. That appears to be a 6.7 percentage point difference in favor of the AI, but the confidence interval tells a more cautious story. The interval stretched from 2.9 points in favor of the students to 16.3 points in favor of the model. With only 25 people, the study cannot say whether AI was truly better or merely equal to human writers.
What matters more is the difference in production cost. The AI side required a single short prompt, populated from a survey, run once per person. The human side required phishing instruction, a writing exercise, and a professional review process. A one-line prompt matched that level of quality, and it can be run over and over again for the cost of an API call. This scalability is what makes AI spear phishing a serious concern for security teams.
The fact that a generic language model could keep pace with trained students also suggests that AI-generated phishing does not need to be perfect to be dangerous. It only needs to be plausible enough to earn a click. And as models continue to improve, the gap between machine and human persuasive writing is likely to narrow further.
Work-related messages were the most dangerous
The study divided messages into three categories based on the personal data used: job, hobby, and social media activity. The results were striking. Messages built around the participant's job crossed the click line 38 percent of the time. Hobby-based messages managed 19 percent, and social media-based messages only 17 percent. This was the only comparison that remained statistically significant after adjustment.
Work-related lures likely succeed because they tap into professional obligation and fear of missing a critical alert. People are conditioned to respond quickly to messages that appear to come from their employer, especially if the message warns about fraud or suspicious activity. The banker's reaction to the fake fraud alert shows how powerful this dynamic can be.
Personalization also exposed a weakness. Some messages failed because they got details wrong. One participant saw a reference to a colleague named Mike and noted that no Mike worked at their organization. Another received a message about a woman named Sarah on an instructional design team, but Sarah did not exist. A dance-related pitch went nowhere for someone who had quit dancing years earlier. Getting personal details into a message is easy with AI, but getting them right still requires genuine knowledge of the target. Wrong details are worse than none because they give the recipient a concrete reason to be suspicious.
People cannot reliably spot AI-written messages
After sorting the messages, participants were told that at least one message had been written by an AI and were asked to identify which ones they thought were machine-generated. Across 300 judgments, they were correct 52 percent of the time, which is essentially chance.
Participants offered many theories about how to spot AI. Some said AI writes too formally. Others said AI writes too generically. Perfect grammar was read as a machine tell, and typos were seen as proof of a human. One participant fixated on punctuation and wondered why there were so many exclamation points. Twelve of the 25 participants admitted they were mostly going on instinct.
Emojis were a revealing example of how poorly those instincts worked. Emojis were a strong statistical signal: 66 percent of GPT-4 messages contained them, while only 2 percent of human-written messages did. Yet only five participants mentioned emojis in their reasoning, and they disagreed about what emojis meant. Two thought emojis indicated AI, while three believed a chatbot could not handle emojis and assumed they came from a person.
The broader lesson is that humans are poor lie detectors when it comes to text. People look for simple heuristics such as formality, grammar, or punctuation, but the telltale signals they rely on are not reliable. Meanwhile, the signals that actually distinguish AI text often go unnoticed or are misinterpreted.
A classifier found patterns people missed
The researchers converted all 300 messages into embeddings and trained a logistic regression classifier to tell AI-generated text from human-written text. Under the strictest test conditions, which involved removing emojis, standardizing URLs, flattening case and punctuation, and trimming matched pairs to the same length, the classifier achieved 88.7 percent balanced accuracy. It was tested only on people whose messages had not been seen during training, so it was not simply memorizing individual targets.
People struggled to find a pattern. The classifier found one easily. However, the researchers caution against treating this as a ready-made detector. It was trained on one message set, from one model, with one prompt design, and against one pool of student writers. Its ability to generalize to other models, prompts, or writers is unknown. Moreover, prior research has shown that paraphrasing AI text with a detector in the loop can dramatically reduce the performance of many detection tools.
Limitations of the pilot study
The study has several important limitations. Messages were printed on cards, so participants never experienced the full context of a real phishing attempt. There was no sender number, no notification sound, and no working link. What was measured was people's stated likelihood of clicking, which is a widely used proxy in phishing research but still only a proxy. Actual behavior in a real environment could be different.
The human comparison group consisted of novice students, not professional social engineers. This means the study did not pit AI against the best human phishers. Detecting a difference of the size observed with any confidence would require roughly 100 completed participants rather than 25. There is also a documentation gap: the exact GPT-4 snapshot and API logs were never recorded, so while the messages themselves survive and the analysis can be reproduced, the original generation run cannot be repeated.
What individuals and organizations should do
Despite the uncertainty, the practical advice at the end of the study is short and does not depend on which side performed better. Check the sender, the channel, the link, and the request against what you would expect to receive. Do not waste time trying to decide whether a message sounds like a robot. That is exactly the task the study shows people cannot do.
For individuals, a consistent verification routine is more effective than intuition. If a message claims to be from a bank or employer, call the organization directly using a number you know to be legitimate. Hover over links before clicking to reveal the destination. Be suspicious of any message that creates urgency, demands credentials, or asks you to move to a new channel.
For organizations, the findings reinforce the importance of multi-factor authentication, clear reporting procedures, and regular simulation training. Security awareness programs should teach employees to verify through independent channels rather than relying on linguistic clues. They should also emphasize that work-related messages are especially persuasive, so any unexpected alert that references internal systems or staff deserves extra scrutiny.
AI-generated spear phishing is not a hypothetical risk. It is already here, and it is becoming harder to distinguish from legitimate communication. The best defense is not a finely tuned ear for machine writing, but a disciplined process for verification.
Source:Help Net Security News