Artificial intelligence has changed cybersecurity in, at the very least, two different ways. It has helped defenders detect threats faster, automate security monitoring, and respond to attacks more efficiently. But, at the same time, it has also given attackers new tools. Criminal groups can now generate convincing emails, clone voices, create fake videos, automate phishing campaigns, and imitate trusted people with very little technical skill. This creates a serious challenge for organisations because traditional security procedures were designed for human attackers, not for attackers supported by artificial intelligence.

One of the biggest problems is that many organisations are still preparing for older forms of cybercrime: they train employees to look for spelling mistakes, suspicious email addresses, and poor-quality scams. AI-assisted attacks often remove these warning signs, leading staff to trust fake messages, fake voices, or even fake video meetings because the attack appears professional and believable.

Imagine a large international company with offices across Europe and Asia. The company uses online meetings constantly because its employees work in different countries and time zones. Senior managers regularly approve payments through virtual calls, instant messaging systems, and email chains. Employees are encouraged to move quickly because delays can affect business operations.

A finance employee receives an email from the company’s chief financial officer. The message explains that a confidential acquisition is taking place and that several urgent transfers will be required over the next few hours. The employee notices that the request is unusual, but before they can question it, they are invited into a video meeting.

Inside the meeting are several senior staff members. The chief financial officer speaks calmly and explains that the transfers are sensitive because the company is trying to secure a competitive deal before another business can intervene. Other executives in the meeting agree with the instructions. They refer to real projects, real colleagues, and real internal procedures.

The employee follows the instructions and transfers millions of pounds into several accounts.

The problem is that none of the people in the meeting are real.

To see this in play, see the above video.

The attackers used artificial intelligence tools to create deepfake video and cloned audio. Public interviews, conference presentations, LinkedIn videos, and company webinars provided enough data for the criminals to reproduce the appearance and voices of senior executives. Generative AI systems helped produce realistic speech patterns, facial expressions, and responses during the meeting.

This attack succeeds because it targets trust rather than computer systems. Traditional cybersecurity focuses heavily on malware, network breaches, and software vulnerabilities. However, many AI-assisted attacks focus on human psychology. The attacker does not need to break through a firewall if they can persuade an employee to cooperate willingly.

This type of attack is known as social engineering. Social engineering manipulates people into revealing information or performing actions that benefit the attacker. AI dramatically strengthens social engineering because it allows criminals to imitate trusted identities at scale.

Several factors make this especially dangerous.

First, AI lowers the skill barrier for attackers. In the past, sophisticated fraud operations required experienced criminals with technical knowledge. Now, publicly available AI tools can generate convincing emails, realistic voice clones, and believable fake images within minutes. A criminal group no longer needs advanced programming expertise to create persuasive scams.

Second, AI increases speed and automation. An attacker can produce thousands of customised phishing emails that imitate the writing style of company executives. Large language models can analyse social media profiles and company websites to personalise messages for individual employees. This makes phishing campaigns more effective because the targets believe the messages are genuine.

Third, AI reduces obvious warning signs. Employees have historically been trained to spot grammatical errors, unusual wording, or poor formatting. Modern AI systems generate professional language that appears legitimate. Voice cloning technology can even reproduce accents, speech rhythms, and emotional tone.

Finally, organisations often rely too heavily on digital trust. Employees assume that a video call proves identity because people can see and hear each other. AI deepfakes challenge this assumption. A convincing fake video meeting can bypass procedures that were originally designed to prevent ordinary fraud.

The “what if?” scenario demonstrates a key issue in modern cybersecurity. The threat is not only technical. It is organisational. Companies may possess strong technical defences while still remaining vulnerable because their employees and procedures are not prepared for AI-assisted deception.

The hypothetical scenario described above is not science fiction. A very similar event took place in 2024 involving the British engineering company, Arup.

In January 2024, an employee in Arup’s Hong Kong office received a suspicious message that appeared to come from senior management. The employee was then invited into a video conference call where they believed they were speaking to the company’s chief financial officer and several colleagues. The individuals in the meeting looked and sounded real.

They were not real.

Criminals used AI-generated deepfake technology to imitate company executives and staff members. During the meeting, the fake executives instructed the employee to transfer funds into several bank accounts. The employee eventually completed 15 separate transactions worth approximately HK$200 million, equivalent to roughly £20 million or $25 million.

The attack became internationally significant because it demonstrated how AI could be used in a large-scale financial fraud operation. According to reports, Hong Kong police described the incident as one of the first known cases in the region involving a fully AI-generated multi-person video conference used for fraud.

The incident highlighted several weaknesses in organisational preparation.

The first weakness was overconfidence in visual communication. Video meetings have become normal in modern workplaces, especially after the expansion of remote and hybrid work. Employees generally assume that seeing someone’s face and hearing their voice provides reliable proof of identity. AI deepfake systems challenge this assumption directly.

The second weakness involved verification procedures. The employee reportedly became suspicious at first, but the presence of multiple apparent colleagues during the video call reduced those concerns. This shows how AI can create a false sense of collective trust—the attack did not depend on only one fake identity, but, rather, it relied on a complete simulated meeting environment.

The third weakness was organisational readiness. Many companies have cybersecurity awareness training, but most traditional training focuses on older threats such as phishing emails or suspicious links. Employees are not always prepared for realistic AI-generated impersonation attacks. An organisation may therefore believe it has strong cyber awareness while still being unprepared for AI-enhanced fraud.

The Arup incident also demonstrates how rapidly the threat landscape is changing. Deepfake technology has improved significantly in a short period of time. Earlier deepfakes were often easy to detect because facial movements looked unnatural or speech patterns sounded robotic. Modern AI systems are far more convincing. They can generate real-time audio and video responses during live conversations.

Another important issue is the availability of training data. Senior executives often appear in interviews, webinars, conference recordings, podcasts, and social media videos. All of this public material can be collected and analysed by AI systems. Attackers can therefore build convincing digital copies of company leaders using information that is already publicly available.

The attack also shows how cybersecurity increasingly overlaps with business operations and corporate culture. If employees are trained to prioritise speed, secrecy, and obedience to senior management, they may become easier targets for social engineering. Attackers understand this. They often create a sense of urgency because urgency reduces critical thinking.

Importantly, the Arup incident was not primarily a failure of antivirus software or network security systems. The attackers manipulated trust relationships within the organisation. This represents a broader shift in cybersecurity threats. AI allows attackers to scale psychological manipulation in ways that were previously difficult or expensive.

The case also attracted wider attention because experts recognised that similar attacks could affect governments, banks, healthcare systems, and infrastructure operators. An AI-assisted attacker might imitate a senior official during a crisis, authorise fraudulent payments, or distribute false instructions. The danger is not limited to financial loss. Deepfake technology could potentially disrupt emergency responses, elections, or public communications.

The real lesson from the Arup case is that organisations cannot rely on old assumptions about identity verification. A familiar face on a screen is no longer enough.

The rise of AI-assisted cybercrime means that organisations need practical response strategies rather than simple awareness campaigns. Traditional cybersecurity guidance is no longer sufficient on its own because the threat environment changes rapidly.

One of the most important solutions is the development of operational playbooks. A cybersecurity playbook is a structured set of procedures that explains how staff should respond to specific threats or incidents. Instead of relying on individual judgement during stressful situations, employees follow predefined steps. In the context of AI-assisted attacks, playbooks are essential because attackers exploit confusion, urgency, and uncertainty. Clear procedures reduce the likelihood of impulsive decisions.

A modern AI-threat playbook should begin with identity verification procedures. Organisations should establish rules that no major financial transfer or sensitive action can be authorised solely through email, messaging platforms, or video calls. Independent verification methods should always be required.

For example, a company could require employees to confirm requests through a secondary communication channel. If a financial instruction arrives during a video meeting, the employee must separately contact the executive using a verified internal number or secure authentication system. This is sometimes called out-of-band verification. Multi-person approval systems are also important. Large transfers or critical operational changes should require approval from several individuals rather than one employee acting alone. This reduces the effectiveness of social engineering because attackers must deceive multiple people simultaneously.

Playbooks should also include escalation procedures. Employees need permission to challenge suspicious requests, even when they appear to come from senior leadership. In some organisations, staff may fear disciplinary action if they delay an executive request. Attackers take advantage of this power imbalance. Cybersecurity training must evolve as well. Many awareness programmes still focus heavily on outdated phishing examples. Training should now include realistic simulations involving AI-generated voice messages, cloned video calls, and advanced impersonation attempts. Employees need experience recognising how these attacks operate.

Another important measure is digital footprint management. Companies should review how much executive audio and video content is publicly available online. Completely removing public content is unrealistic, but organisations can reduce unnecessary exposure and educate executives about the risks of voice and facial data collection. Technical defences also remain important. Security teams are developing AI detection systems that analyse facial movement, speech irregularities, and metadata to identify deepfakes. However, detection technology alone is unlikely to solve the problem completely because AI generation tools continue to improve.

This means organisations must combine technical security with procedural security. The strongest defence is not simply better software. It is a system where employees, policies, and technology work together. Governments and regulators also have a role to play. Financial institutions, infrastructure operators, and public agencies may require updated standards for identity verification and incident reporting. International cooperation will become increasingly important because many AI-assisted cybercrimes involve attackers operating across multiple countries.

There is also a broader cultural issue. Organisations must avoid treating cybersecurity as only the responsibility of IT departments. AI-assisted attacks often target finance staff, human resources teams, executives, and customer service employees. Cybersecurity therefore becomes an organisation-wide responsibility. The speed of AI development creates an additional challenge. Companies cannot rely on static policies that remain unchanged for years. Playbooks need continuous review and testing because attackers adapt quickly. A procedure that works today may become ineffective within a short period of time.

Scenario exercises are particularly valuable. Organisations should run simulated incidents where staff respond to deepfake calls or AI-generated instructions. These exercises expose weaknesses before real attackers can exploit them. Importantly, the goal is not to eliminate trust completely. Modern organisations depend on communication and cooperation. Instead, the objective is to create systems where trust is supported by verification.

The Arup case demonstrates that AI-assisted cybercrime is no longer a future possibility. It is a present reality. Attackers are already using artificial intelligence to manipulate employees, imitate executives, and bypass traditional safeguards. As AI systems become more advanced, these attacks will likely become cheaper, faster, and more convincing. Organisations that continue relying on outdated assumptions about identity and communication will remain vulnerable.

The solution is preparation. Effective cybersecurity in the age of AI requires updated playbooks, stronger verification systems, realistic employee training, and a recognition that social engineering has entered a new phase. Companies must prepare not only for attackers who target computers, but also for attackers who target human trust itself.

Artificial intelligence has transformed cybersecurity into a contest between increasingly sophisticated attackers and increasingly adaptive defenders. The organisations that respond successfully will be those that recognise that technology alone is not enough. Procedures, culture, and preparation are now just as important as software and hardware in defending against cyber threats.

What’s needed now is a community-driven, continuously updated resource. A living document that captures:

Real-world attack patterns

Proven defensive architectures

Tooling evaluations and integrations

Red teaming methodologies

Incident response case studies

If you are working with LLM systems—whether in engineering, security, or product—your insights are valuable. Contribute examples, share failures, document mitigations. The faster we codify collective knowledge, the faster we raise the baseline. Prompt injection is not a problem that any single team will solve in isolation. It requires the same kind of collaborative defence that ultimately matured web security.

The question is not whether prompt injection will be exploited at scale because it already is. The question is concerned with the way we build the playbooks, tools, and share expertise fast enough to stay ahead.

Copyright (C) 2025 Packt Publishing. All rights reserved. 

Our mailing address is: 

Packt Publishing, Grosvenor House, 

11 St Paul's Square, Birmingham, 

West Midlands, B3 1RB, United Kingdom 

Want to change how you receive these emails? 

You can update your preferences or unsubscribe.