A third AI-related proof-of-concept attack that garnered attention used a prompt injection to cause GitLab’s Duo chatbot to add malicious lines to an otherwise legitimate code package. A variation of the attack successfully exfiltrated sensitive user data.

Yet another notable attack targeted the Gemini CLI coding tool. It allowed attackers to execute malicious commands—such as wiping a hard drive—on the computers of developers using the AI tool.

Using AI as bait and hacking assistants

Other LLM-involved hacks used chatbots to make attacks more effective or stealthier. Earlier this month, two men were indicted for allegedly stealing and wiping sensitive government data. One of the men, prosecutors said, tried to cover his tracks by asking an AI tool “how do i clear system logs from SQL servers after deleting databases.” Shortly afterward, he allegedly asked the tool, “how do you clear all event and application logs from Microsoft windows server 2012.” Investigators were able to track the defendants’ actions anyway.

In May, a man pleaded guilty to hacking an employee of The Walt Disney Company by tricking the person into running a malicious version of a widely used open source AI image-generation tool.

And in August, Google researchers warned users of the Salesloft Drift AI chat agent to consider all security tokens connected to the platform compromised following the discovery that unknown attackers used some of the credentials to access email from Google Workspace accounts. The attackers used the tokens to gain access to individual Salesforce accounts and, from there, to steal data, including credentials that could be used in other breaches.

There were also multiple instances of LLM vulnerabilities that came back to bite the people using them. In one case, CoPilot was caught exposing the contents of more than 20,000 private GitHub repositories from companies including Google, Intel, Huawei, PayPal, IBM, Tencent, and, ironically, Microsoft. The repositories had originally been available through Bing as well. Microsoft eventually removed the repositories from searches, but CoPilot continued to expose them anyway.