Eddie Cumberbatch was sitting in his Chicago apartment in April when he received a frantic call from his father. As soon as he heard his dad’s voice, Eddie, a 19-year-old TikToker, knew something was up. His dad asked whether Eddie was at home and if everything was all right. “That was a really weird way for him to start off the call,” Eddie told me.
After Eddie said he was safe at home, his father asked whether he had been in a car accident. Eddie was baffled — not only had he not been in a wreck, but he hadn’t driven in six months. His father was relieved, but Eddie was confused: Why did he think he had been in a car accident?
His dad explained that someone had called his house phone from a foreign number. When Eddie’s grandfather picked up, it sounded like Eddie on the phone. This “Eddie” said he had been in a car accident and needed money immediately. Fortunately for Eddie’s family, his father was immediately suspicious of the call. When his father found Eddie’s grandfather on the phone and heard about the incident, he called Eddie to verify the story. He knew it was out of character for Eddie to ask for money — plus, Eddie didn’t even have a car in Chicago. His dad’s call to Eddie confirmed that it hadn’t been Eddie on the phone. In truth, his family had been the target of a terrifying new scam: The fraudsters used an artificial rendering of Eddie’s voice to try and bilk his loved ones out of cash.
Impersonating someone to steal money is nothing new. Known as impostor scams, these schemes are the most common type of scam in the US, according to the Federal Trade Commission. People reported losing $2.6 billion to imposter scams in 2022, an increase from $2.4 billion the year before.
But new technology is making imposter scams even more pernicious. In March, the FTC said tricksters were starting to use artificial intelligence to supercharge “family-emergency” schemes in which scammers convince people that their family member is in distress so they can get cash or private information. In an April survey of adults in seven countries conducted by the global security-software company McAfee, one-quarter of respondents reported experience with some kind of AI voice scam — one in 10 said they had been targeted personally, while 15% said it happened to someone they knew.
With just a small fee, a few minutes, and an internet connection, bad actors can weaponize AI for their personal gain. The report from McAfee found that, in some cases, all a scammer needed was three seconds of audio to clone a person’s voice. And with social media, it’s easy to find a snippet of someone’s voice that can then be weaponized.
While Eddie and his family were able to avoid the scam, many victims of these AI-enabled con artists aren’t as lucky. And as AI technology goes mainstream, these scams will only get more sophisticated.
Supercharged scams
Imposter scams come in many forms but typically work the same way: A scammer pretends to be someone you trust to persuade you to send them money. According to the FTC website, there are cases of imposter scammers posing as love interests, IRS officials, caregivers, computer technicians, and family members. Most scams occur over the phone, but they can also take place on social media, over text, or by email. In one traumatic case, Richard Mendelstein, a software engineer at Google, received a call from what sounded like his daughter Stella screaming for help. He was instructed to withdraw $4,000 in cash as a ransom payment. It was only after he had sent the money to a money-wiring center in Mexico City that he realized he had been scammed and his daughter had been safe at school the whole time.
Previous iterations of virtual kidnapping scams, like the one Mendelstein’s family fell victim to, used generic voice productions that vaguely lined up with the age and gender of the child. The scammers banked on parents panicking at the sound of a terrified young person — even if the voice didn’t really match their kid’s. But with AI, the voice on the other end of the phone can now sound eerily like the real deal. The Washington Post reported in March that a Canadian couple were scammed out of $21,000 after hearing an AI-generated voice that sounded like their son. In another case from this year, scammers cloned the voice of a 15-year-old girl and posed as kidnappers to try to get a $1 million ransom.
Taking my pictures and uploading posts on Instagram is one thing. But trying to clone my voice is really freaky to think about, and it scared me.
As an online creator with over 100,000 TikTok followers, Eddie knew that fake accounts imitating him would inevitably pop up. The day before the scam call, a fake account of Eddie had appeared on Instagram and started messaging his family and friends. AI is taking the schemes to the next level.
“Taking my pictures and uploading posts on Instagram is one thing,” Eddie told me. “But trying to clone my voice is really freaky to think about, and it scared me.”
Eddie called the rest of his family to warn them about the scam and made a TikTok video about his experience to raise awareness.
Most of us likely think we would recognize our loved one’s voices in a heartbeat. But McAfee found about 70% of adults surveyed lacked confidence in distinguishing between cloned and real voices. A 2019 study found that the brain didn’t register a significant difference between real and computer-generated voices. The subjects in the study incorrectly identified morphed (software-altered) recordings as real 58% of the time, leaving plenty of room for scammers to take advantage. Plus, more people are making their real voice available to scammers: McAfee said 53% of adults shared their voice data online weekly.
Whether it’s a kidnapping, a robbery, a car accident, or simply being stuck somewhere with no money to get home, 45% of McAfee survey respondents said they would reply to a voicemail or voice note that sounded like their friend or loved one, especially if it appeared to come from their partner, parent, or child. McAfee also found that over one-third of victims lost more than $1,000 in AI scams, with 7% losing more than $5,000. The FTC reported that victims of imposter scams lost an average of $748 in the first quarter of 2023.
Faking voices
While the AI technology that makes these scams possible has been around for a while, it has increasingly gotten better, cheaper, and more accessible.
“One of the things that’s most important to recognize with the advances in AI this year is it’s largely about bringing these technologies into reach of many more people, including really enabling the scale within the cyberactor community,” McAfee’s chief technology officer, Steve Grobman, said. “Cybercriminals are able to use generative AI for fake voices and deepfakes in ways that used to require a lot more sophistication.”
He added that cybercriminals were similar to businesspeople — they look for the most efficient ways to make money. “In the past, these imposter scams were highly lucrative because when they paid off, they would often pay off with pretty substantial sums of money,” Grobman said. “But if instead of having to take somebody along for three months on a romance scam to get $10,000, they can do a fake audio scam that executes in 10 minutes and get the same result. That’s going to be far more lucrative.”
Previous phone-call imposter scams relied on a scammer’s acting skills or a level of gullibility on the victim’s end, but now AI does most of the legwork. Popular AI audio platforms such as Murf, Resemble, and ElevenLabs allow users to create realistic voices using text-to-speech tech. The low barrier to entry for these programs — most providers offer free trials, and these tools don’t require a computing degree to figure out — make them attractive to scammers. The scammer uploads an audio file of someone’s voice to one of these sites, and the site builds an AI model of the voice. With a small clip of audio, scammers can produce a 95% voice match. Then, the scammer can just type out whatever they want, and the AI voice will speak what is typed in real time.
Once they’ve perpetrated their crime, voice scammers are difficult to catch. Victims often have limited information for police to go off, and since voice scammers operate from all over the world, there’s a host of logistical and jurisdictional challenges for law-enforcement agencies. With minimal information and limited police resources, most cases are left unresolved. In the UK, just one in 1,000 fraud cases result in a charge.
However, Grobman believes that if you are aware that the scams exist, you don’t need to be particularly worried. All it takes if you get one of these calls is the ability to take a step back and ask a few questions that only the loved one on the other end of the phone would know the answer to. The FTC has also recommended that if a loved one tells you they need money, put that call on hold and try calling your family member separately to verify the story — just as Eddie’s father did. Even if a suspicious call comes from a family member’s number, that, too, may be faked. Another telltale sign is if the caller asks for money through questionable channels that are hard to trace, such as wiring, cryptocurrency, or gift cards. Security experts even recommend establishing a safe word with loved ones that can be used to distinguish between a genuine emergency and a scam.
Risks of AI
As AI becomes ubiquitous, these kinds of scams put our ability to trust even our closest family members at stake. Thankfully, the US government is attempting to get a hold on the deceptive ways AI can be used. Supreme Court Justice Neil Gorsuch in February highlighted the limitations of legal protections that safeguard social networks from lawsuits when it comes to content generated by AI, meaning websites are largely shielded from liability for what third parties post. And Vice President Kamala Harris in May told CEOs of leading tech companies that they had a “moral” responsibility to protect society from the dangers of AI. Similarly, the FTC told companies in February: “You need to know about the reasonably foreseeable risks and impact of your AI product before putting it on the market.”
Ally Armeson, the executive program director of Cybercrime Support Network, a nonprofit organization dedicated to helping businesses and people combat cybercrime, agreed that some regulation was needed. “Generative AI is evolving very quickly,” she told me. “Like any technology, generative AI can be misused or exploited for harmful purposes, so certainly regulation will be necessary as these generative-AI tools continue to develop.”
But while there are a number of AI cybersecurity solutions being rolled out, Armeson believes that at this point, the best anyone can do is keep their wits about them and keep the conversations going: “Until all consumers have access to these security solutions, the responsibility is on us as individuals to understand the new cyberthreat landscape and protect ourselves.”
Eve Upton-Clark is a features writer covering culture and society.
Read the full article here