As Catherine Connolly took part in the final TV debate of the presidential campaign, an AI-generated fake version of her announcing her withdrawal from the race was circulating on social media.
Even experts in the field of AI deepfakes were taken aback by the high quality of the video, sparking fresh concerns over the threat to democracy that the technology now poses.
'Threat level has increased'
Aidan O'Brien is a researcher with the European Digital Media Observatory at Dublin City University. His work involves analysing deepfakes that are posted around the world. He believes the Catherine Connolly video was of a much higher quality compared to what has come before.
An AI-generated video showing a fake RTÉ News report claiming Catherine Connolly has withdrawn from the campaign.
"This was an exceptionally good piece of manipulated media," Mr O'Brien said.
"It involved not just deepfake but also "shallow fake" which is live, genuine footage intermixed with AI-generated footage. They've taken the trouble of getting the RTÉ graphics correct which is very unusual.
"They got the set correct which is something that would have required work, talent and craft. It was of far superior quality than any other deepfake we saw during this election," he added.
Mr O'Brien was also struck by the fact that names were spelt and pronounced correctly, something that AI deepfakes often get wrong.
"It’s very unusual to see names pronounced correctly. This would suggest someone with intimate knowledge of Irish politics and Irish media suggesting that it wasn’t made by somebody from outside of Ireland.
"The quality of this video does seem to indicate that the threat level has increased," he said.
'Surprisingly good quality'
Dr Barry Scannell is AI Law and Policy lead at William Fry and a member of the Government’s Artificial Intelligence Advisory Council.
He spotted the video in its first few hours of circulation on social media on Tuesday evening.
"I thought it was of surprisingly good quality to the point where it seemed so real to me that I actually went on the RTÉ website and other news sites to see if it was being reported elsewhere," Dr Scannell said.
"I think that's a very useful thing to do if you see a surprising story, try to verify it by going to other websites. When I went back to the source to look into it further I saw from the Facebook page that it was AI."
The people behind the account may have just been having some fun with the video or genuinely trying to disrupt the election.
Dr Scannell said it is difficult to speculate about the motivation.
"You have the entire spectrum of people using these tools, you could have somebody just thinking it's funny and stirring the pot.
"You could have someone with genuinely malicious intent and of course, there's the possibility, as we've seen in other elections around the world, where third-party countries use this technology to disrupt elections," he said.
Deepfakes in other countries
In the New York City mayoral race this week, Andrew Cuomo's campaign posted, and quickly deleted, an AI-generated video attacking his opponent Zohran Mamdani.
The ad, which has been branded as racist, depicts Mr Mamdani eating rice with his hands, and portrays his supporters as criminals.
"The video was a proposal that wasn’t approved, wasn’t finished and didn’t go through the normal legal process," a Cuomo spokesperson said.
"It was posted for five minutes before it was taken down."
But, as is always the case with social media, it was widely shared and saved before its removal.
In January 2024, voters in the US state of New Hampshire picked up their phones to hear what sounded like Joe Biden telling Democrats not to vote in the state's primary. It turned out to be an AI-generated voice.
In September 2023, fake audio clips of a Slovakian party leader were circulated just before the country's general election in which he could be heard talking about rigging the election and raising the cost of beer.
Michal Šimečka went on to lose the election sparking concerns that the AI-generated clips had cost him votes.
Isabel Linzer is an Elections and Democracy Fellow at the Centre for Democracy and Technology, a US nonpartisan, nonprofit digital rights organisation.
"We’ve only seen the tip of the iceberg when it comes to AI’s impact on elections," Ms Linzer said.
"The tech is getting better and politicians, and bad actors, are getting more comfortable using it. Anyone who thought the danger had passed after last year’s US election avoided any major AI incidents needs to wake up."
Ms Linzer believes that AI and social media companies need to prioritise election integrity year-round, not just during high-profile national elections.
"We’re a year out from the US midterms and other major international contests like Brazil’s presidential election, and without dedicated strategy and resources from companies, governments, and election administrators, we’ll be unprepared and voters will pay the price," she said.
Response of Irish regulators
An Coimisiún Toghcháin, the Electoral Commission, said it became aware of the Catherine Connolly deepfake video as it was circulating on Tuesday evening.
"Using pre-agreed communications channels with platforms, we escalated the issue immediately to flag these videos with a request that they be urgently reviewed, and I understand that the relevant accounts have since been suspended," a spokesperson said.
The media regulator, Coimisiún na Meán, made contact with Meta to understand the immediate measures it took in response to the video.
An Coimisiún said it also reminded the platform of its obligations under the EU Digital Services Act relating to protecting the integrity of elections.
Meta removed the video from Facebook within hours of it being posted but by then it had been shared widely.
It had attracted a mix of comments. Some dismissed the footage as fake but others appeared to think it was real. The video had originated from a Facebook account called 'RTÉ News AI'. Another post from the account included false news coverage about the high number of spoilt votes being cast on election day.
The account and the videos were removed from Facebook for violating Meta's Community Standards which do not allow content that impersonates or falsely represents people, brands, businesses or organisations.
"I think the response time was pretty good, in my experience platforms rarely respond to complaints and often ignore them," said researcher Aidan O'Brien.
"There is plenty of other AI generated content on these platforms available right now so it is surprising to see that Meta and YouTube acted so quickly on this one," he added.
Dr Barry Scannell believes the system worked well this week.
"It gives me great faith in the Irish democratic process and systems. It just shows that it's great to have the relationships with the big tech firms that are based in Dublin and have their European Headquarters here, having these channels of communication.
"Meta did respond quickly when they were alerted to this issue, but there are other websites and platforms that don't act in the same way and that could be problematic," he added.
The deepfake video remained available to view on X but the platform said it would be labelling the posts as "manipulated media" in accordance with its policies.
"Once these things break containment, once they’ve been posted online and shared you’ll never really be able to pull them back out," said Aidan O'Brien.
"This video is out of the Pandora’s box and we’re going to have to live with it now," he added.
Regulating AI
Governments are trying to police artificial intelligence with legislations like the EU AI Act but when you are dealing with a technology that is evolving at a record speed, regulators have been left struggling to keep up.
Guardrails that are meant to be in place were clearly breached in the making of the Catherine Connolly deepfake.
"Usually the mainstream AI models don't allow you to make deepfakes of known people at least without their consent permission. But there are models and systems available that do allow people to do this. I did find it shocking to see what can happen when you don't have a responsible AI company," said Dr Barry Scannell.
Had the content been different, the response slower, and the timing closer to election day, an AI deepfake video could have had a serious impact on this presidential race.
The events of this week remind us all that bad actors now have the ability to make something fake in the online world have major consequences in the real world.
Read more
Lack of deepfake laws leaving victims 'up against it' - expert
Government urged to pass laws banning AI 'deepfakes'
Deepfake videos of Humphreys removed by Meta
All you need to know about voice spoofing and audio deepfakes
Expert warns parents over AI deepfakes of children
