AI Fakes About Iran-US War Swirl On X Despite Policy Crackdown

AI-created videos circulating on Elon Musk's X depict American soldiers captured by Iran, an Israeli city in ruins, and US embassies ablaze -- a surge of lifelike deepfakes despite a policy crackdown to curb wartime disinformation.

The Middle East war has unleashed an avalanche of AI-generated visuals, dwarfing anything seen in previous conflicts and often leaving social media users unable to distinguish fabrication from reality, researchers say.

In a bid to protect "authentic information" during conflicts, X announced last week that it would suspend creators from its revenue sharing program for 90 days if they post AI-generated war videos without disclosing they were artificially made.

Subsequent violations will result in permanent suspension, X's head of product Nikita Bier warned in a post.

The new policy is a notable pivot for a platform heavily criticized for becoming a haven of disinformation since Musk completed his $44 billion acquisition of the site in October 2022.

It also won praise from senior State Department official Sarah Rogers, who called it a "great complement" to X's Community Notes -- a crowd-sourced verification system -- that results in "less reach (thus monetization)" for inaccurate content.

But disinformation researchers remain skeptical.

"The feeds I monitor are still flooded with AI-generated content about the war," Joe Bodnar of the Institute for Strategic Dialogue told AFP.

"It doesn't seem like creators have been dissuaded from pushing misleading AI-generated images and videos about the conflict," he said.

Bodnar pointed to a post from a premier "blue check" X account -- which is eligible for monetization -- that shared an AI clip depicting an Iranian "nuclear-capable" strike on Israel.

The post garnered more views than Bier's message about cracking down on AI content.

- Incentive for fakes -

X did not respond when AFP asked how many accounts it had demonetized since Bier's announcement.

AFP's global network of fact-checkers -- from Brazil to India -- identified a stream of AI fakes about the Middle East war, many from X's premium accounts with blue checkmarks that can be purchased.

They include AI videos depicting a tearful American soldier inside a bombed-out embassy, captured US troops on their knees beside Iranian flags, and a destroyed US navy fleet.

The flood of AI-fabricated visuals -- mixed with authentic imagery from the Middle East -- continues to grow faster than professional fact-checkers can debunk them.

Grok, X's own AI chatbot, appeared to make the problem worse, wrongly telling users seeking fact-checks that numerous AI visuals from the war were real.

Researchers have also warned that X's model -- allowing premium accounts to earn payouts based on engagement -- has turbocharged the financial incentive to peddle false or sensational content.

One premium account, which posted an AI video of Dubai's Burj Khalifa skyscraper engulfed in flames, ignored a request from Bier that it label the content as AI.

The post remained online, racking up more than two million views.

- 'Countermeasure' -

Last month, a report from the Tech Transparency Project said X appeared to be profiting from more than two dozen premium accounts belonging to Iranian government officials and state-controlled news outlets pushing propaganda, potentially in violation of US sanctions.

X subsequently removed blue checkmarks for some of them, the report said.

Even if X's demonetization policy were strictly enforced, a vast number of X users peddling AI content are not part of the revenue sharing program, researchers say.

Those users are still subject to being fact-checked through Community Notes, a system whose effectiveness has been repeatedly questioned by researchers.

Last year, a study by the Digital Democracy Institute of the Americas found more than 90 percent of X's Community Notes are never published, highlighting major limits.

"X's policy is a reasonable countermeasure to viral disinformation about the war. In principle, this policy reduces the incentive structure for those spreading disinformation," said Alexios Mantzarlis, director of the Security, Trust, and Safety Initiative at Cornell Tech.

"The devil will be in the implementing detail: Metadata on AI content can be removed and Community Notes are relatively rare," he said.

"It is unlikely that X will be able to guarantee both high precision and high recall for this policy."
-AFP