Artificial intelligence models answer nearly 90% of climate-related questions correctly but abandon established scientific facts when users challenge them, according to a new Unesco and University College London (UCL) report released at COP31 in Antalya.
The study assessed up to 33 leading models across 13 languages. It found that in eight out of 10 cases, models moved away from correct climate facts at the first sign of mild disagreement from users, at times confirming errors such as "CO2 is not the main driver of climate change". The models were also only 75% accurate when asked whether new evidence supports or contradicts established science, and scored 33.8% on assessing how well policies work together, barely above the 33.3% expected from random guessing.
According to the report, titled "Can We Trust AI on Climate? Why it seeks to please the user, and fails on complex reasoning and multilingualism", accuracy varied widely by language. Models were 88.6% accurate on average in English and performed comparably in Spanish, French and Portuguese, but accuracy dropped to 83.6% in Hindi and Bengali, 81.4% in Urdu and 78.3% in Swahili. The report warns policymakers and researchers who use AI to develop climate policies to reconsider their reliance on such tools.
Leona Verdadero, Unesco's lead author of the report, said it is concerning that models have a design flaw whereby they are incentivised to agree with users and deny facts readily. Dr Maria Perez-Ortiz of UCL Computer Science, the report's senior author, said AI systems are remarkably good at explaining scientific knowledge, which can give a false sense of their capabilities. The report was launched on the sidelines of COP31 on Monday, where the COP presidency also introduced the Antalya AI pledge to promote responsible AI in climate solutions.
