THE biggest election year in the history of the world is under way, and we have just had our first glimpse at how artificial intelligence is being used by the shadowy world of government-backed hackers. These groups could have significant impacts on democratic processes through hacking, disinformation or leaking sensitive information. In February, Microsoft and OpenAI published a blog in which they identified groups affiliated with…

Researchers manipulated ChatGPT and five other commercial AI tools to create malicious code that could leak sensitive information from online databases, delete critical data or disrupt database cloud services in a first-of-its-kind demonstration.

The work has already led the companies responsible for some of the AI tools – including Baidu and OpenAI – to implement changes to prevent malicious users from taking advantage of the vulnerabilities.

“It’s the very first study to demonstrate that vulnerabilities of large language models in general can be exploited as an attack path to online commercial applications,” says Xutan Peng, who co-led the study while at the University of Sheffield in the UK.

Peng and his colleagues looked at six AI services that can translate human questions into the SQL programming language, which is commonly used to query computer databases. “Text-to-SQL” systems that rely on AI have become increasingly popular – even standalone AI chatbots, such as OpenAI’s ChatGPT, can generate SQL code that can be plugged into such databases.

The researchers showed how this AI-generated code can be made to include instructions to leak database information, which could open the door to future cyberattacks. It could also purge system databases that store authorised user profiles, including names and passwords, and overwhelm the cloud servers hosting the databases through a denial-of-service attack. Peng and his colleagues presented their work at the 34th IEEE International Symposium on Software Reliability Engineering on 10 October in Florence, Italy.

Their tests with OpenAI’s ChatGPT back in February 2023 revealed the standalone AI chatbot could generate SQL code that damaged databases. Even someone using ChatGPT to generate code in order to query a database for an innocent purpose – such as a nurse interacting with clinical records stored in a healthcare system database – might actually be given harmful SQL code that damaged the database.

“The code generated from these tools may be dangerous, but these tools may not even warn the user,” says Peng.

The researchers disclosed their findings to OpenAI. Their follow-up testing suggests that OpenAI has now updated ChatGPT to shut down the text-to-SQL issues.

Another demonstration showed similar vulnerabilities in Baidu-UNIT, an intelligent dialogue platform offered by the Chinese tech giant Baidu that automatically converts client requests written in Chinese into SQL queries for Baidu’s cloud service. After the researchers sent a disclosure report with their testing results to Baidu in November 2022, the company gave them a financial reward for finding the weaknesses and patched the system by February 2023.

But unlike ChatGPT and other AIs that rely on large language models – which can perform new tasks without much or any prior training – Baidu’s AI-powered service leans more heavily on prewritten rules to carry out its text-to-SQL conversions.

Text-to-SQL systems based on large language models seem to be more easily manipulated into creating malicious code than older AIs that rely on prewritten rules, says Peng. But he still sees promise in using large language models for helping humans query databases, even if he describes the security risks as having “long been underrated before our study”.

Neither OpenAI nor Baidu responded to a New Scientist request for comment on the research.

Topics:

ChatGPT can provide researchers with useful feedback on their papers, suggesting it could supplement the human peer review process that helps scientific journals decide which studies to publish. But others are sceptical that ChatGPT could play a role in peer review.

Peer review is a critical component of scientific publishing. However, many journals and research conferences are struggling to recruit enough human peer reviewers – who typically volunteer their time for free – to evaluate the growing number of submitted papers.

“It’s getting harder …

ChatGPT may be as good as or better than students at assessments in around a quarter of university courses. However, this generally only applies to questions with a clear answer that require memory recall, rather than critical analysis.

Yasir Zaki and his team at New York University Abu Dhabi in the United Arab Emirates contacted colleagues in other departments asking them to provide assessment questions from courses taught at the university, including computer science, psychology, political science and business.

These colleagues also provided real student answers to the questions. The questions were then run through the artificial intelligence chatbot ChatGPT, which supplied its own responses.

Next, both sets of responses were sent to a team of graders. “These graders were not made aware of the sources of these answers, nor were they aware of the purpose of the grading,” says Zaki.

In nine out of the 32 courses surveyed, ChatGPT’s answers were rated as good as or better than those of students. At times, its answers were substantially better. For example, it achieved almost double the average score of students when answering questions from a course called Introduction to Public Policy.

“ChatGPT performed much better on questions that required information recall, but performed poorly on questions which required critical analysis,” says Zaki.

The results highlight an issue with the way university assessments are set, says Thomas Lancaster at Imperial College London. They should probe students’ critical thinking, which may not be achieved by ChatGPT. “If [better answers are] possible [with ChatGPT], it suggests that there are flaws in the assessment design.”

Lancaster also says that many of the assessments that are susceptible to being cheated on via ChatGPT could have been vulnerable to existing contract cheating services. This is where students pay professional essay writers to do their work, but they may similarly not perform critical analysis.

Separately, Zaki and his team surveyed academics and students in the UK, US, India, Japan and Brazil about their attitudes towards ChatGPT. Across all of the countries, the students were more likely to say that they would use the chatbot than the academics thought they would.

Topics:

• artificial intelligence/
• ChatGPT

Hollywood actors strike over use of AI in films and other issues

Artificial intelligence can now create images, novels and source code from scratch. Except it isn’t really from scratch, because a vast amount of human-generated examples are needed to train these AI models – something that has angered artists, programmers and writers and led to a series of lawsuits.

Hollywood actors are the latest group of creatives to turn against AI. They fear that film studios could take control of their likeness and have them “star” in films without ever being on set, perhaps taking on roles they would rather avoid and uttering lines or acting out scenes they would find distasteful. Worse still, they might not get paid for it.

That is why the Screen Actors Guild and the American Federation of Television and Radio Artists (SAG-AFTRA) – which has 160,000 members – is on strike until it can negotiate AI rights with the studios.

At the same time, Netflix has come under fire from actors for a job listing for people with experience in AI, paying a salary up to $900,000.

AIs trained on AI-generated images produce glitches and blurs

Speaking of training data, we wrote last year that the proliferation of AI-generated images could be a problem if they ended up online in great numbers, as new AI models would hoover them up to train on. Experts warned that the end result would be worsening quality. At the risk of making an outdated reference, AI would slowly destroy itself, like a degraded photocopy of a photocopy of a photocopy.

Well, fast-forward a year and that seems to be precisely what is happening, leading another group of researchers to make the same warning. A team at Rice University in Texas found evidence that AI-generated images making their way into training data in large numbers slowly distorted the output. But there is hope: the researchers discovered that if the amount of those images was kept below a certain level, then this degradation could be staved off.

Is ChatGPT getting worse at maths problems?

Corrupted training data is just one way that AI can start to fall apart. One study this month claimed that ChatGPT was getting worse at mathematics problems. When asked to check if 500 numbers were prime, the version of GPT-4 released in March scored 98 per cent accuracy, but a version released in June scored just 2.4 per cent. Strangely, by comparison, GPT-3.5’s accuracy seemed to jump from just 7.4 per cent in March to almost 87 per cent in June.

Arvind Narayanan at Princeton University, who in another study found other changing performance levels, puts the problem down to “an unintended side effect of fine-tuning”. Basically, the creators of these models are tweaking them to make the outputs more reliable, accurate or – potentially – less computationally intensive in order to cut costs. And although this may improve some things, other tasks might suffer. The upshot is that, while AI might do something well now, a future version might perform significantly worse, and it may not be obvious why.

Using bigger AI training data sets may produce more racist results

It is an open secret that a lot of the advances in AI in recent years have simply come from scale: larger models, more training data and more computer power. This has made AIs expensive, unwieldy and hungry for resources, but has also made them far more capable.

Certainly, there is a lot of research going on to shrink AI models and make them more efficient, as well as work on more graceful methods to advance the field. But scale has been a big part of the game.

Now though, there is evidence that this could have serious downsides, including making models even more racist. Researchers ran experiments on two open-source data sets: one contained 400 million samples and the other had 2 billion. They found that models trained on the larger data set were more than twice as likely to associate Black female faces with a “criminal” category and five times more likely to associate Black male faces with being “criminal”.

Drones with AI targeting system claimed to be ‘better than human’

Earlier this year we covered the strange tale of the AI-powered drone that “killed” its operator to get to its intended target, which was complete nonsense. The story was quickly denied by the US Air Force, which did little to stop it being reported around the world regardless.

Now, we have fresh claims that AI models can do a better job of identifying targets than humans – although the details are too secretive to reveal, and therefore verify.

“It can check whether people are wearing a particular type of uniform, if they are carrying weapons and whether they are giving signs of surrendering,” says a spokesperson for the company behind the software. Let’s hope they are right and that AI can make a better job of waging war than it can identifying prime numbers.

If you enjoyed this AI news recap, try our special series where we explore the most pressing questions about artificial intelligence. Find them all here:

How does ChatGPT work? | What generative AI really means for the economy | The real risks posed by AI | How to use AI to make your life simpler | The scientific challenges AI is helping to crack | Can AI ever become conscious?

Topics:

The current whirlwind of interest in artificial intelligence is largely down to the sudden arrival of a new generation of AI-powered chatbots capable of startlingly human-like text-based conversations. The big change came last year, when OpenAI released ChatGPT. Overnight, millions gained access to an AI producing responses that are so uncannily fluent that it has been hard not to wonder if this heralds a turning point of some sort.

There has been no shortage of hype. Microsoft researchers given early access to GPT4, the latest version of the system behind ChatGPT, argued that it has already demonstrated “sparks” of the long-sought machine version of human intellectual ability known as artificial general intelligence (AGI). One Google engineer even went so far as to claim that one of the company’s AIs, known as LaMDA, was sentient. The naysayers, meanwhile, insist that these AIs are nowhere near as impressive as they seem.

All of which can make it hard to know quite what you should make of the new AI chatbots. Thankfully, things quickly become clearer when you get to grips with how they work and, with that in mind, the extent to which they “think” like us.

At the heart of all these chatbots is a large language model (LLM) – a statistical model, or a mathematical representation of data, that is designed to make predictions about which words are likely to appear together.

LLMs are created by feeding huge amounts of text to a class of algorithms called deep neural networks, which are loosely inspired by the brain. The models learn complex linguistic patterns by playing a simple game: …

The AI powering ChatGPT may provide completely different answers to the same mathematical problems over time. Those findings from recent experiments have fuelled an ongoing debate about whether the AI chatbot’s performance is getting worse – and have spurred the firm behind it, OpenAI, to reassure customers that applications built on ChatGPT will not continually break.

“The takeaway message is that the behaviour of the ‘same’ large language model can change substantially,” says Lingjiao Chen at Stanford University in California. “It is important …

The artificial intelligence chatbot ChatGPT could automate some aspects of AI training that currently rely on people. The chatbot can accurately classify and label text data used in training AI at a cost of just a third of a cent per work sample – about 20 times cheaper than crowdsourced human labour.

“We expect that in some tasks ChatGPT could replace humans,” says Fabrizio Gilardi at the University of Zürich in Switzerland. In other tasks, ChatGPT could help reduce the “amount of …

Plagiarism detection software that is already screening the work of more than 62 million students worldwide is getting a major upgrade to detect AI-generated writing – though some universities say they don’t want it. This comes several months after the release of the AI chatbot ChatGPT that can generate entire essays prompted major US school districts and universities around the world to ban its use.

It is unclear if AI-generated writing can be reliably detected. Academic researchers have shown that it …