Promise and problems: how do we implement artificial intelligence in clinical settings and ensure patient safety?

In a classic case of finding a balance between costs and benefits of science, researchers are grappling with the question of how artificial intelligence in medicine can and should be applied to clinical patient care – despite knowing that there are examples where it puts patients’ lives at risk.

The question was central to a recent university of Adelaide seminar, part of the Research Tuesdays lecture series, titled “Antidote AI.”

As artificial intelligence grows in sophistication and usefulness, we have begun to see it appearing more and more in everyday life. From AI traffic control and ecological studies, to machine learning finding the origins of a Martian meteorite and reading Arnhem Land rock art, the possibilities seem endless for AI research.

Perhaps some of the most promising and controversial uses for artificial intelligence lie in the medical field.

The genuine excitement clinicians and artificial intelligence researchers feel for the prospect of AI assisting in patient care is palpable and honourable. Medicine is, after all, about helping people and the ethical foundation is “do no harm.” AI is surely part of the equation for advancing our ability to treat patients in the future.

AI is surely part of the equation for advancing our ability to treat patients in the future.

Khalia Primer, who is a PhD candidate at the Adelaide Medical School, points to many areas of medicine where AI is already making waves. “AI systems are discovering crucial health risks, detecting lung cancer, diagnosing diabetes, classifying skin disorders and determining the best drugs to fight neurological disease.

“We may not need to worry about a rise of the radiology machines, but what safety concerns do have to be considered when machine learning meets medical science? What risks and potential harms should healthcare workers be aware of and what solutions can we get on the table to make sure that this exciting field continues to develop?” Primer asks.

These challenges are compounded, Primer says, by the fact “the regulatory environment has struggled to keep up” and “AI training for healthcare workers is virtually nonexistent”.

“AI training for healthcare workers is virtually nonexistent.”

Khalia Primer

As both a clinician by training and an AI researcher, Dr Lauren Oakden-Rayner, Senior Research Fellow at the University of Adelaide’s Australian Institute for Machine Learning (AIML) and Director of Medical Imaging Research at the Royal Adelaide Hospital, balances the pros and the cons of AI in medicine.

“How do we talk about AI?” she asks. One way is to highlight the fact that AI systems are performing as well as or even outperforming humans. The second way is to say AI is not intelligent.

“You might call these, the AI ‘hype’ position and the AI ‘contrarian’ position,” Oakden-Rayner says. “People have made whole careers out of being on one of these positions now.”

Oakden-Rayner explains that both of these positions are true. But how can both be correct?

“You might call these, the AI ‘hype’ position and the AI ‘contrarian’ position. People have made whole careers out of being on one of these positions now.”

Dr Lauren Oakden-Rayner

The problem according to Oakden-Rayner is in the way we compare AI to humans. A fairly understandable baseline given we are human, but the researcher insists that this only serves to confuse the AI-scape by anthropomorphising AI.

Oakden-Rayner points to a 2015 study in comparative psychology – the study of non-human intelligences. That research showed that, for a tasty treat, pigeons could be trained to spot breast cancer in mammograms. In fact, the pigeons took only two to three days to reach expert performance.

Of course, no one would claim for a second that pigeons are as smart as a trained radiologist. The birds have no idea what cancer is or what they are looking at. “Morgan’s Canon” – the principle that the behaviour of a nonhuman animal should not be interpreted in complex psychological terms if it can instead be interpreted with simpler concepts – says that we should not assume a non-human intelligence is doing something smart if there is a simpler explanation. This certainly applies to AI.

“These technologies do not often work the way we expect them to.”

Dr Lauren Oakden-Rayner

Oakden-Rayner also recounts an AI that looked at a picture of a cat and correctly identified it as a feline – before becoming entirely certain it was a picture of guacamole. So sensitive are AI to pattern recognition. The hilarious cat/guacamole mix-up replicated in a medical setting becomes much less funny.

This leads Oakden-Rayner to ask: “Does that put patients at risk? Does that introduce safety concerns?”

The answer is yes.

An early AI tool used in medicine was employed to look at mammograms just like the pigeons. In the early 1990s, the tool was given the green light for use in detecting breast cancer in hundreds of thousands of women based. The decision was based on laboratory experiments that showed radiologists improved their detection rates when using the AI. Great, right?

Twenty-five years later, a 2015 study looked at the real-world application of the program and the results weren’t so good. In fact, women were worse off where the tool was in use. The takeaway for Oakden-Rayner is that “these technologies do not often work the way we expect them to”.

AI tends to perform worst for patients who are most at risk – in other words, the patients that require the most care.

Additionally, Oakden-Rayner notes that there are 350 AI systems on the market, but only about five have been subjected to clinical trials. And AI tends to perform worst for patients who are most at risk – in other words, the patients that require the most care.

AI has also been shown to be problematic when it comes to different demographic groups. Commercially available facial recognition systems were found to perform poorly on black people. “The companies that actually took that on board, went back and fixed their systems by training on more diverse data sets,” Oakden-Rayner notes. “And these systems are now much more equal in their output. No one thought about even trying to do that when they were building the systems originally and putting them on the market.”

Much more concerning is an algorithm used in the US by judges to determine sentencing, bail, parole, and for predicting the likelihood of recidivism in individuals. The system is still in use despite 2016 media reports that it was more likely to be wrong in predicting that a black person would reoffend.

So, where does this leave things for Oakden-Rayner?

“I’m an AI researcher,” she says. “I’m not just someone who pokes holes in AI. I really like artificial intelligence. And I know that the vast majority of my talk is about the harms and the risks. But the reason I’m like this is because I’m a clinician, and so we need to understand what can go wrong, so we can prevent it.”

“I really like artificial intelligence […] we need to understand what can go wrong, so we can prevent it.”

Dr Lauren Oakden-Rayner

Key to making AI safer, according to Oakden-Rayner, is to put standards of practice and guidelines in place for publishing clinical trials involving artificial intelligence. And, she believes, this is all very achievable.

Professor Lyle Palmer, a genetic epidemiology lecturer at the University of Adelaide and also a Senior Research Fellow at AIML, highlights the role that South Australia is playing as a centre for AI research and development.

If there’s one thing you need for good artificial intelligence, he says, it’s data. Diverse data. And lots of it. South Australia is a prime location for large population studies given the large troves of medical history in the state, says Palmer. But he also echoes the sentiments of Oakden-Rayner that these tests have to include diverse samples to capture the differences in different demographics.

“This is all possible. We’ve had the technology to do this for ages.”

Professor Lyle Palmer

“What a cool thing it would be if everyone in South Australia had their own homepage, where all of their medical results were posted and we can engage them in medical research, and a whole range of other activities around things like health promotion,” Palmer says excitedly. “This is all possible. We’ve had the technology to do this for ages.”

Palmer says this technology is particularly advanced in Australia – especially in South Australia.

This historical data can help researchers determine, for example, the lifetime of a disease to better understand what causes diseases to develop in different individuals.

For Palmer, AI is going to be critical in medicine given the “hard times” in healthcare including in the drug delivery pipeline, which is seeing many treatments not reaching the people who need it.

AI can do wonderful things. But, as Oakden-Rayner warns, it’s a mistake to compare it to humans. The tools are only as good as the data we feed them and, even then, they can make many bizarre mistakes because of their sensitivity to patterns.

Artificial intelligence will transform medicine (more slowly than some have suggested in the past, it seems) for sure. But, just as the new technology itself is intended to care for patients, the human creators of the technology are required to ensure that the technology is itself safe and not doing more harm than good.