While artificial intelligence used to be confined to science fiction, it is now part of our reality, found technologies from facial recognition to digital transcription and music recommendation systems. But using artificial intelligence for lifesaving purposes can still be risky.
Researchers at the University of Chicago Medicine are trying to mitigate this risk by developing algorithms that can use deep learning to determine whether images of cancerous tissue come from either of two lung cancers, with high certainty. This technology comes closer to being safe for use with patients because it also estimates how confident it is about its predictions.
“Artificial intelligence systems are great when they work, but you don’t always know when they start to fail. One of the important hurdles that we need to pass is how we make this safe for patients,” said James Dolezal, MD, who is a fellow in the Section of Hematology and Oncology at the UChicago Medicine and the first author of the study.
The research paper, “Uncertainty-informed deep learning models enable high-confidence predictions for digital histopathology,” was published in Nature Communications. Alexander Pearson, MD, PhD, Assistant Professor of Medicine, is the senior author on the study.
Estimating uncertainty improves confidence of algorithm’s predictions
The algorithm that Dolezal and his team developed is a type of artificial intelligence called a deep learning model, which can be given large sets of data and then learn to make ‘predictions’ about similar sets of data. The researchers input around 1,000 microscope slide images of lung cancer tissue from The Cancer Genome Atlas and the Clinical Proteomic Tumor Analysis Consortium. The tissue images were either from lung adenocarcinoma or squamous cell carcinoma of the lung, and the model was told which type of cancer each image represented. The model was then shown more images of lung cancer tissue from an additional set of data, and it applied what it had learned from its training to categorize each new image as belonging to one of the two cancer types. Finally, to determine their accuracy, the study authors compared the model’s predictions to the diagnoses that the images had been manually assigned in the dataset.
Technologies like these have the potential to make more accurate predictions than a human could about microscopic images of cancer tissue, even an expert pathologist with years of training. But an expert pathologist can also explain the reasoning behind their judgments or enlist additional expertise when they are unsure. Algorithms do not have the same discretion, which can make them unsafe to use for diagnosing patients. A wrong prediction could cause a patient to be diagnosed incorrectly, given less effective treatments, or experience worse health outcomes, but Dolezal and his coauthors have addressed this issue by programming the model to estimate how uncertain it is about its predictions.
“The uncertainty estimate basically gives us another branching point for what we do with this information. Without uncertainty, the model is going to give you a prediction no matter what, and you don’t know if the prediction is trustworthy or not,” Dolezal said. “So, if the model is giving you a prediction and it has low confidence, then you can branch out from there and manually verify, or just disregard the prediction. But it’s basically a safety measure to make sure we’re not making decisions based on predictions that are unreliable.” By excluding images that are estimated to have high uncertainty, the accuracy of the model goes from 83.5% to 97.5%.
This study also pushes the field forward by developing a way to estimate uncertainty for full slide images, not just individual regions of an image. Each image that gets analyzed is hundreds of thousands of pixels large and must be analyzed in parts. The algorithm can integrate the many small estimates of uncertainty into a single estimate that can then be used to judge the quality of the prediction for each slide.
“At the end of the day, you have a patient, you have a massive slide image, you put it through the algorithm and the whole slide gets a stamp saying this is a high confidence prediction, or no, this is not trustworthy,” Dolezal said.
Interdisciplinary research moving medical technology forward
Dolezal attributes much of the success of this study to collaboration. “This type of research is really hard to do in a silo and you’re a lot less productive, a lot less innovative when you’re working alone,” he said. “We had computer scientists, a statistician, medical oncologists and trainees, and it took all of us together to make this work.”
One of those researchers, Dmitry Karpeyev, was brought in from outside of the academic research sphere. He was a cancer patient who connected with Dr. Pearson and wanted to volunteer his time and expertise in computer science. His work in the financial sector, predicting market behavior from large data sets turned out to be invaluable. “Dmitry’s experience working in industry was very helpful for us in designing the algorithm. He played a very central role in making this happen, and without him, it would not have been what it is,” Dolezal said.
Going forward, some of the features of the deep learning model described in this study could be applied to other models to improve the predictions they make.
“I think this offers an opportunity for people when they’re testing their deep learning models to add another dimension to what they report,” Dolezal said. “For example, when we’re reporting results for breast cancer risk recurrence, rather than just giving the results for what the predictions are, this offers a tool for researchers to see when the predictions were high confidence and when the predictions were low confidence.”
Artificial intelligence has the potential to make diagnoses of many diseases faster, cheaper, and more accurate. Now, by developing a way to identify low-confidence predictions, the work done in this study has overcome some of the pitfalls of artificial intelligence and brings the medical field a step closer to unlocking its potential.
