Reflection on AI in healthcare, prompted from the recently published JAMA Intern Med comparison of Physician and Artificial Intelligence Chatbot Responses to Patient Questions

 "Comparing Physician and Artificial Intelligence Chatbot Responses to Patient Questions Posted to a Public Social Media Forum" is the title of a new Original Investigation report appeared on JAMA Intern Med (doi:10.1001/jamainternmed.2023.1838)

The article is free online, so we leave it to you to read the full text if you are interested, here however we would like to reflect a bit about the findings, and the context.

In the words of Mark Dredze (one of the authors) on Twitter they"compared ChatGPT responses to people's medical questions with those of doctors. Healthcare professionals preferred ChatGPT 79% of the time; as more empathetic and higher quality".

Since the memory of the web is not eternal, let's paste a screenshot here 😇

Now, before anything else, let's first focus on the limitations of the study, as reported by the authors themselves, and specifically let's devote our attention to the following paragraph: 

Additional limitations of this study include, the summary measures of quality and empathy were not pilot tested or validated; this study’s evaluators despite being blinded to the source of a response and any initial results were also coauthors, which could have biased their assessments; the additional length of the chatbot responses could have been erroneously associated with greater empathy; and evaluators did not assess the chatbot responses for accuracy or fabricated information.

...and to this one:

we did not evaluate patient assessments whose judgements of empathy may differ from our health care professional evaluators and who may have adverse reactions to AI assistant–generated responses

It's arguable, after reading the above text, that we are reading a report about an interesting experience of exploration by fellow colleagues, but any extrapolation of strong claim should be avoided: the experience was lacking any and all foundational criteria of objectivity, and even ignored the accuracy of the responses!!

Why, then, is the study capturing the attention of the community? To use the words of another research group "Patients respond differentially to different information and respond most to information about physicians’ interpersonal and clinical skills" (source: Journal of Marketing, DOI: 10.1177/00222429221146511)  an argument that resonates with a recent reflection appeared on the Italian periodic newspaper Percorsi di Secondo Welfare (maintained by the Dip.to di Scienze Sociali e Politiche dell'Università degli Studi di Milano). Reputation correlates with patients choices when searching for a new care provider, and the feedbacks expressed by patients about their care givers on reputation platforms capture more the empathy and ability to communicate, than the level of competence. This may sound wrong to some at first, but it connects to the transformation from to cure to to care in medical practice and healthcare management, and it has deep roots in the past, captured by the tradition of narrative medicine.

If the interest for the report appeared on JAMA Intern Med is justified (albeit a complex tangle of financial interests and response to patient needs) then, we need to stress that the same report is objectively not the last word about the issue... dare we say more, not even a word to pay much heed to, once the limitations are taken into account.

It is an invitation to do research on the matter though, and as such a very welcome and timely one as well. So do read it, and formulate your own questions and conjectures, share them with your peers (contact us, and we will be glad to find ways to work together), and study... the world of care is changing fast.

Post Scriptum: 

• This one is not about medical AI, but it touches on several sensitivities of the current market of AI and it is well worth reading if you want to catch up to speed https://blog.eladgil.com/p/ai-platforms-markets-and-open-source
• This one is about evaluating innovation, and will walk you through both technical and socio-technical considerations that should be embraced in our R&I effort https://blog.shakirm.com/2023/04/elevating-our-evaluations-technical-and-sociotechnical-standards-of-assessment-in-machine-learning/

• This one about Organizational Governance of Emerging Technologies: AI Adoption in Healthcare, albeit US centric, could also interest many of the readers https://arxiv.org/abs/2304.13081

One of the prize winners of the SIT National 2021 has just published the full manuscript: Usable Homomorphic Encryption for Private Telemedicine on the Cloud

 Some of us remember the spontaneous scientific contribution "Usable Homomorphic Encryption for Private Telemedicine on the Cloud", by José Cabrero-Holgueras and Sergio Pastrana, which was awarded one of the prizes for scientific excellence attributed by the scientific committee of the 2021 SIT National conference

On March the 31st, their manuscript entitled "Towards Realistic Privacy-Preserving Deep Learning over Encrypted Medical Data" presenting progress on the same R&D effort shared with us in 2021, has been accepted for publication in Frontiers in Cardiovascular Medicine (IF=5.846), in the thematic issue "Data Driven and Model Based Computational Futures in Cardiovascular Practice"

If you are technically inclined, or if someone with a more technical profile works in your team, you might love to read that there is a small demonstrator made available to familiarize with the working of the solution: https://github.com/jcabrero/HEFactory 

We feel a parental bond, for having been the first ones to recognize the value of their research, and we wish to celebrate this publication with the authors: congratulations!

We look forward to seeing the first applications soon... maybe in an Italian pilot and in partnership with SIT? ^_^

Machine learning can easily produce false positives when the test set is wrongly used

 Machine learning can easily produce false positives when the test set is wrongly used. Just et al in Nature HumBehav suggested that ML can identify suicidal ideation extremely well from fMRI and there were plenty of reasons to be skeptical. Today retraction and an analysis of what went wrong came out.

Read the refutation here https://www.nature.com/articles/s41562-023-01560-6 (the retracted paper is available here https://www.nature.com/articles/s41562-017-0234-y.epdf if you need it for didactic purposes)

So what went wrong? The authors apparently used the test data to select features. Obvious mistake. A reminder for everyone into ML: never use the test set for *anything* but testing.

Only practical way to do so in medicine? Lock away the test set till algorithm is registered.

Side note: it took 3 years to go through the process of demonstrating that the paper went wrong. Journals need procedures to accelerate this.

BTW, pay attention: Confound Removal in Machine Learning Leads to Leakage https://arxiv.org/abs/2210.09232 Now a quick survey (not by us, but by the source of inspiration for this post) for people in medical ML: which proportion of papers have some kind of leakage? https://twitter.com/KordingLab/status/1645140456655970304

Key healthcare infrastructures are a sensitive target. Security in digital health extends way beyond "data security and privacy"

"A source familiar with the outage said the National Cyber Security Centre, the Cabinet Office and other government agencies had been alerted to the incident, given the group’s role in sensitive areas such as Royal Navy training centres and security at Ministry of Defence bases.

People at sites including critical national infrastructure have resorted to using radios, pens and paper, the source said.

Some employees still have access to computers and email, and the company said its investigation was in the early stages and that it was too soon to tell if the failure was caused by a cyber-attack."

Source: https://www.theguardian.com/business/2023/mar/31/capita-it-systems-fail-cyber-attack-nhs-fears 

The above is an example of why digital health should adopt a strong security culture, and conversations, including those around digital health and telemedicine like the ones we held at the recent meeting in Bologna (read our previous posts here and here) should engage medical practitioners in thinking beyond data policies and privacy, embracing risk qualification and mitigation strategies when thinking about innovation of practices and novel value delivery in healthcare.

We thank again Professor Nicola Dragoni for the stimulating talk about risk and security in healthcare he gave during our session at the Bologna Assembly, which triggered many conversations in the followup, and we hope to launch a devoted activity to promote secure practices in digital health in Europe.