Evaluating the quality and risks of language models for healthcare

There is immense enthusiasm about the potential of large language models to support clinical and administrative workflows in healthcare. In fact, large language models are currently being piloted for several applications in healthcare systems today, including for patient portal messaging and ambient documentation. However, a barrier to effective and safe clinical translation is the lack of standardized approaches to evaluate and monitor the knowledge quality, reasoning ability, and risks of these models. In this lecture, I will discuss current limitations of language model knowledge representation in the context of high-impact clinical applications. I will present our research into measuring language model risks, including bias and logical reasoning errors, in ways that are systematic, generalizable, and clinically-relevant. The intersection of these risks with human factors such as over-reliance and automation bias when implemented in a decision-support capacity will be discussed.