Can co-designed educational interventions help consumers think critically about asking ChatGPT health questions? Results from a randomised-controlled trial

npj Digital Medicine

Julie Ayre, Melody Taba, Brooke Nickel, Geoffrey Edlund, Trang Vu, Julia Yan, Lorna Butters, Ivan C. K. Ma, Kirsten J. McCaffery

Abstract

This randomised controlled trial evaluated two brief co-designed health literacy educational interventions (animation; images) to help people critically reflect on asking ChatGPT health questions. Australian adults with experience of ChatGPT, and without university education, were recruited via an online panel. Primary outcomes were intention to ask ChatGPT questions in ‘lower’ and ‘higher’ risk health scenarios. The analysis sample comprised 592 of 619 participants. The animation group (n = 191) reported lower intention to use ChatGPT for higher risk scenarios (M = 2.42/5, 95%CI: 2.27 to 2.56) compared to the images group (n = 203, M = 2.69/5, 95%CI: 2.54 to 2.83, p = 0.010). Both reported lower intentions compared to a control group who had not viewed the educational content (n = 205, M = 3.12/5, 95%CI: 2.98 to 3.27, p < 0.001). There was no effect on intentions to use ChatGPT for lower risk scenarios (p = 0.800). This study represents an initial step towards addressing health literacy skills for navigating AI tools safely. Australia and New Zealand Clinical Trials registration: ACTRN 12624000641594.