Citation Information :
Cohen SA, Fisher AC, Xu BY, Song BJ. Comparing the Accuracy and Readability of Glaucoma-related Question Responses and Educational Materials by Google and ChatGPT. J Curr Glaucoma Pract 2024; 18 (3):110-116.
Aim and background: Patients are increasingly turning to the internet to learn more about their ocular disease. In this study, we sought (1) to compare the accuracy and readability of Google and ChatGPT responses to patients’ glaucoma-related frequently asked questions (FAQs) and (2) to evaluate ChatGPT's capacity to improve glaucoma patient education materials by accurately reducing the grade level at which they are written.
Materials and methods: We executed a Google search to identify the three most common FAQs related to 10 search terms associated with glaucoma diagnosis and treatment. Each of the 30 FAQs was inputted into both Google and ChatGPT and responses were recorded. The accuracy of responses was evaluated by three glaucoma specialists while readability was assessed using five validated readability indices. Subsequently, ChatGPT was instructed to generate patient education materials at specific reading levels to explain seven glaucoma procedures. The accuracy and readability of procedural explanations were measured.
Results: ChatGPT responses to glaucoma FAQs were significantly more accurate than Google responses (97 vs 77% accuracy, respectively, p < 0.001). ChatGPT responses were also written at a significantly higher reading level (grade 14.3 vs 9.4, respectively, p < 0.001). When instructed to revise glaucoma procedural explanations to improve understandability, ChatGPT reduced the average reading level of educational materials from grade 16.6 (college level) to grade 9.4 (high school level) (p < 0.001) without reducing the accuracy of procedural explanations.
Conclusion: ChatGPT is more accurate than Google search when responding to glaucoma patient FAQs. ChatGPT successfully reduced the reading level of glaucoma procedural explanations without sacrificing accuracy, with implications for the future of customized patient education for patients with varying health literacy.
Clinical significance: Our study demonstrates the utility of ChatGPT for patients seeking information about glaucoma and for physicians when creating unique patient education materials at reading levels that optimize understanding by patients. An enhanced patient understanding of glaucoma may lead to informed decision-making and improve treatment compliance.
Tan SSL, Goonawardene N. Internet health information seeking and the patient-physician relationship: a systematic review. J Med Int Res 2017;19(1):e5729. DOI: 10.2196/jmir.5729
Finney Rutten LJ, Blake KD, Greenberg-Worisek AJ, et al. Online health information seeking among US adults: measuring progress toward a healthy people 2020 objective. Public Health Rep 2019;134(6):617–625. DOI: 10.1177/0033354919874074
Drees J. Google receives more than 1 billion health questions every day. 2019. https://www.beckershospitalreview.com/healthcareinformation-technology/google-receives-more-than-1-billionhealth-questions-every-day.html. Accessed 20 Feb 2024.
Cohen SA, Fisher AC, Pershing S. Analysis of the readability and accountability of online patient education materials related to glaucoma diagnosis and treatment. OPTH 2023;17:779–788. DOI: 10.2147/OPTH.S401492
Cohen SA, Pershing S. Readability and accountability of online patient education materials for common retinal diseases. Ophthalmol Retina 2022;6(7):641–643. DOI: 10.1016/j.oret.2022.03.015
Cohen SA, Tijerina JD, Kossler A. The readability and accountability of online patient education materials related to common oculoplastics diagnoses and treatments. Semin Ophthalmol 2023;38(4):387–393. DOI: 10.1080/08820538.2022.2158039
Hua HU, Rayess N, Li AS, et al. Quality, readability, and accessibility of online content from a google search of “macular degeneration”: critical analysis. J Vitreoretin Dis 2022;6(6):437–442. DOI: 10.1177/24741264221094683
Martin CA, Khan S, Lee R, et al. Readability and suitability of online patient education materials for glaucoma. Ophthalmol Glaucoma 2022;5(5):525–530. DOI: 10.1016/j.ogla.2022.03.004
Cohen SA, Brant A, Rayess N, et al. Google Trends—assisted analysis of the readability, accountability, and accessibility of online patient education materials for the treatment of AMD After FDA approval of pegcetacoplan. J Vitreoretin Dis 2024;8(4):421–427. DOI: 10.1177/24741264241250156
Gupta B, Mufti T, Sohail SS, et al. ChatGPT: a brief narrative review. Cogent Bus Manag 2023;10(3):2275851. DOI: 10.1080/23311975.2023.2275851
De Angelis L, Baglivo F, Arzilli G, et al. ChatGPT and the rise of large language models: the new AI-driven infodemic threat in public health. Front Public Health 2023;11:1166120. DOI: 10.3389/fpubh.2023.1166120
Rosenberg GS, Magnéli M, Barle N, et al. ChatGPT-4 generates orthopedic discharge documents faster than humans maintaining comparable quality: a pilot study of 6 cases. Acta Orthop 2024;95:152–156. DOI: 10.2340/17453674.2024.40182
Singh S, Djalilian A, Ali MJ. ChatGPT and ophthalmology: exploring its potential with discharge summaries and operative notes. Semin Ophthalmol 2023;38(5):503–507. DOI: 10.1080/08820538.2023.2209166
Cohen SA, Brant A, Fisher AC, et al. Dr. Google vs. Dr. ChatGPT: exploring the use of artificial intelligence in ophthalmology by comparing the accuracy, safety, and readability of responses to frequently asked patient questions regarding cataracts and cataract surgery. Semin Ophthalmol 2024;39(6):472–479. DOI: 10.1080/08820538.2024.2326058
Bernstein IA, Zhang Y, Govil D, et al. Comparison of ophthalmologist and large language model chatbot responses to online patient eye care questions. JAMA Network Open 2023;6(8):e2330320. DOI: 10.1001/jamanetworkopen.2023.30320
Cappellani F, Card KR, Shields CL, et al. Reliability and accuracy of artificial intelligence ChatGPT in providing information on ophthalmic diseases and management to patients. Eye 2024;38:1–6. DOI: 10.1038/s41433-023-02906-0
Potapenko I, Boberg-Ans LC, Stormly Hansen M, et al. Artificial intelligence-based chatbot patient information on common retinal diseases using ChatGPT. Acta Ophthalmol 2023;101(7):829–831. DOI: 10.1111/aos.15661
Momenaei B, Wakabayashi T, Shahlaee A, et al. Appropriateness and readability of ChatGPT-4-generated responses for surgical treatment of retinal diseases. Ophthalmol Retina 2023;7(10):862–868. DOI: 10.1016/j.oret.2023.05.022
Cox A, Seth I, Xie Y, et al. Utilizing ChatGPT-4 for providing medical information on blepharoplasties to patients. Aesthet SurgJ 2023;43(8):NP658–NP662. DOI: 10.1093/asj/sjad096
Samaan JS, Yeo YH, Rajeev N, et al. Assessing the accuracy of responses by the language model ChatGPT to questions regarding bariatric surgery. Obes Surg 2023;33(6):1790–1796. DOI: 10.1007/s11695-023-06603-5
Onder CE, Koc G, Gokbulut P, et al. Evaluation of the reliability and readability of ChatGPT-4 responses regarding hypothyroidism during pregnancy. Sci Rep 2024;14(1):243. DOI: 10.1038/s41598-023-50884-w
Sarraju A, Bruemmer D, Van Iterson E, et al. Appropriateness of cardiovascular disease prevention recommendations obtained from a popular online chat-based artificial intelligence model. JAMA 2023;329(10):842–844. DOI: 10.1001/jama.2023.1044
Morahan-Martin JM. How internet users find, evaluate, and use online health information: a cross-cultural review. Cyberpsychol Behav 2004;7(5):497–510. DOI: 10.1089/cpb.2004.7.497
Wang Y, McKee M, Torbica A, et al. Systematic literature review on the pread of health-related misinformation on social media. Soc Sci Med 2019;240:112552. DOI: 10.1016/j.socscimed.2019.112552
Solomon SD, Shoge RY, Ervin AM, et al. Improving access to eye care: a systematic review of the literature. Ophthalmology 2022;129(10):e114–e126. DOI: 10.1016/j.ophtha.2022.07.012
Musa I, Bansal S, Kaleem MA. Barriers to care in the treatment of glaucoma: socioeconomic elements that impact the diagnosis, treatment, and outcomes in glaucoma patients. Curr Ophthalmol Rep 2022;10(3):85–90. DOI: 10.1007/s40135-022-00292-6
Schaeffer K. 10 facts about today's college graduates. Pew Research Center. https://www.pewresearch.org/short-reads/2022/04/12/10-facts-about-todays-college-graduates/. Accessed 20 Feb 2024.
Kianian R, Sun D, Crowell EL, et al. The use of large language models to generate education materials about uveitis. Oph Retina 2024;8(2):195–201. DOI: 10.1016/j.oret.2023.09.008
Killeen OJ, Niziol LM, Cho J, et al. Glaucoma medication adherence 1 year after the support, educate, empower personalized glaucoma coaching program. Ophthalmol Glaucoma 2023;6(1):23–28. DOI: 10.1016/j.ogla.2022.08.001
Newman-Casey PA, Niziol LM, Lee PP, et al. The impact of the support, educate, empower personalized glaucoma coaching pilot study on glaucoma medication adherence. Ophthalmol Glaucoma 2020;3(4):228–237. DOI: 10.1016/j.ogla.2020.04.013
Muir KW, Lee PP. Health literacy and ophthalmic patient education. Surv Ophthalmol 2010;55(5):454–459. DOI: 10.1016/j.survophthal.2010.03.005
Allison K, Patel DG, Greene L. Racial and ethnic disparities in primary open-angle glaucoma clinical trials: a systematic review and meta-analysis. JAMA Network Open 2021;4(5):e218348. DOI: 10.1001/jamanetworkopen.2021.8348
Hickey KT, Creber RMM, Reading M, et al. Low health literacy. Nurse Pract 2018;43(8):49–55. DOI: 10.1097/01.NPR.0000541468.54290.49
Chaudhry SI, Herrin J, Phillips C, et al. Racial disparities in health literacy and access to care among patients with heart failure. J Card Fail 2011;17(2):122–127. DOI: 10.1016/j.cardfail.2010.09.016
Redd TK, Read-Brown S, Choi D, et al. Electronic health record impact on pediatric ophthalmologists’ productivity and efficiency at an academic center. J AAPOS 2014;18(6):584–589. DOI: 10.1016/j.jaapos.2014.08.002
Chiang MF, Read-Brown S, Tu DC, et al. Evaluation of electronic health record implementation in ophthalmology at an academic medical center (an American Ophthalmological Society thesis). Trans Am Ophthalmol Soc 2013;111:70–92. PMID: 24167326.
Chen AJ, Baxter SL, Gali HE, et al. Evaluation of electronic health record implementation in an academic oculoplastics practice. Ophthalmic Plast Reconstr Surg 2020;36(3):277–283. DOI: 10.1097/IOP.0000000000001531
Li Z, Wang L, Wu X, et al. Artificial intelligence in ophthalmology: the path to the real-world clinic. Cell Rep Med 2023;4(7):101095. DOI: 10.1016/j.xcrm.2023.101095
Berkowitz ST, Finn AP, Parikh R, et al. Ophthalmology workforce projections in the United States, 2020 to 2035. Ophthalmology 2024;131(2):133–139. DOI: 10.1016/j.ophtha.2023.09.018
John AM, John ES, Hansberry DR, et al. Analysis of the readability of patient education materials in pediatric ophthalmology. JAAPOS 2015;19(4):e48. DOI: 10.1016/j.jaapos.2015.07.149
Pakhchanian H, Yuan M, Raiker R, et al. Readability analysis of the American Society of ophthalmic plastic & reconstructive surgery patient educational brochures. Semin Ophthalmol 2022;37(1):77–82. DOI: 10.1080/08820538.2021.1919721
