Big Data in Neuro-Ophthalmology: International Classification of Diseases Codes

Advancements in technology, including the widespread adoption of electronic medical record keeping, expanding access to imaging techniques such as nonmydriatic fundus photography the expansion of telemedicine, and the application of artificial intelligence to the field of medicine, promise to revolutionize our ability to diagnose and treat neuro-ophthalmic disease. "Big data," for our purposes referring to the dramatic increase in the amount of medical data accessible to researchers, has the potential to accelerate investigations into diagnosis and treatment of neuro-ophthalmic conditions. Meanwhile, machine learning may provide a feasible avenue for analyzing an ever-increasing volume of data.

Editorial Big Data in Neuro-Ophthalmology: International Classification of Diseases Codes Leanne Stunkel, MD A dvancements in technology, including the widespread adoption of electronic medical record keeping, expanding access to imaging techniques such as nonmydriatic fundus photography the expansion of telemedicine, and the application of artificial intelligence to the field of medicine, promise to revolutionize our ability to diagnose and treat neuro-ophthalmic disease. “Big data,” for our purposes referring to the dramatic increase in the amount of medical data accessible to researchers, has the potential to accelerate investigations into diagnosis and treatment of neuro-ophthalmic conditions. Meanwhile, machine learning may provide a feasible avenue for analyzing an ever-increasing volume of data. As neuro-ophthalmologists, harnessing big data may allow us to access large data sets as an avenue to study uncommon diseases. Thus, it is important for neuro-ophthalmologists to understand both the opportunities and the challenges of embracing big data. This companion article to “Predictive Value of International Classification of Diseases Codes for Idiopathic Intracranial Hypertension in a University Health System” by Khushzad et al (1) aims to discuss opportunities and challenges of using big data, specifically the use of claims-based data, such as International Classification of Diseases (ICD) codes, as a source of big data for neuro-ophthalmology research. BIG DATA What is “Big Data”? “Big data” is an expansive term, used by multiple industries to describe an ongoing, massive expansion of accessible data. Typically, the term is used to describe data sets that are more expansive than we have traditionally been able to analyze and data that are less systematized than traditional data sets (2–5). Big data has been defined as possessing 3 “Vs,” volume, variety, and velocity (3,5,6). Volume refers to the large amount of available data, variety refers to the inclusion of different types of data, which may be structured or unstructured, and velocity refers to the speed of ongoing collection and thus the need for timely analysis. Big data is often heterogeneous, it may be assembled for multiple purposes, and its collection is typically not targeted toward a specific research question. It is continually expanding, exponentially in some cases, in the setting of ongoing data collection (3,4,7). Within the medical field, big data typically involves using large clinical data sets, such as patient medical records, insurance or other payor records, genomic databases, government records, surveys, and more (4,5,8). Big Data and Neuro-Ophthalmology Neuro-ophthalmologists specialize in uncommon diseases (4,9,10). Even the “bread-and-butter” conditions seen by neuro-ophthalmologists are uncommon—optic neuritis and nonarteritic anterior ischemic optic neuropathy each have an incidence of less than 10 in 100,000 (11–14), and idiopathic intracranial hypertension (IIH) has an incidence of less than 5 in 100,000 (15,16). The low incidence of neuro-ophthalmic conditions creates challenges for research. For example, the rarity of a disease may limit the study designs available for studying it and limit study power for both retrospective and prospective studies. As pointed out by Hamedeni et al, this may lead to more reliance on retrospective, case–control study designs and make John F. Hardesty, MD Department of Ophthalmology and Visual Sciences, Washington University in St. Louis, St. Louis, Missouri; and Department of Neurology, Washington University in St. Louis, St. Louis, Missouri. Supported by an unrestricted grant to the Department of Ophthalmology and Visual Sciences from Research to Prevent Blindness. The author reports no conflicts of interest. Address correspondence to Leanne Stunkel, MD, 660 S. Euclid Avenue, Campus Box 8096, St. Louis, MO 63110; E-mail: [e-mail] Stunkel: J Neuro-Ophthalmol 2022; 42: 1-5 1 Copyright © North American Neuro-Ophthalmology Society. Unauthorized reproduction of this article is prohibited. Editorial prospective, cohort designs impractical or financially untenable. It may not be feasible to recruit enough patients for a particular study design (10). Low recruitment can affect both retrospective and prospective studies, affecting study power and even forcing trials to be ended early (17). The small number of practicing neuro-ophthalmologists may additionally limit research on neuro-ophthalmic conditions. Patients with neuro-ophthalmic conditions may not have timely access to a subspecialty-trained neuro-ophthalmologist (18–21). Meanwhile, research on neuro-ophthalmic conditions, typically performed by neuro-ophthalmologists, may only capture patients who eventually present to a neuroophthalmologist. Research limited to only the subset of patients who present to a neuro-ophthalmologist may not be representative of all patients with the neuro-ophthalmic condition and in fact may introduce bias, such as a bias toward atypical or severe presentations. For example, many patients with typical optic neuritis may be cared for by neurologists, ophthalmologists, or both, without ever encountering a neuro-ophthalmologist. The small number of neuroophthalmologists may further limit the sample size even beyond the low incidence of the disease being studied—a single neuroophthalmologist in practice may only be able to collect a small number of patients with a given disease of interest even if they continue to collect data for years. In addition, if patients do not have access to a specialist, they may be misdiagnosed. Despite these challenges, continued research into neuroophthalmic conditions is essential—neuro-ophthalmic conditions are complex and are affected by high rates of diagnostic error (21–26). Even when properly diagnosed, some neuro-ophthalmic conditions currently have inadequate or suboptimal treatment options. Opportunities Leveraging big data may expand research opportunities by providing access to larger sample sizes than achieved in a traditional neuro-ophthalmology study (4). However, the sheer amount of data is not the only potential advantage. Big data, which may be collected from multiple sources and for nonresearch purposes in real-world settings, has the potential to be more generalizable than traditional research models (4). Big data may also provide an avenue to study questions that traditional research models are unable to explore because of financial or ethical constraints (4). For example, it is not ethical to perform a clinical trial in which any group is randomized to receive substandard care, but sources of big data may be able to provide information on how outcomes were affected for patients in real-world settings who presented to the health care system late or did not have access to the best treatments. Challenges Using big data also presents significant challenges. One obvious challenge is the management of such a large volume of data. In fact, being characteristically challenging to manage 2 has even been put forth as a possible definition for big data (2,4). The unstructured, heterogeneous nature of large data sets also presents challenges. Big data sources may not have a standardized collection method and are not designed for the purpose of evaluating the question posed by a particular study (3–5,27). For example, in the medical field, many potential sources of big data are collected for billing purposes, and thus, the data may not be structured ideally for research purposes (3). Even compared with big data sources in other industries, medical big data is more complex (3), access may be limited by privacy concerns (3,5), and data collection may be more costly (3,5). From a practical standpoint, the velocity of ongoing collection characteristic of many big data sources may necessitate an eventual automated process, and there are additional challenges in extracting data from multiple sources, such as variations and a lack of standardization in electronic medical record systems, and the separation of certain detailed genetic information because of privacy concerns (27). Large sample sizes and “real-world” settings do not guarantee that a big data source provides a generalizable sample (4). In fact, big data has the potential to introduce bias, for example, because of missing data points and lack of randomization (4). It remains important that research using big data be grounded in traditional research methods to critically consider the strengths and limitations of a particular data source when designing a research study and to assess the validity of data sources (3,4,27). INTERNATIONAL CLASSIFICATION OF DISEASES CODES In the medical field, health care claims (billing) data are arguably the most promising source of big data (28). ICD codes are devised by the World Health Organization (29) and are modified for use in the United States by the Centers for Disease Control and Prevention (30). Although using ICD codes as a source of big data has the potential to be powerful, it is important to be aware of the potential pitfalls of using ICD codes. Inaccurately assigned ICD codes introduce misclassification bias (4,9,31), a type of bias that occurs when subjects in a study are assigned to an inaccurate category (32). In clinical practice, ICD codes are not assigned with a study protocol in mind, and the level of rigor desired in research may not be applied in clinical settings (33). There are multiple reasons that ICD codes may be inaccurately assigned in the course of clinical practice, including diagnostic error, variability in providers’ threshold for assigning codes, administrative errors, and convenience (1). For neuro-ophthalmic conditions in particular, ICD codes may be incorrectly assigned because of diagnostic error. Previous research has shown high rates of diagnostic error of neuro-ophthalmic conditions, up to 60%–70% (21–26) before contact with neuro-ophthalmology, which is often Stunkel: J Neuro-Ophthalmol 2022; 42: 1-5 Copyright © North American Neuro-Ophthalmology Society. Unauthorized reproduction of this article is prohibited. Editorial delayed (19,21) or may never occur at all. The design of the electronic medical record may also inadvertently incentivize providers to sacrifice accurate documentation to operate efficiently within the system (34). For example, practitioners may be incentivized to assign an ICD code with less than perfect accuracy because of time constraints or to link a diagnosis that will allow the provider to order a specific diagnostic test within the environment of the electronic medical record. Although ICD codes are a convenient source of big data, in isolation they are not guaranteed to provide an acceptable level of accuracy for use in neuro-ophthalmic research. Previous studies (9,35–41) have used ICD codes to collect cases worthy of further review, followed by a time-intensive medical record review by the investigators to accurately identify cases that meet inclusion criteria. However, this level of manual review may not be compatible with an eventual goal of using big data to exponentially increase the number of research subjects or even possible if the big data source is not linked to clinical records. Evaluating ICD Codes Hamedani et al (9) reviewed 11 studies that evaluated the validity of ICD codes for identifying neuro-ophthalmic conditions—IIH, giant cell arteritis, optic neuritis, neuromyelitis optica, ocular motor cranial neuropathies, and myasthenia gravis. The authors found that studies of the same condition varied widely in their assessment of the accuracy of the code, and these articles typically did not offer a full picture of the validity of the ICD code—there were typically a limited number of measures of diagnostic accuracy used, and studies typically did not present complete information about their methods. ICD codes may prove to be more useful within an algorithm incorporating additional variables, such as requiring multiple instances of the ICD code, requiring the ICD code to be in the primary diagnosis position, and requiring diagnostic tests or medication prescriptions specific to the disease of interest (9,37–40). Notably, studies that have evaluated how the inclusion of additional variables changes validity have found that this varies by condition—some codes have relatively high validity even without additional variables, and useful variables may be highly specific to the condition of interest, such as age or acetazolamide prescription for IIH (9,37,38). Although not specific to neuroophthalmology, Wright and colleagues demonstrated that by using multiple variables, including laboratory, medication, problem, billing, and vital sign data, they were able to devise algorithms that outperformed the use of billing codes alone for most of the 17 conditions they investigated (33). Using their algorithm, the investigators were able to improve the positive predictive value of a given diagnosis when compared with billing codes alone. The authors hypothesized that this finding was likely due to the use of Stunkel: J Neuro-Ophthalmol 2022; 42: 1-5 certain billing codes to justify screening tests in patients who were eventually found not to have the disease. Measuring Misclassification Bias In this issue, Khushzad et al presented an important study (1) in which they sought to characterize the misclassification bias introduced by using the ICD code for IIH, measured by the positive predictive value (PPV) of the ICD code for identifying cases with a true diagnosis of IIH. The authors also aimed to evaluate potential additional variables that could improve the PPV, thereby reducing the misclassification bias. This was a retrospective review of medical records from a single academic medical center. They included patients who had at least 1 instance of the ICD code for IIH (including ICD 9 or 10) and who also had optic nerve examination, neuroimaging, and lumbar puncture results included within the medical record at their institution. Limiting the study to only patients who had all relevant testing performed at their institution substantially decreased the study population to approximately 10% of patients with the ICD code for IIH. Within the study population, the authors found that ICD codes for IIH had a PPV of 0.63, compared with whether a detailed review of medical records confirmed that the case met the modified Dandy criteria for diagnosis of IIH. They also found several variables that increased the PPV—if the code was entered by an expert/specialist, if they were treated with acetazolamide, and if the ICD code was entered only after all the testing necessary for diagnosis had been completed. Requiring multiple ICD code instances and requiring a longer duration between the first and last ICD code instance also increased the PPV in a direct relationship. The obvious advantage of including these additional variables in the search process is the improved PPV— patients identified by the algorithms combining ICD codes with additional variables are more likely to have an accurate IIH diagnosis than patients identified by the ICD code alone. However, the disadvantage of these algorithms is that they will inevitably exclude patients who have a true diagnosis of IIH, decreasing the sample size, and potentially introducing bias by excluding patients based on the chosen variables. For example, if treatment with acetazolamide is required to identify patients, then patients with a true diagnosis of IIH who do not receive acetazolamide will not be captured. Strengths of this study include a large potential data pool (1,005 patients with ICD codes for IIH) and capturing data over a long period (1989–2017). The most notable limitation is that this study evaluated only cases from a single academic center, which does not reflect the potential heterogeneity of big data sources. Also notably, the sample was restricted only to patients who had undergone complete testing—including optic nerve examination, neuroimaging, and lumbar puncture—at their institution. Although 3 Copyright © North American Neuro-Ophthalmology Society. Unauthorized reproduction of this article is prohibited. Editorial practical, given the investigators’ plan to confirm through a record review whether patients met the modified Dandy criteria, this method surely excluded patients with a true diagnosis of IIH who had some or all of their testing performed elsewhere. Although incorporating additional variables may improve PPV, it may restrict the study population, which may be undesirable depending on the goals of a particular study. FUTURE DIRECTIONS Neuro-ophthalmology as a field also needs to grapple with what we will define as an acceptable level of accuracy. As practitioners in a diagnostic field, we are accordingly precise in our diagnoses, and thus, we have set a standard for retrospective research case inclusion to near 100% diagnostic accuracy. However, this cannot be achieved in big data research. There will necessarily be a tradeoff between diagnostic accuracy and sample size, but the acceptable threshold is not clear. Future research is needed to further characterize the accuracy of ICD codes for various neuro-ophthalmic conditions and to evaluate how algorithms that include additional variables in conjunction with ICD codes may improve accuracy. Although PPV is a useful measure, it may not be generalizable among various sources of big data because PPV depends on the prevalence of a condition within the data source. Additional research evaluating the sensitivity of using ICD codes or algorithms that include ICD codes with additional variables would also be useful, because in some cases ICD codes may miss relevant cases. Wright et al found the sensitivity of billing codes to be imperfect (33). However, the evaluation of sensitivity may be cumbersome because it would potentially require a record review of all patients in a data source to identify patients missed by an ICD code or, as in the study above, would have required a record review of all 1,000+ patients who had the ICD codes for IIH, including outside hospital records, to identify true IIH patients excluded by the initial search. Future research should also be designed to incorporate multiple sites and heterogeneous data sources to further evaluate how to use ICD codes to enlarge the pool of data available for studies of neuro-ophthalmic conditions. In addition, as an automated process would eventually be desirable, future studies should design processes for automating subject selection and evaluate the validity of the automated processes. CONCLUSIONS Accessing the potential of big data to pool data from multiple sources, examine new research questions, and use new study designs has the potential to advance our understanding of neuro-ophthalmic disease. However, big 4 data cannot simply bypass the limitations of traditional research methods (42)—it must be used thoughtfully, with attention to proper research methodology and with a full understanding of what it can and cannot achieve. Characterizing the limitations of big data sources, such as by evaluating the accuracy of ICD codes and how to improve the selection of patients using ICD codes, important for using big data using big data to surpass the limitations of traditional research models. Using big data sources will likely require a thoughtful, study-specific approach, with consideration of the overall goals and design of the study. Khushzad et al's study, by evaluating the accuracy of ICD codes, is an essential step toward using ICD codes to gather cases for research on a larger scale. REFERENCES 1. Khushzad F, Kumar R, Muminovic I, Moss HE. Predictive value of international classification of diseases codes for idiopathic intracranial hypertension in a university health system. J Neuroophthalmol. 2022;41:e679–e683. 2. Press G. 12 Big Data Definitions: What’s Yours? (Forbes Web Site). Available at: https://www.forbes. com/sites/gilpress/2014/09/03/12-big-data-definitionswhats-yours/?sh=94549ba13ae8. Accessed November 7, 2021. 3. Hulsen T, Jamuar SS, Moody AR, Karnes JH, Varga O, Hedensted S, Spreafico R, Hafler DA, McKinney EF. From big data to precision medicine. Front Med (Lausanne). 2019;6:34. 4. Moss HE, Joslin CE, Rubin DS, Roth S. Big data research in neuro-ophthalmology: promises and pitfalls. J Neuroophthalmol. 2019;39:480–486. 5. NEJM Catalyst. Healthcare Big Data and the Promise of ValueBased Care. (NEJM Catalyst Website). Available at: https:// catalyst.nejm.org/doi/full/10.1056/CAT.18.0290. Accessed November 7, 2021. 6. Laney D. 3D Data Management: Controlling Data Volume, Velocity, and Variety. Stamford, CT: META Group, 2001. 7. Raghupathi W, Raghupathi V. Big data analytics in healthcare: promise and potential. Health Inf Sci Syst. 2014;2:3. 8. Dilsizian SE, Siegel EL. Artificial intelligence in medicine and cardiac imaging: harnessing big data and advanced computing to provide personalized medical diagnosis and treatment. Curr Cardiol Rep. 2014;16:441. 9. Hamedani AG, De Lott LB, Deveney T, Moss HE. Validity of international classification of diseases codes for identifying neuro-ophthalmic disease in large data sets: a systematic review. J Neuroophthalmol. 2020;40:514–519. 10. Hamedani AG, Pineles SL, Moss HE. The case-control study in neuro-ophthalmology. J Neuroophthalmol. 2020;40:144–147. 11. Hattenhauer MG, Leavitt JA, Hodge DO, Grill R, Gray DT. Incidence of nonarteritic anterior ischemic optic neuropathy. Am J Ophthalmol. 1997;123:103–107. 12. Johnson LN, Arnold AC. Incidence of nonarteritic and arteritic anterior ischemic optic neuropathy. Population-based study in the state of Missouri and Los Angeles County, California. J Neuroophthalmol. 1994;14:38–44. 13. Rodriguez M, Siva A, Cross SA, O’Brien PC, Kurland LT. Optic neuritis: a population-based study in Olmsted County, Minnesota. Neurology. 1995;45:244–250. 14. Hassan MB, Stern C, Flanagan EP, Pittock SJ, Kunchok A, Foster RC, Jitprapaikulsan J, Hodge DO, Bhatti MT, Chen JJ. Population-based incidence of optic neuritis in the era of aquaporin-4 and myelin oligodendrocyte glycoprotein antibodies. Am J Ophthalmol. 2020;220:110–114. 15. Radhakrishnan K, Ahlskog JE, Cross SA, Kurland LT, O’Fallon WM. Idiopathic intracranial hypertension (pseudotumor Stunkel: J Neuro-Ophthalmol 2022; 42: 1-5 Copyright © North American Neuro-Ophthalmology Society. Unauthorized reproduction of this article is prohibited. Editorial 16. 17. 18. 19. 20. 21. 22. 23. 24. 25. 26. 27. 28. 29. 30. cerebri). Descriptive epidemiology in Rochester, MN, 1976 to 1990. Arch Neurol. 1993;50:78–80. Mollan SP, Aguiar M, Evison F, Frew E, Sinclair AJ. The expanding burden of idiopathic intracranial hypertension. Eye (Lond). 2019;33:478–485. National Institutes of Health. Surgical Idiopathic Intracranial Hypertension Treatment Trial (Sight). Available at: https:// clinicaltrials.gov/ct2/show/NCT03501966. Accessed November 7, 2021. Frohman LP. A profile of neuro-ophthalmic practice around the world. J Neuroophthalmol. 2018;38:47–51. DeBusk A, Subramanian PS, Scannell Bryan M, Moster ML, Calvert PC, Frohman LP. Mismatch in supply and demand for neuro-ophthalmic care. J Neuroophthalmol. 2021 Mar 23 (epub ahead of print). Williams RD. Neuro-Ophthalmology’s SOS: save our subspeciality. EyeNet Mag. 2020;11:11. Stunkel L, Mackay DD, Bruce BB, Newman NJ, Biousse V. Referral patterns in neuro-ophthalmology. J Neuroophthalmol. 2020;40:485–493. Stunkel L, Newman-Toker DE, Newman NJ, Biousse V. Diagnostic error of neuro-ophthalmologic conditions: state of the science. J Neuroophthalmol. 2021;41:98–113. Stunkel L, Newman NJ, Biousse V. Diagnostic error and neuroophthalmology. Curr Opin Neurol. 2019;32:62–67. Fisayo A, Bruce BB, Newman NJ, Biousse V. Overdiagnosis of idiopathic intracranial hypertension. Neurology. 2016;86:341– 350. Stunkel L, Kung NH, Wilson B, McClelland CM, Van Stavern GP. Incidence and causes of overdiagnosis of optic neuritis. JAMA Ophthalmol. 2018;136:76–81. Schroeder RM, Stunkel L, Gowder MTA, Kendall E, Wilson B, Nagia L, Eggenberger ER, Van Stavern GP. Misdiagnosis of third nerve palsy. J Neuroophthalmol. 2020 (epub ahead of print). Schneeweiss S. Learning from big health care data. N Engl J Med. 2014;370:2161–2163. Ewing C. Unlocking the benefits of ICD-10 through data analytics. J AHIMA. 2016;87:24–26. World Health Organization. International Statistical Classification of Diseases and Related Health Problems (ICD). Available at: www.who.int. https://www.who. int/standards/classifications/classification-of-diseases. Accessed November 7, 2021. Centers for Disease Control and Prevention. International Classification of Diseases, Tenth Revision, Clinical Modification (ICD-10-CM). Available at: https://www.cdc. gov/nchs/icd/icd10cm.htm. Accessed November 7, 2021. Stunkel: J Neuro-Ophthalmol 2022; 42: 1-5 31. Tseng VL, Chang MY. Review of cohort studies: a companion article to Malmqvist et al “progression over 5 years of prelaminar hyperreflective lines to optic disc drusen in the copenhagen child cohort 2000 eye study”. J Neuroophthalmol. 2020;40:286–291. 32. Pham A, Cummings M, Lindeman C, Drummond N, Williamson T. Recognizing misclassification bias in research and medical practice. Fam Pract. 2019;36:804–807. 33. Wright A, Pang J, Feblowitz JC, Maloney FL, Wilcox AR, Ramelson HZ, Schneider LI, Bates DW. A method and knowledge base for automated inference of patient problems from structured data in an electronic medical record. J Am Med Inform Assoc. 2011;18:859–867. 34. Powers EM, Shiffman RN, Melnick ER, Hickner A, Sharifi M. Efficacy and unintended consequences of hard-stop alerts in electronic health record systems: a systematic review. J Am Med Inform Assoc. 2018;25:1556–1566. 35. Sodhi M, Sheldon CA, Carleton B, Etminan M. Oral fluoroquinolones and risk of secondary pseudotumor cerebri syndrome: nested case-control study. Neurology. 2017;89:792–795. 36. Koerner JC, Friedman DI. Inpatient and emergency service utilization in patients with idiopathic intracranial hypertension. J Neuroophthalmol. 2014;34:229–232. 37. Sundholm A, Burkill S, Bahmanyar S, Nilsson Remahl AIM. Improving identification of idiopathic intracranial hypertension patients in Swedish patient register. Acta Neurol Scand. 2018;137:341–346. 38. Sundholm A, Burkill S, Sveinsson O, Piehl F, Bahmanyar S, Nilsson Remahl AIM. Population-based incidence and clinical characteristics of idiopathic intracranial hypertension. Acta Neurol Scand. 2017;136:427–433. 39. Winthrop KL, Chen L, Fraunfelder FW, Ku JH, Varley CD, Suhler E, Hills WL, Gattey D, Baddley JW, Liu L, Grijalva CG, Delzell E, Beukelman T, Patkar NM, Xie F, Herrinton LJ, Fraunfelder FT, Saag KG, Lewis JD, Solomon DH, Curtis JR. Initiation of antiTNF therapy and the risk of optic neuritis: from the safety assessment of biologic ThERapy (SABER) Study. Am J Ophthalmol. 2013;155:183–189.e181. 40. Lee CD, Williams SE, Sathe NA, McPheeters ML. A systematic review of validated methods to capture several rare conditions using administrative or claims data. Vaccine. 2013;31(suppl 10):K21–K27. 41. Muro-Fuentes EA, Moss HE. Factors associated with increased emergency department utilization in patients with acute optic neuritis. J Neuroophthalmol. 2021;41:335–341. 42. Obermeyer Z, Emanuel EJ. Predicting the future—big data, machine learning, and clinical medicine. N Engl J Med. 2016;375:1216–1219. 5 Copyright © North American Neuro-Ophthalmology Society. Unauthorized reproduction of this article is prohibited.