MediCoSpace: Visual Decision-Support for Doctor-Patient Consultations using Medical Concept Spaces from EHRs

Since EHR notes consist of free text, text analysis methods, including approaches that incorporate knowledge graphs or apply language models, can be used to extract the essential concept information. Linking data to an ontology is a common practice. Li et al. [39] present a design framework for named behavioral ontology learning from text. The framework describes linguistic and statistical approaches to address tasks such as variable and synonymous relationship extraction. Knowledge graphs [5, 31, 52] provide insights into (hierarchical) relations and the structure of medical concepts in relation to medical ontology knowledge. For instance, Li et al. [40] present a visual analytics approach by linking medical event sequences to a subgraph in a medical knowledge graph using a domain-knowledge–guided recurrent neural network (DG-RNN) model. Such approaches are effective, yet limited to the information stored in the particular knowledge graph in use.

Deep-learning-based language models (e.g., BERT [11]) have reached high performance in diverse natural language processing tasks. These models are pre-trained on large corpora, learning language structures in an unsupervised manner. Furthermore, domain or task-specific fine-tuning, i.e., adapting the pre-trained weights according to language characteristics of a specific domain (also known as additional pre-training) or downstream task [12], is also commonly used in the medical domain. There are several medical domain-specific adaptations of BERT, such as BioBERT [38] and PubMedBert [23] (both additionally pre-trained models on large-scale biomedical corpora), and ClinicalBERT [29] and clinical-kb-bert [25] (both additionally pre-trained on the MIMIC-III [35] dataset to capture patient-record related information and clinical-kb-bert is also pre-trained on UMLS [7] ontology knowledge). Neural language models can be used for different analysis purposes and downstream tasks. First, we can fine-tune them for the named entity recognition task. For instance, Sun et al. [57] fine-tune BioBERT on a machine reading comprehension task that allows it to predict named entity (chemicals, diseases, and proteins) occurrences. Second, we can use them to generate contextualized embedding representations (e.g., on word, sentence, or even document level). To understand named-entity similarity, we can thus use a medical domain-adapted language model to compute their embedding representations and apply a similarity function to determine their similarity. Since this is a very general approach and is not restricted to specific named-entity categories, we apply it in our work. Also, Loureiro et al. [41] use a language model for a medical entity linking task with the MedMentions [44] dataset, whereby the embedding similarity is one step (in addition to entity classification) in their processing pipeline to link entities to an ontology.

Doctors use information systems to access and extend EHRs. Currently, Epic [14] is one of the most common commercial EHR systems, which, according to doctors, still suffers from problems, see the top of Figure 1. In general, in the research community, interactive EHR visualizations are most often visualized using bar/line/pie charts, glyphs, and timelines [60]. For example, LifeLines was one of the first tools to visualize textual notes from EHRs as events on a timeline [17]. Researchers have used this timeline structure to visualize EHRs of individual patients abundantly [6, 10, 24, 26, 28, 42, 56, 58] to, for example, display cause and effect [48] or disease progression [50]/risk [40] prediction. Also, Sultanum et al. [56] researched the importance of visualizing text for assisting doctors. Moreover, van der Linden et al. [58] visualized EHRs in a multiscale way to find the fragmented narratives based on the different tasks of the doctor.

Furthermore, stepping away from individual patients, many researchers have visualized patient cohorts as flow visualizations [21, 22, 30, 34, 36, 64, 65] for disease progression, which have limitations in identifying relations. Therefore, Jin et al. [34] visualized causal relations between medical events and two groups. Furthermore, many researchers have used text [20] or basic plots (e.g., line plots) [15] to display summary statistics and heatmaps [30] for visualizing research around the diagnosis process using medical concepts. For example, Hur et al. [30] focused on diagnosis predictions, for which they used different heatmaps (one for the entire cohort, one for the patient, or one to show the difference between them) to show the weights of the medical concepts used in their model.

While these tools make important steps, the diagnosis and creation of non-trivial treatment plans are more difficult than more trivial ones. To our knowledge, no medical decision-support system addresses relationships between diseases, drugs, and treatments combined with advanced search support within the patient’s history and across similar patients, and links this back to patient reports to discover and leverage possibly overlooked relations.

By interviewing a cardiologist (D1), a general practitioner (D2), a medical student (D3), and two medical doctors in internal medicine (D4) and cardiology (D5) about their workflows and comparing them to the processes from Balogh et al. [4] and Adler–Milstein et al. [1], we identified the following general workflow for diagnosing and making treatment plans for non-trivial conditions for doctors of all specialties and experience levels, see Figure 1. Accordingly, the interviewed doctors were from different specialties and experience levels. First, the doctor looks up patient appointment details. The patient already went through the experiencing health problems and engaged with the healthcare system stages from Balogh et al. [4]. Second, the doctor reviews the EHR for the medical history and the current disease(s) as preparation (related to the information integration and interpretation stage [1, 4]). Third, the doctor speaks with the patient and might conduct physical tests (related to the information gathering stage [1, 4]). Fourth, the doctor reviews the EHR in more detail to find previous and present diagnoses, issues and physiology, and how the patient appears to progress. Based on this, next steps (related to the formulation of next steps [1]) could be conversations with colleagues and possibly diagnostic testing (related to the information gathering stage [1, 4]). Also, the doctor matches the symptoms to the most probable diseases to form a working diagnosis (related to working/leading diagnosis stage [1, 4]), after which they research the best treatment option online and communicate this to the patient. This is often an iterative process possibly with multiple cycles based on certain outcomes. These final steps also correspond to the final stages of Balogh et al.’s [4] and Adler–Milstein et al.’s [1] processes. We noticed that it differs per specialty how much information the doctors require from the EHR. For example, internal medicine requires a deep dive into the EHR because patients often have vague symptoms, while a cardiologist often needs less information because medical imaging often indicates the main problems directly.

Also, doctors indicated that it is hard to find the correct disease based on ambiguous symptoms. The occurrence frequency of a disease also needs to be taken into account, as well as the patient’s lifestyle context; and sometimes it is hard to mentally let go of an initial diagnosis. Visual analytics can assist the doctor (in stages two and four of our workflow) in finding relations between symptoms, diseases, treatments, and drugs to get a more holistic overview of the patient to guide the doctor in information gathering, finding working diagnoses, and possible treatments.

In designing MediCoSpace, we focus on the fourth workflow step, the in-depth EHR review. From this, we derive the following requirements based on a thematic analysis [8] of the interviews: