Knowledge frame translator.

Building an expert system in certain domain area involves knowledge acquisition, knowledge programming and knowledge refinement --- the tasks of transforming the experts' knowledge into the system. Therefore, appropriate and efficient tools, which may include high-level languages, debugging aids, and friendly editing facilities and so on, to facilitate these tasks, are very essential to an expert system. The Translator is such a tool developed in the Department of Medical Informatics at the University of Utah. It has been designed to aid the knowledge programming and refinement for a medial expert system named Iliad. It provides users a very friendly high level language, allows user to create, review or modify knowledge frames in nearly free test form and translates them into a computer readable from of another language. The Translator is a compiler-like software tool built on the Macintosh. Developing such a tool involves language translation knowledge, compiler construction techniques and software engineering concept. The successful implementation of this tool will make knowledge programming and knowledge refinement for the Iliad system much easier and fast than before, accelerate the process of building, updating and maintaining medical knowledge in Iliad, and make the Iliad a complete, and stand-alone expert system.

KNOWLEDGE FRAME TRANSLATOR by Wenshao Wang A thesis submitted to the faculty of The University of Utah in partial fulfillment of the requirements for the degree of Master of Science Department of Medical Informatics The University of Utah December 1988 Copyright © Wenshao Wang 1988 All Rights Reserved THE UNIVERSITY OF UTAH GRADUATE SCHOOL SUPERVISORY COMMITTEE APPROVAL of a thesis submitted by Wenshao Wang This thesis has been read by each member of the following supervisory committee and by majority vote has been found to be satisfactory. Chair: Homer R. Warner &# Peter J.~ THE UNIVERSITY OF UTAH GRADUATE SCHOOL FINAL READING APPROVAL To the Graduate Council of the University of Utah: I have read the thesis of Wenshao Wang in its ----------------------~~------------- final fonn and have found that (1) its fonnat, citations, and bibliographic style are consistent and acceptable; (2) its illustrative materials including figures, tables, and charts are in place; (3) the final manuscript is satisfactory to the Supervisory Committee and is ready for submission to the Graduate School. Date Homer R. Warner Chair, Supervisory Committee Approved for the Major Department Homer R. Warner OlairlDean Approved for the Graduate Council B. Gale Dick Dean of The Graduate School ABSTRACT Building an expert system in a certain domain area involves knowledge acquisition, knowledge programming and knowledge re­finement - the tasks of transforming the experts' knowledge into the systenl. Therefore, appropriate and efficient tools, which may include high-level languages, debugging aids, friendly editing facilities and so on, to facilitate these tasks are very essential to an expert system. The Translator is such a tool developed in the Department of Medical Informatics at the University of Utah. It has been designed to aid the knowledge programming and refinement for a medical expert system named Iliad. It provides users a very friendly high level language, allows users to create, review or modify knowledge frames in nearly free text form and translates them into a computer readable form of another language. The Translator is a compiler-like software tool built on the Macintosh. Developing such a tool involves language translation knowledge, compiler construction techniques and software engineering concept. The successful implementation of this tool will make knowledge programming and knowledge refinement for the Iliad system much easier and faster than before, accelerate the process of building, updating and maintaining medical knowledge in Iliad, and make the Iliad a complete, and stand-alone expert system. To my parents. T ABLE OF CONTENTS Page ABS1R.ACf ................................................................................................................. iv LIST OF FIGURES ................................................................................................. viii ACKNOWLEDG~NTS .............................................................................................. x Chapter I. WIRODUCfION ............................................................................................... 1 The Tr31lslator .......................................................................................... 1 Background ................................................................................................ 2 Literature Review .................................................................................... 5 MYCIN .................................................................................................... 5 INTERNIST-1/QMR ........................................................................... 7 lIELP ....................................................................................................... 9 TIiad ...................................................................................................... 12 Current Research Goals ...................................................................... 14 II. TIffi TRANSLATOR ...................................................................................... 17 The Features of the Knowledge Representation Models ....... 17 The Syntax of the Language ............................................................ 21 A Guide to the Tr31lslator ................................................................. 24 III. DESIGN OF TIffi TRANSLATOR ................................................................. 32 About Target Language ..................................................................... 32 Software Engineering Approach .................................................... 36 Analysis .............................................................................................. 36 Functional specification ............................................................... 39 Design .................................................................................................. 40 Implementation .............................................................................. 45 V alidation .......................................................................................... 45 Special Techniques Employed in Parsing ................................... 46 Top-down parsing .......................................................................... 46 Left to right parsing ...................................................................... 46 Right to left parsing ...................................................................... 46 Backtracking ..................................................................................... 47 Rearranging the "ELSE" structure ............................................ 48 Error Handling ....................................................................................... 48 IV. TRANSLATOR TESTING AND RESULTS ............................................... 50 Translator Testing and Methods... ....... ........................................ 50 Results.............................................................................. ..... .............. .... 53 Program correctness................................................................... 53 Simplicity.... .... ............ ...................... ........ ....... .......................... ...... 54 SJ>eed....... ....................................... ............. ........................... ............ 54 Accuracy..................................................................................... ...... 55 V. DISCUSSIONS AND CONCLUSIONS ........................................................ 59 Summary ................................................................................................. 59 Conclusion.............................................................................................. 62 Discussion ................................................................................................ 63 REFERENCES ............................................................................................................... 66 vii LIST OF FIGURES Figure Page 1. Basic components in an expert system. ... ... ... ............ ... ... ... ... ... ... ...... 3 2. Example of rule acquisition in the MYCIN....................................... 6 3. QMR disease profile for pulmonary lymphoma ............................... 8 4. A partial PAL program for the frame in Figure 5 ......................... 10 5. A free-text version of an interpretation knowledge frame in Iliad ........................................................................................... 11 6. A RCOM version of the interpretation frame in Figure 5 .......... 13 7. Overall design of the knowledge building tools for Iliad ........... 15 8. Two examples of interpretation frames used in Iliad ............... 19 9. An example of probability frame used in Iliad ............................ 20 10. The "File" menu ........................................................................................... 26 11 . The use of the "Syntax" menu .............................................................. 27 12. The use of the "EXAMPLES" command in "Syntax" menu ....... 28 13. The "Save" procedure ............................................................................... 29 14. The use of the "Show Results" command ........................................ .30 15. A probability frame written in the source language.................. 34 1 6. A frame in target language .................................................................... 35 17. Software lifecycle ..................................................................................... 37 1 8 . A translator.................................. ............................................................... 38 1 9. Operational diagram of the Translator.. ............... ........................... 41 20. Decomposition of the Translator. ............... ........................................ 44 21. Comparisons of speed for HCOM and the Translator................. 57 22. The layering of languages ...................................................................... 60 ix ACKNOWLEDGMENTS I would like to express my thanks and appreciation to my advisor, Dr. Homer R. Warner, for his guidance, support and assistance during my research work and preparation of this thesis as well as for sharing so much with me throughout my stay in this country. I also want to thank Dr. Peter J. Haug and Dr. T. Allan Pryor for their helpful suggestions as my supervisory committee members. I also want to acknowledge the help of Dr. Omar Bouhaddou with the development and testing of the Translator protocol. Finally my sincere thanks goes to my wife, Suhua Wei, and my parents for their encouragement and sacrifice. CHAPTER I INTRODUCTION The Translator The Translator is a compiler-like computer program built for a medical expert system named Iliad [1,2,3]. It is a user-friendly knowledge programming tool, implemented on the Macintosh, which allows knowledge frames created in text form by experts and users for the knowledge base in Iliad system to be translated into computer executable form of another language. Iliad is a Macintosh-based expert system in the field of medicine, and has been developed by Dr. Homer Warner and his team in the Department of Medical Informatics at the University of Utah. Containing nearly 200 knowledge frames, it has been designed to provide decision' support to healthcare practitioners. It may be used in a variety of healthcare settings - as a diagnostic consulting tool for experienced physicians; as an educational tool for medical students in universities and teaching hospitals; and as an on-line, real-time alerting tool packaged as part of a hospital or clinical information system. 2 Background Expert systems, an application area of artificial intelligence, are sophisticated computer programs that use expert knowledge to solve problems efficiently and effectively in a narrow problem area [4]. Nowadays, they are being developed for a variety of fields including the area of medicine, and some examples have already proved to be powerful tools that can provide decision support to healthcare practitioners in certain settings. They have been used to make decisions regarding the diagnosis, treatment, and prognosis of a patient [5]. An expert system usually contains three components [6]: a knowledge base, a database, and an inference engine. These three components are shown in a dotted rectangular in Figure 1. The knowledge base for Iliad contains a set of rules or criteria organized into frames, that model the logic an expert would use to make a set of decisions. The inference engine is a control program that uses the knowledge base to make inferences (as opposed to direct retrieval or computation) from available data. Therefore, it can modify or imply the contents of the database. The control mechanism is called a heuristic -- it is a "rule of thumb" rather than an algorithm that produces an exact solution. A well-designed expert system should have another important component -- the natural language user interface, which is a support environment for the system. See Figure 1. This interface is used by the expert and users to enter their knowledge in text form, as close to natural language as possible, into the system's knowledge base. The process is known as knowledge acquisition or knowledge engineering. This interface is also used by KnoWledge Base Inference Engine Global Database Natural Language User Interface Experts Users Figure 1. Basic components in an expert system. 3 4 the system itself to provide information, results or explanations to its users in natural language. Therefore, a good tool that can facilitate knowledge acquisition, knowledge programming and knowledge refinement is very essential to an expert system. Before we discuss what a good tool is, we should know what the knowledge frame is and what the tool is supposed to do. ILIAD's knowledge base is composed of knowledge frames, each of which is basically a smallest unit of computer program in which human knowledge has been implemented. An example of a simplified knowledge might be: ttIf a patient's temperature exceeds 37.52 C, the patient has fever. tt To implement this knowledge into computer, one might write a computer program (in pseudo form): READ( Temperature ) IF ( Temperature >= 37.5 ) THEN WRI1E( "THIS PATIENT HAS FEVER") The computer does not know what the term T em per at u r e really means. To obtain the value of Temperature, the computer must be informed exactly where to find this variable (in what file, what location, how many bytes and so on). Generally, medical terms like temperature have code representations. This matching is accomplished by creating a dictionary. Each time an expert writes a computer program, the expert, by looking up the dictionary, must specify the variables by codes. With some commitment made in design, the computer knows where to find these variables and consequently accomplish any computational task. At this point, the problem becomes apparent: experts in the field of medicine who want to create knowledge frames not only 5 have to have some experience with computers but also have to understand programming in order to implement their knowledge in the computer. In addition, they have to find correct codes (either by memorizing them or by consulting a dictionary), which represent the variables to be used in the algorithm. Therefore, a good tool for an expert system should reduce the experts' programming effort to a minimum and provide an easy way for experts to access the dictionary. There are many expert systems which are currently operational in the field of medicine. Some have good tools to build the system's knowledge base, and some do not. Let us examine some of them. Literature Review MYCIN MYCIN [7], a successful research system developed at the Stanford University, is a ruled-based expert system specializing in the diagnosis and therapy for infectious disease. It has some tools for helping knowledge engineers to build and modify the knowledge base. One such tool is a rule language (syntax and parser) that allows a user to enter a new rule in a quasi-English form in an interactive mode. Figure 2 shows an example of rule-building by using this tool. This simple method of knowledge acquisition was subsequently expanded in the work of TEIRESIAS [7], which, in addition to knowledge acquisition, also provides static rule checking by using its own knowledge (meta-level knowledge) to ensure the correctness of knowledge base. Therefore, knowledge entry in 6 **NR [The knowledge The new rule will be called RULE200. engineer starts the if: 1-** THE ORGANISM IS A GRAM NEGATIVE ROD rule acquisition and 2-** IT IS ANAEROBIC routine by typing and 3-** IT WAS ISOLATED FROM THE BLOOD NR for new rule] and 4-** YOU THINK THE PORTAL WAS THE GI TRACK and 5-** Then: 1-** IT IS PROBABLY A BACTEROIDES On a scale of 1 to 10, how much certainty would you affix to this conclusion? ** 9 and 2-** This is my understanding of your rule: RULE200 IF: 1) The site of the culture is blood, and 2) The stain of the organism is gramneg, and 3) The morphology of the organism is rod, and 4) The aerobicity of the organism is anaerobic, and 5) The portal of entry of the organism is GI THEN: There is strongly suggestive evidence (.9) that the organism is bacteroides Okay? (YES or NO ) **YES [User: carriage return with no entry] [User: carriage return with no en try] [Note that the original clause 1 has been expanded to separate the 2 attributes, stain and morphology.] Figure 2. Example of rule acquisition in the MYCIN [7]. 7 MYCIN is quite user-friendly and quality is ensured. INTERNIST-! / QMR [8,9,10] INTERNIST -1 is a main frame computer based diagnostic consultation system for internal medicine. It was developed at the University of Pittsburgh and contains a knowledge base of over 600 disease profiles and recognizes over 4000 individual manifestations. As an extension of the INTERNIST -1 medical expert system project, Quick Medical Reference (QMR) has been developed on a micro­computer as a stand-alone system for both diagnostic reference and diagnostic consultation. Knowledge programming and refinement for QMR system are accomplished interactively with on screen help. The authors for the knowledge base are required to be familiar with the dictionary of manifestations before making any attempt to create a new disease profile. This is an added constraint to the author of this knowledge base. In QMR, a disease profile IS developed on paper using QMR's computerized dictionary, and is submitted for approval by the senior domain expert, and finally is typed into the computer by a secretory using a knowledge base editor, this process involves form filling where each finding is defined using the "Completer" a dictionary utility program to help users find correct medical terms used in the system. Figure 3 shows an example of knowledge frame built by the tool for QMR. The process of knowledge building for QMR involves many stages of work and is, therefore, time-consuming. 0 4 1 4 2 2 2 2 1 5 2 2 1 4 2 2 2 2 4 2 2 2 2 2 2 3 2 2 2 1 2 3 4 3 4 3 2 2 4 3 Disease Profile: Pulmonary Lymphoma RACE White COUGH CHEST X-ray Large Rounded Density <IES> CHEST X-ray Linear Density <IES> Radiating from Hilum CHEST X-ray Lung Field <S> Abnormal PLEURA Biopsy Malignant Neoplasm PLEURAL Fluid Lymphocyte <S> Predominant PLEURAL Fluid Malignant Neoplastic Cell <S> SPUTUM Malignant Neoplastic Cell <S> SPUTUM Reed Sternberg Cell <S> BRONCHOGRAM Bronchial Narrowing Diffuse BRONCHOSCOPY Bronchial Narrowing Diffuse BRONCHOSCOPY Brush Biopsy Or washing<S> Malignant Neoplastic Cell<S> BRONCHOSCOPY Endobronchial Biopsy Malignant Neoplasm BRONCHOSCOPY Trachea Narrowing BRONCHOSCOPY Transbronchial Biopsy Malignant Neoplasm LUNG Biopsy Infiltration Immature Lymphocyte <S> or Immunoblast <S> Diffuse LUNG Biopsy Infiltration Immature Lymphocyte <S> or Immunoblast <S> Nodular LUNG Biopsy Malignant Neoplasm With Bronchial Involvement LUNG Biopsy Reed Sternberg Cell <S> Associated Diseases for Pulmonary Lymphoma: CAUS-BY 3 2 Hodgkins Disease Systemic CAUS-BY 3 2 Malignant Lymphoma Non-Hodgkins Type CAUS 3 1 Pleural Effusion Chylous COIN 2 2 Gastric Lymphoma COIN 2 2 Lymphoma of Colon COIN 2 2 Small Intestinal Lymphoma CAUS 1 2 Pleural Effusion Exudative CAUS-BY 1 2 Cutaneous T-Cell Lymphoma CAUS 1 1 Atelectasis CAUS 1 1 Pleural Effusion Transudative PDIS 1 1 Hypertrophic Osteoarthropathy CAUS-BY 1 1 Leukemia Chronic Lymphocytic Figure 3. QMR disease profile for pulmonary lymphoma (Showing only the noteworthy findings and associated diseases). 8 9 HELP [11,12] HELP (Health Evaluation through Logic Processing) is a com­prehensive hospital information system developed at the LDS hospital and the University of Utah in Salt Lake City, Utah. There are several tools provided by the HELP system to build its knowledge base. 1. PAL (Ptxt Application Language) [13] This language, which looks like any general purpose high level language, was originally designed as a report generator, and then was expanded to encompass all of HELP's functions. It has become the database and knowledge base language of the HELP system on the Tandem machine. In the process of programming, the expert must explicitly specify any medical term in code. For example: cough IS represented by code 7 1 120 1 10 and hem ate m e sis by code 7 1 124 2 7 4. There are tools to facilitate dictionary access such as ENTR (for search of code) and PTXT (for modification or creation of code). The advantage of the PAL language is that it can automatically find variables in a patient file when the codes of those variables are specified (while other standard languages such as FORTRAN and Pascal cannot). The disadvantage is that the users have to know the Tandem operating system and programming techniques and have to study the language before any attempt at programming a real application. Figure 4 shows a PAL program (partially) for the frame in Figure 5. The whole program is more than 200 lines. Therefore, coding and debugging are time-consuming. 1 0 1 ?NOPREPROCESS 2 ?NOLIST 3 ?NONIL 4 ?RETRIEVE 2048 5 SECTION DC007A F141S113; 6 BEGIN 7 RELATION A; 8 BEGIN 9 ITEM a CODE (007 001 120 * 001 032 * *) ; 10 ITEM time STRINGTIME; 11 END; 12 RELATION A1; 13 BEGIN 14 ITEM a CODE (007 001 120 002 001 032 000 000); 15 ITEM time STRINGTIME; 16 END; 17 RELATION B; 18 BEGIN 19 ITEM a CODE (007 001 120 * 001 033 * *) ; 20 ITEM time STRINGTIME; 21 END; 22 RELATION B1; 23 BEGIN 24 ITEM a CODE (007 001 120 002 001 033 000 000); 25 ITEM time STRINGTIME; 26 END; 27 RELATION C; 28 BEGIN 29 ITEM a CODE (007 001 120 * 001 005 * *) ; 30 ITEM time STRINGTIME; 31 END; 32 RELATION C1; 33 BEGIN 34 ITEM a CODE (007 001 120 002 001 005 000 000): 35 ITEM time STRINGTIME; 36 END; 37 RELATION D; 38 BEGIN 39 ITEM a CODE (007 001 148 * 102 010 * *) ; 40 ITEM time STRINGTIME; 41 END: 42 RELATION D1; 43 BEGIN 44 ITEM a CODE (007 001 148 002 102 010 000 000); 45 ITEM time STRINGTIME; 46 END; 47 RELATION E; 48 BEGIN 49 ITEM a CODE (007 001 061 * 010 006 * *) ; 50 ITEM time STRINGTIME; 51 END; 52 RELATION E1: 53 BEGIN 54 ITEM a CODE (007 001 061 002 010 006 000 000); 55 ITEM time STRINGTIME; 56 END: Figure 4. A partial PAL program for the frame in Figure 5. TITLE: Pleuritic chest pain TYPE interpretation FINDINGS: a. chest pain worsened by breathing b. pain worsened by cough c. chest pain sharp in quality d. recent chest trauma e. Exam: chest pain reproduced by palpation REQUIREMENTS: True if (a or b) and c and not (d or e) Figure 5. A free-text version of an interpretation know ledge frame in Iliad. 1 1 1 2 2. RCOM [14] This is a somewhat user-friendly knowledge-base editor that is now used mainly in research and by students in the knowledge engineering class. frames interactively. not need to do It allows users to create, review and modify In the process of building a frame, users do any conventional programming. Instead, to create a frame, they need only select commands or options in menus provided by RCOM. To find the code for a finding in a frame, the user provides one or more key words and selects an appropriate item from the list generated from these keys. RCOM also provides a "help" feature for most commands, which tells the user what options are available next, RCOM also shows examples of how to use each command. The advantage of this editor is that it is user friendly, i.e., everything is done interactively. The disadvantage is that one has to spend time reading the manual and learning to use the editor, also, the frame building process is still time­consuming. Figure 6 shows the frame (in Figure 5) in RCOM version in which medical terms in the text frame have been replaced by the dictionary terms. Iliad Currently, the knowledge building process for the Iliad expert system is as follows. Experts and knowledge engineers define the frame logic in text form during knowledge engineering meetings. Using hardcopies of these frames, knowledge engineers convert these frames in the Tandem computer to the RCOM language. These frames are compiled BLOCK #7.141 SECTOR 113 SECTOR LOGIC: PLEURITIC CHEST PAIN A SEARCH: IS YOUR CHEST PAIN INCREASED BY BREATHING DEEPLY? MODIFIER: LAST IF: aAexist FROM: $NOW-2 DAYS TO: $NOW USE:aAval B SEARCH: IS YOUR CHEST PAIN INCREASED BY COUGHING? MODIFIER: LAST IF: aAexist FROM: $NOW-2 DAYS TO: $NOW USE:aAval C SEARCH: IS YOUR CHEST PAIN SHARP OR STABBING? MODIFIER: LAST IF: aAexist FROM: $NOW-2 DAYS TO: $NOW USE:aAval D SEARCH: HAVE YOU INJURED YOUR CHEST RECENTLY? MODIFIER: LAST IF: aAexist FROM: $NOW-2 DAYS TO: $NOW USE:aAval E SEARCH: CHEST PAIN REPRODUCED BY PALPATION MODIFIER: LAST IF: aAexist FROM: $NOW-2 DAYS TO: $NOW USE:aAval F ARITH: IF (AAVAL = 5 OR BAVAL = 5) AND CAVAL = 5 AND (DAVAL = 1 AND EAVAL = 1) THEN 5 ELSE 1 FINAL EVALUATIONS: A FE [VAL] FAVAL 1 3 Figure 6. A ReOM version of the interpretation frame in Figure 5. The final evaluation' (the value of item F which is either 5 for yes or 1 for no) will determine whether the symptom is present or not. 1 4 into pseudo codes (low level code) and then downloaded into the Macintosh where they can be executed by the Iliad system. If, after a period of testing, some frames need change in logic or parameters, knowledge engineers have to modify and recompile them on the Tandem, and download them anew into the Macintosh. This modification and downloading process continues until the decisions made by the system satisfy the experts in the medical subject area. This is not only a very inconvenient way to build and debug a knowledge base but it also means that Iliad is not a stand-alone system based on the Macintosh. Current Research Goals To some extent, the tools/languages described above, whether user-friendly or not, need many user interactions, which proved very time-consuming. For the Iliad system, there is one more trouble, i.e., transferring knowledge frames from one machine to another. Aware of these problems, we started to design and develop a knowledge building tool specific for the Iliad system. This tool, named Translator, would reside on the Macintosh machine and gIve Iliad an even more sophisticated and friendly user interface than those reviewed in the last section. Figure 7 shows a design diagram of this tool and explanations are given in the following paragraph. The knowledge translation process begins with text frames, which can be written using any word processor. The Translator reads a frame at a time; analyzes the structure of that text frame; and translates it into an intermediate language in which the 1 5 KNOWLEDGE BUILDING PROCESS text , intermedi ate , knowledge frames /1' , language r ...... , frames Translator 1 Compiler Di ct i onary Mapper Figure 7. Overall design of the knowledge building tools for Iliad. Knowledge frames in free-text form will be translated by the Translator into an intermediate language in which medical terms used in the text frame are replaced by the dictionary terms (aided by the Dictionary Mapper); and algorithms are automatically generated. Frames in intermediate language will be compiled by the Compiler to become executable code for the system. 1 6 required algorithms and logic have been automatically constructed. The translation process will frequently call a dictionary utility program, which acts as a subroutine of the Translator, in order to find a code to match each text string representing an item of information used to process the decision defined by the frame. The frame in intermediate language form needs further compiling, and finally becomes an executable frame in the system. The goals of this research are : 1. to eliminate totally the need to write a program to represent the frame; 2. to automate the search of the dictionary to find the appropriate code to represent a term in any frame. In this whole design process, I am responsible for the design and implementation of the Translator. Therefore, the first goal above becomes my research goal; that is the success of implementation of the Translator will allow experts/users to build knowledge frames without doing any conventional programming. With this major goal in mind, I started to design and implement the Translator. CHAPTER II THE TRANSLATOR Prior to any compiler or translator design and implementation we must first precisely define the syntax of the language - in our case the knowledge programming language for the expert system Iliad. As mentioned earlier, a good expert system should provide a natural language interface for its users. The Translator is a software tool that acts as such an interface between experts/users and the Iliad expert system and permits experts and knowledge engineers to directly transfer their knowledge into a computer readable form with ease. Therefore, the language to be used by experts should be defined to be as natural as possible and as simple as possible for easy use by the experts/users; yet it still should be powerful enough to allow experts to create knowledge frames without loosing information. In other words, it must meet the requirements of the knowledge representation models used in the expert system. The Features of the KnowledKe Representation Models There are several ways to represent knowledge for an expert system such as rule-based, frame-based, semantic networks and Bayesian probability approaches. Currently, the knowledge re- 1 8 presentation models used in Iliad system are of two types: Interpretation model and Probability model; also called cluster and decision frame. The former is a rule-based model in which clusters of findings, which are conditionally dependent, about a disease entity or pathophysiologic process are collected; and Boolean expressions are used to define the frame logic. Such a frame (or cluster), usually evoked by other cluster or decision frames, returns a Boolean value either 'yes' or 'no,' indicating the presence or absence of a symptom (indicated by the frame title). This clustering approach is considered to closely emulate the logical analysis used by domain experts in making medical decisions in practice [2]. Figure 8 shows two interpretation frames for pulmonary diseases, "lung consolidation" and "pleural effusion." The latter type is referred to as the Bayesian model in which many findings, which are independent or made independent by the use of an "Else" structure (explanations and examples will be given later), are collected together with their sensitivities and specificities; and the Bayes theorem is used to formulate the problem in terms of estimating the probability that a patient does or does not have each disease. This approach makes the analysis more quantitative and objective and facilitates the ranking and comparison of multiple hypotheses [2]. Figure 9 illustrates a· probability frame for the pulmonary disease "Wegener's Granulomatosis," which uses the two clusters, "lung consolidation" and "pleural effusion" as findings. We have designed a knowledge programming language that supports both of these models. TITLE: Lung consolidation TYPE interpretation FINDINGS: a. Coarse crackles (rales) b. Bronchial breath sounds c. Egophony (E-a changes) d. Increased vocal fremitus e. Dullness to percussion f. Whispered pectoriloquy g. CXR - lung infiltrate with alveolar pattern and air bronchograms True if g or if (a and b) and (c or d or e or f) TITLE Pleural effusion TYPE Interpretation FINDINGS: A. @7.141.113 Pleuritic chest pain B. Unilateral dullness to percussion C. Locally diminished breath sounds D. Locally decreased vocal fremitus E. CXR shows free-flowing effusion Requirement for yes TRUE if: e or any 3 of (a,b,c,d) Figure 8. Two examples of interpretation frames used in Iliad. 1 9 In these fram~s, clusters of findings are collected. Each finding may be a symptom, a physical exam, a lab test or another frame which is represented by the frame number and name preceded by a "@" sign. The value of the Boolean expression will determine the presence or absence of the symptom (frame title). TITLE Wegener's Granulomatosis TYPE probability PREVALENCE (a priori) = .0002 FINDINGS a. @7.141.112 Lung consolidation b. @7.141.115 Pleural effusion c. hemoptysis else cough d. Age under 20 20 to 39 40 to 59 60 or older e. arthralgia f. History of red, painful, itchy, or dry eyes or eye drainage or tearing g. History of prolonged sinusitis or sinusitis on X-ray else nasal congestion without sinusitis h. Ulcer on buccal mucosa or palate i. Middle ear fluid or inflamed tympanic membrane else earache or hearing loss j. CXR - cavitation else CXR - multiple nodules k. Pathology - necrotizing granulomas disease no disease .40 .10 .50 .05 .50 .05 .80 .15 .05 .05 .20 .26 .60 .27 .15 .42 .50 .10 .50 .10 .40 .05 .50 .10 .30 .02 .40 .05 .20 .10 .33 .0019 .50 .013 .80 .0001 Figure 9. An example of probability frame used in Iliad. 20 Findings are collected together with sensitivities and complement of specificities to revise the apriori probability. To avoid overconfident results, one can use the "Else" structure to group dependent findings. Findings may be other frames (such as item a and b, which are frames in Figure 8). 21 The Syntax of the Language The best way to describe the syntax is to study some examples. Let us examine both interpretation and probability frames in Figure 8 and 9 again. Both frames begin with the word "Title," which is a key word. This means each time experts write a frame, they must provide this word followed by an English phrase. This phrase is the real title for the frame. It represents either a clusters of findings (interpretation frame) or a decision (probability frame). When an interpretation frame is evoked, it returns either yes or no indicating whether the cluster logic is satisfied or not. When a decision frame is evoked, it returns a probability. Some examples of these frame titles are shown below: Title Acute myocardial Infarction ( probe ) TITLE Infarction Chest Pain ( interp. ) title Bleeding esophageal varices ( probe ) Title upper gi bleeding ( interp. ) The second part in the frame begins with the key word "Type" indicating which kind of knowledge model it is. There are two options: "Probability" and "interpretation, n which follow the key word "Type." The third part follows the key words n A priori" or "Prevalence. It This part is required only when the frame is of probability type. The user provides a number derived from the medical literature, a patient database or an expert's estimate to represent the prevalence of the disease in the population to which the frame will be used for making decisions in future patients. The a priori may be calculated by another frame where prevalence of a 22 disease is dependent on patient age or sex or other risk factors. Optionally users can also provide another number as a multiplier to modify the prevalence. The following are some examples: A priori = 0.08 a priori = @7.142.101 Risk of Coronary Artery Disease (Sign '@' indicates the following is a frame; 7.142.101 is the frame number followed by frame name. ) Prevalence = 0.5 * 0.4 A priori: If sex=male and age <=44 then 0.0035 If sex=male and age 45-46 then 0.075 If sex=female and age <=44 then 0.0011 If sex=female and age 65+ then 0.096 (This prevalence is for Coronary Artery Disease) One important part in any frame is the list of findings that are symptoms, physical exams, lab tests or other frames. This part is labeled by the key word "Finding;" plus "disease" and "no disease" if current frame is a probability one. Each finding starts with an alpha character as a label, and is written in one or more lines; and, if a probability frame, followed by two real numbers that represent sensitivity and the complement of specificity (I-specificity). Sensitivity, represented by "disease," here means the probability of the presence of this finding in a patient with the disease. The complement of specificity, represented by "no disease," here means probability of the presence of this finding in a person without the disease. For example: Findings a. Chest pain, substernal b. Chest pain, lasting > 15 minutes Findings g. @7.143.127 Cholestatic jaundice h. anorexia disease .05 .86 23 no disease .02 .20 In a Bayesian frame one may wish to represent a continuous variable as a series of discrete intervals, each with a sensitivity and I-specificity for the disease. Therefore the language supports the following structure: and Findings a. Age under 35 35 to 49 50 to 69 70 or older k. fasting serum gastrin If two or more findings 0-200 pg/ml 200-1000 > 1000 in a probability disease .001 .09 .66 .25 .02 .48 .5 frame are no disease .15 .30 .35 .20 .98 .019 .001 dependent on each other, they should be grouped by the "else" structure. This means that only one of them will be used to revise the probability. In the following example two findings are dependent on each other. Therefore, key word "Else" is used to group these two findings. The second finding will be used only if the first one is not available since it has the most information to contribute to the decision. Findings disease no disease d. holosystolic or decrescendo murmur, 24 apical radiating to axilla .90 .02 Else murmur as above, radiating to base .10 .06 (Adapted from frame "Mitral regurgitation, chronic" ) The final part of a frame is the logic statement, which is required only when the frame is an interpretation type. This part requires key words "Requirement Jor yes" or "True," followed by a Boolean expression. Any precisely worded Boolean expression is acceptable; but to be more natural, some loosely worded expressions are still valid. The following are some examples: True if a and band (c or d) and not (e and f) True if a or «b or c) and (d or e» True if a, b, c or d True if any 2 of a to c True if 2/3 of a-c or 1/3 of d to f With those features above defined for the language, the Translator was designed and implemented to convert any frame written in this language into a structured knowledge frame capable of compilation into executable pseudo code for execution. A Guide to the Translator Like any Macintosh application the Translator is a menu driven program and easy to use. It provides users with four pull­down menus: File, Edit, Syntax and Compile. The "File" menu allows users to create new frames, review or/and modify existing frames, or print frames in both text and compiled versions. It also provides some other standard commands like "Save," "Revert," "Page setup" and "Quit," etc. Some of these 25 commands, like other applications, have command-key options. Figure 10 shows the "File" menu. When "New Frame" command is selected, an "Untitled" window appears, which allows users to type in their frame. The "Edit" menu provides standard functions like "Cut," "Copy" and "Paste." In addition to this, the Translator also provides a "Syntax" menu that helps users to design their own frame. The first command (Frame structure) in this menu tells users the frame structure in a window and provides more detailed information when the starting key words in that window are double clicked. The second command in this menu (Examples), when selected, will display examples of both probability and interpretation frames. Therefore, a user can easily create frames while taking those examples as references. See Figures 11 and 12. After editing is complete, the user must save the frame before compilation. The saving procedure is standard except that the Translator only accepts file names having the form "number.number.number" which is the format of frame name used in the Iliad system.. Figure 13 shows the "save" procedure. The last menu is "Compile," which includes commands "Parse Frame" and "Review Results." The first command allows a user to compile an already saved frame in the current window; or any existing frame by choosing the frame in a list. If compilation is successful, the user can review the compiled version of the frame by selecting the second command in the menu. The computer will display two windows, one of which is for original text and the other for the compiled version. See Figure 14. This is helpful for users to 1: los(~ S<H'(~ S<H.(~ fJ~", Ih~(J.~rt Page Setup .•. Prin t,,, Figure 10. The "File" menu. 26 27 • File Edit SyntoH Compile TITLE Duodenal Ulcer TYPE Untitled FRAME STRUCTURE TITLE Engl ish phrase TYPE Probabi I ity or Interpretation APRIORI* (or Prevalence in Hospital) FINDINGS Disease N REQUIREMENT FOA YES (or TRUE) ** *** RReeq4q,: u~o~~~~~~~}T!Y~PE[=j= ==~~~ Reserved Word: TYPE 1 Type or type fol lowed by Probabi I ity or Interpretat Examples: Type Probabi I ity TYPE Interpretation Figure 11. The use of the "Syntax" menu. This menu provides users syntax, rules (by selecting menu item "Frame Structure") and examples (by selecting menu item "Examples") for creating new frames. In this figure, the front window "TYPE," which explains how to write the model type, is activated by double-click the key word "TYPE" in the "FRAME STRUCTURE" window, which shows the structures of the knowledge frame. ~ J.. .. File Edit SyntoH Compile Untitled TITLE Duodenal Ulcer TYPE Probabi I ity APRIORI <Prevalence in hospital) 0.0055 FINDINGS a. age 0-29 30-59 60+ b. male sex c. @7. 143.114 Peptic ulcer pain else recent epigastric pain d. Nausea or vomiting e. History of duodenal ulcer disease .01 .54 .40 .72 .85 .95 .05 .60 no disease .09 .32 .59 .57 .02 . 15 .15 . 10 o EHRMPLES APRIORI PREVALENCE IN HOSPITAL = 0.01 FINDINGS a. ~7. 141. 115 Pleural effusion b. else ~7.149. 117 Constitutional sign c. @7. 141. 125 Sol itary pulmonary nodul d. age under 35 35 to 49 50 to 69 70 or older e. Male sex f. History of prolonged dai Iy cough 1.A-o I disease .2 .5 .33 .001 .01 .739 .25 .05 .5 no disease . 12 .2 .021 . 15 .35 .3 .2 .5 .16 28 Figure 12. The use of the "EXAMPLES" command in "Syntax" menu. U sing examples provided by the Translator, one can make one's own frame with ease. r Edit SyntaH Compile TITLE Duodenal Ulcer TYPE Probabi I ity Untitled APRIORI <Prevalence in hospital) 0.0055 FINDINGS disease no disease a. age 0-29 30-59 60+ b. male sex c. t7. 143. 114 else rece d. Nausea or e. History of f . F am i I Y his g. Tenderness physical e h. i7. 143. 113 i. Upper G i b I la project6/201 r.~ 2,2,2 r.~ .~, '~,'~ r.) <1,( r.~ <)~f~Hn J(~ r(~<)d,( r.~ (}d,( r.~ £1£1 ( o.h~ 17.143.1~ Figure 13. The tt Save" procedure. (g) Criclcet ( Eject ) ( Driue ) Salle ( Cancel) 29 After typing a new frame, one has to save the frame through a standard procedure with a name used in Iliad system. • File Edit SyntaH Compile FRRME TEHT TITLE Duodenal Ulcer TYPE Probabi I ity APRIORI <Prevalence in hospital) 0.0056 FINDINGS a. age 0-29 30-59 60+ b. male sex c. @7. 143. 114 Peptic ulcer pain else recent epigastric pain d. Nausea or vomitin disease .01 .54 .46 .72 .85 .95 .05 D RESULT BLOCK *7. 143 SECTOR 10 SECTOR LOGIC: =.===DUODENAL ULCER A ARITH: 0.0056 B SEARCH: AGE no disease .09 .32 .59 .57 .02 . 15 .15 30 MODIFIER: LAST IF: aAexist FROM: $NOW - 2 DAYS TO: $NOW USE C PROB: APRIORI: A-UAL IF BAUAL>O USE: BAUAL MIN (0.. 30, 60) TRUE(0.0100, 0.5400, 0.4600) FALSE(0.0900, 0.3200, 0.5900) D SEARCH: WHAT IS YOUR SEX? MODIFIER: LAST IF: QAexist FROM: $NOW - 2 DAYS TO: SNOW USE E PROB: APRIORI: AAUAL IF DAVAL>O USE: 04VAL MIN(70, 77) TRUE(0.2800, 0.7200) FALSE(0.4300, 0.5700) <Figure 14. The use of the "Show resultstl command. When this command in the "Compile" menu is selected, two windows appear in the screen displaying both compiled and original versions of the frame. 3 1 see how the original text is transferred into terms the computer can deal with. The most difficult task in the process of translation is to find a correct code representation in the Iliad dictionary for each finding. This task is done by an independent procedure. Each time the Translator encounters a finding, it passes it in text form to this procedure and waits for the corresponding dictionary code to be returned. Our goal is to make this process as automatic as possible, but at present the process requires some user interactions. Therefore during the process of translation, there are some text string that require the users to either choose from a list a dictionary code that maps to their terminology or add a new term to the dictionary. This process continues until every finding is matched to a dictionary code. The Translator takes a frame in text form as input and generates two compiled outputs: a text version and a coded version. The former is used for displaying the frame logic in an easily readable form, and the latter is saved in a file ready for compiling into pseudo code for. execution by Iliad. CHAPTER III DESIGN OF THE TRANSLATOR We already know the source language that has been described in the previous chapter. What is the target language of the Translator? As mentioned in Chapter I, during the whole process of knowledge implementation, there is an intermediate language that is generated by the Translator (refer to Figure 7 in Chapter I). Therefore, this language becomes the target language of the Tran sla tor. About Target Lan.:ua.:e The general formula for language translation is SLo -> ILl - > "" -> ILk -> .,', TLn, (n>= 1) where SL is the source language, IL is an intermediate language and TL is the target language [15]. Intermediate languages are conceptual tools used in decomposing the task of translation from the source language to the target language, How many of these intermediate languages and what kind of syntax of these languages depend on many factors such as cost, speed, simplicity, portability and so on. We use an intermediate language in the design of the Translator for simplicity, One might argue why it is simple to use an extra language instead of directly translating the source language into machine code. There are several 33 general approaches to compiler or translator implementation [16]. The simplest is to retarget - modifying a compiler to generate code for a new target language, or rehost - moving a compiler to run on a new machine. The idea is to make use of the existing compiler to render the implementation task simpler and easier. The same idea has been adopted in this design which makes use of an existing compiler on the Tandem computer. The compiler, called HCOM, is a knowledge base editor that is responsible for building knowledge frames on the Tandem. Although it is user-friendly, it can not compile frames in free text form. Now this compiler, after modification, has been moved to the Macintosh machine. Therefore, the syntax of the target language for the Translator is in the format required by this compiler. Our work has proved that it is relatively easier for the Translator to generate this target language than directly generate executable code for the expert system. Figure 6 (in Chapter I) is a target language version of the interpretation frame in Figure 5. Figure 16 is a target language version of the probability frame (in Figure 15). For the Translator, this is the syntax of the target language in one form, which is for the purpose of review. The other form is in code, which is the input to the Macintosh-based compiler. The coded form is the translated version of the text version into ASCII form except that each finding in the frame is replaced by a dictionary code. With this combination of Translator and the compiler, and with the help of the dictionary utility program, the knowledge building task from frames in free text form to executable form becomes possible and quite simple. TITLE Bleeding esophageal varices TYPE probability A PRIORI (PREVALENCE IN HOSPITAL) = .0002 FINDINGS a. age 0-39 40-59 60+ b. male c. @7.l43.113 upper gi bleeding d. hx of bleeding esophageal varices else, hx of varices or cirrhosis else, hx of alcohol abuse e. upper endoscopy: bleeding varices f. angiography: bleeding varices seen disease .10 .60 .30 .70 .98 .30 .60 .70 .82 .90 no disease .16 .25 .59 .57 .02 .001 .01 .05 .005 .005 Figure 15. A probability frame written in the source language. 34 BLOCK #7.143 SECTOR 43 SECTOR LOGIC: =.===BLEEDING ESOPHAGEAL VARICES A ARITH: 0.0002 B SEARCH: AGE MODIFIER: LAST IF: aAexist FROM: $NOW - 2 DAYS TO: $NOW USE: aAval C PROB: APRIORI: AAVAL IF BAVAL>O USE: BAVAL MIN(O, 40, 60) TRUE (0.1000, 0.6000, 0.3000) FALSE(0.1600, 0.2S00, 0.S900) D SEARCH: WHAT IS YOUR SEX? MODIFIER: LAST IF: aAexist FROM: $NOW - 2 DAYS TO: $NOW USE: aAval E PROB: APRIORI : A AVAL IF DAVAL>O USE: DAVAL MIN(70, 77) TRUE (0.3000, 0.7000) FALSE(0.4300, 0.S700) F SEARCH: PCODE:7 3 143 113 MODIFIER: LAST IF: aAexist FROM: $NOW - 2 DAYS TO: $NOW USE: aAval G PROB: APRIORI : A AVAL IF: FAVAL > 0 USE: FAVAL MIN(l, S) TRUE (0.0200, 0.9800) FALSE(0.9800, 0.0200) H SEARCH: HAVE YOU EVER HAD BLEEDING ESOPHAGEAL VARICES? MODIFIER: LAST IF: aAexist FROM: $NOW - 2 DAYS TO: $NOW USE: aAval I SEARCH: DO YOU HAVE CIRRHOSIS OF THE LIVER? MODIFIER: LAST IF: aAexist FROM: $NOW - 2 DAYS TO: $NOW USE: aAval J SEARCH: HAVE YOU BEEN DRINKING (ALCOHOL) HEAVILY RECENTLY? MODIFIER: LAST IF: aAexist FROM: $NOW - 2 DAYS TO: $NOW USE: aAval 35 K ARITH: IF H=S THEN 1 ELSE IF I=S THEN 2 ELSE IF J=S THEN 3 ELSE IF H=l OR I=l OR J=l THEN 4 L PROB: APRIORI : A AVAL IF: KAVAL > 0 USE: KAVAL MIN(l, 2, 3, 4) TRUE(0.3000, 0.6000, 0.7000, 0.3000) FALSE (0.0010, 0.0100, O.OSOO, 0.9S00) M SEARCH: BLEEDING ESOPHAGEAL VARICES (DIAGNOSIS BY UPPER ENDOSCOPY) MODIFIER: LAST IF: aAexist FROM: $NOW - 2 DAYS TO: $NOW USE: aAval N PROB: APRIORI : A AVAL IF: MAVAL > 0 USE: MAVAL MIN(l, S) TRUE (0.1800, 0.8200) FALSE(0.99S0, O.OOSO) o SEARCH: BLEEDING ESOPHAGEAL VARICES (ANGIOGRAPHY) MODIFIER: LAST IF: aAexist FROM: $NOW - 2 DAYS TO: $NOW USE: aAval P PROB: APRIORI: A AVAL IF: OAVAL > 0 USE: OAVAL MIN(l, S) TRUE(0.1000, 0.9000) FALSE(0.99S0, O.OOSO) FINAL EVALUATIONS: A FE [VAL] AAVAL Figure 16. A frame in target language. This is a probability frame (in Figure 15) generated by the Translator, which, at same time, also generates a version in code for the Compiler. 36 Software Engineering Approach The development of a translator for a given language on a computer is a major software project. The Translator writing is an exercise in software engineering - a field comprising disciplines, tools and techniques for producing efficient, correct, reliable software. Therefore, many of the principles in this field have been adopted. In the process of developing the Translator, I followed the general patten of U software life cycle," which is shown in Figure 17, This cycle is a sequence of stages for producing software. Each stage has a definite beginning and a definite ending and treats a specific, defined set of issues. In the following paragraphs, I will briefly describe each of these stages with emphasis on the design stage. Analysis Simply, the analysis stage involves the investigation of the problem in order to know what the program must do. The final result of the analysis is a set of requirements or specification for the program. The requirements for our task are as follows: In order to build knowledge frames for the Iliad expert system, we need a tool to translate knowledge frames from text form into computer executable form. The translator accepts the source text document and transforms it into an object program. The source document is a member of a source language and the object program is a member of an object language. The syntax of the language for the source document has been defined in the previous chapter. The object language is an intermediate language in two forms as described in the previous section. One is in coded form, 37 " / Analysis l/ Functional Speci fi cat ion ,1/ Des; gn ,1/ Implementation ,l/ Validation t Figure 17. Software lifecycle. 38 which needs to be further compiled and then becomes executable code. The other is in text form, which is for the purpose of review and debugging. The Translator parses the source text and generates these two forms (text and code), as well as saves them on disc. As an important part of this translation process, the Translator reports to its user the presence of syntax errors in the source program. Figure 18 illustrates the function of the Translator. As a complete application, it also should provide text editing functions for users to create knowledge frames or modify the existing frames. The Translator runs in an interactive manner on Macintosh computers with a hard disc. Source ProgrBm , Translator "-IV Error Messeges '" TBrget ~-------/~ Progrem Figure 18. A translator. 39 Functional specification This stage is a complete technical specification of the behavior of the proposed Translator, which is as follows: 1. Since the Translator will run on the Macintosh machine, full Macintosh features such as menu bar with menu options, windows and dialog box should be implemented. In other words, the program should be menu driven, with clear, concise options prompting the user for a response. 2. An editor should be included as a part of this software. It will provide standard "Cut," "Copy" and "Paste" functions allowing user to create or/and modify text frames. Users also have options to save (save as), print, revert, open and close any text frames. 3. It should be user-friendly. Users should not need to keep referring to the syntax manual. Instead, they should be able to create text frames simply by requesting on-line help or examining the examples in windows activated by selecting some menu options. 4. Translator will check the validity of the file to be translated. It only accepts text files with names in the form of "number.number.number" (the current representation for frame number used in Iliad), 5. When errors are encountered, it should display an error message along with the whole text line containing that error. 6. It should provide dialog boxes helping users to find or create a dictionary code in the system's dictionary. 7. It should provide windows to display simultaneously the source text and translated version of a frame allowing a user to review the compiled output next to the input. 40 Design This is the most important development stage, which lays the foundation for the coding, testing, and maintenance activities. The principal purpose of this stage is to decide how to build the proposed Translator and let it meet the specifications defined above. This stage is further decomposed into abstract design and detailed design. In this thesis, I only describe the abstract design to show how the Translator works. Detailed design diagrams, algorithms developed and critical data structures used are omitted here. Any compilation or translation can be broken down into two major tasks: * Analysis: Discover the structure and primitives of the source program, and determine their meaning. *Synthesis: Create a target program operationally equivalent to the source program. This breakdown is useful because it separates our concerns about the source and target languages [15]. Commercial compiler developers usually spend much effort on the second part which requires the most specialized techniques for code optimization. In our case, however, we do not need to optimize the intermediate code generation. Therefore, the design stage is concerned only with the first part as is this thesis. Figure 19 shows the simplified operational diagram of the Translator. Descriptions are given in the following paragraphs. Translation begins with some source form as a text file on disc. The Operating System usually provides isolation of physical access characteristics from the Translator. So the Translator first sees some ~ I I I I Tokens Records Box 1 Lexical Analyzer Box 2 Parser Box 3 Computer Computer logic logic in code in text New files Figure 19. Operational diagram of the Translator. 41 Source Records 42 sequence of source records generated by box 1 in the diagram. With these source records as input, the lexical analyzer, box 2, then generates tokens which may be a single character or sequence of characters such as a key word "title," a number "0.55," labels "A," uB" or some special symbols like ">=," "(," etc. The lexical analyzer also performs certain secondary tasks. One such task is eliminating blanks, tabs and newline characters. Another is sending error messages generated by the Translator based on its analysis of the source program. Tokens then are processed by the parser, box 3, which is a critical part of this translator. Its task is to determine the underlying structure of the source program and the operations implied by the source text. Until the parser is reached, the tokens have been collected with little or no regard to the context within the source program as a whole. It is the task of the parser to consider the context of each token and classify groups of tokens into larger entities such as a frame title, a finding to be searched, or an "else" structure, etc. After the structural interpretation, the parser calls two routines, which belong to the synthesis task. These two routines generate two forms of outputs. One of them is a translated version in code ready for further compilation and the other is in text form (currently in ReOM form, but later it will be a simplified version understandable by any person who wants to study the compiled output). Both of these two results are stored as disc files. In the diagram, as well as in the real design, I separated the analysis task into lexical analysis and parsing in order to simplify the design and achieve some Translator efficiency. 43 In addition to the structural interpretation, a symbol table is constructed and managed. The symbol table provides an association of key words appearing in the source program. The principal technique used in this stage is "Top-down design method," that is describing the system from the top down, starting from an overall perspective and working down through levels of increasing detail. In other words, the modules subsidiary to a given module M divide among themselves the responsibility for performing module M's function. For example, the translation task can be decomposed into analysis and synthesis tasks as described above. These two functions can also be further decomposed into smaller functional modules. Figure 20 shows the functionally de­composed translator (but not to the lowest level). Except for the synthesis task, each task also has a subtask of error handling (but not shown in the Figure). An important factor in the development of a translator/compiler is that anew, experimental language may undergo dramatic change during the course of its construction. Decomposition and modular design make the design, im­plementation and . maintenance tasks much easier, and can help the developer cope with any change in the definition of the language. In the construction of this Translator, there were some changes to the language such as adding some new features like "time-constraint." This problem was solved simply by adding a new routine and modifying another existing routine. This task proved very easy. Again, a top-down design method was employed as a coding convention in the implementation stage. Structural Analysis Lexi ca 1 Analysis Analysis Parsing Transl ator Semantic Analysls Synthesi S Figure 20. Decomposition of the Translator. 44 45 Implementation This stage concerns coding and debugging, and the task is to translate the design into a working system. At this stage I used top­down coding and top-down integration of modules developed in the design stage. Also during this stage, more and smaller modules represented by routines were derived from decomposing the bigger ones, more detailed algorithms and small data structures were developed, and experiments were performed to test different strategies. The principal technique used for coding was modular design and structured programming. The Translator IS simply a computer program which, therefore, has to be coded in a programming language. Most compilers were written In low level languages that are difficult to code and understand. I want to use a system-oriented, yet friendlier language. The clear choice is "C." Therefore, the final product of Translator was coded in Light Speed C. Since the Iliad expert system is a Macintosh-based application, naturally the Translator should be implemented on the Macintosh. Such a computer system is easy to use and has a user-friendly operating system that provides a unique graphic-oriented user interface. The Translator was developed in such an environment, it is a truly "Mac-like" application. Validation This is the most important task in any software development project, and is currently under way. This task will be described in more detail in Chapter IV. Special Techniques Employed in Parsine 46 Since the language has been defined to meet the requirements of knowledge representation models used in the Iliad system as well as to be user friendly, its syntax is quite different from those general purpose hight level programming languages. Because of its unique features, some special methods have been developed to accomplish the translation task. Top-down parsing (scanning) This kind of parsing is used globally. That is, parsing starts from the first line of text frame to the end of file. Parsing in this way is consistent with the logic of the frame structure. Left to righ t parsing In general circumstances, for a single source record, parsing starts from left to right until reaching the end of line. This is also consistent with the logic of the frame structure. Right to left parsing In some places, parsing starts from right to left. Take a finding (starting with a label) in a probability frame for example: Findings: disease no disease C. Eosinophils >= 0.06 0.33 0.10 (Adapted from frame uDuodenal Ulcer" ) This finding contains three numbers in the text, the last two of which are the probabilities of the finding in patient with and 47 without the disease. If parsing were from left to right, we might take 0.06 as one probability and cause logic and probability revision errors. However parsing from right to left can overcome the difficulty of obtaining correct probabilities. This method is used in probability frames only. Backtracking The Translator has some "smart" characteristics. If new information is found for a previous finding, control will go back to do some extra tasks. In some findings, we allow numbers to appear in the text. For example: Smoking history > 10 pack-years. This number (10) also appears in the dictionary text. Translator can simply pass the all text to the dictionary search routine to get the matching code. However in some cases, numbers should not be considered as part of the text. The following finding is an example. Findings: k. Fasting serum gastrin 0-200 pg/ml 200-1000 > 1000 (Adapted from frame "Oastrinoma" ) disease .02 .48 .5 no disease .98 .019 .001 Translator should not pass "0-200 pg/ml" to the dictionary search routine or else it will make the search task much more difficult. Backtracking can solve this problem. Whenever Translator finds new information in second line, it will eliminate the phase "0- 200 pg.ml" from the first line, pass the rest text to the search routine, and create multiple bins for the revision. Rearranging the "ELSE" structure 48 subsequent probability In order to avoid overconfident results caused by dependent findings, we use an "ELSE" structure to group such findings. An "ELSE" structure may contain two or more findings, but only one can be used to revise the probability. When this structure is detected in parsing, the Translator will call a sorting routine, the result of which will be used by the synthesis part to generate a structure different from the original frame text in order. The purpose of this sorting is to order the findings in the "Else" list so that the one with the highest information content will be used in the Bayesian calculation if the needed information for this finding is available for the case (patient) being analyzed. Error HandJiD2 One important task for the Translator is to detect errors in the source program and report their presence to its users. This task is called error handling, the purpose of which is to aid the experts and users by highlighting inconsistencies in the source program. It has a low frequency of use in comparison with other Translator tasks, but it can not be regarded as an "add-on" feature of the Translator. What is more, it is a necessary debugging tool during construction of the Translator itself. The error handling task was designed but has not yet been implemented. In the design part of this project, it is assumed that 49 in each stage of translation - lexical analysis, parsing, semantic analysis and so on. The Translator will report any error detected, and furthermore, try to recover the program from error status by providing dialog windows and prompting user interactions. For example, if the Translator detects that a probability is missing for a finding in a probability frame, it will display a dialog box with a warning message and ask the user to input the missing probability. Translation will not proceed until the information required is completed. This feature and other error handling routines remain to be implemented later. Right now there are only dummy routines and most of time the Translator will exit the execution when an error is encountered. CHAPTER IV TRANSLATOR TESTING AND RESULTS Translator Testing and Methods As mentioned in Chapter III, the last stage in the software life cycle is validation or testing, the process of executing a program with the intent of finding errors. This stage involves the comparison of the software product with the specification of intended behavior. It is almost a statistical certainty that some errors will be present in a large software system when the work of the implementation IS finished. Despite our best intentions, errors tend to arise in formulating program specifications, in designing the required system, and in coding the programs. Therefore, testing is a very important activity in the development of a software and should be as thorough as possible. The actual process of Translator testing began during the implementation stage, (in the coding and debugging process where some of the procedures were tested) and continues at this time as a separate stage. There are many software testing strategies to choose from. I used the Bottom-up testing, B lack-box testing and white­box testing [20]. As stated earlier, modular construction has been adopted in the design of the Translator. Naturally, modular testing has been 5 1 used in the testing task, which involved the process of testing the logical units of the Translator (Le., procedures In "cn) individually, and integrating the individual module tests to test the overall system. The objective of module testing is to determine whether the module meets its specification. In contrast to the design stage (where a top-down method was used), bottom-up testing strategy has been used throughout the validation stage. Using this incremen tal testing strategy, I first constructed an acyclic graph to represent relations of all the routines in the system, and started testing with routines at lowest level (routines that do not call any other routines) in the program. Driver routines were first constructed to simulate the function of the routine superordinate to the one being tested (constructing a dummy main routine to call each routine being tested), while providing appropriate parameters according to the specification of that routine. The major advantage of bottom-up testing is that there is no difficulty in creating test data, since driver routines simulate all the calling parameters even if the data flow is not organized into a directed acyclic graph. The major disadvantage of this strategy is that a preliminary version of the system does not exist until the last routine is tested. While global testing strategy was from bottom to top, local testing strategy for each routine was black-box testing. In black­box testing (also called specification-based testing), each routine is viewed as a black box whose internal structure is unknown. The objective is to find out solely when the input-output behavior of the program does not agree with its specifications. In this approach, I first designed test data for all possible situations according to each 52 routine's specification, and ran the program to see if it could produce the expected results. The major drawback of this strategy is its dependence on the specification's correctness (which may not be the case in practice). For most critical routines, in addition to black-box testing, white-box testing (also called program-based testing) has been used. In this approach, the structure of each routine was examined and test data were derived from the program's logic. The objective is to cause execution of a specified group of execution paths within the program. One drawback of this approach is that it is very time­consuming to design test data. Therefore, in real testing process, I first, according to the program's logic, designed test data that could cause some specific and critical statements and conditional branches to be executed, and ran the program to see if any errors occurred. After all individual routines and subsystems were tested, the whole system (Translator) was tested. This task has involved a full­scale use of the Translator in a live environment, i.e., creating, modifying and compiling frames with various structures of both probability and interpretation types; and meanwhile testing Macintosh features like windows, pull-down menus, dialog boxes and so on. The objective of this system testing is to see if it works properly in the Macintosh environment, if it is user-friendly and if it can produce precisely the code in the format required by the Compiler. 53 Results Program correctness Each module works properly and meets its specification. The whole system works properly and meets the requirements defined in the specification stage: 1. Translator runs on the Macintosh machine with full Macintosh features. Menus, windows and dialog boxes all function well. The whole program is menu driven. 2. Like any Macintosh-based compiler, it has an editor that provides standard "Cut", "Copy" and "Paste" functions. Users also have options to save (save as), print, revert, open, and close any frames. 3. It is user-friendly and provides on screen instructions and examples to help users create new frames. 4. Translator checks the validity of each file to be translated. 5. It provides dialog boxes helping users to find or create a dictionary code for each finding. 6. It provides windows to display both text version and translated version of a frame allowing user to study the logic. Only one requirement in the specification has not yet been met. That is the error handling function. As mentioned in Chapter III, this function will be implemented later as our second phase work. Software testing can only prove the presence of errors and never prove the absence of errors. Proving that a program is correct is often more difficult than writing the program. It has been estimated [8] that the stages from analysis through design require 54 40% of the total effort; the implementation stage requires 20% of the effort; validation requires 40% of the effort. Therefore, testing and subsequent maintenance still need much more attention and are a long-term task. Simplicity The implementation of the Translator makes the knowledge building task for the Iliad system much simpler and faster than before. This IS apparent because the task no longer needs the Tandem machine and the downloading procedure; and what is more, it no longer needs any conventional programming effort. Many logic constructions required by the knowledge base editor ReOM such as creating probabilities (with or without multiple bins), arithmetic and final evaluation statements [14] are done by the Translator. A lot of work required by the users previously such as keeping track of the item reference and hand calculation for sorting the findings in an "ELSE" structure, and even some dictionary searches work such as "age," "sex" and calling other frames are automatically performed by the Translator. The language is easy to learn and the Translator is easy to use. These characteristics should make it a convenient environment for physician or subject expert to use when building knowledge frames for the Iliad. Speed Since the Translator is designed to process knowledge frames In text form and produce precompiled forms automatically, the only time needed for creating the knowledge frame is text typing and 55 editing (including logic construction) and dictionary searching. Let us assume that dictionary searching times using the Translator, ReOM or PAL are equal for a probability frame. U sing the Translator, one does not need to do any extra work in addition to dictionary searching; using ReOM, one has to construct sea rch, arithmetic, probability and final evaluation statements explicitly in addition to dictionary searching; using PAL, one has to do traditional programming in addition to dictionary searching. Therefore, the speed of knowledge building using the Translator is significantly increased. The Translator is faster than ReOM, and much faster than PAL. Based on 10 interpretation frames, the ratio of the speed of ReOM to the Translator is 3.75. Based on 7 probability frames without complex structures like "ELSE," the ratio is 5.71. Based on 8 probability frames with complex structures, the ratio is 7.91. So on average, the Translator is almost six times faster than ReOM. These results are shown in Table 1 and Figure 21. Accuracy A translator/compiler must generate correct code. Therefore, accuracy is the most important measurement for the Translator. Since the knowledge frame implementation is divided into two sub­tasks, logic construction and dictionary search, accuracy here means the correctness of logic and algorithms generated. As mentioned earlier, the error handling and recovery subsystem has not yet been implemented, and it is difficult to measure the accuracy without these subsystems fully functioning. There are about 190 knowledge frames operational in Iliad. Because some of them have special 56 Table 1 Speed Comparisons of RCOM and Translator * Time is in minute and second ** frames with complex structure Frame number Type Time*: HCOM Translator Ratio Average 7.141.113 In terpreta ti on 5:00 1 :36 3.125 Dummy In terpreta tion 3:45 0:45 5 7.141.124 Interpretation 10:30 2:30 4.2 7.141.115 In terpretation 7:30 2:00 3.75 7.142.110 In terpretation 7:50 2: 15 3.48 3.75 7.142.120 Interpretation 2: 12 0:35 3.77 7.142.122 In terpretation 4:30 0:45 6 7.141.116 Interpretation 4:47 1 :30 3.19 7.141.126 Interpretation 6:00 2:25 2.48 7.143.122 Interpretation 4:00 1 :36 2.5 7.143.31 Probabili ty 13: 15 2: 15 5.89 Dummy Probabili ty 5:45 0:45 7.67 7.141.14 Probability 10:30 2: 15 4.67 7.143.28 Probability 12:00 2:20 5.14 5.71 7.142.35 Probabili ty 11 :00 2:18 4.78 7.143.41 Probabili ty 9:00 1 :32 5.87 7.143.64 Probabili ty 21 :45 3:40 5.93 7.141.114 Probability** 23:00 2:30 9.2 Dummy Probability** 8:00 0:45 10.67 7.142.71 Probability** 15 :30 3:00 5.17 7.142.101 Probability** 13:30 1 :40 8.1 7.91 7.142.30 Pro babili ty * * 23:30 3:00 7.83 7.143.24 Probability** 15:00 2:00 7.5 7.143.34 Probability** 1 :00 3:45 6.67 7.143.43 Probability** 15:00 1 :50 8.18 Speed Ratio 10 7 5 HCOM ~.------------~.~------------_.Trensletor interpretation frame probability frame (simple) probability frame (complex) Figure 21. Comparisons of speed for HCOM and the Translator. 57 58 structures and some of them were not written in correct syntax (examples and explanations will be given in next chapter), the current version of the Translator can only process about 70% of them. This percentage is based on random sampling of 30 frames. From our past experience with HCOM, we know that knowledge developers did sometimes make mistakes such as typing errors (type a number 0.01 instead of 0.001), reference errors (reference a wrong item number), sorting errors for the "ELSE" structure and so on. However the Translator does not make such errors. I did try some frames (10 interpretation and 15 probability frames) that were written fully consistent with the syntax. The results are satisfactory: 100% correct in logic. This testing strategy is not ideal because I assumed that source frames were correctly written (which is not always the case). Therefore, to make the Translator work correctly and to measure its accuracy adequately, we have to wait until the error handling routines are implemented. CHAPTER V DISCUSSIONS AND CONCLUSIONS Summary An expert system should have a good supporting environment that provides an easy way to capture and encode experts' knowledge, and transfer it into the system's knowledge base. In other words, it should have efficient tools for building the expert system -- the tasks of knowledge acquisition, knowledge pro­gramming and knowledge refinement. Experts' knowledge, when built into the knowledge base, is in the form of computer programs that can be implemented by any programming language. Unfortunately, general purpose high level languages (called application languages), though more user-friendly than low level languages, have proved to be tools that are not easy to use by experts. Neither are they suitable for various kinds of expert systems because of the internal structures of those systems. Therefore, knowledge acquisition, knowledge programming and knowledge refinement become a bottle-neck for most expert systems. In the final analysis, most expert system builders have to design and implement language tools specific to their own systems. These kinds of languages are usually called surveillance or query languages, and in most situations, they have been built on the 60 application languages. The surveillance languages have been designed to be clearer, more user-friendly than the application languages. HCOM (for HELP system) [14], CARE (for Regenstrief system) [21], and MQL (for COSTAR system) [22] are three examples of these languages. During the compilation process, knowledge frames written in these languages are first translated into the application languages, which are then compiled into the machine languages [23]. These languages have proved good for knowledge programming. They are easy to use. We can consider programming languages as layers [23], which are shown in Figure 22. The first layer is machine language the second one is application language and the third one is surveillance or query language. Just as the application language simplifies the expression of many manipulations to be carried out by the machine language so does the surveillance or query language simplify the expression of complex special purpose manipulations to be carried Query Language Application Language Machine Language Figure 22. The layering of languages. 6 1 out by the application language [23]. For example, an expression "IF (TEMPERATURE .GT. 37.5)" in FORTRAN may take many lines of code in machine language while an expression "ITEM itemN arne CODE (007 001 120 002 001 033 000 000);" in PAL may take many lines of code in FORTRAN to do the same task. Consequently, the further out the language resides, the more user-friendly it is. The expert system Iliad does not have a knowledge programming tool for building its knowledge base. Therefore, a language has been designed by domain experts and knowledge engineers, and a language translator has been implemented on the Macintosh for the Iliad system. The syntax of this language is very simple and natural to represent both interpretation and probability models used in the Iliad knowledge base. To take advantage of an existing compiler, this language was built on the HELP knowledge base editor HCOM. In other words, we built a language one layer above the HCOM language. The Translator first translates the knowledge frame written in the Iliad language into HCOM language (the knowledge frame in the language), which is then compiled by a compiler into an executable (frame in the) target language (also called pseudo code). Surrounding this language, there are utilities to aid in its use. There is a text editor for writing, editing and reviewing the frame. There is a function providing on screen instructions and examples of the syntax of this language to aid users in writing. There is a dictionary access routine for helping the users to search the Iliad dictionary file. 62 Conclusion In the Iliad system, two knowledge representation models have been used: interpretation (ruled based) model and probability (Bayesian) model. The knowledge programming language defined for the Iliad system has made it possible to capture and encode the expert's knowledge, and represent this knowledge in one of those two models without loosing any information. The language has been built one layer above the HCOM. Therefore, it simplifies the expression of many special purpose manipulations to be carried out by HCOM. For example, an "ELSE" structure in a text frame will take an "ARITH" item and a "PROB tI item in HCOM (see HCOM manual [14] and the example in Figure 16) plus some hand calculations for sorting (to rearrange the order of) those findings in the structure. It conceals the underlying layer, and users of this language need to know nothing about the HCOM. Beyond simply saving the work required of the HCOM, it allows expression of new levels of abstraction and conceptualization in analysis of the patient database. The syntax of this language is easy to learn, and it allows users to express their knowledge in nearly free text form without any programming effort. Therefore, it provides a more user-friendly environment for the users than its lower layer language HCOM. The Translator works much faster than HCOM and PAL and can produce accurate results. It has been implemented on the Macintosh machine, which is easy to learn and use. In summary, this project has met the initial research goal: to design and implement a translator for use in building a knowledge 63 base for the expert system Iliad. This translator eliminates totally the need for programming. I believe the implementation of the Translator provides an important supporting environment for Iliad, and makes Iliad a complete stand-alone expert system with a useful natural language interface. This Translator has not eliminated the intellectual effort involved in knowledge engineering for the Iliad system, but it has made this task much more efficient and eliminated many of the errors introduced by manual translation. Discussion There are some limitations of this knowledge building system: 1. The frame in HCOM form is an intermediate language in the process of building a knowledge frame. Each time users create a frame, they must first evoke the Translator, and then run the compiler. This two-stage task reduces the overall speed and requires that the user has to learn how to evoke the compiler in addition to running the Translator. Therefore, further enhancement may combine these two programs into a single system, eliminate the intermediate language in the translation process and consequently make the knowledge building process even faster. 2. There are still some special structures used currently in the knowledge frame that the Translator does not handle. The following are two examples: Findings A Gastric ulcer B ", with radiating folds C "with smooth edematous crater margins and D " , with irregular crater margins Requirement for yes 64 True if Predicted FEVl/FVC - Measured FEVl/FVC > 9.06% for a female or > 8.28% for a male. In the first example, quotation marks are used in the findings for typing abbreviation. The current version of the Translator does not support this structure. Further enhancement may be putting a me m 0 r y function in the program and allowing the program to remember what the equivalences for those quotation marks are. The memorizing work can be done because the translation process always keeps two findings at a time in the program. When it detects one or more quotation marks, it can go back to the previous finding and memorize the corresponding equivalencies. Unlike most high level languages, HCOM does not have a requirement for variable declaration. In the second example, variables FEV 1, PVC and sex are referenced. Referencing variables in this way is not consistent with the convention of the HCOM language where labels of findings (such as A, B or C) are used for reference. Therefore, it is very difficult for the Translator to find the values of these variables in the previous findings because it does not know when and where to do the memorizing work. This is different from the first example in that the translation process no longer keeps any previous finding in the program when it reaches the "Requirement for yes" part in a frame. However I belive it is still not an insurmountable task. 65 Al though our final goal is to make the Translator handle structures as complex as possible and let it be a truly natural language processor, we have to accomplish this task step by step. Therefore, our short-term goal is to make the Translator handle structures like time constraints and the following structure: Findings Disease no disease a. @7.142.101 Risk of Coronary Artery Disease low risk 0.0 to 0.25 .05 .45 medium risk 0.26 to 0.45 .25 .35 high risk > 0.45 .70 .20 and so on. Our long-term goal is to make the Translator handle complex structures like the examples discussed above. Although the Translator is a valuable tool for the Iliad system, it is still a long way to go to make it handle natural language and to access data dictionary automatically. Therefore, this work remains to be the future task in the Department of Medical Informatics. REFERENCES 1. HUKILL, M. J., WARD, K. M., HAUG, P. J., AND WARNER, H. R. HELP decision support on the Macintosh. "Proceedings, the Eleventh Annual Symposium on Computer Applications in Medical Care, Washington DC," pp. 155-157, 1987. 2. WARNER, H. R., HAUG, P. J., BOUHADDOU, 0., LINCOLN, M., AND WARNER, H. R. Jr, SORENSON, D., WILLIAMSON, J. W., AND Fan, C. Iliad as an expert consultant to teach differential diagnosis. "Proceedings, the Twelfth Annual Symposium on Computer Applications in Medical Care, Washington DC," pp. 371-376, 1988. 3. SORENSON, D., BOUHADDOU, 0., WANG W, CANFIELD, G., Fu, L., AND WARNER, H. R. Generation and maintenance of Iliad medical knowledge in a Hypercard environment. "Proceedings, the Twelfth Annual Symposium on Computer Applications in Medical Care, Washington DC," pp. 366-370, 1988. 4. WATERMAN, D. A. "A Guide to Expert Systems." Addison­Wesley, Reading, Mass., 1986. 5. WARNER, H. R. Jr. AI solutions. Computers in Healthcare 7, 50 (1987). 6. Blum, B. I. "Clinical Information Systems." Springer-Verlag, New York, 1986. 7. BUCHANAN, B. G., AND SHORTLIFFE, E. H. "Rule-Based Expert Systems - The MYCIN Experiments of the Stanford Heuristic Programming Project." Addison-Wesley, Reading, Mass, 1984. 67 8. :MILLER, R. A., POPLE, H. E., AND MYERS, J. D. INTERNIST-I, an experimental computer-Based diagnostic consultant for general internal medicine. N. Engl. J. Med.307, 468 (1982). 9. MILLER, R. A., MASERIE, F. E., AND MYERS, J. D. Quick medical reference(QMR) for diagnostic assistance. M.D. Computing. 5, 34 (1986). 10. NGUYEN, L. T. "Transferability of Medical Knowledge-Base: A Case Study Between INTERNIST-l AND HELP." Dissertation, The University of Utah, 1986. 11 . WARNER, H. R., OLMSTEAD, C.M., AND RUTIlERFORD, B. D. HELP­a program for medical decision-making. Comput. Biomed. Res. 5, 65 (1972). 12. PRYOR, T. A., GARDNER, R. E., CLAYTON. P. D., AND WARNER, H. R. The HELP system. J. Medical Systems 7, 87 (1983). 13. "PAL: PTXT APPLICATION LANGUAGE An Application Language for the HELP System" (Reference Manual) LDS Hospital. 14. "HELP/HCOM User's Reference Manual version 1.0." The MEDLAB Company, 1983. 15. WAITE, W. M., AND GOOS. G. "Compiler Construction." Springer­Verlag, New York, 1984. 16. AHO, A. V., SETHI, R., AND ULLMAN, J. D. "Compilers-Principles, Techniques, and· Tools." Addison-Wesley, Reading, Mass., 1986. 17. BARRETT, W. A., BATES, R. M., GUSTAFSON, D. A., AND COUCH, J. D. "Compiler Construction: Theory and Practice." Science Re­search Associates, Chicago, 1986. 18. GILBER, P. "Software Design and Development." Science Research Associates, Chicago, 1983. 19. WIENER, R., AND SINCOVEC, R. "Software Engineering with Modular-2 and ADA." John Wiley & Sons, New York, 1984. 68 20. DEMILLO, R. A., McCRACKEN, W. M., MARTIN R. J., AND PASSAFIUME, J. F. "Software Testing and Evaluation." The Benjamin/Cummings Pub. Co., Menlo Park, Calif., 1987. 21. McDONALD, C., BLEVINS, L., GLAZENER, T., HAAS, J., LEMMON. L., AND MEEKS-JOHNSON, J. Data base management, feedback control and the Regenstrief medical record. "Proceedings, the Sixth Annual Symposium on Computer Applications in Medical Care, Washington DC," pp. 52-60, 1982. 22. BARNETT, G. 0., ZEILSTORFF, R. D., PIGGINS, J., McLATCHEY, J., MORGAN, M. M., BARRETT, S. M., SHUSMAN, D., BROWN, K., WEIDMAN-DAHL, F., AND McDONNELL, G. COSTAR: A com­prehensive medical information system for ambulatory care. "Proceedings, the Sixth Annual Symposium on Computer Applications in Medical Care, Washington DC," pp. 8-18, 1982. 23. ADAMS, J. B. Three surveillance and query languages for care. M.D. Computing 3, 11 (1986).