Application binary interface compatability through a customizable language

ZL is a C++-compatible language in which high-level constructs, such as classes, are defined using macros over a C-like core language. This approach is similar in spirit to Scheme and makes many parts of the language easily customizable. For example, since the class construct can be defined using macros, a programmer can have complete control over the memory layout of objects. Using this capability, a programmer can mitigate certain problems in software evolution such as fragile ABIs (Application Binary Interfaces) due to software changes and incompatible ABIs due to compiler changes. ZL's parser and macro expander is similar to that of Scheme. Unlike Scheme, however, ZL must deal with C's richer syntax. Specifically, support for context;-sensitive parsing and multiple syntactic categories (expressions, statements, types, etc.) leads to novel strategies for parsing and macro expansion. In this dissertation we describe ZL's approach to parsing and macros. We demonstrate how to use ZL to avoid problems with ABI instability through techniques such as fixing the size of class instances and controlling the layout of virtual method dispatch tables. We also demonstrate how to avoid problems with ABI incompatibility by implementing another compiler's ABI. Future work includes a more complete implementation of C++ and elevating the approach so that it is driven by a declarative ABI specification language.

APPLICATION BINARY INTERFACE COMPATIBILITY THROUGH A CUSTOMIZABLE LANGUAGE by Kevin Jay Atkinson A dissertation submitted to the faculty of The University of Utah in partial fulfillment of the requirements for the degree of Doctor of Philosophy in Computer Science School of Computing The University of Utah December 2011 Copyright c Kevin Jay Atkinson 2011 All Rights Reserved Th e Un i v e r s i t y o f Ut a h Gr a d u a t e S c h o o l STATEMENT OF DISSERTATION APPROVAL The dissertation of has been approved by the following supervisory committee members: , Chair Date Approved , Member Date Approved , Member Date Approved , Member Date Approved , Member Date Approved and by , Chair of the Department of and by Charles A. Wight, Dean of The Graduate School. Kevin Jay Atkinson Matthew Flatt 11/3/2011 Gary Lindstrom 11/17/2011 Eric Eide 11/3/2011 Robert Kessler 11/3/2011 Olin Shivers 11/29/2011 Al Davis School of Computing ABSTRACT ZL is a C++-compatible language in which high-level constructs, such as classes, are defined using macros over a C-like core language. This approach is similar in spirit to Scheme and makes many parts of the language easily customizable. For example, since the class construct can be defined using macros, a programmer can have complete control over the memory layout of objects. Using this capability, a programmer can mitigate certain problems in software evolution such as fragile ABIs (Application Binary Interfaces) due to software changes and incompatible ABIs due to compiler changes. ZL's parser and macro expander is similar to that of Scheme. Unlike Scheme, however, ZL must deal with C's richer syntax. Specifically, support for context-sensitive parsing and multiple syntactic categories (expressions, statements, types, etc.) leads to novel strategies for parsing and macro expansion. In this dissertation we describe ZL's approach to parsing and macros. We demonstrate how to use ZL to avoid problems with ABI instability through techniques such as fixing the size of class instances and controlling the layout of virtual method dispatch tables. We also demonstrate how to avoid problems with ABI incompatibility by implementing another compiler's ABI. Future work includes a more complete implementation of C++ and elevating the ap-proach so that it is driven by a declarative ABI specification language. CONTENTS ABSTRACT : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : iii LIST OF FIGURES : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : viii LIST OF TABLES : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : x ACKNOWLEDGEMENTS : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : xi CHAPTERS 1. INTRODUCTION: : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 1 1.1 Dissertation Statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 1.2 Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 1.3 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 2. PROBLEMS WITH THE C++ ABI : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 6 2.1 The C++ ABI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 2.2 The Problem of Fragile ABIs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 2.2.1 Solutions Within C++ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 2.2.2 Defining a Better ABI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 2.3 The Problem of Compiler Specific ABIs . . . . . . . . . . . . . . . . . . . . . . . . . . 11 3. SOLVING ABI PROBLEMS : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 12 3.1 Overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 3.1.1 User Roles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 3.2 Adding Private Data Members . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 3.2.1 Reserving Space Ahead of Time . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 3.2.2 Storing the Private Data in a Separate Object . . . . . . . . . . . . . . . . . . . 16 3.2.3 Avoiding Direct Allocation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 3.2.4 Why Not a Fixed Set of Language Extensions? . . . . . . . . . . . . . . . . . 18 3.3 Adding New Virtual Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 3.4 Reordering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 3.5 Removing Members . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 3.6 Migrating Method Upwards . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 3.7 Adding Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 3.8 Other Difficult Transformations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 3.9 A Better ABI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 3.10 Changing Compilers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 4. ZL OVERVIEW : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 25 4.1 ZL Primitives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 4.2 Macros . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 4.3 Parsing and Expanding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 4.4 Procedural Macros . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 4.5 The Class Macro . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 5. USING ZL TO MITIGATE ABI PROBLEMS : : : : : : : : : : : : : : : : : : : : : : 33 5.1 Adding Data Members without Changing Class Size . . . . . . . . . . . . . . . . . 33 5.1.1 Fixing the Size of a Class . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 5.1.2 Allowing Expansion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 5.1.3 Validation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 5.2 Fixing the Size of the Virtual Table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 5.3 A Better ABI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 5.4 Matching an Existing ABI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 5.5 Matching GCC's ABI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 5.6 Matching Another ABI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 5.7 Other ABI Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 6. THE CASE OF A SIMPLE SPELL CHECKER : : : : : : : : : : : : : : : : : : : : : 42 6.1 Simple Spell . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 6.2 The Spell Checker API . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 6.2.1 The Application API . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 6.2.2 The Extension API . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 6.3 A Simple Application and Binary Compatibility . . . . . . . . . . . . . . . . . . . . 47 6.4 Adding a Filter, Compiled with GCC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 6.4.1 The Bridge Class . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 6.4.2 Adding The Email Filter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 6.4.3 Automating the Creation of the Bridge Class . . . . . . . . . . . . . . . . . . . 50 6.5 Adding Support for a Personal Dictionary . . . . . . . . . . . . . . . . . . . . . . . . . 50 6.6 A Better ABI to Allow Future Enhancements . . . . . . . . . . . . . . . . . . . . . . . 54 6.7 A Simple Spell Checker, Version 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59 6.8 An Opportunity for an Even Better ABI . . . . . . . . . . . . . . . . . . . . . . . . . . . 60 6.9 Comparison to a Real Spell Checker: Aspell . . . . . . . . . . . . . . . . . . . . . . . 60 7. USING ZL : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 62 7.1 Classes and User Types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62 7.2 Pattern-Based Macros and Lexical Extensions . . . . . . . . . . . . . . . . . . . . . . 64 7.2.1 Extending the Parser . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65 7.2.2 The Parser . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66 7.2.3 Built-in Macros . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66 7.3 Macro API . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68 7.3.1 The Syntax Object . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68 7.3.2 The Syntax List . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70 7.3.3 Matching and Replacing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71 7.3.4 Match Patterns . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72 v 7.3.5 Creating Marks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73 7.3.6 Controlling Visibility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73 7.3.7 Fluid Binding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74 7.3.8 Partly Expanding Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75 7.3.9 Compile-Time Reflection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76 7.3.10 Misc API Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77 7.4 Procedural Macro Implementation and State Management . . . . . . . . . . . . . 78 7.4.1 The Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78 7.4.2 Macro Libraries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78 7.4.3 State Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80 7.4.4 Symbol Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80 7.5 ABI Related APIs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81 7.5.1 User Type and Module API . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81 7.5.2 User Type Builder . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82 7.5.3 The ABI Switch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83 7.5.4 Mangler API . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84 8. ZL IMPLEMENTATION DETAILS : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 87 8.1 Basic Expander and Hygiene System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87 8.1.1 The Idea . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87 8.1.2 An Illustrative Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87 8.1.3 Multiple Marks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90 8.1.4 Structure Fields . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91 8.1.5 Replacing Context . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92 8.1.6 Fluid Binding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92 8.2 The Reparser . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94 8.2.1 The Idea . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94 8.2.2 Additional Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96 8.2.3 Matching and Replacing with the raw_syntax Form . . . . . . . . . . . . . 97 8.3 Parser Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 8.3.1 Performance Improvements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98 9. IMPLEMENTATION STATUS AND PERFORMANCE : : : : : : : : : : : : : : : 100 9.1 C Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100 9.2 C++ Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101 9.3 Debugging Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101 10. RELATED WORK : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 102 10.1 Binary Compatibility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102 10.2 Scheme . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103 10.3 Other Macro Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103 10.4 Ziggurat . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105 10.5 Extensible Compilers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105 vi 11. DISCUSSION AND FUTURE WORK : : : : : : : : : : : : : : : : : : : : : : : : : : : : 107 11.1 Evaluation of ABI Problems Solved . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107 11.2 Error Messages and Debugging Support . . . . . . . . . . . . . . . . . . . . . . . . . . . 108 11.2.1 Handing of Code Needing the C Preprocessor . . . . . . . . . . . . . . . . . . 108 11.2.2 Source Level Debugging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109 11.2.3 Better Support for Macro Expanded Code . . . . . . . . . . . . . . . . . . . . . 110 11.3 C++ Template Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110 11.4 C++ Support in General . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111 11.5 Enhancements to ZL's Macro System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112 11.5.1 Always Reparsing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112 11.5.2 Matching Literals Hygienically . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113 11.5.3 Using Marks for Inner Namespaces . . . . . . . . . . . . . . . . . . . . . . . . . . 114 11.6 Support for an Extensible Parser . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114 11.7 Beyond ABI Compatibility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115 11.7.1 Type Safe and Extensible printf . . . . . . . . . . . . . . . . . . . . . . . . . . . 115 11.7.2 Variable Interpolation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116 11.7.3 Embedding SQL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116 11.8 Areas of Future Research . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117 11.9 Alternative Research Direction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117 12. CONCLUSION : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 119 APPENDIX: OVERHEAD OF THE PIMPL IDIOM : : : : : : : : : : : : : : : : : : : : : 120 REFERENCES : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 125 vii LIST OF FIGURES 4.1 How ZL compiles a simple program. The body of f is reparsed and expanded as it is being compiled. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 4.2 Procedural macro version of or macro from Section 4.2. . . . . . . . . . . . . . . . . 30 4.3 Basic macro API. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 5.1 Macro to fix the size of a class. All ... in this figure are literal. . . . . . . . . . . 35 6.1 The speller.hpp header file providing the core functionally of Simple Spell. 44 6.2 Other parts of the core simple spell API defined in other header files. . . . . . . 44 6.3 Simple Spell document checker API. All parts of this API use the GCC ABI. 46 6.4 Simple Spell extension API. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 6.5 A bridge class to allow using filters compiled with GCC. . . . . . . . . . . . . . . . 48 6.6 Part of the mk_bridge macro. The real implementation is just under 55 lines of code. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 6.7 Extending the Speller class to include support for a personal dictionary. . . . 53 6.8 The Speller class using the pimpl idiom. . . . . . . . . . . . . . . . . . . . . . . . . . . 55 6.9 Improved Session class to support future enhancements without breaking binary compatibility. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56 6.10 Improved SessionWFilters class. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58 6.11 The Filter class using an enhanced ABI. . . . . . . . . . . . . . . . . . . . . . . . . . . 58 7.1 Macro that iterates over an STL-like container. . . . . . . . . . . . . . . . . . . . . . . . 65 7.2 Simplified PEG grammar. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67 7.3 Version of foreach that returns a helpful error message if the container does not contain the begin or end methods. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68 7.4 Syntax object API. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69 7.5 Syntax list API. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71 7.6 Match and replace API. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71 7.7 Mark API. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73 7.8 Visability API. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74 7.9 Expander API. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76 7.10 Compile time reflection API. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76 7.11 Misc API functions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77 7.12 Symbol properties syntax and API. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80 7.13 User type and module API. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82 7.14 User type builder API. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82 7.15 Overview of the StringBuf class. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85 7.16 Overview of the symbol API . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86 8.1 Example code to illustrate how hygiene is maintained. . . . . . . . . . . . . . . . . . 88 8.2 Example code to show how hygiene is maintained when a macro expands to another macro. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91 A.1 Class used in test. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121 A.2 Same class (Figure A.1) but refactord to use the pimpl idiom. . . . . . . . . . . . . 121 A.3 Simplified version of code used to test the overhead of the pimpl idiom. . . . . 122 ix LIST OF TABLES 3.1 Changes that can affect the ABI. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 5.1 ZL's solution for changes that can affect the ABI. . . . . . . . . . . . . . . . . . . . . . 33 6.1 Approximate lines of code of the various versions of Simple Spell and Aspell. 61 8.1 Improvements in run time and memory usage due to parser optimizations. . . 98 8.2 Effects of individual optimizations in run time and memory usage. . . . . . . . . 98 A.1 Overhead on using the pimpl idiom. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123 ACKNOWLEDGEMENTS I would like to thank my co-advisers Matthew Flatt and Gary Lindstrom-who were also coauthors for works that are part of this dissertation-for their support and contribu-tions to this dissertation. I would also like to thank my other committee members, Eric Eide, Bob Kessler, and Olin Shivers for their support and feedback on this dissertation. In addition I would like to thank Ryan Culpepper, Carl Eastlund, and Jon Rafkind for feedback on works that are part of this dissertation. I also want to thank Jay Lepreau (though he passed away) and Eric Eide for their financial support through the Flux Research Group. This work is based on an earlier work [12]: "ABI Compatibility Through a Cus-tomizable Language", Proceedings of the Ninth International Conference on Generative Programming and Component Engineering (GPCE'10), Eindhoven, The Netherlands, Oct. 2010. c ACM, 2010, http://dx.doi.org/10.1145/1868294.1868316. Parts of this dissertation also appear in the work [11]: "Adapting Scheme-Like Macros to a C-Like Language", Workshop on Scheme and Functional Programming, Portland, Oregon, Oct. 2011. CHAPTER 1 INTRODUCTION There are two types of programming interfaces to a library: the Application Program-ming Interface (API) and the Application Binary Interface (ABI). The API defines the ways a programmer may request services from the library. Some of the constituents of an API in an object-oriented language are the names of classes, the methods they support, and the types of the arguments that methods take. What goes into the API is under the control of the library designer. An ABI is the object-code equivalent of an API. It is the low-level interface between the application and the library. A compiler implements a mapping from a library's API to its ABI. Some of the constituents of the mapping include calling conventions and class layout. Unlike the API, the programmer has little to no control of the ABI in most languages. When a library designer changes an API in a way that preserves backwards compat-ibility with previous releases, source code compatibility is maintained. That is, existing applications that use a library do not need to change at the source level. However, even if source code compatibility is preserved, binary compatibility need not be preserved; existing applications may need to be recompiled because the compiler typically does not guarantee ABI compatibility with API compatibility. In situations when a library is used by a small number of programs that can easily be recompiled, breaking binary compatibility between releases may be acceptable. However, if a large number of programs depend on the library, then recompiling is not an acceptable option as it can take anywhere from hours to days to recompile everything. In addition, in many situations the source code for applications using the library is not available, thus making upgrading impossible unless binary compatibility is preserved. Preserving binary compatibility for C++ programs is difficult because the typical C++ ABI is extremely fragile. Seemingly simple changes, such as adding methods, may break 2 binary compatibility. In fact, almost any change to a class declaration will likely break binary compatibility and require applications that use the library to be recompiled. In addition, the C++ ABI is not well defined as every compiler implements the C++ standard in a slightly different way. Libraries compiled with one compiler, such as Visual C++, generally will not be usable by applications compiled with a different compiler, such as GCC. Furthermore, the ABI may change between releases of the same compiler. Thus, upgrading to a newer compiler may also break binary compatibility. In contrast to C++, the C ABI is simple and well defined for a given architecture and operating system. Since the C ABI is far simpler than the C++ ABI, preserving binary compatibility is much easier. Furthermore, since the C ABI is well defined for a given architecture, compatibility between compilers is a nonissue. In fact, some C++ applications export only a C API for these very reasons. The C ABI is successful because of its simplicity and consistency. That simplicity, in turn, is based in part on the simplicity of the C language. As languages become more complicated, so do the number of choices to be made in an ABI. Thus, ABIs for complicated languages, such as C++, tend to vary among compilers and even among versions of a compiler. Standardizing on one C++ ABI would solve the incompatibility problem. Although some effort has been made in that area with Itanium C++ ABI [7], there are still several C++ ABIs in common use, most notably the GCC and Visual C++ ABIs. Even if all C++ compilers standardized on a single ABI, the problem of preserving binary compatibility between releases of a library would still be a major problem. This is because most C++ ABIs, including the Itanium C++ ABI, are optimized for performance, not for preserving binary compatibility. Previous designs for a less fragile ABI for C++ [48, 38] make significant sacrifices in performance. Thus, library designers must make a choice between breaking binary compatibility between releases or contorting their programs to preserve it by using a variety of programming idioms. We could try to add a few extensions to C++, such as a choice of different ABIs or support for common programming idioms, but a fixed number of extensions will never be enough as the problem of preserving binary compatibility is far too complex. A 3 nonextensible language cannot and should not support every possible rarely needed case. A more general and integrated approach is an extensible compiler. Traditional extensible compiler designs treat a compiler extension as an entity separate from the code to be compiled. On the other hand, a macro system acts as an extensible compiler and also allows the programmer to implement code and compiler extensions together, thus elevating compiler extensions to the level of a library. This, in turn, allows different ABI choices to be incorporated with different parts of an application. For example, one class can use an ABI optimized for performance while another uses an ABI aimed at preserving binary compatibility. A simple macro system, such as the C preprocessor, is not adequate for defining compiler extensions. Rather, the macro system must be an integral part of the language, and it must be able to do more than simply rearrange syntax. In addition to providing macro primitives, a language for giving the programmer control over an ABI must include a care-fully designed core that allows higher-level constructs, such as classes, to be implemented via macros. This capacity enables the programmer to redefine key aspects that affect ABI attributes, such as class layout. ZL, a C++ compatible systems programming language that is the subject of this dissertation, does exactly this. For relatively simple language extensions ZL supports pattern-based macros similar to Scheme's syntax-rules [51]. In addition, ZL supports parser extensions that change the tokenization (roughly) of the input source, so that macro uses need not have the restricted form that Scheme's macro system imposes. Even with such extensions, pattern-based macros are limited. Therefore, in the same way that Scheme provides procedural macros via syntax-case [24], ZL supports procedural macros. ZL's API for procedural macros includes support for reflective tasks such as getting the value of a macro parameter, determining whether a symbol is currently defined and getting basic properties about the symbol, and other necessary tasks to implement a class system. 1.1 Dissertation Statement Fragile and incompatible ABIs are a major problem in software maintenance and evo-lution that can be systematically dealt with and effectively managed using a macro-based 4 system that allows the programmer to control how an API maps to an ABI. 1.2 Approach This dissertation demonstrates the thesis in the context of C++, through the use of ZL. Class layout is a key aspect of the C++ ABI and is hence the focus of our research. However, we support other parts of the ABI as well. For example, we support name mangling, which is how local symbol names are translated in order to make them globally unique. Although, the ZL language gives the user complete control over many things that affect the ABI, since our implementation of ZL compiles to a C like language, it does not give the user control over everything such as exceptions and calling conventions. This is not a problem, however, since exception support is beyond the scope of our research and calling conventions are stable within a given architecture. In addition, C++ is a very complicated language and this research only addressed the features of C++ most relevant to the research question. In particular we did not address multiple-inheritance, exceptions, and templates, which pose unique challenges. 1.3 Contributions Our contributions in this dissertation are two-fold. The first is to demonstrate how ZL can be used to mitigate the problem of binary incompatibility through the use of macros. The second is to demonstrate the adaptation of Scheme-style, hygienic macros to C-style syntax. This dissertation outlines the problems of binary compatibility in C++ (in Chapter 2) and shows how a macro system can help (in Chapter 3). We then, after giving an overview of ZL (in Chapter 4), demonstrate how ZL can be used to mitigate the problem of binary incompatibility (in Chapters 5 and 6). For example, we show how to avoid breaking binary compatibility when adding new data members or methods to a class. We also match GCC's ABI to the point where a simple library can be compiled with ZL and then used with GCC and vice versa. In addition to ZL's native ABI and GCC's ABI, we implement several other specialized ABIs and show how classes with different ABIs can be used in the same program. 5 This dissertation also presents the details of ZL parser and macro expander (in Chapters 4, 7, and 8). Dealing with C's idiosyncratic syntax introduces complexities that are not solved by simply converting the original text into an S-expression intermediate format. Instead, parsing of raw text must be interleaved with the expansion process, and hygiene rules must be adapted carefully to actions such as accessing structure members. CHAPTER 2 PROBLEMS WITH THE C++ ABI This chapter outlines what goes into the C++ ABI, why it is so fragile, and the problems both the fragility and being compiler specific cause. 2.1 The C++ ABI There are many components to the C++ ABI. Of them, the components of most interest to this dissertation are: Data Layout. An ABI specifies how data are laid out in a memory region representing an instance (struct) of a class. These data include the data members of the class but it may also include other auxiliary information needed by the compiler such as a pointer to the virtual table (vtable). If a struct only contains data members and nonvirtual functions, and does not inherit from any other classes, it is generally considered a POD (plain old data) datatype. The layout of POD objects is the same as it would be in the C API. If the structure or class is not a POD data type than the layout is essentially left undefined by the C++ standard. However, in general a pointer to the virtual table is included first, then the data-members of any nonvirtual base classes, or a pointer to the class in the case of a virtual base class, then finally the data members of the current class. Virtual Table Layout. A virtual table (vtable) is the table that is used to dispatch virtual functions and contain run-time type information (RTTI), among other things. A vtable is not part of a C++ standard but it is included in nearly every C++ ABI. The virtual table is generally a static object that is included in the object file and then copied into memory at load time. A virtual table generally includes the following items: 7 - The typeinfo pointer for RTTI - The displacement to the top of the object from the location within the outer object. - Virtual function pointers, which are used for virtual function dispatch - Copies of the virtual tables for any nonvirtual base classes - Pointers to virtual tables for any virtual base classes Construction and Destruction. An ABI defines how objects are created and de-stroyed. In C++ this is done via special member functions known as constructors and destructors. These functions are created automatically by the compiler, but the user can control part of the contents. What is involved in object construction is part of the ABI specification. An ABI, such as GCC's [7], may even emit multiple versions of the same constructor or destructor. New and Delete. An ABI defines how new, delete, and delete [] are imple-mented. Often these operators call ABI specific functions rather than just calling C's malloc and free. Name Mangling. An ABI specifies how a function's local symbol names are mangled in order to make them globally unique. Some of the things that go into the mangled name include 1) the local name of the symbol, 2) the types of the parameters for functions, 3) the class name for member function, and 4) the namespace the symbol is in. Other important components include: RTTI. In addition to information to implement inheritance, an ABI also contains some run-time type information. In C++ the RTTI has three purposes: 1) to support the typeid operation, 2) to match an exception handler with a thrown object, and 3) to implement the dynamic_cast operator [7, §2.9]. Calling Conventions. Calling conventions for nonmember functions are the same as they are in the C ABI. Calling conventions for nonvirtual member functions 8 are generally the same as for nonmember functions with the first parameter being the this pointer. However, this is not always the case. For example, Microsoft Visual C++ passes the this pointer in a register, rather than passing it is as the first parameter. The calling convention for virtual functions involves a lookup in the vtable and thus varies from one ABI to another. Exception Handling. An ABI also specifies how exceptions are handled. There are many different possible ways to implement exceptions. Layout of The Object File. The layout of the object file is generally not part of the C++ ABI specification. It is generally left to other standards such as ELF. Linkage. Symbol lookup in C++ is generally delegated to the standard linker for the platform. However, unlike with C, many objects in C++ are not clearly part of any single object file. Examples include: - Out-of-line Functions - Static Data - Virtual Tables - Typeinfo - Constructors and Destructor - Instantiated Templates These symbols can thus appear in multiple object files. As a result, the C++ compiler needs a way to inform the linker of these special symbols and the linker needs a policy to handle them. Templates. Templates are a large part of the C++ language but a small part of the ABI. As far as the C++ ABI is concerned templates are ordinary objects except for the fact that 1) the symbol names need to be mangled such that they include the template parameters, and 2) the same instantiation can appear in multiple object files and should be combined to save space when linking them together. Keeping track of the size of dynamically allocated arrays. 9 Pointer to member functions. For nonvirtual functions this is generally a function pointer. For virtual functions it is the pointer into the virtual table. 2.2 The Problem of Fragile ABIs The C++ ABI is extremely fragile as seemingly simple changes, such as adding data members or virtual methods to a class, may break binary compatibility. Adding data members breaks binary compatibility because it changes the size of the class, which is used at compile time when allocating objects on a stack or inlining one object in another. Similarly, adding virtual methods changes the size of the vtable, and thus, with most ABIs, changes the offsets of all the methods' function pointers for any subclasses. In fact, most changes to a class will break binary compatibility since they change the object's (or vtable's) layout in one way or another. Due to the extremely fragility of the C++ ABI programmers go to great lengths to avoid breaking the ABI. For example many large software engineering projects have guidelines to that deal with this issue. Examples include KDE [27], BE [49], and Windows. In fact the problem of fragile ABIs was a key consideration when developing the Java ABI as The Java Language Specification [39] has an entire chapter devoted to the issue of Binary Compatibility. The importance of binary compatibility was also recognized in the paper by Forman, et al., which they summarize as "Only application alteration necessitates recompilation" on page 430 [36] . According to Yu, et al. this paper was a precursor to to Java's concept of binary compatibility [58]. 2.2.1 SolutionsWithin C++ For compiled languages like C++ there is no comprehensive solution to this problem; consequently, developers employ a large number of techniques to get around it. One solution is to only export abstract base classes (ABCs), or interfaces, which will never change. When it is necessary to add new methods a new ABC is created. This technique is often accomplished by including version numbers in the name of the ABC. A very similar technique is used by the Microsoft Component Object Model [57]. This however, requires full encapsulation, i.e., no direct access to data members. 10 Another solution is to create a C API on top of the C++ one for the sole purpose of more easily maintaining binary compatibility. C ABIs are inherently less fragile than C++ since they are simpler. For example, we did this with the Aspell project [1]. Yet another solution is to be aware of what exactly will break an ABI and employ techniques to avoid doing so, many of which require planning ahead. For example, a dummy variable can reserve space ahead of time to avoid changing the size of an object when adding new data members. However, these techniques, which will be explored in detail in Chapter 3, can often lead to less maintainable code. 2.2.2 Defining a Better ABI The main reason ABIs are so fragile for compiled languages such as C++ is that offsets and sizes are fixed at compile time. If this information were resolved at load-time then the issue of fragile ABIs would be greatly reduced. Java does just that, by making nearly all references in the compiled Java bytecode symbolic. That is, not only are functions symbolic as they are in C, but so are calls to virtual methods; even data member lookups are symbolic. All of these symbols are resolved when the class is loaded. The Java Language Standard [39] is very careful to define the ABI in such a way that breaking binary compatibility will almost certainly mean breaking source code compatibility also. Resolving any sort of detail that will affect ABI compatibility at load time is possible in Java since Java uses a completely different notion of compilation. In particular Java is compiled to byte code, not object code. This allows more flexibility in the type of information that can be resolved a load time. Even if it is not practical to resolve everything that can possibly affect ABI compatibly at run time, the ABI can still be defined in such as way to make it significantly less fragile. Such an approach is done by Goldstein and Sloan [38]. They define a special ABI known as the Object Binary Interface which will only be used on request. The ABI they define allows for evolutionary steps such as adding new public and protected methods, and adding or removing private data members. However, it does not allow for changing the order or type of public data members. Thus it greatly reduces the problem of a fragile ABI but does not eliminate it. Also, their ABI is not without cost when compared to the more 11 traditional C++ ABI. Thus, it is likely to affect performance, especially since all inheritance is implemented in a manner similar to how virtual inheritance is implemented in traditional C++ ABIs. 2.3 The Problem of Compiler Specific ABIs Due to the complexity of the C++ ABI, the implementation is compiler specific. Hence, changing compilers can also break binary compatibility. Thus, when using C++ libraries, not only is the specific version of the library important, but so is the compiler used to compile it. The fact that a C++ library is tied to a particular ABI implementation is a particular problem in Windows when a large amount of code written is using Microsoft's VC++. Because so much code is written using VC++ in Windows many other Windows compilers often conform to at least part of this ABI, making it less of an issue. But it is still an issue since this ABI is not universally used by all C++ compilers. For example GCC uses a different ABI. Thus it is impossible to use GCC when developing Windows code that uses Windows C++ libraries. We ran into this problem a while ago, when we wanted to write some filters for AviSynth, a program for scriptable video processing. The source code for AviSynth is freely available but it will only compile on VC++. We wanted to write the filters using GCC. So in order to do this we had to write a special filter whose sole purpose was to bridge the gap by using a more stable C (as opposed to C++) interface. We then used this filter to write filters that could be compiled using GCC. These filters were written in C++. Thus we had to write a special "C" interface in order to interface with "C++" code, which seems silly, but was necessary since VC++ and GCC C++ ABIs are incompatible. There is no real solution to the problem of incompatible ABIs. The general solution is to simply avoid the issue by just using a compiler that is compatible with the C++ ABI deployed. When this is not an option, then the only other solution is to write a C API as was done for AviSynth. CHAPTER 3 SOLVING ABI PROBLEMS Table 3.1 lists changes that can break binary compatibility without affecting source code compatibility. Except for the last item, this list is from the paper by Forman, et al. [36]. This chapter discusses each of the problems, the solutions used in practice, and how a macro system can improve on them. 3.1 Overview For each ABI compatibility problem there are often several different solutions, and which one to choose depends on the situation. If there were one really good solution to the particular problem then it could easily be added as an extension to the language. In fact, since C++ is a fairly mature language, there is a good chance that the solution already Table 3.1. Changes that can affect the ABI. Solutions to many of these problems can be supported to some extent within the constraints of the existing C++ ABI, but for some the only solution is a better ABI. Change Can Support Section add instance variable Yes 3.2 add new method Yes 3.3 reorder methods Yes 3.4 reorder instance variables Yes 3.4 remove private method Yes 3.5 remove private instance variable Yes 3.5 migrate method upward in class hierarchy Yes 3.6 add parameters Yes 3.7 insert new class in class hierarchy New ABI 3.8 migrate parent downward in class hierarchy New ABI 3.8 change compilers - 3.10 13 would have been added. However, since there are many solutions, with none of them being clearly better than the other, language designers would have to add them all in order to deal with the problem. Adding all solutions is not an attractive option, as it would severely, possibly unnecessarily, bloat the language. Furthermore, there may be additional creative solutions, specific to a particular problem that language designers cannot possibly think of. Consequently it is essential to give the programmer as much control as practical for implementing the best solution for a given situation. Giving programmers control is a task ideally suited to a macro system where the implementation of the classes and other key parts of the ABI are under the programmer's control. 3.1.1 User Roles A good macro system can benefit all users, but not everyone needs to know the full details of how macros work. There are three primary classes of users: 1) End Users or Library Consumers, who just use the library, but can benefit from increased binary com-patibility; 2) Library Implementers, who can use the macro libraries to provide increased binary comparability, but do not need to know the details of the macro libraries themselves; and 3) Tool Implementers, who provide the macro libraries for the library implementers. With traditional compiler designs, tool implementers are in relatively short supply, and they face a daunting task on two fronts: they must modify the compiler, and they must convince users of the library to use the modified compiler. Our approach to improving ABI compatibility is to simplify the tool implementer's job, so that library implementers will have better tools and end users will have more compatible libraries. Specifically, with a macro-extensible compiler that can express ABI details through the macro layer, tool implementers gain a simpler framework for implementing more interoperable designs, and they get a more composable framework so that multiple tools can be combined. In this way, a tool becomes more like a library. Indeed, just as library consumers can become library implementers when they want to generalize their application code so that others can use it, library implementers can become tool implementers when they need to do something unusual for which a macro library does not yet exist. The key benefit of a macro system in this case is that it allows a library implementer to easily become a tool implementer. 14 3.2 Adding Private Data Members A C++ ABI can change by adding new private data members since that changes the size of the object. For example, changing class X { private: int old_var; public: ... }; to class X { private: int old_var; int new_var; public: ... }; breaks binary compatibility, because the size of X changes from the size of one integer to the size of two integers. This change is a problem when the size of the object is needed at compile time, such as when the object is allocated directly on the stack, embedded inside another object, or even allocated with new. 3.2.1 Reserving Space Ahead of Time One solution in C++ is to use a dummy variable to reserve space ahead of time. For example, changing X to: class X { int old_var; unsigned long _reserved[3]; }; reserves enough space for three additional variables. Then to add a new variables simply decrease the size of _reserved: class X { int old_var; int new_var; unsigned long _reserved[2]; }; 15 This approach works but requires a bit of planning ahead. It also depends on knowing the size of the members, which varies amoung architectures. The above example relies on the fact that long is the same size as an int, which is not always the case. For example, on 64-bit processors, an int is 4 bytes while a long is 8. However, long is used, as opposed to an int, since on most architectures a long is the same size as a pointer. Things get interesting when we run out of space. To deal with this situation, the last reserved slot is used as a pointer. For example: class X { private: int old_var; unsigned long _reserved[3]; public: ... }; becomes class X { private: int old_var; int new_var_1; int new_var_2; class D {int new_var_3; int new_var_4;}; D * d; public: X(); X(const X & x); X & operator= (const X & x); ~X(); ... }; X::X() {d = new D();} X::X(const X & x) {...; d = new D(*x.d);} X & X::operator= (const X & x) {...} X::~X() {delete d;} But accessing the data members is now cumbersome, since we must always use d->new_var_3 instead of just new_var_3. There is also a slight performance hit due to the extra layer of indirection. 16 With a macro system this solution can easily be automated by writing a macro to do the same thing. For example we could write a macro to recognize a fix_size flag to a class as so: class X : fix_size(sizeof(long)*4) { private: int old_var; public: ... }; Then, no end-user visible tricks are needed to add a new member as the size of the object will not change. Thus binary compatibility is maintained. Furthermore, since macros handle the low-level details and not the programmer, this solution will be portable across different architectures; the user does not have to know the exact size the types involved. If a programmer tries to use more space than is preallocated, a compile-time error will be emitted. If the programmer wishes to allow additional private data members an additional flag can be specified: class X : fix_size(sizeof(long)*4), allow_expansion { private: int old_var; int new_var_1; int new_var_2; int new_var_3; int new_var_4; public: ... }; and the macro responsible for implementing this feature will allocate additional space if necessary. In this case the implementation will look a lot like the C++ example just given. 3.2.2 Storing the Private Data in a Separate Object Another solution is simply to keep all the private data members in a separate object, for example: 17 class X { private: class D { int var_1; int var_2; ... }; Data * d; public: X() {d = new D();} X(const X & x) {d = new D(*x.d);} X & operator= (const X & x) {...} ~X() {delete d;} }; This solution is simpler than reserving space ahead of time since it does not require foresight in how large the object may be now, and in the future, and it also does not depend on the size of the types. This solution, known as the pimpl idiom, is in fact a very common solution used in practice. The only downside is that there is additional overhead involved in the creating, copying, and deleting of X, and that all accesses to private data must be done through. Based on our own tests the overall slowdown from using this idiom is anywhere between a factor of 1.0 and 1.8. In practice the slowdown is likely to be closer to 1.0 than 1.8. Appendix A gives more details on the tests performed. This solution can easily be implemented via macros using a similar syntax as before: class X : fix_size, allow_expansion { int var_1; int var_2; } Since no size is given to fix_size, it will be assumed that only enough space should be allocated to maintain a pointer to an additional object which will store all the private data members. 3.2.3 Avoiding Direct Allocation As previously described, the problem with changing the size of the object is that the information is needed if the object is directly allocated. This problem can be avoided 18 by disallowing the object to be directly allocated using the standard C++ trick of making constructors, assignment, and destructors private, and instead provide methods to create, copy, and destroy the function: class X { private: X(); X(const X &); void operator=(const X &); ~X(); public: X * clone(); static X * allocate(); static X * destroy(); }; This strategy means that the object cannot be directly allocated on the stack or embedded in other objects. However, it also means that the object cannot be allocated using C++ builtin new and delete, since new and delete cannot be overloaded on a per-class bases. If new and delete are implemented via macros, then they can easily be modified to use a different approach for a particular object. 3.2.4 Why Not a Fixed Set of Language Extensions? Reserving space ahead of time is a good solution when performance really matters. However since it requires planning ahead and depends on knowing the exact size of types it is not a very attractive option. Storing all the private data in a separate object is easier to implement, but it does have a small performance overhead which some may find unacceptable. Finally, preventing direct allocation is an undesirable alternative to users of the library. Since none of these solutions is perfect, none of them are good candidates for language extensions. However, in a system where higher-level objects are implemented using macros the programmer is free to extend these macros to support whichever solution is best suited to the problem. 19 3.3 Adding New Virtual Methods A C++ ABI can also change by adding new virtual methods since that changes the size of the vtable. This is a problem because the vtable is often included in the object file instead of being created dynamically at load time. The C++ solution is similar to the one for adding new data members, except that space is reserved by using dummy methods. For example, class X { public: virtual old_method(); private: virtual void _dummy_1(); virtual void _dummy_2(); virtual void _dummy_3(); }; reserves enough space for three new methods. Then to add a new method simply replace one of them with the real method: class X { public: virtual old_method(); virtual new_method(); private: virtual void _dummy_2(); virtual void _dummy_3(); }; In a system where macros will be used to implement inheritance, these macros can easily be expanded to reserve space ahead of time for additional methods by recognizing syntax similar to: class X : vtable_slots(4) { ... } which will create a virtual table with four slots in it. However, unlike the case of adding private data members, running out of slots is a serious problem as there is no way to simply add a pointer to another object to add more 20 virtual functions, as we could before. There are still ways to add functionality to the class; however, it can not be through adding more virtual methods. Thus, the idea of fixing the size of the vtable is not an attractive solution since it requires the programmer of the library to have foresight into how many virtual functions they will ever need for this object. Therefore, support for this strategy is not something that is likely to get added to as a language extension. But as before, in some situations, it may still be a viable option. 3.4 Reordering Another way to break ABI compatibility in C++ is to reorder the methods or the instance variables since that will change the offsets in the vtable or the object's instance, respectively. For example changing: class X { public: virtual a(); virtual b(); }; to class X { public: virtual b(); virtual a(); }; will change the offset of a and b. There is no real solution to this in C++ other than to just be aware of this fact and not do it. However there are several ways this problem can be solved by modifying how inheritance is implemented: 1. One idea is to use something like: class X : freeze_virtual_table { ... } 21 which will in a separate interface file store the offset of each virtual function. Once a function is defined its offset will never change. 2. Another idea is to put the virtual methods into groups, something like: class X { public (group 1): virtual a(); virtual b(); public (group 2): virtual c(); virtual d(); }; then sort each of the groups alphabetically. When new methods are added put them into another group. 3. Finally, a Java-like approach can be used where the offsets are determined when the class is first used. However, this will involve a completely different ABI from the one generally used in C++. The problem of reordering instance variables can be solved using similar techniques, except that changes will affect class instances and not the vtable. None of these solutions are particularly attractive; thus, any one of them is unlikely to be implemented as an extension to C++. However, just because no single solution is attractive does not mean that they are not useful. With a macro system in which classes are implemented via macros, the programmer is free to chose which solution, if any, is best for the particular situation. 3.5 Removing Members Removing methods and instance variables breaks ABI compatibility in C++ since it changes the order the vtable or the object's instance, respectfully. The only way to solve these problems in C++ is to avoid removing the method or instance variables by instead replacing it with a dummy member. This way the layout is preserved. The unused slot can then later be replaced with a new member in order to save, or it can simply be left unused. 22 A macro system can help automate this method by freezing the layout in a very similar fashion as discussed in Section 3.4. When this is done the macro will automatically insert a dummy member in place of the removed member. Later on the unused slot can be replaced with a new member. 3.6 Migrating Method Upwards Migrating a method upwards in the class hierarchy breaks binary compatibility in C++. Solving this problem in C++ is not really possible, but a macro system can help as long as space is reserved ahead of time so that it is is possible to add new virtual methods to the class (as discussed in section 3.3) and single inheritance is used. The idea is as follows: instead of actually removing the method from the old class, the vtable for the old class is adjusted to simply point to the method in the new class. This technique will not work when multiple inheritance is used, since the pointer to the class instance may need to be adjusted; in that case, a proxy method can be created that will just call the new method. 3.7 Adding Parameters Adding parameters to a function in C++ will break binary compatibility because it changes the name of the symbol used to represent the function. The name changes because, to support overloading, the types of the parameters are encoded as part of the name. However, since C++ allows for overloading it is possible to define a new function with the added parameter. The old function can then call the new one. Macros can automate this technique. 3.8 Other Difficult Transformations Unfortunately, some program transformations are difficult if not impossible to support within the constraints of an existing ABI. Such transformations include inserting a new class and migrating a parent downwards in the class hierarchy. The transformations are difficult because the parent class is directly embedded in the child class, for both instances of the classes and the vtable. The only real way to support these transformations is to define a better API. 23 3.9 A Better ABI All of the solutions in the previous sections work within an existing ABI; thus none of the solutions were ideal. However, the problems can all be solved by defining a new ABI that takes these transformations into account. The new ABI can support transformations that are difficult if not impossible to solve within the existing C++ ABI such as those mentioned in section 3.8. With a system where a large part of what affects the ABI is written using macros, it is possible to write a new ABI from scratch in order to minimize the issue of ABI fragility, for example implementing something similar to the Object Binary Interface [38], or maybe even implementing something close to what Java does. However, using a new ABI is not always an option. For example, if a library developer wants code to be usable by existing C++ applications, the developer must use the existing C++ ABI. Thus it is necessary to give programmers the tools to work within an existing ABI when necessary, but also give programmers the option of creating a new ABI when appropriate. 3.10 Changing Compilers Finally, changing compilers also breaks binary compatibility, since ABIs differ between compilers and sometimes between different versions of the same compiler. Thus, when using C++ libraries, not only is the specific version of the library important, but so is the compiler used to compile it. Unfortunately, there is no good solution to this problem in C++, other than always using a compatible compiler when compiling the library. The only way to support a different, incompatible, compiler is to avoid directly using the C++ ABI altogether. A typical work-around is to create a C API on top of the internal C++ API, and then only export the C API. This technique effectively defines a program-specific ABI that the library developer has complete control over. It may seem silly for a C++ program to have to use a C API to use another C++ library, but currently there is no other way around the problem. However, if classes are implemented via macros, then the programmer has control of how classes are implemented, and thus has control over which ABI is used. In fact, a programmer can use classes with different ABIs within the same program. For example, 24 the ABI used can be specified as part of the class declaration. For using existing code, the ABI can be specified on a per header-file basis. Via the right macro hooks, additional parts of the the ABI, such as name mangling, can be brought under control of the programmer. CHAPTER 4 ZL OVERVIEW ZL is a C++-compatible language that solves ABI compatibility problems by giving the programmer as much control as possible. ZL provides a C-like core and enough of C++ to let the type-checker and compiler do its job without committing to key parts of the ABI such as class layout. The rest is defined using a sophisticated macro system. The ZL library provides a default implementation of language constructs such as classes. The implementation can be overridden or extended by defining new macros in a source file or by importing a macro library. Macros, including those that define the behavior of a language construct, are scoped and can be shadowed. This means it is possible to use two different class ABIs by loading one class library and defining some classes, then loading another library and defining some more classes. A more convenient solution is to add some syntax for selecting the ABI for a class, which ZL also supports. This chapter gives an overview of ZL. Chapters 7 and 8 will give a more detailed description of ZL and how it is implemented. 4.1 ZL Primitives Most of the class implementation in ZL is left to macros, but since classes are an integral part of the C++ type system, ZL still needs to have some notion of what a class is. User types are ZL's minimal notion of classes. A user type has two parts: a type, generally a struct, to hold the data for the class instance, and a collection of symbols for manipulating the data. The collection of symbols is a module. For example, module M { int x; int foo(); } 26 defines a module with two symbols. Module symbols are used by either importing them into the current namespace, or by using the special syntax M::x, which accesses the x variable in the above module. A user type is created by using the user_type primitive, which serves as the mod-ule associated with the user type. A type for the instance data is specified using associate_type. As an example, the class1 class C { int i; int f(int j) {return i + j;} }; roughly expands to: user_type C { struct Data {int i;}; associate_type struct Data; macro i (:this ths = this) {...} macro f(j, :this ths = this) {f‘internal(this, j);} int f‘internal(...) {...} } which creates a user type C to represent a class C; the structural type Data is used for the underlying storage. The macro i implements the i field, while the f macro implements the f method by calling the function f‘internal with ths as the first parameter. The next section will explain the syntax of the macros and Section 7.1 will give the full expansion of class C and a more complete picture of how classes are implemented. ZL also supports syntax for creating macros, of which there are two kinds: pattern-based macros that simply rearrange syntax, and procedural macros that are functions that perform more complex manipulation of syntax or take action based on the input, as is necessary to implement classes. 4.2 Macros The simplest form of a macro is a pattern-based macro, which is simply a transforma-tion of one piece of syntax to another. For example, consider an or macro that behaves 1 For simplicity, we leave off access control declarations and assume all members are public in this dissertation when the distinction is unimportant. 27 like C's || operator, but instead of returning true or false, returns the first nonzero value. Thus, or(0.0, 6.8) returns 6.8. To define it, one uses ZL's macro form, which declares a pattern-based macro: macro or(x, y) { ({typeof(x) t = x; t ? t : y;}); } In ZL, as in GCC, the ({...}) is a statement expression whose value is the result of the last expression, and typeof(x) gets the type of a variable. Like Scheme macros [24], ZL macros are hygienic, which means that they respect lexical scope. For example, the t used in or(0.0, t) and the t introduced by the or macro remain separate, even though they have the same symbol name. The or macro above has two positional parameters. Macros can also have keyword parameters and default values. For example: macro sort(list, :compar = strcmp) {...} defines the macro sort, which takes the keyword argument compar, with a default value of strcmp. A call to sort will look something like sort(list, :compar = mycmp). 4.3 Parsing and Expanding The macros shown so far are pattern-based macros. Writing more sophisticated pro-cedural macros, such as those required to implement classes, requires some knowledge of parsing and macro expansion in ZL. This section gives the necessary background material, while the next section details how to write such macros. To deal with C's idiosyncratic syntax while also allowing the syntax to be extensible, ZL does not parse a program in a single pass. Instead, it uses an iterative-deepening approach to parsing. The program is first separated into a list of partly parsed declarations by a Packrat [34, 35] parser that effectively groups tokens at the level of declarations, statements, grouping curly braces, and parentheses. Each declaration is then parsed. As it is being parsed and macros are expanded, subparts, such as code between grouping characters, are further separated. ZL's iterative-deepening strategy is needed because ZL does not initially know how to parse any part of the syntax involved with a macro. When ZL encounters something that looks like a function call, such as f(x + 2, y), it does not know if it is a true function 28 call or a macro use. If it is a macro use, the arguments could be expressions, statements, or arbitrary syntax fragments, depending on the context in which they appear in the expansion. Similarly, ZL cannot directly parse the body of a macro declaration, as it does not know the context in which the macro will ultimately be used. More precisely, the ZL parsing process involves three intertwined phases. In the first phase raw text, such as (x+2), is parsed. Raw text is converted into an intermediate form known as a syntax object, which can still have raw-text components. (Throughout this paper we show syntax objects as S-expressions, such as ("()" "x+2").) In the second phase, the syntax object is expanded as necessary and transformed into other syntax objects by expanding macros until a fixed point is reached. In the third phase, the fully expanded syntax object is compiled into an AST. Figure 4.1 demonstrates ZL's parsing and expansion process. The top box contains a simple program as raw text, which is first parsed. The result is a syntax list (internally represented as a @) of stmt's where each stmt is essentially a list of tokens, as shown in the second box. Each statement is then expanded and compiled in turn, and is added to the top-level environment (which can be thought of as an AST node). The third box in the figure shows how this is done, which requires recursive parsing and expansion. The first stmt is compiled into the fun f, while the body of the function is left unparsed. Next, fun is compiled into an AST (shown as a rounded rectangle). During the compilation, the body is expanded. Since it is raw text, this process involves parsing it further, which results in a block. Parsing the block involves expanding and compiling the subparts. Eventually, all of the subparts are expanded and compiled, and the fully parsed AST is added to the top-level environment. This process is repeated for the function main, after which the program is fully compiled. 4.4 Procedural Macros Some macros must take action based on the input. One example is the built-in class macro. Another example is a macro that fixes the size of the class, since the amount of padding it needs to add depends on the numeric value of the size passed in. For these situations, ZL provides procedural macros, which are functions that transform syntax 29 inline int f() {int x = 10; return x;} int main() {return f();} #PARSE# (@ (stmt inline int f ("()" "") ("{}" "int x = 10; return x;") (stmt int main ("()" "") ("{}" "return f();"))) #EXPAND & COMPILE# TOP-LEVEL ENVIRONMENT (stmt inline int f ...) #EXPAND# (fun f "()" int :inline ("{}" "int x = 10; return x;"}) #COMPILE# FUN inline true id f type int body ("{}" "int x = 10; return x;") #EXPAND & REPARSE# (block (stmt int x = 10) (return (exp x))) #COMPILE# BLOCK (stmt int x = 10)) #EXPAND# (var x (int) (exp 10)) #C OMPILE# VAR ... (return (exp x)) #...# (stmt int main ...) #...# Figure 4.1. How ZL compiles a simple program. The body of f is reparsed and expanded as it is being compiled. 30 objects. Figure 4.2 demonstrates the essential parts of any procedural macro. The macro is defined as a function that takes a syntax object and environment, and returns a transformed syntax object. Syntax is created using the syntax form. The match function is used to decompose the input while the replace function is used to rebuild the output. Finally, make_macro is used to create a macro from a function. More interesting macros use additional API functions to take action based on the input. Figure 4.3 defines the key parts of the macro API, which we describe in the rest of this section. Syntax is created using the syntax and raw_syntax forms. The different forms create different types of code fragments. In most cases, the syntax {...} form can be used, such as when a code fragment is part of the resulting expansion; the braces will not be in Syntax * or(Syntax * p, Environ *) { Match * m = match(NULL, syntax (_, x, y), p); return replace(syntax {({typeof(x) t = x; t ? t : y;});}, m, new_mark()); } make_macro or; Figure 4.2. Procedural macro version of or macro from Section 4.2. Types: UnmarkedSyntax, Syntax, Match, and Mark Syntax forms: new_mark() - returns Mark * syntax (...)|{...}|ID - returns UnmarkedSyntax * raw_syntax (...) - returns UnmarkedSyntax * make_macro ID [ID]; Callback functions: Match * match(Match * prev, UnmarkedSyntax * pattern, Syntax * with) Match * match_args(Match *, UnmarkedSyntax * pattern, Syntax * with) Syntax * match_var(Match *, UnmarkedSyntax * var); Syntax * replace(UnmarkedSyntax *, Match *, Mark *) size_t ct_value(Syntax *, Environ *) Figure 4.3. Basic macro API. 31 the resulting syntax. If an explicit list is needed, for example, when passed to match as in Figure 4.2, then the syntax (...) form should be used (in which the commas are part of the syntax used to create the list). Neither of these forms create syntax directly, however; for example, syntax {x + y;} is first parsed as ("{}" "x + y;") before eventually becoming (plus x y). When it is necessary to create syntax directly, the syntax ID form can be used for simple identifiers. For more complicated fragments the raw_syntax form can be used in which the syntax is given in S-expression form. The match function decomposes the input. It matches pattern variables (the second parameter) with the arguments of the macro (the third parameter). If it is successful, it prepends the results to prev (the first parameter) and returns the new list. If prev is NULL, then it is treated as an empty list. In the match pattern a _ can be used to mean "don't care." The match is done from the first part of the syntax object. That is, given (plus x y), the first match is plus. Since the first part is generally not relevant, ZL provides match_args, which is like match except that the first part is ignored. For example, match_args could have been used instead of match in Figure 4.2. The replace function is used to rebuild the output. It takes a syntax object (the first parameter, and generally created with syntax), replaces the pattern variables inside it with the values stored in the Match object (the second parameter), and returns a new Syntax object. The final argument to replace is the mark, which is used to implement hygiene. A mark captures the lexical context at the point where it is created. Syntax objects created with syntax do not have any lexical information associated with them, and are thus unmarked (represented with the type UnmarkedSyntax). It is therefore necessary for replace to attach lexical information to the syntax object by using the mark created with the new_mark primitive (the third parameter to replace). Match variables exist only inside the Match object. When it is necessary to access them directly, for example, to get a compile-time value, match_var can be used; it returns the variable as a Syntax object, or NULL if the match variable does not exist. If the compile-time value of a syntax object is needed, ct_value can be used, which will expand and parse the syntax object and return the value as an integer. 32 Once the function for a procedural macro is defined, it must be declared as a macro using make_macro. This section only gives a small part of the macro API. A more detailed description is given in Chapter 7. Some of the more important functions not shown here include functions for controlling the visibility of macros and partly expanding syntax. 4.5 The Class Macro We have now presented most of the necessary parts that make up the class macro. Sections 4.1 and 4.2 give a representation of the code generated, while Sections 4.3 and 4.4 give a representation of what is necessary to generate that code. The remaining details are given in Chapter 7, which includes more of ZL's macro API. The class macro also uses ZL's support for syntax macros, which work with arbitrary syntax, as opposed to function-call macros, which only work with syntax that takes the shape of a function call or identifier. The core class macro is currently around 900 lines of code. The implementation is highly reusable, because it is a class itself that is organized around methods that can be overridden to extend its functionality. The bootstrapping problem of writing methods to implement classes is solved by having a simpler, more compact class system just to implement the class macro. In addition to overriding individual methods, the class syntax object can be declared to expand to a completely different macro. The class macro is defined using the function parse_class, which can be called directly so that the new macro can reuse the original implementation. CHAPTER 5 USING ZL TO MITIGATE ABI PROBLEMS ZL can be used to mitigate key ABI problems discussed in Chapter 3. This chapter gives the details of how key techniques from that chapter are implemented in ZL (see Table 5.1 for an overview). The next chapter demonstrates how these techniques can be used to mitigate binary compatibility problems through the evolution of a simple spell checker. 5.1 Adding Data Members without Changing Class Size Adding data members to a class changes the size of the class, which breaks binary compatibility. To avoid this problem we must somehow fix the size of the class. Table 5.1. ZL's solution for changes that can affect the ABI. ZL can implement all of the techniques discussed in Chapter 3 (and shown in Table 3.1, page 12). However, only a key subset of the techniques discussed are currently implemented. An outline of how ZL can implement the other techniques is given in Section 5.7. Change Solution Implemented Section add instance variable Yes 5.1, 5.3 add new method Yes 5.2, 5.3 reorder methods - 5.7 reorder instance variables - 5.7 remove private method - 5.7 remove private instance variable - 5.7 migrate method upward in class hierarchy - 5.7 add parameters - 5.7 insert new class in class hierarchy - 5.7 migrate parent downward in class hierarchy - 5.7 change compilers Yes 5.5, 5.6 34 5.1.1 Fixing the Size of a Class As described in Section 3, one common technique to fix the size of the class is to add dummy data members as placeholders to allow for future expansion. Using the ZL macro system, it is possible to automate this solution, as shown in Figure 5.1. To support this extension the ZL grammar has been enhanced to support specifying the size. The syntax for the new class form is: class C : fix_size(20) {...}; which allows a macro to fix the size of the class C to 20 bytes. The macro in Figure 5.1 redefines the built in class macro. It works by parsing the class declaration and taking its size. If the size is smaller than the required size, an array of characters is added to the end of the class to make it the required size. The details are as follows. Lines 2-7 decompose the class syntax object to extract the relevant parts of the class declaration. A @ by itself in a pattern makes the parts afterward optional. The pattern form matches the subparts of a syntax object; the first part of the object (the {...} in this case) is a literal1 to match against, and the other parts of the object are pattern variables. A @ followed by an identifier matches any remaining parameters and stores them in a syntax list; thus, body contains a list of the declarations for the class. Finally, :(fix_size fix_size) matches an optional keyword argument; the first fix_size is the keyword to match, and the second fix_size is a pattern variable to hold the matched argument. If the class does not have a body (i.e., a forward declaration) or a declared fix_size, then the class is passed on to the original class macro in line 9. Line 11 compiles the fix_size syntax object to get an integer value. Lines 13-22 involve finding the original size of the class. Due to alignment issues the sizeof operator cannot be used, since a class such as "class D {int x; char c;}" has a packed size of 5 on most 32 bit architectures, but sizeof(D) will return 8. Thus, to get the packed size, a dummy member is added to the class. For example, the class D will become "class D {int x; char c; char dummy;}" and then the offset of the dummy 1 ZL matches literals symbolically (i.e., not based on lexical context). Matching sensitive to lexical context is future work. (See 11.5.2) 35 1 Syntax * parse_myclass(Syntax * p, Environ * env) { 2 Mark * mark = new_mark(); 3 Match * m = match_args 4 (0, raw_syntax(name @ (pattern ({...} @body)) 5 :(fix_size fix_size) @rest), p); 6 Syntax * body = match_var(m, syntax body); 7 Syntax * fix_size_s = match_var(m, syntax fix_size); 8 9 if (!body || !fix_size_s) return parse_class(p, env); 10 11 size_t fix_size = ct_value(fix_size_s, env); 12 13 m = match(m, syntax dummy_decl, 14 replace(syntax {char dummy;}, NULL, mark)); 15 Syntax * tmp_class = replace(raw_syntax 16 (class name ({...} @body dummy_decl) @rest), 17 m, mark); 18 Environ * lenv = temp_environ(env); 19 pre_parse(tmp_class, lenv); 20 size_t size = ct_value 21 (replace(syntax(offsetof(name, dummy)), m, mark), 22 lenv); 23 24 if (size == fix_size) 25 return replace(raw_syntax 26 (class name ({...} @body) @rest), 27 m, mark); 28 else if (size < fix_size) { 29 char buf[32]; 30 snprintf(buf, 32, "{char d[%u];}", fix_size - size); 31 m = match(m, syntax buf, 32 replace(string_to_syntax(buf), NULL, mark)); 33 return replace(raw_syntax 34 (class name ({...} @body buf) @rest), 35 m, mark); 36 } else 37 return error(p,"Size of class larger than fix_size"); 38 } 39 make_syntax_macro class parse_myclass; Figure 5.1. Macro to fix the size of a class. All ... in this figure are literal. 36 member with respect to the class D is taken. This new class is created in lines 13-17. Here, the @ before the identifier in the replacement template splices in the values of the syntax list. To take the offset of the dummy member of the temporary class, it is necessary to parse the class and get it into an environment. However, we do not want to affect the outside environment with the temporary class. Thus, a new temporary environment is created in line 18 using the temp_environ macro API function. Line 19 then parses the new class and adds it to the temporary environment. The pre_parse API function partly expands the passed-in syntax object and then parses just enough of the result to get basic information about symbols. With the temporary class now parsed, lines 20-22 get the size of the class using the offsetof primitive. Lines 24-37 then act based on the size of the class. If the size is the same as the desired size, there is nothing to do and the class is reconstructed without the fix_size property (lines 24-27). If the class size is smaller than the desired size, then the class is reconstructed with an array of characters at the end to get the desired size (lines 28-35). (The string_to_syntax API function simply converts a string to a syntax object.) Finally, an error is returned if the class size is larger than the desired size (lines 36-37). The last line declares the function parse_myclass as a syntax macro for the class syntax form. 5.1.2 Allowing Expansion The example in Figure 5.1 demonstrates one technique for preserving binary compat-ibility when adding new data members. However, this technique requires planning ahead and reserving enough space for all future extensions. If there is not enough space reserved but enough space for a pointer, then the remaining space can be used to point to the rest of the data. For example: class C : fix_size(12) { int x; int y; int i; int j; }; could become: class C { int x; int y; struct {int i; int j;} * data; } 37 To do this, we modify the macro definition in Figure 5.1 to use the last bit of available space for the overflow pointer instead of returning an error. To a user of the class, the fact that some data members are stored in a separate object is completely transparent. In the above example, if x is an instance of class C, then data member i can be accessed using x.i. The full expansion of class C is something like: class C { int x; int y; class Overflow { struct Data { int i; int j; }; struct Data * ptr; Overflow() {ptr = malloc(sizeof(Data));} Overflow(const Overflow & o) {ptr = malloc(sizeof(Data));} ~Overflow() {free(ptr);} }; Overflow overflow; pseudo_member i int overflow.ptr->i; pseudo_member j int overflow.ptr->j; }; The key to making this work is the use of pseudo_member (which is built into the default class macro) to create pseudo members that behave like normal members for most purposes. This support includes properly calling the constructor and destructor for the member if it has one. Thus, the members in C::Overflow::Data will get properly initialized even though malloc/free is used instead of new and delete. In principle, the fix_size macro can work without the pseudo_member extension, but doing so greatly increases the complexity of fix_size, and implementing pseudo_member in the class macro was accomplished in around 6 lines of code. In addition, a closely related feature, alias, is useful for implementing other features such as anonymous unions. An alias is like a pseudo_member except that the constructor and destructor for the member are not called. We chose to implement pseudo_member in the default macro class. However, since the class macro is built using its own class system, extending the class macro to support pseudo_member is fairly straightforward, and would still be less work than trying to do all the work in the fix_size macro. The enhanced fix_size macro can also be used to store all the private data, i.e. the "pimpl idiom," in a separate object by specifying a size of zero, which the fix_size macro would recognize as a special case. 38 5.1.3 Validation Both previously mentioned techniques have been implemented in ZL as a macro library. All the end user needs to do is include the library, which will replace the class implementation with one that supports fixing the size. We have verified that the size does not change under various scenarios and hence binary comparability will be maintained. 5.2 Fixing the Size of the Virtual Table Adding new virtual methods can break binary compatibility in essentially the same way as adding data members. Since the macro that implements classes uses another class to implement the vtable, all of the techniques previously discussed can easily be used to fix the size of the vtable. To make this strategy work, the ZL class macro provides a way to specify the implementation of the class used to implement the virtual table. We have written a macro that uses the technique just described to allowing fixing the vtable size using the special syntax: class X : fix_vtable_size(8) {...} which will fix the vtable size to 8 bytes. We have verified that the macro does indeed fix the size of the vtable and hence maintains that aspect of binary compatibility. We have also written a a more sophisticated macro that, amount other things, allows the size to be implemented in terms of slots, which is discussed in Section 6.6. 5.3 A Better ABI Adding new data members or methods breaks binary compatibility because the sizes of the class and vtable are needed at compile time. The size of the class is needed when directly allocating an object on the stack, or when inlining one object into another. The first can be avoided by dynamically allocating the class on the heap. However, the second is a problem with most C++ ABIs as a typical C++ ABI defines class layout to be something like: class Parent {...}; class Child { Parent parent; ...}; which inlines the parent in the child class. This means adding new data members to the parent class will break binary compatibility for any code that depends on the child class. 39 We defined a new ABI to avoid this problem. Our new ABI defines class layout to be something like: class Parent {void * child_ptr; ...}; class Child { Parent * parent_ptr; void * child_ptr; ...}; where the parent class is dynamically allocated when the child class is created, and child_ptr is used to downcast. This strategy preserves binary compatibility when new data members are added to the parent. A similar strategy is used for the vtable. The code to implement the new ABI is under 60 lines of code. It overrides three methods from the core class macro; the method that adds the parent info to the user type was rewritten, and some additional information was added to every user type to include the child pointer. We verified that the new ABI maintains binary comparability when adding new data members by creating a situation in which adding data members would cause problems with the more traditional ABIs. For example, in the following code: class X {int x;} class Z : public X {int z;} adding a new data member, say y, to X will break binary compatibility with programs that use Z since the addition will change the offset of z. Therefore, accesses to the data member z will report an incorrect value. We verified that this was indeed a problem with ZL's default ABI, by setting the value of z with object code compiled against the new API (the one with the new y data member) but reading the value with object code compiled against the original API (without y) and verified that a different value was returned. We then did the same thing with the new ABI and verified that the same value was returned. We did a similar test to verify that adding new virtual methods will not break binary comparability. For many purposes, this ABI can impose too much overhead. For example, each class must have a pointer to the child to support down casting, and virtual-method dispatch is slower. When binary compatibility is a primary concern, however, this ABI can be a good choice. Furthermore, since ZL can use more than one ABI at a time, a programmer can choose this ABI for just the parts of a program where the benefits in binary compatibility outweigh the costs in performance. 40 5.4 Matching an Existing ABI Because classes are just user types to the compiler, it is possible to construct classes to match an existing ABI. This includes specialized ABIs which are really a C implementation of classes (such as done in GNOME [5]) or C wrappers around a C++ API (such as done in Aspell [1]). Doing so provides a more class-like interface to the C API. For example, ZL's macro API is a pure C API for simplicity; however, a more class-like interface is also provided. ZL provides a class-like interface to many of the API types including Match, Syntax, and UnmarkedSyntax. For example, instead of using match_var(m, syntax x), one can use m->var(syntax x). This is done by creating a user type Match that looks something like: user_type Match { associate_type struct Match; macro var(str, :this ths) { match_var(ths, str);} }; 5.5 Matching GCC's ABI Just as it is possible to match a C ABI, it is possible to match other compilers' ABIs. It is even possible to use classes with different ABIs in the same program with some restrictions, which depend on fundamental incompatibilities between different ABIs. For example, while it is possible to mix classes with different ABIs through composition, doing so via inheritance is unlikely to work. This is due to differences in how inheritance is implemented, and in particular, how the vtable is laid out. To demonstrate that ZL is complete enough to match another compiler's ABI we have matched the GCC ABI. The vtable layout turned out to be compatible with ZL's default ABI. However, there were still some key differences between ZL's default ABI and GCC. The most significant one is that each class has multiple implementations of each constructor and destructor. In particular there is an allocating constructor that calls new and then constructs the object, the constructor that is called by derived classes, and the normal constructor. In a similar fashion there are multiple destructors. If the destructor is virtual than there are also multiple destructors in the vtable, hence affecting vtable layout. In addition to class layout, the mangling scheme used by GCC is different from that of ZL. 41 The code to implement the GCC ABI consists of around 150 lines of code to extend the class macro and around 300 lines of code to implement the alternative mangling scheme. A demonstration that we indeed matched the GCC ABI is given throughout the next chapter. 5.6 Matching Another ABI In addition to the ABIs we have already implemented, we implemented one additional ABI. We defined a new ABI by building on the existing class macro to pass the this parameter as a global variable. This implementation simulates passing the this parameter in a register, as the Microsoft C++ ABI does, as opposed to passing it as the first parameter, as GCC does. We then used both ABIs in the same program, and even embedded classes with one ABI in another via composition. The code to implement the new ABI was under 45 lines. The only methods from the core class macro that needed to be overridden were the ones involved with constructing and calling member functions-three in all. 5.7 Other ABI Problems This chapter illustrated how ZL enabled solutions to key ABI problems outlined in Chapter 3. There are no fundamental limitations to solving the other ABI problems outlined in that chapter. Regarding techniques for maintaining binary compatibility, it is just a matter of writing the macros to implement the additional techniques. The main difficulty in the unimplemented techniques is the bookkeeping to keep track of previous states in the ABI. For example to avoid breaking binary compatibility when reordering methods it is necessary to keep track of the old layout somehow. While possible with ZL macros-as they can perform I/O-there is still some issues to work out before they can be made reliable. CHAPTER 6 THE CASE OF A SIMPLE SPELL CHECKER In this chapter we use the techniques of the previous chapter to mitigate binary com-patibility problems through the evolution of a simple spell checker (which we refer to as Simple Spell). In addition, we demonstrate ABI compatibly with GCC. 6.1 Simple Spell Simple Spell is a spell checker that provides basic spell checker functionally. It can check that a word is in a dictionary and handles case in a intelligent fashion; for example, if a first letter is upper case, Simple Spell will first try to match the word in a case-sensitive fashion, and if that fails, it looks for an all lower-case version of the word; thus, it will reject "Mcdonald" as the correct spelling is "McDonald," but still accept "Color" and "Dog." If a word is not found in the dictionary then Simple Spell will offer a list of words which are within one edit-distance of the misspelled word. For example, it will suggest "the" when given "teh" or "color" when given "colr." If the misspelled word is indeed the correct spelling, Simple Spell can remember the word to avoiding flagging it again via a session dictionary. In addition to offering basic spell checker services such as checking if a word is the dictionary, Simple Spell also provides an API for checking documents. The document checker provides the ability to skip over parts of the document that should not be spell checked, such as URL's, via a plugable filter interface that selectively blanks out part of the document. A simple URL filter is provided. 6.2 The Spell Checker API With concern to binary compatibility there are two API's of interest; there is the API for applications simply wishing to use the spell checker and there is the extension API for 43 those wishing to extend the functionally of the spell checker. So that Simple Spell can be used by more than just ZL, we will use GCC's ABI for the application ABI. The extension ABI will be in ZL's own ABI so that we have more flexibility in the techniques used to mitigate ABI compatibility problems. Nevertheless, we will still be able to make use of extensions compiled with GCC through the use of a bridge class which will be discussed in a latter section. 6.2.1 The Application API The most important class in Simple Spell is the Speller class, which is responsible for checking that a word is correctly spelled-and when it is not, coming up with a list of suggestions. The definition of the Speller class is defined in speller.hpp as shown in Figure 6.1. The basic usage of Simple Spell is to create a new instance of the Speller class and then initialize it via the init method by giving it a language and dictionary class. The API of those two classes is of no interest to the application writer, instead new instances are created via the new_lang and new_master_dict factory functions, which are shown in Figure 6.2. Once a new instance of the Speller class is created, the check method is used to check if a word is the correct spelling. If the word is not the correct spelling the suggest method can be used to come up with a list of possible replacements, or if the word is indeed correct, the add_to_session method can be used to ignore the word for the rest of the session. The Suggestions struct is used for iterating through the suggestion results. It is a simple wrapper class and as such the implementation details are completely exposed via the header file for the sake of efficiency. The SugsData struct is an internal class used by ZL, but its existence must be exposed in the header file since it is a data member of the Speller class. The preprocessor macros GCC_ABI_BEGIN and GCC_ABI_END ensure that the GCC ABI is used when compiled with ZL. They are defined in the config.hpp header file as such: 44 ... #include "config.hpp" ... GCC_ABI_BEGIN struct Suggestions { ... const_iterator begin() const {return begin_;} const_iterator end() const {return end_;} const char * operator[](unsigned n) const {return begin_[n];} unsigned size() const {return end_ - begin_;} }; struct SugsData; class Speller { ... SugsData * sugs_data; ... public: Speller(); void init(Language * lang, Dictionary * main); bool check(const char *); void add_to_session(const char *); Suggestions * suggest(const char *); ~Speller(); private: Speller(const Speller &); // no copy }; GCC_ABI_END Figure 6.1. The speller.hpp header file providing the core functionally of Simple Spell. Language * new_lang(const char * name); Dictionary * new_master_dict(Language *, const char * fn); WritableDict * new_session_dict(Language *); Figure 6.2. Other parts of the core simple spell API defined in other header files. 45 #ifdef __zl # define GCC_ABI extern "C++" : "GCC" # define GCC_ABI_BEGIN extern "C++" : "GCC" { # define GCC_ABI_END } #else # define GCC_ABI # define GCC_ABI_BEGIN # define GCC_ABI_END #endif where __zl is defined by the zlc compiler when prepossessing ZL code and "extern "C++" : "GCC"" selects a macro-pluggable ABI implementation (details in 7.5.3). The other important part of the spell checker API is the document checker interface, which is shown in Figure 6.3. The Session class provides basic document checker support and the SessionWFilters class extends it with basic filter support. A document is checked one line at a time by passing in a line with the new_line method. The next_misspelling method is then used to advance to the next misspelled word on the current line, assuming there is one, otherwise it returns false. When there is a misspelled word misspelled_word returns the word, and misspelled_offset and misspelled_len can be used to find the word in the current line. If the misspelled word was replaced with another, supposedly correct word, the replace method needs to be used to inform the document checker of the correct spelling. When this method is used the checker will recheck the word to make sure it is correct before advancing on. The extended document checker interface SessionWFilters functionally is identical to Session except that before checking the document one or more filters needs to be added using the add method. The function new_url_filter returns a new instance of the URL filter. The details of the Filter class are part of the extension API. 6.2.2 The Extension API Simple Spell supports the ability to provide custom filters by extending the Filter class defined in Figure 6.4. New filters simply define the filter method, which blanks out any part of the line that should not be spell checked. New filters can then be added via the add method of the already shown SessionWFilters class (Figure 6.3). 46 class Session { protected: char * word; unsigned misspelled_start; unsigned misspelled_stop; ... public: Session(Speller * sp); virtual void new_line(const char *); virtual bool next_misspelling(); virtual void replace(const char * new_word); const char * misspelled_word() {return word;} unsigned misspelled_offset() {return misspelled_start;} unsigned misspelled_len() {return misspelled_stop - misspelled_start;} virtual ~Session(); }; class SessionWFilters : public Session { ... public: SessionWFilters(Speller * sp); virtual SessionWFilters & add(Filter *); virtual void new_line(const char *); virtual ~SessionWFilters(); }; Filter * new_url_filter(); Figure 6.3. Simple Spell document checker API. All parts of this API use the GCC ABI. class Filter { public: Filter() : next() {} virtual void filter(char * line) = 0; virtual ~Filter(); Filter * next; }; Figure 6.4. Simple Spell extension API. 47 6.3 A Simple Application and Binary Compatibility To demonstrate the functionally of Simple Spell, and that we matched another compil-ers ABI, we wrote a simple application that uses the Simple Spell library. When given a file name (via the command line) the application checks the current document. Otherwise, it enters a simple demonstration mode that accepts one word per line and reports it as either correct or incorrect and then offers a list of suggestions. The interface for checking a document is simple but functional. It checks the provided text file for spelling errors and when one is found, prints out the line with the misspelled word highlighted, offers a list of suggestions, and then prompts the user for what to do next. For example: *Teh* dog swm up the stream. 1) The 2) Tea 3) Ted 4) Tee 5) Tel 6) Ten 7) Tet 8) TeX 9) Tech i) Ignore I) Ignore all r) Replace a) Abort The user can then either accept one of the suggestions, ignore the word this time, ignore the word for the rest of the document (i.e., add it to the session dictionary), offer a replacement, or abort. When done checking the document, a new file is written out that has the same name as the original file but with the .new extension added. We have compiled this simple application with GCC and linked it with a version of Simple Spell compiled with ZL, thus demonstrating that we have indeed matched GCC ABIs with ZL. In addition we have compiled the application with ZL and linked it with a version of Simple Spell compiled with GCC, thus further demonstrating ABI compatibility. 6.4 Adding a Filter, Compiled with GCC Due to the choice of using ZL's ABI for the Filter class, the Simple Spell library, when compiled with ZL, can not make direct use of a Filter class that is compiled with GCC. However, we can rectify this situation through the use of a simple bridge class. 6.4.1 The Bridge Class The bridge class is shown in Figure 6.5. The Filter class with ZL's ABI is included in the file filter.hpp, while the Filter class in GCC's ABI in wrapped in a module so that we can refer to both at the same time. Normally, the module name will appear as part of 48 #include "filter.hpp" module GCC :asm_hidden { GCC_ABI class Filter { public: Filter() : next() {} virtual void filter(char * line) = 0; virtual ~Filter(); Filter * next; }; } class FilterBridge : public Filter { GCC::Filter * GCC_filter; public: FilterBridge(GCC::Filter * f) : GCC_filter(f) {} void filter(char * line) {GCC_filter->filter(line);} ~FilterBridge() {delete GCC_filter;} }; extern "C" Filter * filter_bridge (void * filter) { return new FilterBridge(reinterpret_cast<GCC::Filter *>(filter)); } Figure 6.5. A bridge class to allow using filters compiled with GCC. 49 the mangled name of symbols defined within it; however, this is clearly not what we want in this case. Thus, the :asm_hidden flag is used to make the module invisible as far as external names go. Normally, this will cause name conflicts, but since a different mangling scheme is used for the ZL and GCC ABIs, there is no conflict. The actual bridge class is fairly simple and should be self explanatory. It implements ZL's Filter interface by simply forwarding the filter method to GCC Filter class. Directly including header files for any parts involved with the GCC ABI is extremely problematic since there will now be two filter classes, one of which is meant to be in ZL's ABI and others GCC's. A module is not the same thing as a C++ namespace; for example, the following will not work: module GCC {class Filter {...};} module GCC {class EmailFilter : public Filter {...};} as the second module will shadow the first rather than extending in. Thus, wrapping the header files in the module will not work. When ZL implements C++ namespaces this might be made to work, but for now we simply avoid the need by not referring to the GCC Filter class in the parameter for the filter_bridge factory function, and instead cast the void pointer to the correct type. 6.4.2 Adding The Email Filter With this bridge class now written we make use of it to add a filter that is compiled with GCC to our application. The new filter, the email filter, is a simple filter that skips quoted lines. For example given: > This line will be skippd. This line will be checkd. the word "skippd" will not be checked but the word "checkd" will. We avoid including the actual definition of the class itself and instead only include the declaration of the factory function: extern "C" void * new_email_filter(); We then make use of the email filter by passing in the pointer returned by new_email_filter into filter_bridge to create a new Filter instance using ZL's ABI. 50 6.4.3 Automating the Creation of the Bridge Class The filter class is fairly simple with only one real method. The creation of the bridge class for more complicated methods would be a lot more tedious. In addition, there is the burden of keeping the bridge class up-to-date as methods are added or removed from the interface class. Fortunately it is fairly easy to automate the creation of the bridge class with a procedural macro. Figure 6.6 shows the essential part of the bridge class. In order to avoid having to include the definition of the class in the macro call we extract the original syntax object from the class definition by using get_symbol_prop to extract the syntax_obj property from the module used to implement the class. The syntax_obj property is one of many properties added by the class macro. Once we have the syntax object for the class we extract the virtual method definition and create the necessary bridge code. We then return the code to define the bridge class. The symbols OtherAbi and Bridge are lexically scoped and thus we do not need to worry about conflicts with other bridge classes. With this macro now written we replace the code in Figure 6.5 with mk_bridge(Filter, filter_bridge, "GCC"); in the Simple Spell library. 6.5 Adding Support for a Personal Dictionary No spelling dictionary can include every possible valid word; thus a key feature of almost any spell checker is the ability to maintain a personal dictionary. We would like to be able to add this feature to Simple Spell without breaking binary compatability. Unfortunately, since we allow direct allocation of the Speller class (by exposing the class definition, private data members and all) we cannot easily extend the Speller class without breaking binary compatability. For one thing, we cannot add private data members as that will change the size of the class instance. Fortunately, all is not lost as we can still extend the the Speller class, we just need to be careful not to change of the size of the class. Doing so using traditional C++ can be very tedious and error prone. However, assuming we are willing to require that the library 51 Syntax * parse_mk_bridge(Syntax * p, Environ * env) { Mark * mark = new_mark(); Match * m = match_args(0, syntax (class_n, fun_n, abi_name), p); Syntax * class_syn = get_symbol_prop(m->var(syntax class_n), syntax syntax_obj, env); m = match_args(m, raw_syntax (_ @ (pattern ({...} @body)) @_), class_syn); SyntaxList * bridges = new_syntax_list(); SyntaxEnum * itr = partly_expand_list(m->varl(syntax body), FieldPos, env); Syntax * member; while ((member = itr->next)) { // if memeber is a virtual method, create a // forwarding method and append it to bridges list } UnmarkedSyntax * res = syntax { module OtherAbi :asm_hidden { extern "C++" : abi_name $1; } class Bridge : public class_n { OtherAbi::class_n * obj; Bridge(OtherAbi::class_n * o) : obj(o) {} $2; ~Bridge() {delete obj;} }; extern "C" class_n * fun_n (void * o) { return new Bridge(reinterpret_cast<OtherAbi::class_n *>(o)); } }; return replace(res, match_local(m, class_syn, bridges, 0), mark); } make_macro mk_bridge parse_mk_bridge; Figure 6.6. Part of the mk_bridge macro. The real implementation is just under 55 lines of code. 52 is compiled with ZL, we can use the fix_size macro from Section 5.1 to maintain the size of the class and thus maintain binary compatibility. Note that while we require that the library be compiled with ZL, we will still be matching the GCC ABI. Thus applications that use Simple Spell can still be compiled with either GCC or ZL. Figure 6.7 shows the part of the header file defining the improved Speller class. The header file is designed to be used by both GCC and ZL. To be able to fix the size of the class, we first must determine what the size of the old class was; thus we create a dummy class, SpellerOld for the sole purpose of taking its size. We then use fix_size to fix the size of the new Speller class. With the class size fixed, we are free to add (or remove) new private data members. We can even reorder existing onces since we do not expose any code (in the form of inline functions) that use the private data members. We will also naturally need to add some additional methods to the class, but this will not break binary compatability since the Speller class does not have a vtable. We can even even overload an existing method, as is done with init, without a problem as it is equivalent to adding a new method since the two methods will be mangled differently. However, since fix_size is a ZL construct and this header file is also used by applications compiled with GCC we must also fix the class size in GCC's eyes. To do this we replace the entire class with a character array of the correct size when the header is read by GCC (or other non-ZL compiler). Since the application has no need to access the private data members this is all that is needed to preserve binary compatibility. It is important to note that while the header file is slightly complicated, it is far simpler than any solution would of been without the aid of the fix_size macro. In particular, the changes shown here are the only changes necessary to fix the size of the class. The library code does not need to worry about the fact the some of the private data members are now likely in a separate object, nor does it need to worry about maintaining the object which is likely to be heap allocated. To test the new functionally and to verify that we still match the GCC ABI, we enhanced our sample application to take advantage of the new personal dictionary. We then compiled the application with both ZL and GCC and linked it with the same library (which now must 53 class SpellerOld { // original Speller class private data members }; class Speller #ifdef __zl : fix_size(sizeof(SpellerOld)) #endif { public: // but don't use #ifdef __zl // private data members SavableDict * personal; // more private data members #else union { void * datap; // to make sure the structure is aligned char data[sizeof(SpellerOld)]; }; #endif public: Speller(); void init(Language * lang, Dictionary * main); void init(Language * lang, Dictionary * main, SavableDict * personal); // NEW bool check(const char *); void add_to_session(const char *); void add_to_personal(const char *); // NEW void save_personal(); Suggestions * suggest(const char *); ~Speller(); private: Speller(const Speller &); // no copy }; Figure 6.7. Extending the Speller class to include support for a personal dictionary. 54 be compiled with ZL). We also verified that we indeed maintained binary compatability by linking the original application (before the changes in this section) with the new library without recompiling and verified that everything worked. 6.6 A Better ABI to Allow Future Enhancements Through the use of ZL we were able to extend the Speller class without breaking binary compatability. However, the only reason we were able to do this was because we did not need to add any virtual methods to the Speller class. If we did, we would not have been able to extend the Speller class as adding any virtual methods will change the offset in the vtable for any derived classes. In addition, we would not be able to fix the size of Speller's vtable as we did with the class itself since-unlike with the private data members-the application does directly use the vtable. As such GCC will not be able to use any methods whose pointer is not directly stored in the vtable. However, with some planning ahead we can create a better initial ABI which allows for easier expansion. For every class whose definition is exposed to the application we will fix the size of the class from the start. We will also fix the size of the vtable to allow for future expansion. As long as the vtable size is larger than the required size we will not create ABI problems for GCC as a separate object will not need to be used. We will also take use this opportunity to hide some unnecessary implementation details from the application. Figure 6.8 shows the new Speller class definition. Since the applicaton has no need to access any of the private data members we chose to use fix_size to implement the pimpl idiom. By doing so we also able to avoid a layer of indirection in the orignal API; in the orignal API (Figure 6.1) the Speller class contained a pointer to the SugsData class to avoid having to expose the SugsData definaton in the header file. However, with the pimpl idiom this is unnecessary since only the Speller class needs access to the private data members. Any source file that does not implement the Speller class only needs to know the size of the class (just like application using the Simple Spell library). Thus, unless __speller_impl is defined all the other source files will see is void * impl. Any source file that defines part of the spell checker includes an alternative header file speller_impl.hpp which defines the SugsData class, the __speller_impl preprocessor 55 class Speller #ifdef __speller_impl : fix_size(0) #endif { #ifdef __speller_impl ... SugsData sugs_data; #else void * impl; #endif public: Speller(); void init(Language * lang, Dictionary * main); // will take ownership of both bool check(const char *); void add_to_session(const char *); Suggestions * suggest(const char *); // result only valid to next call to suggest ~Speller(); private: Speller(const Speller &); // no copy }; Figure 6.8. The Speller class using the pimpl idiom. macro and then includes speller.hpp, for example: struct SugsData {...}; #define __speller_impl #include "speller.hpp" #undef __speller_impl Figure 6.9 shows the new Session class definition. Since, unlike like the Speller class, some of the private data members are used by the application (via inline functions) we can not use the pimpl idiom. Instead we fix the size of the class to be just large enough to include the pointer-to-vtable and the private data members used by inline functions. To make the header file more readable we define a few preprocessor macros that are defined differently depending on whether ZL is being used as follows: 56 class Session FIX_SIZE(sizeof(void *) /* vptr */ + sizeof(unsigned) * 2 + sizeof(char *) + sizeof(void *)) VTABLE_SLOTS(16) { protected: unsigned misspelled_start; unsigned misspelled_stop; char * word; #ifdef __zl // other private data members #else void * impl; #endif void reset_line(); public: Session(Speller * sp); virtual void new_line(const char *); virtual bool next_misspelling(); virtual void replace(const char * new_word); const char * misspelled_word() {return word;} unsigned misspelled_offset() {return misspelled_start;} unsigned misspelled_len() {return misspelled_stop - misspelled_start;} virtual ~Session(); #include "vtable_pad-Session.inc" }; Figure 6.9. Improved Session class to support future enhancements without breaking binary compatibility. 57 #ifdef __zl # define FIX_SIZE(size) :fix_size(size) # define VTABLE_SLOTS(size) :vtable_slots(size) #else # define FIX_SIZE(size) # define VTABLE_SLOTS(size) #endif In addition, and unlike the Speller class, the Session class has a vtable; thus, we also fix the size of the vtable so we can add new methods without breaking binary compatability. However, since we also need to match the GCC ABI and provide a header file that can be used with GCC we need to use a method slightly more complicated than the method described in Section 5.2 in which we used the same fix_size macro on the vtable class. The problem is that we need to fix the size of the vtable in GCC's eyes but we can not just provide a dummy character array as we can not directly specify what goes into the vtable, and even if we could the application needs to access the vtable for virtual dispatch. But we can still fix the size by including dummy virtual methods, which is what is included in the file vtable_pad-Session.inc: #ifndef __zl virtual void * dummy__0001_(); virtual void * dummy__0002_(); ... virtual void * dummy__0011_(); #endif Thus, to create the vtable_pad-Session.inc file we use a specialized macro de-signed specially for the vtable class. This macro will, as a side effect, write out a file with the correct number of dummy methods to be used by GCC. In addition, since we are using a specialized macro, we can also allow the user to specify the size in terms of available slots for virtual methods rather than raw size. Thus we use :vtable_slots instead of :fix_vtable_size. Since using dummy methods is the only viable method to fixing the size of the vtable with GCC we need to plan ahead and make sure we reserve enough slots to allow for future expansion. We choose to use 16, but we could easily make that larger since the vtable is only allocated once per class (as opposed to once per class instance) and thus does not waste a lot of memory. 58 The SessionWFilters class, shown in Figure 6.10, gets similar treatments except that we fix the size to 0 (and thus effectively use the pimpl idiom) since none of the private data members need to accessed by the application. Since we already decided that we will not match the GCC ABI for the extension interface we will use the more complicated ABI described in Section 5.3 for the filter class so we don't have to worry about reserving enough vtable slots ahead of time. For reference the the new definition is included in Figure 6.11. While we changed the ABI from the orignal we have not made any changes to the API, thus we can reuse the same application after a simple recompile. class SessionWFilters : public Session FIX_SIZE(0) VTABLE_SLOTS(16) { #ifdef __zl // private data members #else void * impl; #endif public: SessionWFilters(Speller * sp); virtual SessionWFilters & add(Filter *); virtual void new_line(const char *); ~SessionWFilters(); #include "vtable_pad-SessionWFilters.inc" }; Figure 6.10. Improved SessionWFilters class. extern "C++" : "better" class Filter { public: Filter(); virtual void filter(char * line) = 0; virtual ~Filter(); Filter * next; }; Figure 6.11. The Filter class using an enhanced ABI. 59 6.7 A Simple Spell Checker, Version 2 Now that we have defined a better ABI we can make two key enhancements with minimal effort: 1) add a personal dictionary and 2) better support filters with state. The changes made for the first enhancement are identical to the ones we made in Section 6.5, except that we no longer need any of the tricks in that section as we already fixed the size. All we need to is add the necessary private data members and methods. So far the filters we have added are stateless, that is they they do not need to maintain any state between lines. But most useful filters will need to maintain sort of state between line; for example, a filter to only check the comments a C or C++ source file will need to know if the previous line started a C style comment. The current API will support such filters as long as only a single document is checked and the lines are checked in sequential order. However, it is sometimes necessary to recheck the same document and often useful to check more than one document without having to create a new session. Thus, to better support stateful filters we add a new virtual method to the Session and Filter class, reset(), which simply resets the state. Since not all filters need to implement this method we provide a no-op default implementation. Adding the reset method to Session would normally break binary compatability since it will change the offset of any derived classes. In particular, in SessionWFilters, the offset of the add method will change. Thus, if an application calls the add using the new SessionWFilters class with a header file for the old implementation, the wrong method will be called. As we already discussed, without planning ahead and reserving spots there is little we could have done to avoid this problem while still using the GCC ABI. Fortunately, we did plan ahead and adding the new method does not cause a problem. To take advantage of the new filter API we added a new filter to our sample application that simple checks all comments in a C or C++ source file and ignores the rest. To verify that we indeed maintained binary compatability, we linked the original application (before the changes in this section) with the new library without recompiling and verified that everything worked. In addition we tried adding the reset method without fixing the size of the vtable and verified that the wrong method was called as predicted. 60 6.8 An Opportunity for an Even Better ABI The enhanced ABI we used for the Filter class (from section 5.3) goes a long way towards preserving binary compatibility. The new ABI will avoid changing the offsets of any virtual methods of derived classes when adding new methods to the base class. But unfortunately this is still not enough to allow us to use filters compiled with the old ABI with the new ABI, at least without some extra care. That is, we can use a filter from the old ABI as long as the new reset method is never called. If we do try to call the reset method on a filter with the old ABI, the application will crash. The problem is that filters compiled with the old ABI will still use the original vtable since they are created statically when the application starts. Hence, the application will crash since the original vtable does not contain that slot. A better ABI, which could be implemented in ZL, could avoid this problem by dy-namically creating the vtable so that all derived classes will use the vtable for the Filter class of the new ABI regardless of which ABI they where originally compiled with. This change will allow new virtual methods to be called without a problem provided that they have a default implementation. If they do not (i.e., they are pure virtual) then source code compatability will also be broken and hence there is no point in trying to maintain binary compatibility. 6.9 Comparison to a Real Spell Checker: Aspell Simple Spell is modeled after a real spell checker, Aspell [2]. In many ways the interface to Simple Spell mirrors that of Aspell. Of course, Aspell is far more complex than Simple Spell, with Aspell containing around 30,000 lines of code and Simple Spell containing between 1,100 and 1,700 lines of code (depending on which version) (see Table 6.1). To mitigate ABI compatibility problems Aspell does not expose a C++ interface. Instead, a Perl script is used to generate the C interface. The Perl script is around 1,900 lines of code, with around a 1,000 line input file. The Perl script generates around 3,000 lines of code, of which 400 lines consist of code that is now manually maintained. An additional 1,900 lines of manual interface code is also used in the C interface (not including the 400 61 Table 6.1. Approximate lines of code of the various versions of Simple Spell and Aspell. Spell Checker Version Section Lines Of Code Simple Spell (Initial Version) 6.1 1,100 Simple Spell w/ Email Filter 6.4 1,300 Simple Spell w/ Personal Dictionary 6.5 1,300 Simple Spell w/ Better ABI 6.6 1,600 Simple Spell, Version 2 6.7 1,700 Aspell - 30,000 lines once generated with the Perl script). The large line count for the interface code reflects the fact that Aspell is a moderately complex program, and also that the bridge between the internal C++ interface and external C interface is rather involved; for example, it includes support for managing memory and conversion of the input and output from one encoding (such as UTF-8) to Aspell's internal 8-bit encoding. Due to its use of advanced C++ features (such as templates) Aspell is currently unable to compile under ZL. Had it been able to compile, the bridge code (from the internal C++ ABI to the external C ABI) could be written using ZL; the information in the interface file could become part of the class, and a modified class macro can be used to extract it. It remains to be seen if the end result will be any simpler. More importantly, by using many of the techniques outlined in this chapter, it will likely be possible to directly expose a stable C++ that