Considerations in the choice of interobserver reliability estimates

Two types of interobserver reliability values may be needed in treatment studies in which observers constitute the primary data-acquisition system: trial reilability and the reliability of the composite unit or score which is subsequently analyzed, e.g., daily or weekly session totals. Two approaches to determining interobserver reliability are described: percentage agreement and "correlational" measures of reliability. The interpretation of these estimates, factors affecting their magnitude, and the advantages and limitations of each approach are presented.