Study of inter-rater reliability using various patient categorization methods

Nurses have determined staffing patterns in the hospital environment for years by patient census at its historical peak. Some nursing administrators questions this as being the most appropriate way to manage the workload had have suggested identifying patient's needs for care and placing the patients into categories based on those needs. Once the category of classification has been determined, the assigned category for each patient is converted to pre-determined hours of care. This system then is dependent upon the reliability of nurses using the tools for categorization and the validity of the tools themselves. This study was designed to look at the inter-rater reliability of staff nurses using various types of categorization methods. The tools in the study ranged from a subjective, intuitive nursing assessment, to more structured format with suggest criteria (Modified Georgette), to a checklist format, (Pardee Checklist), to a point system multiple dimension tool with specific criteria (Poland Point System). These methods were use by six registered nurse raters in a general hospital assigned to two medical divisions. Each nurse rated a randomly designated patient using all four of the methods. Pearson product-moment correlations were calculated to determine the inter-rater reliability. Of the four tools, the intuitive, subjective method of identifying patient needs was the least reliable with correlations of .18 to .85 and an overall correlation of .57. The reliability coefficients for the Modified Georgette ranged from .00 to 1.00 with an overall mean correlation of .73 and the Pardee Checklist had correlations between .09 and 1.00 with an overall mean correlation of .70. The Poland Point System, which is the most structured tool and has the greatest specificity of patient needs, was used most reliably as indicated by correlations of .49 to 99 and an overall mean correlation of .83. Experience seemed to be one factor that influenced the rater's ability to use the various tools. For example, nurse with less than one year of experience were much more consistent with other raters and themselves on the more structured tool, the Poland Point System. Raters with experience of three to twenty-five years and raters with less than one year of experience were not consistent with each other. Further study on the role of experience in assessing patient needs in indicated and deemed necessary to fully understand these findings. Nurses who assumed administrative roles in addition to their clinical assignment were not as consistent in their use of the various tools with other raters as were those nurses with primary clinical assignments. Further study is suggested on the time involved and the influence of administrative tasks on clinical management care. Rater intra-divisionally had high reliability coefficients. This study suggested high agreement with other rates on a single division or unit but low consistency existed between raters inter-divisionally. This certainly suggests the influence of leadership and peer group suggestion on assessment of patient needs. The scores do not suggest whether the influence was positive or not, only that raters on a unit tended to assess a patient in a similar way.