The cost of saliency: how salience interacts with relevance to support or impede information search

Technological innovations have increased the ability to collect and store information. However, these innovations potentially create a psychological problem because the increasing amounts of information must be managed within the still limited human cognitive processing capacity. One common data visualization approach to improve information search is the concept of increased bottom-up stimulus-driven saliency to guide the user. But is increasing the salience of an item enough to produce a sufficient, efficient search with high decision accuracy? How does increasing the salience of an item without regard to its relevance affect the search for information? Is there a difference between lists and tag clouds and what role does the system context; play? To answer these questions we adapted the concepts of the Wason selection task (WST) and considered the propositional logic values of P, Q, not P and not Q, to analyze search sufficiency, efficiency, decision accuracy and search patterns. We found that the incongruence or congruence of salience and relevance can impede or support the search for information. However, increasing the salience of relevant items is not enough to insure a sufficient or efficient search with an accurate decision. The search for information is affected by interactions between the display format, the system context; and the congruence conditions.

THE COST OF SALIENCY: HOW SALIENCE INTERACTS WITH RELEVANCE TO SUPPORT OR IMPEDE INFORMATION SEARCH by Heidi S. Kramer A thesis submitted to the faculty of The University of Utah in partial fulfillment of the requirements for the degree of Master of Science Department of Psychology The University of Utah December 2010 Copyright © Heidi S. Kramer 2010 All Rights Reserved The University of Utah Graduate School STATEMENT OF THESIS APPROVAL The thesis of Heidi S. Kramer has been approved by the following supervisory committee members: Frank Drews , Chair 8/25/2010 Date Approved Sarah Creem-Regehr , Member 8/25/2010 Date Approved Matthew Samore , Member 8/25/2010 Date Approved and by Cynthia Berg , Chair of the Department of Psychology and by Charles A. Wight, Dean of The Graduate School ABSTRACT Technological innovations have increased the ability to collect and store information. However, these innovations potentially create a psychological problem because the increasing amounts of information must be managed within the still limited human cognitive processing capacity. One common data visualization approach to improve information search is the concept of increased bottom-up stimulus-driven saliency to guide the user. But is increasing the salience of an item enough to produce a sufficient, efficient search with high decision accuracy? How does increasing the salience of an item without regard to its relevance affect the search for information? Is there a difference between lists and tag clouds and what role does the system context play? To answer these questions we adapted the concepts of the Wason selection task (WST) and considered the propositional logic values of P, Q, not P and not Q, to analyze search sufficiency, efficiency, decision accuracy and search patterns. We found that the incongruence or congruence of salience and relevance can impede or support the search for information. However, increasing the salience of relevant items is not enough to insure a sufficient or efficient search with an accurate decision. The search for information is affected by interactions between the display format, the system context and the congruence conditions. TABLE OF CONTENTS ABSTRACT ............................................................................................................................. iii INTRODUCTION .................................................................................................................... 1 Hypotheses ............................................................................................................................ 5 METHOD ................................................................................................................................ 9 Participants ............................................................................................................................ 9 Design .................................................................................................................................. 9 Procedure ............................................................................................................................ 15 RESULTS ............................................................................................................................... 17 Issues and Overview ........................................................................................................... 17 Congruence ......................................................................................................................... 18 Salience Display Formats ................................................................................................... 19 System Contexts ................................................................................................................. 21 Confidence .......................................................................................................................... 24 DISCUSSION ......................................................................................................................... 39 Congruence ......................................................................................................................... 39 Salience Display Formats ................................................................................................... 40 System Contexts ................................................................................................................. 42 Implications and Conclusions ............................................................................................. 43 APPENDICES A VIGNETTES ................................................................................................................... 44 B APPLICATION SCREEN SHOTS ................................................................................. 46 REFERENCES ....................................................................................................................... 49 INTRODUCTION Technological innovations including software applications and the internet have increased the ability to collect and store information. In May 2009 the digital content on the internet was estimated at around 500bn gigabytes (GB), which is roughly equivalent to 10 stacks of books reaching from the earth to Pluto (Wray, 2009). Clearly, the amount of information storage required for this large amount of data is significant. The psychological problem for applied cognitive psychology is one of increasing amounts of information that need to be managed within the limited human cognitive processing capacity, in a limited time period. In addition to the technical and theoretical challenge, an important question for Human Factors is how to find the information that we are looking for among all of these data. Compounding the problem is the additional issue of knowing what to look for. Finally, it is not quite clear if it is even possible to search this amount of information, even knowing what we are looking for under time constraints that are always present in the context of information search. Consequentially, it is an important research question of theoretical and applied relevance to examine what affects search sufficiency and efficiency in order to reach an appropriate and accurate resolution to a question or problem. Fortunately, there is a body of literature that provides some guidance. The presentation format affects how information is searched, how that information is interpreted, and how it is shared with others (F. R. Brown & Lovejoy, 1955; P. E. Brown, 1991; Carey & White, 1991; Carter, 1947; McCabe & Castel, 2008; Schaubroeck & 2 Muralidhar, 1991; Smith, Best, Stubbs, Archibald, & Roberson-Nay, 2002). For example, there is a rich history of information visualization tools such as charts, maps and pictures being used to present geographical information to aid human cognition and to communicate important information (Tufte, 2000; Wainer, 2005). One commonality of modern and older data visualization approaches is the concept of increased saliency to guide the user's search for information. The intent is to provide information in such way that the important or relevant information receives initial attention, and is processed early because of the properties of its appearance. The consequence is more effective guidance of the information recipient in searching, integrating and interpreting the information (Kleinmuntz & Schkade, 1993). Saliency has been defined as how perceptually distinctive the information is in the context in which it is presented (Fishbein, Haygood, & Frieson, 1970; Sanfey & Hastie, 1998). This perceptual distinctiveness or salience is accomplished by stimulus-driven or bottom-up processing which is based on features such as size, position, line orientation, form, sound levels, stimulus-onset, color hue and intensity, all of which can affect the search for information (Huang & Pashler, 2005; Nothdurft, 1993; Treisman & Gelade, 1980; Treisman, Gelade, & Yantis, 2000). Although there is evidence of an insignificant difference in reading speed between 10 and 14 pt fonts (Beymer, Russell, & Orton, 2007, July), larger size is generally accepted as increasing saliency (Huang & Pashler, 2005). Stimulus-driven salience is not the only factor that affects the search for information. Other factors that play an important role relate to top-down cognitive processes that are based on experience, expectancies and goals (Itti, 2007; Yantis, 1993). Instructions or setting a goal to attend to a unique stimulus among other similar stimuli affects the search for 3 information (Itti & Koch, 2001; Posner, 1980; Treisman & Gelade, 1980). For example, when searching for a yellow square in a display of multicolored shapes, goal driven top-down processing affects the attention allocated to yellow and/or square shapes over other colors or shapes such as red triangles (Itti, 2007). These top-down factors can determine the relevance of the data (Parkhurst, Law, & Niebur, 2002). Pirolli and Card (1999) state that when foraging for information a person assesses the relevance of the information with the intent to determine what particular information should be given further consideration. Considerable debate exists on the interaction between bottom-up and top-down processing when searching for information (Folk, Remington, & Johnston, 1993; Theeuwes, 2004; Yantis, 1993) Some researchers believe that top down goal directed processes cause the stimulus-driven attentional capture (Gibson & Kelsey, 1998). Others believe that top-down processing factors cannot override attention capture (Theeuwes, 2004). Still others believe that goal directed or relevance factors affecting attention can modulate or override the stimulus salience involved in bottom-up processing, thus making top-down selectivity possible during searches for information (Bacon & Egeth, 1994; Itti & Koch, 2001; Lamy, Tsal, & Egeth, 2003; Richard, Wright, & Ward, 2003) In sum, there are three competing predictions for the interaction between bottom-up and top-down processing: 1. Saliency is determined by goal driven top-down processing. 2. Bottom-up processing determines the attentional capture regardless of top-down processing. 3. Top-down processing modulates or overrides the bottom-up processing. Researchers continue to explore the factors and contexts that lead to these disparate models (Biggs & Gibson, 2010). The ongoing controversy highlights the risks of attempting to "improve" information search by simply increasing the 4 stimulus-driven salience of data by changing features such as the size and position without understanding the search strategies and/or the relevance of the data. An important challenge when determining the appropriate level of salience for data visualization is that the relevance of the data may not be known a priori. One common approach to overcome this problem is to equate frequency, or number of occurrences in the context, with relevance, and simply increase the stimulus-driven salience of the most frequent occurrences. An example of this approach is tag clouds. Tag clouds are a collection of words or word phrases that appear in different sizes and colors. Tag clouds with hypertext links are frequently used as a method to support navigation or information search. Unfortunately, this approach of equating the number of occurrences of individual pieces of information with the relevance of the information may not be a valid aid for information search in all cases. Consider the tag cloud created from President Barack Obama's January 2010 State of the Union address (Figure 1) (MSCHW, 2010, January 28). The most salient word "will" may be the most frequently used word in the speech, but it is unlikely to be the most relevant word to aid a search to understand the concepts presented in the speech. The saliency of the generic words "people" and "Americans" also overshadows topics such as the "deficit," the "economy" and the very small text size of "recession" and "recovery." The incongruence of stimulus-driven saliency and relevance might have a negative impact on sufficiency and efficiency of the search for information using tag clouds. Another issue when considering information search is the context of the search. Durso and Drews (2010) make a distinction between three types of system contexts: 1. Natural systems consisting of elements that occur without intentional design or manufacturing and can adapt; 2. Social 5 systems pertaining to human communities or interactions; 3. Technical systems pertaining to machines or man-made devices. The context has been shown to affect reasoning and search patterns (Cheng & Holyoak, 1985; Cosmides, Tooby, & Gazzaniga, 2004; Dawson, Gilovich, & Regan, 2002; George, 1991; Roberts, Schaeken, Vandierendonck, Schroyens, & d'Ydewalle, 2007). For example, context and framing affect whether searches are conducted as confirming or disconfirming when selecting and evaluating information (Cosmides & Tooby, 2008; Fodor, 2000; Gigerenzer & Hug, 1992; Johnson-Laird, Byrne, & Girotto, 2009; Roberts et al., 2007; Wason & Johnson-Laird, 1970). Searches in the social system context are generally more disconfirming and this may be due to familiarity and/or variability. The system context can also affect confidence in the conclusions reached. For example, conclusions about medical conditions (natural system) may be viewed with less confidence than conclusions for technical systems (Politzer & Bourmaud, 2002). In addition, the saliency of the data affects the confidence viewers have in the data (McCabe & Castel, 2008). Thus, stimulus-driven salience and the context may affect the confidence that is placed in a decision reached after completion of an information search. Hypotheses The present study is motivated by a concern for the effects of increasing the stimulus-driven salience without matching that salience with optimal relevance. The hypotheses regarding search sufficiency, efficiency and accuracy relate to three concepts and related questions: 1. Congruence - What is the overall effect of the salience and relevance congruence conditions? 2. Salience display format - Is there a difference in the effect of 6 salience manipulated by size in a tag cloud and position in a list? How does do the salience and relevance congruence conditions affect each display type? 3. Context - How do the context and the congruence conditions act and interact to affect how information is searched? In addition, confidence in the decision is measured to assess how accurately people can self-assess the impact of salience and relevance congruence and context on search accuracy. Congruence Conditions Based on the reviewed literature, the first hypothesis was that the incongruent condition (i.e., high salience of low relevance items) impedes information search and the congruent condition (i.e., high salience of high relevance items) supports information search. The incongruent condition was predicted to have the following results: (1a) The search will be less sufficient in that the selection of all items necessary for a correct decision will occur less frequently. (1b) The search will be less efficient in that more unnecessary items (low relevance and distracter items) will be selected, and elapsed time will increase. (1c) Without a sufficient search, decision accuracy will be lower. Salience Display Format Two display formats were used to examine the effects of the congruence conditions when manipulating stimulus driven salience. Tag clouds were used to manipulate stimulus-driven salience by size and lists were used to manipulate stimulus-driven salience by position. The hypotheses related to these two display format were: (2a) Manipulating stimulus-driven salience by size (tag clouds) will result in searches that are less sufficient and efficient than manipulating salience by position (list). (2b) The differences between the 7 congruent and incongruent conditions will be greater in the tag cloud display format than the list display format. (2c) The display format will affect decision accuracy. Decisions will be more accurate in lists. Context The differences between contexts seen in prior studies relating to confirming and disconfirming search patterns were expected to be replicated. These differences in search patterns were expected to have the following effects: (3a) Because of the familiarity and variability of social systems, searches in the social context will be more disconfirming resulting in more sufficient and efficient searches with more accurate decisions than technical or natural systems. (3b) In contrast, the natural systems context will lead to more confirming searches that are the least sufficient and efficient, with fewer accurate decisions. (3c) Because searches are expected to be more sufficient and efficient in the social system context, the incongruent condition will affect search performance more negatively in the social system context than the natural system context. Confidence The analyses of the participants' confidence ratings were expected to show that: (4a) Confidence in the decision reached will be high with no correlation to the accuracy of the decision. (4b) The congruence condition will not affect confidence levels. (4c) The salience display format will not affect the confidence level. (4d) Due to inherent ambiguity, confidence levels will be lower for decisions in the natural system context than in the social or technical systems contexts. 8 Figure 1. Tagcloud 2010 State of the Union Address Reprint Courtesy of International Business Machines Corporation, © International Business Machines Corporation METHOD Participants Male and female participants from the University of Utah psychology undergraduate participant pool were recruited to participate in this study. Each of the 96 participants completed the tasks individually. The duration of each experiment was approximately one-half hour and participants received credit towards a psychology course requirement. Design Wason Selection Task We adapted the Wason Selection Task (WST) to define and measure the sufficiency, efficiency, and type of search performed. The WST uses propositional logic and has been used extensively for over 40 years to study information search behavior (Leighton, 1999; Wason & Johnson-Laird, 1970). The WST typically presents the participant with a rule in the form of if P then Q. Participants are asked to select from four cards with statements that represent the P, Q, not P and not Q values. When the cards are "turned over" they reveal a correlating P, Q, not P or not Q value that either supports the statement or proves the statement false. Participants are directed to select the fewest number of cards to prove the statement false. In propositional logic, a conditional if P then Q is false only if P is true and Q is false. Therefore, a logically correct (optimal) method of information search would be to 10 select only the P and not Q values. Since the value of not P is irrelevant and the value of Q either confirms the statement or is irrelevant, the values of not P and Q are not necessary for a correct decision. A search is sufficient when either one card (P or not Q value) proves the statement false or both the P or not Q cards are selected. An efficient search is one that includes only the P and not Q values. Using the WST, searches can be defined as confirming or disconfirming. Confirming searches include selection of P values which are necessary items and Q values with are not necessary for an accurate decision. Disconfirming searches include selection of P and not Q values which are both necessary items. Therefore, disconfirming searches are more likely to be sufficient and efficient. Overview Twelve vignettes were used (see Appendix A). There were four vignettes for each of the three system contexts. Each vignette contained background information and a hypothesis framed in the form of a statement. A list of items was displayed below the vignette (see Appendix B). Participants followed on-screen instructions to search and select items from this list to determine the veracity of the given statement. The study included four independent variables: 1. System context condition (natural, social or technical system), 2. congruence conditions (salience and relevance levels were congruent or incongruent) 3. statement veracity (statement was supported or falsified), and 4. salience display format (list or tag cloud). The independent variable salience display format was varied as a between subject factor. Each participant searched information that was displayed consistently per session in either a list or a tag cloud format. All other independent variables were varied as 11 within subject factors. Therefore, all participants viewed all levels of the other three independent variables. The order of presentation of all levels of the independent variables was counterbalanced across participants. The independent variables are defined in more detail below. System Contexts We considered three contexts: 1. Natural systems which for the purposes of this study are limited to diseases; 2. Social systems pertaining to human communities or interactions; 3. Technical systems pertaining to machines or man-made devices. Salience Display Format Stimulus-driven saliency is manipulated by position or size (visual angle) in one of two display formats: lists and tag clouds. Each display format includes 18 items (single words or a word group). When the items are presented as a list, saliency is manipulated by list order using early list position to reflect salience that is potentially driven by primacy effects. High salience is defined as being one of the first six items in the list. Low salience is defined as being one of the last six items in the list. In the list format the 18 items are arranged vertically in a consistent size and color. The corresponding tag cloud display format contains the same 18 items. However, the selectable items are displayed in alternating colors of green and blue to differentiate between the items. The stimulus-driven saliency of each item is manipulated by differences in the text size or visual angle of the item. Text is displayed in one of three sizes. The smallest text size is defined as low saliency with a visual angle of approximately 0.25° 12 (calculated using an average viewing distance of 22 inches). The largest text size is defined as high saliency and is 2.5 times larger with a visual angle of approximately 0.65°. Congruence Conditions The relevance of each of the 18 items is determined by the content of the vignette. Adapting the Wason task selection paradigm, there are three each of the P, Q, Not P and Not Q values. The P and not Q items are necessary and sufficient for determining an accurate decision and are therefore of high relevance. The not P and Q value items are not necessary for an accurate decision and are therefore of low relevance. Additionally, six items that could plausibly be associated with the given context are displayed. These items are not members of the set of P, Q, Not P or Not Q values and serve as distracters and are not relevant to determining the veracity of the statement. For each vignette there are two congruence conditions: 1. Congruent condition, where there is high salience for high relevance items and low salience for low relevance items. (The distracters are of medium saliency); and 2. incongruent condition, where there is high salience for low relevance items and low salience for high relevance items. (The distracters are again of medium saliency.) List. When presented as a list, the items were randomly but consistently ordered within their relevance group (blocked) for all list displays (i.e., the high relevance items always appear in the order Not Q, P, P, Not Q, P, Not Q in list displays for all vignettes). The high relevance items are the first six items in the congruent condition and the last six items in the incongruent condition. Conversely, the low relevance items are the first six items in the incongruent condition and the last six items in the congruent condition. The six 13 distracters are always positioned in the middle of the list between the high relevance and low relevance values. Tag cloud. When presented as a tag cloud, the position of the items was randomly but consistently ordered for all tag cloud displays (i.e., the P, Q, Not P, Not Q and distracter values of the 18 items are in the same position in all tag cloud displays for all vignettes). The size or visual angle of an item varies based on its relevance and the congruence condition. Specifically, in the congruent condition the items of high relevance are the largest size and the low relevance items are the smallest size. In the incongruent condition the items of high relevance are the smallest size and low relevance items are the largest size. The six distracters are always of medium size (visual angle of approximately 0.45° when viewing distance is 22 inches). Statement Veracity (Supported or False) Supported. When the statement is supported, selecting any of the P values displays a corresponding Q value. Selecting any of the not Q values displays a not P value. Therefore, all of the high relevance items support the statement. False. A false statement is defined by one of the three not Q values corresponding to a P value (which falsifies the statement). Selecting either of the other not Q values displays a not P value (which supports the statement). Selecting any of the P values displays a corresponding Q value (also supporting the statement). Therefore, only one of the high relevance values falsifies the statement. (The text displayed after selecting a low relevance or distracter item is not necessary to determine the statement veracity.) 14 Dependent Variables Five values were measured to evaluate the impact of the manipulation. These values are : (a) Items selected (P, Q, Not P, Not Q and distracters); (b) The sequence of selection; (c) Time to decision; (d) Decision - statement false or supported; (e) Confidence in the decision (scale of 1-10). These variables were used to determine the sufficiency, efficiency, decision accuracy, search pattern performed and confidence in the decision. Sufficiency. The sufficiency of the search considers the conditions of necessary and sufficient. For supported statements, selecting each of the six high relevance values (three P and three not Q values) is necessary and together they are sufficient for determining the veracity of the statements. For false statements, finding the single not Q value that corresponds to a P value is the only selection necessary for a sufficient search. Sufficiency was calculated and analyzed using a binary categorical measure, either the search was sufficient or it was not sufficient. Decision accuracy. Accuracy is defined as correctly determining whether the statement is supported or false. Accuracy is measured by comparing the participant's selection (decision) with the actual statement veracity. Efficiency. Efficiency is measured in three ways: (a) By considering the inefficiency of the search as a ratio of the total unnecessary items selected (Not P, Q and distracters) to total number of items selected (higher ratio is less efficient); (b) Total number of items selected, including duplicate selections; (c) Time efficiency defined by the time elapsed from vignette display to the time the participant made an assessment of the statement and identified the statement as supported or false. 15 Search pattern. The traditional Wason selection task uses four cards, one card corresponding to each value, P, not P, Q, not Q. A search is defined to be confirming if the P and the Q values are selected. A search is defined to be disconfirming if the participant selects the P and the not Q values. We included three items for each P, not P, Q, not Q value. Participants were not limited in the number of items selected or the number of times each item could be selected. To determine the search pattern, duplicate selections were removed and the sum of unique P and not Q values selected was divided by the total of unique selections to form a disconfirming ratio for each vignette per participant. A confirming ratio was calculated in a similar manner by summing the selection of unique P and Q values and dividing by the total of unique selections. Confidence. Participants rated their confidence in their decision on a 1-10 scale (10 being the most confident). Procedure Upon entry into the lab, participants were greeted and informed consent was obtained for this IRB approved study. As part of the consent process, participants were informed they could drop out of the study at any time with no penalty. Moreover, participants were also informed they had the option to not participate in this study but still receive research credit by reading and summarizing a related research article. Because the presentation of information required the detection of different colors of text, students were given an online color vision screening test (Waggoner). The students were allowed to continue the study regardless of the test results. However, based on the results, some students were not included as participants in this study. 16 Participants completed the task individually. The application introduced participants to the task, presented instructions and then all 12 vignettes were presented in either a tag cloud or list display format. The task consisted of completing the following steps for each of the 12 vignettes: • A vignette was presented on the computer screen to be read by the participant. • Participants selected from the 18 items to reveal additional information (P, Q, Not P, or Not Q values or data related to the distracter items). Participants were instructed to select the fewest number of items possible for a correct decision. However, participants were free to click on the items in any order with no limit to the number of items or the number of times an item could be selected. • Participants determined whether the statement was false or supported and recorded that decision by selecting one of two buttons ("False" or "Supported"). • A dialog appeared where participants rated their confidence in the decision using a scale of 1-10 (10 being the most confident). After completing the tasks the application presented a questionnaire to collect socio-demographic information. Information that would identify individuals was not collected or stored as part of the study. RESULTS Issues and Overview The analyses, unless otherwise stated, were performed using a general linear model repeated measures test with the between-subjects factor of salience display format (list or tag cloud) and the within-subject factors of system context, congruence condition and statement veracity. Table 1 summarizes all of the hypotheses and corresponding results. Decision accuracy ranged from 50% to 69% for 11 of the 12 vignettes. The notable exception was a vignette where 85% of participants correctly identified as "supported" the statement, "Children who are vaccinated will not develop mumps." Because there is no theoretical basis for the higher decision accuracy for this vignette, the vignette was excluded and decision accuracy was recalculated as a ratio of the number correct (see Table 2 and Table 3 to compare accuracy analyses with and without the mumps vignette). Both the number of items selected and time required for decision each only differed in one condition. Therefore, the complete results of those analyses are reported in Table 4 and Table 5. 18 Congruence Sufficiency of Search A nonparametric Friedman test was used to analyze how congruence affected sufficiency.1 The sufficiency measures for each vignette were aggregated within each congruence condition. Contrary to hypothesis 1a, search sufficiency did not differ between congruence conditions, χ 2(1, N = 96) = .73, p = .39 (congruent: M = 3.08 of 6 searches sufficient, SD =1.90; incongruent: M = 2.68 of 6 searches sufficient, SD =1.74). Efficiency: Inefficiency Ratio More unnecessary items were selected in the incongruent condition, (M=.55 inefficiency ratio, SE=.02) than the congruent condition (M=.41 inefficiency ratio, SE = .02) F(1, 94) = 48.87, p < .001, ηp 2 =.34. This finding supports hypothesis 1b. Efficiency: Number of Items Selected There was no difference in the total number of items selected between congruence conditions (see Table 4). Efficiency: Time There was no difference in time between congruence conditions (see Table 5). Overall, analyses of the measures of efficiency for hypothesis 1b revealed a mixed pattern: The inefficiency ratio indicates lower search performance in the incongruent 1 Some of the data in this study were not normally distributed (e.g., categorical measures such as sufficiency and accuracy). Therefore those data were analyzed using nonparametric tests. The Friedman test can be seen as a repeated measures analysis of variance for one group. 19 condition. However, no differences in search performance were found between congruence conditions in the number of items selected or the time to decision. Decision Accuracy Contrary to hypothesis 1c which predicts decision accuracy to be lower in the incongruent condition, a related samples Friedman test revealed no difference in accuracy between the congruent condition (M = .61, SD = .17) and the incongruent condition (M = .61, SD = .21), χ 2(1, N = 96) = .85 p = .41. Salience Display Formats Sufficiency To analyze the effect of the display format on sufficiency, measures for each vignette were first aggregated across all vignettes per participant then a nonparametric independent sample, grouped by display format (list or tag cloud) Kruskal-Wallis test was performed. Contrary to hypothesis 2a, no difference was found in search sufficiency between the list display format and the tag cloud display format. (See Table 6 for comparisons.) Contrary to hypothesis 2b, a related samples Friedman test did not show an effect of the congruence condition on search sufficiency in the tag cloud format. However, also contrary to hypothesis 2b the congruence condition may affect the search sufficiency of searches in the list display format.2 Specifically, searches were more sufficient in the list display format when salience and relevance were congruent than when salience and relevance were incongruent. 2 Using a Bonferroni correction for multiple comparisons the significance level would be .025 (.05/2) and this value would not be considered significant. 20 Efficiency: Inefficiency Ratio Contrary to hypothesis 2a, the analysis of the effect of the display format shows a trend for more inefficient searches when manipulating salience by position (list) than the display format where salience was manipulated by size (tag cloud), F(1,94) = 3.52, p = .06. (See Table 7 for comparisons.) The analysis revealed a significant interaction between the display format and the congruence condition, F(1,94) = 16.11, p = .001. While fewer unnecesary items were selected in the congruent condition for both tag clouds and list displays, contrary to hypothesis 2b, the effect of incongruence was numerically greater in lists. Post hoc analysis revealed the difference between congruence conditions to be significant for lists and potentially significant for tag clouds.3 Efficiency: Number of Items Selected Contrary to hypothesis 2a, participants selected significantly more items when the items were displayed in lists (M = 9.94 items, SE =.54) rather than tag clouds (M = 8.29 items, SE =.54), F(1, 94) = 4.70, p = .03, ηp 2 = .05. Also contrary to hypothesis 2b, there was no indication that the congruence conditions affected the number of items selected differently for the display formats (see Table 4). Efficiency: Time Also contrary to hypothesis 2a, there was no difference in time to decision between display formats (see Table 5). 3 Using a Bonferroni correction for multiple comparisons the significance level would be .025 (.05/2) and this value would not be considered significant. 21 Decision Accuracy To determine the effect of display format on decision accuracy, the accuracy for each search was aggregated as a sum across all vignettes per participant (see Table 8 for all comparisons). Contrary to hypothesis 2c, a nonparametric independent sample Kruskal- Wallis test showed no difference between display formats in decision accuracy. Aggregating within congruence conditions then performing a Friedman related samples test revealed that contrary to hypothesis 2b, there are no differences in accuracy between congruence conditions for the tag cloud or for the list formats. System Contexts Search Patterns: Disconfirming Ratio Consistent with prior research and hypothesis 3a, searches in the social context (M =.57 disconfirming ratio, SE = .02) were significantly more disconfirming than searches in natural contexts (M =.50 disconfirming ratio, SE = .02) or technical contexts (M =.47 disconfirming ratio, SE = .02), F(2, 188) = 16.52, p < .001, ηp 2 = .15. Pairwise comparisons revealed differences between the social and natural systems contexts (p < .001) and the social and technical systems contexts (p < .001). However, there was no difference in disconfirming search patterns between natural and technical systems contexts (p = .12). Search Patterns: Confirming Ratio In support of hypothesis 3b, the analysis of the confirming ratio revealed searches in the natural system context (M = .55 confirming ratio, SE = .02) to be more confirming than 22 searches in the social (M = .50 confirming ratio, SE = .02) or technical (M = .50 confirming ratio, SE = .02) contexts, F(2, 188) = 3.85, p = .02, ηp 2 =.04. Sufficiency A nonparametric Friedman test was used to analyze the differences of search sufficiency between contexts (see Table 9 for comparisons.). As predicted in hypothesis 3a the search context significantly affected the sufficiency of the search, χ 2 (2, N = 96) = 19.32, p <.001. More specific comparisons revealed that the sufficiency of searches in social systems was greater than the sufficiency of searches in either natural systems, χ 2 (1, N = 96) = 16.75, p <.001, or technical systems, χ 2 (1, N = 96) = 7.9, p <.01. However, while searches in natural systems were, as expected, the least sufficient these searches were not significantly less sufficient than searches in technical systems. Consistent with hypothesis 3c, Friedman tests revealed that the incongruent condition negatively affected search sufficiency in the social system context, χ 2(1, N = 96) = 6.48, p = .02, while there was no difference in search sufficiency between the congruence conditions in the natural system context. Search sufficiency in technical systems was also affected by the congruence condition4, χ 2(1, N = 96) = 4.60, p = .04. Efficiency: Inefficiency Ratio The context of the search significantly affected the number of unnecessary items selected, F(2,188) = 19.39, p <.001. (See Table 10 for comparisons) Consistent with hypothesis 3a, pairwise comparisons showed that fewer unnecessary items were selected in 4 Using a Bonferroni correction for multiple comparisons the significance level would be .0167 (.05/3) and this value would not be considered significant. 23 searches in the social system context than searches in the natural systems context or the technical system context. Consistent with disconfirming search patterns and sufficiency results, pairwise comparison showed there was no significant difference in the selection of unnecessary items between natural and technical systems contexts. Finally, there was an interaction between context and the congruence condition. More unnecessary were items selected in the incongruent condition for each context. Consistent with hypothesis 3c, the incongruent condition affected searches in the social system context, F (1,95) = 27.97, p <.001, ηp 2 = .23; however, the congruence condition also affected search efficiency in the natural system context, F (1,95) = 9.67, p <.01, ηp 2 = .09, and the technical system context, F (1,95) = 44.54, p <.001, ηp 2 = .32. (Analysis of the difference between the congruent and incongruent conditions was performed using post-hoc tests for each context.) Efficiency: Number of Items Selected Contrary to hypotheses 3a and 3b there was not a significant difference between contexts in the total number of items selected (see Table 4). While the difference between congruence conditions in the natural and technical systems context was not significant, the difference approached significance in the social system context, F (1, 94) = 3.33, p = .07. Fewer items were selected in the incongruent condition in the social system context (incongruent: M = 8.49; congruent: M = 9.32). Contrary to hypothesis 3c the incongruent condition did not negatively affect efficiency measured by the number of items selected in the social system context (see Table 4). 24 Efficiency: Time Contrary to the expectations of hypothesis 3a and 3b, efficiency measured by time for decision did not differ between contexts (see Table 4). The only condition with a significant difference in time was between the congruence conditions in the social systems context (8.14 second difference), F (1, 95) = 4.63, p = .03, with search times being less in the congruent condition (M = 72.54 seconds, SE = 3.96) compared to the incongruent condition (M = 80.68 seconds, SE = 4.89). Decision Accuracy The binary value of decision accuracy was aggregated within each context per participant. Consistent with the predictions of hypothesis 3a and 3b, a related samples Friedman test showed decisions in the social context were the most accurate while decisions in the natural system context were the least accurate χ 2 (2, N = 96) = 16.25, p <.001 (see Table 11 for comparisons). Decision accuracy in the technical system was comparable to the social system context. Contrary to hypothesis 3c, the incongruent condition did not impact decision accuracy in any of the system contexts. Confidence Accuracy After each decision, participants rated their confidence in their decision on a scale of 1 to 10 (10 being the most confident). As predicted by hypothesis 4a, the overall confidence levels were high (7.77) and the confidence interval of judgments was small (CI: 7.4 - 8.1). 25 Also as predicted by hypothesis 4a, a nonparametric two-tailed Spearman's rho correlation computed pairwise for decision accuracy and confidence for all 12 vignettes did not reveal any correlations between accuracy and confidence (see Table 12). Congruence Consistent with hypothesis 4b, the analysis showed no difference in confidence between the congruence conditions (see Table 8). Display Format Contrary to hypothesis 4c, participants were more confident in decisions made after searching items displayed in a tag cloud (see Table 13). While the congruence condition alone did not significantly affect confidence, there was an interaction between the congruence condition and the display format. When items were selected from a tag cloud confidence was higher in the congruent condition than the incongruent condition. However, when items were selected from a list there was a trend for confidence to be higher in the incongruent condition. System Context Contrary to hypothesis 4d confidence was not lower for decisions in the natural system context (see Table 13). There was no difference in confidence levels between any of the contexts. The difference between the highest confidence level and the lowest was only 0.24 points. The difference between congruence conditions in the technical system context 26 approached significance while there was no difference in confidence between congruence conditions for the natural and social system contexts. 27 Table 1 Hypotheses and Results Hypothesis Results Congruence (1a) Sufficiency - incongruent condition less sufficient than congruent condition Not supported No difference in search sufficiency between congruence conditions. (1b) Efficiency - incongruent condition less efficient Mixed results • More unnecessary items were selected in the incongruent condition. • No difference in total number of items selected. • No difference in overall time to reach a decision. (1c) Decision accuracy - incongruent condition less accurate Not Supported No difference in accuracy between congruence conditions Salience display format (2a) Tag cloud display less sufficient and efficient than list display Not supported Sufficiency No difference found in search sufficiency between the tag cloud and list display formats. Efficiency - Opposite effect • Trend for more unnecessary items selected in lists. • Overall more items were selected in the list format display than the tag cloud display • No difference in time (2b) The differences in sufficiency and efficiency between the congruent and incongruent conditions will be greater for tag clouds than lists Not supported Sufficiency The congruence conditions did not affect search sufficiency in the tag cloud display format. However searches were less sufficient in the incongruent condition for lists. Efficiency • More unnecessary items were selected in the incongruent condition for both tag clouds and lists. The numerical difference between congruence conditions was greater for lists than tag clouds. • The congruence conditions did not affect the total number of items selected for either tag clouds or list display formats • Time was not affected by congruence conditions and display formats. (2c) The display format will affect decision accuracy Not Supported • No difference in decision accuracy between display formats. • No difference in accuracy between congruence conditions for lists or tag cloud display formats 28 Table 1 Cont. Hypothesis Results Context (3a) Searches in the social context will be more disconfirming, more sufficient, more efficient and more accurate than searches in natural or technical systems contexts Disconfirming - Supported Searches in social systems context were more disconfirming Sufficient - Supported. Searches in social systems were more sufficient that searches in natural or technical systems contexts. Efficiency - Mixed • Fewer unnecessary items were selected in searches in the social system context. • Not a significant difference between contexts in the total number of items selected. • No difference between contexts in the time to decision. Decision Accuracy - Supported Decision accuracy was highest in the social system context. (3b) Searches in the natural systems context more confirming, least sufficient, least efficient and least accurate Confirming - Supported Searches in the natural systems context were more confirming than searches in social or technical systems contexts Sufficient - Mixed Searches in the natural systems context were less sufficient than searches in the social systems context, but not significantly less sufficient than searches in the technical systems context. Efficiency - Mixed • More unnecessary items were selected in searches in the natural system context than searches in the social system context, but not more than searches in the technical system context. • No difference between contexts in the number of items selected or time to decision. Accuracy - Supported Decision accuracy was lowest in the natural system context. 29 Table 1 Cont. Hypothesis Results 3c) Incongruent condition will affect search performance more negatively in social context than the natural system context Sufficient - Supported The incongruent condition negatively affected search sufficiency in the social and technical systems contexts but did not affect search sufficiency in the natural system. Efficiency - Mixed • The number of unnecessary items selected increased in the incongruent condition in the social system context more than in the natural system context. However, the efficiency of searches in the technical system context was the most negatively impacted by the incongruent condition. • The congruence conditions did not impact the total number of items selected more negatively in the social system context. In fact there was a trend for fewer items to be selected in the incongruent condition of the social system context. • The largest difference in elapsed time between congruence conditions was in the social system context. However, time to decision in the natural system context was greater in the congruent condition. Decision accuracy - Not supported No difference in accuracy between the congruence conditions in any of the system contexts. Confidence (4a) Confidence will be high with no correlation to decision accuracy Supported No correlations between accuracy and confidence for 11 of the vignettes. (4b) The congruence condition will not affect confidence levels Supported No difference in confidence between the congruence conditions. (4c) The salience display format will not affect the confidence level Not supported • More confident in decisions made after searching in a tag cloud. • Interaction with congruence condition. Confidence higher in the congruent condition in a tag cloud. Trend for higher confidence in the incongruent condition in a list. (4d) Confidence levels will be lower for decision in the natural system context than social or technical systems contexts Not supported No difference in confidence levels between contexts. 30 Table 2 Accuracy Comparisons With the Mumps Vignette Outlier Condition Mean Total |% SD χ 2 value p value Congruent (of 6) 3.64 | 65% 1.04 (1, N = 96) 5.07 .03 Incongruent 3.90 | 61% 1.08 Tag clouds (of 12) 7.77 | 65% 1.72 (1, N = 96) 1.34 .25 List 7.29 | 61% 1.66 Tag cloud Congruent Incongruent (of 6) 3.65 | 61% 4.13 | 69% 1.16 .94 (1, N = 48) 4.84 .04 List Congruent Incongruent (of 6) 3.63 | 61% 3.67 | 61% .91 1.17 (1, N = 48) 1.13 .38 Natural Social Technical (of 4) 2.46 | 62% 2.66 | 67% 2.42 | 61% .85 .94 1.01 (2, N =96) 3.30 .19 Social Congruent Incongruent (of 2) 1.33 | 67% 1.32 | 66% .61 .59 (1, N=96) .11 .87 Technical Congruent 1.25 | 63% .6 (1, N=96) 1.4 .31 Incongruent 1.17 | 59% .59 Natural Congruent 1.05 | 53% .57 (1, N=96) 15.51 <.001 Incongruent 1.41 | 71% .57 31 Table 3 Accuracy Comparisons Without the Mumps Vignette Outlier Condition Mean (% correct) SD χ 2 value p value Congruent .61 .17 (1, N = 96) .85 .41 Incongruent .61 .21 Tag clouds .63 1.66 (1, N = 96) 1.03 .31 List .59 1.65 Tag cloud Congruent Incongruent .61 .65 .19 .18 (1, N = 48) .53 .56 List Congruent Incongruent .60 .57 .15 .23 (1, N = 48) .33 .67 Natural .53 .27 (1, N = 96) 16.25 <.001 Social .66 .23 Technical .60 .25 Social Natural .66 .53 .23 .27 (1, N = 96) 10.89 .001 Social Technical .66 .60 .23 .25 (1, N = 96) 2.77 .12 Natural Technical .53 .60 .27 .25 (1, N = 96) 7.35 <.01 Social Congruent .67 .30 (1, N = 96) .11 .87 Incongruent .66 .29 Technical Congruent .63 .30 (1, N = 96) 1.4 .31 Incongruent .58 .30 Natural Congruent .53 .31 (1, N = 96) .053 .91 Incongruent .55 .50 32 Table 4 Number of Items Selected Condition Mean (items) SE F value p value Congruent Incongruent 9.12 9.11 .37 .47 (1,94) .002 .96 Tag clouds Lists 8.29 9.94 .54 (1,94) 4.70 .03 Display x Congruence (1,94) .057 .81 Tag cloud Congruent Incongruent List Congruent Incongruent 8.26 8.32 9.99 9.90 .52 .63 .52 .63 Natural Social Technical 9.30 8.91 9.13 .51 .46 .44 (2,188) .37 .70 Context x Congruence (2,188) 2.75 .07 Natural Congruent Incongruent 9.01 9.62 (1,94) 1.36 .25 Social Congruent Incongruent 9.32 8.49 (1,94) 3.33 .07 Technical Congruent Incongruent 9.04 9.22 (1,94) .19 .66 33 Table 5 Time to Decision Condition Mean (seconds) SE F value p value Congruent 77.07 3.01 (1,94) .45 .50 Incongruent 78.56 3.19 Tag clouds 80.51 5.38 (1,94) .87 .35 List 75.11 5.38 Natural 76.54 4.07 (2, 188) .49 .61 Social 76.61 4.03 Technical 80.29 3.33 Social Congruent 72.54 3.96 (1,95) 4.63 .03 Incongruent 80.68 4.89 Technical Congruent 79.66 3.62 (1,95) .14 .71 Incongruent 80.91 3.83 Natural Congruent 79.01 4.16 (1,95) 2.4 .12 Incongruent 74.07 4.57 Table 6 Sufficiency: Display Format Condition Mean Sufficiency Total |% SD χ 2 value p value Tag clouds (of 12) 5.52 | 46% 3.0 (1, N = 96) .59 .44 List 6.0 | 50% 3.68 Tag cloud Congruent Incongruent (of 6) 2.73 | 46% 2.79 | 47% 1.87 1.83 (1, N = 48) .86 .35 List Congruent Incongruent (of 6) 3.44 | 57% 2.56 | 43% 1.89 1.65 (1, N = 48) 3.79 .05 34 Table 7 Inefficiency Ratio: Display Format Format/Condition Mean Inefficiency Ratio SE F value p value Tag clouds .45 .02 (1, 94) 3.52 .06 List .51 .02 Display Format x Congruence Condition (1, 94) 16.11 <.001 Tag cloud Congruent Incongruent .42 .48 .02 .03 (1,47) 4.01 .05 List Congruent Incongruent .40 .62 .02 .03 (1,47) 67.66 <.001 Table 8 Accuracy: Display Format Format/Condition Mean (% correct) SD χ 2 value p value Tag clouds .63 1.66 (1, N = 96) 1.03 .31 List .59 1.65 Tag cloud Congruent Incongruent .61 .65 .19 .18 (1, N = 48) .53 .56 List Congruent Incongruent .60 .57 .15 .23 (1, N = 48) .33 .67 35 Table 9 Sufficiency: Context Context/Condition Mean Sufficiency SD χ 2 value p value Natural (of 4) 1.66 | 42% 1.28 (2, N = 96) 19.32 < .001 Social 2.25 | 56% 1.34 Technical 1.85 | 46% 1.26 Social Natural 2.25 | 56% 1.66 | 42% 1.34 1.28 (1, N = 96) 16.75 < .001 Social Technical 2.25 | 56% 1.85 | 46% 1.34 1.26 (1, N = 96) 7.9 < .01 Natural Technical 1.66 | 42% 1.85 | 46% 1.28 1.26 (1, N = 96) 2.12 .19 Natural Congruent (of 2) .82 | 41% .79 (1, N = 96) .09 .88 Incongruent .83 | 42% .78 Social Congruent (of 2) 1.23 | 62% .79 (1, N = 96) 6.48 .02 Incongruent 1.02 | 51% .78 Technical Congruent 1.03 | 52% .75 (1, N = 96) 4.60 .04 Incongruent .82 | 41% .74 36 Table 10 Inefficiency Ratio: Context Context/Condition Mean Inefficiency Ratio SE F value p value Natural .50 .02 (2, 188) 19.39 < .001 Social .42 .02 Technical .52 .02 Social Natural .42 .50 .02 .02 <.001 Social Technical .42 .52 .02 .02 <.001 Natural Technical .50 .52 .02 .52 .13 Context x Congruence Condition (2, 188) 5.78 < .01 Natural Congruent .45 .02 (1, 95) 9.67 <.01 Incongruent .54 .03 Social Congruent .35 .02 (1,95) 27.97 < .001 Incongruent .49 .02 Technical Congruent .43 .02 (1,95) 44.54 <.001 Incongruent .62 .02 37 Table 11 Accuracy: System Context Condition Mean (% correct) SD χ 2 value p value Natural 53 .27 (2, N = 96) 16.25 <001 Social 66 .23 Technical 60 .25 Social Natural 66 53 .23 .27 (1, N = 96) 10.89 .001 Social Technical 66 60 .23 .25 (1, N = 96) 2.77 .12 Natural Technical 53 60 .27 .25 (1, N = 96) 7.35 <.01 Social Congruent 67 .30 (1, N = 96) .11 .87 Incongruent 66 .29 Technical Congruent 63 .30 (1, N = 96) 1.4 .31 Incongruent 58 .30 Natural Congruent 53 .31 (1, N = 96) .053 .91 Incongruent 55 .50 Table 12 Confidence and Accuracy Correlations Vignette Confidence Accuracy Spearman's rho p N1CS 7.5 .52 -.11 .27 N2IS* 7.95 .85 .20 .06 N3CF 7.97 .53 .137 .18 N4IF 7.69 .55 .136 .19 S1CS 7.34 .65 -.04 .69 S2IS 7.7 .67 .16 .11 S3CF 8.3 .69 .04 .71 S4IF 8.3 .66 .10 .35 T1CS 7.38 .64 .14 .17 T2IS 7.5 .50 -.05 .66 T3CF 8.2 .61 .11 .29 T4IF 7.49 .67 .13 .21 *N2IS is the excluded mumps vignette 38 Table 13 Confidence Comparisons Condition/Context Mean Confidence SE F value p value Congruent 7.78 .17 (1, 94) .04 .85 Incongruent 7.76 .18 Tag clouds 8.11 .24 (1,94) .04 List 7.4 .24 4.02 Display Format x Congruence Condition (1, 94) 7.46 <.01 Tag cloud Congruent Incongruent 8.22 8.00 .25 .25 (1,47) 4.62 .04 List Congruent Incongruent 7.31 7.55 .25 .25 (1,47) 3.28 .08 Natural 7.78 .18 (2, 188) 1.68 .19 Social 7.89 .19 Technical 7.64 .19 Context x Congruence Condition (2, 188) 2.36 .10 Natural Congruent 7.74 .20 (1, 95) .23 .63 Incongruent 7.81 .20 Social Congruent 7.89 .21 (1, 95) 1.43 .23 Incongruent 7.97 .20 Technical Congruent 7.79 .20 (1, 95) 3.17 .08 Incongruent 7.50 .22 DISCUSSION This study addresses a concern for the effect of salience when searching for information and seeks to answer questions related to the concepts of salience/relevance congruence, display formats and system context. Congruence Increasing the salience without regard to the relevance of the items can impede the search for information. The results for the general effect of the congruence conditions were surprising in that the incongruent condition did not impact the total number of items selected. Although the number of items selected remained the same, more unnecessary items were selected in the incongruent condition. Increasing the salience of relevant items is clearly not the panacea for information search. While the selection of necessary items increased in the congruent condition, the increase was not enough to insure a sufficient and efficient search. Specifically, the search was insufficient since not all of the necessary items were selected. The search was also inefficient since many unnecessary items were selected. Furthermore, the time required for a decision and decision accuracy were the same whether or not salience and relevance were congruent. The lack of a difference in search sufficiency between the congruence conditions may be partially explained by the participants' suboptimal search strategies. The optimal search 40 strategy of the WST is to select an item if and only if the item is necessary to prove the statement false. However, it is not generally understood that P and not Q are the required values. Einhorn and Hogarth (1978) showed this lack of understanding; they presented statisticians, theoretically trained in examining disconfirming evidence, with a WST. Only 5 of the 23 statisticians performed a sufficient and efficient search by selecting both the P and the not Q values. If people trained in falsification did not perform an optimal search it should not be surprising that the search behavior of our participants also did not reflect an optimal search strategy. Attempting to support an optimal search by increasing salience of relevant items is not enough to change general search strategies. Salience Display Formats Generally there is no difference between tag clouds and lists in search sufficiency, time to decision or decision accuracy. However, there is a trend for the selection of fewer unnecessary items in a tag cloud. This trend may be due to fewer items overall being selected from a tag cloud. While there is a lack of main effect, there is a clear interaction between display formats and the congruence conditions. Tag Cloud: Salience by Size Even though tag clouds are thought of as a method to improve information search by providing structure to large amounts of information, our results show that while the use of tag clouds may be more efficient in the congruent condition,5 even the congruent condition fails to promote sufficient searches and accurate decisions. This lack of a difference between 5 Using a Bonferroni correction for multiple comparisons the significance level would be .025 (.05/2) and this value would not be considered significant. 41 congruence conditions in tag clouds seems to oppose the theory that preattentive processing determines the attentional capture regardless of top-down processing. Selection of only the highly salient items would have resulted in significant differences in sufficiency and efficiency between the congruence conditions. However, another interpretation is that the mechanisms that apply to preattentive processing may not apply to tag clouds because typically there are no singletons in tag clouds. In general, tag clouds have multiple items with the same characteristics with the consequence that no single item pops out of the stimulus array. This reduction of saliency is consistent with Theeuwes (2004) argument that a noisy stimulus field reduces the saliency of singletons. Therefore, attempting to increase salience by increasing the size of a group of items is not as effective as increasing the size of a single item. The use of tag clouds is expanding across the Web and increasing in general popularity (Sinclair & Cardew-Hall, 2008). This increased use may be explained in part by the higher-though misplaced-confidence in the decision reached after searching in a tag cloud compared to searching in a list. Lists: Salience by Position While a general comparison between lists and tag clouds failed to reveal a difference in search sufficiency, the position of high relevance items in a list may affect search sufficiency. Items at the top of the list are selected more frequently, thus more list searches are sufficient and more necessary items are selected in the congruent condition. However as mentioned, more items, including unnecessary items, are investigated in both congruence conditions of list presentations. 42 While we did not analyze the selection of items by position in a tag cloud, other research has shown that position also affects the selection of items in a tag cloud. Items in the center of tag clouds are selected more frequently (Bateman, Gutwin, & Nacenta, 2008) and are found more quickly (Halvey & Keane, 2007). Consistent with other studies comparing tag clouds and lists (Rivadeneira, Daniel, Michael, & David, 2007), we conclude that simple lists may be the best option when searching for specific items. System Contexts Increasing the salience of items relevant to an optimal search is clearly not the complete solution for improving search performance. Other factors must be considered, including the context of the search. Search strategies change as a result of the context, thus affecting what items are selected. The findings of this study support and expand previous work on the differences between system contexts. Search patterns are unique for all three contexts considered in this study. The system contexts differ by inducing a range of search patterns with varying degrees of confirming and disconfirming selections. Consistent with prior research, search patterns in the social system context are the most disconfirming, while search patterns in the natural system context are the most confirming. The search patterns in the technical system context share characteristics of both the social and natural system contexts. Specifically, the search patterns in the technical system context are similar to the disconfirming pattern of the natural system context and the confirming search pattern of the social system context. Search sufficiency and efficiency, and even decision accuracy, differ between system contexts. A disconfirming search pattern supports the optimal search strategy as defined in 43 the WST (the selection of P and not Q values). Therefore, it is not surprising that the disconfirming search pattern of the social system context produces the most sufficient and efficient search overall. What is surprising is that the congruence conditions affect search sufficiency and efficiency differently within each system context. The incongruent condition negatively affects search sufficiency in the technical and social systems contexts but does not affect search sufficiency in the natural system context. A potential explanation is that there is a floor effect in the natural system context since search sufficiency is already low because a confirming search strategy does not include the selection of not Q items. Conversely, in the social and technical systems context a disconfirming search strategy includes the selection of Not Q values. In the incongruent condition the selection of P and not Q values was impeded by the high salience of unnecessary items and the low salience of necessary items. The negative effect on the technical and social system contexts again reflects that incongruence of salience and relevance impedes an optimal search. However, the suboptimal search strategy induced by the natural system context could not be overcome by congruent salience and relevance. Implications and Conclusions It is important to recognize that the congruence of salience and relevance can interact to support or impede information search. However, salience and relevance are clearly not independent factors. How the salience is manipulated (e.g., size and position) also affects the sufficiency and efficiency of the search. The impact of the salience clearly depends on the context of the search. APPENDIX A VIGNETTES Vignette Information • Four vignettes for each system • Each vignette contained a scenario with a rule (see rules below). • Salience and relevance conditions and statement veracity were consistent for all three systems and are outlined below: o Vignette 1 was supported and congruent o Vignette 2 was supported and incongruent o Vignette 3 was false and congruent o Vignette 4 was false and incongruent Natural System Rules Vignette 1 If P (Has H1N1) then Q (Test positive) "Patients with the novel H1N1 influenza infection have a positive swab test." Vignette 2 If P (vaccinated) then Q (does not develop mumps) "Children who are vaccinated will not develop mumps." Vignette 3 If P (ill with shigella) then Q (ate at Bob's Burger) "All persons who became ill with shigellosis ate at Bob's Burger." Vignette 4 Adapted from Cosmides & Tooby (2008) If P (has Ebbinghaus disease) then Q (is forgetful) "When a person has Ebbinghaus disease that person will be forgetful." 45 Social System Rules Vignette 1 If P (make-up test) then Q (student was ill) "Students took the make-up test because they were ill." Vignette 2 If P (blue box) then Q (must weigh less than 5 pounds) "A blue ‘5 for 5 box' weighs 5 pounds or less." Vignette 3 Adapted from Cheng and Holyoak (1985) If P (At pool) then Q (tenant of complex) "All individuals at the swimming pool are tenants." Vignette 4 If P (at the mall unsupervised) then Q (Must be at least 14 years old) "Unsupervised children at the mall are at least 14 years old." Technical System Rules Vignette 1 If P (oil temp sensor malfunctions) then Q (engine overheats) "When the oil temperature sensor malfunctions the engine overheats." Vignette 2 If P (car in garage) then Q (false alarm) "When the car is in the garage there will be a false alarm." Vignette 3 If P (wind speeds >20 mph) then Q (fountain show cancelled) "When the wind speed is over 20 mph the fountain show is cancelled." Vignette 4 If P (MP3 player on) then Q (battery won't charge) "When the MP3 player is on the battery won't charge." APPENDIX B APPLICATION SCREEN SHOTS 47 Tag Cloud Display You are the m.nager of.n .p.rtment comr"ex. The .men~ i es for the comr"ex include voll eyb. 1I . nd tenn is courts wh ich . re """ il.ble to ten. nt s . nd guests . • nd • swimming pool th . t is """ il.ble fo r ten. nt s only The li st below represents inform.tion on 18 indMdu. ls Each item represent s one indMdu. 1 . nd is preceded by th eir initi. ls Clicking on each item wi ll pr""de one p<ece of add ition. 1 informat ion (e g th eir locat ion . t the comr"ex; wheth er or not th ey . re. ten. ntl. Click on items to se<> if . ny of these cases pr""" th is st. tement f. lse -All individuals allhe swimming pool a re tenants.­Select only the items req uired to determ ine wheth er th e st. tement is f. lse or suWOrted "'""'"' CD: Not Tenant DJ: Parked in Lot ES Volleyball Court IR: Parked in Lot J M: Not Tenant JR: Swimming Pool "' '"""''''"" MD: Swimming Pool "' '""'"' PA: Student RG: Swimming Pool SI: Student m '""'"' UY: Not Tenant WB Volleyball Court WZ: Parked in Lot YR: Student False II s""","oo I 48 List Display O\J a re the manage< of an a partme nt com~ex. The ame nities for the com~ex include "'lIeyball and tenn is courts which are """il al>e to ten ants and gue sts, and a swimmiog pool that is """ila l>e for teoarns ooly e list below reI"eseots information on 18 individua ls Each item reI"eseots ooe individu al and is p<eceded by their initi als. Clicking on each Jlem will provide one pi""e of addjlion ~ 1 inlo",uolion (e.9 thei, location", the ~omplex ; whethe, '" nolthey ,,'" " ten ~ nI ) . Click on items to see if any of the se cases 1"""'" this statement false: -All individuals a t the s winwning pool are tenants_­Select only the items required to determine whethe< the statement is false Of supported BC: Not Tenant CD: Swimming Pool OJ: Swimming Pool ES: Not Tenant IR: Swimming Pool JM: Not Tenant JR: Parked in l ot KW: Parked in lot MD: Student Nl: Parked in l ot PA Student RG: Student SI: Volleyball Court TO: Tenant UY: Tenant WB: Tennis Court WZ: Tenant YR: Volleyball Court False I [ S"pported REFERENCES Bacon, W. F., & Egeth, H. E. (1994). Overriding stimulus-driven attentional capture. Perception & Psychophysics, 55(5), 485-496. Bateman, S., Gutwin, C., & Nacenta, M. (2008). Seeing things in the clouds: The effect of visual features on tag cloud selections. Paper presented at the 19th ACM conference on Hypertext and Hypermedia. Beymer, D., Russell, D. M., & Orton, P. Z. (2007, July). An eye tracking study of how font size, font type, and pictures influence online reading. Paper presented at the INTERACT 2007. Retrieved February 3, 2010, from dmrussell.googlepages.com/FontSizeTypePictures-long-Beymer-Rus.pdf Biggs, A. T., & Gibson, B. S. (2010). Competition between color salience and perceptual load during visual selection can be biased by top-down set. Attention, Perception & Psychophysics, 72(1), 53-64. Brown, F. R., & Lovejoy, W. L. (1955). Current developments in improving cockpit informational presentations. Journal of Aviation Medicine, 26, 413-416. Brown, P. E. (1991). The relationship between graphic aids in a business report and decision-making and cognitive style of a report reader. ProQuest Information & Learning, US. Carey, J. M., & White, E. M. (1991). The effects of graphical versus numerical response on the accuracy of graph-based forecasts. Journal of Management, 17(1), 77-96. Carter, L. F. (1947). An experiment on the design of tables and graphs used for presenting numerical data. Journal of Applied Psychology, 31(6), 640-650. Cheng, P. W., & Holyoak, K. J. (1985). Pragmatic reasoning schemas. Cognitive Psychology, 17, 391-416. Cosmides, L., & Tooby, J. (2008). Can a general deontic logic capture the facts of human moral reasoning? How the mind interprets social exchange rules and detects cheaters. In Moral psychology, Vol 1: The evolution of morality: Adaptations and innateness (pp. 53-119). Cambridge, MA US: MIT Press. 50 Cosmides, L., Tooby, J., & Gazzaniga, M. S. (2004). Social exchange: The evolutionary design of a neurocognitive system. In The cognitive neurosciences (3rd ed.). (pp. 1295-1308). Cambridge, MA US: MIT Press. Dawson, E., Gilovich, T., & Regan, D. T. (2002). Motivated reasoning and performance on the Wason selection task. Personality and Social Psychology Bulletin, 28(10), 1379- 1387. Durso, F., & Drews, F. A. (2010). Healthcare, aviation, and ecosystems: A socio-natural systems perspective. Current Directions in Psychological Sciences, 19, 71-75. Einhorn, H. J., & Hogarth, R. M. (1978). Confidence in judgment: Persistence of the illusion of validity. Psychological Review, 85(5), 395-416. Fishbein, H. D., Haygood, R. C., & Frieson, D. (1970). Relevant and irrelevant saliency in concept learning. American Journal of Psychology, 83(4), 544-553. Fodor, J. (2000). Why we are so good at catching cheaters. Cognition, 75(1), 29-32. Folk, C. L., Remington, R. W., & Johnston, J. C. (1993). Contingent attentional capture: A reply to Yantis (1993). Journal of Experimental Psychology: Human Perception and Performance, 19(3), 682-685. George, C. (1991). Facilitation in the Wason selection task with a consequent referring to an unsatisfactory outcome. British Journal of Psychology, 82(4), 463-472. Gibson, B. S., & Kelsey, E. M. (1998). Stimulus-driven attentional capture is contingent on attentional set for displaywide visual features. Journal of Experimental Psychology: Human Perception and Performance, 24(3), 699-706. Gigerenzer, G., & Hug, K. (1992). Domain-specific reasoning: Social contracts, cheating, and perspective change. Cognition, 43(2), 127-171. Halvey, M., & Keane, M. T. (2007). An assessment of tag presentation techniques. Paper presented at the WWW 2007 Banff, Alberta, Canada. Huang, L., & Pashler, H. (2005). Quantifying object salience by equating distractor effects. Vision Research, 45(14), 1909-1920. Itti, L. (2007). Visual salience. 2(9):3327, revision #58499. from http://www.scholarpedia.org/article/Visual_salience Itti, L., & Koch, C. (2001). Computational modelling of visual attention. Nature Reviews Neuroscience, 2(3), 194-203. Johnson-Laird, P. N., Byrne, R. M. J., & Girotto, V. (2009). The mental model theory of conditionals: A reply to Guy Politzer. Topoi, 28, 75-80. 51 Kleinmuntz, D. N., & Schkade, D. A. (1993). Information displays and decision processes. Psychological Science, 4(4), 221-227. Lamy, D., Tsal, Y., & Egeth, H. E. (2003). Does a salient distractor capture attention early in processing? Psychonomic Bulletin & Review, 10(3), 621-629. Leighton, J. P. (1999). An alternate approach to understanding formal reasoning: Thinking according to the inductive-coherence model. Ph.D. dissertation, University of Alberta, Canada (Publication. Retrieved October 6, 2009, from Dissertations & Theses: http://proquest.umi.com.tproxy01.lib.utah.edu/pqdweb?did=784320951&sid=6&Fmt =2&clientId=9456&RQT=309&VName=PQD McCabe, D. P., & Castel, A. D. (2008). Seeing is believing: The effect of brain images on judgments of scientific reasoning. Cognition, 107(1), 343-352 MSCHW. (2010, January 28). Visualizations : President Barack Obama, State of the Union, January 27, 2010: Retrieved from http://manyeyes.alphaworks.ibm.com/manyeyes/visualizations/president-barack-obama- state-of-th. Nothdurft, H.-C. (1993). Saliency effects across dimensions in visual search. Vision Research, 33(5-6), 839-844. Parkhurst, D., Law, K., & Niebur, E. (2002). Modeling the role of salience in the allocation of overt visual attention. Vision Research, 42(1), 107-123. Pirolli, P., & Card, S. K. (1999). Information foraging. Psychological Review, 106, 643-675. Politzer, G., & Bourmaud, G. (2002). Deductive reasoning from uncertain conditionals. British Journal of Psychology, 93(3), 345-381. Posner, M. I. (1980). Orienting of attention. The Quarterly Journal of Experimental Psychology, 32(1), 3-25. Richard, C. M., Wright, R. D., & Ward, L. M. (2003). Goal-driven modulation of stimulus-driven attentional capture in multiple-cue displays. Perception & Psychophysics, 65(6), 939-955. Rivadeneira, A. W., Daniel, M. G., Michael, J. M., & David, R. M. (2007). Getting our head in the clouds: Toward evaluation studies of tagclouds. Paper presented at the Proceedings of the SIGCHI conference on Human factors in computing systems. Roberts, M. J., Schaeken, W., Vandierendonck, A., Schroyens, W., & d'Ydewalle, G. r. (2007). Mental models and falsification: It depends on the task. In The mental models theory of reasoning: Refinements and extensions. (pp. 85-113). Mahwah, NJ US: Lawrence Erlbaum Associates Publishers. 52 Sanfey, A., & Hastie, R. (1998). Does evidence presentation format affect judgment? An experimental evaluation of displays of data for judgments. Psychological Science, 9(2), 99-103. Schaubroeck, J., & Muralidhar, K. (1991). A meta-analysis of the relative effects of tabular and graphic display formats on decision-making performance. Human Performance, 4(2), 127-145. Sinclair, J., & Cardew-Hall, M. (2008). The folksonomy tag cloud: When is it useful? Journal of Information Science, 6(1), 15-23. Smith, L. D., Best, L. A., Stubbs, D. A., Archibald, A. B., & Roberson-Nay, R. (2002). Constructing knowledge: The role of graphs and tables in hard and soft psychology. American Psychologist, 57(10), 749-761. Theeuwes, J. (2004). Top-down search strategies cannot override attentional capture. Psychonomic Bulletin & Review, 11(1), 65-70. Treisman, A. M., & Gelade, G. (1980). A feature-integration theory of attention. Cognitive Psychology, 12(1), 97-136. Treisman, A. M., Gelade, G., & Yantis, S. (2000). A feature-integration theory of attention. In Visual perception: Essential readings. (pp. 347-358). New York, NY US: Psychology Press. Tufte, E. R. (2000). Visual explanations (4th ed.). Cheshire, CT: Graphics Press. Waggoner, T. L. PseudoIsochromatic Plate Ishihara Compatible (PIP) Color Vision Test 24 Plate Edition. Retrieved January 28, 2010, from http://colorvisiontesting.com/ishihara.htm Wainer, H. (2005). Graphic discovery: A trout in the milk and other visual adventures Princeton, NJ: Princeton University Press. Wason, P. C., & Johnson-Laird, P. N. (1970). A conflict between selecting and evaluating information in an inferential task. British Journal of Psychology, 61(4). Wray, R. (2009). Internet data heads for 500bn gigabytes. Monday, 18 May 2009. Retrieved 11/23/2009, 2009, from http://www.guardian.co.uk/business/2009/may/18/digital-content- expansion Yantis, S. (1993). Stimulus-driven attentional capture. Current Directions in Psychological Science, 2(5), 156-161.