Supporting scientific reasoning about real-world phenomena

The current study examined the impacts of concreteness fading and explanation activity on learning from visualizations of real-world physics experiments. Results indicated that concreteness fading may make the relationships between concrete and symbolic representation elements more salient, yielding improvements in representation integration and in the quality of students' scientific explanations. However, it did not improve the accuracy or depth of students' comprehension of domain principles. Meanwhile, encouraging more active explanation activities during learning did not provide students with observable learning benefits in the current research. Compared to building or selecting explanations during learning, viewing provided explanations was equally effective for learning and was the only explanation activity that did not show a decrease in domain enjoyment following the intervention.

SUPPORTING SCIENTIFIC REASONING ABOUT REAL-WORLD PHENOMENA by Sarah Davies A dissertation submitted to the faculty of The University of Utah in partial fulfillment of the requirements for the degree of Doctor of Philosophy Department of Educational Psychology The University of Utah May 2019 Copyright © Sarah Davies 2019 All Rights Reserved The University of Utah Graduate School STATEMENT OF DISSERTATION APPROVAL The dissertation of Sarah Davies has been approved by the following supervisory committee members: , Chair Kirsten Renee Butcher May 10, 2018 Date Approved , Member Anne E. Cook May 10, 2018 Date Approved , Member Dan J. Woltz May 10, 2018 Date Approved , Member Eric Gilbert Poitras May 10, 2018 Date Approved , Member David E. Johnson May 10, 2018 Date Approved and by the Department/College/School of , Chair/Dean of Anne E. Cook Educational Psychology and by David B. Kieda, Dean of The Graduate School. ABSTRACT The current study examined the impacts of concreteness fading and explanation activity on learning from visualizations of real-world physics experiments. Results indicated that concreteness fading may make the relationships between concrete and symbolic representation elements more salient, yielding improvements in representation integration and in the quality of students’ scientific explanations. However, it did not improve the accuracy or depth of students’ comprehension of domain principles. Meanwhile, encouraging more active explanation activities during learning did not provide students with observable learning benefits in the current research. Compared to building or selecting explanations during learning, viewing provided explanations was equally effective for learning and was the only explanation activity that did not show a decrease in domain enjoyment following the intervention. TABLE OF CONTENTS ABSTRACT ....................................................................................................................... iii LIST OF TABLES ............................................................................................................. vi LIST OF FIGURES ......................................................................................................... viii Chapters 1: INTRODUCTION .......................................................................................................... 1 1.1 Learning With Visual Representations ................................................................ 3 1.1.1 Learning With Concrete Visualizations ................................................... 7 1.2 Producing Explanations During Learning ......................................................... 13 1.2.1 Self-Explanation .................................................................................... 14 1.2.2 Computer-Based “Self-Explanation” ..................................................... 16 1.2.3 Generating Scientific Explanations........................................................ 20 1.3 The Current Experiment .................................................................................... 23 1.3.1 Research Questions and Hypotheses ..................................................... 24 2: METHODS ................................................................................................................... 27 2.1 Design ................................................................................................................ 27 2.2 Participants ......................................................................................................... 27 2.3 Materials ............................................................................................................ 28 2.3.1 Digital Learning Environment ............................................................... 28 2.3.2 Knowledge Assessments ........................................................................ 33 2.3.3 Perceptions of Physics Learning ............................................................ 40 2.4 Procedure ........................................................................................................... 41 2.4.1 Experimental Protocol ........................................................................... 41 2.4.2 Collection of Data From the Digital Learning Environment ................. 43 3: RESULTS ..................................................................................................................... 55 3.1 Instructor and Lab Effects .................................................................................. 55 3.2 Learning Outcomes ............................................................................................ 57 3.2.1 Pre- to Posttest Overall Knowledge Gain .............................................. 58 3.2.2 Hypothesis 1........................................................................................... 58 3.2.3 Hypothesis 2........................................................................................... 60 3.2.4 Exploratory Analyses: Interactions ........................................................ 62 3.3 Post Hoc Exploration ......................................................................................... 62 3.3.1 Behaviors During Learning.................................................................... 62 3.3.2 Perceptions of Physics Learning ............................................................ 68 4: DISCUSSION ............................................................................................................... 77 4.1 How Does Concreteness Fading Impact Learning? ........................................... 77 4.1.1 Representation Integration ..................................................................... 78 4.1.2 Explanation Quality ............................................................................... 81 4.1.3 Reported Reliance on Real-World Examples ........................................ 83 4.2 How Does Explanation Activity Impact Learning? ........................................... 84 4.2.1 Knowledge Assessments ........................................................................ 84 4.2.2 Behaviors During Learning.................................................................... 86 4.2.3 Perceptions of Physics Learning ............................................................ 91 4.3 Does the Combination of Concreteness Fading and Explanation Activity Impact Learning? ................................................................................................................. 94 4.4 Limitations ......................................................................................................... 95 4.5 Future Research Directions ................................................................................ 97 4.6 Conclusions ...................................................................................................... 100 APPENDIX ..................................................................................................................... 102 REFERENCES ............................................................................................................... 108 v LIST OF TABLES Tables 2.1 Rubric for scoring explanation relevance and depth of reasoning .............................. 54 2.2 Intraclass correlation coefficients (ICCs) for scoring explanation accuracy and completeness ..................................................................................................................... 54 3.1 Number of students by instructor per condition ......................................................... 73 3.2 Means (and standard deviations) for pre- and posttest overall scores ........................ 73 3.3 Means (and standard deviations) for posttest representation integration scores......... 74 3.4 Means (and standard deviations) for explanation quality ........................................... 74 3.5 Percentage means (and standard deviations) for submission accuracy (during learning) ............................................................................................................................ 75 3.6 Means (and standard deviations) for times on task (in seconds) ................................ 75 3.7 Means (and standard deviations) for video interaction measures ............................... 75 3.8 Means (and standard deviations) for video interactions by diagram explanation difficulty............................................................................................................................ 76 3.9 Means (and standard deviations) for user experience and metacognitive selfassessments ....................................................................................................................... 76 A.1 Skewness and kurtosis values of nonnormally distributed variables before and after square root transformation .............................................................................................. 103 A.2 Means (and standard deviations) for easy and difficult measures of concrete and symbolic translation ........................................................................................................ 103 A.3 Correlations of dependent variables from RM-MANOVA for pre- to posttest overall knowledge measures ....................................................................................................... 104 A.4 Correlations of dependent variables from MANOVA for posttest representation integration ....................................................................................................................... 105 A.5 Correlations of dependent variables from MANOVA for posttest explanation quality ............................................................................................................................. 105 A.6 Correlation of dependent variables from MANOVA for video study time on task . 106 A.7 Correlations of dependent variables from MANOVA for video interactions.......... 106 A.8 Correlations of dependent variables from MANOVA for video interactions by explanation difficulty ...................................................................................................... 107 vii LIST OF FIGURES Figures 1.1: An example of a free body diagram. The arrows represent the directionality of forces, with length indicative of magnitude. In this diagram, the upward and downward arrows are of equal magnitude and sum to 0; the magnitude of the right-facing arrow is subtracted from the left-facing arrow, yielding a net force (and thus, acceleration) in the leftward direction .............................................................................................................. 26 2.1: Integrated concrete and symbolic visual representations. A series of video frames shows a hands-on physics experiment with an overlay of red arrows depicting real-time forces. The visualization is dynamic, updating over time ................................................ 46 2.2: Video frames from three versions of the same scenario: an object falling to the ground. Each version consists of the same underlying symbolic structure but varies in concreteness. The left and middle frames vary in superficial concrete detail; the right frame contains no concrete imagery at all ........................................................................ 46 2.3: The view explanation interface. When “View Explanation” button is clicked, the correct answer is revealed ................................................................................................. 46 2.4: A system-provided correct explanation in the view explanation condition............... 47 2.5: Selecting an explanation with the select explanation interface, showing the initial view (top frame), selection choices (middle frame), and submission view (bottom frame) ................................................................................................................................ 47 2.6: System-provided feedback following a correct explanation submission (for both the select and build explanation interfaces) ............................................................................ 48 2.7: System-provided feedback following an incorrect explanation submission (for both the select and build conditions)......................................................................................... 48 2.8: Explanation interface for the build explanation condition, showing initial view (top frame), component selection (middle frame), and submission view (bottom frame) ....... 49 2.9: A video viewing page. A caption at top left describes the video content. The “Play Video” button is clicked to remove the mask and play the video ..................................... 50 2.10: A diagram explanation page. The diagram to be explained is pictured at top left. To the right of the diagram is a masked video, which can be activated and explored using the video controls. A the bottom of the screen is the explanation interface. A timer at top right counts down .............................................................................................................. 51 2.11: A declarative knowledge item ................................................................................. 51 2.12: An application item .................................................................................................. 52 2.13: A difficult concrete translation item: symbolic elements in the correct answer (d) do not clearly align with information about the direction of motion ..................................... 52 2.14: A symbolic translation item ..................................................................................... 53 2.15: A written explanation item ...................................................................................... 53 3.1: Accuracy of the causal elements component of participants’ submitted explanations for the select and build conditions during part 1 and part 2 of the diagram explanation activity............................................................................................................................... 72 3.2: Accuracy of the causal interaction component of participants’ submitted explanations for the select and build conditions during part 1 and part 2 of the diagram explanation activity............................................................................................................................... 72 ix CHAPTER 1 INTRODUCTION Physics seeks to explain the nature of phenomena encountered in the material world. Although physics principles underlie observable phenomena encountered in real world situations, novice learners do not find physics principles intuitive or easy to comprehend. This is due in part to the challenge of making accurate physics observations in the real-world; in real situations, it is easy to conflate outcomes and causal mechanisms. For example, novice learners observing an object in motion might infer that the presence of motion indicates the presence of force. This leads to the erroneous conclusion that an object will stop moving when it “runs out” of force (American Association for the Advancement of Science, 2017). In actuality, forces influence and change motion are not necessary to sustain it; once set in motion, an object can continue to move without being propelled by force. Understanding the nature of the relationship between force and motion requires knowledge of underlying principles that are not visibly obvious. In order to illustrate the underlying principles driving observable phenomena in the world, visual representations are used in physics to depict the invisible forces at play in real-world situations. One common example is the free-body diagram. Free-body diagrams are abstract representations that simplify a real-world phenomenon, reducing it 2 to two components: an object, and the forces affecting that object’s state of motion (see Figure 1.1). Free-body diagrams are idealized representations in that they portray the target object, such as an automobile, as a simple shape (e.g., a square). Arrows connected to the shape are used to represent the magnitude and direction of the various forces acting upon that object. The relationship between depicted forces gives information about resulting changes in the object’s motion. When forces that are opposite in direction (e.g., left vs. right) are not equal in magnitude, one is subtracted from the other to yield a net force. A nonzero net force indicates that an object is accelerating. In other words, its velocity is changing. Opposing forces that are equal in magnitude cancel each other out, yielding a net force of zero. When net force is zero, an object may be still or moving (e.g., at a constant speed), but it is not accelerating. Free-body diagrams attempt to make causal mechanisms salient and easily understood by simplifying the information portrayed. Yet, research suggests that even after learning with free-body diagrams, many physics students develop only a shallow, compartmentalized understanding of the basic principles of force and motion (Brown, 1989; Kim & Pak, 2002; Savinainen, Scott, & Viiri, 2005). When explaining real-world phenomena, students often fall back on superficial explanations that fail to move them beyond intuitive misconceptions. Thus, a key challenge in physics education is understanding how and when visual representations can support meaningful understanding and integration of abstract ideas related to real-world phenomena. 3 1.1 Learning With Visual Representations Visual representations include a variety of media such as illustrations, photographs, physical objects, videos, and symbolic notations. Such representations frequently are used to communicate scientific ideas (Mathewson, 1999). When presented in addition to textual information, visual representations often result in improved comprehension; this is widely known as the multimedia effect (Clark & Mayer, 2016). Much research has examined the mechanism underlying the multimedia effect. Some research suggests that the combination of verbal and visual information supports learning by taking advantage of dual channels for information processing (Paivio, 1990). Other research suggests that visual representations support learning by making explicit relationships that are not always clear in writing (Larkin & Simon, 1987). Other studies have focused on the support of multimedia materials for effective cognitive processing. Ainsworth and Loizou (2003) found that learning with a complementary combination of text and pictures elicited a self-explanation effect: students engaged in more frequent self-explanation and subsequently evidenced greater comprehension of the material. Butcher (2006) found that students who learned with diagrams in addition to text were more likely to engage in higher-level cognitive processes (e.g., inference generation) than students who learned with text alone. Taken together, these studies suggest that the multimedia effect may involve both enhanced processing of content as well as differential encoding. Although the multimedia effect is fairly robust and frequently observed in the research literature (Clark & Mayer, 2016; Mayer, 2002), not all visual representations are equally effective for learning. If imagery distorts or interferes with the construction of an 4 appropriate mental model, it impedes learning. Using a geography learning task, Schnotz and Bannert (2003) found that visual representations only aided comprehension inasmuch as the structure of those representations was well-aligned with the goals of the task. In their study, students performed better on a circumnavigation problem-solving task when using a circle diagram especially designed for that task than when using an informationally-equivalent but less well-suited carpet diagram. Furthermore, structural misalignment between visual imagery and learning goals actually harmed performance more than offering no visual images during learning. Thus, care must be taken to ensure that visual representations are well-aligned with a given learning task. Even visual representations that are accurate and appropriate to the learning goals can present a sense-making challenge. Visualizations that are easily understood by domain experts often are difficult for novices to comprehend because novices lack the foundational knowledge and practiced skill needed to identify the most relevant features. In a comparison of expert and novice reasoning about physics problems, Chi, Feltovich, and Glaser (1981) found significant differences in how the two groups categorized physics problems. Given the same information, novices categorized problems according to their superficial and easily observed components (e.g., real-world objects like inclined planes), while experts categorized problems by the underlying physics law or principle needed to solve a given problem (e.g., the Law of Conservation of Energy). This distinction is important because problem categorization affects how a student will approach the problem and the solutions that they will consider. In order for novices to make sense of visual information appropriately, they likely need support in determining which features or components are the most conceptually relevant. 5 Simply altering learners’ locus of attention can have a significant effect on how students approach problem-solving. For example, Grant and Spivey (2003) examined the eye movements of students tasked with solving a visual problem (Duncker’s radiation problem). They found that successful problem-solving was related to greater attention to specific, conceptually-relevant elements in the picture. Yet, the majority of participants were unable to identify the most important element spontaneously (as indicated by the proportion of gaze time spent looking at that element). Instead, they tended to focus on the most prominent, central element of the diagram and the space around it. The authors utilized visual cues (via a highlighting animation) to successfully draw learner attention to the conceptually-relevant component in the problem, and in so doing significantly improved problem-solving performance. Without scaffolding, novices tend to rely upon feature salience (e.g., prominence, distinctiveness) to infer importance. Again, this is problematic when the most obvious visual features are not necessarily the most conceptually relevant. Drawing students’ attention to relevant problem features seems promising as a learning strategy, but can be difficult to implement if relevant features are not visually detectable. This is the case with many learning activities in physics, in which factors underlying concrete properties and interactions may be invisible (e.g., an object falling to the ground due to gravity). Moreno, Ozogul, and Reisslein (2011) examined student learning about the physics phenomena of circuits; students learned with concrete-only, abstract-only, or concrete and abstract visual representations. In the latter condition, abstract representations were paired with concrete representations to make explicit (with symbols) the scientific concepts at play in a real-world situation. Results of the study 6 showed that students studying circuits with a combination of concrete and abstract visual representations outperformed students studying with concrete-only representations on measures of both problem-solving and transfer. This suggests that when visualizations do not clearly depict important concepts, student learning may be hindered. An important question is how and under what conditions unseen conceptual components can be visualized to contribute to a better understanding of physics principles. For physics learning, this challenge extends to hands-on laboratory experiments where students connect domain principles to dynamic real-world phenomena. Inquiry-driven learning experiences such as hands-on laboratory experiments serve a fundamental role in science (Hofstein & Lunetta, 2004). A key goal of these activities is to help students apply scientifically-grounded concepts to the physical phenomena they experience daily. Real-world experiments provide students with the opportunity to investigate the operation of abstract scientific principles through learning experiences situated in concrete, real-world contexts. Consistent with this instructional goal, hands-on experiments have been shown to support memory and conceptual understanding in science (Kontra, Lyons, Fischer, & Beilock, 2015). Despite the potential benefits of hands-on experiments for knowledge application in physics, some studies suggest students’ grasp of foundational physics principles is disappointing—even after completing a physics course that included typical laboratory learning experiences (Hake, 1998). The immaterial nature of conceptually important features (i.e., invisible forces) may help to explain why students have a difficult time thinking beyond the physical nature of hands-on labs. Kozma (1999) found that novice students completing a hands-on laboratory experiment more often discussed the operation 7 of equipment and the physical properties of the experiment than the underlying, conceptual nature of the processes being demonstrated. For example, students might reason about how to set the incline of a plane and the subsequent speed of a rolling cart on the plane without reasoning about the underlying forces driving the results. 1.1.1 Learning With Concrete Visualizations Although hands-on experiments typically are designed to illustrate conceptual information (including abstract ideas and principles that are common across multiple situations), they do so with concrete features and representations that differ at a surface level. For example, rolling a ball and then a car down a ramp differ in terms of surface features (ball vs. car) but the underlying principles being demonstrated remain the same. Even within a single experiment (e.g., the ball rolling down the ramp), the physical elements of a real-world experiment are marked by surface details which may or may not be relevant. Consider the example of a physics student rolling a ball down a ramp. The student may notice the texture, color, and material make-up of each component (the ball and the ramp). The student may also note that the ball seems to move slowly or quickly, but likely will not visualize the abstract forces underlying that movement or how they dynamically update as the experiment progresses. Concrete visualizations are useful because the ultimate goal of science learning is to better understand the material world we inhabit. Indeed, relevance to everyday experience has been proposed as a beneficial component of concrete visualizations, increasing their potential effectiveness for learning (Clements, 2000). Utilizing objects and interactions with which students have had informal, everyday learning experiences 8 provides a way to activate and build upon students’ prior knowledge (Donovan & Bradford, 2005). Furthermore, the use of concrete materials has been shown to improve long-term retention of learned information, particularly among novice students (Joseph & Dwyer, 1984). While concrete visual representations may provide some learning advantages, they also come with significant drawbacks. Concrete imagery consistently has been shown to be less effective than abstracted visual representations for promoting deep understanding. For example, Scheiter, Gerjets, Huk, Imhof, and Kammerer (2009) tested the comprehension of students who had learned about a biological concept (cell replication) via realistic or schematic pictures. Students learning with schematic images outperformed students learning with realistic images on all measures. Furthermore, they reported fewer problems in integrating the visual information with verbal explanations provided. The authors concluded that the amount of detail present in the realistic images was overwhelming for students. This is consistent with the findings of Butcher (2006), who showed that while both detailed and simplified diagrams supported comprehension and learning, simplified, schematic diagrams were more effective for knowledge integration: by reducing unnecessary concrete details, simplified visualizations can emphasize key relationships. Retaining accurate detail may not only fail to promote knowledge integration, but actually may impede integration processes. Sloutsky, Kaminski, and Heckler (2005) examined the use of concrete images that included conceptually irrelevant features. Findings demonstrated that both learning and transfer were impeded with such images, suggesting that irrelevant concrete features disrupt encoding of transferable knowledge. 9 Although the mechanism of this effect is yet unknown, it may be that irrelevant concrete details tax cognitive resources by encouraging processing of perceptual elements rather than conceptual relationships. Another possibility is that the irrelevant concrete details may be included in the encoded representation, thereby limiting the flexibility and transfer of resulting knowledge. If concrete details compromise the development of transferable knowledge, one might be tempted to make the general conclusion that concrete visualizations are inferior to their abstract counterparts. But, as discussed earlier, novices tend to have difficulty processing abstract visual representations (Hegarty & Just, 1993), and they may benefit from the ways in which concrete representations activate prior knowledge (Donovan & Bradford, 2005). In order for learners to process concrete objects conceptually rather than merely perceptually, learning tasks can be designed to emphasize the symbolic nature of portrayed information. One approach is to reduce tactile interaction with concrete objects, relying instead upon visual observation or digital manipulation (Uttal, O’Doherty, Newland, Hand, & DeLoache, 2009). Another approach is to utilize multiple, complementary representations of information to reduce learners’ attention to superficial object properties, focusing attention instead on important commonalities and encouraging learners to abstract generalizable features (Ainsworth, 2006). When representations are understood in isolation, resulting knowledge often is compartmentalized (Ainsworth, 2006). Providing students with multiple representations of information in concert is one way to provide learners access to the benefits of each individual representation, while helping to deepen understanding of conceptual relationships shared between representations (Ainsworth, 2008). For example, using 10 static diagrams of electric circuits, Moreno et al. (2011) found that learning with a combination of abstract and concrete diagrams was more effective than learning with either abstract or concrete representations alone. Multiple representations – including concrete and abstract visualizations – may be presented simultaneously or sequentially. When multiple visual representations are presented simultaneously, their very proximity may support students in identifying connections between concrete and abstract features (Ginns, 2006). The advantage of using simultaneously-presented concrete and abstract representations can be seen not only with static diagrams, but also with dynamic (updating) content. Van der Meij and de Jong (2006) found that when multiple abstract and concrete visual representations were dynamically linked and visually integrated (e.g., superimposed one on top of another), students performed better on measures of domain knowledge and inter-representational translation. This finding suggests that explicitly integrating the shared features of multiple representations facilitates connection-making among novice students, resulting in a more cohesive mental model. However, other research has suggested that while presenting multiple visual representations (concrete + abstract) in an integrated, dynamically-linked format does indeed support representational flexibility, it is not sufficient for promoting deep understanding. In a previous study (Davies & Butcher, in preparation), students learned with videos of concrete physics experiments that either had been overlaid with dynamic, abstract annotations or that had no annotations (the control condition). Students who viewed videos that integrated abstract and concrete representations outperformed students in the control condition on several measures, most notably on items that required 11 translating concrete situations (e.g., a car driving) into abstract symbols. However, this superior representational flexibility was not associated with demonstrable improvements in comprehension of physics principles. One possible explanation for the lack of improvement in conceptual understanding may be overdependence on concrete, contextual information. That is, students may be mentally "translating" the abstract representations into extensions of the concrete representation (e.g., "That arrow reflects the car's speed") rather than processing the conceptual meaning of the abstract symbol. Indeed, previous research has shown that students with low prior knowledge struggle to deeply process abstract ideas when concrete properties are salient (Goldstone & Sakamoto, 2003). For example, consider the movement of an ice cube pushed across a table. Students may understand the motion more in terms of the ice cube’s “slipperiness” rather than resulting from a specific relationship between the propelling force and opposing friction. For deep understanding, students must not only be able to identify and interpret key features of concrete and abstract representations, but also to understand the underlying domain principles connecting them. Thus, it may not be enough to combine abstract and concrete representations in a single visualization. One presentation strategy using concrete and abstract representations that has been shown to improve understanding of domain principles as represented in visual simulations is called concreteness fading (Goldstone & Son, 2005). Concreteness fading refers to the sequential presentation of multiple visual representations, beginning with concrete visualizations that are subsequently replaced by their abstract counterparts (i.e., the same visual situation that has been stripped of realistic detail). In this way, information initially 12 is grounded in an accessible, concrete context but that concrete context ultimately is removed so that student can focus their processing less on concrete details and more on domain principles. Goldstone and Son (2005) examined student learning using four visualization categories: consistently concrete, consistently abstract, concrete to abstract, and abstract to concrete. Results showed that while concrete and abstract visualizations had unique advantages, the best combination of visualizations for promoting comprehension and transfer were presented with concreteness fading (concrete followed by abstract visualizations). Concreteness fading is one way to ground scientific knowledge in real-world examples while increasing attention to conceptually important features and relationships and decreasing reliance on irrelevant information. However, previous research has examined this strategy using concrete and abstract representations that illustrate the same objects. For example, Goldstone and Son (2005) represented ants and food in two ways: concretely (images of ants and food) as well as symbolically (black dots and green patches). Although the representations were presented in sequence (temporally and spatially separate), there was a 1:1 correspondence between concrete and abstract components. In the physics domain of force and motion, however, there is not 1:1 correspondence between concrete and abstract representations. Forces operating on realworld objects cannot be observed and have no concrete form to be represented. Therefore, any attempt to integrate visible forces into a visualization is abstract in terms of the underlying concepts being represented. We do not know the extent to which concreteness fading may support student learning and transfer when the information being represented includes the interaction of concrete objects and abstract forces, 13 regardless of the visualization chosen to represent them. The physical objects from the real-world are the only part of this representation that can be represented in both concrete and abstract ways. Thus, physics representations can start out as a mix of concrete (objects) and abstract (force) components and then can be faded to a fully abstract representation. This work addresses the potential impact of concreteness fading in dynamically-linked visualizations for physics. 1.2 Producing Explanations During Learning As described above, the design of visual representations has strong impact on students’ cognitive processes during learning. A well-designed visualization can support novices in important activities such as reasoning more often about relevant scientific concepts, generating inferences, and integrating visual and verbal information. These are outcomes that can be observed and even encouraged by engaging students in selfexplanation. Self-explanation is a well-known strategy for capturing and promoting student learning (Roy & Chi, 2005). That said, some types of self-explanation are more useful than others (Renkl, 1997). Even well-designed visualizations may not support meaningful learning outcomes if learners lack sufficient prior knowledge to produce specific, conceptually-relevant explanations of the provided visualizations. Indeed, studies suggest that students rarely construct high-quality scientific explanations spontaneously (Chi, Bassok, Lewis, Reimann, & Glaser, 1989; Renkl, 1997). This finding is consistent with prior research showing that students’ verbalized interpretations of physics experiments tended to be more descriptive than explanatory, even when the explanations are categorized as scientifically-relevant (Davies & Butcher, in preparation). 14 Yet, articulation and persuasive argumentation are considered critical domain skills in physics and other science domains (Berland & Reiser, 2009). This suggests that scaffolding may be needed to help students reason in ways that are more accurate and explanatory when learning with STEM-focused visualizations. 1.2.1 Self-Explanation Self-explanation refers to a metacognitive learning process in which learners engage in monitoring their understanding of studied material as well as generating inferences to connect that material with prior knowledge (Roy & Chi, 2005). In their seminal study of “good” and “poor” physics students, Chi and colleagues (1989) found that more successful problem-solvers spontaneously engaged in self-explanation during learning. They more often verbalized connections between domain principles and specific practice exercises, and ultimately demonstrated a greater understanding of domain principles. Subsequent studies demonstrated that learners achieved better outcomes when trained to use self-explanation and encouraged to use it during study (Chi, De Leeuw, Chiu, & Lavancher, 1994). The benefits of self-explanation have led many researchers to examine its impact in a variety of learning contexts (Ainsworth & Loizou, 2003; Chi et al., 1994; de Koning, Tabbers, Rikers, & Paas, 2011; Wong, Lawson, & Keeves, 2002). But, even when students effectively engage in more frequent self-explanation, not all explanations generated by learners are equally beneficial. In other words, verbosity is not effective for learning in and of itself. Renkl (1997) found that only a small fraction of students' spontaneously-produced self-explanations could be categorized as highly-beneficial–that 15 is, self-explanations that make connections between the problem or materials at hand and larger conceptual principles or explanations. In physics, the most effective forms of selfexplanation require connecting physical features of a problem (e.g., car accelerating down a ramp) to conceptual principles (e.g., net force). But students are unlikely to be able to complete this mapping without sufficient prior knowledge. Instead, a majority of students’ utterances tend to consist of less-useful content, such as elaborating the problem at hand. Several other studies have similarly demonstrated that, while good explanations improve learning, they are not easy for students to construct (see for example: Cho & Jonassen, 2012; de Koning, Tabbers, Rikers, & Paas, 2010). Even with training, useful explanatory statements may be few and far between. The need to consider the content of self-explanations during a study session is highlighted by research showing that, in some circumstances, self-explanation can hurt learning. Self-explanation prompts can lead students to focus on certain types of information at the exclusion of others, resulting in selective encoding (Große & Renkl, 2006; Williams & Lombrozo, 2010). This may result in incomplete knowledge that limits the quality of learning outcomes. Another possibility is that a student engaged in selfexplanation may explain their own flawed thinking without recognizing erroneous content. The process of self-explaining flawed ideas can reinforce misconceptions, impeding the acquisition of new, accurate information that is necessary for mental model revision (Rittle-Johnson & Loehr, 2016). Given the prevalence of misconceptions and naïve ideas that novices have about physics phenomena, encouraging well-composed, meaningful, and high-quality self-explanations is a special concern for physics education. Because learners do not naturally self-explain and rarely engage in the highest-quality 16 self-explanation, coupled with the potential danger of self-explaining erroneous ideas, there is a strong need for thoughtful design and effective instructional scaffolding to promote high-quality self-explanation during physics learning. One of the most effective methods for scaffolding self-explanation in previous studies has been through the use of a human tutor or facilitator, who verbally prompts a learner during an individual study session (Litman et al., 2006). Although effective, a one-on-one facilitator represents a high response cost for eliciting verbal selfexplanations, diminishing its utility in classroom settings. One potential alternative is computer-supported self-explanation, where self-explanation prompts and instructional scaffolds are provided automatically by a computer system. Computer-supported selfexplanation is promising because it reduces the human cost of a useful learning strategy while still allowing for personalized learning. 1.2.2 Computer-Based “Self-Explanation” One way to solicit self-explanation in a digital environment is to provide generic prompts similar to those used to facilitate verbal self-explanation, requiring typed explanations in response. Yet, research suggests that typed self-explanations tend to be more inference-impoverished than verbal self-explanation: typed self-explanations often consist of statements of fact and knowledge-telling (Hausmann & Chi, 2002). Still, the generative nature of typed self-explanation seems to provide some benefits. Using the geometry Cognitive Tutor, students prompted to explain domain principles in their own words showed comparable learning to students who used drop-down menus to justify their answers (Aleven, Ogan, Popescu, Torrey, & Koedinger, 2004), and they also 17 produced more complete explanations during problem-solving at posttest. Thus, even if typing explanations does not provide a learning advantage in terms of domain content, it may still provide an advantage in training a transferable learning strategy. However, this method is compromised by some of the same issues as verbal self-explanation. If students’ explanations are inaccurate, erroneous ideas may be reinforced by explaining them. This would suggest that feedback is necessary for truly effective self-explanation; but it can be difficult to administer appropriate feedback for open-ended responses. As noted above, an alternative to soliciting typed responses utilizes menu selections or preconstructed statements to support self-explanation. This approach often is used to require students to justify their choices during problem-solving. Although this has been referred to as self-explanation in the intelligent tutoring community, it should be noted that students who justify their choices during problem-solving usually are engaged in limited generation. Therefore, there is limited depth and personalization to this form of computer-supported self-explanation. For example, Aleven and Koedinger (2002) used an ITS for geometry learning that asked learners to name the geometry principle (e.g., alternate interior angles) that justified their answer in each problem-solving step. The principal name could be typed or chosen from a provided glossary. Similarly, Conati and Vanlehn (2000) used a physics intelligent tutoring system (ITS) that prompted students to name and also to elaborate the meaning of specific physics principles. As students solved problems, they were prompted to “explain” their answers. A dialog box asked students to complete the statement “This choice is correct because…” by selecting a rule from a list of options. Once the rule was selected, an additional dialog box opened containing an incomplete definition of that rule. Students clicked on blanks in the definition to see a list 18 of optional fillers. They completed the definition by selecting a phrase to fill in the blank. In both the physics and geometry examples, students who justified their actions during problem-solving evidenced better learning. However, while focusing on the names or definitions of domain principles evidences comprehension, it does not necessarily gauge or promote deeper, inferential reasoning related to domain principles. A deeper approach to principle naming might require learners to choose from among several precrafted explanatory statements. Similar to other types of menu selection, this type of self-explanation would allow automation both in requiring an explanation and in providing feedback on the explanation. Meanwhile, exposing learners to multiple explanations that vary in accuracy might cause them to more thoughtfully consider the relationships between domain principles and specific problems or examples. There is some evidence that evaluating an explanation during learning can support meaningful outcomes. D’Mello, Lehman, Pekrun, and Graesser (2014) investigated the learning of students exposed to multiple contradictory statements about a scientific phenomenon. They found that contradictions actually promoted learning by inducing confusion. But, providing fully-formed contradictory statements to students may also limit the degree to which they engage in generative processing during learning. Some research supports the idea that generation is a necessary process during effective self-explanation; this research has found that although students do learn with provided explanations and show more confidence in their understanding, they actually learn less with provided content than when they are required to generate explanations themselves (Schworm & Renkl, 2006). Since generating inferential content is one of the hallmarks of self- 19 explanation (Roy & Chi, 2005), one can question whether provided explanations should be considered a true form of self-explanation unless the learner is producing observable inferences. Balance between student-generated and computer-provided content is key in designing an explanation task that is challenging enough to elicit student inferences but not overly demanding on cognitive resources (Koedinger & Aleven, 2007). With too much support, students often engage in superficial problem-solving strategies that undermine learning (e.g., rapidly inputting several possible answers until the correct solution is identified). Provided with full scientific explanations to choose from, students may make decisions based upon superficial strategies such as vocabulary matching or explanation length without fully processing the conceptual content of the explanation. But without adequate assistance, novice learners may be unsure of how to proceed and can become overwhelmed or frustrated. Tasked with generating an explanation from scratch, novices may focus on shallow details. This is particularly the case in the absence of immediate system feedback (Aleven & Koedinger, 2000). The challenge of providing too much or not enough support is an “assistance dilemma,” a known concern when designing computer-based learning environments (Koedinger & Aleven, 2007). If students are expected to produce explanations during a computer-based learning task, a moderate level of assistance may be needed to prompt learning that is thoughtful and also effective for learning. One possible approach used in previous research consists of a multistep explanation task to assist learners in producing more complete explanations during learning. Such an approach was used in a study of problem-solving in algebra (Booth, 20 Lange, Koedinger, & Newton, 2013). In this study, learners solved an algebra problem and then were provided with a selection of content segments with which to create an explanation of their work. First, students explained their problem-solving strategy (operation) by making selections from three consecutive drop-down menus. Next, students provided a justification for their problem-solving strategy by making selections from two-more drop-down menus. The provided segments served as building blocks, eliminating the demand of from-scratch generation. But, learners still had to act constructively in order to identify needed content and chain it together appropriately. This multistep explanation strategy seems like it could be a useful method for improving student explanations in physics, both in terms of quality and effectiveness for learning. However, it should be noted that when implemented in the algebra study, the multistep explanation interface was not a part of the experimental intervention; it was used by all participants. Thus, it is not known how scaffolding learner explanations in this way compares against other methods for computer-based explanation. Research examining the challenges of constructing scientific explanations lends insight into how content might be segmented to provide the most useful scaffolding during a physics learning task. 1.2.3 Generating Scientific Explanations As noted above, while generating explanations has been shown to benefit students in many learning situations, free generation may not be the most effective approach when working with multiple, complex representations of science information (Kuhn & Katz, 2009). When learning from real-world activities, for example, science students may be especially prone to neglect relevant principles and data in favor of what is more 21 perceptually obvious (Nilsson, 2013). Sandoval (2003) found that even when students made an accurate scientific claim, they often substantiated the claim with inappropriate evidence. The problem was compounded when students were provided with data from multiple representations that were not explicitly linked. When students struggle to identify relevant features and relationships, generating explanations may actually do more harm than good by reinforcing incorrect conclusions. Scaffolding may be needed in helping students to focus on relevant elements of physics problems in order to generate appropriate scientific explanations. And, corrective feedback likely is also needed. As students gain conceptual understanding, particularly through the analysis of experimental data, they are able to produce more complex, scientifically-sound explanations of real-world phenomena (Klein, 2004; McNeill, Lizotte, Krajcik, & Marx, 2006). Yet, classroom studies suggest that few students construct scientific explanations which contain all the necessary elements for establishing causality (Berland & Reiser, 2009; Ruiz-Primo, Li, Tsai, & Schneider, 2010). Instead, learners often focus solely on making claims, without providing supporting data or explanatory reasoning (Ruiz-Primo et al., 2010; Sandoval, 2003). Investigations of students’ scientific explanations suggest that even when unconstrained explanatory tasks yield desirable integrative reasoning, good explanations tend to make up a relatively small proportion of generated content. For example, Chi and colleagues (1994) found that the majority of student utterances during a science learning task consisted of paraphrasing and monitoring statements; few explanations evidenced inferential reasoning about the content, even among better learners. Research specific to physics (Davies & Butcher, in preparation) found that when students were provided with 22 prompts but allowed to freely generate explanations of laboratory activities, about one in three utterances was irrelevant to the learning material. Furthermore, those explanations that were relevant rarely elaborated causal mechanisms. A growing body of research suggests that even slight constraints can improve the quality of student-generated content. Several scaffolds have been implemented specifically to improve science explanation, with encouraging results. McNeill et al. (2006) asked students to construct scientific explanations using a framework that made the necessary components of a complete explanation explicit (claim, evidence, and reasoning). Using such a framework over the course of an 8-week science unit led to significant improvements in all aspects of students’ scientific explanations, even in the absence of the prompting framework stipulating necessary explanation components. However, posttest performance indicated that students still struggled to substantiate their arguments appropriately, typically using a combination of relevant but insufficient evidence along with some irrelevant evidence. Furthermore, students rarely used sufficient scientific principles to elaborate the relationships between their conclusions and data. Sandoval (2003) also utilized a framework requiring students to construct scientific explanations using similar components during inquiry learning activities. The conceptual scaffolds were domain- and problem-specific, reducing the demands of identifying and integrating relevant pieces of information by providing guiding goals and questions as well as representations of data from which students could make selections. In this study, findings suggested that the combination of explicit conceptual scaffolds and implicit epistemic scaffolds did indeed help students to construct more coherent and bettersupported scientific explanations. Furthermore, evidence suggested that students were 23 supported in identifying important pieces of data. However, students continued to struggle with interpreting that data. This suggests that in order to construct appropriate scientific explanations, learners may benefit from scaffolding that not only focuses students on relevant information and links the necessary components of a causal claim, but also supports students in making sense of the information presented. 1.3 The Current Experiment This research sought to examine how online scaffolding strategies that combine visualization support for transfer with support for high-quality, conceptually-relevant explanations impact the quality and depth of physics learning for novice students viewing real-world phenomena. First, this study examined how concreteness fading may impact learning with visualizations of real-world experiments that have been overlaid with symbolic annotation. In other words, it examined the impact of concreteness fading on learning when the abstract and concrete visual representations utilized were complementary rather than redundant, and when they were also initially integrated rather than portrayed separately. Second, this research examined how scaffolding student explanations of physics phenomena may impact comprehension of physics concepts and the quality of subsequent scientific explanations. This study varied the amount of generation required by constrained, conceptually-targeted explanation scaffolds in an online environment. Students either made zero, one, or three selections from provided content to yield complete scientific explanations of real-world. The current study addressed two research questions, detailed below along with 24 corresponding hypotheses. 1.3.1 Research Questions and Hypotheses 1.3.1.1 Research Question 1 Can implementing concreteness fading during learning help students to comprehend physics concepts, integrate symbolic and concrete representations, and produce better scientific explanations of real-world phenomena? 1.3.1.1.1 Hypothesis 1. Because concreteness fading has been shown to promote principle abstraction, it was expected that learners who studied with concreteness fading would gain a deeper understanding of physics principles such that they could more successfully be interpreted and also applied to novel contexts. It was predicted that students who saw concreteness fading would demonstrate higher scores on comprehension, integration, and scientific explanation. 1.3.1.2 Research Question 2 To what extent does scaffolding conceptually-relevant input during explanation of physics experiments help students to comprehend physics concepts and to produce scientific explanations about real-world phenomena? 1.3.1.2.1 Hypothesis 2. Requiring students to build explanations by selecting multiple system-provided segments reduces the effort of from-scratch content generation while promoting constructive cognitive processing and thus was hypothesized to yield the greatest improvements in comprehension and scientific explanation. Requiring students to select the best choice from a set of system-provided explanations is still highly 25 constrained and not generative, but requires student input and thus was expected to prompt more active cognitive processing, yielding better comprehension and scientific explanation than requiring no input at all. 26 Figure 1.1: An example of a free body diagram. The arrows represent the directionality of forces, with length indicative of magnitude. In this diagram, the upward and downward arrows are of equal magnitude and sum to 0; the magnitude of the right-facing arrow is subtracted from the left-facing arrow, yielding a net force (and thus, acceleration) in the leftward direction. . CHAPTER 2 METHODS 2.1 Design This study utilized a 2x3 factorial design. The independent variables were visual concreteness (continued or faded) and explanation activity (view explanation, select explanation, or build explanation). 2.2 Participants One hundred undergraduate students participated in this study: 57 females, 43 males. Participants were students from an introductory physics laboratory course at Saint Louis College of Pharmacy (STLCOP). There were six sections of the course, with about 15-20 students per section. The students participated in the research during class time and received course credit for their participation. At the time of the study, participants were studying a unit that included the topic of force and motion. STLCOP is a small private college in Saint Louis, Missouri. All students of the college are enrolled in a 7-year program through which they earn Bachelor of Science and (subsequently) Doctor of Pharmacy degrees. Admitted students have an average ACT score of 27, which suggests they may have better than average performance in STEM subjects. Most participants (86%) reported having completed a physics course previously 28 (e.g., high school physics). However, STLCOP students do not enroll in a physics course until their junior year of college. Physics professors report low prior knowledge for enrolled students in the introductory courses. 2.3 Materials 2.3.1 Digital Learning Environment A web-based digital learning environment was created to deliver physics visualizations and collect students’ explanations of physics phenomena during study. The digital learning environment was designed for individual computer workstations in a networked laboratory. Scaffolding features included in the digital learning environment varied by experimental condition, as described below. 2.3.1.1 Physics Visualizations The digital learning environment was created to facilitate study of both static and dynamic physics visualizations; to that end, videos of hands-on physics experiments as well as diagrams drawn from the videos were created. 2.3.1.1.1 Videos of hands-on physics experiments. Each video in this study depicted one of three different scenarios that could be observed via real-world laboratory activities. The scenarios were defined in consultation with a college physics professor to represent a variety of force-motion relationships. The scenarios were the following: 1) an object initially suspended in air, then released to fall to the ground; 2) an object on a horizontal plane that initially was still, accelerated to a point, and then maintained a constant velocity; and, 3) an object on a horizontal plane that initially was still, 29 accelerated to a point, then decelerated until forward motion ceased. Each scenario was designed to incorporate dynamic interactions that varied over time in terms of the location of the target object, the motion of the target object, and/or the forces acting on the target object (see an example in Figure 2.1). Hands-on laboratory activities were carried out and video recorded to capture each scenario. Video recordings of the scenarios lasted from 1.5 to 6 seconds. The Adobe Creative Suite was used to add a symbolic overlay representing forces at play. Three video versions of each scenario were created to represent the same underlying symbolic structure but with varied concreteness (see Figure 2.2). The first version was considered the primary example and consisted of real-world imagery overlaid by symbols of conceptual elements. For example, version one of scenario one depicted a black basketball overlaid with red arrows representing the forces at play. The second version also integrated concrete and abstract representations, but was designed to vary the superficial (concrete) detail from version 1. For example, version two of scenario one depicted a brown cardboard box overlaid with red arrows representing the forces at play. The third version was designed to remove the concrete imagery entirely, leaving only abstract symbols (see Figure 2.2). For example, version three of scenario one depicted the hands-on activity shown in version 1 as a free body diagram, with an open black square representing the basketball and contiguous black arrows representing forces. 2.3.1.1.2 Static diagrams. Frames sampled directly from the videos were selected to present students with key relationships between force and motion. Per scenario, two force-motion relationships were extracted from the video and presented to students as a 30 static diagram. This means that a total of six force-motion relationships were presented across the three scenarios. From scenario one, the selected relationships were the following: 1) an object with equal, opposing vertical forces that is still; and, 2) an object with an unopposed downward force that is moving downward. From scenario two, the selected relationships were the following: 1) an object with equal, opposing horizontal forces that is still; and, 2) an object with equal and opposing horizontal forces that is moving. From scenario three, the selected relationships were the following: 1) an object with a propelling force that is moving forward; and, 2) an object with no propelling force that is moving forward. 2.3.1.2 Explanation Interfaces The explanation interface provided a text prompt and interactive elements designed to invoke self-explanation of a specific force-motion relationship. Three different forms of the explanation interface were developed to vary student explanation activity as required for the experimental conditions: view explanation, select explanation, or build explanation. The text prompts did not vary by experimental condition. 2.3.1.2.1 View. The view explanation interface collected no explanation input and thus required minimal activity and provided maximum support. As seen in Figure 2.3, this interface provided a “View Explanation” button to click in order to view the correct answer to the prompt. Once clicked, the “View Explanation” button was replaced by an accurate explanation as well as a button labeled “I’m ready to move on” that could be clicked to advance through the system (see Figure 2.4). All provided explanations were complete, consisting of three components: causal elements, causal interaction, and effect. 31 2.3.1.2.2 Select. The select explanation interface required learner input but no generative activity. As seen in Figure 2.5, this interface provided a drop-down menu consisting of four possible explanations. All choices represented complete causal explanations consisting of three relevant components: causal elements, causal interaction, and effect. Only one choice was fully accurate; the remaining choices could be fully inaccurate or partially inaccurate (e.g., with at least one component being incorrect but one or two components being correct). Once a selection had been made, it was submitted by clicking “Check Answer.” After “Check Answer” was clicked, the drop-down menu disappeared and the system provided correct or incorrect feedback. If the system indicated that the selection was correct, the correct explanation remained on screen and a button labeled “I’m ready to move on” appeared; this button could be clicked to advance through the system (see Figure 2.6). If the system indicated that the selection was incorrect, a “View Correct Explanation” button appeared (see Figure 2.7). Once clicked, the “View Correct Explanation” button was replaced by a complete, accurate explanation as well as an “I’m ready to move on” button, which could be clicked to advance through the system—this feedback was identical to the provided explanation in the view explanation condition (see Figure 2.4). 2.3.1.2.3 Build. The build explanation interface required generative learner activity: an explanation was “constructed” through making selections for each of the three explanation components: causal elements, causal interaction, and effect. As seen in Figure 2.8, this interface provided three drop-down menus corresponding to the three components, respectively. Each menu consisted of three possible options. Once selections had been made from all three drop-down menus, a complete explanation was input to the 32 system by clicking “Check Answer.” As with the select explanation condition, all submissions represented complete causal explanations consisting of causal elements, causal interaction, and effect, but only one choice was fully accurate. The remaining choices could be fully inaccurate or partially accurate (e.g., with at least one component being incorrect but one or two components being correct). After “Check Answer” was clicked, the drop-down menus disappeared and the system provided correct or incorrect feedback. If the system indicated that the submission was fully correct, the correct explanation remained on screen and a button labeled “I’m ready to move on” appeared; this button could be clicked to advance through the system (see Figure 2.6). If the system indicated that the submission was not fully correct and thus incorrect, a “View Correct Explanation” button appeared (see Figure 2.7). Once clicked, the “View Correct Explanation” button was replaced by a complete, accurate explanation as well as an “I’m ready to move on” button that could be clicked to advance through the system. This final view was the same explanation as viewed initially by the view explanation condition (see Figure 2.4). 2.3.1.3 User Interfaces Within the digital learning environment, two user interfaces were used to present physics visualizations and to collect explanation input: a video viewing page and a diagram explanation page. The video viewing page was designed to provide first exposure to a hands-on physics experiment. It consisted of a caption, a masked video, and a “Play Video” button (see Figure 2.9). The caption described the video content. The play button could be 33 clicked to remove the mask and begin playing the video. No other video controls were provided. Each video viewing page was associated with two diagram explanation pages. The diagram explanation pages were designed to prompt self-explanation about specific moments drawn from the associated video. Each diagram explanation page consisted of a static diagram, a space that displayed the user-controlled masked video of the full handson task, and an explanation interface (see Figure 2.10). The static diagram (at left in the interface) cued the learner to the specific force-motion relationship that was to be explained; this was necessary, because a video will contain several different force-motion relationships with distinct explanations. The associated video, available on the right side of the screen, was provided as an optional reference; clicking on any of the provided video controls activated the video. Four video controls were provided: play, pause, frame forward, and frame backward. The diagram explanation page was designed as a selfpaced activity, but a timer counted down a maximum time allotment. 2.3.2 Knowledge Assessments To gauge participant learning during the study, knowledge assessments were administered at pretest and posttest; all assessments were collected by computer. Assessments included measures of domain knowledge, representation integration, and explanation quality. 34 2.3.2.1 Domain Knowledge Assessments To evaluate participants’ understanding of general domain principles, two kinds of assessment items were administered: declarative knowledge items and application items. 2.3.2.1.1 Declarative knowledge items. To measure participants’ recall of foundational physics concepts relevant to the learning material, declarative knowledge items required selection of the appropriate definition for four physics terms selected by a physics professor. The declarative knowledge items were administered as multiple choice with five response options, with one point possible per item (see Figure 2.11). Both pretest and posttest consisted of the same four items, for a total possible score of four points. To reduce practice effects, the order of items and answer choices was changed from pre- to posttest. 2.3.2.1.2 Application items. As a measure of deeper comprehension, application items presented students with hypothetical scenarios. Selecting answers about the scenarios required students to apply the physics concepts they had studied (see Figure 2.12). Six items were drawn from a bank of multiple choice items developed and tested by the American Association for the Advancement of Science. National testing suggests that for this set of items, American secondary students achieve an accuracy rate of 36% (American Association for the Advancement of Science, 2017). Both pretest and posttest consisted of the same six items, for a total possible score of six points. To reduce practice effects, the order of items and answer choices was changed from pre- to posttest. 35 2.3.2.2 Representation Integration Assessments Two kinds of integration items were used to assess participants’ ability to relate multiple visual representations: concrete translation items and symbolic translation items. Together, these items serve as complementary measures for evaluating participants’ understanding of the relationships between visual representations. 2.3.2.2.1 Concrete translation items. The concrete translation items assessed students’ ability to translate a concrete situation into a symbolic visual representation. This measure presented learners with text descriptions of various real-world scenarios (e.g., an automobile driving on a highway at a constant speed of 70 mph). Along with each description, participants were shown several free-body diagrams. They were asked to identify the diagram that correctly represented the forces at play in the given scenario (see Figure 2.13). These items tested participants’ accuracy in representing symbolic elements (specific, individual forces at play), as well as their understanding of the relationship between elements (e.g., relative differences in magnitude and/or direction). The pretest consisted of three concrete translation items, for a total possible score of three. The posttest consisted of all three pretest items plus an additional five concrete translation items, for a total possible score of eight. To reduce practice effects, the order of items and answer choices was changed from pre- to posttest. Performance on posttest concrete translation was examined for differences between items categorized as easy versus difficult. “Easy” concrete translation items were items for which symbolic and concrete representations are in alignment. For example, imagine a box as it begins to be pushed to the right: the symbolic representation includes a large arrow (representing a force) pointing to the right, while the concrete 36 description indicates movement in the same direction. In contrast, “difficult” concrete translation items were items for which the symbolic and concrete representations were not in alignment. For example, the item in Figure 2.13 describes a box that continues to move to the right after it has been shoved: the concrete description indicates movement to the right, but the symbolic representation does not include an arrow pointing in that direction. Out of the eight concrete translation items at posttest, four were classified as easy and four as difficult. Both easy and difficult concrete translation measures were analyzed as percentage correct out of four. 2.3.2.2.2 Symbolic translation items. The symbolic translation items assessed students’ ability to interpret a symbolic visual representation and determine its alignment with concrete descriptions of motion. Items depict a free body diagram along with several descriptions of object motion (e.g., “the object is increasing in speed as it moves to the right”); participants must select all of the descriptions that could be accurate for the given free body diagram (see Figure 2.14). These items require cognitive flexibility because one diagram can accurately reflect several different real-world interpretations. The pretest consisted of three symbolic translation items, with a total of seven correct interpretations in all. The posttest consisted of all three pretest items plus an additional five items, with a total of 18 correct interpretations in all. To reduce practice effects, the order of items was changed from pre- to posttest. At both pre- and posttest, overall performance on concrete translation was analyzed as percentage of correct interpretations selected. Because participants could make multiple selections for each symbolic translation item, each item consisted of multiple correct as well as incorrect choices. As with the concrete translation items, correct symbolic translation selections could be classified as 37 “easy” or “difficult” to the extent that the symbolic elements aligned with the concrete descriptions of object motion. For example, the easy correct interpretation of the diagram in Figure 2.14 would be “moving to the right and speeding up” since the larger horizontal arrow is to the right; a difficult correct interpretation of the same diagram would be “moving to the left and slowing down.” Each item had a single “easy” correct interpretation and one or more “difficult” correct interpretations. The remaining interpretations were incorrect. Across 8 items, there were 8 easy symbolic translation selections, 11 difficult symbolic translation selections, and 32 incorrect selections. Easy, difficult, and incorrect symbolic translation measures were calculated as percentage of total possible selections. 2.3.2.3 Explanation Quality Written explanation items asked participants to generate physics explanations for various phenomena (e.g., the upward movement of a baseball that had been thrown into the sky). Assessment items consisted of a description of a real-world phenomenon involving a target object (indicated by underline), a prompt to relate force and motion for the target object, and a text box for typing the explanation (see Figure 2.15). The pretest included two written explanation items, and the posttest included both pretest items as well as four additional items for a total of six items. To assess the quality of students’ written explanations, two rubrics were developed. The rubrics assessed written explanations in terms of 1) relevance and depth, and 2) accuracy and completeness, respectively. 2.3.2.3.1 Explanation relevance and depth. The coding rubric for relevance and 38 depth was adapted from previous research on verbal explanations generated during video learning in physics (Davies & Butcher, in preparation). The rubric consisted of four mutually-exclusive codes: no content, shallow reasoning, descriptive physics reasoning, and causal physics reasoning. Each written explanation received one of the four codes. Items received a code of no content if no answer was submitted or if the meaning of the written content was unclear. Items received a code of shallow reasoning if the submitted content posed an answer to the question but did not apply physics principles to do so. For example, if a change of motion was said to result from running into an object rather than from the application of an opposing force. Items received one of two physics reasoning codes if the answer contained any application of physics principles: descriptive or causal physics reasoning. The code of descriptive physics reasoning was assigned when the answer elaborated features of the problem but did not provide a causal explanation linking force and motion. The code of causal physics reasoning was assigned when the answer did provide a causal explanation linking force and motion. The causal physics reasoning code was considered deeper than descriptive physics reasoning because it evidenced inferential processing to relate cause and effect. See Table 2.1 for examples of each of the four codes. In order to complete the coding, raters were blinded to condition. The author coded all written participant explanations. To establish reliability, a second rater categorized all explanations for 20% of participants; participants in the 20% sample were selected randomly. An analysis of interrater reliability showed excellent agreement, with Cohen’s ĸ = .83. To examine the effects of visual concreteness and explanation scaffolding on 39 explanation quality, analyses were conducted for posttest performance in shallow reasoning, descriptive physics reasoning, and causal physics reasoning. After explanations were coded, the number of explanations in each category were divided by the total number of explanations (6) to determine the percent of explanations that fell into each type of reasoning. For example, if a student wrote one shallow explanation, their shallow score would be 17% (1/6). Similarly, if the student generated three causal explanations, their causal reasoning score would be 50% (3/6). 2.3.2.3.2 Explanation accuracy and completeness. A second rubric was developed in consultation with a physics professor to score students’ written explanations for accuracy and completeness. This rubric was applied to the subset of explanations consisting of physics reasoning (as determined by the first rubric). Each explanation was evaluated with respect to the components of a complete causal explanation: causal elements, causal interaction, and effect. For each of the three components, two scores were assigned: one for missing content (0 = not missing, 1 = missing), and one for erroneous content (0 = contains no erroneous content, 1 = contains erroneous content). Thus, a total of six dichotomous scores were assigned for each written explanation. Again, raters were blinded to condition. The author scored all physics reasoning explanations. A second rater scored all physics reasoning explanations for 20% of participants; participants in the 20% sample were selected randomly. Intraclass correlation coefficients (ICCs) ranged from .86 to .99, suggesting good to excellent agreement. ICC estimates and their 95% confidence intervals were calculated using SPSS statistical package based on a single-rating, absolute-agreement, two-way mixed-effects model. See Table 2.2 for all ICC estimates. 40 Explanations that contained no erroneous content were deemed accurate. For each explanation that was not missing content for any of the three components and which contained no erroneous content, 1 point was awarded for accuracy. For each explanation that was missing content for at least one component but contained no erroneous content, .5 point was awarded for accuracy. Accuracy was analyzed as percentage of total items. The pretest had two explanation items, for a total possible accuracy score of 2. The posttest had six explanation items, for a total possible accuracy score of 6. Explanations that were not missing any components (causal elements, causal interaction, and effect) received 1 point for completeness, regardless of accuracy. Explanations that were missing at least one component received no points for completeness. Because of the limited number of explanation items at pretest (2), explanation completeness was analyzed for posttest only. The posttest had six explanation items, for a total possible completeness score of 6. 2.3.3 Perceptions of Physics Learning A Likert scale survey administered before and after the learning intervention assessed participants' self-ratings for four items: domain enjoyment (“I enjoy learning physics.”), reliance on real-world examples (“I learn best from real-world examples.”), reliance on hands-on learning (“I learn best from experiences that are hands-on.”), and conceptual understanding for the topic being studied (“I understand the relationship between force and motion.”). The items asked for ratings of agreement with the statements using a 5-point scale, where 1 = Strongly Disagree, 2 = Disagree, 3 = Neither Agree nor Disagree, 4 = Agree, and 5 = Strongly Agree. 41 2.4 Procedure 2.4.1 Experimental Protocol The study used an opt-out consent model. In a physics class prior to the date of the study, students were provided with an information form about the research. They were given the opportunity to opt out of participation by contacting the principal investigator or their physics instructor. No students opted out. The study was run during regular class time in a physics laboratory class on the Saint Louis College of Pharmacy campus. During the study, each student used a college-provided laptop to access all experimental materials and completed the protocol individually. Within each lab section, students were randomly assigned to research conditions. Students used unique, randomly assigned identification numbers to log in to the digital learning environment. Once logged in, they were provided with a summary of the information form and a reminder of the opt-out consent process. Next, participants completed a short survey containing demographic questions and Likert scale items gauging perceptions about physics learning. There was no time limit on this survey, but most students took less than 2 minutes to complete it. Following the survey, participants were allotted 10 minutes to complete a pretest assessing prior knowledge. The assessment was self-paced, so students who completed the pretest in under 10 minutes could move on when finished. After completing the survey and pretest, the learning portion of the study began. The learning intervention was administered in two parts. For the first part of the learning intervention, all participants viewed the same visualizations (integrated concrete and symbolic representations – version one for each of the three scenarios). For the second 42 part, participants viewed visualizations that varied according to condition: students in the continued visual concreteness condition viewed a second form of integrated concrete and symbolic visual representations (version two for each of the three scenarios); students in the faded visual concreteness condition viewed symbolic-only visual representations (version three for each of the three scenarios). Throughout the entire learning intervention, explanation interface was consistent per individual student but differed between students according to their assigned explanation condition (view explanation, select explanation, or build explanation). Each part of the learning intervention consisted of instructions followed by a learning sequence for the three physics scenarios: video viewing, first diagram explanation, and second diagram explanation. There was no time limit on the video viewing page, but the only user control provided was a play button. Once the video had been viewed, the system automatically advanced to the next screen. Each video was followed by two diagram explanations, each targeting a different force-motion relationship depicted in the video (and cued by a static frame sampled from the video). For each diagram explanation, a maximum of 2.5 minutes was allotted. When users had completed the diagram explanation or when 2.5 minutes had passed, the system advanced to the next screen. Overall, 35 minutes were allotted for the learning intervention, with about half the time allotted for part one (concrete visualizations) and half for part two (condition-specific visualizations). After completing the learning portion of the experiment, participants were given a 20-minute posttest. As with the pretest, the assessment was self-paced; students who completed the posttest in under 20 minutes could move on when finished. Next, a brief 43 survey was administered to collect students’ perceptions about their own physics learning. A final screen served to debrief: participants were provided with the contact information of the principal investigator and were invited to contact the PI or their instructors with any questions. 2.4.2 Collection of Data From the Digital Learning Environment Log data were collected for various learner behaviors during the learning intervention, including time on task, video interactions, and – for participants in the select and build explanation conditions – submission accuracy. 2.4.2.1 Time on Task Two times on task variables were calculated using data automatically logged by the interface: study time (in seconds) prior to viewing the correct explanation, and reflection time (in seconds) after the correct explanation appeared on screen. Although students are likely to spend more time prior to viewing the correct explanation when more activity is required of them, increased reflection time after an explanation is provided also can be an indicator of thoughtfulness and deliberation. 2.4.2.2 Video Interactions All students watched each video one time through before being moved on to diagram explanation pages. Once on diagram explanation pages, several functions were available for further exploration of the videos: a play button allowed students to watch the video again in real time, a pause button allowed them to freeze the real-time video 44 during play, and two more buttons allowed students to advance forward or backward frame-by-frame. The interface automatically logged all interactions with timestamps. To examine condition differences in video interactions during diagram explanation, a dichotomous variable called video interaction was used to indicate whether participants interacted with the videos provided (0 = no video interactions at all, 1 = interactions with 1 or more videos during study). Another variable, interaction frequency, captured the percentage of 12 videos with which a participant interacted in any way. The variable of passive interaction was another dichotomous variable that differentiated between participants who interacted with videos exclusively through the “play” function and those who used additional functions to explore the videos (0 = used more functions than “play” to interact with the videos, and 1 = used only “play” to interact with the videos). A fourth variable, reflection interaction, captured the percentage of all videos explored by learners for which they completed a video interaction during reflection time (after the correct explanation was available). 2.4.2.3 Submission Accuracy During learning, students in the select and build explanation conditions were required to submit an explanation for each of 12 video diagrams. Visual concreteness was the same across conditions for the first six diagrams (part 1) but varied between conditions for the second six diagrams (part 2). Across conditions, a single full explanation was correct for each diagram. For the select explanation condition, achieving the correct full explanation required making one correct selection; for the build explanation condition, it required making three correct selections. In both cases, an 45 incorrect submission could be erroneous in one of the components but correct in one or two of the other components; for example, in the select condition a student could select an explanation with one incorrect explanation component or all three incorrect components. Making an incorrect submission did not mean that students’ understanding was completely erroneous and not all incorrect submissions were equivalent in terms of their relevance to the to-be-explained physics diagram. To examine how level of scaffolding impacted the accuracy of explanation selection, submission accuracy was scored on a full submission level as well as a component level (for causal elements, causal interaction, and effect components). To examine accuracy differences related to visual concreteness, the submission accuracy for each of these measures was calculated separately for part 1 and part 2 of the learning intervention. There were six explanations during part 1 and another six explanations during part 2. For part 1 as well as part 2, participants received percentage correct scores (out of 6) for the following measures: full submission, causal elements component, causal interactions component, and effect component. 46 Figure 2.1: Integrated concrete and symbolic visual representations. A series of video frames shows a hands-on physics experiment with an overlay of red arrows depicting real-time forces. The visualization is dynamic, updating over time. Figure 2.2: Video frames from three versions of the same scenario: an object falling to the ground. Each version consists of the same underlying symbolic structure but varies in concreteness. The left and middle frames vary in superficial concrete detail; the right frame contains no concrete imagery at all. Figure 2.3: The view explanation interface. When “View Explanation” button is clicked, the correct answer is revealed. 47 Figure 2.4: A system-provided correct explanation in the view explanation condition. Figure 2.5: Selecting an explanation with the select explanation interface, showing the initial view (top frame), selection choices (middle frame), and submission view (bottom frame). 48 Figure 2.6: System-provided feedback following a correct explanation submission (for both the select and build explanation interfaces). Figure 2.7: System-provided feedback following an incorrect explanation submission (for both the select and build conditions). 49 Figure 2.8: Explanation interface for the build explanation condition, showing initial view (top frame), component selection (middle frame), and submission view (bottom frame). 50 Figure 2.9: A video viewing page. A caption at top left describes the video content. The “Play Video” button is clicked to remove the mask and play the video. 51 Figure 2.10: A diagram explanation page. The diagram to be explained is pictured at top left. To the right of the diagram is a masked video, which can be activated and explored using the video controls. A the bottom of the screen is the explanation interface. A timer at top right counts down. Figure 2.11: A declarative knowledge item. 52 Figure 2.12: An application item. Figure 2.13: A difficult concrete translation item: symbolic elements in the correct answer (d) do not clearly align with information about the direction of motion. 53 Figure 2.14: A symbolic translation item. Figure 2.15: A written explanation item. 54 Table 2.1 Rubric for scoring explanation relevance and depth of reasoning Code No Content Shallow Reasoning Descriptive Physics Reasoning Causal Physics Reasoning Definition Missing or unclear response. Contains no evidence of physics application. Uses physics concepts to identify and elaborate problem features. Uses physics concepts to explain a causal relationship. Example “I don’t know.” “The ball is moving because it was thrown.” “Gravity is not equal to the tension of the spring.” “The forces acting on the squirrel are balanced which is why it is moving at a constant velocity and not accelerating.” Table 2.2 Intraclass correlation coefficients (ICCs) for scoring explanation accuracy and completeness Component Causal Elements Causal Interaction Effect Subcategory Missing Erroneous Description Does not identify the forces at play. Mentions at least one erroneous force. ICC .94 .93 Missing Erroneous Does not identify a relationship between forces. Identifies an incorrect relationship between forces. .99 .96 Missing Does not identify outcome of causal interaction. Identifies shallow or incorrect outcome of causal interaction. .90 Erroneous .86 CHAPTER 3 RESULTS A value of p = .05 was set as the alpha level for all analyses. Most dependent variables were within acceptable limits for skewness and kurtosis (between ±2). Variables that exceeded ±2 for skewness and/or kurtosis were adjusted by means of a square root transformation (with reflection for negatively skewed variables); the transformed variables were all within acceptable limits for kurtosis and skewness. See the appendix for skewness and kurtosis values for the variables that were transformed. Pre- to posttest analyses examined the percent correct for each of the five knowledge measures (to account for different numbers of items on pre- and posttests). Because of the limited number of concrete translation, symbolic translation, and explanation items at pretest, performance on easy versus difficult representation integration (concrete and symbolic translation) and additional measures of explanation quality were analyzed for posttest only. 3.1 Instructor and Lab Effects Participants were enrolled in lecture sections with one of two instructors. Each student also was enrolled in one of six laboratory sections independent of lecture section. To examine the impact of instructor and lab section on participants’ prior 56 knowledge, a multivariate analysis of variance (MANOVA) was conducted. Betweensubjects factors were instructor (1 or 2) and lab section (1, 2, 3, 4, 5, or 6). Dependent variables were pretest scores for declarative knowledge, application, concrete translation overall, symbolic translation overall, and explanation accuracy. Multivariate results showed a significant effect of instructor (F(5, 84) = 4.22, p < .01; ƞ2p = .20). Univariate results showed that students of instructor 1 significantly outperformed students of instructor 2 on several measures: declarative knowledge (Mdiff = .15; F(1, 88) = 8.64, p < .01; ƞ2p = .09); concrete translation overall (Mdiff = .11; F(1, 88) = 4.46, p = .04; ƞ2p = .05), and explanation accuracy (Mdiff = .18; F(1, 88) = 17.57, p < .001; ƞ2p = .17). A nonsignificant trend in favor of instructor 1 was found for application (Mdiff = .07; F(1, 88) = 2.69, p = .11) and symbolic translation overall (Mdiff = .08; F(1, 88) = 2.96, p = .09). The multivariate effect of lab section was not significant (F < 1), nor were there any significant univariate effects of lab section (explanation accuracy: F(5, 88) = 1.11, p = .36; all other Fs<1). There was no significant interaction between instructor and lab section (F(25, 440) = 1.01, p = .46). Overall, 62% of participants were students of instructor 1. Given the effect of instructor, it is important to note that the students in each lab section – which enrolled students from both instructors -- were randomly assigned to experimental condition. Because lecture instructors were not considered during random assignment, the proportions of students who had each instructor varied by condition. The number of students by instructor per condition is shown in Table 3.1. 57 3.2 Learning Outcomes Learning outcomes were assessed via two rounds of analysis. The first round of analysis consisted of a repeated measures multivariate analysis of variance (RMMANOVA) conducted for all knowledge assessment types: declarative knowledge, application, concrete translation overall, symbolic translation overall, and explanation accuracy. Between-subjects factors were visual concreteness (continued or faded) and explanation activity (view explanation, select explanation, or build explanation). The within-subjects factor was test time (pretest and posttest). As noted earlier, more integration and explanation items were administered at posttest than at pretest, allowing for finer-grained investigation. Thus, the second round of analysis further probed the impact of the between-subjects variables (visual concreteness: continued or faded, and explanation activity: view explanation, select explanation, or build explanation) with a MANOVA examining impact on several aspects of representation integration (easy versus difficult concrete translation, easy versus difficult symbolic translation, and incorrect symbolic translation), and a MANOVA examining impact on measures of explanation quality (descriptive physics reasoning, causal physics reasoning, and explanation completeness). Overall means and standard deviations for easy and difficult measures (concrete and symbolic translation) are shown in the appendix. One measure of explanation quality—shallow reasoning— was excluded from this analysis because it occurred rarely during student explanations (<1% of all responses received a code of “shallow reasoning”). 58 3.2.1 Pre- to Posttest Overall Knowledge Gain The RM-MANOVA showed a significant multivariate effect of test time (F(5, 90) = 8.23, p < .001; ƞ2p = .31). Means and standard deviations are shown in Table 3.2. Univariate results for test time showed significant pre- to posttest improvement for concrete translation overall (Mdiff = .13; F(1, 94) = 26.77, p < .001; ƞ2p = .22), symbolic translation overall (Mdiff = .06; F(1, 94) = 12.32, p < .01; ƞ2p = .12), and explanation accuracy (Mdiff = .06; F(1, 94) = 7.22, p < .01; ƞ2p = .07). There was no significant pre- to posttest improvement on scores for declarative knowledge (F < 1) or application (F(1, 94) = 1.42, p = .24). 3.2.2 Hypothesis 1 It was expected that learners who studied with concreteness fading would gain a deeper understanding of physics principles such that they could more successfully be interpreted and also applied to novel contexts. It was predicted that students who saw concreteness fading would demonstrate higher scores on comprehension, integration, and scientific explanation. 3.2.2.1 Concreteness Fading: Pre- to Posttest Improvement Results from the RM-MANOVA showed no significant main effect for visual concreteness (F(5, 90) = 1.53, p = .19) and no interaction effect of time and visual concreteness (F < 1). Univariate tests also showed no significant main effects of visual concreteness on comprehension (declarative knowledge: F(1, 94) = 1.72, p = .19; and, application: F < 1), on one of two integration measures (concrete translation: F(1, 94) = 59 1.16, p = .28), or on scientific explanation (explanation accuracy: F(1, 94) = 1.49, p = .23). There was a significant main effect of visual concreteness for symbolic translation (F(1, 94) = 4.08, p = .05; ƞ2p = .04). But, there were no univariate interaction effects of time and visual concreteness (explanation accuracy: F(1, 94) = 1.09, p = .30; all other Fs < 1). 3.2.2.2 Concreteness Fading: Easy Versus Difficult Integration Items and Explanation Quality MANOVA results for easy versus difficult integration showed no significant multivariate effect of visual concreteness (F(5, 90) = 1.87, p = .11). Although the multivariate results showed that the main effect of visual concreteness was not significant, univariate results showed a significant effect of visual concreteness on two integration measures: easy concrete translation and difficult symbolic translation. Students who learned with concreteness fading in the digital learning environment outperformed their counterparts on easy concrete translation (Mdiff = .10; F(1, 94) = 4.00, p < .05; ƞ2p = .04). They also performed better on difficult symbolic translation (Mdiff = .12; F(1, 94) = 4.11, p < .05; ƞ2p = .04). The effect of visual concreteness was not significant for incorrect symbolic translation (F(1, 94) = 2.57, p = .11) or for difficult concrete translation or easy symbolic translation (Fs < 1). Means and standard deviations are shown in Table 3.3. MANOVA results for explanation quality showed a nonsignificant trend for visual concreteness (F(3, 92) = 2.49, p < .07). Although the overall multivariate effect of visual concreteness did not reach the level of statistical significance, univariate results 60 showed a significant effect of visual concreteness on causal physics reasoning: students who studied with faded concreteness generated a significantly higher percentage of written explanations evidencing causal physics reasoning (Mdiff = .14; F(1, 94) = 4.44, p < .04; ƞ2p = .05). There was also a nonsignificant trend indicating that those who studied with faded concreteness tended to produce a higher proportion of written explanations that were complete (Mdiff = .09; F(1, 94) = 2.97, p < .09). There was no significant effect of visual concreteness for descriptive physics reasoning (F < 1). Means and standard deviations are shown in Table 3.4. 3.2.3 Hypothesis 2 It was hypothesized that the build explanation condition would yield the greatest improvements in comprehension and scientific explanation, and that the select explanation condition would yield better comprehension and scientific explanation than the view explanation condition. 3.2.3.1 Explanation Activity: Pre- to Posttest Improvement Results from the RM-MANOVA showed no significant main effect for explanation activity (F < 1); univariate tests also showed no significant univariate effects for explanation activity (Fs < 1). There was no significant interaction effect of time and explanation activity (F(10, 162) = 1.09, p = .37); univariate tests showed no significant interactions of time and explanation activity for comprehension (declarative knowledge: F < 1; and, application: F(2, 94) = 1.16, p = .32), one of two integration measures (symbolic translation: F < 1), and explanation quality (explanation accuracy: 61 F(2, 94) = 1.08, p = .35). There was a significant interaction of time and explanation activity for concrete translation (F(2, 94) = 3.29, p=.04; ƞ2p = .07). Means and standard deviations are shown in Table 3.2. 3.2.3.2 Concreteness Fading: Easy Versus Difficult Integration and Explanation Quality MANOVA results for easy versus difficult integration showed no significant multivariate effect of explanation activity (F < 1); univariate tests also showed no significant univariate effects for explanation activity (Fs < 1). Means and standard deviations are shown in Table 3.3. MANOVA results for explanation quality showed no main effect of explanation activity on explanation quality (F(6, 186) = 1.22, p = .30). Univariate results showed no significant effects on descriptive physics reasoning (F(2, 94) = 2.29, p = .11) or causal physics reasoning (F(2, 94) = 2.27, p = .11). There was a nonsignificant trend of explanation activity on explanation completeness (F(2, 94) = 2.92, p = .06). A pairwise comparison (with Bonferroni’s adjustment) indicated an underlying, nonsignificant trend such that participants in the view explanation condition wrote more complete written explanations at posttest than those in the build explanation condition (Mdiff = .15; p = .06). There was no significant difference in explanation completeness between the view explanation condition and the select explanation condition (Mdiff = .04; p > .99), nor between the select explanation condition and the build explanation condition (Mdiff = .11; p < .27). Means and standard deviations are shown in Table 3.4. 62 3.2.4 Exploratory Analyses: Interactions Multivariate results from the RM-MANOVA (pre- and posttest assessments) showed no significant two-way interactions for visual concreteness and explanation activity (F < 1) and no three-way interaction between visual concreteness, explanation activity, and test time (F < 1). The MANOVA on easy versus difficult integration items showed no interaction between visual concreteness and explanation activity (F < 1). Univariate results also showed no interactions between visual concreteness and explanation activity (easy symbolic translation items: F(2, 94) = 1.61, p = .21; all other Fs < 1). MANOVA results for explanation quality showed no multivariate or univariate interactions between visual concreteness and explanation activity for explanation quality (Fs < 1). 3.3 Post Hoc Exploration Because learning process differences often correspond to observable differences in learning outcomes (Butcher, 2006) a series of post hoc analyses were conducted to explore potential differences in learners’ behaviors during the study. Post hoc analyses also examined students’ perceptions of physics learning and possible interactions between students’ perceptions and learning behaviors. 3.3.1 Behaviors During Learning Students’ behaviors as they studied with the digital learning environment were examined to determine potential impact of visual concreteness and explanation activity 63 on specific behaviors associated with system usage. Submission accuracy, time on task, and video interaction measures were examined. 3.3.1.1 Submission Accuracy A repeated measures multivariate analysis of variance (RM-MANOVA) was conducted for submission accuracy on parts 1 (concrete visualizations) and 2 (condition-specific visualizations) of the learning intervention. Since students in the view explanation condition did not submit explanations during learning, only students in the select and build explanation conditions were included in this analysis (n = 66). Between-subjects factors were visual concreteness (continued or faded) and explanation activity (select explanation or build explanation). The within-subjects factor was time (part 1 and part 2). Dependent variables were percentage correct for full submissions and for each of the submission components: causal elements, causal interactions, and effects. The RM-MANOVA showed a significant multivariate effect of explanation activity (F(4, 59) = 8.50, p < .001; ƞ2p = .37). Univariate results showed a significant effect of explanation activity on the accuracy of full submissions (F(1, 62) = 7.56, p < .01; ƞ2p = .11); students who selected explanations were more accurate than students who built explanations, across time. There was no significant effect of explanation activity when analyzing accuracy of individual components: causal elements (F < 1), causal interactions (F(1, 62) = 1.27, p = .27), or effects (F(1, 62) = 1.06, p = .31). In other words, component-level accuracy did not differ between the two explanation activity conditions. 64 Results also showed a significant multivariate effect of time (F(4, 59) = 22.14, p < .001; ƞ2p = .60). Univariate results showed significant improvement from part 1 to part 2 on the accuracy of all measures: full submission (Mdiff = .14; F(1, 62) = 20.4, p < .001; ƞ2p = .25); causal elements component (Mdiff = .20; F(1, 62) = 57.0, p < .001; ƞ2p = .48); causal interaction component (Mdiff = .23; F(1, 62) = 75.6, p < .001; ƞ2p = .55); and, effect component (Mdiff = .14; F(1, 62) = 26.5, p < .001; ƞ2p = .30). There was a significant multivariate interaction of time and explanation activity (F(4, 59) = 3.57, p = .01; ƞ2p = .20). Univariate results showed a significant interaction for the causal elements component (F(1, 62) = 4.02, p < .05; ƞ2p = .06) as well as a nonsignificant trend for the causal interaction component (F(1, 62) = 3.03, p < .09). As shown in Figures 3.1 and 3.2, the build explanation condition scored lower than their select explanation condition counterparts on causal elements and causal interaction components during part 1, but evidenced higher gains in achieving comparable scores for both components at part 2. There was no significant interaction effect of time and explanation activity for full submissions (F(1, 62) = 1.28, p = .26) or for the effect component (F < 1). Results showed no significant multivariate effects of visual concreteness, no interaction between visual concreteness and explanation activity, and no interactions between time and visual concreteness or time, visual concreteness, and explanation activity (Fs < 1). Means and standard deviations are shown in Table 3.5. 65 3.3.1.2 Time on Task A MANOVA was conducted to assess how visual concreteness (continued or faded) and explanation activity (view explanation, select explanation, or build explanation) influenced students’ time on task during self-paced study. Dependent variables were study time prior to viewing the correct explanation and reflection time after the correct explanation was provided. Five participants were excluded from this analysis because technical issues resulted in missing timestamp data (n = 95). Multivariate results showed a significant main effect of explanation activity (F(4, 178) = 41.7, p < .001; ƞ2p = .48). Univariate results showed significant effects of explanation activity for study time prior to viewing the correct explanation (F(2, 89) = 100.8, p < .001; ƞ2p = .69). A pairwise comparison (with Bonferroni’s adjustment) found that students using the view explanation interface spent significantly less time in the interface prior to viewing the correct explanation than students who were tasked with selecting input (p < .001) or constructing input (p < .001). In addition, students who selected explanations spent significantly less time than students who built explanations (p < .01). Univariate results also showed a significant main effect of explanation activity on the amount of reflection time after the correct explanation was provided (F(2, 89) = 31.3, p < .001; ƞ2p = .41). Pairwise comparisons (with Bonferroni’s adjustment) found that students using the view explanation interface spent significantly more time reflecting after a correct explanation was provided than students who selected explanations (p < .001) or built explanations (p < .001). Students who built explanations spent significantly more time in reflection than students who selected explanations (p < .01). There were no main effects of visual concreteness (F(2, 88) = 1.26, p = .29), and 66 no significant interaction between visual concreteness and explanation activity (F < 1) for time on task measures. Means and standard deviations are shown in Table 3.6. 3.3.1.3 Video Interaction All participants viewed each video one time through prior to diagram explanation. But, only 89 of 100 participants interacted with at least one video during diagram explanation. A MANOVA was conducted with this subset of participants to examine the impact of visual concreteness (continued or faded) and explanation activity (view explanation, select explanation, or build explanation) on how learners interacted with the videos during diagram explanation. Dependent variables were interaction frequency, passive interaction (dichotomous), and reflection interaction. The MANOVA for video interaction showed a significant multivariate effect of explanation activity (F(6, 164) = 4.11, p < .01; ƞ2p = .13). Univariate results indicated a significant effect of explanation activity for reflection interaction (F(2, 83) = 11.9, p < .001; ƞ2p = .22). A pairwise comparison (with Bonferroni’s adjustment) found that students in the view explanation condition more often interacted with videos during reflection time than did participants in the select explanation condition (p < .001) and build explanation condition (p < .001). There was no significant difference in the frequency of reflection interaction between the select explanation condition and build explanation condition (p > .99). There was no significant univariate effect of explanation activity for interaction frequency (F < 1) or passive interaction (F(2, 83) = 2.40, p = .10). Results showed no significant multivariate main effect of visual concreteness 67 (F(3, 81) = 1.01, p = .40) and no significant multivariate interaction between visual concreteness and explanation activity (F(6, 164) = 1.40, p = .22). Overall, the findings indicated relatively low rates of video interaction during diagram explanation across conditions. Participants who interacted with videos while they explained a video frame typically did so only for about half of the videos available to them (M = .48), and 80% of the participants who interacted with videos did so only in a passive way (exclusively using the “play” function to rewatch the video). Means and standard deviations are shown in Table 3.7. 3.3.1.4 Video Interaction and Diagram Explanation Difficulty As noted above, there was relatively low video exploration during diagram explanation, across conditions. Given that reviewing videos as students explained a target frame was optional and utilized only occasionally, an important question is raised: under what circumstances did participants opt to interact with videos during diagram explanation? More specifically, was there a relationship between diagram explanation difficulty and video interaction? To address this question, a post hoc analysis compared the frequency of video interactions between “easy” and “difficult” diagram explanations. Easy and difficult categorizations were categorized in relation to full submission accuracy performance among the select explanation and build explanation conditions. Overall, students from these conditions submitted about five fully accurate submissions out of twelve. Thus, the five explanations with the highest accuracy rate (M = .55) were categorized as “easy,” while the remaining seven explanations were categorized as “difficult” (M = .30). A MANOVA was conducted to 68 examine the effects of visual concreteness (continued or faded) and explanation activity (view explanation, select explanation, or build explanation) on learners’ video interactions with respect to explanation difficulty. Dependent variables were percentage of easy videos explored, percentage of difficult videos explored, number of interactions per easy video explored, and number of interactions per difficult video explored. The MANCOVA found no significant multivariate effect of visual concreteness or explanation activity and no interaction effect between the two factors (Fs < 1). Results showed that across conditions, learners reviewed videos associated with difficult explanations (M = .48) more often than videos associated with easy explanations (M = .36). The average number of interactions per video viewed was also higher for difficult explanations (M = 2.06) than for easy explanations (M = 1.10). Means and standard deviations are shown in Table 3.8. 3.3.2 Perceptions of Physics Learning A RM-MANOVA was conducted to examine the impact of experimental conditions on students’ perceptions about learning physics, including their own comprehension. Between-subjects factors were visual concreteness (continued or faded) and explanation activity (view explanation, select explanation, or build explanation). The within-subjects factor was time (before or after the intervention). Dependent variables were self-ratings of domain enjoyment, reliance on real-world examples, reliance on hands-on learning, and conceptual understanding. The RM-MANOVA showed a significant multivariate effect of time (F(4, 91) = 7.94, p < .001; ƞ2p = .26). Univariate results showed a decrease in perceptions of domain 69 enjoyment (F(1, 94) = 7.59, p < .01; ƞ2p = .08), reliance on real-world examples (F(1, 94) = 8.74, p < .01; ƞ2p = .09), and conceptual understanding (F(1, 94) = 28.2, p < .001; ƞ2p = .23) following use of the digital learning environment. There was no significant effect of time for reliance on hands-on learning (F(1, 94) = 1.23, p = .27). The analysis found no significant multivariate effects of visual concreteness or explanation activity (Fs < 1). The main effect of time was subsumed in a significant multivariate interaction between time and explanation activity (F(8, 184) = 1.98, p = .05; ƞ2p = .08). Univariate results showed a significant interaction effect for domain enjoyment (F(2, 94) = 3.96, p = .02; ƞ2p = .08), as well as a nonsignificant trend for the interaction effect on conceptual understanding (F(2, 94) = 2.66, p = .08). There was no univariate interaction effect of time and explanation activity on students’ self-reported reliance on real-world examples (F(2, 94) = 1.46, p = .24) or hands-on learning (F < 1). Post hoc analyses examined significant interaction effects and are described in Section 3.3.2.1 Time Interaction Effects. While there was no significant multivariate interaction for time and visual concreteness (F(4, 91) = 1.52, p = .20), univariate results did show a single significant interaction: reliance on real-world examples (F(1, 94) = 4.97, p < .03; ƞ2p = .05). Both conditions reported lower reliance on real-world examples following the intervention. There were no other significant interactions for time and visual concreteness (all Fs < 1). Post hoc analyses (described below) examined the significant interaction effect. There was no interaction between visual concreteness and explanation activity (F < 1), or between time, visual concreteness, and explanation activity (F(8, 184) = 1.36, p = .22). Means and standard deviations are shown in Table 3.9. 70 3.3.2.1 Time Interaction Effects A RM-MANOVA was conducted for each explanation activity condition (view explanation, select explanation, and build explanation) with the within-subjects factor of time (before or after the intervention) to examine the effect of the experimental intervention on two dependent variables: domain enjoyment and conceptual understanding. A RM-ANOVA was conducted for each visual concreteness condition (continued concreteness and faded concreteness) with the within-subjects factor of time (before or after the intervention) to examine the effect of the intervention on one dependent variable: reliance on real-world examples. 3.3.2.1.1 Interaction effects between time and explanation activity. The RMMANOVA for the view explanation condition showed no main effect of time (F(2, 32) = 1.71, p = .20). Univariate tests showed no significant change from before to after the intervention for reported domain enjoyment (F < 1) or conceptual understanding (F(1, 33) = 2.34, p = .14). The RM-MANOVA for the select explanation condition showed a main effect of time (F(2, 31) = 5.78, p < .01; ƞ2p = .27). Univariate tests showed a significant decrease in self-reported conceptual understanding from before to after the intervention (Mdiff = -.60; F(1, 32) = 11.4, p < .01; ƞ2p = .26) and a nonsignificant trend for a decrease in domain enjoyment (Mdiff = -.22; F(1, 32) = 3.52, p = .07). The RMMANOVA for the build explanation condition also showed a main effect of time (F(2, 31) = 10.32, p < .001; ƞ2p = .40). Univariate test showed a significant decrease in selfreported domain enjoyment following the intervention (Mdiff = -.42; F(1, 32) = 10.5, p < .01; ƞ2p = .25), as well as conceptual understanding (Mdiff = -.58; F(1, 32) = 15.9, p < .001; ƞ2p = .33). 71 3.3.2.1.2 Interaction effect between time and visual concreteness. The RMANOVA for the faded concreteness condition showed a main effect of time (F(1, 48) = 8.62, p < .01; ƞ2p = .15) such that participants in this condition reported significantly lower reliance on real-world examples following the intervention (Mdiff = -.49). Participants in the continued concreteness condition did not report a significant change in their reliance on real-world examples (Mdiff = -.08; F < 1). 72 Figure 3.1: Accuracy of the causal elements component of participants’ submitted explanations for the select and build conditions during part 1 and part 2 of the diagram explanation activity. Figure 3.2: Accuracy of the causal interaction component of participants’ submitted explanations for the select and build conditions during part 1 and part 2 of the diagram explanation activity. 73 Table 3.1 Number of students by instructor per condition Condition Instructor 1 Instructor 2 1) Faded Concreteness + View Explanation 14 3 2) Continued Concreteness + View Explanation 9 8 3) Faded Concreteness + Select Explanation 10 6 4) Continued Concreteness + Select Explanation 12 5 5) Faded Concreteness + Build Explanation 9 7 6) Continued Concreteness + Build Explanation 8 9 Table 3.2 Means (and standard deviations) for pre- and posttest overall scores Test Time Pre Post Continued Concreteness View Select Build (n=17) (n=17) (n=17) .59 (.29) .54 (.27) .57 (.26) .56 (.26) .49 (.36) .66 (.20) Faded Concreteness View Select Build (n=17) (n=16) (n=16) .63 (.24) .64 (.29) .58 (.25) .68 (.23) .66 (.29) .61 (.33) Application Pre Post .52 (.19) .63 (.23) .71 (.18) .67 (.28) .66 (.23) .70 (.23) .67 (.24) .69 (.24) .61 (.23) .65 (.23) .63 (.22) .60 (.26) Concrete Translation Pre Post .37 (.26) .57 (.22) .49 (.24) .60 (.20) .53 (.21) .56 (.24) .47 (.29) .70 (.21) .54 (.32) .62 (.19) .48 (.24) .59 (.15) Symbolic Translation Pre Post .32 (.13) .41 (.18) .37 (.17) .39 (.20) .39 (.13) .42 (.16) .43 (.25) .50 (.27) .45 (.23) .52 (.25) .38 (.18) .46 (.25) Explanation Accuracy Pre Post .18 (.25) .30 (.17) .22 (.20) .35 (.20) .25 (.20) .25 (.18) .28 (.31) .35 (.21) .31 (.30) .31 (.22) .27 (.19) .30 (.18) DV Declarative Knowledge 74 Table 3.3 Means (and standard deviations) for posttest representation integration scores Continued Concreteness View Select Build (n=17) (n=17) (n=17) DV View (n=17) Faded Concreteness Select Build (n=16) (n=16) Easy Concrete Translation .79 (.30) .84 (.22) .75 (.34) .91 (.15) .89 (.18) .88 (.16) Difficult Concrete Translation .35 (.28) .37 (.31) .37 (.27) .49 (.37) .34 (.29) .31 (.27) Easy Symbolic Translation .68 (.24) .64 (.26) .74 (.24) .76 (.27) .76 (.17) .66 (.24) Difficult Symbolic Translation .21 (.20) .21 (.26) .19 (.24) .31 (.38) .35 (.37) .32 (.32) Incorrect Symbolic Translation .09 (.10) .09 (.09) .10 (.11) .06 (.08) .07 (.06) .07 (.07) Table 3.4 Means (and standard deviations) for explanation quality DV Continued Concreteness View Select Build (n=17) (n=17) (n=17) View (n=17) Faded Concreteness Select Build (n=16) (n=16) Shallow Reasoning .01 (.04) .03 (.12) .00 (.00) .00 (.00) .01 (.04) .02 (.06) Descriptive Physics Reasoning .31 (.30) .45 (.37) .49 (.29) .29 (.29) .35 (.32) .45 (.34) Causal Physics Reasoning .56 (.36) .45 (.39) .38 (.29) .70 (.30) .60 (.35) .52 (.35) Explanation Completeness .31 (.30) .26 (.30) .16 (.16) .39 (.26) .36 (.27) .25 (.25) 75 Table 3.5 Percentage means (and standard deviations) for submission accuracy (during learning) Full Submissions Part 1 Part 2 Continued Concreteness Select Build (n=17) (n=17) .43 (.23) .32 (.29) .54 (.30) .39 (.31) Causal Elements Part 1 Part 2 .61 (.24) .73 (.24) .55 (.27) .75 (.22) .58 (.20) .76 (.21) .46 (.22) .76 (.18) Causal Interaction Part 1 Part 2 .59 (.24) .75 (.21) .54 (.27) .78 (.19) .60 (.17) .80 (.20) .46 (.21) .76 (.21) Effect Part 1 Part 2 .55 (.23) .68 (.22) .50 (.20) .60 (.25) .49 (.22) .69 (.25) .48 (.19) .64 (.19) DV Faded Concreteness Select Build (n=16) (n=16) .35 (.24) .22 (.20) .60 (.31) .36 (.23) Table 3.6 Means (and standard deviations) for times on task (in seconds) Continued Concreteness View Select Build (n=16) (n=16) (n=17) DV View (n=15) Faded Concreteness Select Build (n=16) (n=15) Study Time (Prior to the Correct Explanation) 6.9 (3.8) 31.0 (10.1) 39.4 (11.2) 10.8 (4.5) 33.6 (10.9) 41.7 (10.8) Reflection Time (After the Correct Explanation) 9.2 (4.6) 3.0 (1.7) 5.7 (3.4) 9.7 (4.0) 3.9 (2.1) 5.8 (2.6) Table 3.7 Means (and standard deviations) for video interaction measures DV Continued Concreteness View Select Build (n=17) (n=15) (n=15) View (n=14) Faded Concreteness Select Build (n=14) (n=14) Interaction Frequency .48 (.36) .42 (.25) .41 (.27) .55 (.36) .52 (.30) .54 (.27) Passive Interaction .71 (.47) .73 (.46) 1.00 (.00) .71 (.47) .79 (.43) .86 (.36) Reflection Interaction .46 (.36) .04 (.11) .07 (.10) .21 (.28) .14 (.22) .09 (.27) 76 Table 3.8 Means (and standard deviations) for video interactions by diagram explanation difficulty Continued Concreteness View Select Build (n=17) (n=17) (n=17) .42 (.39) .31 (.27) .29 (.29) Faded Concreteness View Select Build (n=17) (n=16) (n=16) .36 (.40) .40 (.39) .38 (.32) .51 (.35) .41 (.30) .40 (.32) .52 (.41) .49 (.32) .54 (.33) Interactions per Easy Video Explored 1.02 (0.74) 0.94 (0.78) 0.99 (0.86) 1.45 (1.83) 0.92 (0.96) 1.29 (1.21) Interactions per Difficult Video Explored 1.81 (1.19) 2.10 (1.75) 1.56 (1.03) 2.24 (1.64) 1.97 (1.96) 2.71 (3.34) DV Percent of Easy Videos Explored Percent of Difficult Videos Explored Table 3.9 Means (and standard deviations) for user experience and metacognitive selfassessments. Scale of 1 (strongly disagree) to 5 (strongly agree). DV Time Domain Enjoyment Pre Post Reliance on Real-World Examples Pre Reliance on Hands-on Learning Pre Conceptual Understanding Pre Post Post Post Continued Concreteness View Select Build (n=17) (n=17) (n=17) 3.71 3.59 3.53 (0.92) (0.87) (0.87) 3.71 3.53 3.00 (0.92) (1.01) (1.28) View (n=17) 3.06 (1.09) 3.18 (1.02) Faded Concreteness Select Build (n=16) (n=16) 3.44 3.37 (0.89) (1.15) 3.06 3.06 (1.18) (1.34) 4.12 (0.78) 4.06 (0.66) 4.00 (1.00) 4.18 (1.13) 4.35 (0.49) 4.00 (0.87) 4.18 (0.73) 3.82 (1.07) 4.19 (0.54) 3.63 (1.36) 4.13 (0.89) 3.56 (0.89) 4.06 (0.83) 3.76 (0.83) 4.18 (0.73) 4.24 (0.75) 4.06 (1.03) 3.94 (0.90) 4.12 (0.70) 4.00 (1.00) 3.75 (0.86) 3.56 (1.09) 3.81 (0.98) 4.00 (0.52) 3.94 (0.83) 3.59 (0.71) 4.29 (0.47) 3.59 (1.28) 4.00 (0.79) 3.59 (1.00) 3.65 (0.79) 3.65 (1.12) 4.00 (0.37) 3.50 (0.89) 4.13 (0.62) 3.38 (0.96) CHAPTER 4 DISCUSSION The purpose of this study was to investigate whether deep understanding of physics principles could be enhanced through concreteness fading and explanation scaffolding during study of physics visualizations. Overall, results demonstrated that concreteness fading offered some (albeit limited) benefits to students. However, contrary to expectations, scaffolding learners’ construction of physics explanations did not improve learning. 4.1 How Does Concreteness Fading Impact Learning? Results from this study indicated that concreteness fading did not affect all learning measures, but it did provide significant benefits in terms of students’ representation integration and the quality of their physics explanations. Postlearning surveys also showed that students who learned with concreteness fading reported lower reliance on real-world examples after the intervention. The implications of these findings are discussed below. 78 4.1.1 Representation Integration Concrete translation items assessed students’ representation integration by asking them to read a text description of a real-world (concrete) situation and identify the corresponding symbolic visualization. This is a hallmark skill in physics, demonstrating clear understanding of how physics forces are applied to everyday situations. Results showed that concreteness fading supported student performance on easy concrete translation items but not on difficult concrete translation items. Thus, concreteness fading supported knowledge application only when visual connections were easier to make between the two representations (e.g., a concrete description of movement to the right corresponding to a symbolic depiction that included a right-facing arrow). At first glance, this seems like a limited success. However, it should be noted that students who only saw realistic visualizations (the continued concreteness condition) during learning did not achieve the same level of performance for even these easier items. Thus, students may need targeted support provided by concreteness fading to make initial progress in developing this integration skill. One possible explanation for this finding is that concreteness fading may make overt alignment between corresponding concrete and symbolic elements more salient, thereby freeing up cognitive resources for processing the meaning of the symbolic structure as a whole. That is, removing detail from the visualizations (as when realistic videos are replaced with symbolic representations) may make it easier for novice learners to focus on the represented forces and their relationships. This explanation is consistent with previous research showing that although novice learners may struggle to process abstract visualizations (Hegarty, Carpenter, & Just, 1991), comprehension of abstract 79 principles improves when links between multiple representations are clear (Ainsworth, 2006). In this case, concreteness fading may work for simple scenarios because these representations are easier for learners to link. Complex scenarios may require additional scaffolding or additional exposure to support these beginning learners. An alternative explanation for the benefit of concreteness fading on “easy” but not difficult concrete translation items is that concreteness fading may reinforce a faulty (intuitive) heuristic, limiting the benefit of concreteness fading to situations in which the heuristic happens to work. Chi et al. (1981) found that novice learners tend to characterize problems in terms of salient problem features rather than relevant domain principles. To the extent that concreteness fading makes the connection between concrete and symbolic elements more salient, it may lead students to misunderstand the meaning of symbolic elements (for example, encoding an arrow representing force instead as a representation of motion). Relating multiple representations is problematic if learners do not understand the form of each individual representation (Ainsworth, 2008). Student performance on symbolic translation lends support to the former explanation, providing evidence that concreteness fading can actually help students to move beyond naïve physics reasoning. Symbolic translation items assessed students’ representation integration by providing a symbolic depiction and asking them to identify corresponding text descriptions of concrete situations. This task depends not on understanding a specific context but on understanding of domain principles. Whereas a concrete situation can be represented by a single symbolic visualization, a single symbolic visualization can represent multiple concrete situations: each symbolic representation corresponds to a single correct easy interpretation (where there is overt 80 alignment between multiple representations) but also to one or more correct difficult interpretations (where the alignment between multiple representations is not obvious at the surface level). Across conditions, students were fairly successful at identifying more intuitive correct answers (M = .71), but they struggled to identify additional, difficult correct answers (M = .26). However, in the latter case, participants who studied with concreteness fading outperformed their counterparts by 13% (MFC = .33 versus MCC = .20). Thus, students who viewed concreteness fading were better able to connect visual and verbal information related to domain concepts. This finding provides additional evidence that concreteness fading helped learners to process the meaning of symbolic elements. This is consistent with the findings of Goldstone and Son (2005). They found that implementing concreteness fading helped learners to gain a better understanding of domain principles and improved their performance on items that required them to transfer what they had learned to new situations. To the extent that concreteness fading helps learners to abstract meaning from symbolic representations, they are able to apply that meaning to diverse contexts. It is important to note that while concreteness fading did improve performance on some measures of representation integration, performance was low across all conditions for difficult items in both concrete translation (M = .37) and symbolic translation (M = .26). There was room for improvement on both types of easy items as well, particularly with regard to symbolic translation. These scores highlight the difficulty of these tasks and suggest the need for additional support in making sense of physics concepts and visualizations. It may be that additional mechanisms are needed to better engage learners with the provided scaffolds. This idea will be discussed in further detail in the future 81 research section. 4.1.2 Explanation Quality Concreteness fading had some positive effects on participants’ written physics explanations at posttest, although benefits were limited to evidence of causal reasoning and explanation completeness. Overall accuracy of physics explanations was not influenced by concreteness fading of visualizations during study. Analysis of students' written physics explanations at posttest showed that students who viewed concreteness fading during study of physics visualizations were more likely to show evidence of causal reasoning. Among students who had learned with concreteness fading, the majority of their written explanations identified causal relationships between force and motion (MFC = .61 versus MCC = .46). This result has important potential implications because it demonstrates movement away from the intuitive descriptions normally employed by beginning learners in this domain—students learning physics often mischaracterize force as an attribute rather than as a causal interaction (American Association for the Advancement of Science, 2017). Increased use of causal reasoning to explain how invisible forces underlie real-world, perceptible phenomena shows development of students' knowledge application in the domain, with movement toward more expert-like understanding of force and motion. In this study, concreteness fading removed the concrete imagery from a previous visualization that integrated symbolic and concrete representations. While integrating symbolic and concrete physics representations is thought to support physics comprehension by visualizing unseen conceptual elements, it does so by adding visual content that could 82 increase the difficulty of processing depicted information, including differentiating roles and relationships between representation elements. This finding might suggest that fading out concrete detail serves to make salient the nature of the relationship between visible, real-world elements (outcomes) and invisible elements (underlying cause). This provides additional support for the hypothesis that concreteness fading facilitates principle abstraction, helping students to reason in terms of general domain principles (e.g., force as a causal mechanism) across varied contexts (e.g., different types of object motion). Concreteness fading also increased the completeness of students’ written explanations. Participants who studied with concreteness fading produced explanations that more often included all three conceptual components that are necessary for a meaningful physics explanation: causal elements, causal interaction, and resulting effect (MFC = .34 versus MCC = .25). One possible explanation for this finding is that by supporting learners in making sense of the relationships between symbolic and concrete representations, concreteness fading may reinforce comprehension of the contributions of individual pieces of information. In this way, concreteness fading may increase students’ understanding of which information is relevant, resulting in explanations that are more complete. As an example, someone who understands force as a causal element might recognize that an explanation identifying relevant forces at play is incomplete without articulating the outcome to which those forces correspond (the effect) and the mechanism by which that outcome is effected (the causal interaction). In contrast, someone who identifies force as an object attribute might think of it as a corollary to motion, failing to understand and articulate the role of force interactions. Consistent with the findings for representation integration, the benefits of 83 concreteness fading on students’ written physics explanations left room for improvement in overall explanation quality. Across conditions, both the accuracy (M = 31%) and completeness (M = 29%) of written explanations were low. Thus, while concreteness fading can help students to reason scientifically about domain principles, more support is needed for students to achieve proficiency levels consistent with what would be expected in a university-level physics course. 4.1.3 Reported Reliance on Real-World Examples Reliance on real-world examples was assessed by self-reported level of agreement with this statement: “I learn best from real-world examples.” Before the learning intervention, students across conditions generally agreed with the statement (M = 4.16, on a scale of 1 = Strongly Disagree to 5 = Strongly Agree). After the experimental intervention, students who had learned with concreteness fading reported significantly lower agreement with this statement (M = 3.67), responding more neutrally than their peers (M = 4.08). Thus, participants in the concreteness fading condition may have begun to appreciate and understand the value of analyzing abstract representations, given that participants who saw concreteness fading were better able to understand the real-world situations represented by abstract visuals. This possibility is consistent with the observed benefits of concreteness fading for representation integration items. Although intriguing, more investigation is needed to fully understand the underlying reasons for this change. 84 4.2 How Does Explanation Activity Impact Learning? The current study examined the impact of scaffolding explanation activity during learning –different levels of learner input were required during explanations that varied the amount of generation required of students. Results indicated that although explanation activity did impact student behaviors during learning, it did not have a significant impact on their learning outcomes. Postlearning surveys revealed decreases in self-reported domain enjoyment and conceptual understanding for the select and build explanation conditions. These findings are discussed in more detail below. 4.2.1 Knowledge Assessments Results showed no significant effect of explanation activity on overall pre- to posttest learning, nor on any other posttest knowledge measure. Explanation activity may have failed to produce an effect because all conditions saw the same explanation prompts (12 in all); it is possible that the prompts produced comparable cognitive processing among learners, regardless of the level of observable activity. Providing principle-based explanation prompts during a learning task has been shown to increase the relevance and richness of student reasoning as well as conceptual understanding, compared to providing no prompts (Berthold, Röder, Knörzer, Kessler, & Renkl, 2011). While research has shown that different levels of observable activity frequently correspond with different depths of cognitive processing (Chi, 2009), this is not always the case. Renkl and Atkinson (2007) noted that when explanation prompts directed learners to focus on central domain concepts and the relationships between them, they were likely to engage in the same deep cognitive processes elicited by learning environments that are overtly 85 constructive. Another possible explanation for the lack of condition differences is that all conditions (eventually) viewed the same set of correct explanations (12 in all), provided on demand in the view explanation condition and provided as feedback in the other conditions. While generative learning activities such as self-explanation have been shown to promote knowledge integration and transfer (Chi et al., 1994), providing learners with instructional content (including expert explanations) is a reliable method for facilitating fundamental understanding of domain principles (Mayer, 2004). In a study examining the effectiveness of instructional explanations during a problem-solving task, Renkl (2002) found that providing learners with instructional explanations improved posttest performance compared to providing no explanations. Thus, it may be that providing correct explanations was independently effective for learning and may even have been more powerful than any generative activity that preceded it. While it is certainly possible that comparable learning outcomes resulted from common experiences that prompted thoughtful study across conditions, overall time on task results do not indicate thoughtfulness and deliberation. On average, students completed all learning tasks in under 12 minutes – far less than the half hour allotted. Furthermore, the overall pattern of none-to-modest gains on all five pre- to posttest knowledge measures makes both of these interpretations unlikely. If prompts or provided explanations had prompted deep reflection and inference, we might expect to find significant improvements across conditions in students’ understanding of domain principles. Yet from pre- to posttest, there was no overall improvement in measures of domain knowledge, including application, and only slight improvement in explanation 86 accuracy. Thus, a third explanation for comparable learning outcomes between explanation activity conditions seems more likely: that the explanation intervention was simply too weakly enforced to realize an effect. Differences in submission accuracy, time on task, and video interactions suggest that – across the explanation conditions – students did engage differently with the content between conditions. However, these behavioral differences did not result in differential learning outcomes. The following discussion addresses key behavioral differences and why they may have failed to impact learning. 4.2.2 Behaviors During Learning The select explanation condition consistently outperformed the build explanation condition in terms of full submission accuracy during diagram explanation, but this benefit ultimately was not effective for learning. While the former result is not surprising given that there are fewer opportunities for error in the select explanation interface, it is somewhat unexpected that greater accuracy during learning did not translate to knowledge gains. One possible explanation is that making a selection from a small set of preconstructed options facilitated student “success” without prompting adequate reflection; this is supported by time on task data showing that students in the select explanation condition spent the least amount of time in reflection after the correct explanation was indicated on screen. The select condition was designed to prompt thoughtful evaluation by requiring students to make a selection from several different possible explanations. In previous research where students learned through exposure to multiple and contradictory explanations of a scientific phenomenon, the contradictions supported learning only to the extent that they successfully confused learners (D’Mello et 87 al., 2014). Confusion was induced by allowing students to agree with multiple statements. In the current study, students could only select a single correct explanation; this forced choice may have made it easier to weigh options, thereby reducing potentially beneficial confusion. In line with theory about the assistance dilemma, the select explanation condition may have provided too much support such that learning was undermined (Koedinger & Aleven, 2007). Conversely, the build explanation interface did appear to increase task difficulty, but the difficulty was not productive for learning. One indicator of difficulty for the build explanation condition was the lower accuracy of full submissions during diagram explanation. Another indicator of difficulty comes from time on task data, showing that students in the build explanation condition spent longer in study time as well as reflection time compared to their counterparts. The difference in study time prior to viewing the correct explanation could be explained by increased activity requirements relative to the other conditions. An increase in reflection time, though, could indicate thoughtfulness and deliberation; however, as evident in the comparable learning outcomes, there is no indication that the additional seconds in study and reflection were beneficial for learning. It may be that any potential benefits of this generative activity were offset by reduced support for explanation coherence. An interaction of time and explanation activity revealed a difference in performance trajectory between the select and build explanation conditions from part 1 to part 2 of the diagram explanation task: although component-level performance was comparable between conditions, the build explanation condition evidenced more significant improvement from part 1 to part 2 for the accuracy of the causal elements component. A similar but weaker trend was also seen for the 88 causal interaction component; there was no interaction of time and explanation activity for the effect component. This pattern of results could mean that for the build explanation condition, where students were tasked with constructing an explanation through the selection of individual components, the effect of explanation activity was especially powerful for improving the accuracy of specific explanation components. A common concern in science education is that student understanding consists of knowledge fragments that are not well integrated (Savinainen & Viiri, 2008). By requiring students to construct explanations through serial selection, the build explanation condition was expected to provide learners needed practice in identifying each of the relevant pieces of information needed for a complete explanation as had been found with previous research using implicit epistemic scaffolds (Sandoval, 2003). Furthermore, it was expected that requiring multiple selections would more effectively activate self-explanation processes like inference and integration by increasing generative demands relative to a single dropdown selection. Instead, the observed results provide weak evidence that building explanations may have perpetuated knowledge compartmentalization by focusing student reasoning on the content of individual components rather than on their holistic integration. Additional support for this interpretation comes from the finding that explanations written by the build explanation condition at posttest were less complete than those written by the (nongenerative) view explanation condition. The above finding is surprising because it provides evidence that the only explanation activity that provided any learning benefit was also the least generative: viewing explanations. Behavioral data shed light on how this may have occurred. The view explanation condition, which was not tasked with any overt explanation activity 89 during learning, spent significantly less time during initial study but spent longer and interacted more with videos during reflection time than did the other conditions. Given the lack of activity requirements in the view explanation condition relative to the select and build explanation conditions, it is not very surprising that students in the view explanation condition spent significantly less study time up front. However, it is interesting to note that the view explanation condition spent significantly longer than others reflecting on the correct explanations provided, and they more frequently interacted with videos during this reflection time. The explanation activity seems to have shifted their learning away from mostly focusing on ineffective interpretation of the visual materials, and instead towards increased processing of instructional content. The increased time and activity in reflection observed for the view explanation condition is similar to how learning appears to be fostered through worked examples. Worked example research has found that for complex learning tasks, novices struggle to solve problems on their own and often resort to ineffective strategies. For such tasks and domains, learners have been shown to benefit from exposure to worked examples that are progressively faded (Atkinson, Renkl, & Merrill, 2003), thereby scaffolding students’ knowledge construction and problem-solving skill (Paas & Van Gog, 2006). Time spent processing a worked example generally ultimately helps reduce the cognitive demands of applying domain principles (Sweller, 1988). In the current study, the previously noted trend such that students in the view explanation condition produced more complete written explanations than students in the build explanation condition provides limited evidence that increased reflection time improved learners’ scientific reasoning. However, no other learning benefits were observed. 90 One possible explanation for the overall ineffectiveness of increased reflection time and activity is that students in the view explanation condition, like their counterparts, simply did not spend sufficient time processing the provided explanations. On average, the view explanation condition spent under 10 seconds in reflection time per correct explanation. This suggests that students were more oriented toward achieving results and progressing through the system (e.g., revealing and reading the correct explanation) than on processing the content to understand concepts and the relationships between them (Reiser, 2004). A second, related explanation is that even though these learners interacted with videos more often during reflection than did other conditions, they did not engage in ways that were effective for learning (as evidenced by the lack of condition differences in knowledge gains). Interactive learning environments that provide multiple representations of information – including text, images, and video – can increase the extraneous cognitive demands of a learning task if students are not adequately supported in making use of each representation and moving between them (Moreno & Mayer, 2007), inhibiting learning. The results of this study indicate the need for additional support to facilitate effective reflection and use of video content during evaluation of provided explanations. 4.2.2.1 Video Interactions As noted above, student interactions with videos were ineffective for learning, and this was true across conditions. All students viewed each video one time prior to diagram explanation, but they also had the option of examining videos further during study time and reflection. Previous research in which students had access to videos 91 during a learning task showed that learners engaged in spontaneous video interactions to support their information-seeking (Merkt, Weigand, Heier, & Schwan, 2011). In the current study, there was evidence to suggest that videos were indeed spontaneously consulted in an attempt to support learning. Students tended not to re-view all videos available to them; instead, they selectively interacted with videos during the explanation task – interacting only about half the time. Across conditions, videos that were accessed by students more often were associated with difficult diagram explanations; students conducted a higher number of interactions with these videos than with videos that were associated with easier diagram explanations. This pattern of behavior suggests two things: 1) students did seek to use the videos to support their learning; but, 2) the ways in which students interacted with videos were not effective for their learning (given that students performed poorly on difficult diagram explanations). It may be that the type of information provided in the videos was too difficult to connect to its associated static diagram. Merkt et al. (2011) found that students made use of videos in pursuit of declarative knowledge, whereas in the current study videos provided dynamic representations that provided no explicit declarative or conceptual knowledge and required interpretation. Additional scaffolds that support learners in making sense of the dynamic information likely are needed. 4.2.3 Perceptions of Physics Learning Self-assessments collected before and after the learning intervention indicate that more generative explanation activities had a negative impact on students’ reported enjoyment of the domain. Following the study, students in the build explanation 92 condition reported significantly lower agreement with the statement “I enjoy learning physics”; a similar trend was evident for the select explanation condition. In contrast, there was no change in reported domain enjoyment for the view explanation condition. More investigation is needed to understand how the learning experiences of each condition contributed to these reported outcomes. Previous research suggests that positive emotions including enjoyment of learning are affected by student achievement (Pekrun, Goetz, Titz, & Perry, 2002); so, it is possible that the opportunity to err and the frequency of its occurrence among the build explanation condition (and to a lesser extent, among the select explanation condition) reduced students’ sense of enjoyment. But, lower domain enjoyment due to hampered performance is not necessarily detrimental to student learning. Research has shown that while a certain level of difficulty may appear to impair achievement during learning, it can promote long-term comprehension (Bjork, Dunlosky, & Kornell, 2013). That said, a decrease in positive emotions after completing a difficult task could indicate that students tend toward a performance goal orientation rather than a learning goal orientation (Steele-Johnson, Beauregard, Hoover, & Schmidt, 2000). Students with a learning goal orientation operate from the perspective that competence is achievable through practice, while students with a performance goal orientation are more focused on high achievement and avoiding error (Nicholls, 1984); not surprisingly, the latter orientation is associated with poorer learning outcomes (Steele-Johnson et al., 2000). If decreased enjoyment in the current study corresponded to a performance goal orientation, this would suggest the need for additional support to help students see the value of increasing in competence over time through practice. Explanation activity also decreased self-reported conceptual understanding for 93 students in the build and select explanation conditions, though not for the view explanation condition. This finding is interesting because knowledge assessments did not show corresponding decreases in understanding; on the contrary, performance improved or stayed the same from pre- to posttest across conditions. Still, overall performance was mediocre, leaving significant room for improvement. Thus, lower assessments of conceptual understanding following the intervention likely were more accurate than previous, higher assessments of self-knowledge. This is consistent with previous research that found that actively generating an explanation was more effective for metacomprehension accuracy than expecting to be required to generate an explanation (Fukaya, 2013). When the accuracy of student assessments about their own understanding increases, this provides evidence of metacognitive calibration (Hacker, Bol, & Keener, 2008). In the current study, metacognitive calibration may have resulted from cognitive processing associated with actively providing explanation input to the system or with visibly confronting errors committed, two learning experiences not supported in the view explanation condition. Such experiences may also have served to adjust students’ frame of reference for interpreting the meaning of “force and motion” (Azevedo, 2009). Although decreases in self-assessed conceptual understanding were not associated with condition differences in learning outcomes, it may help to explain why the explanation activities were ineffective for learning. To the extent that students’ previous self-assessments were initially overinflated, they may have been more likely to engage in knowledge retrieval (assuming mastery) rather than knowledge acquisition or construction (Hacker et al., 2008). However, even though metacognitive calibration may result from an ineffective learning experience (as in the current study), it can help 94 students identify gaps in their knowledge and serve as an important precursor to future learning (Bjork et al., 2013; Pintrich, 2002). More investigation is needed to identify whether explanation activities experienced by the build and select explanation conditions would more effectively support subsequent learning than an activity involving only viewing provided explanations. 4.3 Does the Combination of Concreteness Fading and Explanation Activity Impact Learning? There was not a specific hypothesis about the combination of concreteness fading and explanation activity in the current study, but this question was addressed via exploratory analyses. Although previous research has shown that increasing the generative nature of a learning activity promotes deep cognitive processes (Chi, 2009), the build explanation condition in the current study did not enhance the benefit of learning with concreteness fading. It is worthwhile to question why a generative explanation activity with visual examples (that included abstract notations from physics diagrams) failed to combine to produce meaningful learning in this study. One possibility is that generation of explanations may be more effective in domains where common misconceptions do not limit the depth and accuracy of students’ explanations of visual materials. Indeed, other research has highlighted the need to scaffold and constrain the accuracy of explanations when students’ situation models are flawed. Students tasked with self-explanation of complex content may inadvertently reinforce misconceptions by verbalizing them (Rittle-Johnson & Loehr, 2016), making explanation of expert solutions that are (by definition) highly accurate and well-articulated an effective alternative 95 (Berthold & Renkl, 2010). The current study attempted to reduce the demands of fromscratch generation by constraining input while still requiring some form of constructive activity, but found little evidence that the drop-down building activity promoted effective generative processing. It also could be assumed that concreteness fading, thought to support principle abstraction, may promote integration and inferential processing for students who generated explanations. However, the current results showed no evidence of this. It may be the case that students’ prior knowledge was insufficient for connecting conceptually meaningful explanations with visual content. Jonassen and Ionas (2008) suggest that the effectiveness of explanation in supporting causal reasoning is limited by prerequisite understanding of the roles and relationships between each component in a causal relationship. Thus, it may be necessary to use generative activities that more clearly highlight connections between global explanations and conceptual information in visual materials. Results suggest that, with the current instructional materials, the effects of concreteness fading were independent of learners’ explanatory activity in promoting learning outcomes. 4.4 Limitations One potential limitation of this research is that the participant sample consisted of students currently enrolled in a physics course who had already begun learning about the topic at hand. While the use of authentic participants in a realistic context lends validity to the learning context, it is worth noting that pretest performance on measures of domain knowledge was relatively high. For example, on application items drawn from the AAAS 96 item bank, the overall mean score at pretest was 63% correct – much higher than the mean of 36% for the same set of items among a national sample of secondary students (American Association for the Advancement of Science, 2017). These items are considered measures of deeper knowledge, since they require students to apply domain concepts to varied situations. Given that all students were enrolled in a degree program in the sciences, additional research with participants who have less prior knowledge and are more novice in STEM domains may be warranted. Another limitation of the study was the limited amount of time that participants spent using the experimental interface. As appropriate to the authentic population and context, students self-paced their study with the learning materials in the experimental interface. However, students in this study spent far less than the allotted time studying and explaining each diagram; on average, the diagram explanation portion of the experimental protocol was completed in less than 12 minutes. It seems likely that the exposure to the intervention was insufficient for effective learning; increased exposure— via system-controlled pacing or increased numbers of examples (diagrams) to be explained —may be necessary to see full benefits, particularly when generative explanations are combined with feedback. Future research should increase exposure to the interventions to more fully understand its potential effects. It is also important to note that participants who were students of instructor 1 performed better at pretest than did students of instructor 2; it is possible that preexisting differences may have influenced results. In this study, it was too complicated to account for nesting given that each participant attended one of three lecture sections taught by one of two instructors and also attended one of six laboratory sections taught by one of three 97 instructors. Because lecture instructors were not considered during random assignment, the distribution of each instructor’s students across conditions is not proportionate. The implications of instructor assignment on knowledge and behavioral outcomes is unknown. Another limitation of the study is high correlation between some of the dependent variables used. Specifically, descriptive and causal physics reasoning had a correlation of -.85. Participant scores for descriptive and causal physics reasoning were mutually exclusive and additive (percents of total); thus, they should not both have been included as dependent measures of a MANOVA. Another high correlation found was between the percentage of easy and difficult videos explored (r = .80). Although these values were not mutually exclusive, it is not surprising that participants would evidence consistency in video exploration frequency. Dependent variable correlations for all MANOVAs are shown in the Appendix. 4.5 Future Research Directions Previous research indicates that students’ naïve conceptions about physics phenomena are persistent and difficult to change (Vosniadou, 2002). Although concreteness fading shows promise as a strategy for facilitating principle abstraction and comprehension, performance on nearly all knowledge assessments was disappointing and suggests the need for additional exposure and/or scaffolds that increase students’ processing of coherent conceptual explanations that connect to visual examples in physics. One area in which students may benefit from additional intervention is in their 98 engagement with multimedia content, including diagrams and videos. Although evidence suggests that students did interact with videos more often when they encountered difficult problems, their increased engagement did not show corresponding improvements to learning outcomes. Future research might examine how to better scaffold student interactions with the provided visual representations. Previous research with static and dynamic diagrams showed that when students used diagrams as a locus for problemsolving interactions, their comprehension improved—particularly when interactions served to link visual elements and domain principles (Butcher & Aleven, 2013). Thus, generative actions in a learning system with important diagrammatic content may be more useful when those actions help students engage with and process information contained within the visual representations. Future research should investigate the use of interactions with visual content during physics explanation activities, whether it be the symbolic elements of a static diagram or with the dynamic video content intended to contextualize that diagram. Another question for future research is how to support effective processing of provided instructional content (i.e., correct explanations). In the current study, selecting and building explanations to explain physics diagrams were no more effective for learning than simply viewing provided explanations. In fact, there was some evidence to suggest that simply viewing provided explanations without engaging in prior generative activity may have shifted students’ approach away from the ineffective interpretation of visual representations (limited by students’ prior knowledge) toward more reflective processing of accurate instructional content. In turn, this increase in reflective processing may have supported students in subsequently producing written explanations that were 99 more complete. However, advantages in reflection and explanation completeness were slight; students across conditions evidenced minimal interactivity and a tendency to complete problems much faster than anticipated during interface design. Future research could investigate how to promote more deliberate and thoughtful reflection on provided instructional content. For example, a future interface may use the study of provided explanations as a learning activity in and of itself rather than as an affordance for validation or corrective feedback. One approach that has been shown to help students gain comprehension of provided content while also increasing their proficiency in applying domain principles to specific problems is faded or worked examples. Worked examples, which progressively remove scaffolds during problem-solving, have been shown to support subsequent problem-solving (Renkl, Atkinson, & Maier, 2000). Worked examples are thought to be effective for learning in that they minimize the initial cognitive demands of a learning task, gradually increasing generative requirements as students gain in competence (Paas & Van Gog, 2006; Sweller, 1988). Thus, instead of prompting generative effort first followed by reflection as in the current study, future work might investigate a worked example approach to scientific explanation: reflection of expert explanations first, with increasingly generative activity as students gain competence with diagrams and explanations. A third area for future research could be in using students’ performance on explanation selection or construction as a method to help learners accurately predict their own learning needs. In the current study, neither selecting nor building physics explanations yielded learning gains, but both served to identify errors or gaps in student knowledge (e.g., inaccuracies at the component and full submission level), and both 100 appeared to decrease learners’ self-assessments of their conceptual understanding. To the extent that this change in self-assessment represented true metacognitive calibration, it may have had an impact on learners’ response to subsequent learning opportunities (Hacker et al., 2008; Pintrich, 2002). Thus, future work might examine the outcome of using student explanation activities with a different instructional goal: as a formative assessment activity to foster accurate metacomprehension and promote the effectiveness of follow-up learning activities through increased self-regulation (Butler & Winne, 1995; Nicol & Macfarlane‐Dick, 2006). 4.6 Conclusions Previous research by the author found that presenting integrated concrete and symbolic representations during study of real-world physics experiments improved students’ representation integration and the relevance of their verbalized explanations, but it did not significantly improve their conceptual understanding (Davies & Butcher, in preparation). The current study examined the utility of concreteness fading and explanation scaffolding for augmenting the benefits of learning with integrated representations, with the expectation that both representation integration and conceptual understanding would be enhanced; hypotheses were only partially supported. Results indicated that concreteness fading may function to make the relationships between concrete and symbolic representation elements more salient, yielding improvements in representation integration and in the quality of students’ scientific explanations. However, there was no evidence that concreteness fading improved the accuracy or depth of students’ comprehension of domain principles. In contrast, scaffolding students’ 101 explanation activities during learning did not provide any learning benefits—if anything, shifting students’ explanation activity away from interpretation and generation and toward evaluation and reflection via provided instructional content was more favorably received by the learner and was equally effective for learning outcomes. Thus, interventions may be best focused on providing students with coherent explanations until they can generate such content more independently, using online interactions to enhance reflection and processing of provided content until mastery. Future research should examine how students can more effectively be supported in engaging with conceptual principles to overcome naïve reasoning and move towards more expert understanding. APPENDIX 103 Table A.1 Skewness and kurtosis values of nonnormally distributed variables before and after square root transformation Skewness Kurtosis Variable Before After Before After Easy concrete translation* -1.60 1.41 2.25 1.38 Reflection time after the correct explanation 1.36 .40 3.22 .09 Reflection interaction 1.81 .87 2.56 -.56 Interactions per Easy Video Explored 1.81 -.09 4.92 -.69 Interactions per Difficult Video Explored 2.64 .06 9.75 1.36 Reliance on real-world examples (before)* -1.16 .44 2.91 .49 Reliance on real-world examples (after)* -1.22 .63 1.46 .23 *Due to negative skew, these variables were reflected before the square root transformation was performed. Table A.2 Means (and standard deviations) for easy and difficult integration items Type of Item Concrete Translation Symbolic Translation Easy .84 (.24) .71 (.24) Difficult .37 (.30) .26 (.30) 104 Application, Pretest .389** Concrete Translation (overall), Pretest .262** .523** Symbolic Translation (overall), Pretest .281** .384** .429** Explanation Accuracy, Pretest .355** .429** .386** .269** Declarative Knowledge, Posttest .562** .418** .323** .287** .431** Application, Posttest .393** .650** .392** .349** .391** .435** Concrete Translation (overall), Posttest .438** .495** .501** .332** .357** .439** .494** Symbolic Translation (overall), Posttest Concrete Translation (overall), Posttest Application, Posttest Declarative Knowledge, Posttest Explanation Accuracy, Pretest Symbolic Translation (overall), Pretest Concrete Translation (overall), Pretest Application, Pretest Declarative Knowledge, Pretest Table A.3 Correlations of dependent variables from RM-MANOVA for pre- to posttest overall knowledge measures Symbolic Translation (overall), Posttest .390** .477** .544** .642** .416** .451** .472** .547** Explanation Accuracy, Posttest * p values less than .05 ** p values less than .01 .269** .438** .381** .290** .496** .437** .531** .586** .467** 105 Difficult Concrete Translation Easy Symbolic Translation Difficult Symbolic Translation Easy Symbolic Translation Difficult Concrete Translation Easy Concrete Translation Table A.4 Correlations of dependent variables from MANOVA for posttest representation integration .151 .331** .319** Difficult Symbolic Translation .146 .503** .258** Incorrect Symbolic Translation -.296** -.370** -.592** -.301** * p values less than .05 ** p values less than .01 Causal Physics Reasoning -.847** Explanation Completeness -.618** * p values less than .05 ** p values less than .01 Causal Physics Reasoning Descriptive Physics Reasoning Table A.5 Correlations of dependent variables from MANOVA for posttest explanation quality .744** 106 Reflection Time Table A.6 Correlation of dependent variables from MANOVA for video study time on task Study Time -.247* * p values less than .05 ** p values less than .01 Passive Interaction .223* Reflection Interaction -.162 * p values less than .05 ** p values less than .01 Passive Interaction Interaction Frequency Table A.7 Correlations of dependent variables from MANOVA for video interactions .123 107 Percent of Difficult Videos Explored .800** Number of Interactions per Easy Video Explored .480** .423** .165 .269** Number of Interactions per Difficult Video Explored * p values less than .05 ** p values less than .01 Number of Interactions per Easy Video Explored Percent of Difficult Videos Explored Percent of Easy Videos Explored Table A.8 Correlations of dependent variables from MANOVA for video interactions by explanation difficulty .336** REFERENCES Ainsworth, S. (2006). DeFT: A conceptual framework for considering learning with multiple representations. Learning and Instruction, 16(3), 183-198. Ainsworth, S. (2008). The educational value of multiple-representations when learning complex scientific concepts. In Visualization: Theory and practice in science education (pp. 191-208). Dordrecht, Netherlands: Springer. Ainsworth, S., & Loizou, A. T. (2003). The effects of self-explaining when learning with text or diagrams. Cognitive Science, 27(4), 669-681. Aleven, V., & Koedinger, K. R. (2000). The need for tutorial dialog to support selfexplanation. In Building Dialogue Systems for Tutorial Applications, Papers of the 2000 AAAI Fall Symposium (pp. 65-73). Aleven, V., & Koedinger, K. R. (2002). An effective metacognitive strategy: Learning by doing and explaining with a computer-based Cognitive Tutor. Cognitive Science, 26(2), 147-179. Aleven, V., Ogan, A., Popescu, O., Torrey, C., & Koedinger, K. (2004). Evaluating the effectiveness of a tutorial dialogue system for self-explanation. In International conference on intelligent tutoring systems (pp. 443-454). Berlin, Germany: Springer. American Association for the Advancement of Science. (2017). AAAS science assessment: Force and motion. Retrieved 2/6/2017, from http://assessment.aaas.org/misconceptions/FMM107/259. Atkinson, R. K., Renkl, A., & Merrill, M. M. (2003). Transitioning from studying examples to solving problems: Effects of self-explanation prompts and fading worked-out steps. Journal of Educational Psychology, 95(4), 774-783. Azevedo, R. (2009). Theoretical, conceptual, methodological, and instructional issues in research on metacognition and self-regulated learning: A discussion. Metacognition and Learning, 4(1), 87-95. Berland, L. K., & Reiser, B. J. (2009). Making sense of argumentation and explanation. Science Education, 93(1), 26-55. Berthold, K., & Renkl, A. (2010). How to foster active processing of explanations in instructional communication. Educational Psychology Review, 22(1), 25-40. 109 Berthold, K., Röder, H., Knörzer, D., Kessler, W., & Renkl, A. (2011). The double-edged effects of explanation prompts. Computers in Human Behavior, 27(1), 69-75. Bjork, R. A., Dunlosky, J., & Kornell, N. (2013). Self-regulated learning: Beliefs, techniques, and illusions. Annual Review of Psychology, 64, 417-444. Booth, J. L., Lange, K. E., Koedinger, K. R., & Newton, K. J. (2013). Using example problems to improve student learning in algebra: Differentiating between correct and incorrect examples. Learning and Instruction, 25, 24-34. Brown, D. E. (1989). Students' concept of force: The importance of understanding Newton's third law. Physics Education, 24(6), 353. Butcher, K. R. (2006). Learning from text with diagrams: Promoting mental model development and inference generation. Journal of Educational Psychology, 98(1), 182-197. Butcher, K. R., & Aleven, V. (2013). Using student interactions to foster rule–diagram mapping during problem solving in an intelligent tutoring system. Journal of Educational Psychology, 105(4), 988-1009. Butler, D. L., & Winne, P. H. (1995). Feedback and self-regulated learning: A theoretical synthesis. Review of Educational Research, 65(3), 245-281. Chi, M. T. H. (2009). Active-constructive-interactive: A conceptual framework for differentiating learning activities. Topics in Cognitive Science, 1(1), 73-105. Chi, M. T. H., Bassok, M., Lewis, M. W., Reimann, P., & Glaser, R. (1989). Self‐ explanations: How students study and use examples in learning to solve problems. Cognitive Science, 13(2), 145-182. Chi, M. T. H., De Leeuw, N., Chiu, M.-H., & Lavancher, C. (1994). Eliciting selfexplanations improves understanding. Cognitive Science, 18(3), 439-477. Chi, M. T. H., Feltovich, P. J., & Glaser, R. (1981). Categorization and representation of physics problems by experts and novices. Cognitive Science, 5(2), 121-152. Cho, Y. H., & Jonassen, D. H. (2012). Learning by self-explaining causal diagrams in high-school biology. Asia Pacific Education Review, 13(1), 171-184. Clark, R. C., & Mayer, R. E. (2016). E-learning and the science of instruction: Proven guidelines for consumers and designers of multimedia learning. Hoboken, New Jersey: John Wiley & Sons. Clements, D. H. (2000). ‘Concrete’ manipulatives, concrete ideas. Contemporary Issues in Early Childhood, 1(1), 45-60. Conati, C., & Vanlehn, K. (2000). Toward computer-based support of meta-cognitive 110 skills: A computational framework to coach self-explanation. International Journal of Artificial Intelligence in Education, 11, 389-415. D’Mello, S., Lehman, B., Pekrun, R., & Graesser, A. (2014). Confusion can be beneficial for learning. Learning and Instruction, 29, 153-170. Davies, S., & Butcher, K. R. (in preparation). Using dynamically annotated videos to support learning from hands-on experiments in physics. de Koning, B. B., Tabbers, H. K., Rikers, R. M., & Paas, F. (2010). Learning by generating vs. receiving instructional explanations: Two approaches to enhance attention cueing in animations. Computers & Education, 55(2), 681-691. de Koning, B. B., Tabbers, H. K., Rikers, R. M. J. P., & Paas, F. (2011). Improved effectiveness of cueing by self-explanations when learning from a complex animation. Applied Cognitive Psychology, 25(2), 183-194. Donovan, M. S., & Bradford, J. D. (2005). How students learn: History, mathematics, and science in the classroom. Washington, DC: National Academies Press. Fukaya, T. (2013). Explanation generation, not explanation expectancy, improves metacomprehension accuracy. Metacognition and Learning, 8(1), 1-18. Ginns, P. (2006). Integrating information: A meta-analysis of the spatial contiguity and temporal contiguity effects. Learning and Instruction, 16(6), 511-525. Goldstone, R. L., & Sakamoto, Y. (2003). The transfer of abstract principles governing complex adaptive systems. Cognitive Psychology, 46(4), 414-466. Goldstone, R. L., & Son, J. Y. (2005). The transfer of scientific principles using concrete and idealized simulations. Journal of the Learning Sciences, 14(1), 69-110. Grant, E. R., & Spivey, M. J. (2003). Eye movements and problem solving guiding attention guides thought. Psychological Science, 14(5), 462-466. Große, C. S., & Renkl, A. (2006). Effects of multiple solution methods in mathematics learning. Learning and Instruction, 16(2), 122-138. Hacker, D. J., Bol, L., & Keener, M. C. (2008). Metacognition in education: A focus on calibration. In J. Dunlosky & R.A. Bjork (Eds.), Handbook of metamemory and memory (429-455). New York, NY: Psychology Press. Hake, R. R. (1998). Interactive-engagement versus traditional methods: A six-thousandstudent survey of mechanics test data for introductory physics courses. American Journal of Physics, 66(1), 64-74. Hausmann, R. G., & Chi, M. H. (2002). Can a computer interface support self-explaining. Cognitive Technology, 7(1), 4-14. 111 Hegarty, M., Carpenter, P. A., & Just, M. A. (1991). Diagrams in the comprehension of scientific texts. In R. Barr, M. L. Kamil, P. B. Mosenthal, & P. D. Pearson (Eds.), Handbook of reading research, volume 2 (641-668). Hillsdale, NJ: Lawrence Erlbaum Associates, Inc. Hegarty, M., & Just, M. A. (1993). Constructing mental models of machines from text and diagrams. Journal of Memory and Language, 32(6), 717-742. Hofstein, A., & Lunetta, V. N. (2004). The laboratory in science education: Foundations for the twenty-first century. Science Education, 88(1), 28-54. Jonassen, D. H., & Ionas, I. G. (2008). Designing effective supports for causal reasoning. Educational Technology Research and Development, 56(3), 287-308. Joseph, J. H., & Dwyer, F. M. (1984). The effects of prior knowledge, presentation mode, and visual realism on student achievement. The Journal of Experimental Education, 52(2), 110-121. Kim, E., & Pak, S.-J. (2002). Students do not overcome conceptual difficulties after solving 1000 traditional problems. American Journal of Physics, 70(7), 759-765. Klein, P. D. (2004). Constructing scientific explanations through writing. Instructional Science, 32(3), 191-231. Koedinger, K. R., & Aleven, V. (2007). Exploring the assistance dilemma in experiments with cognitive tutors. Educational Psychology Review, 19(3), 239-264. Kontra, C., Lyons, D. J., Fischer, S. M., & Beilock, S. L. (2015). Physical experience enhances science learning. Psychological Science, 26(6), 737-749. Kozma, R. B. (1999). Students collaborating with computer models and physical experiments. In Proceedings of the 1999 Conference on Computer Support for Collaborative Learning (p. 39). International Society of the Learning Sciences. Kuhn, D., & Katz, J. (2009). Are self-explanations always beneficial? Journal of Experimental Child Psychology, 103(3), 386-394. Larkin, J. H., & Simon, H. A. (1987). Why a diagram is (sometimes) worth ten thousand words. Cognitive Science, 11(1), 65-100. Litman, D. J., Rosé, C. P., Forbes-Riley, K., VanLehn, K., Bhembe, D., & Silliman, S. (2006). Spoken versus typed human and computer dialogue tutoring. International Journal of Artificial Intelligence in Education, 16(2), 145-170. Mathewson, J. H. (1999). Visual-spatial thinking: An aspect of science overlooked by educators. Science Education, 83(1), 33-54. Mayer, R. E. (2002). Multimedia learning. Psychology of Learning and Motivation, 41, 112 85-139. Mayer, R. E. (2004). Should there be a three-strikes rule against pure discovery learning? American Psychologist, 59(1), 14. McNeill, K. L., Lizotte, D. J., Krajcik, J., & Marx, R. W. (2006). Supporting students' construction of scientific explanations by fading scaffolds in instructional materials. The Journal of the Learning Sciences, 15(2), 153-191. Merkt, M., Weigand, S., Heier, A., & Schwan, S. (2011). Learning with videos vs. learning with print: The role of interactive features. Learning and Instruction, 21(6), 687-704. Moreno, R., & Mayer, R. (2007). Interactive multimodal learning environments. Educational Psychology Review, 19(3), 309-326. Moreno, R., Ozogul, G., & Reisslein, M. (2011). Teaching with concrete and abstract visual representations: Effects on students' problem solving, problem representations, and learning perceptions. Journal of Educational Psychology, 103(1), 32-47. Nicholls, J. G. (1984). Achievement motivation: Conceptions of ability, subjective experience, task choice, and performance. Psychological Review, 91(3), 328-346. Nicol, D. J., & Macfarlane‐Dick, D. (2006). Formative assessment and self‐regulated learning: A model and seven principles of good feedback practice. Studies in Higher Education, 31(2), 199-218. Nilsson, P. (2013). Challenges in seeing data as useful evidence in making predictions on the probability of a real-world phenomenon. Statistics Education Research Journal, 12(2), 71-83. Paas, F., & Van Gog, T. (2006). Optimising worked example instruction: Different ways to increase germane cognitive load. Learning and Instruction, 16(2), 87-91. Paivio, A. (1990). Mental representations: A dual coding approach. New York, NY: Oxford University Press. Pekrun, R., Goetz, T., Titz, W., & Perry, R. P. (2002). Academic emotions in students' self-regulated learning and achievement: A program of qualitative and quantitative research. Educational Psychologist, 37(2), 91-105. Pintrich, P. R. (2002). The role of metacognitive knowledge in learning, teaching, and assessing. Theory Into Practice, 41(4), 219-225. Reiser, B. J. (2004). Scaffolding complex learning: The mechanisms of structuring and problematizing student work. The Journal of the Learning Sciences, 13(3), 273304. 113 Renkl, A. (1997). Learning from worked‐out examples: A study on individual differences. Cognitive Science, 21(1), 1-29. Renkl, A. (2002). Worked-out examples: Instructional explanations support learning by self-explanations. Learning and Instruction, 12(5), 529-556. Renkl, A., & Atkinson, R. K. (2007). Interactive learning environments: Contemporary issues and trends. An introduction to the special issue. Educational Psychology Review, 19(3), 235-238. Renkl, A., Atkinson, R. K., & Maier, U. H. (2000). From studying examples to solving problem: Fading worked-out solution steps helps learning. In Proceedings of the 22nd Annual Conference of the Cognitive Science Society (pp. 393-398). Mahwah, New Jersey: Lawrence Erlbaum Associates, Inc. Rittle-Johnson, B., & Loehr, A. M. (2016). Eliciting explanations: Constraints on when self-explanation aids learning. Psychonomic Bulletin & Review, 24(5), 1501-1510. Roy, M., & Chi, M. T. (2005). The self-explanation principle in multimedia learning. The Cambridge Handbook of Multimedia Learning, 271-286. Ruiz-Primo, M. A., Li, M., Tsai, S.-P., & Schneider, J. (2010). Testing one premise of scientific inquiry in science classrooms: Examining students' scientific explanations and student learning. Journal of Research in Science Teaching, The Official Journal of the National Association for Research in Science Teaching, 47(5), 583-608. Sandoval, W. A. (2003). Conceptual and epistemic aspects of students' scientific explanations. The Journal of the Learning Sciences, 12(1), 5-51. Savinainen, A., Scott, P., & Viiri, J. (2005). Using a bridging representation and social interactions to foster conceptual change: Designing and evaluating an instructional sequence for Newton's third law. Science Education, 89(2), 175-195. Savinainen, A., & Viiri, J. (2008). The force concept inventory as a measure of students conceptual coherence. International Journal of Science and Mathematics Education, 6(4), 719-740. Scheiter, K., Gerjets, P., Huk, T., Imhof, B., & Kammerer, Y. (2009). The effects of realism in learning with dynamic visualizations. Learning and Instruction, 19(6), 481-494. Schnotz, W., & Bannert, M. (2003). Construction and interference in learning from multiple representation. Learning and Instruction, 13(2), 141-156. Schworm, S., & Renkl, A. (2006). Computer-supported example-based learning: When instructional explanations reduce self-explanations. Computers & Education, 46(4), 426-445. 114 Sloutsky, V. M., Kaminski, J. A., & Heckler, A. F. (2005). The advantage of simple symbols for learning and transfer. Psychonomic Bulletin & Review, 12(3), 508513. Steele-Johnson, D., Beauregard, R. S., Hoover, P. B., & Schmidt, A. M. (2000). Goal orientation and task demand effects on motivation, affect, and performance. Journal of Applied Psychology, 85(5), 724. Sweller, J. (1988). Cognitive load during problem solving: Effects on learning. Cognitive Science, 12(2), 257-285. Uttal, D. H., O’Doherty, K., Newland, R., Hand, L. L., & DeLoache, J. (2009). Dual representation and the linking of concrete and symbolic representations. Child Development Perspectives, 3(3), 156-159. Van der Meij, J., & de Jong, T. (2006). Supporting students' learning with multiple representations in a dynamic simulation-based learning environment. Learning and Instruction, 16(3), 199-212. Vosniadou, S. (2002). On the nature of naive physics. In Reconsidering conceptual change: Issues in theory and practice (pp. 61-76). Dordrecht, Netherlands: Springer. Williams, J. J., & Lombrozo, T. (2010). The role of explanation in discovery and generalization: Evidence from category learning. Cognitive Science, 34(5), 776806. Wong, R. M. F., Lawson, M. J., & Keeves, J. (2002). The effects of self-explanation training on students' problem solving in high-school mathematics. Learning and Instruction, 12(2), 233-262.