Verification of national weather service spot forecasts using surface observations

Software has been developed to evaluate National Weather Service spot forecasts issued to support prescribed burns and early-stage wildfires. Fire management officials request spot forecasts from National Weather Service Weather Forecast Offices to provide detailed guidance as to atmospheric conditions in the vicinity of planned prescribed burns as well as wildfires that do not have incident meteorologists on site. This open source software with online display capabilities is used to examine an extensive set of spot forecasts of maximum temperature, minimum relative humidity, and maximum wind speed from April 2009 through November 2013 nationwide. The forecast values are compared to the closest available surface observations at stations installed primarily for fire weather and aviation applications. The accuracy of the spot forecasts is compared to those available from the National Digital Forecast Database (NDFD). Spot forecasts for selected prescribed burns and wildfires are used to illustrate issues associated with the verification procedures. Cumulative statistics for National Weather Service County Warning Areas and for the nation are presented. Basic error and accuracy metrics for all available spot forecasts and the entire nation indicate that the skill of the spot forecasts is higher than that available from the NDFD, with the greatest improvement for maximum temperature and the least improvement for maximum wind speed.

VERIFICATION OF NATIONAL WEATHER SERVICE SPOT FORECASTS USING SURFACE OBSERVATIONS by Matthew Robert Lammers A thesis submitted to the faculty of The University of Utah in partial fulfillment of the requirements for the degree of Master of Science Department of Atmospheric Sciences The University of Utah August 2014 Copyright © Matthew Robert Lammers 2014 All Rights Reserved The University of Utah Graduate School STATEMENT OF THESIS APPROVAL The following faculty members served as the supervisory committee chair and members for the thesis o f__________________Matthew Robert Lammers_________ . Dates at right indicate the members' approval of the thesis. John D. Horel Chair 5/14/2014 Date Approved Courtenay Strong Member 5/14/2014 Date Approved Timothy J. Brown Member 5/14/2014 Date Approved The thesis has also been approved b y _______ Kevin Perry Department o f_______ Atmospheric Sciences____________ and by David B. Kieda, Dean of The Graduate School. , Chair of the ABSTRACT Software has been developed to evaluate National Weather Service spot forecasts issued to support prescribed burns and early-stage wildfires. Fire management officials request spot forecasts from National Weather Service Weather Forecast Offices to provide detailed guidance as to atmospheric conditions in the vicinity of planned prescribed burns as well as wildfires that do not have incident meteorologists on site. This open source software with online display capabilities is used to examine an extensive set of spot forecasts of maximum temperature, minimum relative humidity, and maximum wind speed from April 2009 through November 2013 nationwide. The forecast values are compared to the closest available surface observations at stations installed primarily for fire weather and aviation applications. The accuracy of the spot forecasts is compared to those available from the National Digital Forecast Database (NDFD). Spot forecasts for selected prescribed burns and wildfires are used to illustrate issues associated with the verification procedures. Cumulative statistics for National Weather Service County Warning Areas and for the nation are presented. Basic error and accuracy metrics for all available spot forecasts and the entire nation indicate that the skill of the spot forecasts is higher than that available from the NDFD, with the greatest improvement for maximum temperature and the least improvement for maximum wind speed. TABLE OF CONTENTS ABSTRACT........................................................................................................................................... iii LIST OF TABLES............................................................................................................................... vi ACKNOWLEDGEMENTS.............................................................................................................. vii Chapters 1. INTRODUCTION............................................................................................................................. 1 1.1 Prescribed and Wildland Fires in the United States..................................................... 2 1.2 Forecast Verification............................................................................................................... 4 1.3 Summary.................................................................................................................................... 7 2. DATA AND METHODS................................................................................................................. 13 2.1 Data........................................................................................................................................... 13 2.1.1 Spot Forecasts and Requests....................................................................................13 2.1.2 NDFD Forecasts.........................................................................................................14 2.1.3 Validation Datasets.................................................................................................. ..15 2.2 Verification................................................................................................................................16 2.2.1 Text Parsing................................................................................................................ 16 2.2.2 Verification Procedures........................................................................................... 20 2.2.3 Online Web Tools..................................................................................................... 22 3. RESULTS............................................................................................................................................... 31 3.1 Specific Fire Case Studies.................................................................................................... 31 3.1.1 Box Creek Prescribed Fire................................................................................... 31 3.1.2 Other Case Studies................................................................................................... 34 3.1.2.1 River Breaks One............................................................................................. 34 3.1.2.2 Figueroa Mountain.......................................................................................... 35 3.1.2.3 Jack Springs........................................................................................................ 35 3.1.2.4 Pasture 3B/Pautre............................................................................................ 36 3.1.2.5 Twin Hat............................................................................................................ 37 3.1.2.6 Yarnell Hill..........................................................................................................38 3.1.2.7 Rim Fire............................................................................................................... 39 3.2 Weather Forecast Offices.....................................................................................................40 3.2.1 Eureka, CA (EKA).................................................................................................... 40 3.2.2 Tucson, AZ (TWC)................................................................................................. 43 3.2.3 Melbourne, FL (MLB).............................................................................................45 3.3 Overall Statistics.................................................................................................................... 46 3.3.1 Prescribed Burns....................................................................................................... 46 3.3.2 Wildfires...................................................................................................................... 49 4. SUMMARY, CONCLUSIONS, AND FUTURE WORK.................................................. 112 4.1 Summary................................................................................................................................ 112 4.2 Prescribed and Wildfire Spot Forecasts....................................................................... 113 4.3 Recommendations............................................................................................................... 114 4.3.1 Separating Numerical Values and Text in the Spot Forecast.................... 116 4.3.2 Consistency in the Request Form...................................................................... 117 4.3.3 Allowing Users to Specify the Verifying Data............................................... 117 4.4 Future Work......................................................................................................................... 118 4.4.1 Evaluating Other Variables................................................................................. 118 4.4.2 Expanding the Verification Time Window..................................................... 119 4.5 Conclusion............................................................................................................................ 120 REFERENCES.................................................................................................................................. 123 v LIST OF TABLES 3.1 Spot forecast cases explored in detail.............................................................................................. 52 3.2 Bulk statistics for prescribed burn and wildfire forecasts issued in the continental United States for maximum temperature, minimum relative humidity, and maximum wind speed..................................................................................................................................................................53 3.3 Smallest mean error and median absolute error among the forecasts issued by the 123 WFOs for prescribed burns........................................................................................................................ 54 3.4 Marginal distributions for accurate prescribed burn and wildfire spot and NDFD forecasts as relative to surface observations...........................................................................................55 3.5 As in Table 3.3, but for wildfire forecasts.......................................................................................56 ACKNOWLEDGEMENTS I would like to thank my advisor, Dr. John Horel, for his guidance, as well as the opportunities he has given me, first by hiring me as a researcher at MesoWest, and then by taking me on as a graduate student. I would also like to thank Drs. Courtney Strong and Tim Brown for serving on my committee and helping me develop the scope of this work. Thanks are due to the rest of the MesoWest group, as well, for their assistance with accessing the database of surface observations and gridded datasets. I would like to thank Virgil Middendorf from NWS Billings WFO for packaging and sending all of the spot forecasts and their requests. Finally, I would like to thank my friends and family for their support - moral, emotional, and otherwise - in this endeavor. Funding for this project was provided by the Joint Fire Science Program under grant 12-1-05-3. CHAPTER 1 INTRODUCTION A 2008 National Oceanic and Atmospheric Administration (NOAA) report entitled, "Fire Weather Research: A Burning Agenda for NOAA," outlined the need for more robust forecast verification for wildland fire incidents (NOAA SAB 2008). National Weather Service (NWS) forecasters at Weather Forecast Offices (WFOs) have issued 103,370 forecasts, often at very short notice, requested by fire and emergency management professionals for specific locations, or "spots," during the April 2009 - November 2013 period. Fulfilling spot forecast requests requires NWS forecasters to apply their expertise in a manner distinct from their other duties, which include issuing more general guidance for their entire County Warning Area (CWA). Those guidance products rely extensively on gridded forecasts of meteorological parameters created by the NWS forecasters, issued for each County Warning Area, and then blended together nationally as part of the National Digital Forecast Database (NDFD) (Glahn and Ruth 2003). Spot forecasts are requested for prescribed burns, wildfires, search and rescue operations, and hazardous material incidents (Fig. 1.1). For example, the New Orleans WFO issued over 4400 hazardous material spot forecasts associated with cleanup operations after the Deepwater Horizon spill in April 2010. Figs. 1.2-1.4 break down the number of prescribed burn forecasts, wildfire forecasts, and combined search and rescue and hazardous material spot forecasts issued by NWS forecasters within each CWA. Medford, OR (MFR) has issued the most prescribed burn forecasts while Missoula, MT (MSO) has been responsible for the most wildfire forecasts during this period. Nationwide, spot forecasts are issued twice as often for prescribed burns as for wildfires. This is due to the sheer number of forest and rangeland management activities that occur relative to wildfire ignitions. Also reducing the number of wildfire spot forecasts is that NWS Incident Meteorologists take over the responsibility to relay weather information to fire professionals once they are assigned to a fire. Finally, the agency responsible for the burn plays a role in whether a spot forecast is likely to be requested. NWS forecasters rarely receive detailed feedback from fire and emergency management professionals on the usefulness of their spot forecasts and no quantitative evaluation of spot forecasts has been undertaken nationwide. The objectives of this research are to: (1) facilitate the transfer from research to operations of methodologies to verify such forecasts and (2) develop approaches to assess the degree of improvement provided by such forecasts relative to those available from the NDFD. This work is intended to aid forecasters in understanding discrepancies between their spot forecasts and what took place as well as to improve end user confidence in the accuracy of the spot forecasts. This study focuses on the vast majority of spot forecasts requested for prescribed and wildland fires. 1.1 Prescribed and Wildland Fires in the United States Of the three elements in the fire environment triangle (terrain, fuels, and weather), weather changes on the shortest time scales (Whiteman 2000). Understanding the current and future state of the atmosphere has played an integral role in fire management in the United States since the early 20th century. S.B. Snow recommended establishing a fire weather warning service within the Weather Bureau (the predecessor to the NWS) as early as 1916 (Snow 1931) 2 and fire weather forecasts routinely incorporated temperature, humidity, and wind direction information (Calvert 1925). These variables remain central to fire weather forecasts, featuring in nearly every request by fire management officials. Prescribed fires on federal or state land have operating plans that contain thresholds for atmospheric variables such as wind speed and relative humidity beyond which they should not or will not burn. Spot forecasts play a central role in determining whether a burn is initiated on a given day. Of the 16,600+ prescribed burns undertaken in 2012, only 14 escaped (WFLLC 2013). However, public reaction to this small number of escapes is overwhelmingly negative. Outcry from the Lower North Fork Fire, which broke out in smoldering litter four days after the prescribed burn work, destroyed 23 homes, caused 3 fatalities, and led to modifications of the Colorado state constitution allowing victims of prescribed burn escapes to sue the state and recoup losses collectively greater than a $600,000 cap (Ingold 2012). Case studies in Section 3.1 will demonstrate that even if a forecast is issued that anticipates exceeding burn go/no go thresholds, this guidance is not always heeded. Budgetary and personnel concerns often weigh heavily on these decisions (WFLLC 2013). Having confidence in the forecast guidance is critical for fire management officials to appropriately balance all of the competing factors related to go/no go decisions. The nation is increasingly at risk for loss of life and damage to property as a result of wildfires (Calkin et al. 2014). During 2003, fires near San Diego, California destroyed over 3500 homes and killed 22 people (Hirschberg and Abrams 2011). Three fires (High Park, Waldo Canyon, and Black Forest) in the Front Range of Colorado in 2012 and 2013 destroyed a total of 1117 homes. Forecast guidance, issued by WFO forecasters initially and later by Incident Meteorologists as wildfires grow in extent and potential for harm, helps to determine the magnitude and placement of responding firefighters. In some circumstances, there is little 3 that can be done to contain explosively developing conflagrations, but even when the ability to control a fire is diminished, accuracy in forecasting timing and intensity is essential. The deaths of 19 firefighters in Yarnell, AZ, caused in part by a sudden wind shift outflowing from a thunderstorm, underscores the need for capturing the wide range of possible fire weather conditions. This case will be explored briefly in Section 3.1. The fire community has developed a broad spectrum of computational resources to evaluate fire behavior that require weather observations and forecasts as input. Rothermel (1972) synthesized the research from the 1950s and 1960s on fire spread and growth. The first fire model deployed for use in the field, BEHAVE, was based on Rothermel's work (Andrews 1986). Advanced wildfire forecasting tools now available, such as the SFIRE module for the Weather Research and Forecasting model, remain based to a large extent on this work (Mandel et al. 2011). In the development of the National Fire Danger Rating System (NFDRS, NWCG 2002), Deeming et al. (1972) defined fire danger as "The resultant descriptor of the combination of both constant and variable factors which affect the initiation, spread and difficulty of control of wildfires on an area." By collapsing fuel and weather information into interrelated indices, management officials have tools available to more easily convey the difficulty with which a given fire can be contained. Weather data have the greatest impact on day-to-day changes in NFDRS indices such as 1- and 10-hour fuel moisture, spread component, and ignition component. 1.2 Forecast Verification As reviewed by Myrick and Horel (2006), the goals of forecast verification fall into three categories: 4 • administrative (assess overall forecast performance for strategic planning), • scientific (improve understanding of the nature and causes of forecast errors to improve future forecasts), • economic (assess the value of the forecasts to the end users). This research is focused on the first two categories. Joliffe and Stephenson (2003) and Wilks (2006) define objective estimates of forecast quality that are appropriate for administrative-oriented verification at the national level as well as scientific-oriented verification that can provide feedback directly to the forecasters. Both needs can be addressed as outlined by Murphy and Winkler (1987) either in terms of measures-oriented or distributions-oriented verification. The former is centered on statistics such as Root Mean Squared Error (RMSE) or other skill scores developed to contrast forecasts with verifying data. These sorts of statistics have been used for decades and have definite advantages in terms of their ability to draw simple conclusions from complex series of variables, e.g., the mean absolute error of a set of forecasts from "truth." The European Center for Medium-Range Weather Forecasting (ECMWF) Technical Committee advised forecast offices in member states to compute measures-oriented values as a baseline for in-office verification (Nurmi 2003). Nevertheless, as Murphy and Winkler (1987) state, "they are not particularly helpful when it comes to obtaining a more detailed understanding of the strengths and weaknesses in forecasts or to identifying ways in which the forecasts might be improved." The distributions-oriented method alleviates some of these concerns in part by presenting more detailed information about the relationships between the forecasts and the verifying observations. It allows for any type of forecast to be examined, whether for a discrete or continuous variable and whether done in a categorical or probabilistic manner. The locations of errors are also exposed more effectively, as breaking up the joint, marginal, and 5 conditional distributions allows for the inspection of categorical errors that only occur under certain conditions. Myrick and Horel (2006) used both measures-oriented and distributions-oriented metrics to compare NDFD forecasts in the western United States to verifying data sets. Their cumulative statistics exposed biases relative to both surface observations and gridded analyses. However, metrics such as RMSE and mean absolute error handle outliers of forecast error poorly, which can affect the values averaged over an entire season. Conversely, if errors are normally distributed with large spread, relying on the Mean Error (ME), or bias, can deceive the user into thinking that there are no issues with the forecasts (Brooks and Doswell 1994). When broken down in a distributions-oriented fashion, however, outliers become marginalized, and a more nuanced understanding of conditional accuracy comes out. Horel et al. (2014) illustrate how the skill of NDFD forecasts for fire weather applications can be evaluated using both measure- and distribution-oriented statistics. Verification often assumes an unwarranted level of trust regarding the data used to compare to the forecasts. Observation representativeness in complex terrain was recognized early on as one of the impediments to generating fire weather forecasts that depended at that time largely upon the modification of observed values to reflect anticipated synoptic trends (Alexander 1930). Casati et al. (2008) outline how much the community has begun relying on model data assimilation and analyses, which is described as leading to "an incestuous verification" in which the analysis used for verification can potentially be based upon the background field derived from the model being verified. Like most studies, it is assumed here that observations are "truth," i.e., perfect, ignoring representativeness and instrumentation errors. The spot and NDFD forecasts are compared to observations at the nearest observation location as well as gridded analyses for the requested spot forecast location. 6 7 Brown and Murphy (1987) provide an excellent example of evaluating fire weather forecasts. Forecasts issued by the Boise WFO in 1984 were evaluated for the Black Rock Ranger Station in Wyoming. The forecasters were instructed to issue not only an anticipated value, but also projected 25th and 75th percentile values. They found a slight warm/dry bias in maximum temperature and minimum relative humidity forecasts. This can be seen in the conditional quartile plots (not shown), as the spread above the line for maximum temperature is less than below (suggesting forecasting higher values than observed). The opposite is present in the minimum relative humidity graph, as more spread is observed above the one to one line than below, especially at low relative humidity values. Maximum temperature forecasts had the highest percentage within the narrowest error category, followed by minimum relative humidity, and then wind speed (which was correct just over 50% of the time). They suggest that the biases are due to the forecasters' perceptions of the consequences to fire professionals of underforecasting the maximum temperature and maximum wind speed, while overforecasting minimum relative humidity, such that fire danger calculations would then be underestimated. The forecaster does not desire to leave the fire officials ill prepared for potential curing of fuels. Brown and Murphy (1987) also suggested that difficulties in quantifying uncertainty by the forecasters (i.e., predicting the upper and lower quartile values) led to negative skill in relative humidity and wind speed relative to climatological forecasts. 1.3 Summary Forecasters require verification of their spot forecasts to help improve those forecasts and fire and emergency management personnel need to be able to develop confidence regarding the skill of those forecasts. This research is intended to show how such forecast verification can be undertaken and eventually transitioned to an operational entity, such as the NWS Performance Branch. To demonstrate the capabilities of the tools developed, we limit this study to evaluating quantitatively selected forecast variables (temperature, relative humidity, and wind speed). As described earlier, these variables are central to estimates of fire spread rates and hence affect fire management and containment activities. Data and methods are summarized in Chapter 2. Case studies of spot forecasts issued for prescribed burns and wildfires are introduced in Chapter 3. Cumulative statistics for selected WFOs and the nation as a whole follow in Chapter 3. A summary of the results and issues associated with implementing a spot forecast verification system are presented in Chapter 4. The online web tools and extensive results available beyond what are possible to show here are available online at: http://meso1.chpc.utah.edu/jfsp/. 8 a) c) b) d) Figure 1.1. Locations of spot forecasts in the continental United States, April 2009 to November 2013. a) all spot forecasts, b) wildfire spot forecasts, c) prescribed burn spot forecasts, and d) hazardous materials (black) and search and rescue (orange). 9 Figure 1.2. Counts of prescribed burn spot forecasts by county warning area (CWA). CWAs are labeled by their three letter abbreviations. 10 Figure 1.3. As in Figure 1.2, but for wildfire spot forecasts. 11 Figure 1.4. As in Figure 1.2, but for hazardous materials and search and rescue spot forecasts. 12 CHAPTER 2 DATA AND METHODS 2.1 Data 2.1.1 Spot Requests and Forecasts As introduced in Chapter 1, spot forecasts are issued by forecasters at NWS WFOs for four primary purposes: prescribed burns, wildfires, search and rescue, and hazardous materials (Figs. 1.1-1.4). Professionals submit an online request form outlining the reason for needing the forecast and other pertinent information (Fig. 2.1). These forms are similar for most WFOs, with changes resulting from differing regional needs of the end user communities. Every WFO's request form presents the opportunity for the user to select common atmospheric variables such as temperature, relative humidity, and wind speed. Certain derived products, such as Haines Index, Atmospheric Dispersion Index, and Low Visibility Occurrence Index, are relevant for management purposes only in certain regions. For fires, information concerning the aspect of the slope, fuel types, sheltering, and anticipated size are generally also provided by the requestor. Observations of meteorological variables can be included to assist the forecaster in understanding any disparities between information available at the WFO and conditions near the incident. The resulting request is stored as a text document (Fig. 2.2). The spot forecast itself contains four primary sections, each of which is represented in the example product in Fig. 2.3. The first contains basic information: name of the fire, land ownership, time the forecast was issued, and contact information for the forecast office. The second section is a freeform discussion of anticipated conditions, including wind shifts, trends, potential for thunderstorms and lightning, or simply providing context for the forecasted conditions relative to recent observed values. Detailed forecasts follow of requested values for the requested time periods. Often these periods are "Today" or "Rest of Today," "Tonight," and whatever the next day is. Finally, the spot forecast identifies the forecaster responsible, the requestor, and the type of request. From the Graphical Forecast Editor (GFE) within their Automated Weather Interactive Processing System workstation, forecasters can choose to populate the requested specific forecast values for each time period from the locally stored gridded fields at the WFO or enter the requested values manually. The forecast grid files at the WFOs are often at higher spatial resolution than those stored as part of the NDFD national products. Considerable effort is spent by forecasters adjusting numerical guidance and previous forecast fields to update their local grids several times per day (Myrick and Horel 2006; Horel et al. 2014). After reviewing additional information, the forecaster may then choose to adjust the values initially populated by the GFE as needed based on their interpretation of the forecast situation. Whether by request or forecaster prerogative, the "Today" forecast regularly includes more detailed hourly or bi-hourly values, which can prove highly useful to end users in the case of a frontal passage or anticipated wind shift. 2.1.2 NDFD Forecasts As mentioned earlier, NWS WFOs release their forecasts for their respective CWAs as gridded products, which are stored nationally as part of the NDFD at 5 km horizontal resolution during the majority of the period evaluated in this study (Glahn and Ruth 2003). A 14 goal of this study is to assess the extent to which the numerical components of the spot forecasts provide improved forecast guidance relative to the NDFD forecasts. Of course, the NDFD forecasts can replace neither the critical Discussion section provided by the forecaster nor the valuable information on terrain-relative flows (e.g., up-slope/up-valley) often provided within the forecast guidance, broken down by time period, that take into account local knowledge of terrain features. The online web tools developed as part of this project (http://meso1.chpc.utah.edu/jfsp/) make it possible to compare NDFD and spot forecasts for all available forecasts. However, in order to evaluate a consistent set of NDFD and spot forecasts, the 0900 UTC NDFD forecasts for the afternoon/evening (6-, 9-, 12-, and 15-hour forecasts for 15, 18, 21, and 24 UTC) are used as a baseline for comparison with spot forecasts issued commonly in the early morning. NDFD values are extracted from the nearest neighbor grid points to the spot forecast locations. Deficiencies of the NDFD grids noticed frequently are spatial discontinuities across CWA boundaries (Myrick and Horel 2006; Horel et al. 2014). While large wildfires may cross CWA boundaries and forecasts issued by neighboring CWAs may conflict with one another (see Section 3.1), this study is focused on the locations requested for the spot forecasts. 2.1.3 Validation Datasets Fire professionals rely most heavily on surface observing stations installed by land agencies as part of the Remote Automated Weather System (RAWS, Horel and Dong 2010). There were, as of November 2013, 2277 RAWS stations in operation from which data are archived in the MesoWest database (Horel et al. 2002). Equally relevant for this study to validate the spot and NDFD forecasts are the additional 2289 NWS/Federal Aviation Administration 15 stations as of November, 2013. As shown in Fig. 2.4, the density of the observations from these two networks varies across the nation, with the highest number in California. While data from an additional 25,000 surface observing stations are available in MesoWest (see http://mesowest.utah.edu), the RAWS and NWS/FAA networks are relied on most heavily by NWS forecasters issuing spot forecasts. In addition, they rely on standardized equipment and maintenance standards (Horel and Dong 2010), e.g., both networks report temperature and relative humidity at ~2 meters (~6.6 feet). RAWS stations report wind speed at 6.1 meters (20 feet), which has been the desired height for fire management operations, as well as the height at which wind speed is generally forecast in spot forecasts. NWS/FAA stations report wind speed at 10 meters to meet the goals of aviation applications (33 feet). The National Center for Environmental Prediction (NCEP) has generated the Real­Time Mesoscale Analysis (RTMA) since 2006, providing hourly analyses of surface atmospheric variables (de Pondeca et al. 2011). This study uses the operational 5 km gridded fields available during most of this study period, although operational RTMA grids are now available at 2.5 km resolution. While it can be generally assumed that nearly all NWS/FAA and most RAWS observations are used in the RTMA analyses, some RAWS observations are not received in time for the RTMA due to latencies in satellite data transmission. The analyses provide a point of comparison within at most a few km of the location requested for the spot forecast. 2.2 Verification 2.2.1 Text Parsing The mix of textual and numerical values contained in spot forecasts (Fig. 2.3) makes it difficult to extract pertinent information for verification. The numerical values contained 16 within the spot forecasts are not separated and sent to a centralized online database. NWS forecasters rely on the GFE to translate quantitative information into text products for the general public and other customers. However, validating spot forecasts requires the inverse, reverting from text products back to numerical values. Hence, methodologies needed to be developed as part of this project to parse the forecast values from the freeform text of the spot forecasts. Computer scientists have for decades labored to generate sensical analysis from written language, creating the Natural Language Processing (NLP) field (Dale 2000). Following World War II, computer scientists attempted to translate languages using rules and modes of representation easily understood by computers. NLP has evolved into a wide range of computerized tools, including intelligent search engines, web translation services, and Apple's Siri. While very robust toolkits for NLP have been developed to analyze vocabulary and commonalities within a block of text (e.g., "nltk," Bird et al. 2009), using such toolkits was found to not be helpful to extract numerical values out of the spot forecasts. The nuances of dealing with even relatively well formatted spot forecasts from some CWAs necessitated developing text extraction and type identification code, and relying on series of "if-then" statements to handle atypical situations. The resulting code was found to be adequate to develop representative samples of spot forecasts for all CWAs, and minimize the number of forecasts dropped due to inability to parse the text properly (i.e., 9854 forecasts of the 71,070 forecasts issued during the study period were not able to be processed). Relative to the spot forecasts, the spot requests are easier to parse, since each NWS WFO coordinates with their incident management customers to define a standardized web form. Those web forms vary among the CWAs as far as which fields the customers in those 17 regions would like to see in the forecast. Since some of the text inputs on the web form allow the user to type freely, differences can arise within the request responses that are difficult for a text parser to handle (e.g., accidental nonprinting characters, inclusion of a carriage return, or presence of a line overflow). From the request form, the critical variables needed for properly verifying the forecast include: request type, latitude, longitude, bottom elevation, top elevation, size, and requested parameters. There is no guarantee that the information provided by the requestor through this form is correct, and the NWS forecasters typically have to double check that the information provided matches other geolocation information provided. Development of the validation web tools has focused on analyzing those spot forecasts that are labeled "WILDFIRE" or "PRESCRIBED." Large sections of text for those spot forecast types are ignored because they are outside the scope of the research, e.g., the Discussion section. Most spot forecasts for prescribed burns are issued in the morning for the remainder of the day, such that the section following the Discussion focuses on "Today" or "Rest of Today." Requests for prescribed spot forecasts often are submitted the night before scheduled burn operations, but the forecasts are not required nor desired until early morning. The parser keys on "Today" to begin looking for values of maximum temperature, minimum relative humidity, and wind speed. The word "Night" helps to define that the remainder is going to be ignored (see Fig. 2.3). Within the TODAY or REST OF TODAY block, relevant numerical values are sought for maximum temperature, minimum relative humidity, and maximum wind speed by searching for a number of variants of TEMPERATURE, HUMIDITY, and WINDS. Once the line containing those forecasted values is identified, the text is captured and stored to be parsed in subsequent blocks of code. If a single number is listed, or a number along with the word "Around," this number is stored for verification. If 18 two numbers separated by a hyphen are listed, the average of those two values is stored. If they are not separated by a hyphen, only the maximum value is kept. If a list of hourly or bi-hourly numbers is given, the maximum temperature or minimum relative humidity is taken from that list of numbers. Conveniently, time is never a reasonable value for temperature or relative humidity, so any numbers that correspond to 24-hour clock times (such as 1100) are ignored in a validity check. Handling wind is more complicated than what is required for temperature or humidity. Consider the following snippets of content from spot forecasts. "LIGHT AND VARIABLE WINDS BECOMING SOUTHWEST 5 MPH EARLY IN THE AFTERNOON.. .THEN BECOMING LIGHT AND VARIABLE LATE IN THE AFTERNOON." Or: "UPSLOPE/UPVALLEY 6 to 11 MPH. GUSTY AND ERRATIC IN THE VICINITY OF THUNDERSTORMS." While an end user can glean useful information from such forecasts, the lack of specificity makes it difficult to validate against observations that are reported at typically hourly intervals. What is the wind speed corresponding to light and variable? When specifically is early or late afternoon? What direction is upslope or upvalley? What is gusty and erratic? Hence, a pragmatic approach was adopted to simply focus on the maximum wind speed forecasted, ignoring directional terms or phrases related to wind gusts. The maximum wind speed value forecasted before the character string "GUST" appears in the relevant line of text is sought. If gusts are forecast prior to sustained winds, it is unlikely the forecast will be evaluated. Forecasters in a specific CWA may be required to forecast winds at a single level or multiple levels using different definitions (e.g., "20 FT," "20 FOOT," "EYE LEVEL," or "GENERAL"). To obtain the most reasonable maximum wind speed forecast value for validation, 20 ft winds are preferred. If there are two forecasts for wind speed for the day, one 19 that is more free flowing, and one that is specific by hour or every two hours, then the maximum of all the values is kept. 2.2.2 Verification Procedures As described in Section 2.1.3, the spot and NDFD forecasts are compared to RAWS and NWS/FAA observations as well as RTMA analyses. It is important to distinguish between the capabilities of the online web tools described in the next subsection and the more restrictive limits used in this specific study. For this study, the latitude and longitude extracted from the request form are used to define the four stations nearest to the spot forecast location. If there are not four stations within a horizontal distance of 50 km and vertical distance of 333 m, then only those stations within those thresholds are stored. While the distance limit was arbitrarily selected, the vertical threshold ensures that a forecast with no error will not, assuming an average lapse rate of 6oC km-1, be declared inaccurate. Only 1054 forecasts were removed from the analysis because they did not have any stations that met those requirements. Although data from all four stations are archived, this study relies on the nearest station to the spot forecast location. While occasionally a fire management official will designate a more representative station than the nearest one to assist in forecasting, attempting to extract that identifier from the request was not possible for this study. Upon transition to operations, specification of desired verification station should improve forecast performance. The maximum temperature and wind speed and minimum relative humidity are determined and stored from all values available between 16 UTC and 24 UTC. Although some of these stations occasionally report at irregular intervals, as long as there was at least one observation within that period, the station is used. Stations that do not have any reasonable observations are removed from the set. Simple range checks are used to eliminate occasionally erroneous 20 values. For temperature, this is between -100oF (-73oC) and 150oF (66oC). For relative humidity, the limits are 0% and 100%, and for wind speed, the limit is 100 mi hr-1 (45 m s-1). Winds are not adjusted for height of the sensor, which varies between 6.1 meters (20 feet) for RAWS stations and 10 meters (33 feet) for ASOS. The maximum temperature and wind speed and minimum relative humidity from all RTMA values between 16 UTC and 24 UTC at the nearest neighboring gridpoint to the spot forecast location were also obtained. Similar values were also extracted from the NDFD grids for comparison to the spot forecasts. Other methods to determine the optimal value for verification were explored, including bilinear interpolation from the surrounding grid points, averaging the surrounding grid points, and weighted averages over the 16 nearest grid points. The nearest neighbor approach was not only the most computationally efficient and allowed for the inclusion of the most spot forecasts, but also was the preferred approach based on conversations with several NWS forecasters. Results from other approaches are not shown, but the alternative approaches were not significantly better at the expense of the ability to verify some forecasts. A metric to estimate the local variation in wind speed near the spot forecast locations was extracted from the RTMA and NDFD grids, but results using that metric will not be shown here. The local standard deviation within a 20 km x 20 km box centered on the spot forecast location at the same time of maximum wind speed was determined in order to assess the variability of the local winds. The metric was intended to help define representativeness errors as well as the potential for "gusty" winds, but the results were inconclusive. 21 2.2.3 Online Web Tools As described by Murphy (1991), the large dimensionality implicit in forecast verification inhibits documenting all of the characteristics of these spot forecasts in this single study. For the April 2009 - November 2013 period, there were 44,901 prescribed burn and 16,280 wildfire forecasts that could be verified. These occur at all times of the year, within 121 CWA boundaries, and over elevation ranges from 0 to 4315 m. Some spot forecasts are for locations near an observing station, others have an observing site located tens of km away and differing in elevation by hundreds of meters. It is important as well to be able to examine forecast skill as a function of the particular values of the forecasts or the verifying observations or analyses. Hence, a central goal of this study was to develop tools that forecasters and end users can use to evaluate the forecasts of interest to them, rather than attempting to relate cumulative statistics over a limited sample of broad categories to their needs. In order to be able to rapidly query such a large data set that is continually updating, a comma-separated text file containing every valid forecast with the corresponding nearby observations, NDFD forecasts, and RTMA values is created for prior forecasts. The file is also being continually updated the day after each forecast is made. To alleviate the complexity of the multivariate nature of the spot forecasts, the open source Crossfilter code developed by Square, Inc., is used. This JavaScript package surfaces functions that store points of data in a binary heap with an indexing scheme that allows for near-instantaneous slicing on each axis of a multidimensional data set. That allows users to create histograms conditioned on ranges of values in multiple dimensions, i.e., within selected elevation ranges, times of year, values of variables (for example, max temperature in the range 20-25oC), etc. These histograms then can be adjusted dynamically by the user based on selections in other histograms. The Crossfilter 22 object is instantiated by simply pulling in the necessary information in comma-separated format. Filters are generated on one or more of the variables so that the user can make selections based on ranges of values, but also visualize the impact of other selections on these variables. Consider the verification data available at http://meso1.chpc. utah.edu/jfsp/statsAllWF.html for All Wildfires starting 1 April 2009 and updating daily. A short description of the forecasts available for this page is provided, followed by a histogram of the number of forecasts broken down by date, a series of other tabs, and a map with red markers for accurate spot forecasts issued during that period. Black markers are forecasts that are assumed to have less skill since they deviated from the surface observation by user-selectable values that default to ±2.5oC, ±10% relative humidity, and ±5 m s-1 (Fig. 2.5). By clicking on any of the markers, a window is displayed that contains the parsed values from each of the data sets that were used for verifying that forecast. There are also links to the MesoWest page for that station for the day of the forecast and for the spot forecast itself as a text document. On either side of the histogram of forecasts binned by month are two "brushes." Dragging them to restrict the range of allowable months adjusts the markers on the map to only reflect those forecasts that were issued during that time frame. It also modifies all of the other multivariate histograms that are initially hidden within the clickable tabs. Many of these tabs can be opened as are desired by the user, and brushes can be used on every histogram to pare down the number of forecasts to only those they wish to view on the map and see reflected in the histogram lengths. By leveraging these web tools, basic questions about the distributions of errors and the relationships between variables can be addressed without 23 searching endless archived figures. Since the intention is for such tools to be used operationally, they must be dynamic such that recent forecasts are constantly being provided to the forecasters and end users. As a simple example of allowing the end user the ability to make informed assessments, consider the arbitrary decision used in this study to allow verifying observations to be used if the station location is within ±333 m of the spot forecast location. Fig. 2.6 contrasts the 1262 prescribed burn forecasts for all elevation differences (Fig. 2.6a) with the smaller sample in which the spot forecast location was 200 m or more higher than the verifying station (Fig. 2.6b). It is not surprising that the spot forecasts located at higher elevations than the verifying stations tend to have lower temperature and higher relative humidity (Fig. 2.6b) compared to the complete sample (Fig. 2.6a). Without online tools such as this one, reaching this conclusion would have required writing extra lines of code to isolate forecasts with the correct elevation difference characteristics and even more lines of code to format and output the histogram of the temperature and humidity differences. With Crossfilter, this process is left to the user, and can be integrated with other filters to further understand under what circumstances different forecast behaviors are evident. More robust documentation is being developed to aid the user in their leveraging of the Crossfilter technology. Another benefit is that the sample size in the histograms is always evident as the user manipulates the filters. Hence, users are less likely to overinterpret the statistical results when it becomes evident that the underlying sample size may be very low for a particular combination of filters. 24 25 Figure 2.1. The online spot forecast request form for Salt Lake City (SLC) WFO. 26 BMBB91 KSLC 202121 STQSLC A SPOT FORECAST REQUEST HAS JUST BEEN RECEIVED FOR A WILDFIRE NAMED "Patch Springs" PRIORITY: IMMEDIATE DATE: 8/20/13 TIME: 1S21 PROJECT NAME: Patch Springs PROJECT TYPE: WILDFIRE REQUESTING AGENCY: USFS REQUESTING OFFICIAL: Chris Church REQUEST REASON: WILDFIRE FAX: EMERGENCY PHONE: [phone] LOCATION: STATE: UT DLAT: 40.3413 DLON: 112.6699 EXPOSURE: s FUEL TYPE: Grass,Brush,Tli SHELTERING: PARTIAL BOTTOM ELEVATION: S312 TOP ELEVATION: 8400 SIZE (ACRES): 31000 WEATHER CONDITIONS AT PROJECT OR FROM NEARBY STATIONS Cedar Mountain RAWS ELEV=4650 TIME=145S WIND=NE11 Gtol9 T=86 t w = RH=28 t d = nostly cloudy ELEV= TIME= WINDr T= TW= RH= TD= ELEV= TIME= WIND= T= TW= RH= TD= ELEV= TIME= WIND= T= TW= RH= TD= ...REMARKS... Please Include tomorrow night, LAL and CWR In all forecasts. Thanks, new tean taking fire tomorrow. Thanks for all the help the last 3 days. Really appreciated the phone callsand the spots Chris. ...WEATHER PARAMETERS REQUESTED... LAL: 0,0,1 HAINES INDEX: 0,0,1 CLEARING INDEX: 0,0,1 SKY/WEATHER: 0,0,1 TEMPERATURE: 0,0,1 HUMIDITY: 0,0,1 WIND - 20 FOOT: 0,0,1 SITE: SLC Figure 2.2. Example request form for Patch Springs Wildfire, 20 August 2013. 27 FNUS75 ECSLC 202145 FH5SLC SPOT FORECAST FOR PATCH SPRINGS NATIONAL WEATHER SERVICE SALT LAKE CITY UT 323 PM MET TUE AUG 20 2013 .DISCUSSION...SHOWERS AND THUNDERSTORMS WILL CONTINUE ACROSS NORTHERN UTAH INIO IHE OVERNIGHT HOURS. THERE IS THE POIENTIAL FOR THESE STORMS TO IMPACT THE PATCH SPRINGS FIRE. THESE STORMS ARE WEI...AND WILL BE ACCOMPANIED BY MODERATE TO HEAVY RAIN. EVEN IF A STORM DOES NOT DIRECTLY IMPACT THE FIRE...GUSTY AND ERRATIC WINDS FROM STORMS IN THE VICINITY MAY AFFECT THE FIRE. THE POTENTIAL FOR THUNDERSTORMS TO DEVELOP DECREASES TOMORROW THROUGH FRIDAY AS MOISTURE DECREASES ACROSS THE AREA. WINDS WILL INCREASE FROM THE SOUTH ON FRIDAY. .REST OF TODAY. . . LAL................ 3 , HAINES INDEX....... 3 ..VERY LOW. CLEARING INDEX..... 1000+. SKY/WEATHER........ PARTLY CLOUDY (65-75 PERCENT CLOUD COVER). SCATTERED SHOWERS AND IHUNDERSIORMS. MAX TEMFERAIURE.... 87-91. MIN HUMIDITY....... 22-24 PERCENT. WINDS - 20-FOOT.... UPSLOPE/UFVALLEY 6 TO 11 MPH. GUSTY AND ERRATIC IN THE VICINITY OF THUNDERSTORMS. . TONIGHT, . LAL................ 3 , HAINES INDEX....... 3 ..VERY LOW. SKY/WEATHER........ MOSTLY CLOUDY (75-85 PERCENT CLOUD COVER) . SCATTERED SHOWERS AND THUNDERSTORMS. MIN TEMPERATURE.... 65-67. MAX HUMIDITY....... 52-54 PERCENT. WINDS - 20-FOOT.... DOWNSLOPE/DOWNVALLEY 5 TO 9 MPH. .OUTLOOK FOR WEDNESDAY... LAL................ 2 , HAINES INDEX....... 3 ..VERY LOW. CLEARING INDEX..... 1000+. SKY/WEATHER........ PARTLY CLOUDY (40-50 PERCENT CLOUD COVER). A SLIGHT CHANCE OF SHOWERS AND THUNDERSTORMS. MAX TEMPERATURE.... 92-94. MIN HUMIDITY....... 19-21 PERCENT. WINDS - 20-FOOI.... UPSLOPE/UFVALLEY 6 TO 11 MPH. .WEDNESDAY NIGHT... LAL................ 4 UNTIL MIDNIGHT...THEN 3. HAINES INDEX....... 3 ..VERY LOW. SKY/WEATHER........ MOSTLY CLOUDY (75-85 PERCENT CLOUD COVER) . MIN TEMFERAIURE.... 69-72. MAX HUMIDITY....... 40-45 PERCENT. WINDS - 2Q-FOOT.... DOWNSLOPE/DOWNVALLEY 5 TO 9 MPH. FORECASTER...HOSENFELD REQUESTED BY...CHRIS CHURCH REASON FOR REQUEST...WILDFIRE .TAG 20130820.PATCH.01/SLC $ $ Figure 2.3 Example spot forecast from Patch Springs Wildfire, 20 August 2013. 28 Figure 2.4. Locations of NWS/FAA and RAWS stations in MesoWest. Figure 2.5. Map section of the website showing the different colored markers and the information contained within the popup window. 29 a) b) Figure 2.6. Example of using the website "brushes" on a histogram to adjust other histograms by selecting only those cases where the station was over 200 meters lower than the forecast location. a) website before selection, b) website after selection. 30 CHAPTER 3 RESULTS 3.1 Case Studies As part of this study, many spot forecasts were examined subjectively. Selected case studies are presented to highlight some of the issues associated with spot forecast verification. Evaluation of these specific cases is aided by Facilitated Learning Analysis (FLA) reports, which are issued by fire agencies when unintended outcomes, injuries, or fatalities occur as a result of prescribed or wildfire activities (USFS 2014, available online at http://wildfirelessons.net/). Six prescribed burns that eventually escaped the intended burn areas are examined followed by two wildfires that had large societal impacts (Table 3.1). 3.1.1 Box Creek Prescribed Fire The Box Creek Prescribed Fire occurred in the Fishlake National Forest of Southern Utah in May 2012 (USFS 2012, available online at http://wildfirelessons.net/). A crew ignited a test fire on 15 May that burned for a few days under containment. According to the FLA, spot forecasts were requested "and referenced against observed weather conditions and feedback was given to the meteorologist. The spots lined up with conditions on the ground very well. This provided the RXB2 (Burn Boss) with much confidence in the meteorologist's forecasts" (USFS 2012). According to the FLA, ignitions were halted for several days due to unfavorable winds and did not resume until 29 May. Mop-up and patrol operations followed until 4 June , when torching and spotting were observed to an extent that on-site resources could not contain it within the prescription boundary. Weather conditions in this area were warmer and drier on 4 June than typical for this time of year. No prescribed burn spot forecast was requested on the morning of 4 June since the fire was assumed to be contained. A wildfire spot forecast was requested later that afternoon and subsequent ones continued to be issued until June 17. As an illustration of the web tools developed for verifying prescribed and wildfire forecasts, the sample of 23 spot forecasts and verifying data for this case are accessible via the following web page: http://meso1.chpc.utah.edu/jfsp/BoxCreek.html. Fig. 3.1 contrasts the spot forecasts of temperature, relative humidity, and wind speed issued for the Box Creek fire to the observations from the portable RAWS (FISHLAKE PT #4, assigned MesoWest identifier TT084) deployed 3 km and 56 m above the average burn elevation, which was placed to support the prescribed fire operations. It also contains the NDFD gridpoint values and RTMA values at the location. As described in the FLA, the spot forecasts of temperature and relative humidity tended to track well with those observed at TT084, except for the expectation of warmer, drier, and windier conditions expected by the forecaster on 26 May. Spot forecast winds tend to be higher than those observed at TT084 or analyzed by the RTMA for this location. Fig. 3.2 shows histograms of differences between the 23 spot forecasts and the corresponding conditions observed at TT084 and analyzed by the RTMA available using the web page http://meso1.chpc.utah.edu/jfsp/BoxCreek.html. The user-controlled whisker filters can be used to isolate, for example, which forecasts are outliers (i.e., 26 May with a ~7oC temperature error, see also Fig. 3.1) or the date when the location requested for the spot 32 forecasts shifted several km further south (29 May). Using the default thresholds for accuracy for temperature, relative humidity, and wind speed spot forecasts of 2.5oC, 5%, and 2.5 m s-1, respectively, then Figs. 3.3a,c,e indicate that 18 temperature, 19 relative humidity, and 18 wind speed forecasts would be considered accurate relative to the observations for this sample of 23 forecasts. However, 3 temperature, 12 relative humidity, and 21 wind speed forecasts would be considered accurate using the same thresholds when verified against the RTMA (Figs. 3.2b,d,f). Hence, the RTMA has a warm, dry bias in this case due to a lower elevation specified for the verifying gridpoint (2690 m) compared to that used by the forecaster (2896 m) or that of TT084 (2952 m). In order to evaluate the accuracy of the spot forecasts for the Box Creek fire relative to the values available from the NDFD, Fig. 3.3 contains tabulations of the departures of the spot and NDFD forecasts from the TT084 observations into bins defined in terms of their absolute error following the approach of Myrick and Horel (2006). Note that the sample size is reduced to 19 since four NDFD forecasts are not available in the NDFD archive at the University of Utah. Columns reflect increasing error from left to right of the spot forecasts while rows indicate increasing error from top to bottom of the NDFD forecasts. Each bin is split further such that the sample above (below) the diagonal line indicates forecasts for which the forecaster made no or small (large) changes relative to the NDFD guidance. The thresholds for distinguishing between small and large deviations from the NDFD guidance are set for temperature, relative humidity, and wind speed by default to 1oC, 5%, and 1 m s-1, respectively. It is readily apparent from Fig. 3.4 that 17 (7) of the 19 temperature spot (NDFD) forecasts would be considered accurate. As with the RTMA analyses, the warm bias of the NDFD forecasts likely results from a lower elevation specified in the grid than observed (Fig. 33 3.1) and is also very evident using the web tools available for this case. While the forecasters provided improved temperature guidance on 10 occasions, only 1 relative humidity and 3 wind speed forecasts were improved to the point they would be considered accurate by the metrics used for this example. The Box Creek prescribed burn/wildfire was relatively unique with so many spot forecasts to verify. Although it is not possible to rigorously evaluate their performance, the remaining cases are selected to highlight specific characteristics of the spot forecasts, including: subjective estimates of forecast accuracy relative to what the verification metrics used in this study are able to assess as well as the end users' perceptions of the forecast accuracy that affect fire decisions. 3.1.2 Other Case Studies 3.1.2.1 River Breaks One Part of a larger program of prescribed burns in the River Breaks region of East Central Idaho, the Breaks One Unit Four burn was initiated on 25 September 2010 with a test fire, followed by sustained firing on 27 September. It was declared a wildfire on 29 September after unexpected westerly winds blew firebrands onto the east side of a ridge that had been declared the boundary of the burn (USFS 2010a). The escape eventually burned 100 acres of Forest Service land in steep terrain. While a relatively small and innocuous escape, this case illustrates the limitations of the verification tools applied to the spot forecasts in complex terrain. The nearest RAWS station, Kriley Creek (KRCI1) is located 9 km east-southeast of the burn at an elevation representative of the middle of the prescribed burn. Of the five spot forecasts issued by the Pocatello WFO that included daytime as their first forecast period, all 34 of them are within 8% RH and 2oC of the RAWS observation. The station observed maximum 20 ft winds (3.6 m s-1) in line with the spot forecast (4 m s-1). Based on the validation metrics, this was an accurate forecast. However, the FLA notes that winds shifted around 2 PM on 29 September from southerly to westerly and increased to 5-7 m s-1, which was neither observed at KRCI1 nor forecasted (USFS 2010a). The wind shift enabled fires to develop outside of the burn area. Hence, without evaluating wind direction and corroborating observations, the interpretation of an accurate forecast as defined here misses the critical influence of the wind shift on fire behavior. 3.1.2.2 Figueroa Mountain The FLA for the Figueroa Mountain prescribed burn in southern California on 16 November 2010 states "The reliability of spot weather forecasts in the project area is questionable" (USFS 2010b). The escape comprised only 6.5 acres beyond the boundaries of the burn area and was caused by spotting driven by easterly winds that were not forecasted. Again, the basic verification metrics used here indicate accurate forecasts from the Los Angeles/Oxnard WFO for temperature, relative humidity, and wind speed relative to the nearest station, Figueroa RAWS, which is located only 2 km from the prescribed burn (not shown). 3.1.2.3 Jack Springs The Jack Springs II prescribed burn located in Northwest Colorado was scheduled for 28 September 2010. The spot forecast issued by Grand Junction WFO for that day called for maximum temperatures from 28-31oC, a minimum relative humidity of 5-10%, and maximum sustained winds of 3.6 m s-1. These values verified well relative to observations from the 35 36 Ladore RAWS station (LODC2) located 9 miles to the north, except that the observed maximum temperature reached 33oC. The FLA reported that not only had conditions been dry in the days prior to the burn, curing the fuels and making them more susceptible to ignition, but also that the prescribed burn plan suggested optimal relative humidity values from 15-35% and temperatures within 10-29oC (BLM 2010). Hence, accurate forecasts corroborated by nearby observations were largely ignored in this case. 3.1.2.4 Pasture 3B/Pautre The largest escape explored in this study, the Pasture 3B prescribed burn in April 2013 (which became the Pautre Wildfire) illustrates how forecast differences between WFOs can affect a prescribed burn. The public response to this escape was strongly negative since several thousand acres of ranch land were burnt. The Rapid City Journal reported that some farmers lost as much as 95 percent of their land, even after "ranchers said they urged the Forest Service not to conduct burns during drought conditions" (Simmons-Ritchie 2013). The burn escape resulted from the passage of a dry cold front the afternoon of 3 April 2013 (USFS 2013). This feature was outlined in the discussion section of spot forecasts issued by Bismarck WFO (Fig. 3.4) and a subsequent one from Rapid City WFO (Fig. 3.5). Per the burn plan, Rapid City was authorized to be the WFO responsible for forecast support, but management officials were more familiar with requesting forecasts from the Bismarck WFO (Fig. 3.6). The FLA notes that between the offices, "Forecasts are inconsistent and do not always agree" (USFS 2013). The station used for verification, Hettinger Airport (KHEI), lies in North Dakota (Fig. 3.6). The minimum relative humidity forecasts of ~32% from Bismarck and Rapid City on 3 April did not anticipate the drop to 23% observed at KHEI, although both improved upon the minimum relative humidity forecast available from the NDFD (~38%) for the prescribed burn location (Fig. 3.7). Neither did any of the forecast guidance capture the observed sustained maximum wind speed of ~13 m s-1 associated with the frontal passage (Fig. 3.8). The Bismarck hourly wind forecast guidance was effective in its assessment of how wind speeds would increase through the day, only falling short in the late afternoon. Conditions were extreme enough prior to the frontal passage that firefighters reported seeing dust devils and fire whirls within the prescribed burn area (USFS 2013). Subsequent spot forecasts issued for the Pautre Wildfire were more accurate for wind speed and temperature, at times overestimating peak sustained winds (not shown). 3.1.2.5 Twin Hat The Twin Hat Unit Prescribed Burn was part of a larger program of burns around the periphery of the town of Williams, AZ in October 2009 (USFS 2009). Spot forecasts were critical for this burn to limit smoke dispersal from southerly winds into Williams and across the nearby interstate highway. It had been determined that Twin Unit was determined to be the optimal location for burning on 1 October 2009 based on the forecasted east-northeast winds. Firing proceeded as expected on 1 October, but when firefighters returned on the morning of 2 October, a couple of spot fires were detected outside of the prescription area in rugged terrain. With daytime heating and erratic winds, these spot fires continued to spread, leading to the wildfire declaration. Comparing the spot forecasts for 1 and 2 October with the observations taken from Kaibab-SK Micro #3 RAWS (TR492) located to the north of the prescribed burn, the forecasts failed to anticipate the extreme dryness observed (minimum relative humidity of 4%), and also underrepresented maximum wind speeds. The FLA notes that the spot forecast with very low relative humidity values (~7-10%) should have raised concerns, since the burn plan 37 called for a minimum 12% relative humidity. As stated in the FLA, "the consequence of low relative humidity is that embers that did drift outside the burn unit found receptive fuels with elevated probability of ignition" (USFS 2009). Another factor was a westerly wind shift during the afternoon that was not addressed in the spot forecast. 3.1.2.6 Yarnell Hill On 28 June 2013, a lightning strike ignited a rocky area west of Yarnell, AZ (State of Arizona 2013). The following day only two acres had burned by noon, and several of the resources brought in were released to other fires in the region. That afternoon, as winds picked up in concert with exceedingly low relative humidity values, the fire began to spread, burning 100 acres through the evening hours. On 30 June, several dozen firefighters were deployed to contain the fire and prevent it from reaching the town of Yarnell. A spot forecast was requested at 939 MST that morning and the forecast issued at 0945 MST called for highs above 380C, minimum relative humidity values between 11 and 15%, and light easterly winds shifting to out of the southwest in the afternoon with gusts to 9 m s-1. The Flagstaff WFO was in regular contact with the fire behavior analyst on site and provided updates during the day as conditions became more unstable to the north of the fire. These included reports of the potential for 16-20 m s-1 gusts at 1402 and 18-22 m s-1 at 1526 (State of Arizona 2013). The gusts were caused by northerly outflows generated by dry thunderstorms causing the fire to spread at ~5 m s-1 and overtaking 19 fire fighters. In verifying the spot forecasts for the Yarnell Hill Fire, the closest observation within 333 vertical meters came from Crown King RAWS (QCKA3). Crown King RAWS is in a region of tall conifers and lies at a slightly higher elevation than Yarnell, which is surrounded by mostly sagebrush and short pine. Stanton RAWS (QSTA3), while a mere 4 miles from the 38 town of Yarnell, lies in a valley to the south at an elevation of only 1097 meters. The fire elevation was closer to 1646 meters. As it turned out, each of the three spot forecasts issued for the Yarnell Hill Fire lined up more closely with QSTA3 than QCKA3. The forecast for 30 June called for a maximum temperature of 39.40C, a minimum relative humidity of 11%, and maximum sustained winds of 2.2 m s-1, with no mention of higher sustained winds in the afternoon (not shown). The verifying observation from Crown King reported a maximum temperature of 35.60C, 16% minimum relative humidity, and a maximum sustained wind of 4 m s-1. Stanton RAWS observed a high temperature of 39.40C, a minimum relative humidity of 14%, and a maximum sustained wind of 6.3 m s-1. The latter lines up quite a bit better with the forecast, although the wind speed forecast issued significantly underestimated afternoon thunderstorm outflows. In the Investigation Report, very little is made of the underforecasting of wind speed, with the only spot forecast featured being the overnight forecast discussed at the 0700 MST meeting on 30 June. 3.1.2.7 Rim Fire Ignited on 17 August 2013, the Rim Fire eventually burned over 250,000 acres in and north of Yosemite National Park in central California. Response to the fire and property destruction cost over $127 million. Because it escalated quickly, only a couple spot forecasts were issued by local WFOs before an Incident Meteorologist was deployed. Of the two that were generated for daytime conditions, both erred on the cool/wet side relative to the nearest station. This station, El Portal RAWS (EPWC1) observed maximum temperatures 7.50C and 11.60C higher than forecast, with minimum relative humidities 4.5% and 8.5% drier than forecast on 18 and 19 August, respectively. El Portal RAWS is on the low end of the elevation range given in the request, so warmer conditions are expected in the comparison, but not to 39 the extent observed. The curing of fuels in the first days of the fire helped it to spread more quickly than anticipated. 3.2 Weather Forecast Offices The case studies in the previous section provide specific examples of spot forecast performance for fire events that had unexpected outcomes. To evaluate spot forecasts more generally, the web tools make it possible to examine all the prescribed burn (http://meso1.chpc.utah.edu/jfsp/statsWFOPrescribed.html) or wildfire spot forecasts (http://meso1.chpc.utah.edu/jfsp/statsWFOWildfire.html) issued by each WFO, thereby obtaining samples of interest both to the WFO forecasters and NWS management as well as end users. We illustrate how such samples can be evaluated for three WFOs (Eureka, CA, Tucson, AZ, and Melbourne, FL). 3.2.1 Eureka, CA WFO (EKA) Samples of 853 prescribed burn forecasts (Fig. 3.9a), and 62 wildfire forecasts (Fig. 3.9b) are examined for the Eureka WFO. Prescribed burns are common from October-April with wildfires in late summer. Almost half of the wildfire spot forecasts occurred during the 2013 summer. The histogram of the differences between maximum temperature spot forecasts and nearby observations (Fig. 3.10a) appears bimodal simply because of the conversion of whole number temperature forecasts issued in degrees Fahrenheit into degrees Celsius. If plotted as error in Fahrenheit, the histogram is normally distributed. Binning in the histograms was decided based on a balance between space available in the web tools for a given graph and the maximum resolution. The majority of the maximum temperature forecasts would be judged 40 accurate (within ±2.5oC of the closest observations) with occasionally large errors (the 2.2oC Median Absolute Error, MAE, of these forecasts is large compared to other WFOs). The prescribed burn forecasts issued by Eureka forecasters exhibit an excessively wide spread in relative humidity errors (Fig. 3.10b) compared to other WFOs. There were cases in which the relative humidity was forecast more than 40% too dry and others nearly 40% too wet, leading to a 10% MAE, which is ~5% greater than the MAE for the national sample of prescribed burn forecasts. Forecasts of maximum wind speed are positively skewed towards higher maximum wind speeds than observed (Fig. 3.10c). This is likely due to the preference of the forecaster to anticipate higher winds to ensure fire officials are prepared for the potential for higher fire danger. Figs. 3.11 (a) and (b) tabulate the percentages of maximum temperature forecast errors for the 853 prescribed burns and 62 wildfires, respectively. These forecast errors are categorized into bins, which are defined by the relative magnitudes of the spot and NDFD errors. Each bin is further subdivided into whether the spot forecasts reflect small or large changes from the NDFD forecasts. The marginal percentage of accurate temperature spot forecasts for prescribed burns is defined by whether the forecasts fall within 2.5oC of the nearest observation and is the sum of all the percentages in the first column of Fig. 3.11a (59.6%) while the marginal percentage of accurate NDFD temperature forecasts is the sum of all the percentages in the first row (52.6%). Hence, 7% of the Eureka WFO maximum temperature forecasts for prescribed burns are at least one error class more accurate than those available from the NDFD forecasts. For NDFD forecasts that are accurate, forecasters submitted spot forecasts that were nearly the same as the NDFD 23.9% of the time (above the diagonal in the upper left bin) and modified other similarly accurate NDFD forecasts by 41 more than 1oC on 17.9% other occasions (below the diagonal in the upper left bin). Inaccurate NDFD forecasts (ones with departures from the verifying observations greater than 2.5oC) were substantively modified by more than 1oC 16% of the time (sum of the values below the topmost bin below the diagonal lines in the left column). However, 9.4% of the accurate NDFD forecasts were substantively modified as well by the forecasters, leading to degraded forecasts. For the smaller sample of spot forecasts for wildfires (Fig. 3.11b), the forecasters improved on values available from the NDFD by only 1.1%. As evidenced by the broad distribution of relative humidity errors in Fig. 3.10b, there were fewer accurate (errors less than 5%) spot forecasts for prescribed burns (26.3%- the sum of the leftmost column) than very inaccurate forecasts (31.9%- the sum of the rightmost column) (Fig. 3.12a). NDFD forecast accuracy was worse (23.8% accurate and 36.6% of the forecasts with errors larger than 15%). For the smaller sample of wildfire spot forecasts, both the NDFD and spot forecasts exhibited fewer extremely large errors (Fig. 3.12b). Maximum wind speed spot forecasts for Eureka prescribed burns tend to be less accurate than the NDFD ones (Fig. 3.13a): the forecasters improved 14.4% of the NDFD forecasts while degraded 18.9% of them with a tendency to overforecast the maximum wind speeds (Fig. 3.10c). For wildfire forecasts (Fig. 3.13b), Eureka forecasters tend to let accurate NDFD forecasts "ride," i.e., 65.4% of the wind forecasts are accurate for both. If they do adjust the NDFD maximum wind forecasts, the tendency is not to lead to an overall improvement in forecast accuracy (7.3% improvement and 10.9% degradation). To examine possible causes for the broad spread in relative humidity spot forecast errors, all Eureka prescribed burn forecasts with MAEs greater than 30% were examined subjectively. The cases with large errors did not exhibit strong dependence on location, time 42 of year, proximity of the validating station observations, or validation by RTMA values rather than station observations (compare Fig. 3.10b to Fig. 3.14a). The apparent dry bias of the spot forecast for the smaller wildfire sample when those forecasts are compared to the RTMA grid values (Fig. 3.14b) suggests that the RTMA values tend to have higher relative humidity values, which is also apparent by the negative skewness for the larger prescribed burn sample (Fig. 3.14a). Eureka forecasters appear to have difficult challenges to provide accurate relative humidity guidance whether in terms of the NDFD gridded values for the entire CWA or specific spot forecasts. As a representative case, consider the Mill Creek prescribed burn for 24 October 2011, which anticipated a minimum humidity between 26 and 31% (Fig. 3.15). This was related to "a dry airmass and light offshore flow" mentioned in the discussion. However, at the verifying station 8 km away (Big Hill RAWS (BIIC1, Fig. 3.16), the observed maximum temperature was 80C lower than forecast (Fig. 3.17), and the relative humidity never dropped below 63%, a discrepancy of 35% (Fig. 3.18). This particular case is a relatively rare one, since the NDFD guidance was substantively better than the spot forecast. 3.2.2 Tucson, AZ WFO (TWC) The Tucson CWA in the southeast corner of Arizona experiences, not surprisingly, hot and dry conditions (i.e., there are no spot forecasts issued for maximum temperature below 100C or minimum relative humidity above 60%) with 214 prescribed burn forecasts during the period and 258 wildfire forecasts. Of interest here is that Tucson forecasters tend to overforecast maximum temperature and underforecast minimum relative humidity more so than forecasters at other WFOs (Fig. 3.19). While the Tucson warm bias of ~1.7oC for both prescribed burn and wildfire forecasts is not too surprising (Brown and Murphy 1987), the 43 majority of WFOs exhibit a slight cool, wet bias relative to the observations. Further, only ~10% of prescribed burn forecasts and ~20% of wildfire forecasts called for maximum temperatures less than what was observed (Fig. 3.19). However, the NDFD gridded forecasts for these locations have a slight cool, wet bias for the same locations for both prescribed burn and wildfire cases (Fig. 3.20). Forecasters adjust on average the temperatures to be nearly 30C warmer and ~7% drier than the corresponding NDFD values (Fig. 3.21). The frequencies of these adjustments are corroborated by the prescribed burn and wildfire distributions of temperature errors (Fig. 3.22) and relative humidity errors (Fig. 3.23). About 82% of temperature forecasts were adjusted by more than 1oC and half of relative humidity forecasts were adjusted by more than 5% relative humidity. In only 18 of the 217 prescribed burn forecasts did the human forecaster expect conditions to be cooler than suggested by the NDFD grids (Fig. 3.21a). In both prescribed burns and wildfires, the wind speed spot forecasts improve upon the NDFD gridded values more often than they degrade them (Fig. 3.24). For prescribed burns, the improved cases comprise 15.6% of the sample while the degraded forecasts represent only 10.6%. With wildfires, the disparity is increased, with spot forecasts improving in 20.4% of cases while they degraded versus the NDFD only 6.1% of the time. Hence, Tucson forecasters tend to supply spot forecasts that are less accurate than the gridded values they provide for general applications. These forecasts tend to err conservatively for fire weather operations by anticipating higher fire danger via higher maximum temperature and lower minimum relative humidity forecasts. 44 3.2.3. Melbourne, FL WFO (MLB) As shown in Fig. 3.25, prescribed burns are frequent in central Florida (between April 2009 and November 2013, 808 spot forecasts issued by Melbourne forecasters) and wildfires are common as well (164 spot forecasts). Most of the wildfire forecasts were issued during 2011. The need for accurate metadata to be provided by the requesters is evident by the forecast locations in Fig. 3.25a off the Atlantic coast. Feedback from forecasters at many WFOs suggested that latitude and longitude values supplied by fire professionals are occasionally wrong, which requires the forecaster to contact them to obtain more accurate location information. The accuracy of the spot temperature and relative humidity forecasts for prescribed burns made by Melbourne forecasters is quite good (Fig. 3.26), including one of the lowest temperature MAEs among all CWAs (~1.10C) and a relative humidity bias of only 1.3% when compared to nearby observations. 91.4% of Melbourne prescribed burn spot forecasts have errors less than 2.5oC (Fig. 3.27a), and 47.9% of prescribed burn relative humidity spot forecasts have errors less than 5% (Fig. 3.27a). Similar skill is displayed for wildfire forecasts, with 88% of temperature forecasts within 2.5oC of the nearest station (Fig. 3.27b) and 43.2% of relative humidity forecasts within 5% of the observation (Fig. 3.28b). In prescribed burn cases, 79.1% of the wind speed spot forecasts were accurate whether supplied from the NDFD or spot forecasts (Fig. 3.29a). However, for wildfire forecasts, forecasters improved on the NDFD wind forecasts 20.4% of the time and degraded them only 6.1% of the time. Hence, Melbourne forecasters tend to find less need to adjust their gridded guidance for prescribed burns but spot wildfire forecasts often require changing their gridded values and those changes tend to be beneficial. 45 As shown in Fig. 3.30, verifying the spot forecasts against the RTMA values suggests that the RTMA grid values in Melbourne's CWA tend to be too cool and wet (e.g., a bias between spot forecasts and RTMA values of -6.5%). As a specific example, the spot forecast for the Comp 292 Atloona prescribed burn on 3 August 2012 predicts a minimum relative humidity of 46% and a maximum temperature of 340C, which was nearly identical to what was observed from a nearby portable RAWS station (TS959). The nearest RTMA gridpoint values for these same variables are 70% minimum RH and 310C. This issue for the RTMA grids is not limited to Melbourne. The relative humidity bias for prescribed burn forecasts for all NWS Eastern Region WFOs is 1.6% compared to nearby observations but -4.4% when compared to RTMA values (not shown). 3.3 Overall Statistics 3.3.1 Prescribed Burns Cumulative statistics for as many of the prescribed burns as possible are now summarized. A total of 44,901 prescribed burn spot forecasts were analyzed for the afternoon forecast period between 1 April 2009 and 30 November 2013 with at least one forecast issued in every state as well as Puerto Rico (Fig. 3.31). The months with the most prescribed burn forecasts were April 2010 and March and April 2012 (Fig. 3.32). Forecasts have been supplied for burns with maximum observed temperatures lower than -250C in a burn in November 2012 near Fairbanks, AK, and as high as 420C on a burn near the Texas/Mexico border in May 2010. The highest observed wind speed during the afternoon of a prescribed burn was 21.9 m s-1 in southern New Mexico in April 2009. As summarized in Table 3.2, the temperature spot forecasts for prescribed burns have a slight cool bias (-0.50C mean error, ME) and a 1.3oC MAE when verified against the observed 46 maximum temperatures. The slight cool bias is evident in the forecast error histogram (Fig. 3.33a) with the bimodal peak surrounding zero again resulting from the rounding resulting from the conversion from English to metric units (see the discussion of Fig. 3.9a). Comparing NDFD forecasts to the observations suggests that the NDFD forecasts are more biased (- 1.70C ME), less accurate (1.70C MAE), and their errors relative to observations skewed negatively than the spot forecasts (Table 3.2 and Fig. 3.33b). Focusing on WFOs with at least 100 prescribed burn forecasts, WFOs in the western United States and those containing large sections of the Appalachian Mountains tend toward higher MAE values than those in the Great Plains and the South (Fig. 3.34). We previously discussed the accuracy of the spot forecasts out of Melbourne, FL for temperature, but another equally strong office for temperature accuracy is Springfield, MO, with a Median Absolute Error of just 1.090C (Table 3.3). Only one forecast issued by Springfield had a temperature error greater than 50C out of 165 forecasts. Fig. 3.35 tabulates the errors of spot forecasts compared to those for NDFD forecasts. The values in the upper left bin are ones where the NDFD and spot forecasts were accurate and the forecaster either made only minor changes of less than 1oC (40.5%) or else they made slightly more substantive changes (18.8%). Of greater interest are the sums excluding the upper left bin of: (1) the values in the left column (i.e., where the forecasters made changes relative to the NDFD gridded values that resulted in accurate forecasts) and (2) the values in the uppermost row (i.e., where the NDFD forecasts were accurate and the manual adjustments provided by the forecasters degraded the skillful forecast available from the NDFD). For maximum temperature forecasts those values are 16.1% compared to 6.5%, which suggests that the manual intervention by the forecasters improved the spot forecasts compared to 47 NDFD forecasts by 9.6% (Table 3.4). Of particular note are the 2.8% of the forecasts where the NDFD forecasts deviated from the verifying observations by more than 7.5oC while the forecasters adjusted those values substantively and provided spot forecasts within 2.5oC. As summarized in Table 3.2, spot forecasts perform better than the NDFD gridded forecasts for minimum relative humidity in terms of both bias (1.5% wet bias for spot forecasts, 6.0% wet bias for NDFD) and accuracy (5.3% MAE for spot forecasts, 6.0% for NDFD). The cumulative error histograms confirm the slight wet biases of both spot and NDFD forecasts (Fig. 3.36). The regions with less accurate minimum relative humidity forecasts are those with generally higher relative humidity values in general: the Pacific Coastal states, the Central Appalachian Mountains, and parts of the Great Plains (Fig. 3.37). CWAs in the desert southwest and other regions where relative humidity values tend to be low exhibit higher accuracy in terms of MAE. The top office for relative humidity was Midland/Odessa, due in part to the consistently dry conditions under which forecasts are issued (Table 3.3). The smaller the range of possible values for relative humidity, the less likely large errors can occur. The relative accuracy of the spot vs. NDFD forecasts for minimum relative humidity forecasts is less than that for maximum temperature, as shown in Fig. 3.39. Forecasters improved substantively upon 15.7% of the NDFD forecasts and degraded 11%, which suggests an improvement in accuracy of 4.7% as a result of forecasters adjusting the NDFD values for the nation as a whole (Table 3.4). A smaller sample of 38,017 prescribed burn forecasts for maximum wind speed is available due to the greater difficulty in parsing the spot wind speed forecasts. As evident in the error histograms (Fig. 3.39) and Table 3.2, both spot and NDFD forecasts exhibit slight 48 overforecasting errors (biases of 0.2 m s-1 and 0.4 m s-1, respectively). The histograms in Fig. 3.39 appear skewed negatively simply because they are plotted at the lowest values of the 1oC bins. The positive biases and larger number of prescribed burns in the western CWAs dominate over the greater number of CWAs in the central and eastern United States with negative biases calculated from smaller sample sizes (Fig. 3.40a). There is less regional homogeneity in terms of MAE, although Rocky Mountain offices are slightly less accurate (Fig. 3.40b). Jackson, MS WFO issued the most accurate maximum wind speed forecasts, with a Median Absolute Error of only 0.85 m s-1 over 537 forecasts (Table 3.3). As shown in Fig. 3.41, accurate forecasts were provided 65% of the time by both the spot forecasters and NDFD forecasters. Adjustments by the forecasters for 11% of the poor NDFD forecasts result in accurate spot forecasts but 9.4% of similar adjustments degrade the forecasts provided by the NDFD (Table 3.4). 3.3.2 Wildfires Relative to prescribed burns, less than half as many wildfire spot forecasts (16,280) could be verified for temperature and relative humidity (Fig. 3.42). Whereas prescribed burn forecasts are spread fairly evenly throughout the country, wildfire forecasts are concentrated in the western United States with sizeable numbers in Florida and from Eastern Michigan through North Dakota as well (Fig. 3.42). As shown in Fig. 3.43, the months with the largest number of spot forecasts issued for wildfires are July and August with 1,043 spot forecasts that could be verified during August 2011; only 3 forecasts were verified during December 2009. The lowest observed maximum temperature for a wildfire was -16.70C (in Northern Montana in January 2012), while the highest observed temperature was 46.70C near Blythe, CA, close to the Arizona border in July 2009. The highest observed maximum wind speed 49 for verification of a wildfire spot forecast was 19 m s-1 for a fire in Southeast Colorado during April 2011. The maximum temperature forecasts for wildfires are quite unbiased (Fig. 3.44a) with a -0.30C ME (Table 3.2) while the NDFD forecast errors are more negatively skewed (Fig. 3.44b) with a cool bias of -1.460C (Fig. 3.44b). The MAE of the wildfire spot (NDFD) forecasts versus the observations is -0.370C (1.990C), suggesting that the spot forecasts improve upon NDFD gridded values for wildfire maximum temperature (Table 3.2). Most CWAs exhibit a slight cool bias (Fig. 3.45a) and the large samples of maximum temperature wildfire forecasts for CWAs in the western United States tend to be less accurate relative to the smaller samples for wildfire forecasts in other regions of the country (Fig. 3.45b). A number of offices had low MAE temperature values with the least biased one being Melbourne, FL, (see Section 3.2.3) with a mean error of only -0.05oC (Table 3.5). As shown in Fig. 3.46, 66.6% (59.1%) of the wildfire spot (NDFD) maximum temperature forecasts are judged to be accurate (within 2.50C of nearby observations), reflecting a substantive improvement of accuracy for 7.5% of the wildfire forecasts (Table 3.4). Not surprisingly, the observed relative humidity during wildfires tends to be lower than that observed for prescribed burns (not shown). The histograms for spot and NDFD wildfire forecasts relative to the observed minimum relative humidity in Fig. 3.47 suggests both types of forecasts exhibit a tendency for higher relative humidity than observed, but the NDFD forecasts are even more clearly biased towards higher relative humidity. As shown in Table 3.2, the ME for wildfire spot relative humidity forecasts is only 0.69%, while NDFD forecasts have a Mean Error of 4.1% with a 1.05% lower MAE for spot forecasts compared to NDFD forecasts. 50 The few offices that have dry biases for their relative humidity forecasts are dispersed widely (Fig. 3.48a). While most of the WFOs with high MAE values for wildfire forecasts are in the eastern half of the United States (except for Seattle, SEW, and Oxnard/Los Angeles offices, LOX, Fig. 3.48b), areas that are generally drier are predisposed to having less error in relative humidity as mentioned earlier. The top offices in terms of accuracy (MAE ~2%) in minimum relative humidity are Phoenix and Midland/Odessa (Table 3.5). Phoenix never forecast a value above 40% for minimum relative humidity, and Midland/Odessa never exceeded 35%. Forecasters provided accurate minimum relative humidity forecasts 51.3% of the time, an increase of 3.8% compared to NDFD forecasts (Fig. 3.49). Forecasting maximum wind speed effectively is crucial for containing and combatting wildfires, especially in their early stages. As shown in Table 3.2, the ME for spot (NDFD) forecasts is 0.72 m s-1 (0.79 m s-1) and corresponding MAE values of 1.5 m s-1 (1.59 m s-1), respectively (see also Fig. 3.50). All of the western United States offices save San Diego have positive biases for wind speed, while the Eastern offices have varying ME values (Fig. 3.51a). MAE values vary widely across the country (Fig. 3.51b). Miami and Birmingham both had the highest accuracy in wildfire wind speed forecasts, with MAE values of only 0.9 m s-1 (Table 3.5). Similar to the prescribed burn maximum wind speed forecasts, 58.2% the wildfire maximum wind speed forecasts supplied by the NDFD grids are equally accurate to those provided by the spot forecasts (Fig. 3.52). Adjustments to the NDFD grid values by the forecasters provided only a net increase in accurate forecasts of 1.6%. 51 52 Table 3.1 Spot forecast cases explored in detail Name Start Date Containment Date State WFO Prescribed Burn or Wildfire Box Creek 15 May 2012 14 June 2012 Utah SLC Prescribed Breaks One 25 Sep 2010 ~7 Oct 2010 Idaho PIH Prescribed Figueroa Mountain 16 Nov 2010 18 Nov 2010 California LOX Prescribed Jack Springs II 28 Sep 2010 29 Sep 2010 Colorado GJT Prescribed Pasture 3B /Pautre 3 April 2013 3 April 2013 South Dakota UNR Prescribed Rim 17 August 2013 24 October 2013 California STO Wildfire Twin 1 Oct 2009 7 Oct 2009 Arizona FGZ Prescribed Yarnell Hill 28 June 2013 10 July 2013 Arizona FGZ Wildfire Table 3.2. Bulk statistics for prescribed burn and wildfire forecasts issued in the continental United States for maximum temperature, minimum relative humidity, and maximum wind speed. Number o f Forecasts (Spot - Observation) (Spot - Observation) Mean Error (Spot - Observation) Median Absolute Error Number o f Forecasts (NDFD - Observation) (NDFD - Observation) Mean Error (NDFD - Observation) Median Absolute Error Number o f Forecasts (Spot - RTMA) (Spot - RTMA) Mean Error (Spot - RTMA) Median Absolute Error Prescribed Burn Temperature 44,901 -0.53 °C 1.33 °C 42,924 -1.72 °C 1.69 °C 39,457 0.40 °C 1.35 °C Prescribed Burn Relative Humidity 44,901 1.46% 5.29% 42,924 6.04% 6.64% 39,457 -3.26% 5.71% Prescribed Burn Wind Speed 38,017 0.22 m s-1 1.34 m s-1 35,979 0.42 m s-1 1.42 m s-1 33,298 0.21 m s-1 1.14 m s-1 Wildfire Temperature 16,280 -0.37 oC 1.67 oC 14,680 -1.46 oC 1.99 oC 15,885 0.20 oC 1.63 oC Wildfire Relative Humidity 16,280 0.69% 4.00% 14,680 4.10% 5.05% 15,885 -1.82% 4.23% Wildfire Wind Speed 8,860 0.72 m s-1 1.50 m s-1 8,075 0.79 m s-1 1.59 m s-1 8,872 0.33 m s-1 1.37 m s-1 53 Table 3.3. Smallest mean error and median absolute error among the forecasts issued by the 123 WFOs for prescribed bums. Mean Error - Maximum Temperature Median Absolute Error - Maximum Temperature Mean Error - Minimum Relative Humidity Median Error - M Relative H Absolute inimum humidity Mear MaximS i Error - um Wind jeed Median Absolute Error - Maximum Wind Speed GRB -0.040C MLB 1.080C SEW -0.06% MAF 2.34% OAX -0.05 ms-1 JAN 0.85 ms-1 OUN -0.060C SGF 1.090C CAE 0.11% FGZ 3.00% GRB 0.06 ms-1 CAE 0.89 ms-1 LCH -0.070C MOB 1.110C BYZ 0.13% TWC 3.00% AKQ 0.07 ms-1 LIX 0.89 ms-1 LIX -0.070C FGZ 1.110C LIX -0.13% PSR 3.00% LCH 0.07 ms-1 AKQ 0.89 ms-1 PDT -0.080C MFL 1.110C FFC 0.13% LKN 3.00% MLB -0.08 ms-1 LCH 0.89 ms-1 54 Table 3.4. Marginal distributions for accurate prescribed burn and wildfire spot and NDFD forecasts as relative to surface observations. Cutoff values for "accurate" forecasts are 2.5oC, 5% relative humidity, and 2.5 m s-1. Accurate Spot Forecasts Accurate NDFD Forecasts Difference (Spot - NDFD) Prescribed Burn Temperature 75.4% 65.8% 9.6% Prescribed Burn Relative Humidity 43.9% 39.2% 4.7% Prescribed Burn Wind Speed 76% 74.4% 1.6% Wildfire Temperature 66.6% 59.1% 7.5% Wildfire Relative Humidity 53.3% 49.5% 3.8% Wildfire Wind Speed 70.4% 68.8% 1.6% 55 Table 3.5. As in Table 3.4, but for wildfire forecasts. Mean Error - Maximum Temperature Median Absolute Error - Maximum Temperature Mean Error - Minimum Relative Humidity Median Error - M Relative H Absolute inimum humidity Mear MaximS i Error - um Wind jeed Median Absolute Error - Maximum Wind Speed MLB -0.050C MFL 0.990C TFX -0.04% PSR 2.00% MFL -0.06 ms-1 MFL 0.90 ms-1 FGZ 0.060C JAX 1.110C OUB 0.06% FGZ 3.00% MLB 0.26 ms-1 BMX 0.90 ms-1 BYZ -0.110C LCH 1.110C JAX -0.18% GJT 3.00% PDT 0.40 ms-1 MSO 1.34 ms-1 JAX -0.110C BRO 1.110C DLH -0.20% EPZ 3.00% JAX 0.40 ms-1 OTX 1.34 ms-1 PUB -0.120C MLB 1.110C FGZ 0.25% BOI 3.00% MFR 0.42 ms-1 PDT 1.34 ms-1 56 57 a) b) Figure 3.1. Forecasts and verifying data during the Box Creek prescribed burn and subsequent wildfire. Data are for a) maximum temperature (oC), b) minimum relative humidity (%), and c) maximum wind speed (m s-1). 58 a) c) e) b) d) f) Spot-RTMA Max Temperature r10 - 9 - S -7 Spot-RTMA Min RH 10 9 8 7 Spot-RTMA Max Wind Speed Figure 3.2. Histograms of differences between Box Creek spot forecasts and observations at TT084. Histograms are for a) maximum temperature (oC), c) minimum relative humidity, and e) maximum wind speed (m s-1); b) as in a) except verified against the RTMA, d) as in c) except verified against the RTMA, and f) as in e) except verified against the RTMA. 59 a) b) Figure 3.3. Count of the number of cases for differences between spot forecasts and observations (columns) and NDFD forecasts and observations (rows). These are for a) maximum temperature (oC), b) minimum relative humidity (%), and c) maximum wind speed (m s-1). Values above (below) the diagonal lines in each bin indicate spot forecasts that are within (greater than) specified ranges of values. These ranges are 1oC, 5%, and 1 m s-1 of the NDFD forecast for temperature, relative humidity, and wind speed, respectively. x = |Spot-Obs| in C~| x<2.5 2.5<=x<5 5<=x<7.5 x>7.5 y y o y o y / - y o y o / z o / / / y o / ° / o y ' / o y / • y o X y o 0 / • / o / 1 / / 0 / ■ y o / ° | x = |Spot-Obs| in % | x<5 5<=x<10 10<=x<15 x>15 y y o y o y y o y o x o / / A / * / 0 0 / 0 / ' / o y / 0 y o / ' y o 0 / y 0 / l / / 0 / ■ y o y o 60 c) mr s i V> > Hi m E V >■ c II __ (/> V n in o rs i Qu _ in QZ V '--- >V II II >- V in m A> x<2.5 2.5<=x<5 5<=x<7.5 x>7.5 / • / 0 S ° / / » / o / 0 0 S • / ° / • / / 2 / • / i * / ° * / 0 / / ■ o / 0 0 / 0 / 0 / s 0 / 0 / 0 Figure 3.3. Continued 61 FNUS73 KBIS 031526 FWSBIS SPOT FORECAST FOR PASTURE 3B...USFS N ATIONAL W E A T H E R SERVICE BISMARCK ND 1026 A M CDT WED A P R 3 2013 F ORE C A S T IS BASED ON IGNITION TIM E OF 0940 CDT ON A P R I L 03. IF CONDITIONS BECOME U N R E P R E S E N T A T I V E ... C O N T A C T THE NATIONAL W E A T H E R SERVICE. . D I S C U S S I O N . . .A GRADUAL WIND S H I F T WIL L DEVELOP TO D A Y BEHIND A COLD FRONT. M A X IMUM NORTH W E S T E R L Y WIN D GUSTS UP TO 30 MPH ARE E XPECTED THIS AFTERNOON. .T O D A Y . .. S K Y / W E A T H E R .......... P ARTLY SUNNY. T E M P E R A T U R E .......... 42 AT I G N I T I O N ... M AX 66. R H ..................... 59 P ER CENT A T IGNITION ... MIN 33 PERCENT. W I N D ...................SOUTH WINDS 5 TO 11 MPH SHIFTING TO THE SOUTHWEST AROUND 14 MPH LATE IN THE M O R N I N G . ..THEN SHIFTING TO THE NORTHWEST AROUND 25 MPH WITH 30 MPH GUSTS THIS AFTERNOON. SMOKE D I S P E R S A L ..... EXCELLENT. C W R .................... 0 PERCENT. TI M E (CDT) 9AM 10A 11A 12P 1PM 2PM 3PM 4PM 5PM 6PM SKY ( % ).......... 39 W E A T H E R C O V ..... W E A T H E R T Y P E ____ T E M P ..............35 R H ................ 59 20 FT WIND DIR..S 20 FT WI N D S P D ..10 20 FT WIND G U S T . 15 C W R ............... 0 44 51 65 72 72 70 70 67 62 42 49 54 59 63 65 66 65 62 50 44 41 37 35 34 33 34 36 S SW W W W HW HW NW N 12 15 18 20 21 22 24 25 25 15 20 25 25 25 30 30 30 30 G 0 0 0 0 0 0 0 0 .TONIGHT. S K Y / W E A T H E R .......... P ARTLY C LOUDY THEN BECOMING M O S T L Y CLOUDY. T E M P E R A T U R E .......... MIN 29. R H ..................... M AX 75 PERCENT. W I N D ...................NORTH WINDS 15 TO 23 MPH DECREASING TO 8 TO MPH IN THE LATE EVENING AND OVERNIGHT. SMOKE D I S P E R S A L ..... POOR. L A L .................... 1. C W R .................... 0 PERCENT. 13 TI M E (CDT) SKY ( % ) ........... W E A T H E R COV.., W E A T H E R T Y P E . R H .......... 20 FT WIND. 20 FT WIND C W R ......... L A L ......... 7 PM 9 PM 11 PM 1 AM 3 AM 5 AM 60 52 50 51 61 75 NONE NONE NONE NONE NONE NONE 59 48 39 34 32 30 39 52 62 66 69 72 N 25 N 17 N 11 N 11 N 10 N 9 30 20 15 15 10 0 0 0 0 0 0 1 1 1 1 1 1 .THURSDAY... Figure 3.4. Bismarck WFO spot forecast for Pasture 3B Prescribed Burn, 3 April 2013. 62 FNUS73 KUNR 031638 FWSUNR SPOT FORECAST FOR PAUTRE 3B...USFS NATIONAL WEATHER SERVICE RAPID CITY SD 1038 AH MDT WED APR 3 2013 FORECAST IS BASED ON IGNITION TIME OF 1035 MDT ON APRIL 03. IF CONDITIONS BECOME UNREPRESENTATIVE CONTACT THE NATIONAL WEATHER SERVICE OFFICE IN RAPID CITY. .DISCUSSION...A COLD FRONT CROSSING FAR WESTERN MONTANA WILL CONTINUE TO PUSH INTO THE DAKOTAS EARLY THIS AFTERNOON. THE WINDS WILL GRADUALLY SHIFT TO THE WEST IN THE NEXT COUPLE HOURS...AND THEN CONTINUE TO SHIFT TO THE NORTHWEST BY MID AFTERNOON. THE ACTUAL COLDER AIR WILL NOT PUSH INTO THE AREA UNTIL LATE THIS AFTERNOON. .TODAY... SKY/WE A T H E R ..........PARTLY CLOUDY. MAX T E M P E R A T U R E.....AROUND 63. MIN H U MIDITY........ 31 PERCENT. WIND (20 F T ) ........ SOUTHWEST WINDS 8 TO 16 MPH SHIFTING TO THE NORTHWEST 17 TO 20 MPH IN THE AFTERNOON. MIXING H E I G H T ....... 3500 TO 4500 FT AGL. TRANSPORT W I N D S .....WEST AROUND 20 MPH. SMOKE D I S P E R S A L.....GOOD. .TONIGHT... SKY/WEATHER..........MOSTLY CLOUDY. SLIGHT CHANCE OF RAIN SHOWERS IN THE E V ENING.. .THEN SLIGHT CHANCE OF SNOW SHOWERS AFTER MIDNIGHT. MIN T E M P E R A T U R E.....AROUND 25. MAX H U MIDITY........ 88 PERCENT. WIND (20 F T ) ........ NORTH WINDS 10 TO 20 MPH SHIFTING TO THE NORTHEAST 7 TO 8 MPH AFTER MIDNIGHT. C W R ................... 12 PERCENT. MIXING H E I G H T ....... 900 TO 1400 FT AGL. TRANSPORT W I N D S .....NORTH AROUND 15 MPH. SMOKE D I S P E R S A L.....POOR. .THURSDAY... SKY/WEATHER..........MOSTLY CLOUDY. CHANCE OF SNOW IN THE MORNING. MAX T E M P E R A T U R E.....AROUND 47. MIN H U MIDITY........ 38 PERCENT. WIND (20 F T ) ........ EAST WINDS 8 TO 12 MPH. C W R ................... 18 PERCENT. SMOKE D I S P E R S A L.....POOR. S$ FORECASTER.. .CALDERON REQUESTED BY...MARTI DAHLIN TYPE OF R E QUEST...PRESCRIBED .TAG 20130403.PAUTR.01/UNR Figure 3.5. Rapid City WFO spot forecast for Pasture 3B Prescribed Burn, 3 April 2013. 63 Figure 3.6. The location of the Pasture 3B Prescribed Burn and its verifying observation (KHEI) relative to the CWAs of surrounding WFOs. 64 Figure 3.7. Forecasts and verifying data for minimum relative humidity (%) for the Pasture 3B prescribed burn on 3 April 2013. BIS forecasts are available at hourly intervals while only the minimum relative humidity value is available from UNR. 65 14 Hour(UTC) Figure 3.8. As in Fig. 3.7, but for maximum wind speed (m s-1). 66 Figure 3.9. EKA spot forecasts used in the analysis. Maps for a) prescribed burns and b) wildfires. 67 a) b) Figure 3.10. Errors for prescribed burn spot forecasts for the EKA CWA. Histograms for a) maximum temperature (oC), b) minimum relative humidity (%), and c) maximum wind speed (m s-1). 68 c) Figure 3.10. Continued 69 a) b) Figure 3.11. Percentages of the total number of cases for differences between spot forecasts and observations (columns) and NDFD forecasts and observations (rows) for EKA. a) contains percentages for prescribed burns and b) for wildfires. Values above (below) the diagonal line indicate the percent of the spot forecasts that are within (greater than) specified ranges of the NDFD forecast values. These ranges are 1oC, 5%, and 1 m s-1 of the NDFD forecast for temperature, relative humidity, and wind speed, respectively. | x = |Spot-Obs| in~C~| x<2.5 2 . 5 < = x < 5 5 < = x < 7 . 5 x>7.5 23.9% / / 17.9% 1.4% / / 7.5% 0 % / / 1.6% o% y / 0.3% 1 .8% y / 10.9% 8.0% / / 4.4% 1.0% / / 2.6% 0% y / 0.4% 0 % / / 2.9% 0.2% y / 3.7% 2.4% y / 0.5% 0.3% y / 0.9% 0 % / / 2.2% 0 % / / 1.6% 0.1% y / 1 .1% 0.6% y / 1.6% | x = |Spot-Obs| in C~| x<2.5 2 . 5 < = x < 5 5 < = x < 7 . 5 x>7.5 10.5% y / 16.8% 3.2% y / 10.5% 0 % / / 4.2% 0 % / / 1.1% 3.2% y / 11.6% 7.4% y / 5.3% o% y / n % 0% y / o% 0% y / 4.2% i.i% y / 6.3% 3.2% y / n % o% y / 1.1% 0 % / / 1.1% 0% y / 1 .1% o% y / 3.2% i.i% y / 2.1% 70 a) b) |x = |Spot-Obs| in ° | x<5 5<=x<10 10<=x<15 x>15 cn e iA o Ql_L_ Q in V V >> II V in v II Vo rLHTi A >» 10.5% / / L 0% 2.5% y / 3.9% 0% / / 4.0% 0% / / 19% 2.9% y / 4,4% 6.5% / / 1.2% 2.4% / / 1.6% 0% / / 3.6% 0% / / 3.2% 1.6% / / 1.5% 42% / / 0.4% 2.5% / / 2.8% 0% / / i . m o% y / 4,1% 1.3% / / 5.8% 9-1% / / 12.0% |x - |Spot-Obs| in % | x<5 5<=x<10 10<=x<15 x>15 _Q o QU _Q LTJ V>s V >. Ill V m il-ni V>, 111 V o m A 24.2% / / 1.1% 7.4% / / 1.1% 0% y / 6.3% o% y / 0% 5.3% S / 8.4% 10.5% / / 1.1% 1.1% y / 0% 0% y / 4.2% 0% / / 5.3% 2.1% y / 2.1% e.3% y y ^ 0% 1.1% y / 0% 0% S / 0% 0% y / 0% 0% y / 1.1% e.3% y y 5.3% Figure 3.12. Same as Fig. 3.11, but for minimum relative humidity. 71 a) b) | x = |Spot-Obs| in ms 11 x<2.5 2.5<=x<5 5<=x<7.5 x>7.5 LO r\j V>, tn LO s- V c_ V n Ln O (M Q LL- Ln n r-z- V>, II >i V LO lo A>1 27.9% S / 15.1% 2.0% / / 13.9% 0% / / 1.8% 0% / / 1.2% 2.1% S / 11.2% 7.3% / / 3.4% 0% y / 3.4% 0% / / 2.5% 0% / / 0.7% 0.2% / / 3.2% 0.7% / / 0.4% o% y / 1.2% 0% / / 0.4% 0% / / 0.4% 0% / / 0.4% 0.5% / / 0.4% | x = |Spot-Obs| x<2.5 Tn ms 1 | 2.5<=x<5 5<=x<7.5 x>7.5 y < 2,5 lh s= V >. c. II _ V tn Q in O CM □LL- LO n i~- z- V II ... V LO in A 41.8% 23.6% 5.5% 3.6% 0 % 1.8% 0 % 0 % 0 % 7.3% 3.6% 5.5% 1.8% 3.6% 0 % 0 % 0 % / / 0 % 0 % 1.8% 0 % 0 % 0 % / / 0 % 0 % 0 % 0 % 0 % 0 % / / 0 % 0 % 0 % Figure 3.13. Same as Fig. 3.11, but for maximum wind speed. 72 a) b) Figure 3.14. As in Fig. 3.10b, but versus the RTMA gridded analysis. a) for prescribed burns, b) as in a) but for wildfires. 73 FNUS76 KEKA 2 41741 EWSEKA SPOT FORECAST FOR MILL CREEK...3IA NATIONAL WEATHER SERVICE EUREKA CA 1041 AM PDT MON OCT 2 4 2 011 FORECAST IS BASED ON IGNITION TIME OF 1045 PDT ON OCTOBER 24. IF CONDITIONS BECOME UNREPRESENTATIVE... CONTACT THE NATIONAL WEATHER SERVICE IN EUREKA AT [phone]. .DISCUSSION... A DRY AIRMASS AND LIGHT OFFSHORE FLOW WILL COMBINE TO KEEP THE REGION UNSEASONABLY DRY THROUGH AT LEAST THURSDAY. THE NEXT CHANCE OF RAIN WILL BE ON FRIDAY NIGHT INTO SATURDAY WITH A WEAK COLD FRONT PASSING THROUGH...BUT THIS FRONT WILL LACK MOISTURE WITH LITTLE OR NO RAINFALL EXPECTED AT THE BURN AREA. .TODAY... SKY/WEATHER.............MOSTLY SUNNY. MAX TEMPERATURE......70 TO 75. MIN HUMIDITY...........2 6 TO 31 PERCENT. EYE LEVEL WINDS......UPSLOPE OR WEST 3 TO 5 MPH. SURROUNDING RIDGE...NORTHWEST 4 TO 7 MPH...GUSTS TO 10 MPH IN THE AFTERNOON. WIND (20 FT)...........UPSLOPE OR WEST 4 TO 6 MPH. .TONIGHT... SKY/WEATHER.............MOSTLY CLEAR. MIN TEMPERATURE......42 TO 46. MAX HUMIDITY...........80 PERCENT DRAINAGE... 60 TO 70% UPPER SLOPES. EYE LEVEL WINDS......DOWNSLOPE OR NORTHEAST 1 TO 3 MPH. SURROUNDING RIDGE...NORTHEAST 3 TO 4 MPH. WIND (20 FT)...........NORTHWEST WINDS AROUND 5 MPH IN THE EVENING BECOMING NORTHEAST AND LIGHT. FORECASTER...TONKIN REQUESTED BY...ROCKY COLEGROVE TYPE OF REQUEST...PRESCRIBED ■TAG 20111024.MILLC.02/EKA. Figure 3.15. Mill Creek prescribed burn spot forecast for 24 October 2011. 74 Figure 3.16. Location of Mill Creek prescribed burn (black marker) relative to Big Hill RAWS (red-filled circle). 75 Figure 3.17. Forecasts and verifying data for maximum temperature (oC) for the Mill Creek prescribed burn on 24 October 2011. 100 20 °L5 16 17 18 19 20 21 22 23 24 Hour (UTC) Figure 3.18. As in Fig. 3.17, but for minimum relative humidity (%). 76 a) b) Figure 3.19. Errors for prescribed burn spot forecasts for the TWC CWA. For a) maximum temperature (oC), b) minimum relative humidity (%), c) as in a) but for wildfire forecasts, and d) as in b) but for wildfire forecasts. 77 c) d) Figure 3.19. Continued 78 a) b) Figure 3.20. As in Fig. 3.19a-b, but for NDFD forecasts. 79 a) b) NDFD-Spot Max Temperature (C) = B0 - 70 -60 -50 -40 -30 ^20 -10 0 10 20 30 40 Figure 3.21. As in Fig. 3.19a-b, but for differences between NDFD forecasts and spot forecasts. 80 a) b) | x - |Spot-Obs| in C | x<2.5 2.5<=x<5 5<=x<7.5 x>7.5 in <N U □ in in 12.1% / / 38.3% 2.3% y / 20.6% 0 % / / 0.9% 0 % / / 0 % / 9.8% 2.3% / / 2.3% o % y / 2.8% 0 % / / 0 % 0 % / / 0.5% 0 % / / 1.9% 0 % / / 0 % 0% y / 1.4% 0 % / / 2.8% 0 % / / 0 % 0% y / 0 % 0 % / / 0.5% | x = |Spot-Obs| in C | x<2.5 2.5<=x<5 5<=x<7.5 x>7.5 u m Ln 11.7% 26.4% 0.7% 20.5% 0 % 4.4% 0 % 0 % 0.4% 16.5% 2.9% 2.9% 0 % 3.7% 0 % 0.4% 0 % / / 1.5% 0 % 0.7% 0.7% 1.1% 0 % 0.7% 0 % / / 1.8% 0 % 0.7% 0 % 0.7% 0.4% 1.1% Figure 3.22. As in Fig. 3.11, but for TWC forecasts. 81 a) b) | x = |Spot-Obs| in ms 11 x< 2 .5 2 .5< = x< 5 5< = x< 7 .5 x> 7 .5 y < 2 .5 yi Lfl F V c >. II _ V a Ln O rsi Q LL- Lfl Q I'' 7-* >V, IIV in in A> 1 28.9% / / 25.6% 0% y / 7.8% 0% / / 2.8% 0% / / 0% 3.3% / / 11.7% 7.2% / / 3.9% 0.6% / / 0.6% 0% / / 0% 0% / / 0.6% 0.6% / / 2.8% 1 .1% y / 0% 0.6% / X °-6% 0% / / 0% 0% / / 0.6% 0% J / 1.1% 0% / / 0% |x = |Spot-Obs| in % x<5 5< = x< 1 0 1 0< = x< 1 5 x>15 LO V> . V>NII mV ii-ni V VII V o LT)HA 27.1% y / 15.0% 8.1% / / 12.5% 0% y / 2.6% 0% y / 0.4% 5.5% / / 8.4% 2.9% / / 1.5% 0.7% y / 0.4% o% y / 1.8% 0% / / 1.8% 0.7% y / 2.2% 1.1% y / 0% 0.7% y / 0.4% 0% / / 1.8% o% y / 2.6% 0.4% y / 0.4% 1.1% y y 0% Figure 3.23. As in Fig. 3.11, but for TWC minimum relative humidity forecasts. 82 a) b) x = |Spot-Obs| in ms 1' x< 2 .5 2 .5< = x< 5 5< = x< 7 .5 x>7 .5 y< 2 .5 kO Lfl S= V c >. II _ V in O rN Q LL_ un n I"' z V - >. II - V LO in A >. 28.9% / / 25.6% 0% y / 7.8% 0% y / 2.8% o% y / 0% 3.3% y / 11-7% 7.2% / / 3.9% 0.6% y / 0.6% 0% y / 0% 0% / / 0.6% 0.6% y / 2.8% i.i% y ^ y o% 0.6% y / 0.6% 0% / / 0% 0% y / 0.6% o% y / 1.1% o% y / 0% | x = |Spot-Obs| in m s 1 | x< 2 .5 2 .5< = x< 5 5< = x< 7 .5 x> 7 .5 y < 2 , 5 kO Lfl S= V > c II _ V nt/) i n O r\j □ LL-Lfl n h-z V - > . ii II V i n in A 27.5% y / 19.9% 0.9% y / 3.3% o% y / 1.4% o% y / 0.5% 1.4% y / 14.7% s.5% y / 3.8% 0% y / 1.9% o% y / 2.4% 0% / / 3.8% 0% y / 4.3% 0% y y y 0% o% y / 1.9% 0% y / 0.5% 0% y / 0.5% o% y / 1-4% 0.5% y / 0.9% Figure 3.24. As in Fig. 3.11, but for TWC maximum wind speed forecasts. 83 a) b) Figure 3.25. As in Fig. 3.9, but for MLB forecasts. 84 a) b) Figure 3.26. As in Fig. 3.10a-b, but for MLB forecasts. 85 a) b) Figure 3.27. As in Fig. 3.11, but for MLB forecasts. 86 a) b) | x = |Spot-Obs| in % | x<5 5< = x< 1 0 1 0< = x< 1 5 x>15 V) _Q o 6Ll. Q in V V>H II mV V>, 111 V o in ■-^ A 38.6% S / 0.9% 4.0% / / 1.0% 0% / / 0.3% 0% / / 0% 5.0% / / 1.5% 20.3% / / 0% 3.1% y / 0.1% 0% / / 0.1% 0% / / 0.5% 3.1% S / 0.6% 7.4% S / 0% 1.3% / / 0% 0% / / 1-4% 0% / / 1.0% 1.8% / / 0.6% 6.8% / / 0.6% |x - |Spot-Obs| in % | x<5 5< = x< 1 0 1 0< = x< 1 5 x>15 to _Q 01QLL_ C l mv >. > II V ID V > II V o in *=H A 25.4% / / 0.5% 3.8% / / 3.2% 0% y / 0% 0% y / 0.5% 7.6% / / 3.2% 17.8% / / 0% 1.1% y / 0% 0% y / 0% 0% / / 1.1% 3.8% y / 2.2% 8.1% y / 0% 0.5% y y o% 0% / / 5.9% 0% y / 3.8% 0.5% y y 2.2% 7.6% y / 1.1% Figure 3.28. As in Fig. 3.11, but for MLB minimum relative humidity forecasts. 87 a) b) Figure 3.29. As in Fig. 3.11, but for MLB maximum wind speed forecasts. 88 a) b) Figure 3.30. As in Fig. 3.19, but for MLB forecasts compared to RTMA. 89 c) d) Figure 3.30. Continued Figure 3.31. As in Fig. 1.2, but limited to forecasts in the final analysis. 90 Figure 3.32. Prescribed burn spot forecasts by month. 91 a) b) Figure 3.33. All maximum temperature errors (oC) for prescribed burn forecasts. Histogram a) contains spot forecast errors and histogram b) contains NDFD forecast errors. 92 93 Figure 3.34. Mean error and median absolute error (MAE) for prescribed burn spot forecasts for maximum temperature (oC) as a function of CWA. a) features mean error values while b) contains median absolute errors. 94 x = |Spot-Qbs| in C LO fNj V> , Lfl u V> c , II _U_) V -Q in o r-j Q LL- ln Q i-- -Z--- V> , II II > , V LO LO 1^ A>, x<2.5 2.5< = x<5 5<=x<7.5 x>7.5 40.5% / 1.3% / o% y o% y X 18.8% / 4.6%y 0.5% y 0.1% 2.2% / 7.3% / o.4% y 0% y / 9.4% / 2.5%y i.o%y 0.2% 0% / 0.2% / 1.4% y o.i% y / 1-7% / 1.0%y o.5% y 0.3% 0% y 0% y o% y 0.5% y X 2.8% y o.o%y o.6%y o.4% Figure 3.35. As in Fig. 3.11a, but for forecasts from all CWAs. a) b) Figure 3.36. As in Fig. 3.33, but for minimum relative humidity (%). 95 96 Figure 3.37. As in Fig. 3.34, but for minimum relative humidity (%). 97 x = |Spot-Obs| in % x<5 5<=x<10 10<=x<15 x>15 26.2% / / 2.0% 5.2% / / 3.4% o% y y i.7% o% y y 0.7% 5.0% / / 4.8% 12.9% / / 0.6% 2.4% y y i.o% o% y y 1.1% 0% / X 2.7% 2.6% / / 2.1% s.6% y y o.i% i.i% y y o.o% 0% y / 3.2% 0% y y 3.1% 1.1% y y 2.2% s.6% y y 2.9% Figure 3.38. As in Fig. 3.11a, but for minimum relative humidity (%) forecasts from all CWAs. 00 On •( s uj) paads puiA\ ujnujixEuj jo} jnq ‘££'£ 'S y ui s y '(,£'£ ajn S y (q 99 Figure 3.40. As in Fig. 3.34, but for maximum wind speed (m s-1). 100 x = |Spot-Qbs| in ms 1 LO (N V> , in F V> c , II __ V ■Q in o r-j Q LL- LO Q I-- Z V -- > , II II > , V LO x < 2.5 2 .5 < = x < 5 5 < = x < 7.5 x > 7.5 LO 1^ A>, 43.8% y / 21.2% 1.0% / / 6.5% o% y y 0.7% o% y y 0.3% 1.9% / X 8.1% 7.1% / X 2.3% o.i% y y 0.9% o% y y 0.3% 0% / / 0.8% 0.2% y y 1.3% o.8% y y 0.2% o.i% y y 0.2% 0% y / 0.2% o% y y 0.3% o% y y 0.3% 0.2% y y 0.2% Figure 3.41. As in Fig. 3.11a, but for maximum wind speed (m s-1) forecasts from all CWAs. Figure 3.42. As in Fig. 1.3, but limited to spot forecasts in the final analysis. 101 Figure 3.43. As in Fig. 3.32, but for wildfire forecasts. 102 a) b) Figure 3.44. As in Fig. 3.33, but for wildfire forecasts. 103 104 Figure 3.45. As in Fig. 3.34, but for wildfire forecasts. 105 x = |Spot-Qbs| in C x < 2.5 2 .5 < = x < 5 5 < = x < 7.5 x > 7.5 L f l fNj V> , L f l u V> , c II _ _ V U ) -Q i n o r \ i QL L- L / l Q i - - Z V ----- > , II II > , V LO in 1^ A>, 27.6% 20.7% 1.5% 7.7% 0% 1.3% 0% 0.3% 1.7% 11.9% 7.4% 2.9% 0.6% 1.8% 0% 0.3% 0% / / 2.7% 0.1% 2.5% 1.9% 0.9% 0.1% 0.5% 0% 2.0% 0% 1.0% 0.1% 0.9% 0.6% 0.8% Figure 3.46. As in Fig. 3.11b, but for forecasts from all CWAs a) b) Figure 3.47. As in Fig. 3.33, but for wildfire minimum relative humidity forecasts. 106 107 Figure 3.48. As in Fig. 3.34, but for wildfire minimum relative humidity forecasts. 108 x = |Spot-Obs| in % x<5 5< = x< 10 10< = x< 15 x>15 34.0% y X 3.1% e.3% y X 3.0% 0% / / 1.7% 0% / X 0.5% 5.9% / X 5.8% 11.6% / X 0.5% 2.2% y yX 0.7% 0% / 1.0% 0% / / 2.5% 2.3% y X 1.8% 3.6% / X 0.1% 1.0% / X 0.5% 0% / X 2.0% 0% / X 1.8% o.e% / X 1.2% 3.7% y >y i.6% Figure 3.49. As in Fig. 3.11b, but for minimum relative humidity forecasts from all CWAs. a) b) NDFD-Obs Max Wind Speed - 3,000 - 2,500 - 2,000 - 1,500 - 1,000 Figure 3.50. As in Fig. 3.33, but for wildfire maximum wind speed forecasts. 109 110 Figure 3.51. As in Fig. 3.34, but for wildfire forecasts. 111 x = |Spot-Qbs| in ms 1 x < 2.5 2 .5 < = x < 5 5 < = x < 7.5 x > 7.5 LO (N V> , lfl in F V> c , II __ V ■Q in O r-j Q Ll_ ln Q i-- Z V -- > , II II > , V LO LO 1^ A>, 36.4% y 21.0% 2 ,o% y y 6.e% o% y y 1 .2% o% y y o.6% 2.0% / / 9.1% 8.o% y y 3 .2% 0.3% y y 1 .4% o% y y 0.7% 0% / / 1.0% 0.3% y y 1 .9% o.8% y y 0.3% o% y y o.6% 0% y y o.i% o% y y 0.4% o% y y 0.5% 0.3% y y 0.2% Figure 3.52. As in Fig. 3.11b, but for maximum wind speed forecasts from all CWAs. CHAPTER 4 SUMMARY, CONCLUSIONS, AND FUTURE WORK 4.1 Summary The objective of this study was to develop a framework to evaluate spot forecasts for the benefit of the forecasters who provide them as well as the fire and emergency management professionals who request them. While commercial software is available for individuals to quantify metrics and distributions of forecast errors, flexible open source web tools were developed to allow users to evaluate the cases of interest to them, which help identify the causes and ramifications of forecast errors. To implement these verification tools is beyond the scope of this study and requires transitioning or developing similar capabilities in an operational entity, such as the NWS Performance Branch. To demonstrate the capabilities of the tools developed, case studies of specific spot forecasts are examined as well as statistics aggregated by NWS CWA and nationwide for spot forecasts issued for prescribed burns and wildfires for three variables (maximum temperature, minimum relative humidity, and maximum wind speed) that are critical for estimating fire danger and behavior. The statistical summaries reviewed briefly in the next section are not definitive evaluations of spot forecast accuracy for a number of reasons, as discussed further in Section 4.3. 4.2 Prescribed and Wildfire Spot Forecasts Prescribed burns are anticipated in management plans developed by wildland fire management officials at lead times of months or even years for publicly owned locations. Wildfires are spontaneous and can occur anywhere there is fuel to burn. Forecasters are called upon to provide detailed forecasts conditions for both types of fires. Because of the need for immediate assessment of potential fire danger, wildfire forecasts are turned around quickly, with 71% having lead time (the difference between the recorded receipt of the request and spot forecast issuance) less than 50 minutes (Fig. 4.1b). For prescribed burns, only 27% are issued in 50 minutes or less from the recorded receipt of the request, suggesting that forecasters have more time to evaluate the forecast situation (Fig. 4.1a). If accuracy is dependent upon lead time, it is likely that prescribed burn spot forecasts would be on the whole more accurate than wildfire spot forecasts. While there are mitigating factors, such as the geographic extent of larger prescribed burns relative to small, recently discovered wildfires, that also impact accuracy, it is the case for two of the study variables that prescribed burns do show increased accuracy. Accepting the limitations inherent in the aggregate statistics for prescribed and wildfire spot forecasts for the nation as a whole as summarized in Tables 3.2 and 3.3, prescribed burn spot forecasts tend to be more accurate than wildfire spot forecasts for maximum temperature (75.4% versus 66.6%) and maximum wind speed (76% versus 70.4%) while the reverse is apparent for minimum relative humidity (43.9% of prescribed burns versus 53.3% of wildfires). Of course, the overall percentages summarized in Table 3.3 of "accurate" forecasts are affected by the thresholds selected (2.5oC, 5%, and 2.5 m s-1 for temperature, relative humidity, and wind speed, respectively). The lower accuracy for minimum relative humidity forecasts for prescribed burns is more pronounced 113 for the forecasters in the NWS Western region: 42.6% (55.8%) accuracy for prescribed burn (wildfire) forecasts (not shown). Similarly, the MAEs for temperature and wind speed forecasts are lower for prescribed burns than wildfires with the reverse true for relative humidity (Table 3.2). Hence, relative humidity is the most difficult of the three variables to forecast and having additional lead time to prepare relative humidity forecasts does not appear to provide additional benefit. An objective of this study was to assess the specific guidance provided by the forecasters relative to that available from the grids they develop for more general purposes that are archived as part of the NDFD. This evaluation is far from a comprehensive evaluation of the skill of the spot forecasts, since it is restricted to assessing the spot forecasts of temperature, relative humidity, and wind speed during the afternoon relative to NDFD grids available earlier in the day at 0900 UTC. Every summary metric (e.g., see Tables 3.2 and 3.3) indicates that the accuracy of spot forecasts is higher than that of the values obtained from the NDFD grids for both prescribed burns and wildfires. The improvement is largest for maximum temperature (9.6% and 7.5% for prescribed burns and wildfires, respectively) and lowest for maximum wind speed (1.6% for both categories). Hence, while forecasters often adjust their forecasts to deviate from the NDFD values, those adjustments do not display substantial improvement for maximum wind speed while adding considerable value for maximum temperature. 4.3 Recommendations Forecast verification is a continual, ongoing process. Tools must be in place that make it possible for the forecasters and users of the forecasts to quickly examine cases and aggregate statistics of interest to them using their experience and local knowledge, rather than depending 114 on bulk statistical metrics accumulated on national scales as summarized in the previous section. In order to develop useful verification tools for spot forecasts operationally requires minimizing some of the underlying limitations identified during this study. The principal recommendation of this study is to leave the decisions as to what to verify and how to verify the forecasts in the hands of the forecasters and end users by developing flexible methods to explore the multidimensional nature of the forecasts. Foremost is simply the need to be able to examine in a centralized framework: the requests; the forecasts; geolocation information; and nearby observations and other information relevant to analyzing the forecast situation. While this may seem obvious, it has never been possible before this study developed such capabilities. Then, the user should be able to explore and control interactively key parameters (e.g., distance to the verifying observations, forecast lead times, magnitudes of the parameters, or magnitudes of the errors). Currently, much of the verification performed on the federal level boils down to aggregate statistics that fail to capture the nuance necessary for evaluating spot forecasts, something that the online tools enable. In order to make the tools described in this study more appropriate for operational use, several limitations need to be overcome, as summarized in bullet points here and detailed below: • Isolate quantitative numerical values separately from qualitative alphabetical descriptors. • Make forecast wind level a numerical parameter adjustable within the request form, so that even when it is not "20-Foot," the level is known for evaluation. • Store the name of or abbreviation referencing the specific station for verification as part of the request form. This should include stations from networks not used in this study. 115 4.3.1 Separating Numerical Values and Text in the Spot Forecast As every WFO coordinates locally with its user community to specify the fields to be included in the spot request, creating a text parser that correctly captures the intention of the forecasts on a national basis is a complex and frustrating endeavor. While the parsing algorithms developed here for maximum temperature, minimum relative humidity, and maximum wind speed were found to be adequate after numerous iterations, they occasionally still fail either by eliminating forecasts from consideration (most often for wind speed) or populating spurious values for verification. The former prevents the forecaster from being able to view the results from specific cases, discouraging them from continuing to use the real­time web products we have developed. The latter corrupts aggregate statistics and also frustrates the forecaster trying to observe their verification values. Of course, the factors that make parsing difficult add value for fire and emergency management professionals, contributing layers of detail on timing and variability that help to make critical personnel and containment decisions. This free form information should not be removed from the spot forecasts, nor should all the requests be standardized into a single request form on a national basis. Rather, alternative methods to separate the basic numerical information from the free form information should be straightforward to implement. One way that this is occasionally done in spot forecasts is by including hourly or bi-hourly values for the meteorological fields, from which maximum and minimum values can be easily extracted. An example of this can be seen in the Bismarck forecast for the Pasture 3B Prescribed Burn (Fig. 3.5). There is the possibility of underrepresenting extreme values in some cases, as the anticipated extrema might occur during an intermediate time not listed in the forecast. Nevertheless, demarcating critical numbers from text fields would eliminate the 116 117 difficulty with handling words and times that hampered this study. 4.3.2 Consistency in the Request Form As mentioned in the discussion of wind speed parsing in Chapter 2, the adjustments required during this study to capture all the variations to simply define the height of the wind forecasts should be avoided in the future. Initially, it was assumed after inspection of many request and forecast pairs that to parse maximum wind speed from the forecasts required searching for "20 FOOT" or "20 FT." However, forecast offices changed over time their definitions (e.g., SLC WFO in late-2010 switched to "20-FOOT" while others switched to 20FT) or, due to requests from their users, other offices use "EYE LEVEL" (Boise) or "GENERAL" (Marquette). Presumably, in the latter cases, communication with those specific user communities could lead to standardization of what is provided to them. More generally, the level of the wind should be specified in the request form (not input or selectable independently by the forecaster in the spot forecast) so that the values provided in the spot forecast are numerical values supplemented by the height descriptor from the request form. 4.3.3 Allowing Users to Specify the Verifying Data The number of forecasts that were not analyzed in this study because of lack of nearby observations was relatively small because observations at often large distances from the spot location were allowe