Leveraging the Uintah Computational Framework for Commercial Simulation of Industrial Flares

Paper from the AFRC 2018 conference titled Leveraging the Uintah Computational Framework for Commercial Simulation of Industrial Flares

With funding from the Department of Energy, to further the use of government funded high; performance computing (HPC) software in the analysis of industrial problems, REI is working; with the University of Utah to leverage the Uintah Computational Framework for simulation of; industrial flares. The Phase I program leveraged the strength of the LES basis of the Arches; component of the UCF, along with its demonstrated scalability to O(100K) processes, to simulate; industrial two industrial flares that were tested as part of the Texas Commission on Environmental; Quality (TCEQ) flare study completed in 2010.; The air-assist flare design from the TCEQ study was the basis of an Arches simulation that was; carried out on the Nimbix commercial HPC facility. The Nimbix platform was selected as the; commercial HPC provider due to its containerized cloud computing capabilities which streamlined; our efforts in developing a web-based interface between the computing resources and Arches.; During the Phase I program, a basic user interface was developed for easing the flare case; definition and setup, case monitoring, and post processing. In addition to the air-assist flare; simulation, a simulation of a steam-assist flare design from the TCEQ study was also carried out.; This paper will present the results of the two flare simulations using resources from a commercial; HPC facility.

AFRC 2018 Industrial Combustion Symposium Leveraging the Uintah Computational Framework for Commercial Simulation of Industrial Flares Marc Cremer, Dave Wang, Matthew McGurn Reaction Engineering International [e-mail] 801-875-4314 Jeremy Thornock University of Utah [e-mail] Abstract With funding from the Department of Energy, to further the use of government funded high performance computing (HPC) software in the analysis of industrial problems, REI is working with the University of Utah to leverage the Uintah Computational Framework for simulation of industrial flares. The Phase I program leveraged the strength of the LES basis of the Arches component of the UCF, along with its demonstrated scalability to O(100K) processes, to simulate industrial two industrial flares that were tested as part of the Texas Commission on Environmental Quality (TCEQ) flare study completed in 2010. The air-assist flare design from the TCEQ study was the basis of an Arches simulation that was carried out on the Nimbix commercial HPC facility. The Nimbix platform was selected as the commercial HPC provider due to its containerized cloud computing capabilities which streamlined our efforts in developing a web-based interface between the computing resources and Arches. During the Phase I program, a basic user interface was developed for easing the flare case definition and setup, case monitoring, and post processing. In addition to the air-assist flare simulation, a simulation of a steam-assist flare design from the TCEQ study was also carried out. This paper will present the results of the two flare simulations using resources from a commercial HPC facility. 1 Introduction Waste gases generated at industrial facilities must be controlled using safe and economically sound methods to protect the health of the public and the environment. Flaring, the high-temperature oxidation process used to burn combustible components of waste gases from industrial operations is used throughout the world to reduce the impact of waste gases such as volatile organic compounds (VOCs), toxic compounds, and other pollutants. The technology is widely used in oil and gas production and in petroleum refining and chemical processing. It is the standard practice for mitigating the release of unburnt waste gases into the atmosphere for safety reasons often associated with upset conditions, process startup or shutdown, or during process emergencies to avoid over-pressurization [1]. Flaring is the more desirable option compared with venting, where unburned hydrocarbons are vented directly to the atmosphere. AFRC 2018 Industrial Combustion Symposium Flare simulation software that is available today must simulate combustion under a very complex set of environmental conditions and often lack the detailed physics that are required to fully describe the complex behavior involved. Much more detailed, super-computer-based simulations are required to meet the needs of these users. These tools exist, but have traditionally been limited to expert-level users at universities and national labs. Although modeling and simulation tools and the development of high performance computing (HPC) facilities, have seen dramatic advances in recent years in the US, their use by US industry for engineering and manufacturing has been severely lagging [2, 3]. For the most part, use of high fidelity computational fluid dynamics (CFD) tools, such as those incorporating use of large eddy simulation (LES), for analysis of industrial combustion processes such as industrial flares, has remained largely an academic research exercise [4, 5, 6]. These investigations have used high end HPC facilities developed and utilized through significant governmental funding, such as through the DOE Office of Advanced Scientific Computing (ASCR), and available only to research institutions. In most cases, small to medium sized companies do not have the resources to develop and maintain the in-house expertise or hardware to support these advanced modeling and simulation capabilities [7]. However, economical access to these high-fidelity modeling and simulation tools as well as to the HPC facilities able to support them could have significant impact on the ability to improve design, manufacturing, and operational know-how. This access will improve the ability of small to medium sized US businesses to meet long term US goals associated with improved air quality while maintaining an economical advantage in the global market. This paper describes REI's application of Arches and other high-fidelity modeling components within the UCF to simulate industrial flare performance on a publicly available HPC platform. This was achieved for 2 elevated flare types, air-assisted and steam-assisted. Additionally, REI developed and demonstrated the use of a user interface intended to allow a user to prescribe simulation inputs, such as flare geometry, operating conditions, and atmospheric conditions; initiate and continuously monitor computed parameters as the simulation progresses; and postprocess the final results. 2 Methods, Assumptions, and Procedures The development of a user accessible simulation framework requires the specification of the modeling packages, user interface, and specific modeling cases. This section outlines the methods, assumptions, and procedures used to define, select, and develop these components. 2.1 Uintah Computational Framework REI's approach was to leverage the use of the highly advanced UCF along with existing and developing publicly available HPC resources to demonstrate that these advanced simulation tools and commercially available HPC facilities can be applied to commercial simulation of industrial flares. The UCF [8, 9] is a massively-parallel software infrastructure designed to solve partial differential equations (PDEs) [9, 5] such as those arising in fluid dynamics, reacting flows, and combustion [6]. Uintah provides a set of tools that scientists and engineers can use to develop physics-based utilities (called components) to simulate real-world phenomena [10]. To that end, several components have been developed over the years to leverage the scalability and parallel features of Uintah. These range from tools to simulate material crack propagation to buoyancydriven flows and fully compressible flows. In particular, research efforts using the UCF (e.g. AFRC 2018 Industrial Combustion Symposium Arches) have been carried out demonstrating the ability of UCF in accurately capturing flare combustion dynamics not achievable with RANS based models. Arches is a UCF component that focuses on solving reacting flow problems such as those encountered in flares. Arches has well-documented and verified LES capabilities and has been shown to accurately predict turbulent flows. It was originally developed for simulation of the buoyancy driven flows associated with pool fires along with the combustion and radiation heat transfer models for prediction of heat transfer to objects within or near the pool fire including the impacts of soot. The UCF was developed to be highly scalable and has been shown to be scalable to thousands of processors. 2.2 User interface development The original plan for the UI was to utilize three primary components; end-user's machine, a control server, and a HPC facility. The user's machine would be presented with the web-interface through any standards compliant browser. The end-user would input parameters, start simulations, and monitor cases through this user-friendly web interface. The control server would act as web-host and job manager for the cloud-based services. However, recent developments in the industry have removed the requirement for three separate components with a single Hybrid Provider. Supercomputing providers such as Nimbix have recognized a market opportunity to provide a method to sell compute time and 3rd party software together. With this paradigm, companies host their software and user interfaces on provides such as Nimbix. This platform is already being used by developers such as Ansys, Abaqus, CD-adapco along with others. A comparison was performed based on the traditional proposed solution and the hybrid provider, with a summary provided in Table 1. Based upon the comparison, it was decided to perform the fast development/prototyping with the hybrid approach. This approach will allow REI to incorporate feedback from university and industry partners in an iterative fashion. As they use beta versions of the software any comments or requests can be incorporated back into the design much faster because of this paradigm, resulting in a software package tailored for the end customers. Table 1: Comparison between Proposed and Hybrid UI/HPC Providers Proposed Web/Controller/HPC Solution Hybrid Provider Separate web, controller, HPC providers Single web/HPC provider Pros: Pros:  Multiple HPC providers available  More scalable Cons:    Increase development time Slower implementation/ development cycle Increased communication complexity    Faster prototype/incorporate more user feedback Simpler implementation Billing/data management/accounts handled by provider Cons:   Less deployment flexibility (one provider at a time) Scalability limited to provider AFRC 2018 Industrial Combustion Symposium 2.2.1 High Performance Computing Nimbix provides industry standard HPC hardware with high speed interconnects. A variety of node types is available to the end user for an appropriate workload, from low CPU count/high memory to high CPU count/high memory, and various combinations with multiple GPU types. However, developers can customize their application to guide the user into using appropriate node types. The JARVICE platform encompasses several components: a web interface (Material Compute), application deployment (PushToCompute), a web-based Application Programming Interface (API) for job control/monitoring, and several job UI access methods. 2.2.2 Application Development Traditional user interface development tightly ties the specific interface to the simulation framework, in this case Uintah. This paradigm results in an interface that takes significant time and customization for the problem at hand. During this project a dynamic schema-based UI was developed that remedies many of these problems. Many modern computational frameworks utilize Extensible Markup Language (XML) for providing the problem statement. The XML specification is an open standard governed by World Wide Web Consortium (W3C) [11] and therefore allows easy communication between products and services from multiple companies. Once the selection on the UI methodology was chosen the development language had to be selected. A survey of UI frameworks was performed including PHP, RubyOnRails, and Electron which allows apps to be built using JavaScript, HTML, and CSS. It was decided to build the application in the cross-platform JavaFX framework. JavaFX, originally developed by Oracle, is part of the open source OpenJDK project licensed under a GPL based license with only minor exceptions [12]. The open source nature allows for the longevity of the standard due to not being owned by a single cooperation such as Microsoft, Apple, or Oracle. JavaFX applications can be run locally, on windowing servers such as Nimbix, or on webservers using packages such as JPro [13]. This versatility allows for the developed interface to be used on a wide variety of platforms and applications. This also speeds development allowing for the application UI to be run and changed locally on end users' machines. For visualization, it was decided to utilize the VisIt visualization framework within the UI. VisIt is an open source tool originally developed by the Department of Energy [14]. VisIt is available under the BSD license, thereby allowing the distribution and use of VisIt within the interface. VisIt includes a series of features that allow for direct communication using a Java Application Programming Interface (API). This allows control of the visualization directly within the developed application producing a seamless experience for the user. The development of the UI was divided into two primary stages; the development of the primary shared framework (henceforth referred to as DynamicUI) and the flare modeling specific implementation of said framework (FlaresUI). The base framework DynamicUI can be utilized and customized for any XML based simulation software package. The DynamicUI can be customized for applications such as FlaresUI without modifying the base DyamicUI. 2.3 Industrial Flare Cases Two flares were selected to examine the real-world time & cost of running flare simulations on commercial HPC facilities and to help guide the development of the UI and show Arches capabilities. An air and steam assisted flare were chosen to be modeled from the TCEQ 2010 Flare AFRC 2018 Industrial Combustion Symposium Study Final Report [15]. Details of the physical and environmental parameters of each case are outlined in this section. 2.3.1 Air Assist Flare Simulation A simulation of air assisted flare described in the TCEQ 2010 study final report [15] as A2.1 Run 2 was chosen. The air assisted flare was Model LHTS-24/60 and was provided by John Zink. The technical drawing in the TCEQ final report was used to construct the geometrical representation of the flare. Figure 1 shows the overall target industrial flare (John Zink model LHTS-2/60 air assisted flare). The LHTS-24/60 Flare has a tip diameter of 24 inch with 3 Pilots with maximum capacity of fuel flow rate of 144,000 lb/hr. This flare size and design configuration represents a large number of flare models currently in the field. The flare exit tip is 33 feet above ground level and the flare bottom diameter is 12.3 feet. The flare tip detail is shown in Figure 2. Figure 1: John Zink Model LHTS-24/60 air assisted flare tested in TCEQ 2010 Flare Study Figure 2: Flare Tip of John Zink Model LHTS-24/60 air assisted flare AFRC 2018 Industrial Combustion Symposium The operating condition of A2.1 Run 2 in the TCEQ 2010 study final report is listed in Table 2. A2.1 Run 2 uses pure propylene as the fuel at flow rate of 355 lb/hr, ~0.25% of maximum capacity (144,000 lb/hr). This fuel gas flow rate is in the range of operation for typical flow rates (less than 0.5% of maximum capacity) used in industry. Table 2: Operating Condition of A2.1 Run2 Parameter Value Fuel Fuel C3H6 Mole Fraction 1.00 Fuel Flow Rate (lb/hr) 355 o Fuel Initial Temperature ( F) 95 Environments Barometer Pressure (psia) 14.24 Ambient Temperature (oF) 81.5 Wind Speed (MPH) 12.8 Assisted Air Assisted Air Flow Rate (lb/hr) o Assisted Air Temperature ( F) 83,818 81.5 Figure 3 shows the overall chosen simulation domain (66 ft x 48 ft x 36 ft). The simulation domain is constructed as following: a) Wind direction is chosen as the direction from -y to +y b) Domain height is chosen as twice the height of flare (2*33= 66 ft), starting from the ground level c) In the wind direction, the length is chosen as four times of flare bottom diameter (4*12 = 48 ft), the flare center location then is placed as 1.5 times of flare bottom diameter form wind inlet (-Y) d) In the other direction, the length is chosen as three times of flare bottom diameter (3*12 = 36 ft), the flare center location is placed in the center of that direction. e) Uniform resolution was chosen in three directions. Figure 3: Simulation Domain of A2.1 run2 AFRC 2018 Industrial Combustion Symposium ARCHES is a LES based code with a chemistry model that takes advantage of the increased representation of the temporal scales. This model is best described as a rate controlled constrained equilibrium (RCCE) approach [16]. Full kinetics combustion representation is difficult for LES because the reaction set is typically composed of several tightly coupled reactions with rates that span a range of timescales and are often not well defined. The numerical stiffness present in such a system leads to an inability to obtain a converged result. The other extreme in combustion modeling is to use chemical equilibrium. The use of full chemical equilibrium is not appropriate in cases where slower time scales have a significant impact on the combustion chemistry, as in the case of industrial flare. The RCCE model attempts to inject a time-scale dependence by representing a sub-set of the full chemical mechanism directly and then using the result to constrain the resulting equilibrium concentration. In other words, the resolved rates on the CFD mesh are the rate limiting steps for the combustion. All other combustion reactions proceed instantaneously to chemical equilibrium as constrained by the products of the slow reactions. With this model, ignition and quenching behavior can be predicted. Such behaviors are critical in predicting the effects of flare fuel stripping, where aerodynamic effects between the flare stack and head along with interactions with the plume of hot combustion gasses cause eddies to strip unburnt fuel away from the combustion zone. Subsequent dilution with air and cooling of the stripped fuel permanently results in combustion inefficiency. In the RCCE model for ARCHES simulation, additional constraints are placed on the rate terms for the LES resolved processes. Most notably it has included the effect of a flammability limit. Regions of mixture that are outside of the flammability limit are not allowed to progress, thus allowing for further quenching. The flammability limits of C1, C2, and C3 hydrocarbons, as described in the Ref. [17] "nose-plots". 2.3.2 Steam‐Assist Flare Two flare simulations were performed with the Arches code using the steam-assist geometry as reported in the Texas Commission on Environmental Quality (TCEQ) Flare Study 2010 [15]. The purpose of the simulations was to evaluate the code and to perform a qualitative comparison to data presented in the TCEQ report. Photographs of the flare stack, as taken from the TCEQ report, are presented in Figure 4. Two scenarios from the test were considered; with and without steam-assist where the mass flow rates of fuel were nearly the same. Sampled combustion efficiencies for the steam-assist on scenario (TCEQ Test Index S3.5) measured ~96%. Sampled combustion efficiencies for the steamassist-off scenario (TCEQ Test Index S3.6) measured ~99.8%. Crosswind speed for both cases was measured to be around 3MPH. Two steam assist zones are used in the steam-assist flare. The first is a center steam inlet, which is a simple pipe recessed in the flare stack positioned along the axis of the flare. The second is a series of star-shaped steam nozzles together in a manifold surrounding the circumference of the flare stack exit. The purpose of the steam nozzles is to enhance the mixing between the fuel and air and promote good combustion efficiency. However, it is suspected that the steam may actually be a factor in promoting flame quenching, resulting in a lower combustion efficiency. This is particularly true at low fuel flow rates. AFRC 2018 Industrial Combustion Symposium Figure 4: Photographs of the steam assist flare (taken from the TCEQ Flare Study Report 2010 [15]). The photo on the left shows the entire flare stack. The photo on the right shows a close-up of the steam injectors. Compared to the resolution of the LES simulation (2.8cm), the upper steam manifold introduces steam through unresolved, drilled holes of about 15mm diameter. For this simulation, a fairly crude approximation was used, representing the entire manifold with an annulus surrounding the flare exit. A handoff file was pre-generated then read into Arches, which introduced steam at the appropriate mass flow rate and velocity vector angles. The steam mass fraction field, including the boundary conditions, is shown in Figure 5. Figure 5: The instantaneous steam mass fraction field, volume rendered. The flare stack has been cut away in the visualization to illustrate the center and upper steam boundary conditions (highest steam mass fraction). AFRC 2018 Industrial Combustion Symposium 3 Results and Discussion During the Phase I effort significant work was performed in the development of the UI and simulation of the flares. The results of this effort are presented with focus on using the new UI, hardening of Arches, and simulation results. 3.1 Using the FlaresUI The generic DynamicUI can be applied to any problem that can be described using an XSD document. As part of this project an extension to the DynamicUI was implemented for modeling flares, henceforth referred to as FlaresUI. The FlaresUI only exposes a subset of the capabilities in Uintah and Arches using a custom XSD document. This section documents the usage of said interface for setting up and running a simulation. 3.1.1 Setting up a Simulation When the user first logs in to start the simulation, they will be presented with a library of preconfigured flare simulations. As shown in Figure 6, the application knows of two preconfigured flares, outlined in Section 2.3. The library window includes an image of each flare and a short description. Figure 6: The preconfigured flare library. This section will outline the modeling the TCEQ 2010 Air Assist flare, but instead of using the preconfigured library flare the input will entered from scratch. This is started by selecting Empty AFRC 2018 Industrial Combustion Symposium from the library window. This creates an empty project with the required elements of Geometry and Environment shown in Figure 7. The user will select Geometry to see the required parameters where they are expected to provide the STL Geometry of the Flare Geometry and to specify the up direction. The completion action is shown in Figure 8. The environmental parameters are specified in the edit area according to the problem outline in Section 2.3.1. Two inlets must be specified for this flare simulation, the fuel and air inlets. This is done using the "+" menu and selecting inlet. Three types of inlets are available to the user; steam, fuel, and air. Using the resulting drop-down menus the user can switch to the desired inlet type. This is shown in Figure 10 where air and fuel are selected. Selection of a fuel also requires the specification of the fuel components (i.e. mass fraction) also shown. The user is required to specify the flow/fuel rate and temperature for each inlet. The physical location of the fuel and air inlets on the flare geometry must be specified. This can be done by first selecting the inlet and then selecting the areas on the geometry that are represented by that inlet. Figure 11 shows the selection of the air inlet (shown in purple) and the fuel in red. This approach allows for an arbitrary number of fuel, air, and steam inlets with different flowrates and temperature nonmatter how complex the flare geometry is. One the inlets are specified the user can select the RunSimulation button. Figure 7: The empty FlareUI case with the required elements. AFRC 2018 Industrial Combustion Symposium Figure 8: Selection of the flare geometry. Figure 9: Adding an inlet to the Flare. Figure 10: Selection of air and fuel as inlets. AFRC 2018 Industrial Combustion Symposium Figure 11: Selection of the air and fuel geometries. 3.1.2 Running a Simulation Upon completion of the specification of the flare geometry, environmental parameters, and inlets the users can start the simulation. When this is done using the but "Run Simulation" they are taken to the simulation and monitoring page. The user may enter any platform specific Run Parameters and then start and stop the simulation using the areas in the top left of Figure 12. Once the simulation starts the user is presented with a dynamic list of possible outputs. In this example the user may visualize 3D results with VisIt, the simulation text log, and line plots of Fuel Mole Fraction and Average Fuel Mole Fraction. These line plots are automatically updated and dynamic allowing the user to zoom and select portions of the chart. Figure 12: Simulation monitoring using the FlaresUI. AFRC 2018 Industrial Combustion Symposium The selected monitor point depending upon the geometry and environmental conditions. For the Air Assisted simulation the monitor point was picked as the cell (306, 577, 276) or in xyz coordinate (x=13.5 m, y= 6.0 m, and z = 0.0 m). The average value of unburned fuel mass fraction as a function of time was plotted in Figure 13. The average unburned fuel mass fraction stage starts to oscillate around 0.0015 after physical time of 10 seconds. This figure, together with the overall DRE/CE plot (Figure 14), is used to determine the convergence of the case and is used to stop the case. Figure 13: The average unburned fuel mass fraction as a function of time at monitor point Another plot available to the users will include the Destruction and Removal Efficiency (DRE). The DRE is the mass percentage of hydrocarbon species X that is destroyed relative to the quantity of hydrocarbon species X entering the flare. Numerically, this is represented as DRE % 100 1 Eq. 1 Where, DRE (%) = destruction and removal efficiency (%); Xplume = mass flow rate of hydrocarbon species X found in the flare plume after combustion has ceased; Xin = mass flow rate of hydrocarbon species X in the vent gas entering the flare. CE is the percentage of the total hydrocarbon stream entering the flare that burns completely to form only carbon dioxide and water. Numerically, if CO2 in the air is assumed to be negligible, this is represented as CE % ∑ 100 Eq. 2 Where, CE (%) = combustion efficiency (%); CO2 (plume) = volume concentration of carbon dioxide in the plume (ppmv) after combustion has ceased; CO (plume) = volume concentration of carbon monoxide in the plume (ppmv) after combustion has ceased; hydrocarbons (plume) = volume concentration of all the unburned hydrocarbons in the plume after combustion has ceased multiplied by the number of carbons in the hydrocarbon (ppmCv). The mass flow rate of hydrocarbon fuel is integrated over the outlet boundary of simulation domain and DRE (%) is calculated and is plotted as a function of time (Figure 14). Again, the overall DRE starts to oscillate around 60% after physical time of 10 seconds. Thus, determined by the average AFRC 2018 Industrial Combustion Symposium unburned fuel concentration at the monitor and overall DRE, the case can be considered as a converged case after physical time of 10 seconds. Selecting the 3D results pulls up the VisIt results shown in Section 3.3.1 with a sub-selection of predefined field variables. A user may opt to exert with full control of VisIt to investigate parameters not provided. Figure 14: The Overall DRE as a function of time 3.2 Hardening of Uintah and Arches Although Uintah and Arches component are a mature, some development was necessary to simplify input specification and prevent user error due to poor specification. The geometry of industrial flare tips range from simple, nearly open pipes to very geometrically complex structures, typically involving assist (steam or air) piping and injectors. In the LES calculations, the geometry itself is present as a boundary condition to the solution domain. These boundary conditions interact with the solution as sources of mass, momentum or energy. As such, the boundaries can influence several aspects of the simulation results. As far as flares go, the Arches code has historically simulated relatively simple geometries for flares. Through Phase I, boundary conditions enhancements were required to more faithfully represent the entire flare stack geometry and inlets considered in this project. Two major development thrusts were pursued; pairing the Arches simulation with CAD geometry for representing the solid surfaces of the flare and pairing the Arches simulation with precomputed solutions of finer grained boundary conditions information, specifically at inlets, for representing fine geometric detail that would otherwise go unresolved in the LES simulation. These two developments are discussed next. From a computational scaling perspective, Arches and the Uintah framework has been successful in scaling to very large numbers of computational cores (O(100K)) because of the use of a structured meshing strategy. The structured mesh strategy, along with a domain patch decomposition allows the framework to equally distribute work across the total number of cores to obtain a high-level of efficiency. In the case of flare simulations, however, this results in a coarsening effect of the geometry of the system. In a best case scenario, curved geometric surfaces become "stair-stepped" (so-named based on their resemblance of stairs). In a worst case, geometric detail is lost or not represented at all because the detail may fall below (i.e., subgrid) the physical AFRC 2018 Industrial Combustion Symposium dimension of a computational cell. To address the later issue, a surface detection algorithm was implemented into the Arches that uses imported CAD representations of the flare geometry to determine the location of all boundaries, including those that may be finer than the computational mesh. Once identified, the geometry is coarsened to the LES mesh resolution in the usual stairstepped approach. This ensures that CAD imported geometry to Arches is fully intact. The CAD surface detection algorithm in Arches was tested for simple geometric shapes generated from CAD files as well as two flares (steam and air assist) as used in the simulation of the TCEQ simulations. The second development thrust was to represent flow inlet boundary conditions for under resolved inlets. Arches has demonstrated this capability in the past for what is termed domain-edge boundary conditions. That is, those boundary conditions that are present on the furthest extent of the computational domain boundaries. In this case, it was desired to introduce boundary conditions on the interior of the domain, typically from geometric features of the flare (e.g., steam injector). However, since many of these geometric details might go under resolved, a fine-grained simulation near the geometric feature could be produced and then scaled up to the final LES resolution through a information hand-off procedure. This so-called handoff procedure at interior geometric features was demonstrated for the steam assist flare. 3.3 Results of Flare Simulations 3.3.1 Air Assisted Flare This section will visualize and discuss the flare simulation results. The mesh resolution of the simulation of the case finished in the publicly available HPC facility (Nimbix) is 460 x 730 x 552. The case was run in Nimbix HPC facility with 1920 cores and was stopped at ~ 130 hr, or total ~249,600 core-hours. RANS based turbulence model is routinely used to analyze internal combustion system since steady operation is more likely given the system operates in a controlled environment and RANS based turbulence model gives the excellent trade-off between representation of physics and computational tractability. LES based turbulence model is more suitable to analyze external combustion systems (such as flare), where the combustion process is impacted by ambient conditions such as cross wind, and accurately predicting the behavior of ignition and quenching is critical. Figure 15 shows the instantaneous temperature contour and a plane cut through the flare center in the wind direction at physical time of 14.6 seconds. The high temperature flame is observed at the locations near the fuel tip and the flare is bending due to the effect of wind. Figure 16 shows the instantaneous unburned fuel mass fraction contour and a plane cut through the flare center in the wind direction at physical time of 14.6 seconds. For this simulation, noticeable unburned fuel is observed leaving the domain. Aerodynamic interactions associated with the flare stack and flare tip geometries and the ambient air conditions (e.g. cross-wind), along with interactions with the plume of hot combustion gasses cause eddies to strip unburnt fuel away from the combustion zones. Subsequent dilution with air and cooling of the stripped fuel permanently results in combustion inefficiency. This physical behavior can't be predicted by RANS based turbulence model, which is critical in accurately predicting the combustion efficiency of the flare operation. AFRC 2018 Industrial Combustion Symposium Figure 15: Instantaneous temperature (K) field (310 K ~ 1800 K) at 14.6 sec Figure 16: Instantaneous unburned fuel field (mass fraction 0 ~ 0.004) at 14.6 sec Figure 17 shows the instantaneous CO2 mass fraction contour and a plane cut through the flare center in the wind direction at physical time of 14.6 seconds. CO2 is formed due to the combustion of fuel. Figure 17 shows that the majority of combustion happens in the region near the flare fuel tip, which is consistent with the high temperature near the flare fuel tip. Figure 18 shows the instantaneous vorticity contour and a plane cut through the flare center in the wind direction at physical time of 14.6 seconds. It's shown that the structure of flare has significant effect on the downstream field in the wind direction, less effect on the upper stream in the wind direction. That information should be used to guide us in the future to form optimal domain of a flare simulation. AFRC 2018 Industrial Combustion Symposium Figure 17: Instantaneous CO2 field (mass fraction 0.0006 ~ 0.02) at 14.6 sec Figure 18: Instantaneous vorticity field (20 s-1 ~ 100 s-1) at 14.6 sec Figure 19 shows the instantaneous velocity magnitude in a plane cut through the flare center in the wind direction at physical time of 14.6 seconds. It shows the disrupted velocity downstream in the wind direction due to the combustion of fuel and due to the physical structure of flare. For this run, overall DRE or CE is predicted to be 60% (see Figure 14), calculated from the integration of unburned fuel exiting the outlet boundary. The experimental extraction probe measurements from TCEQ 2010 study were converted to 95.9% combustion efficiency or 97.1% Destruction and Removal Efficiency. However, it is important to note that the combustion efficiency of 95.9% or Destruction and Removal Efficiency of 97.1% was computed based on the plume composition from the extraction probe, and therefore, is highly dependent on the location AFRC 2018 Industrial Combustion Symposium Figure 19: Instantaneous velocity magnitude (0 ~ 30 m/s) at 14.6 sec of the probe in the plume. Nevertheless, the predicted overall DRE seems to be low in comparison with experimental extraction probe measurements, it may due to 1) The dimensions of the fuel tips are estimated. Since the majority of combustion happens in the region near the fuel tip; those details could potentially influence simulation results. 2) The researchers at University of Utah has confirmed that modification of empirical values for the RCCE model may be necessary in order to simulate the industrial flare. We used the original empirical values for the phase I project and intend to modify them in the phase II project based on more simulations of TCEQ 2010 study cases 3.3.2 Steam Assisted Flare Two simulations were performed for the modeling of the steam assisted flare; with and without steam-assist where the mass flow rates of fuel were nearly the same. The following are the list of features of both simulations:       Spatial resolution - 2.8cm resolution on a side (dx,dy,dz) Total cell resolution - 7.6M computational cells Temporal resolution - 3.6e-4 sec/timestep Total resolved time - ~37sec for both cases Total processors - 416 cores Total CPU hours - 28.3K The simulations were run on the University of Utah's Ash cluster (2.8 GHz Intel Xeon (Westmere X5660) processors) as well as the post-processing of the simulation data. AFRC 2018 Industrial Combustion Symposium Following the TCEQ report, combustion efficiency is defined as the ratio of the carbon contained in CO2 over the sum of the carbon contained in CO2, CO, and any unburnt fuel. These species were time-averaged using a 17.4sec window in the simulation and then used to computed a local combustion efficiency (i.e., at each grid cell). Slices of the average combustion efficiency are shown in Figure 20 for the steam-off and steam-on cases. The effect of the crosswind can be seen, including a fairly low-efficiency region below the main body of the flame and attached to the rearward portion of the stack. This low efficiency region is caused by vortical structures shedding off the flare-stack, which pull unburnt fuel down to a cool region away from the main combustion zone. This effect appears to be exacerbated in the steam-on case. Figure 21 shows the same combustion efficiencies at the vertical exit plane of the simulation. Again, the distribution of the lower-efficiency regions for the steam case is clearly seen. Note that these figures are turned on their side, with gravity acting from right to left. Plots of time-averaged temperature are shown in Figure 22. The temperature ranges are fairly similar for both cases. The horseshoe shape of the higher temperature zone is due to the interactions of the crosswind with the flare stack and the effects of buoyancy. While these temperatures are fairly low, it may be possible that the Arches combustion model may still consume fuel in these zones. Future simulations should include a study of the effects of the domain box on the combustion efficiency. Figure 20: Time averaged combustion efficiencies for the steam-off case (left) and steam-on case (right). The efficiencies are time averaged over the same time window of 17.4sec. AFRC 2018 Industrial Combustion Symposium Figure 21: Time-averaged combustion efficiencies for the steam-off case (top) and steam-on case (bottom). Note that these images have been turned on their side so that gravity is acting from right-to-left. Figure 22: Time average temperature at a slice downstream of the flare (exit of computational domain) for the steam-off case (top) and steam-on case (bottom). Note that these images have been turned on their side so that gravity is acting from right-to-left. AFRC 2018 Industrial Combustion Symposium Combustion efficiencies in the TCEQ tests were computed using sampled species measurements downstream from the flare stack. Species were sampled using a large collection probe suspended from a crane. The probe was positioned in an attempt to capture a representative sample of the combustion effluent based on visual observation. The probe position is not reported for each test in the TCEQ report. In addition, the sample probe employed a fan to draw sample volume into the device. These two facts constitute the biggest challenges in attempting to represent the combustion efficiency measurement from the perspective of the simulation. Several virtual instrument models could be explored in an attempt to compare simulation and measured data. In this work, the collection probe was represented using the profile of time-averaged combustion efficiency across a circular cross sectional area at the vertical exit of the computational domain. The elevation of the center of the circular cross-section was changed to represent different potential heights of the probe. A visualization of the virtual collector is shown in Figure 23. Figure 23: Volume rendering of temperature and a pseudocolor slice of combustion efficiency (masked with a circle) at the domain exit. The circular cross-section is meant to represent the TCEQ probe sample. Instantaneous combustion efficiencies for one position of the virtual collector are shown as a function of time in Figure 24. The means and width of one standard deviation are also shown on the plot. This plot shows that the data appear to be at steady-state, with the bulk of the variation occurring within the first standard deviation. Final combustion efficiencies for the steam-on and steam-off cases are shown in Figure 25 as a function of probe elevation. The experimentally measured combustion efficiencies are shown with the solid lines for reference (not intended to show an elevation functionality). As shown in Figure AFRC 2018 Industrial Combustion Symposium 20 and Figure 22, the effect of the steam on the combustion efficiency is captured in the simulation, with the steam assist showing a consistent degradation of combustion efficiency. In the case of the steam-on, a potential consistent region is observed with the cross-over of the solid and dotted red lines. However, no cross-over appears for the steam-off scenario. A more thorough analysis should include sensitivity testing of the instrument modeling itself given the uncertainties in experimental setup. For example, one may use a mass flow rate averaged combustion efficiency at the exit plane. In this case, considering the entire vertical exit of the computational domain, a combustion efficiency of 99.2% is computed for the steam-off scenario, while a combustion efficiency of 98.9% is computed for the steam-on scenario. Figure 24: Instantaneous values (black line) of efficiency over the masked exit zone. Also shown is the mean value (red dotted line). The shaded region shows the range of one standard deviation. Figure 25: Plot of combustion efficiency vs. elevation of virtual sample position. The experimentally measured values are shown for reference with solid lines and are not a function of elevation for this plot. AFRC 2018 Industrial Combustion Symposium 4 Conclusions In the Phase I program, four key questions were posed as a basis for demonstrating the technical feasibility of our approach. In the Phase I program, each of these questions were addressed as follows: Question 1: Are HPC facilities, which are able to achieve the high level of scalability of the UCF in simulating industrial flares, commercially available? In the phase I program, REI carried out simulations of two flare test geometries from the Texas Commission on Environmental Quality (TCEQ) flare study [15]: a) air-assisted flare (John Zink Model LHTS-24/60 test case A2.1) b) steam-assisted flare (John Zink EEF-QS-36C, test case S3.5) The air-assisted simulation was carried out at the NIMBIX commercial HPC facility. The simulation domain (66 ft x 48 ft x 36 ft) was relatively large with a domain height double the flare height, a length of a factor of 4 of the flare base diameter and a width of a factor of 3 of the flare base diameter. The resulting cell count for the structured, cartesian, uniform grid was 185 million cells (730 x 552 x 460), a very large simulation. We were able to converge the simulation using 1920 cores and 130 hours (5.4 days) or a total of 249,600 core-hours, obtaining 14.6 seconds of physical time for the LES simulation. Statistically stationary state was achieved following 10 seconds of physical time allowing for analysis of steady state properties of the flare. In summary, based on completion of this relatively large flare simulation in 5 days, running on 1920 cores, we were able to demonstrate a high level of scalability of the Arches code on the commercially available NIMBIX HPC facility. Question 2: Can a UCF based full-scale flare simulation be completed on commercially available HPC resources in a time and budget that makes these simulations feasible for commercial design, operational guidance, and trouble-shooting? The air-assist flare which was simulated using the NIMBIX HPC platform required 5.4 days on 1920 cores or a total of 249,600 core-hours to achieve 14.6 seconds of physical time. The steamassist flare which was simulated using the University of Utah Center for High Performance Computing required 2.8 days on 416 cores or a total of 28,300 core-hours to achieve 37 seconds of resolved physical time. These two cases incorporated large differences in the simulation domain, leading to large differences in cell count (185 million vs. 7.6 million). It is notable that the ratio of core-hours per physical time scales well with the ratio of cell size for these two cases indicating that the Arches flare simulations scaled very well on up to the 1920 cores used on the Nimbix facility. At a commercial computing cost of $0.09 per core-hour on the Nimbix facility, the two case costs for the air-assist and steam-assist flares were $22,464 and $2547, respectively. Although the turnaround times of 5.4 days and 2.8 days for these two simulations, would certainly be considered acceptable for commercial analysis, a computing cost of $22,464 would be considered prohibitive in many instances, while a cost of $2547 is significantly more commercially acceptable. In summary, these two simulations demonstrated that these high fidelity LES flare simulations can be completed on a commercial time scale. The cost is dependent on the chosen domain and mesh resolution which is a trade-off with accuracy. AFRC 2018 Industrial Combustion Symposium Question 3: Do the results of the flare simulation show improved accuracy of the simulation in comparison with standard legacy simulations involving use of RANS models? For the Arches LES based flare simulations carried out in the phase I program, corresponding RANS simulations were not carried out. However, RANS simulations have been showing difficulties in accurately predicting the external combustion systems (such as flare), where the combustion process is impacted by ambient conditions such as cross wind, and where accurately predicting the behavior of ignition and quenching is critical. The Arches LES-based flare simulations carried out in the phase I program show that the models capture the complex aerodynamic interactions associated with the flare stack and flare tip geometries and the ambient air conditions (e.g. cross-wind). These interactions affect the behavior in the plume of hot combustion gasses where eddies can strip unburnt fuel away from the combustion zones. Subsequent dilution with air and cooling of the stripped fuel permanently results in combustion inefficiency. The ability of the Arches software to predict this behavior is a significant improvement in accuracy of the LES based flare simulations over standard legacy simulations involving use of RANS models. Question 4: Can a web-based interface to the UCF be constructed to ease the input specification, case monitoring, and output analysis to provide meaningful flare simulation results to a non-expert user of the software? The phase 1 program demonstrated the use of a relatively basic user interface constructed to: 1) specify geometry, inlet conditions, and boundary conditions for an Arches simulation, 2) initiate an Arches flare simulation on the Nimbix HPC facility, 3) monitor case progress, and 4) post process the case results. The interface was demonstrated for the air-assist flare. The phase I program identified specific desirable enhancements to the basic interface which are planned for the phase II program, including capability for accessing multiple commercial HPC facilities based upon current load and pricing. The current work plan for the phase II program will be critical to advance the capability of the web based interface as well as to harden Arches for simulation of a wider range of flare types including multipoint ground flares (MPGF). AFRC 2018 Industrial Combustion Symposium 5 References   [1] B. C., in The John Zink Combustion Hand Book, CRC Press, 2001, pp. 589- 634. [2] Council on Competitiveness, "US Manufacturing-Global Leadership through Modeling and Simulation," 2009. [3] President's Information Technology Advisory Committee, Computational Science: Ensuring America's Competitiveness. Report to the President, June 2005, 2005. [4] P. Smith, J. Thornock, Y. Wu, S. Smith, B. Isaac, P. Chapman, D. Sloan, D. Turek, Y.-M. Chen and A. Levasseur, "Oxy-Coal Power Boiler Simulation and Validation Through Extreme Computing," in AFRC Industrial Combustion Symposium, Houston, 2014. [5] Q. Meng and M. Berzins, "Scalable Large-scale Fluid-structure Interaction Solvers in the Uintah Framework via Hybrid Task-based Parallelism Algorithms," SCI Technical Report No. UUSCI-2012-004 (SCI Institute, University of Utah)/ Concurrency and Computation (Submitted), 2012. [6] J. Schmidt, M. Berzins, J. Thornock, T. Saad and J. Sutherland, "Large Scale Parallel Solution of Incompressible Flow Problems using Uintah and hypre," in Cluster, Cloud and Grid Computing (CCGrid) - IEEE/ACM International Symposium, 2013. [7] "What's Behind the HPC Justification Problem," HPCwire, 9 October 2014. [Online]. Available: http://www.hpcwire.com/2014/10/09/behind-hpc-justification-problem. [8] M. Berzins, "Status of Release of the Uintah Computational Framework. No. UUSCI-2012001," SCI Institute, University of Utah, 2012. [9] M. Berzins, J. Schmidt, Q. Meng and A. Humphrey, "Past, Present, and Future Scalability of the Uintah Software," Proceedings of the Blue Waters Workshop, 2012. [10] A. Humphrey, Q. Meng, M. Berzins and T. Harman, "Radiation Modeling Using the Uintah Heterogeneous CPU/GPU Runtime System," in (Submitted) XSEDE Conference, 2012. [11] "Extensible Markup Language (XML)," [Online]. Available: www.w3.org/XML/. [Accessed Jan 2018]. [12] "OpenJFX Main," [Online]. Available: wiki.openjdk.java.net/display/OpenJFX/Main. [Accessed 01 2018]. [13] "JavaFX in the Browser," [Online]. Available: jpro.io. [Accessed Jan 2018]. [14] "VisIt," [Online]. Available: wci.llnl.gov/simulation/computer-codes/visit/. [Accessed Jan 2018]. AFRC 2018 Industrial Combustion Symposium [15] D. T. Allen and V. M. Torres, "TCEQ 2010 Flare Study Final Report," Texas Commission on Environmental Quality. PGA No. 582-8-862-45-FY09-04, August 1, 2011. [16] J. C. Keek and D. Gillespie, "Rate-controlled partial-equilibrium method for treating reacting gas mixtures.," Combustion and Flame, vol. 17, pp. 237-241, 1971. [17] M. Zabetakis, "Flammability characteristics of combustible gases and vapors. Bulletin 627," Bureau of Mines, 1965. [18] "https://wordpress.premieroilfieldlabs.com/stimplan-software/," [Online]. [19] "https://www.software.slb.com/products/petrel," [Online]. [20] "XML Schema," [Online]. Available: www.w3.org/XML/Schema. [Accessed Jan 2018]. [21] C. K. Westbrook and F. L. Dryer, "Chemical kinetic modeling of hydrocarbon combustion.," Combustion Science and Technology, vol. 10, pp. 1-57, 1984. [22] "PushToCompute Pipeline Overview," [Online]. Available: http://jarvice.readthedocs.io/en/latest/cicd/. [Accessed Jan 2018]. [23] J. S. J. a. Y. L. Cain, "Overview of flaring efficiency studies," [Online]. Available: http://content.lib.utah.edu:81/cgibin/showfile.exe?CISOROOT=/iriradmin2&CISOPTR=21350&filename=21067.pdf. [24] Federal Register (FR) 2701, Jan. 21, 1986, as amended at 63 FR 24444, May 4, 1998; 65 FR 61752, Oct. 17, 2000; 73 FR 78209, Dec. 22, 2008. [25] S. Evans, "A critique of the flare provisions of EPA's proposed refinery sector rule," in AFRC Industrial Combustion Symposium, Houston, TX, 2014.