Large Eddy Simulation/Computing Needs

Paper from the AFRC 2018 conference titled Large Eddy Simulation/Computing Needs

As computing power becomes more affordable and available, Large Eddy Simulation (LES) becomes attractive as a simulation tool for industrial combustion equipment. LES, by explicitly modeling the time-dependent evolution of the largest turbulent scales, avoids the approximations inherent in the Reynolds-Averaged Navier-Stokes (RANS) models that have been commonly used for decades. ; Academic LES practitioners have listed criteria for LES reliability and validity such as mesh resolution necessary to capture a sufficient portion of the turbulent energy spectrum. Also important are the use of high-order time and space discretization methods, and sufficiently small time steps to ensure accurate temporal evolution of the reacting flowfield. Strict adherence to the many criteria put forth leads to heavy computational demands. ; The engineer interested in industrially useful answers is left to wonder how strictly the academic criteria for valid LES must be followed to get answers sufficient to guide engineering design. In this paper we will attempt to address this issue, drawing from academic literature, published industrial work and our own experience.

Large Eddy Simulation / Computing Needs Presented at the 2018 Conference of the American Flame Research Committee Mike Henneke, John Zink Hamworthy Combustion Chris Montgomery, John Zink Hamworthy Combustion Benedicte Cuenot, CERFACS Eleonore Riber, CERFACS Sreejith Nadakkal, CERFACS Introduction Computational Fluid Dynamics (CFD) simulation of combustion processes is an important engineering tool for design, application and troubleshooting of many types of combustion equipment. This includes in-cylinder flows for IC engines, gas turbine combustors, and industrial combustion systems which are the focus of this paper. Large eddy simulation (LES) of combustion systems is a methodology that was initially proposed in 1963 by Joseph Smagorinsky to simulate atmospheric flows. Conceptually, the goal of Large Eddy Simulation is to apply a low-pass filter to the governing equations of fluid flow to make the solution of these equations computationally tractable. The use of this low-pass filtering approach requires submodels for the unresolved scales which are called sub-grid scale models. Groups of high-performance compute servers are the tool most engineers use for computing. These groups are called compute clusters and can range from a few individual servers to clusters with millions of compute cores. The website top500.org keeps an updated list of the clusters with the highest calculation speed, for example the June 2018 list (https://www.top500.org/list/2018/06/) indicates that a Department of Energy cluster at Oak Ridge National Laboratory with 2.3 million compute cores is at the top of the list. Background Combustion processes of engineering interest are usually turbulent. Turbulence is a characteristic of a flow that creates apparently random fluid motions. CFD simulation of turbulent flows falls into a couple of categories: 1. Reynolds (or Favre) averaged Navier Stokes equations (RANS), steady or transient 2. Large eddy simulation (LES), inherently transient 3. Direct numerical simulation (DNS), inherently transient Steady RANS models are extensively used as they have the lowest computational cost. RANS simulations require a turbulence model to model the effects of turbulence. Common turbulence models include the k-epsilon model, the k-omega model, and the Reynolds Stress model. There are countless alternative models and variations on these models. These models often require special considerations at wall boundaries and the CFD engineer must be aware of the y+ mesh requirements for the various wall treatments. Large eddy simulations are based on the concept of resolving (in both time and space) large turbulent length and time scales while modeling the small length and time scales. This approach is attractive in part because the small length and time scales are often thought to be universal in nature (homogeneous and isotropic) while the larger length scales are problem dependent and will vary in time, space and direction. Since the smallest length and time scales are universal the submodels developed will be widely applicable. Large eddy simulation brings with it the potential to directly simulate sound produced by flow and combustion processes. Flow-induced noise and combustion-instability induced noise can both be captured by LES with suitable combustion submodeling. Direct numerical simulation (at the current time) is extremely limited in application due to the range of length scales in a turbulent flow even at moderate Reynolds' numbers. To the authors' knowledge, DNS has only been used as a tool to study turbulent flow and improve models - it has never been used for an engineering application. Not discussed in this paper are hybrid models such as V-LES (very large eddy simulation) and DES (detached eddy simulation). These are methods that aim to blend LES and RANS in a way that retains the advantages of LES at a reduced computational cost. Quality metrics for Large Eddy Simulation McDermott (2011) reviewed LES quality metrics in the context of a particular LES software tool called FDS. However the quality metrics reviewed are applicable to any LES simulation. A criterion that has seen widespread use in the research community is the so-called Pope criterion (Pope, 2004). This criterion requires that the fraction of modeled turbulent kinetic energy be less than 20% of the total turbulent kinetic energy. This can be written as: 𝑘"#" < 0.2 𝑘"#" + 𝑘&'" Where 𝑘"#" is the modeled (sub-grid scale) turbulent kinetic energy and 𝑘&'" is the resolved kinetic energy. Other criteria that are commonly used are: 1. y+<1 at the walls. This requirement means that near-wall cells in the LES mesh are less than 1 wall unit away from the wall. 2. Convective Courant number <1. This means that for every cell in the simulation (with a cell length of ∆𝑥), the numerical timestep is such that the flow travels a distance less than ∆𝑥. 3. It is common practice to run a RANS model prior to the LES simulation so that the turbulent length scales can be estimated from k and e. The turbulent length scale computed from a RANS solution is l0=Cμk1.5/e then the LES mesh size should be on the order of l0. Example jet flow Circular jets are a well-studied fluid mechanics problem and also heavily utilized in the design of industrial combustion systems. In many applications, the jets may be choked so that the flow is compressible with the jet discharge being at or above sonic velocity. Choked jets are not well studied in the scientific literature so this discussion is limited to incompressible jet flows. For example, a typical process burner might have 10-50 individual choked jets that are used by the burner designer to shape the flame, mix the fuel and air, and entrain surrounding flue gas to control NOx emissions. In Jewkes et al (2012), LES simulations of a single jet were performed on a mesh of 6 million cells. The authors used the Pope criterion discussed in a previous section. This type of analysis is a useful way to verify the adequacy of the computational mesh. Pope (2004) has suggested that the metric plotted in Figure 1 should be less than 20% for well-resolved LES. Figure 1: Percentage of modeled turbulent kinetic energy for simulation reported in Jewkes et al (2012) An example result from Jewkes et al (2012) is shown below. This example shows the type of turbulence that can be found in a circular jet - the bulk of which should be resolved on the LES mesh. Figure 2: Centerline slice showing instantaneous vortical structures, vorticity magnitude scaled from 0 (white) to 10 (from Jewkes et al 2012) Length and Time Scales As discussed above, turbulent length scales based on dimensional arguments can be used to estimate the length scales that should be captured in a LES simulation. For example, in sonic jets with a diameter of 5mm, the turbulent length scales would be on the order of 10µm (estimated with k1.5/ε from internal work). The time scales in a turbulent flow must also be captured. A typical numerical requirement for capturing time scales is to require that the Courant number be less than 1, .∆/ ∆0 < 1. This is equivalent to saying that during a timestep, the fluid should travel less than the length of one computational cell. Given the spatial requirements discussed above and considering a choked velocity, it is not unusual to need a numerical timestep that is < 0.1µs. In industrial combustion problems of engineering interest, it is not unusual to have length scales of 10 meters or more. More challenging to the simulation is the timescales related to industrial combustion. For example the residence time in a furnace could be 10 seconds or more. In principle we would like to have a LES model consider several ‘flow-through' times to ensure that the results are statistically stationary. This can lead to needing several hundred million timesteps to produce a result. Application to a typical furnace geometry Applying LES to large-size furnaces raises a new issue, linked to the large scale ratio between the injection and turbulent flame scales (of the order of the mm) and the furnace height (of ten meters). This large length scale range could be managed with sophisticated mesh techniques, but it also involves a similar time scale range, which is also problematic. An alternative is to use a multi-scale LES method that decomposes the computational domain into different zones calculated separately, and then coupled via a recovery zone. The independent calculation of each sub-domain allows to desynchronize them, i.e., compute each with its own time step, leading to a substantial gain of computing time. This method has been extensively tested and validated for the simulation of multi-stages turbines (de Laborderie, 2016) or to simulate torch flames about ten meters long. In the present furnace case, illustrated in Figure 3, the domain is decomposed in two zones. The first, small sub-domain (red box) contains the burner and is meshed with small cells able to describe the injection and flame length scales. The second, larger subdomain (blue box) contains the plume of hot gas and has a much coarser mesh. The time step is accordingly much larger in the blue sub-domain, which requires therefore less iterations for the same physical time than the red sub-domain. Figure 3: View of the full furnace, the two sub-domains and zoom on the burners. Example simulations have run with the code AVBP of CERFACS, which resolves the filtered Navier-Stokes equations and the conservation equations for energy and species to describe turbulent reacting flows. AVBP uses 3rd-order discretization schemes on unstructured meshes (Colin, 2000) It integrates a number of physical models for the turbulence, the turbulent combustion, wall flows, etc. The present simulation has been run with the SIGMA subgrid scale model (Toda, 2012), the thickened flame approach (Colin, 2000) and simple 2-step chemistry. Figure 2 shows the flames stabilizing on each burner. As all flames are attached to the injector, the combustion regime is of the non-premixed type. At the integration time shown the flame is quite short but may get longer as time advances. Figure 4: LES of one burner : Iso-surface of heat release rate colored by temperature The computational cost of the simulations shown here ranges from 30,000 cpu-hours for the multi-scale approach up to 750,000 cpu-hours for a single-scale model. Also note that the physical time simulated at this point is only 0.173 seconds. The furnace residence time is on the order of 10 seconds so a significantly longer simulation is required to approach a statistically stationary solution. Table 1: Mesh size, timestep, and compute details for LES furnace example For comparison purposes, Reynolds Averaged Navier-Stokes (RANS) simulations done by John Zink CFD engineers for this application would require about 5-10 million cells and 5-10,000 cpu-hours. Conclusions: This paper has provided a framework for estimating the computing requirements and cost for Large Eddy Simulation of industrial combustion systems based on previously published quality metrics. The focus has been on computing requirements but it should be noted that the engineering time required for LES is expected to be similar to the engineering time required for RANS modeling. The example shown here indicates that a well-done LES simulation of a furnace could require upwards of 100 times the computing resources of a RANS simulation. This is however affordable with the use of High Performance Computing (HPC) and highly scalable codes. The trend toward cloud-based HPC makes this compute power available without any upfront investment.. With hybrid methods such as VLES and DES this computational cost can be reduced. The use of LES is radically different from the use of RANS: LES gives a predictive, accurate solution for a detailed understanding of the complex physics while RANS gives only a trend based on prior calibration. Therefore LES is a unique and essential tool for innovation. Further work is required to better understand and define the best use of LES for industrial systems and to determine the quality metrics necessary to use LES as an engineering tool for various applications. References: Smagorinsky, Joseph (March 1963). "General Circulation Experiments with the Primitive Equations". Monthly Weather Review. 91 (3): 99-164. Pope, SB, "Ten questions concerning the large-eddy simulation of turbulent flows". New Journal of Physics 6 (2004) 35 McDermott, RJ, "Quality Assessment In The Fire Dynamics Simulator: A Bridge To Reliable Simulations". PROCEEDINGS, Fire and Evacuation Modeling Technical Conference 2011 J. Meyers, B. Geurts, and P. Sagaut, editors. Quality and Reliability of Large-Eddy Simulations. Springer, 2008. 2 Jewkes, JW, AJ King, YM Chung, Large Eddy Simulation of a Steady Circular Jet Issuing Into Quiescent Fluid, 18th Australasian Fluid Mechanics Conference, Launceston, Australia,3-7 December 2012 de Laborderie, J., Duchaine, F., Vermorel, O., Gicquel, L. and Moreau, S., 2016, June. Application of an overset grid method to the large eddy simulation of a high-speed multistage axial compressor. In ASME Turbo Expo 2016: Turbomachinery Technical Conference and Exposition (pp. V02CT39A015V02CT39A015). American Society of Mechanical Engineers Colin O, Rudgyard M. Development of high-order Taylor-Galerkin schemes for LES. Journal of Computational Physics. 2000 Aug 10;162(2):338-71. Toda, H.B., Cabrit, O., Balarac, G., Bose, S., Lee, J., Choi, H. and Nicoud, F., 2010. A subgrid-scale model based on singular values for LES in complex geometries. In Proc. of the Summer Program (pp. 193-202) Colin O, Ducros F, Veynante D, Poinsot T. A thickened flame model for large eddy simulations of turbulent premixed combustion. Physics of fluids. 2000 Jul;12(7):1843-63.