| Description |
As spatiotemporal datasets grow, accessing and processing them for analysis and visual- ization are increasingly the primary bottlenecks for their use. Challenges include retrieving, resampling, and analyzing large and often disparately located data. Utilization of large-scale computing resources can be helpful, but may still incur delays due to extensive data transfers, job scheduling, and remote access. Furthermore, some applications, such as those for public safety, must remain interactive even as data sizes increase. To enable utilization of increasingly massive datasets, it is worthwhile to invest in the creation of work ows that guarantee interactivity, making the broadest set of inquiries possible at minimal cost. In this work, I present a framework that addresses several common pitfalls of interactive data analysis and visualization. It is comprised of an embedded domain-speci c language (EDSL) and associated runtime speci cally designed for the interactive exploration of large, remote data ensembles. The EDSL is an extension of JavaScript, which allows users to express a wide range of analyses in a simple and abstract manner. The underlying runtime utilizes a streaming, multiresolution data layout, transparently resolving issues such as remote data access and resampling, and maintaining interactivity through progressive, interruptible computation. This framework enables interactive exploration of massive, remote datasets, such as the 3.5 petabyte 7km NASA GEOS-5 \Nature Run" simulation, for which remote users have previously been able to analyze only oine or at reduced resolution. Most available climate data are stored using legacy le formats that prohibit incremental, multiresolution access. In order for the framework to automatically read these datasets, I developed an on-demand conversion module, currently deployed at Lawrence Livermore National Lab as part of the Earth System Grid Federation (ESGF) platform. Based on the techniques used for this framework, I also propose a general purpose model to aid creation and evaluation of other interactive work ows for large, remote data. I present the necessary components of such work ows along with important considerations regard- ing their design and integration, including comprehensive runtime management, e ective communication, interruptibility, appropriate data formats, and programming models that facilitate progressive re nement of results. |
