26 Matching Results

Search Results

Advanced search parameters have been applied.

Formulas for robust, one-pass parallel computation of covariances and arbitrary-order statistical moments.

Description: We present a formula for the pairwise update of arbitrary-order centered statistical moments. This formula is of particular interest to compute such moments in parallel for large-scale, distributed data sets. As a corollary, we indicate a specialization of this formula for incremental updates, of particular interest to streaming implementations. Finally, we provide pairwise and incremental update formulas for the covariance. Centered statistical moments are one of the most widely used tools in descriptive statistics. It is therefore essential for statistical analysis packages that robust and efficient algorithms be devised and implemented. However, robustness and speed of execution, in this context as well as in others, tend to be orthogonal. For instance, it is well known1 that algorithms for calculating centered statistical moments that utilize sum of powers for the sake of execution speed (one-pass algorithms) lead to unacceptable numerical instability.
Date: September 1, 2008
Creator: Pebay, Philippe Pierre
Partner: UNT Libraries Government Documents Department

Parallel contingency statistics with Titan.

Description: This report summarizes existing statistical engines in VTK/Titan and presents the recently parallelized contingency statistics engine. It is a sequel to [PT08] and [BPRT09] which studied the parallel descriptive, correlative, multi-correlative, and principal component analysis engines. The ease of use of this new parallel engines is illustrated by the means of C++ code snippets. Furthermore, this report justifies the design of these engines with parallel scalability in mind; however, the very nature of contingency tables prevent this new engine from exhibiting optimal parallel speed-up as the aforementioned engines do. This report therefore discusses the design trade-offs we made and study performance with up to 200 processors.
Date: September 1, 2009
Creator: Thompson, David C. & Pebay, Philippe Pierre
Partner: UNT Libraries Government Documents Department

Current parallel I/O limitations to scalable data analysis.

Description: This report describes the limitations to parallel scalability which we have encountered when applying our otherwise optimally scalable parallel statistical analysis tool kit to large data sets distributed across the parallel file system of the current premier DOE computational facility. This report describes our study to evaluate the effect of parallel I/O on the overall scalability of a parallel data analysis pipeline using our scalable parallel statistics tool kit [PTBM11]. In this goal, we tested it using the Jaguar-pf DOE/ORNL peta-scale platform on a large combustion simulation data under a variety of process counts and domain decompositions scenarios. In this report we have recalled the foundations of the parallel statistical analysis tool kit which we have designed and implemented, with the specific double intent of reproducing typical data analysis workflows, and achieving optimal design for scalable parallel implementations. We have briefly reviewed those earlier results and publications which allow us to conclude that we have achieved both goals. However, in this report we have further established that, when used in conjuction with a state-of-the-art parallel I/O system, as can be found on the premier DOE peta-scale platform, the scaling properties of the overall analysis pipeline comprising parallel data access routines degrade rapidly. This finding is problematic and must be addressed if peta-scale data analysis is to be made scalable, or even possible. In order to attempt to address these parallel I/O limitations, we will investigate the use the Adaptable IO System (ADIOS) [LZL+10] to improve I/O performance, while maintaining flexibility for a variety of IO options, such MPI IO, POSIX IO. This system is developed at ORNL and other collaborating institutions, and is being tested extensively on Jaguar-pf. Simulation code being developed on these systems will also use ADIOS to output the data thereby making it easier for other systems, ...
Date: July 1, 2011
Creator: Mascarenhas, Ajith Arthur & Pebay, Philippe Pierre
Partner: UNT Libraries Government Documents Department

Parallel tetrahedral mesh refinement with MOAB.

Description: In this report, we present the novel functionality of parallel tetrahedral mesh refinement which we have implemented in MOAB. This report details work done to implement parallel, edge-based, tetrahedral refinement into MOAB. The theoretical basis for this work is contained in [PT04, PT05, TP06] while information on design, performance, and operation specific to MOAB are contained herein. As MOAB is intended mainly for use in pre-processing and simulation (as opposed to the post-processing bent of previous papers), the primary use case is different: rather than refining elements with non-linear basis functions, the goal is to increase the number of degrees of freedom in some region in order to more accurately represent the solution to some system of equations that cannot be solved analytically. Also, MOAB has a unique mesh representation which impacts the algorithm. This introduction contains a brief review of streaming edge-based tetrahedral refinement. The remainder of the report is broken into three sections: design and implementation, performance, and conclusions. Appendix A contains instructions for end users (simulation authors) on how to employ the refiner.
Date: December 1, 2008
Creator: Thompson, David C. & Pebay, Philippe Pierre
Partner: UNT Libraries Government Documents Department

Performance of a streaming mesh refinement algorithm.

Description: In SAND report 2004-1617, we outline a method for edge-based tetrahedral subdivision that does not rely on saving state or communication to produce compatible tetrahedralizations. This report analyzes the performance of the technique by characterizing (a) mesh quality, (b) execution time, and (c) traits of the algorithm that could affect quality or execution time differently for different meshes. It also details the method used to debug the several hundred subdivision templates that the algorithm relies upon. Mesh quality is on par with other similar refinement schemes and throughput on modern hardware can exceed 600,000 output tetrahedra per second. But if you want to understand the traits of the algorithm, you have to read the report!
Date: August 1, 2004
Creator: Thompson, David C. & Pebay, Philippe Pierre
Partner: UNT Libraries Government Documents Department

Scalable descriptive and correlative statistics with Titan.

Description: This report summarizes the existing statistical engines in VTK/Titan and presents the parallel versions thereof which have already been implemented. The ease of use of these parallel engines is illustrated by the means of C++ code snippets. Furthermore, this report justifies the design of these engines with parallel scalability in mind; then, this theoretical property is verified with test runs that demonstrate optimal parallel speed-up with up to 200 processors.
Date: December 1, 2008
Creator: Thompson, David C. & Pebay, Philippe Pierre
Partner: UNT Libraries Government Documents Department

Computational algebraic geometry for statistical modeling FY09Q2 progress.

Description: This is a progress report on polynomial system solving for statistical modeling. This is a progress report on polynomial system solving for statistical modeling. This quarter we have developed our first model of shock response data and an algorithm for identifying the chamber cone containing a polynomial system in n variables with n+k terms within polynomial time - a significant improvement over previous algorithms, all having exponential worst-case complexity. We have implemented and verified the chamber cone algorithm for n+3 and are working to extend the implementation to handle arbitrary k. Later sections of this report explain chamber cones in more detail; the next section provides an overview of the project and how the current progress fits into it.
Date: March 1, 2009
Creator: Thompson, David C.; Rojas, Joseph Maurice & Pebay, Philippe Pierre
Partner: UNT Libraries Government Documents Department

Scalable k-means statistics with Titan.

Description: This report summarizes existing statistical engines in VTK/Titan and presents both the serial and parallel k-means statistics engines. It is a sequel to [PT08], [BPRT09], and [PT09] which studied the parallel descriptive, correlative, multi-correlative, principal component analysis, and contingency engines. The ease of use of the new parallel k-means engine is illustrated by the means of C++ code snippets and algorithm verification is provided. This report justifies the design of the statistics engines with parallel scalability in mind, and provides scalability and speed-up analysis results for the k-means engine.
Date: November 1, 2009
Creator: Thompson, David C.; Bennett, Janine C. & Pebay, Philippe Pierre
Partner: UNT Libraries Government Documents Department

Scalable multi-correlative statistics and principal component analysis with Titan.

Description: This report summarizes existing statistical engines in VTK/Titan and presents the recently parallelized multi-correlative and principal component analysis engines. It is a sequel to [PT08] which studied the parallel descriptive and correlative engines. The ease of use of these parallel engines is illustrated by the means of C++ code snippets. Furthermore, this report justifies the design of these engines with parallel scalability in mind; then, this theoretical property is verified with test runs that demonstrate optimal parallel speed-up with up to 200 processors.
Date: February 1, 2009
Creator: Thompson, David C.; Bennett, Janine C.; Roe, Diana C. & Pebay, Philippe Pierre
Partner: UNT Libraries Government Documents Department

An Exodus II specification for handling gauss points.

Description: This report specifies the way in which Gauss points shall be named and ordered when storing them in an EXODUS II file so that they may be properly interpreted by visualization tools. This naming convention covers hexahedra and tetrahedra. Future revisions of this document will cover quadrilaterals, triangles, and shell elements.
Date: November 1, 2007
Creator: Thompson, David C.; Jortner, Jeffrey N. & Pebay, Philippe Pierre
Partner: UNT Libraries Government Documents Department

Visualization of higher order finite elements.

Description: Finite element meshes are used to approximate the solution to some differential equation when no exact solution exists. A finite element mesh consists of many small (but finite, not infinitesimal or differential) regions of space that partition the problem domain, {Omega}. Each region, or element, or cell has an associated polynomial map, {Phi}, that converts the coordinates of any point, x = ( x y z ), in the element into another value, f(x), that is an approximate solution to the differential equation, as in Figure 1(a). This representation works quite well for axis-aligned regions of space, but when there are curved boundaries on the problem domain, {Omega}, it becomes algorithmically much more difficult to define {Phi} in terms of x. Rather, we define an archetypal element in a new coordinate space, r = ( r s t ), which has a simple, axis-aligned boundary (see Figure 1(b)) and place two maps onto our archetypal element:
Date: April 1, 2004
Creator: Thompson, David C.; Pebay, Philippe Pierre; Crawford, Richard H. & Khardekar, Rahul Vinay
Partner: UNT Libraries Government Documents Department

Parallel auto-correlative statistics with VTK.

Description: This report summarizes existing statistical engines in VTK and presents both the serial and parallel auto-correlative statistics engines. It is a sequel to [PT08, BPRT09b, PT09, BPT09, PT10] which studied the parallel descriptive, correlative, multi-correlative, principal component analysis, contingency, k-means, and order statistics engines. The ease of use of the new parallel auto-correlative statistics engine is illustrated by the means of C++ code snippets and algorithm verification is provided. This report justifies the design of the statistics engines with parallel scalability in mind, and provides scalability and speed-up analysis results for the autocorrelative statistics engine.
Date: August 1, 2013
Creator: Pebay, Philippe Pierre & Bennett, Janine Camille
Partner: UNT Libraries Government Documents Department

Determining the Bayesian optimal sampling strategy in a hierarchical system.

Description: Consider a classic hierarchy tree as a basic model of a 'system-of-systems' network, where each node represents a component system (which may itself consist of a set of sub-systems). For this general composite system, we present a technique for computing the optimal testing strategy, which is based on Bayesian decision analysis. In previous work, we developed a Bayesian approach for computing the distribution of the reliability of a system-of-systems structure that uses test data and prior information. This allows for the determination of both an estimate of the reliability and a quantification of confidence in the estimate. Improving the accuracy of the reliability estimate and increasing the corresponding confidence require the collection of additional data. However, testing all possible sub-systems may not be cost-effective, feasible, or even necessary to achieve an improvement in the reliability estimate. To address this sampling issue, we formulate a Bayesian methodology that systematically determines the optimal sampling strategy under specified constraints and costs that will maximally improve the reliability estimate of the composite system, e.g., by reducing the variance of the reliability distribution. This methodology involves calculating the 'Bayes risk of a decision rule' for each available sampling strategy, where risk quantifies the relative effect that each sampling strategy could have on the reliability estimate. A general numerical algorithm is developed and tested using an example multicomponent system. The results show that the procedure scales linearly with the number of components available for testing.
Date: September 1, 2010
Creator: Grace, Matthew D.; Ringland, James T.; Boggs, Paul T. & Pebay, Philippe Pierre
Partner: UNT Libraries Government Documents Department

Monitoring computational clusters with OVIS.

Description: Traditional cluster monitoring approaches consider nodes in singleton, using manufacturer-specified extreme limits as thresholds for failure ''prediction''. We have developed a tool, OVIS, for monitoring and analysis of large computational platforms which, instead, uses a statistical approach to characterize single device behaviors from those of a large number of statistically similar devices. Baseline capabilities of OVIS include the visual display of deterministic information about state variables (e.g., temperature, CPU utilization, fan speed) and their aggregate statistics. Visual consideration of the cluster as a comparative ensemble, rather than as singleton nodes, is an easy and useful method for tuning cluster configuration and determining effects of real-time changes.
Date: December 1, 2006
Creator: Gentile, Ann C.; Brandt, James M.; Wong, M. H. & Pebay, Philippe Pierre
Partner: UNT Libraries Government Documents Department

A data storage model for novel partial differential equation descretizations.

Description: The purpose of this report is to define a standard interface for storing and retrieving novel, non-traditional partial differential equation (PDE) discretizations. Although it focuses specifically on finite elements where state is associated with edges and faces of volumetric elements rather than nodes and the elements themselves (as implemented in ALEGRA), the proposed interface should be general enough to accommodate most discretizations, including hp-adaptive finite elements and even mimetic techniques that define fields over arbitrary polyhedra. This report reviews the representation of edge and face elements as implemented by ALEGRA. It then specifies a convention for storing these elements in EXODUS files by extending the EXODUS API to include edge and face blocks in addition to element blocks. Finally, it presents several techniques for rendering edge and face elements using VTK and ParaView, including the use of VTK's generic dataset interface for interpolating values interior to edges and faces.
Date: April 1, 2007
Creator: Doyle, Wendy S.K.; Thompson, David C. & Pebay, Philippe Pierre
Partner: UNT Libraries Government Documents Department

Meaningful statistical analysis of large computational clusters.

Description: Effective monitoring of large computational clusters demands the analysis of a vast amount of raw data from a large number of machines. The fundamental interactions of the system are not, however, well-defined, making it difficult to draw meaningful conclusions from this data, even if one were able to efficiently handle and process it. In this paper we show that computational clusters, because they are comprised of a large number of identical machines, behave in a statistically meaningful fashion. We therefore can employ normal statistical methods to derive information about individual systems and their environment and to detect problems sooner than with traditional mechanisms. We discuss design details necessary to use these methods on a large system in a timely and low-impact fashion.
Date: July 1, 2005
Creator: Gentile, Ann C.; Marzouk, Youssef M.; Brandt, James M. & Pebay, Philippe Pierre
Partner: UNT Libraries Government Documents Department

A half-explicit, non-split projection method for low Mach number flows.

Description: In the context of the direct numerical simulation of low MACH number reacting flows, the aim of this article is to propose a new approach based on the integration of the original differential algebraic (DAE) system of governing equations, without further differentiation. In order to do so, while preserving a possibility of easy parallelization, it is proposed to use a one-step index 2 DAE time-integrator, the Half Explicit Method (HEM). In this context, we recall why the low MACH number approximation belongs to the class of index 2 DAEs and discuss why the pressure can be associated with the constraint. We then focus on a fourth-order HEM scheme, and provide a formulation that makes its implementation more convenient. Practical details about the consistency of initial conditions are discussed, prior to focusing on the implicit solve involved in the method. The method is then evaluated using the Modified KAPS Problem, since it has some of the features of the low MACH number approximation. Numerical results are presented, confirming the above expectations. A brief summary of ongoing efforts is finally provided.
Date: February 1, 2004
Creator: Pousin, Jerome G. (National Institute for Applied Sciences, France); Najm, Habib N. & Pebay, Philippe Pierre
Partner: UNT Libraries Government Documents Department

The OVIS analysis architecture.

Description: This report summarizes the current statistical analysis capability of OVIS and how it works in conjunction with the OVIS data readers and interpolators. It also documents how to extend these capabilities. OVIS is a tool for parallel statistical analysis of sensor data to improve system reliability. Parallelism is achieved using a distributed data model: many sensors on similar components (metaphorically sheep) insert measurements into a series of databases on computers reserved for analyzing the measurements (metaphorically shepherds). Each shepherd node then processes the sheep data stored locally and the results are aggregated across all shepherds. OVIS uses the Visualization Tool Kit (VTK) statistics algorithm class hierarchy to perform analysis of each process's data but avoids VTK's model aggregation stage which uses the Message Passing Interface (MPI); this is because if a single process in an MPI job fails, the entire job will fail. Instead, OVIS uses asynchronous database replication to aggregate statistical models. OVIS has several additional features beyond those present in VTK that, first, accommodate its particular data format and, second, improve the memory and speed of the statistical analyses. First, because many statistical algorithms are multivariate in nature and sensor data is typically univariate, interpolation of data is required to provide simultaneous observations of metrics. Note that in this report, we will refer to a single value obtained from a sensor as a measurement while a collection of multiple sensor values simultaneously present in the system is an observation. A base class for interpolation is provided that abstracts the operation of converting multiple sensor measurements into simultaneous observations. A concrete implementation is provided that performs piecewise constant temporal interpolation of multiple metrics across a single component. Secondly, because calculations may summarize data too large to fit in memory OVIS analyses batches of observations at a time and aggregates ...
Date: July 1, 2010
Creator: Mayo, Jackson R.; Gentile, Ann C.; Brandt, James M.; De Sapio, Vincent; Thompson, David C.; Roe, Diana C. et al.
Partner: UNT Libraries Government Documents Department

OVIS 2.0 user%3CU%2B2019%3Es guide.

Description: This document describes how to obtain, install, use, and enjoy a better life with OVIS version 2.0. The OVIS project targets scalable, real-time analysis of very large data sets. We characterize the behaviors of elements and aggregations of elements (e.g., across space and time) in data sets in order to detect anomalous behaviors. We are particularly interested in determining anomalous behaviors that can be used as advance indicators of significant events of which notification can be made or upon which action can be taken or invoked. The OVIS open source tool (BSD license) is available for download at ovis.ca.sandia.gov. While we intend for it to support a variety of application domains, the OVIS tool was initially developed for, and continues to be primarily tuned for, the investigation of High Performance Compute (HPC) cluster system health. In this application it is intended to be both a system administrator tool for monitoring and a system engineer tool for exploring the system state in depth. OVIS 2.0 provides a variety of statistical tools for examining the behavior of elements in a cluster (e.g., nodes, racks) and associated resources (e.g., storage appliances and network switches). It calculates and reports model values and outliers relative to those models. Additionally, it provides an interactive 3D physical view in which the cluster elements can be colored by raw element values (e.g., temperatures, memory errors) or by the comparison of those values to a given model. The analysis tools and the visual display allow the user to easily determine abnormal or outlier behaviors. The OVIS project envisions the OVIS tool, when applied to compute cluster monitoring, to be used in conjunction with the scheduler or resource manager in order to enable intelligent resource utilization. For example, nodes that are deemed less healthy, that is, nodes that exhibit outlier ...
Date: April 1, 2009
Creator: Mayo, Jackson R.; Gentile, Ann C.; Brandt, James M.; Thompson, David C.; Roe, Diana C.; Wong, Matthew H. et al.
Partner: UNT Libraries Government Documents Department

OVIS 3.2 user's guide.

Description: This document describes how to obtain, install, use, and enjoy a better life with OVIS version 3.2. The OVIS project targets scalable, real-time analysis of very large data sets. We characterize the behaviors of elements and aggregations of elements (e.g., across space and time) in data sets in order to detect meaningful conditions and anomalous behaviors. We are particularly interested in determining anomalous behaviors that can be used as advance indicators of significant events of which notification can be made or upon which action can be taken or invoked. The OVIS open source tool (BSD license) is available for download at ovis.ca.sandia.gov. While we intend for it to support a variety of application domains, the OVIS tool was initially developed for, and continues to be primarily tuned for, the investigation of High Performance Compute (HPC) cluster system health. In this application it is intended to be both a system administrator tool for monitoring and a system engineer tool for exploring the system state in depth. OVIS 3.2 provides a variety of statistical tools for examining the behavior of elements in a cluster (e.g., nodes, racks) and associated resources (e.g., storage appliances and network switches). It provides an interactive 3-D physical view in which the cluster elements can be colored by raw or derived element values (e.g., temperatures, memory errors). The visual display allows the user to easily determine abnormal or outlier behaviors. Additionally, it provides search capabilities for certain scheduler logs. The OVIS capabilities were designed to be highly interactive - for example, the job search may drive an analysis which in turn may drive the user generation of a derived value which would then be examined on the physical display. The OVIS project envisions the capabilities of its tools applied to compute cluster monitoring. In the future, integration with ...
Date: October 1, 2010
Creator: Mayo, Jackson R.; Gentile, Ann C.; Brandt, James M.; Houf, Catherine A.; Thompson, David C.; Roe, Diana C. et al.
Partner: UNT Libraries Government Documents Department

A framework for graph-based synthesis, analysis, and visualization of HPC cluster job data.

Description: The monitoring and system analysis of high performance computing (HPC) clusters is of increasing importance to the HPC community. Analysis of HPC job data can be used to characterize system usage and diagnose and examine failure modes and their effects. This analysis is not straightforward, however, due to the complex relationships that exist between jobs. These relationships are based on a number of factors, including shared compute nodes between jobs, proximity of jobs in time, etc. Graph-based techniques represent an approach that is particularly well suited to this problem, and provide an effective technique for discovering important relationships in job queuing and execution data. The efficacy of these techniques is rooted in the use of a semantic graph as a knowledge representation tool. In a semantic graph job data, represented in a combination of numerical and textual forms, can be flexibly processed into edges, with corresponding weights, expressing relationships between jobs, nodes, users, and other relevant entities. This graph-based representation permits formal manipulation by a number of analysis algorithms. This report presents a methodology and software implementation that leverages semantic graph-based techniques for the system-level monitoring and analysis of HPC clusters based on job queuing and execution data. Ontology development and graph synthesis is discussed with respect to the domain of HPC job data. The framework developed automates the synthesis of graphs from a database of job information. It also provides a front end, enabling visualization of the synthesized graphs. Additionally, an analysis engine is incorporated that provides performance analysis, graph-based clustering, and failure prediction capabilities for HPC systems.
Date: August 1, 2010
Creator: Mayo, Jackson R.; Kegelmeyer, W. Philip, Jr.; Wong, Matthew H.; Pebay, Philippe Pierre; Gentile, Ann C.; Thompson, David C. et al.
Partner: UNT Libraries Government Documents Department

Bayesian methods for estimating the reliability in complex hierarchical networks (interim report).

Description: Current work on the Integrated Stockpile Evaluation (ISE) project is evidence of Sandia's commitment to maintaining the integrity of the nuclear weapons stockpile. In this report, we undertake a key element in that process: development of an analytical framework for determining the reliability of the stockpile in a realistic environment of time-variance, inherent uncertainty, and sparse available information. This framework is probabilistic in nature and is founded on a novel combination of classical and computational Bayesian analysis, Bayesian networks, and polynomial chaos expansions. We note that, while the focus of the effort is stockpile-related, it is applicable to any reasonably-structured hierarchical system, including systems with feedback.
Date: May 1, 2007
Creator: Marzouk, Youssef M.; Zurn, Rena M.; Boggs, Paul T.; Diegert, Kathleen V. (Sandia National Laboratories, Albuquerque, NM); Red-Horse, John Robert (Sandia National Laboratories, Albuquerque, NM) & Pebay, Philippe Pierre
Partner: UNT Libraries Government Documents Department

A variational solution to the transport equation subject to an affine constraint.

Description: We establish an existence and uniqueness theorem for the transport equation subject to an inequality affine constraint, viewed as a constrained optimization problem. Then we derive a Space-Time Integrated Least Squares (STILS) scheme for its numerical approximation. Furthermore, we discuss some L{sup 2}-projection strategies and with numerical examples we show that there are not relevant for that problem.
Date: February 1, 2004
Creator: Pousin, Jerome G. (National Institute of Applied Sciences, Villeurbanne Cedex, France); Najm, Habib N.; Picq, Martine (National Institute of Applied Sciences, Villeurbanne Cedex, France) & Pebay, Philippe Pierre
Partner: UNT Libraries Government Documents Department

The verdict geometric quality library.

Description: Verdict is a collection of subroutines for evaluating the geometric qualities of triangles, quadrilaterals, tetrahedra, and hexahedra using a variety of metrics. A metric is a real number assigned to one of these shapes depending on its particular vertex coordinates. These metrics are used to evaluate the input to finite element, finite volume, boundary element, and other types of solvers that approximate the solution to partial differential equations defined over regions of space. The geometric qualities of these regions is usually strongly tied to the accuracy these solvers are able to obtain in their approximations. The subroutines are written in C++ and have a simple C interface. Each metric may be evaluated individually or in combination. When multiple metrics are evaluated at once, they share common calculations to lower the cost of the evaluation.
Date: March 1, 2006
Creator: Knupp, Patrick Michael; Ernst, C.D. (Elemental Technologies, Inc., American Fork, UT); Thompson, David C. (Sandia National Laboratories, Livermore, CA); Stimpson, C.J. (Elemental Technologies, Inc., American Fork, UT) & Pebay, Philippe Pierre
Partner: UNT Libraries Government Documents Department