Integrating parallel file I/O and database support for high-performance scientific data management Metadata

Metadata describes a digital item, providing (if known) such information as creator, publisher, contents, size, relationship to other resources, and more. Metadata may also contain "preservation" components that help us to maintain the integrity of digital files over time.

Available Formats

Dublin Core: XML Dublin Core: JSON Dublin Core: Text Dublin Core: RDF/XML UNTL: XML Access METS: XML

Title

  • Main Title Integrating parallel file I/O and database support for high-performance scientific data management

Creator

  • Author: No, J.
    Creator Type: Personal
  • Author: Thakur, R.
    Creator Type: Personal
  • Author: Choudhary, A.
    Creator Type: Personal

Contributor

  • Sponsor: United States. Department of Energy.
    Contributor Type: Organization
    Contributor Info: US Department of Energy (United States)

Publisher

  • Name: Argonne National Laboratory
    Place of Publication: Illinois
    Additional Info: Argonne National Lab., IL (United States)

Date

  • Creation: 2000-04-03

Language

  • English

Description

  • Content Description: Many scientific applications have large I/O requirements, in terms of both the size of data and the number of files or data sets. Management, storage, efficient access, and analysis of this data present an extremely challenging task. Traditionally, two different solutions are used for this problem: file I/O or databases. File I/O can provide high performance but is tedious to use with large numbers of files and large and complex data sets. Databases can be convenient, exible, and powerful but do not perform and scale well for parallel supercomputing applications. The authors have developed a software system, called Scientific Data Manager (SDM), that combines the good features of both file I/O and databases. SDM provides a thin layer of database-like functionality on top of a high-performance, parallel file-I/O interface (MPI-IO). As a result, users can access data with the convenience of databases and the performance of MPI-IO, without having to bother with the details of either. In t his paper, they describe the design and implementation of SDM. With the help of two parallel application templates, ASTRO3D and an Euler solver, they illustrate how some of the design criteria affect performance.
  • Physical Description: 12 pages

Subject

  • Keyword: Management
  • Keyword: Storage
  • STI Subject Categories: 99 General And Miscellaneous//Mathematics, Computing, And Information Science
  • Keyword: Implementation
  • Keyword: Design
  • Keyword: Performance

Source

  • Conference: 9th IEEE International Symposium on High Performance Distributed Computing (HPDC-9), Pittsburgh, PA (US), 08/01/2000--08/04/2000; Other Information: PBD: 3 Apr 2000; PBD: 3 Apr 2000

Collection

  • Name: Office of Scientific & Technical Information Technical Reports
    Code: OSTI

Institution

  • Name: UNT Libraries Government Documents Department
    Code: UNTGD

Resource Type

  • Article

Format

  • Text

Identifier

  • Report No.: ANL/MCS/CP-101504
  • Grant Number: W-31109-ENG-38
  • Office of Scientific & Technical Information Report Number: 764221
  • Archival Resource Key: ark:/67531/metadc723513
Back to Top of Screen