pyGlobus: A python interface to the globus toolkit(TM) Page: 3 of 8
This article is part of the collection entitled: Office of Scientific & Technical Information Technical Reports and was provided to UNT Digital Library by the UNT Libraries Government Documents Department.
Extracted Text
The following text was automatically extracted from the image on this page using optical character recognition software:
Information Power Grid (IPG) and the NCSA Alliance's National Technology Grid have demonstrated the
feasibility of providing persistent Grid Services to the scientific community. One similarity amongst these
Grids is the use of the Globus toolkit to provide many of the underlying Grid Services. The Globus toolkit
provides a number of modules that implement Grid Services for security, resource discovery, data
transfer/management, etc. It has become the most popular solution to providing these services.
Although many of the necessary services are becoming available, they can still be very challenging to
use. To fully realize the goal of allowing application scientists to routinely use Grid Services, more must be
done to ease the burden of Grid application development. The careful use of appropriate abstractions and
higher-level constructs such as objects and components can help hide much of the complexity of Grid
programming from the application scientist.
We now consider a motivating example from the field of observation cosmology that illustrates many of
the requirements we see for 21st century science.
Recent studies of distant supernova have shown that the expansion of the universe is accelerating under
the influence of a new force, called dark energy. Current studies are conducted by geographically
distributed research teams, and involve the coordinated use of several ground-based observatories, the
Hubble space telescope and multiple distributed compute and storage resources. For example, the
Supernova Factory at LBNL utilizes instruments in Hawaii and California, and storage and compute
resources at Cal Tech and LBNL. As the program progresses, it will incorporate resources in Chile and the
Canary Islands. This program is a stepping-stone to the next generation search, the space-based Supernova
Acceleration Probe (SNAP).
As the scale of these searches has increased, a number of new requirements have emerged. The first is
the shear scale of the data handling and compute tasks involved. Raw, uncorrected sky images must be
transferred nightly from the remote observatories to compute facilities. The images are then corrected and
calibrated to remove any atmospheric effects or tracking errors. The results are then compared to baseline
sky catalogs to eliminate asteroids and man-made satellites. Finally algorithms are applied to the images to
search for increases in stellar magnitude that may indicate a supernova event. The resulting data is then
analyzed manually be researchers to find the most promising candidates to observe. This process involves
approximately 50 gigabytes of data in 500 files to be transferred, processed, and archived daily for the life
of the project - 5 to 10 years.
Secondly, as the accuracy of supernova models increases, it should be possible to allow the tight
integration of simulation data with experimental data to help filter out candidate supernovas. As more
accurate supernova simulations are developed over the next year, it should become possible to use these to
filter out candidate supernovas for further observation. This process of comparing simulation with
experiment must happen within a 24 hour time period to be of use in filtering out candidate supernovas.
To create large scale science applications such as those described above requires the close coupling of a
variety of technologies. It is necessary to move and access large amounts of data, access high-performance
compute resources, and integrate one-of-a-kind scientific instruments. Many of these functions are
provided by a combination of open-source and proprietary software that must be used as an integrated
whole. The Python language offers excellent support for this type of "glue" coding.
pyGlobus Overview
The rest of the paper will focus on the Python CoG Kit, pyGlobus, and explain; what the high-level goals
for the project are, and how Globus concepts are mapped to the Python idiom. It will also discuss some of
the underlying implementation details before examining a number of the most commonly used interfaces.
This project began with a number of important high-level goals in mind. First, we wanted to ensure
performance levels at or near the native Globus C code. To do this we have relied on the use of native
extension modules in Python. This allows Python code to cleanly interface with the underlying C code. By
using Python solely as a very thin control proxy, we can minimize the performance cost associated with the
wrapping. Second, where possible we have mapped the underlying C code to a natural Python idiom. For
example, in C it is normal to return an int status code and use pointers to pass in other output variables. In
Python functions may return multiple values. The wrapper functions take care of mapping between these
two styles. Another important example is the use of exceptions. Python provides support for catching and
throwing exceptions to indicate error conditions. The pyGlobus wrappers convert the underlying Globus
error codes into Python exceptions, allowing for much cleaner error handling at the Python level. The third
Upcoming Pages
Here’s what’s next.
Search Inside
This article can be searched. Note: Results may vary based on the legibility of text within the document.
Tools / Downloads
Get a copy of this page or view the extracted text.
Citing and Sharing
Basic information for referencing this web page. We also provide extended guidance on usage rights, references, copying or embedding.
Reference the current page of this Article.
Jackson, Keith R. pyGlobus: A python interface to the globus toolkit(TM), article, July 1, 2002; Berkeley, California. (https://digital.library.unt.edu/ark:/67531/metadc733624/m1/3/: accessed April 24, 2024), University of North Texas Libraries, UNT Digital Library, https://digital.library.unt.edu; crediting UNT Libraries Government Documents Department.