Deep in Data: Empirical Data Based Software Accuracy Testing Using the Building America Field Data Repository: Preprint Page: 3 of 10
This article is part of the collection entitled: Office of Scientific & Technical Information Technical Reports and was provided to Digital Library by the UNT Libraries Government Documents Department.
The following text was automatically extracted from the image on this page using optical character recognition software:
DEEP IN DATA: EMPIRICAL DATA-BASED SOFTWARE ACCURACY TESTING USING
THE BUILDING AMERICA FIELD DATA REPOSITORY
Joel Neymarki, David Roberts2
J. Neymark & Associates, Golden, Colorado, U.S.
2National Renewable Energy Laboratory, Golden, Colorado, U.S.
An opportunity is available for using home energy
consumption and building description data to develop a
standardized accuracy test for residential energy analysis
tools. That is, to test the ability of uncalibrated simulations
to match real utility bills. Empirical data collected from
around the United States have been translated into a
uniform Home Performance Extensible Markup Language
format that may enable software developers to create
translators to their input schemes for efficient access to the
data. This may facilitate the possibility of modeling many
homes expediently, and thus implementing software
accuracy test cases by applying the translated data. This
paper describes progress toward, and issues related to,
developing a usable, standardized, empirical data-based
software accuracy test suite.
Background: Why We Are But Not Where, or, Where
We Are But Not Why?
Software accuracy tests play a vital role in the continuous
improvement of residential building energy analysis
[Judkoff and Neymark 2006, Judkoff et al. 2010, Polly et al
2011, RESNET 2006]. Historically, established software
accuracy tests are based on the Building Energy Simulation
and Diagnostic Test (BESTEST) methodology [Judkoff and
Neymark 2006, ASHRAE 2009]. These types of tests are
included in ANSJ/ASHRAE Standard 140, Method of Test
for the Evaluation of Building Energy Analysis Computer
Programs [ASHRAE 2011], and comprise idealized test
suites where programs are compared to each other and/or to
analytical or quasi-analytical solutions. Such
deterministically oriented test cases work well for finding
and diagnosing software errors; however, without direct
comparisons to empirical data there is no physical truth
standard of comparison with respect to overall accuracy.
So, BESTEST can tell us "why we are" (or at least help
diagnose why we are having errors), but cannot evaluate
true accuracy relative to how a real building performs as
built and as occupied.
A carefully conceived laboratory-based empirical
validation study can provide both prediction accuracy
testing and diagnostic capability, i.e., it addresses both the
"where" and the "why." However, such procedures have
been developed with only limited success. This is because
such tests are an order of magnitude more expensive to
This report is available at no cost from the
National Renewable Energy Laboratory (NREL)
develop than BESTEST-type tests, requiring substantial
dedicated multi-year funding. Because of the expense of
constructing facilities, such tests can be accomplished in
only a limited number of climates and configurations. Also,
many previously published empirical validation studies
failed to empirically determine fundamental inputs (in
addition to the outputs), and therefore can contain
substantial bias errors [Neymark et al. 2005].
Proposed new test cases with measured audit (not
laboratory) data for multiple buildings, applying a
stochastic approach, provide an as-built, as-occupied
energy-use target, but not much precision. Figure 1
illustrates a preliminary example of the type of accuracy
observable with current data. The blue solid line and the
blue dashed lines represent perfect agreement and +40%
disagreement between predicted and measured data,
respectively. Here we can discern some signal (correlation
of predicted versus measured energy consumption) from
the noise (data scatter related to bias and random error, e.g.,
occupant behavior). This type of test suite addresses the
"where we are, but not why." That is, we see how well we
can hit the target, but when disagreement between
predictions and measured data occurs, there is only limited
diagnostic capability based on statistical analysis for
identifying causes of disagreements.
The remainder of the paper describes development of the
new empirical data-based software accuracy test.
Predicted v. Measured Natural Gas Use Data Set Y
500 - -
0 500 1000 1500 2000 2500 3000
Measured Natural Gas Use (Therms)
Figure 1. Predicted versus Measured Natural Gas Use
from A Preliminary Study [Roberts et al. 2012]
Here’s what’s next.
This article can be searched. Note: Results may vary based on the legibility of text within the document.
Tools / Downloads
Get a copy of this page or view the extracted text.
Citing and Sharing
Basic information for referencing this web page. We also provide extended guidance on usage rights, references, copying or embedding.
Reference the current page of this Article.
Neymark, J. & Roberts, D. Deep in Data: Empirical Data Based Software Accuracy Testing Using the Building America Field Data Repository: Preprint, article, June 1, 2013; Golden, Colorado. (digital.library.unt.edu/ark:/67531/metadc841135/m1/3/: accessed January 21, 2019), University of North Texas Libraries, Digital Library, digital.library.unt.edu; crediting UNT Libraries Government Documents Department.