Discovering metamorphic relations from OpenSWMM

Scientific software development can last for decades and needs to adjust as the underlying software evolves. Adding to and improving scientific software, such as SWMM requires testing later versions of the code to previous ones.  Unit testing, regression testing, and other testing need to be performed. One challenge is that there is insufficient automated testing because of the oracle problem. In software testing, an oracle defines the mechanism for checking whether the program under test produces the expected output when executed using a set of test cases. Test oracles are often not available due to reasons such as the inherent uncertainties in simulation models and the exploratory nature of scientific software written to find answers previously unknown. Metamorphic testing alleviates the oracle problem by specifying the relation that a source test case and its follow-up test case shall meet. For example, although the absolute value of sine(12) may depend on how the specific scientific software handles round-off and truncation, the relation sine(12)=sine(pi-12) must hold when the outcomes of the source test case (x=12) and follow-up test case (x=pi-12) are compared. The assertion, sine(x)=sine(pi-x), is called a metamorphic relation, and contemporary approaches mainly rely on software developers' and testers' expertise to create metamorphic relations. In this talk, we present our ongoing work to extract metamorphic relations from OpenSWMM, a forum where SWMM stakeholders post and comment on software usage questions. Our results show the valuable information available in user forums like OpenSWMM, help develop metamorphic relations to support the verification and validation for SWMM.

Disclaimer: This abstract has been reviewed in accordance with the United States Environmental Protection Agency (USEPA) policy and approved for publication.  The views expressed in this abstract are those of the authors and do not necessarily represent the views or the policies of the USEPA.  Any mention of trade names, manufacturers, or products does not imply an endorsement by the U.S, Government or the USEPA.