Computing Community Consortium Blog

The goal of the Computing Community Consortium (CCC) is to catalyze the computing research community to debate longer range, more audacious research challenges; to build consensus around research visions; to evolve the most promising visions toward clearly defined initiatives; and to work with the funding organizations to move challenges and visions toward funding initiatives. The purpose of this blog is to provide a more immediate, online mechanism for dissemination of visioning concepts and community discussion/debate about them.

Great Innovative Idea- Known Unknowns: Testing in the Presence of Uncertainty

June 3rd, 2015 / in awards, research horizons, Research News / by Helen Wright

Screen Shot 2015-06-03 at 8.39.49 AM

The following Great Innovative Idea is from Sebastian Elbaum, Professor of Computer Science and Engineering at the University of Nebraska-Lincoln and David S. Rosenblum, Dean of the School of Computing at the National University of Singapore.

Their paper Known Unknowns: Testing in the Presence of Uncertainty won second place at the Computing Community Consortium (CCC) sponsored Blue Sky Ideas Conference Track series at the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE), November 16-22, 2014 in Hong Kong.

The Innovative Idea

Uncertainty is present in most systems we build today, whether introduced by human decisions, machine learning algorithms, external libraries, or sensing variability. This uncertainty leads to occasional misbehavior or incorrect output that is deemed to be acceptable. In the context of software testing, this uncertainty makes it difficult to distinguish acceptable from unacceptable misbehaviors, and to determine when there are faults in the system that are being masked by acceptable misbehaviors. Existing approaches to deal with uncertainty in testing have been partial and of limited scope, leaving the systematic treatment of uncertainty in testing still open.  In the paper, we explore the use of Hidden Markov Models and statistical reasoning about behaviors observed during testing in order to distinguish between acceptable misbehavior and behavioral errors that are due to latent faults.


The idea will allow us to deal better with uncertainty that arises in modern complex software systems in many different forms, such as imprecision in readings from hardware sensors or imprecision in machine learning-based classifiers, and correspondingly in a broad range of applications from human activity recognition to quadcopter stabilization.

Other Research

We both work in testing and analysis of complex software systems.  Sebastian has interests in the development and assessment of automated techniques to improve software dependability. David has interests in the design, verification and testing of distributed systems and context-aware ubiquitous computing systems.

Researcher’s Background

Sebastian is a Professor of Computer Science and Engineering at the University of Nebraska – Lincoln (UNL). He is a co-founder of the E2 Software Engineering Group at UNL, and the Nimbus UAV Lab at UNL. He is the Program Co-Chair for the 2015 International Conference in Software Engineering, and a member of the ACM Transactions on Software Engineering and Methodology Editorial Board.


David is Professor of Computer Science and Dean of the School of Computing at the National University of Singapore (NUS).  He previously held academic positions at University College London and the University of California, Irvine, and he also was a research scientist at AT&T Bell Laboratories (Murray Hill) and CTO at a technology startup called PreCache, Inc.  He is the Editor-in-Chief of the ACM Transactions on Software Engineering and Methodology (ACM TOSEM) and is Past Chair of the ACM Special Interest Group on Software Engineering (ACM SIGSOFT).


Sebastian’s homepage is at

David’s homepage is at

To view more Great Innovative Ideas, please click here.

Great Innovative Idea- Known Unknowns: Testing in the Presence of Uncertainty