Micrometrics as a solution to software invisibility

Software is central to modern science. But software is also largely invisible, and in consequence, is undervalued, poorly understood, and subject to what appear to be underinvestment and policy decisions that are not driven by data. We must do better if we want science to address the challenges faced by humankind in a time of massive scientific opportunity but limited resources. I argue here that micrometrics can help us do better.

Invisibility of scientific software

It is software that makes it possible to collect, organize, and analyze large quantities of data; to operate sophisticated instrumentation; to simulate the behavior of complex systems; to work within far-flung collaborative teams; and to document and communicate research results. Without software, essentially all modern research would grind to a halt. 

Yet to a large extent, the software that underpins research is invisible. Descriptions of research discoveries often describe unique instrumentation that made them possible—genome sequencing machines, telescopes, or supercomputers; name datasets analyzed; refer to innovative algorithms; and cite publications that introduce key ideas and methods. But they rarely discuss in any detail the code that was acquired and/or developed in order to perform these various steps.

One plausible reason for this invisibility is that good software is invisible: like the electrical supply and plumbing that are also necessary to research success, it operates silently and automatically. However, while some software used in science may have these characteristics (e.g., operating systems and word processors, at least on good days), much of that software is quite different. It is expensive (to create, if not to acquire), diverse, and dynamic—and, in contrast to other systems viewed as infrastructure, is frequently central, not peripheral, to innovation. Indeed, software is in many respects more like unique instrumentation—and, like much laboratory instrumentation, will often be jury rigged to meet the needs of a specific experiment. (It can require a lot of work to make software general purpose.) So perhaps another reason for silence around software is a reluctance to draw back the veil and reveal its inherent messiness.

Software invisibility is bad for science

Whatever the reason for this state of affairs, invisibility of software is bad for science. In broad-brush terms, it is bad because stakeholders in the scientific process must frequently make decisions that depend upon or concern software, and in the absence of clarity concerning software’s role in that process, such decisions are likely to be sub-optimal. More specifically, we encounter three types of problem:

1. Software invisibility can result in researchers not taking seriously the tasks of software acquisition, creation, maintenance, and dissemination. One symptom of this problem is that far too much software is written by students, postdocs, and researchers with limited expertise in software engineering. Another is that training in software engineering rarely forms part of student education.
2. Software invisibility can result in uninformed and probably poor funding policy. Do funding agencies spend too much on software or too little? Should software be open source or proprietary? We cannot answer these questions if we do not know (and I assert that, to a large extent, we do not know) what software is important for science, who writes and maintains that software, how much it cost to produce, how much it is used, and how good it is. Funding agencies may well aspire to support the software that will lead to the most science, but they cannot know how to do this today.
3. Software invisibility can hinder understanding of science as a process. Social scientists studying the formation and dissemination of knowledge, or seeking to understand the impact of research funding on innovation and employment, focus on papers and patents. To what extent is software an important mechanism for knowledge transfer? How does standardization on common software packages encourage (or hinder) innovation? We do not know.

Others have proposed a range of solutions to these problems: for example, that all scientists should be trained in software engineering (Merali, 2010); that scientists should not be allowed to write software—the task instead being assigned to professional engineers; that funding for software should be increased (Keyes et al., 2011); that all scientific software should be open (Morin et al., 2012); that the provenance of all research data should be documented in a way that indicates the software used to produce it (Miles et al., 2005); and that software should be delivered as services rather than downloadable packages (Foster, 2005). There is merit to all of these proposals. But each is challenging to implement, and in the absence of data it has historically proven hard to persuade people of the merits of any one approach.

Micrometrics as a possible solution

I propose here a more modest (although I think potentially transformative) approach, which is to increase dramatically the amount of data available about the software used in science and how it is produced and used. And because we live in a computer-mediated and Internet-connected world, I propose further that we seek to obtain this data via what I call micrometrics: low-level data about software usage that is collected by instrumenting software to provide low-level usage information. For example, we can obtain micrometric data regarding the location and nature of program invocations by configuring software to “phone home” each time it is use, and we can determine the size, download counts, and contribution patterns of open source software packages by accessing repository logs.

The resulting micrometrics will lack the specificity of explicit acknowledgements of use. For example, they will not tell us that a particular discovery was due to the use of package A. However, they can tell us that package A was used 10 times more than any other package by the discovering team. Similarly, we may see that use of a package B at institution C emerges at the same time that student D joins that institution. Neither correlation necessarily corresponds to causation, but in both cases we obtain useful data without any user effort.

While micrometrics may be trivial to generate, they may well be politically (and, in some cases, perhaps legally) challenging to collect, subject to gaming, and technically difficult to analyze. I do not want to downplay the importance of these issues, but nor do I see them as fatal flaws. Suitable incentives can overcome political resistance: for example, federal funding agencies can encourage integration of micrometric collectors into software that they fund by making funding decisions depend in part on intensity of usage, while auto-update mechanisms (already common in many software packages) can increase acceptance among users. Well-known methods can be used to isolate micrometric data to guard against privacy violations.

My team at Argonne National Laboratory and the University of Chicago has experience with micrometrics. At the request of the National Science Foundation, we incorporated some years ago usage logging into the Globus software that we distribute to the scientific community. This decision initially led to concerns  (Europeans complained about “US spyware”), but once we explained that it was easy to opt out of the reporting, all has proceeded smoothly. Today we obtain more than 10 million usage report records per day. These records have helped us improve software quality in several respects. They have not informed science policy more broadly, in part I think because the science policy community is not looking at software usage, and in part because there is little other data to compare with. But the potential is there.

A trend that can facilitate the collection of high-quality micrometric is the emergence of software-as-a-service (SaaS) methods for software delivery. SaaS services such as Google mail and Amazon ecommerce (in industry) and Globus Online (Allen et al., 2012) and nanoHUB (Klimeck et al., 2008) (in science) have the big advantage that all user interactions are mediated by the hosted service. Thus, much more information can be collected and worries about compromised data quality due to missing data are reduced. We again have experience with this approach due to our work with the Globus Online research data management service, and again, our experiences are positive.

Three concrete steps

I propose three steps to realizing this vision of software made visible via micrometrics:

1. Start adding micrometrics to our software.
2. Encourage funding agencies to mandate micrometric data collection and to support concrete experiments with the use of micrometric data to evaluate software utility.
3. Engage social scientists in examining questions relating to the relationship of software to innovation, in order to identify the data sources that will be most useful in answering questions such as those above.

Progress in these three areas will then allow us to pursue our ultimate goal of making scientific software visible in the eyes of its users, the policy makers who shape policies that influence it, and the scientists who study it.

Acknowledgments

I thank Dan Katz for comments on an early draft of this note. This work was supported in part by grants NSF ACI-1240726 and DOE DE-AC02-06CH11357.

References

Allen, B., Bresnahan, J., Childers, L., Foster, I., Kandaswamy, G., Kettimuthu, R., Kordas, J., Link, M., Martin, S., Pickett, K. and Tuecke, S. Software as a Service for Data Scientists. Communications of the ACM, 55(2):81-88, 2012.

Foster, I. Service-Oriented Science. Science, 308(5723):814-817, 2005.

Keyes, D. and others National Science Foundation Advisory Committee for CyberInfrastructure Task Force on Software for Science and Engineering: Final Report, http://1.usa.gov/1fMYINU, 2011.

Klimeck, G., McLennan, M., Brophy, S.P., Adams III, G.B. and Lundstrom, M.S. nanoHUB.org: Advancing Education and Research in Nanotechnology. Computing in Science and Engineering, 10(5):17-23, 2008.

Merali, Z. Computational science: … Error: Why scientific programming does not compute. Nature, 467:775-777, 2010.

Miles, S., Groth, P., Branco, M. and Moreau, L. The requirements of using provenance in e-science experiments. Journal of Grid Computing, 5(1):1-25, 2005.

Morin, A., Urban, J., Adams, P.D., Foster, I., Sali, A., Baker, D. and Sliz, P. Shining Light into Black Boxes. Science, 336(6078):159-160, 2012.