April 25, 2006

Sue M. Cobb
Secretary of State
Florida Department of State
R. A. Gray Building
500 S. Bronough St.
Tallahassee, FL 32399-0250

Dear Ms. Cobb,

We are writing as practitioners in the field of information quality management, to express our concerns about the circumstances surrounding the administration of elections in numerous locations throughout the country and in your State. Below we comment on the developments you are presently confronting in Leon County. We believe that these circumstances are the result of misunderstandings regarding the nature of processes that produce information (such as elections) and a lack of application of established principles and practices for the improvement of such processes.

Unfortunately, the problems that are arising reflect fundamental misunderstandings built into the basic nature of the Help America Vote Act (HAVA) and its administrative processes. HAVA has incentivized the replacing of known-defective technology with new technology, without adequate provisions to effectively observe and control the impact of these changes. While this may not be the intent, the result is that election processes are being placed further out of control throughout the country.

Some of these misunderstandings include an apparent tendency to think of automation as a means of assuring reliability and accuracy in itself, a general focus on developing standards that apply to voting technology in isolation, and misconceptions about how to assess accuracy.

We have several urgent recommendations to make:

1) Implement reliable measures of the election process that will enable observing and controlling the impact of changes being introduced into it.

2) Establish accountability for all process performers, both election administrators and providers of technology systems and services, in terms of the quality of the information being produced (the election results) by the process as a whole.

3) Join us in calling legislators, the Election Assistance Commission (EAC) and the National Institute of Standards and Technology (NIST), to act now to put these processes under control, based on a proper understanding of the nature of elections as information production processes.

NIST has been a champion of the application of effective principles for quality assurance ever since they were given stewardship of the Malcolm Baldrige National Quality Award in 1987(1). However, in their efforts to foster improvement of election processes, we do not see the EAC and NIST applying these principles to elections in a way that properly addresses the nature of the process.

We offer the following comments in the hopes of bringing greater understanding of these issues and helping to correct these circumstances.

Troubling Circumstances

As you know, Leon County Elections Supervisor Ion Sancho recently authorized a security test of the Diebold machines in his jurisdiction. Subsequently, arrangements between Diebold and the Leon County Elections Department were interrupted, and Mr. Sancho also experienced a failure on the part of the other two major vote technology vendors, ES&S and Sequoia, to respond to his efforts to employ their services instead.

At this time, election administrators who test for and discover the kinds of issues Mr. Sancho has highlighted often find themselves in a difficult position. The presuppositions embodied in HAVA appear to have led to a context in which such disclosures are not appropriately appreciated. When vendors are able to disengage in the way Mr. Sancho has encountered, the situation only appears to grow more difficult.

The issues Mr. Sancho revealed in testing the Diebold devices in Leon County are not the only issues associated with voting technology that will reveal themselves, whether as a result of such tests or in the course of the live conduct of elections. Many, including Florida's Governor Jeb Bush, have expressed concerns related to the automated voting devices (3), and there are many efforts underway to manage the effects of various characteristics of the devices in themselves.

We recommend a wholistic approach. There are ways to handle the consequences of such disclosures - most notably, the implementation of quality controls in the form of measures that show the quality of the process as a whole and thereby reveal whether the process has been affected by changes introduced in it.

Part of the reason issues are guaranteed to continue to arise is that such means for observing the impact of changes in technology are not being employed. In fact, the critical nature of elections and the complexity of information production processes themselves, means that these techniques must be used when introducing changes in election processes.

Many have expressed concerns regarding the use of voting technology in elections, noting the effect of the technology on transparency. While this has clear implications for how the technology impacts the process, we do not comment on this line of analysis here. Instead, we wish to simply observe that the present methods being employed are defective from the standpoint of recognized, well-established principles for managing the quality of complex information production processes.

Placing Election Information Production Processes Under Control

On Assuring Reliability and Automation: Simply adding automation to a process will not assure its reliability and accuracy. While automated devices in themselves generally execute the functions they are given very consistently, the reliability of an information production process depends on many more factors than the automation that may be added to it. Information production processes are complex processes, with a non-technical side - called "election administration" in this case - and a technical side, responsible for systems and applications used to produce information. Both sides work together to produce information, and both sides must be held accountable to the requirements for the product of the process - the vote count results, in this case.

On Assessing Technology in Isolation: The attempt to establish countermeasures for all the potential types of defects that can arise associated with computerized voting devices in themselves is an inherently complex enterprise. However, from the standpoint of managing the impact of changes in processes such as elections, the degree of confidence one may have in computerized voting devices in themselves is not the key concern. What matters is the reliability of the process as a whole, into which the devices may be introduced. The effect of the introduction of technology into a process can only be understood in the context of the process as a whole, performed live.

Measure the Election Results to Control the Process: The recognized professional practice for placing a complex process under control is to measure the product of the process in terms of the requirements it must fulfill. In the case of elections, election results are the product. Accuracy is among the most important characteristics that election results must meet.

Vendors who take part in this process must meet expectations, not only for their devices and applications, but for the information produced by the process. If any vendor has withdrawn from Leon County as a result of Mr. Sancho's security test, this is only made possible if accountability is not established for the role vendors play in producing reliable election results. Any vendor that seeks to operate free of assessments such as this should be rebuffed, even if these assessments open up profound questions and concerns regarding their technology.

Measuring Accuracy

Accuracy is a measure of the information produced by the process, not of the technology employed by the process. It is also a characteristic of information that cannot be measured by automatic tests. Accuracy is defined as "the degree to which data correctly reflects the real world object or event being described" (4).

Characteristics of information that can be measured automatically include completeness of values, validity in terms of business rules, acceptable values, reasonability tests, nonduplication, timeliness, and consistency across distributed sources (5).

A measure of accuracy, on the other hand, requires comparing of the electronic representation against the real world entity (or event) being represented. Automatic assessments cannot judge the agreement of the electronic representation with what it represents, and cannot be regarded as a definitive means of assessing accuracy.

This principle applies for any form of information production. As an example, the health care field deals broadly with three types of information: member data, health care provider data, and health care services delivered to members. The accuracy of data describing health plan members and health care providers is assured by those who interact directly with the members and providers, and the accuracy of data related to care provided to members is assured by the care providers themselves, who record their direct experience as they deliver care. Only participants in the process who are in a position to compare the data against the attributes of the real world entity or event being represented can be expected to create or assess accuracy.

For elections, which encompass a requirement of anonymous voting and therefore direct comparisons of individual electronic records against voter intent are not appropriate, this entails comparing vote total percentage results of the live process against the percentage results of a manual count of a representative random sample of human-readable ballots.

Information Quality Management in Health Care: The NCQA's Baseline Assessment Tool

Presently, the field of health care management is subject to more appropriate forms of accountability and quality control for information production than elections are held to.

The National Committee for Quality Assurance (NCQA) implements extensive reporting requirements for information production processes as part of its stewardship of health care quality measures in the Health Plan Employer Data and Information Set (HEDIS) (6). NCQA has recognized that if they are to assure that health plans provide reliable information regarding the quality of health care, controls must also be established over their information production processes.

The reporting requirements under HEDIS include information quality measures such as accuracy, completeness and timeliness, and all roles involved in producing information, including vendors and private practice health care practitioners.

Conclusion

We hope our comments have shown the urgency of our recommendations. Election processes are presently at grave risk, and they will only be increasingly placed out of control if reliable measures are not established to observe the impact of changes, and if accountability is not established for all process performers in relation to the quality of the election results being produced.

These problems are not local in scope.

NIST and EAC have taken an approach largely focused on developing standards addressing characteristics of voting devices, rather than fostering an approach based on a full understanding of how to assure the quality of election information production processes as a whole.

The Voter Confidence and Increased Accessibility Act (H.R. 550) presently advocates the establishment of paper records of voters' ballot choices, but it articulates audit protocols that do not provide valid means for revealing whether changes in election processes have affected the quality of the election results, particularly with respect to accuracy. The impact of changes in voting technology cannot be observed without such controls.

We ask that you take these comments in consideration and seek to implement them in your State, as well as join in advocating that Federal agencies and related legislation that propose to introduce voting technology into election processes apply these principles to place the processes under control.

Notes:

(1) See http://www.quality.nist.gov/

(2) See http://www.ncqa.org/programs/HEDIS/

(3) See http://www.sptimes.com/2005/12/18/State/Gov_Bush_calls_for_re.shtml

(4) See Larry English, "Defining and Measuring Accuracy", http://www.infoimpact.com/articles/DMR_7.03DefiningandMeasuringAccuracy.pdf

(5) See Larry English, "Improving Data Warehouse and Business Information Quality", Wiley & Sons, 1999, pp. 178-179

(6) See the NCQA/HEDIS Baseline Assessment Tool: http://www.ncqa.org/programs/HEDIS/2006/Volume5/BAT/main.htm

The following list their names in support of the above statement:

Matthias Groh, Six Sigma Master Black Belt

Seth Johnson, Information Quality Consultant

Bruce J. McTavish, Data Architect

David Rafner, Vice President for Industry Services, DAMA