Essay About Analysis Of Actual Cases And Csli-94-188

Bias in Computer Systems
Essay Preview: Bias in Computer Systems
Report this essay
Bias in Computer Systems
BATYA FRIEDMAN
Colby College and The Mina Institute
HELEN NISSENBAUM
Princeton University
From an analysis of actual cases, three categories of bias in computer systems have been
developed: preexisting, technical, and emergent. Preexisting bias has its roots in social
institutions, practices, and attitudes. Technical bias arises from technical constraints or
considerations. Emergent bias arises in a context of use. Although others have pointed to bias
in particular computer systems and have noted the general problem, we know of no comparable
work that examines this phenomenon comprehensively and which offers a framework
for understanding and remedying it. We conclude by suggesting that freedom from bias should
be counted among the select set of criteria—including reliability, accuracy, and efficiency—
according to which the quality of systems in use in society should be judged.
Categories and Subject Descriptors: D.2.0 [Software]: Software Engineering; H.1.2 [Information
Systems]: User/Machine Systems; K.4.0 [Computers and Society]: General
General Terms: Design, Human Factors
Additional Key Words and Phrases: Bias, computer ethics, computers and society, design
methods, ethics, human values, standards, social computing, social impact, system design,
universal design, values
INTRODUCTION
To introduce what bias in computer systems might look like, consider the
case of computerized airline reservation systems, which are used widely by
travel agents to identify and reserve airline flights for their customers.
These reservation systems seem straightforward. When a travel agent
types in a customer’s travel requirements, the reservation system searches
This research was funded in part by the Clare Boothe Luce Foundation.
Earlier aspects of this work were presented at the 4S/EASST Conference, Goteborg, Sweden,
August 1992, and at InterCHI ’93, Amsterdam, April 1993. An earlier version of this article
appeared as Tech. Rep. CSLI-94-188, CSLI, Stanford University.
Authors’ addresses: B. Friedman, Department of Mathematics and Computer Science, Colby College,
Waterville, ME 04901; email: [email protected]; H. Nissenbaum, University Center for Human
Values, Marx Hall, Princeton University, Princeton, NJ 08544; email: [email protected].
Permission to make digital /hard copy of part or all of this work for personal or classroom use
is granted without fee provided that the copies are not made or distributed for profit or
commercial advantage, the copyright notice, the title of the publication, and its date appear,
and notice is given that copying is by permission of the ACM, Inc. To copy otherwise, to
republish, to post on servers, or to redistribute to lists, requires prior specific permission
and/ or a fee.
© 1996 ACM 1046-8188/96/0700–0330 $03.50
ACM Transactions on Information Systems, Vol. 14, No. 3, July 1996, Pages 330–347.
a database of flights and retrieves all reasonable flight options that meet or
come close to the customer’s requirements. These options then are ranked
according to various criteria, giving priority to nonstop flights, more direct
routes, and minimal total travel time. The ranked flight options are
displayed for the travel agent. In the 1980s, however, most of the airlines
brought before the Antitrust Division of the United States Justice Department
allegations of anticompetitive practices by American and United
Airlines whose reservation systems—Sabre and Apollo, respectively—dominated
the field. It was claimed, among other things, that the two reservations
systems are biased [Schrifin 1985].
One source of this alleged bias lies in Sabre’s and Apollo’s algorithms for
controlling search and display functions. In the algorithms, preference is
given to “on-line” flights, that is, flights with all segments on a single
carrier. Imagine, then, a traveler who originates in Phoenix and flies the
first segment of a round-trip overseas journey to London on American
Airlines, changing planes in New York. All other things being equal, the
British Airlines’ flight from New York to London would be ranked lower
than the American Airlines’ flight from New York to London even though in
both cases a traveler is similarly inconvenienced by changing planes and
checking through customs. Thus, the computer systems systematically
downgrade and, hence, are biased against international carriers who fly
few, if any, internal U.S. flights, and against internal carriers who do not
fly international flights [Fotos 1988; Ott 1988].
Critics also have been concerned with two other problems.