(There are exceptions, of course, like the writings of Cory Doctorow.)
But in “Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy,” Cathy O’Neil presents a concise case about the perils of Big Data through the examples she offers over decades of technological development, and this text will remain critically relevant in the years ahead. She addresses the pattern of fundamental flaws at the core of many of these systems and her cautionary remarks about increasing surveillance are perhaps the most pertinent points of the entire book.
Details of Big Bang Data exhibit at CCCB (Photo Credit: By Kippelboy (Own work) CC BY 3.0 (http://creativecommons.org/licenses/by/3.0), via Wikimedia Commons)
Each of the examples of Weapons of Math Destruction are characterized by intrinsic flaws. To identify these traits, she poses three questions to ask when examining any Big Data system:
First – Even if the participant is aware of being modeled, or what the model is used for, is the model opaque, or even invisible?
Second – Does the model work against the subject’s interest? In short, is it unfair? Does it damage or destroy lives?
And finally – [has] the model the capacity to grow exponentially? As a statistician would put it, can it scale?
Throughout the book, O’Neil explores several examples of WMDs and their socio-economic consequences. The introduction presents how IMPACT scoring unfairly resulted in the termination of good teachers, and how WMDs routinely target the poor where they hurt the most. The first chapter outlines her work as a hedge fund quantitative analyst leading up to the collapse of the housing market. Predatory lending is a key example of a WMD. Next, she examines the feedback loop created by the U.S. News college ranking report, and the resulting skyrocketing of college tuition, as well as the predatory nature of enrollment marketing campaigns.
From there, she dives into UCLA’s PredPol system, designed to optimize police patrol of areas where crime is statistically most likely to occur, and how the system inherently targets impoverished neighborhoods, creating yet another feedback loop of increased incarceration. Another chapter outlines the negative consequences of automated resume analysis and job performance metrics, and how the “optimization” of work shifts negatively impacts the middle class and the working poor. The final chapters present similar flaws in data systems determining insurance rates and credit eligibility, as well as Big Data’s Orwellian impact on the political process of voter targeting.
While the world painted by these flawed systems may appear dour, the text is not without hope. Scott Galloway’s book, “The Four: The Hidden DNA of Amazon, Apple, Facebook and Google” painted the apocalyptic near-future where Apple, Google, Amazon, and Facebook serve as the four horsemen of the end times. But O’Neil’s concluding chapter offers a number of proposed solutions to implement checks and balances into these systems to prevent that sort of abuse and exploitation. O’Neil presents the informed insight of a woman in a field severely dominated by men, and her perspective of big data through the lens of moral conscience. She humanizes and personalizes the societal effect of these systems and makes the subject of algorithms engaging and impactful.
“Weapons of Math Destruction” effectively outlines the characteristic flaws shared by many Big Data systems throughout history, and presents practical measures to reign in these unchecked operations. It’s a sharp and relevant text for anyone interested in the way these technologies shape our culture.
The Innerspace Connection 