Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy
By Cathy O'Neil
Review by Dave Wineberg
We model everything now. Teacher evaluations, job applicants, credit applications, online purchasing, voting patterns, crime – pretty much anything you can think of is modeled in some opaque black box of unaccountable algorithms. They are so inherently faulty, discriminatory and racist as to be shameful. They cut off the poor from keeping up, and provide the wealthy with all kinds of advantages. Data Scientist Cathy O’Neil says in Weapons of Math Destruction: ”Big Data processes codify the past. They do not invent the future.”
Something as simple as a zip code can tell a system what kind of neighborhood you live in, and make assumptions. Search history, social media activity, purchase record – all contribute to an instant decision that you are worthy or not. These values are plugged in to school applications, job applications, and personal evaluations such as HR records, personality tests and even dating sites. Even purged, forgiven, and expired details remain active. Police model neighborhoods. They harass residents for every little thing in poorer neighborhoods, while giving a free pass to wealthier ones, where crimes are far bigger, but mostly white collar. Only ten states have outlawed the use of credit checks on job applications. For shopping at downscale stores, credit cardholders had their limits slashed, making them poorer and making them poorer risks – as in higher interest rates. It is computer models that schedule shifts, without concern for the needs of the employee in terms of child care, time off between shifts, or advance notice. Managers are paid to optimize revenue per hour worked, so memos from above go unheeded.
That models are often incorrect, badly designed, misinformed and misconstrued, means that people are denied service, or not hired, or outright fired. But there’s always someone else behind them, so it’s just the cost of doing business. “Unfairness is the black stuff belching out of the smokestacks. It’s an emission, a toxic one,” O’Neil says. We are all just collateral damage.
One insurance company instantly evaluates whether a customer is likely to shop around. If it judges not, it charges them more. It actually has 100,000 microsegments (buckets) depending on instant customer scores. In Florida, a driver with a clean record but a poor credit score pays $1552 more for insurance than a driver with a high credit score and a drunk driving conviction. Shopping sites won’t offer you a discount if you are already logged in. Payday loans and for profit schools prey on the disadvantaged and the desperate, extracting billions from them. The games are endless.
WMD is extremely fast paced, fact packed, and depressing. It has come to the point that machines dictate who may have a successful life, right out of the gate. Initiative, courage, creativity, drive, human kindness – don’t enter into it. We are all typecast by Big Data - assigned values mathematically that can stymie a life. There is no appeal. There isn’t even any knowing. The poor get poorer. The rich find the new era refreshing.
And of course, none of this is transparent. Customers cannot arrive at these prices, these decisions or these scores themselves. It’s all in the math, manipulating us. And yet, 73% of Americans believe search engine results are “accurate and impartial”. 62% believe Facebook posts their submissions to everyone. Nothing could be farther from the truth. Worse, data banks draw on each other, multiplying their errors, sometimes creating completely false profiles of a person, who then cannot get a job, rent an apartment or buy a car.
O’Neil says she is outraged by her own profession. You will be too.
Editor's note: This review was has been reposted with permission of David Wineberg. Like what you read? Subscribe to the SFRB's free daily email notice so you can be up-to-date on our latest articles. Scroll up this page to the sign-up field on your right.