Scan barcode
relin's review
challenging
informative
sad
slow-paced
5.0
Content Warnings
Graphic: Racism and Ableism
Moderate: Child abuse
This book goes in depth into how technology has been used to make it easier to encode systemic -isms into our systems that effect people now.ege's review
informative
medium-paced
4.0
This book discusses three examples of algorithms that guide decision making in government services. It is split into five chapters. Chapter 1 covers the history of welfare in the United States. Chapters 2-4 cover three examples of welfare distribution algorithms that have had either intended or unintended negative consequences for the people who need their services. Chapter 5 discusses a few ways that we can prevent these negative consequences in the future, specifically emphasizing how data scientists need to "do no harm".
First, we learn about the automation of the Indiana welfare system. This section depicts several stories of Hoosiers who were directly impacted by the loss of their promised benefits after a state-implemented algorithm automatically kicked thousands of people out of their system. This is the most viscerally upsetting chapter, as you would expect from stories of people who lose their Medicaid unexpectedly. This level of despair is relatively toned down once you get past this chapter.
Next, Dr. Eubanks discusses the system by which the City of Los Angeles prioritizes specific people for homelessness services. As in most cities, LA has more people that need Section 8 housing than landlords that are willing to take Section 8 tenants. Her central thesis is to emphasize that collecting highly personalized data from people that you have no realistic intention to help is a violation of both the central tenets of informed consent and common decency.
The final chapter is about an algorithm used by the CPS department in Pittsburgh (they don't actually use the CPS acronym). As you might expect from an algorithm that predicts the probability that somebody commits child abuse or neglect before it happens, it's fundamentally produces high error rates based on flawed assumptions within the data, and Dr. Eubanks picks apart these issues in detail. As a data scientist, if this system is still used as described, the fact that actual people use this algorithm to make actual decisions is pretty appalling. Dr. Eubanks gives examples of child neglect due to poverty, but does not describe any examples of abuse.
I read this at the recommendation of a fellow data scientist, and I would recommend this to you if you do a "big data" sort of job, or if you come in contact with algorithms that promise the moon in your day-to-day life. I would also recommend this if you use any kind of welfare or government services, especially if your state or country uses these sorts of automated systems.
First, we learn about the automation of the Indiana welfare system. This section depicts several stories of Hoosiers who were directly impacted by the loss of their promised benefits after a state-implemented algorithm automatically kicked thousands of people out of their system. This is the most viscerally upsetting chapter, as you would expect from stories of people who lose their Medicaid unexpectedly. This level of despair is relatively toned down once you get past this chapter.
Next, Dr. Eubanks discusses the system by which the City of Los Angeles prioritizes specific people for homelessness services. As in most cities, LA has more people that need Section 8 housing than landlords that are willing to take Section 8 tenants. Her central thesis is to emphasize that collecting highly personalized data from people that you have no realistic intention to help is a violation of both the central tenets of informed consent and common decency.
The final chapter is about an algorithm used by the CPS department in Pittsburgh (they don't actually use the CPS acronym). As you might expect from an algorithm that predicts the probability that somebody commits child abuse or neglect before it happens, it's fundamentally produces high error rates based on flawed assumptions within the data, and Dr. Eubanks picks apart these issues in detail. As a data scientist, if this system is still used as described, the fact that actual people use this algorithm to make actual decisions is pretty appalling. Dr. Eubanks gives examples of child neglect due to poverty, but does not describe any examples of abuse.
I read this at the recommendation of a fellow data scientist, and I would recommend this to you if you do a "big data" sort of job, or if you come in contact with algorithms that promise the moon in your day-to-day life. I would also recommend this if you use any kind of welfare or government services, especially if your state or country uses these sorts of automated systems.
Content Warnings
Graphic: Ableism and Medical trauma
Minor: Terminal illness and Police brutality
More...