June 2016

Study Finds Race-related Errors in COMPAS Recidivism Algorithms

A recent study by ProPublica published May 23, 2016, examined one of the commercially-available risk assessment tools – the Correctional Offender Management Profiling for Alternative Sanctions (COMPAS), developed by Northpointe, Inc. – for predictive accuracy and racial bias in pretrial application.  The researchers focused upon two of the COMPAS scores, the risk of recidivism and the risk of violence, for the study, and did not use the third score, the risk of failure to appear.

The study utilized data from Broward County, Florida, covering a two-year period from 2013 to 2014, and for subsequent offense data through April 1, 2016.  Traffic tickets and some municipal ordinance violations and failures to appear in court were excluded from the recidivism definition; recidivism was defined as those criminal offenses resulting in a jail booking and that occurred after the initial COMPAS testing.  ‘Violent recidivism’ included the offenses of murder, manslaughter, forcible rape, robbery and aggravated assault.  The racial classifications used were for defendants identified as black, white, Hispanic, Asian and Native American in the records.

Looking at the records of 6,172 defendants who did not recidivate, the researchers found that the scores for white defendants were skewed towards lower-risk categories, while the scores for black defendants were evenly distributed.  Age was the strongest predictor of a high score (young defendants were 6.4 times more likely to be assessed a higher score than middle-age defendants).  However, the study found that black defendants were twice as likely to be misclassified as higher risk of general recidivism than were whites, and blacks were 77% more likely to be classified at higher risk of violent recidivism than were whites.  Whites were almost twice more likely to receive lower-risk predictive scores than were blacks.  Blacks who did score higher were slightly more likely to recidivate than whites (63% to 59%).

The researchers found that the predictive accuracy of the COMPAS system for general recidivism was between 63.6% and 66.4% (depending upon which particular methodology was used), and was 65.1% for violent recidivism.  The study noted that all of the percentages were below what Northpointe described as a threshold for reliability: “A rule of thumb according to several recent articles is that AUCs [“area under the curve”] of .70 or above typically indicate satisfactory predictive accuracy, and measures between .60 and .70 suggest low to moderate predictive accuracy.”

COMPAS scoring predictive accuracy was consistent between the races, with a finding of 62.5% for whites and 62.3% for blacks.

Sources:  Jeff Larson, Surya Mattu, Lauren Kirchner and Julia Angwin, “How We Analyzed the COMPAS recidivism Algorithm,” propulica.org, May 23, 2016: https://www.propublica.org/article/how-we-analyzed-the-compas-recidivism-algorithm; Julia Angwin, Jeff Larson, Surya Mattu and Lauren Kirchner, “Machine Bias: There’s software used across the country to predict future criminals. And it’s biased against blacks,” ProPublica, May 23, 2016: https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing; Link to Northpointe, Inc., “Research Synthesis Reliability and Validity of COMPAS,” September, 2007: http://www.northpointeinc.com/files/research_documents/reliability_validity.pdf

by Neil Leithauser
Associate Editor