A Comment on Wisconsin v Loomis
Facts of the Case:
The State contends that Loomis was the driver in a drive-by shooting. It charged him with five counts, all as a repeater: (1) First-degree recklessly endangering safety (PTAC); (2) Attempting to flee or elude a traffic officer (PTAC); (3) Operating a motor vehicle without the owner’s consent; (4) Possession of a firearm by a felon (PTAC); (5) Possession of a short-barreled shotgun or rifle (PTAC). Loomis denies involvement in the drive-by shooting. He waived his right to trial and entered a guilty plea to only two of the less severe charges, attempting to flee a traffic officer and operating a motor vehicle without the owner’s consent. The plea agreement stated that the other counts would be dismissed but read in. After accepting Loomis’s plea, the circuit court ordered a presentence investigation. The Presentence Investigation Report (“PSI”) included an attached COMPAS risk assessment. Loomis’s COMPAS risk scores indicated that he presented a high risk of recidivism on all three bar charts. His PSI included a description of how the COMPAS risk assessment should be used and cautioned against its misuse, instructing that it is to be used to identify offenders who could benefit from interventions and to target risk factors that should be addressed during supervision. The PSI also cautions that a COMPAS risk assessment should not be used to determine the severity of a sentence or whether an offender is incarcerated. At sentencing, the State argued that the circuit court should use the COMPAS report when determining an appropriate sentence. Ultimately, the circuit court referenced the COMPAS risk score along with other sentencing factors in ruling out probation. In addition to the COMPAS assessment, the circuit court considered the read-in charges at sentencing. Although a review of the transcript of the plea hearing reveals miscommunications and uncertainty about the consequences of a dismissed but read-in offense, the circuit court ultimately quoted directly from a then-recent decision of this court explaining the nature of such a read-in offense. It explained to Loomis that a circuit court can consider the read-in offense at sentencing and that such consideration could increase a defendant’s sentence. This Loomis said he understood. The plea questionnaire/waiver of rights form stated that the maximum penalty Loomis faced for both charges was seventeen years and six months imprisonment. The court sentenced him within the maximum on the two charges for which he entered a plea. Loomis filed a motion for post-conviction relief requesting a new sentencing hearing. He argued that the circuit court’s consideration of the COMPAS risk assessment at sentencing violated his due process rights. Loomis further asserted that the circuit court erroneously exercised its discretion by improperly assuming that the factual bases for the read-in charges were true. The circuit court held two hearings on the post-conviction motion. At the second hearing, the circuit court addressed the due process issues. The defendant offered the testimony of an expert witness, Dr. David Thompson, regarding the use at sentencing of a COMPAS risk assessment. Dr. Thompson opined that a COMPAS risk assessment should not be used for decisions regarding incarceration because a COMPAS risk assessment was not designed for such use. According to Dr. Thompson, a circuit court’s consideration at sentencing of the risk assessment portions of COMPAS runs a “tremendous risk of over estimating an individual’s risk and… mistakenly sentencing them or basing their sentence on factors that may not apply…” Dr. Thompson further testified that sentencing courts have very little information about how a COMPAS assessment analyzes the risk. In denying the post-conviction motion, the circuit court explained that it used the COMPAS risk assessment to corroborate its findings and that it would have imposed the same sentence regardless of whether it considered the COMPAS risk scores. Loomis appealed and the court of appeals certified the appeal to the Supreme Court of Wisconsin.
My review of the Case:
The Supreme Court’s affirmation of the order of the Circuit Court was persuasively justified in this case. There is some cogency to maintaining that reliance on COMPAS should not be challenged (at least on grounds of “due process” as initiated by the defendant). My reasons: One, the Circuit Court’s consideration of the charges (which are independent of COMPAS), actually employed and recognized the appropriate legal standards (especially in exercising its power of discretion). Two, the COMPAS risk assessment scores were not relied on by the court exclusively. They were supported by other independent factors, which were not seminal to the decision (for example on whether Loomis could be supervised safely in the community or not). In fact, regardless of the COMPAS risk scores, Loomis would still have obtained the same sentence. This is supported by the court’s demonstration of awareness on the limitations of COMPAS, and how they were admonished by precedence. Three, despite establishing COMPAS’s constitutionality, the Court went further to place numerous restrictions on its use, including mandating an elaborate, five-part warning on the Pre-Sentence Investigation (PSI) (highlighting the algorithm’s controlled utility). This is fair enough caution and justification for the algorithm: I agree with the Court that if used properly, COMPAS may not violate a defendant’s right to due process.
However, due to limitations on the scope of the dataset, COMPAS’s implicit bias, lack of cross-validation and misuse, (amongst other factors enumerated below), the consideration of COMPAS is laden with discrepancies.
First, the COMPAS risk assessment does not predict the specific likelihood that an individual offender will reoffend. It only provides prediction based on a comparison of information about the individual to a similar data-group. Data is “king” here. It is critical for the prediction that the COMPAS will provide. An unrepresentative dataset will definitely affect the outcome. Such that, if African Americans are poorly represented (not just in few samples but with biased samples) in the dataset used to train the COMPAS algorithm (of which there are claims that posits thus), then the PSI may reproduce or reaffirm such patterns of bias.
Second, since the COMPAS risk assessment on Loomis was based on specific information that was fed into it (i.e. data from his criminal file record and information from the interview with him), it leaves out other valuable sources of information (which COMPAS can’t assess and may be influential to the outcome of the report) such as socioeconomic and sociocultural influences that affect/affected the defendant). In fact, by using the interview transcripts of the defendant as one of the two sources for evaluation, the algorithm is perhaps not given an opportunity to consider sensitivities such as length (and conditions) of the interview (e.g. if the defendant was interviewed for extremely long hours, or was in a state of fear or torture); the articulation level of the defendant, mental health status and behavioral control during the interview (including emotional and sociological factors such as the expression of remorse or regret etc.).
Third, central to Loomis claim is the AI challenge of “explainability”. This is evident. The inability of COMPAS to fully explain how it analyzes the risk, poses an accountability problem too. Beyond the need for the defendant-appellant to understand how COMPAS arrived at an assessment of him, even the judges (as end-users too), need to have an understanding to be able to write explainable rulings – in non-technical terms – for the benefit of the public. It may be hard for an AI to fully report, explain, and justify its algorithmic decision-making process. This is because machine learning applies huge volumes of data to non-linear models which are sometimes said to be “beyond the realm of human comprehension.” But, any decision-making algorithm with a potential for significant human impact should be explainable, especially for this case where the consequence is incarceration for more than seventeen years. Explainable AI is still an ethical challenge, but as with this case; it is crucial for establishing transparency and discouraging judicial skepticism. Also, public justice systems should never suffer an accountability-deficit simply because a private capital want to protect their intellectual property interest.
And fourth, Since COMPAS uses a person’s criminal history and some dynamic variables (such as a person’s associates or previous substance abuse), this prods a concern about the sociology of the crime and how we engage justice in the Fourth Industrial Revolution. Should current sentencing be influenced by people’s past (even after they have “served time” or become “clean”)? What portions of history (or not) should an algorithm evaluate when determining new sentences (and why)? With concerns about the dangers of “digital permanence”, does criminal historicity not threaten the “right to be forgotten” and promote recidivism? I think that the value-add of COMPAS could outweigh its limitations as it helps to (perhaps better, and definitely faster) assess low-risk prison-bound offenders to a non-prison alternative; measures whether an offender is a threat to public safety (or not) etc. However, the proprietary nature of COMPAS which prevents defendants from challenging its validity is unquestionably problematic. As an evidence-based assessment tool, it should be subject to challenge. COMPAS and the law governing same should be subject to trial. They are not immune from error. Challenging them (over and over again) gives us an opportunity to test their relevance and possibly improve criminal justice outcomes with AI. With the data emanating from cases in court, we can contribute to improving the iterations of machine learning algorithms – to better represent the relationship between data and justice.
We need to also reframe how (and where) we want to engage AI – in both the substantive and procedural components of legal practice. As a prism through which our own prejudices, inequities and ignorance are reflected, we need to recognize that tools like COMPAS are progressive and reliable for meeting our socio-criminogenic needs, but, are far from being neutral or unobjectionable (therefore must be cautioned). An idea: How about making the use of COMPAS optional and at the instance of the defendant (and not the State)?