Algorithmic impact assessment of the predictive system for risk of homelessness for the Allegheny County

Case Studies

Algorithmic Impact Assessment of the Predictive System for Risk of Homelessness for the Allegheny County

recommender systems

Eticas audited the algorithmic system that predicts homelessness risk in Allegheny County, Pennsylvania, USA. The model took into consideration 964 variables coming from different departments and regulatory bodies of the county (demographic and mental health variables, justice, prison, hospital visits and substance use registers, amongst others).

This audit was focused on determining the algorithmic model accuracy, as well as evaluating different metrics to be able to measure system bias regarding protected groups and sociodemographic discrimination such as age, race, gender, and disability.

The audit revealed:

Overall accuracy of the model

Eticas discovered the predictive model was accurate, except for the 0–17 age group which was heavily underrepresented in the training data used to build the model. For this reason, the County was advised to handle this group separately by a business rule, and not by the model.

Algorithmic discrimination

Throughout the audit, the model was evaluated to determine if the algorithm and the risk score it produced introduced a large disadvantage for potentially already disadvantaged groups, to which Eticas did not find such disadvantage or discrimination for the following reasons: Disability, age, race, and property-based discrimination.

There were no clear indicators of discrimination based on gender and veteran status, but some inconsistencies suggest that these factors’ performance should closely monitored.

Recommendations

Re-modeling: Due to the potential change in composition of the homeless population, Eticas recommended that DHS conduct re-modeling at least once a year with new data. Additionally, Eticas recommended documenting carefully the training dataset and the modeling process through model cards, and making this documentation public, allowing researchers access to microdata that allows them, with appropriate data sharing agreements in place, to examine the models closely.
Explainability: Establishing a clear communication strategy with stakeholders about the tool, its development, implementation, and impact.
Desirability: To improve acceptance of the tool by workers and clients, Eticas recommended that DHS consider conducting formal training for frontline workers who will use the Allegheny Housing Assessment (AHA) tool.

Allegheny County Department of Human Services’ Response to Eticas’ report

The county committed to make a series of changes to enhance the accuracy of the system and the protection of its stakeholders, including but not limited to:

Closely monitor the performance of the model for women and single veterans as there is a potential disadvantage for these groups.
Regular re-modelling of the Predictive System for Risk of Homelessness.
Regular quality assurance monitoring.
Designing a clear communication strategy with stakeholders before, during and after implementation of the AHA.
Providing multiple trainings for all frontline workers involved in the homeless coordinated entry system.