Search ST Engineering

ST Engineering
ST Engineering

December 15, 2020 - By now, many would’ve heard of the firing of Dr Timnit Gebru from Google. She is a leading AI bias and ethics researcher famous for a landmark study that exposed racial and gender biases in facial recognition systems. The incident centered on a research paper that mentioned biases inherent in large-scale language models like BERT, used in many Google products. While we don’t know the whys and wherefores as details are still unfolding, the debacle brings renewed focus to her work and the importance of fairness in AI systems.

Fairness is one of the key pillars of responsible AI, just as privacy, security and transparency/ explainability are. As we extend the use of AI models to more critical industries, they have the potential to be fairer than decisions made from ad-hoc human judgements; but they can also amplify pre-existing biases in models. Creators of AI systems need to actively mitigate bias and ensure fairness to the best of their abilities. Not doing so can have revenue impacts and involve reputational and regulatory risks. This is particularly so as we extend our solutions in Public Safety & Security (PSS), cybersecurity & fraud detection and smart cities etc. to other geographies with different demographics and societal dynamics.

Lack of fairness can impede the adoption of valuable AI
COMPAS, which was an AI tool used by several US states to predict recidivism, was found to have double the rate of false positives for African Americans than for Caucasians. Because this implicit bias was not caught before the system was deployed, many African Americans were incorrectly predicted to re-offend. Some states stopped using COMPAS. However, algorithms like COMPAS have been shown to be much more effective than humans in predicting recidivism. This excellent article from Brookings also explains how when creators actively take steps to mitigate bias in the training dataset and combine it with transparency, risk assessment instruments like COMPAS can be extremely valuable in steering criminal justice decisions. Not managing the implicit bias in the model prevented COMPAS from being fully utilised.

Facial recognition is another contentious technology. Evidence of bias in these systems have been well-flagged. Civil liberties groups have been pushing against the use of such technology due to implicit bias in the system and concerns over privacy and misuse. The cities of SF, Oakland and Boston have all banned government entities from using them. In Sept ’20, the city of Portland even banned corporate use of facial recognition technology in public spaces. Major tech groups like Amazon, IBM and Microsoft have temporarily halted sales of such technology to law enforcement. Facial recognition technology is highly valuable in biometrics, surveillance, fraud protection at ATMs, gated access etc. and isn’t going away. Companies who can imbue a greater level of trust in their facial recognition technology can glean a bigger slice of this market.

These are but some examples of lost revenue opportunities or reputational hits when AI models do not actively mitigate bias and endeavor to ensure fairness.

How does bias get introduced in the first place?

Bias gets introduced into the model via three main avenues:
1) Framing of the problem
Bias can be introduced when one first defines the problem, decides what target variable to use and what cost function to minimise. When predicting creditworthiness, should the goal of the bank’s AI model be to maximise repayment rate of loans or maximise profits? If the latter, the bank might decide to maximise the extension of subprime loans which sets them up for predatory behavior.

2) Biased dataset

Skewed samples
If the historical dataset contains certain bias, the model would replicate these biases. In crime risk assessment tools, if police tended to patrol areas with initially higher crime rates, a crime risk assessment model is likely to record more instances of crime in the same areas.
Unrepresentative training data or class imbalance

In this research by Buolamwini and Gebru (Gender Shades), the higher error rates for women and minorities in facial recognition technologies can likely be traced to unrepresentative training data. The datasets used by two facial analysis benchmarks were overwhelmingly made up of white faces (~80%). Darker-skinned females ended up being the most misclassified group (error rates of up to 34.7%) vs 0.8% for lighter-skinned males.

Proxies
Biases can occur when the model uses a statistical proxy for a protected attribute.
In 2015, Amazon developed a ML tool to automate the screening of resumes for hiring decisions. It found that the model was not vetting candidates for technical jobs in a gender-neutral way. The model had actually taught itself that male candidates were preferable; penalising resumes that included the word “women” in say “women’s chess club”. Even after adjusting for these explicit gender terms, the model still favored candidates who used descriptors like “captured”, “executed”; “aggressive” terms more often found in male engineers’ resumes. Amazon eventually decided to can the use of the hiring algorithm.

In this research paper, the researchers tried to predict the crime rate per community based on the Communities & Crimes dataset (composed of U.S. census, law enforcement and crime data from the FBI). From 122 variables, they removed 32 features explicitly linked to race. Yet they still managed to find a proxy for race consisting of a combination of 58 features out of the remaining 90. This proxy had an association with race of 0.85, while the single feature with the strongest association in the dataset only had an association of 0.73. Checking combinations of features to proxies is very difficult in real life. Fortunately, the paper presented an example of a tool that exists to discover such proxies that are influential in model decisions.

3) Data Preparation
Finally, what features to include or leave out is unique to the problem. Age might be discriminatory in hiring software, but predictive in healthcare models. Debt-to-income ratio might be correlated to race, but influential in predicting creditworthiness; humans-in-the-loop might therefore decide to still include debt-to-income ratios as a feature.

Mitigation Strategies
All that being said, the definition of fairness is not a straightforward one. There really is no standard definition for fairness. Should we extend loans at the same rate to two different groups, regardless of payment risks profiles; or should we extend loans that are proportional to each group’s payback rate? Should we consider the interests of one key under-represented minority group, or all groups? Doing the latter becomes very hard especially if what is fair is defined differently across different people groups. In addition, mathematically, there are also many different ways to account for fairness, some of which are mutually exclusive (introduction to the math). Finally how far do we sacrifice model accuracy to incorporate fairness attributes? Ultimately these are calculated tradeoffs that have to be made by the team.

1. Diverse data science team
This is why having a diverse data science team is important. Members from different backgrounds with different experiences can help detect blind spots in the design and execution of experiments.

2. Red-teaming approach
Similar to red-teaming in cybersecurity, it involves a group of engineers, data scientists and business folks thinking together — of what can go wrong. This could be the same group of people who are involved in the design of a synthetic dataset (one that is broad enough to incorporate edge cases), or who are also involved in detecting drift or spotting predictions that don’t make sense.

3. Establish processes and utilise tools to test and mitigate AI bias
Some tools to help mitigate bias and ethical issues:

  • IBM has an open-source library to detect and mitigate biases in unsupervised learning algorithms called AI Fairness 360. It is currently designed for binary classification models and needs to be extended to multi-class and other ML problems.
  • IBM Watson’s OpenScale performs bias checking and mitigation in real-time
  • Model cards which similar to nutritional labels on food, clarify the intended use of an ML model, its limitations, bias checks and performance evaluation. This was developed by Gebru at Google.
  • Framework for algorithmic auditing
  • Keeping abreast with such research via Alan Turing Institute for Fairness, Transparency and Privacy and Partnership on AI

Ethical and fairness in AI necessitates a case-by-case approach. It is however, increasingly an important pillar of building trustworthy systems. As corporations who build AI systems, we need to be cognizant of ethics and fairness as we conceive ML projects, being aware of contexts that are prone to exacerbating biases and asking the right questions so as to make the appropriate tradeoffs.