Ramesh Srivatsava Arunachalam
Introduction
The use of artificial intelligence (AI) and machine learning models for making lending decisions is increasing rapidly. These models can analyze large amounts of data to make faster and more accurate credit assessments. However, there are also risks around fairness, bias, and explainability of the decisions made by these black-box algorithms. This is especially concerning for lending decisions impacting financially vulnerable populations with low incomes.
As AI continues to expand in the lending sector, it is critical that appropriate governance frameworks are established to validate that these models are, in fact, fair, unbiased, and explainable in their decision-making. This will build trust amongst applicants and uphold ethical AI standards. Regulatory bodies are still playing catch-up in this fast-moving domain. In the meantime, lending companies themselves need rigorous in-house validation procedures.
This article, based on my experiences globally, provides a comprehensive overview of the issues and methodologies around validating AI lending models, with a specific focus on low-income populations. It will cover the following key aspects:
- Fairness and bias considerations for AI lending models
- Testing lending dataset bias
- Establishing explainability and transparency requirements
- Accuracy metrics and outlier detection
- Monitoring model drift over time
- A/B testing against existing systems
- Simulating decision-making on synthetic applicant profiles
- Consulting with consumer advocacy groups
- Implementing human-in-the-loop approval processes
- Analysis of model interpretation methods
- Case studies of validation frameworks in practice
- Limitations and open challenges
Fairness and Bias Considerations
Fairness is a critical requirement for any AI system making impactful decisions about peoples’ lives, such as access to credit. However, the standard machine learning paradigms aim to maximize accuracy without any inherent concept of what is ethical or fair. It is up to lending companies to ensure additional constraints around fairness, diversity, and inclusion are programmed into the modeling process.
This requires grappling with some complex questions around what fairness actually means in this context. There are various mathematical definitions of algorithmic fairness, with different trade-offs and limitations. Some key criteria relevant for lending include:
Statistical parity: Approval rates/loan terms should be equal across different groups based on ethnicity, gender, age etc. However, this may not account for genuine risk differences.
Individual fairness: Similar individuals should receive similar decisions. But the metrics for similarity are subjective.
Counterfactual fairness: Applicants from different groups but identical credentials/risk profiles should get the same decisions. However, relevant risk criteria could be unobservable.
There are also distinctions between group fairness across segments of applicants and fairness towards individuals. AI models can seem fair on average but still present issues for some individuals.
The first step is testing for sources of unfair bias throughout the modeling pipeline, including:
- Biased lending datasets for training models
- Proxy discrimination through facially neutral variables
- Overreliance on narrow credit scoring
- Feedback loops encoding historical inequities
- Poor model interpretability hiding unfairness
- Incorrect assumption of model neutrality
Models with intrinsic biases or proxy discrimination can wrongly associate certain groups with higher default risk. Religious names, ethnic minorities, low-income neighbourhoods, public assistance recipients and even consumers with little credit history tend to get unfairly profiled, continuing financial exclusion.
Beyond this, biased models also present financial and operational risks for lenders through inaccurate risk pricing, lower approval rates, and lack of portfolio diversity.
So what determines model fairness exactly? It is a complex, multi-dimensional challenge with no universally agreed-upon solution. Context also matters - fairness for mortgage lending may require different standards than pay-day loans or non-profit financing for low-income groups.
At a minimum, AI systems should not discriminate on the basis of ethnicity, gender, religion etc. But eliminating bias needs nuanced and thoughtful approaches tailored to the lender’s goals. I will cover some leading methodologies later in the article.
Testing Lending Dataset Bias
For supervised machine learning algorithms, historical lending data is used to train predictive models on past examples of good and bad customers. But if the training dataset itself is biased, those unfair biases become encoded within the model’s logic and get amplified in the real world.
“Rubbish in, rubbish out” is a genuine concern, especially given unexamined assumptions that big data accurately reflects ground realities. Legacy lending data built up over decades may advantage Groups with easier historical access to finance while excluding poorer people and neighbourhoods.
Dataset bias testing methodologies are critical to avoid perpetuating past inequities:
1) Data profiling on labels like ethnicity, gender, income, geography etc. to quantify dataset representation across groups and detect imbalances. Are lower income segments adequately captured?
2) Checking correlation of input variables with protected group attributes. For example, does the prevalence of certain postcodes strongly correlate with certain ethnic profiles. Such indirect proxies can enable discrimination through the backdoor.
3) Testing model performance specifically for previously under-represented groups, not just overall accuracy. Break out approval rates, default rates, average loan size etc. by income segment, geography, race etc.
4) Comparing aggregate lending outcomes of the model versus real world. Does overall lending distribution match census data demographics and financial inclusion benchmarks?
5) Testing for “reject inference” where unexplained application declines reduce certain groups in the training data itself. This requires comparing the input dataset with the raw application data.
Overall, the training data may require additional sourcing, cleaning, relabelling and augmentation to correct sampling biases before it can serve as the foundation for fair lending models.
Establishing Explainability
Interpretability refers to the ability to explain in human terms why an AI model makes certain decisions. This is especially important for lending decisions determining financial outcomes for consumers. Both regulators and applicants expect transparency on the logic and risk criteria behind approvals or denials.
However, techniques like deep neural networks are complex black-box systems with billions of parameterized interactions. Their working cannot be distilled into simple ‘if-then’ rules. This inherent lack of explainability poses challenges on multiple fronts:
1) Applicants cannot contest unfair or erroneous decisions if the logic is inscrutable. Lack of recourse can exacerbate mistrust and helplessness.
2) Lenders themselves may not fully understand model failures or irregularities. Undetected biases can systematically distort certain decisions.
3) Debugging production issues with black-box models can be extremely challenging. Performance monitoring depends wholly on observed outputs.
4) Model governance becomes difficult when the processing is opaque even to creators. Harder to ascertain fairness, detect manipulation etc.
5) Regulations increasingly demand explainability for credit decision processes impacting consumers. This includes rights to credit denial information.
So there is certainly a business and compliance incentive also for lending companies to make their models more interpretable, in addition to ethics considerations.
Various techniques are emerging to improve explainability of complex AI models:
1) Simplified models – Using linear regression, decision trees and traditional statistical techniques instead of neural networks prioritizes transparency, at the cost of some accuracy.
2) Local explanations - Though the full model is opaque, individual decisions can be explained by approximations like LIME based on local deviations.
3) Example-based explanations – Previously predicted similar applicants and their outcomes can provide intuition on new decisions.
4) Model distillation – Complex models are approximated into more interpretable versions which mimic their workings.
5) Attention layers – Neural networks can be modified to also output attention heatmaps highlighting influential variables behind each choice.
6) Explanation by example – Users provide hypothetical applicant profiles alongside desired decisions to understand model logic.
7) Explanation engines – Dedicated software condenses complex models into more intuitive formats like decision trees, flowcharts, rule lists etc. tailored to different stakeholders.
The appropriate level of explainability and interfaces can vary based on the target audience - applicants, business users, regulators etc. But priority for transparent design needs to be baked into the model development process itself.
Accuracy Metrics and Outlier Detection
While fairness and explainability considerations are critical, AI systems deployed in the real world still need to function accurately and reliably. Model performance metrics beyond raw accuracy scores require thoughtful selection, with emphasis on outliers and errors.
Some key metrics for lending models include:
- Overall application approval rates
- Relative group-wise approval rates
- Sub-segment analysis – low income, minority areas etc.
- Approval rate for thin-file applicants
- Default rates/credit losses
- Group default rate differences
- ROC-AUC curve analysis
- Confusion matrix for decision types
- False positive/false negative ratios
- Prediction error rates
- Mean average error for predicted risk rating
- R2 – variance explained for risk rating
Many metrics can indicate potential biases or performance issues hidden behind a supposedly high accuracy score. Wide confidence intervals between groups, highly skewed errors against minorities or thin-file applicants etc. require deeper investigation.
Analyzing perplexing model errors and outliers is also critical:
- Unexpected defaults from very high-risk applicants
- Review of declined applicants with stellar credentials
- Cluster analysis to detect unusual sub-segments
In deployment, models run on 100% new data which may differ from the training population. Continual monitoring can detect examples that are statistically anomalous compared to previously seen cases. Techniques like density-based outlier algorithms can help identify exceptionally high risk profiles the model may not be equipped to score reliably.
The accuracy bar for lending models is also higher than other AI use cases because errors have significant financial and reputational consequences. Beyond technical metrics, decision quality ultimately depends on user feedback in the field. Hence the next section on A/B testing with live applicants.
A/B Testing Against Existing Systems
Machine learning papers sometimes present unbelievable results on benchmark datasets. But experimental performance often fails to translate into the real world. Models trained on sliver subsets of available data within closed environments frequently struggle when exposed to messy, high dimensional production data.
Before AI systems are entrusted with independent lending decisions at scale, extensive A/B testing against existing systems is prudent. Running superiority testing in tandem on incoming applications provides assurance on multiple fronts - not just better decisions but also detecting potential model degradation.
Some guidelines on comparative testing:
1) Sample population - Mix of thin-file, low/moderate income, minority applicants
2) Decision types - Both accept and reject decisions
3) Duration - Multi-month testing on latest applicant data
4) Metrics - Approval rates, risk rating accuracy, default rates, group fairness
5) Incremental rollout - slowly divert traffic while monitoring outlier decisions
6) Interpretability tools - Compare group-wise importance of features, attribution maps
Essentially, the AI system should demonstrably perform as well or better than previous systems on all aspects, not just overall accuracy. Evidence of unfair bias or unexplainable errors during A/B testing warrants re-evaluation even for high performing models.
And rather than a one-time validation, continuous monitoring against existing systems even post-deployment provides a reliable safeguard against production issues. The next section covers additional ways to simulate model decisions.
Simulating Decisions on Synthetic Applicant Profiles
While A/B testing reveals model performance on actual applicants, internally generated synthetic applicant data offers useful supplementary validation:
1) Simulate group-wise discrimination - Same risk profile with different names/geographies should get same decisions
2) Edge case testing - Good and bad synthetic applicants clearly within approval thresholds
3) Vary attributes like income, loan amount within set bands to check for cliffs/thresholds
4) Ensure predicted rating and decisions change smoothly not sharply with gradual variable changes
5) Test sensitivity to missing variables, data errors, typos, numeric/text variants etc.
The ability to internally simulate applicant data with controlled differentiation enables scenario testing difficult during live A/B trials. It provides greater visibility into potentially problematic edge configurations related to bias vulnerabilities or rating integrity.
However, synthetic data itself carries risks around inaccurate assumptions, population representation issues and over-tuning models on fake applicant profiles. So it should complement not replace comparative testing on incoming real applications.
The next few sections discuss additional perspectives to incorporate into the validation process...
Consulting with Consumer Advocacy Groups
Lending institutions make independent decisions on credit model validation based on internal checks and priorities. However, greater engagement with consumer advocacy groups as external stakeholders can provide crucial perspectives on fairness and potential issues.
1) Help define fairness – Advocacy groups directly engage affected communities on the ground to better represent their concerns and goals around algorithmic systems.
2) Ensure dataset diversity – They can examine training data characteristics to see if lower income consumer data is adequately reflected.
3) Detect exclusion criteria – Comparing decline reasons and credit criteria can identify unacceptable or illegal reasons for financial exclusion.
4) Provide balanced applicant samples and user research panels – Not just customers familiar and comfortable with the banking system.
5) Incorporate feedback on transparency, recourse and communications – Essential for trust and procedural fairness.
6) Guide situation testing for bias – Provide inputs on sensitive use cases, customer scanning systems etc. to check for embedded discriminations.
7) Suggest under-represented applicant profiles – For synthetic testing on group specific cases that may be systemically disadvantaged.
Often, algorithm harms disproportionately affect groups with lower visibility and agency. Hence it is vital lending institutions consciously seek inputs from consumer rights advocates oriented towards public interest. The validation process should incorporate their perspectives right from the problem formulation stage itself.
Implementing Human-in-the-Loop Approval Processes
While AI systems demonstrate computational accuracy on historical datasets, can they evaluate risk and make nuanced decisions contextualized for individual applicant needs? Do inputs like personal letters adequately inform creditworthiness for thinner files? Can questions be answered to clarify data gaps? Essentially is automated adjudication fair and ethical?
The solution lies in combining algorithmic insights with human intelligence and empathy. AI scores and rules provide decision consistency, speed and scale while people integrate qualitative judgement and discretion to remedy inevitable model limitations.
Some principles on establishing human-in-the-loop frameworks:
1) Rules-based system separates applications into rules-based approvals, declinations and further review buckets.
2) Risk-based prioritization surfaces applicants closer to the approval threshold for manual review.
3) Human reviewers contextualize scores with supplemental data like bank statements, correspondence etc. overlooked in automated processing.
4) Applicants can answer questions, correct data errors, provide explanations etc. to reviewers.
5) Outcomes analyzed to improve model training data and rules. Decline reasons also help focus manual review.
The dual review structure maximizes computational efficiency for the bulk of applications while allowing bespoke assessment when beneficial. Over time, the system can potentially transition fully automated with enough training data demonstrating generalizability, safety and fairness even for outlier cases.
Analysis of Model Interpretation Methods
As discussed earlier, interpretation tools are indispensable for evaluating model fairness and trust. But these proxy explanation techniques also come with their own limitations relative to true working complexity. Their generated explanations reveal an approximation of the key logic rather than an exact feature-to-outcome mapping.
So it is critical lending institutions rigorously assess explanation methods themselves before fully relying on them to detect biases – both when selecting software and interfacing with model owners. Different classes of interpretability techniques and their pros-cons merit examination:
1) Local explanations like LIME – Interpret model predictions for individual applicants by intelligently perturbing inputs and observing impact. However, edge cases may be challenging to evaluate comprehensively via perturbations.
2) Global explanation methods like Shapley values – Summarize average model behavior across the applicant population instead of case-by-case. But may hide unfairness towards subgroups.
3) Example-based approaches – Elucidate model logic through user-provided applicant examples and corresponding desired decisions. Limited by example set coverage and still model approximations.
4) Model distillation algorithms – Compress inscrutable models into inherently interpretable versions mimicking their decisions. But simpler models bound to lose fidelity.
5) Attention layers in neural networks – Quantify contribution of input variables towards model output. But attribution values themselves result from complex intermediate computations.
A combination of global and local methods analyzingfeature importance, decision trees, textual/visual summaries etc. provides well-rounded explanatory depth. But simplified explanatory interfaces should come with necessary qualification of inherent limitations. And ideally, models should be transparent by design rather than post-hoc explanations tacked on opaque systems.
The next section illustrates how some of these validation principles are being implemented by lending companies and fintech firms developing credit risk models.
Case Studies
While AI validation is still an evolving, individual organizations are operationalizing combinations of the methodologies referenced above within their model development pipelines. Here we analyze three case studies from different segments of lending industry leveraging automation:
Regional Bank – Risk rating models for consumer loans
- A/B tested models to ensure parity with existing credit procedures on approval rates, defaults, group fairness metrics
- Further testing simulating biased decisions and edge cases on synthetically generated consumer profiles based on census data
- Interpretability module outputs decision trees and Shapley values highlighting key variables and their contribution to the risk rating prediction
- Declined applicants can request simplified model re-evaluation along with additional clarification or data
Payday Lender – Automated affordability and anti-fraud checks
- Compared input dataset to raw applicant pool to ascertain and adjust for potential sampling biases causing under-representation
- Calibrated decision thresholds and graded cutoff relaxations for higher risk cases to balance default likelihood and financial inclusion goals
- Added manual review step for moderate risk applicants to incorporate additional signals like bank statement checks, income stability assessment etc. before automated approval/decline
- Consumer advocacy panels with specialized focus on unbanked and underbanked communities provide consultation across process spanning data, decisions, communications etc.
Microlender – Custom credit models for informal economy
- Field testing of model with actual applicants across urban slums, villages etc. rather than just historical training dataset
- Worked with nonprofit field officers to get qualitative human judgement and context to complement model decisions
- Feedback loops allow periodic applicant data captured to improve model predictions for thin-file segments
- Local board with community/borrower representatives monitors lending activity for potential disparate impact across social groups
The case studies highlight that holistic validation requires zooming out beyond technical accuracy to focus on decisions experienced by actual people. It necessitates transparency to affected individuals, engagement with consumer advocates, and participative oversight from the ground-up rather than just the top-down.
Limitations and Open Challenges
While this article has extensively covered AI validation techniques for ethical and fair lending, significant open challenges remain from both a technical and governance perspective:
No consensus on universal fairness metrics – Group fairness targets like statistical parity can conflict with individual fairness ideals of similar treatment for similar people. And different cultures and stakeholders define fairness differently based on history, priorities and value judgments.
Hard to combat indirect discrimination – Without awareness of applicants’ legally protected group identities, proxy variables can stand in for ethnicity, disabilities, gender etc. Preventing opaque proxies requires rigorous feature engineering and statistical testing.
Explanation methods have limitations – Simplified local and global interpretation approaches can miss unfairness or provide misleading rationales on model inner workings. Their inherent approximations may instill false confidence.
Difficult to secure sensitive training data – Thin-file segments often correspond to marginalized communities with lower digital footprints. But sourcing adequate and representative data faces privacy barriers given suspicion of exploitation or discrimination.
Concrete recourse still lacking – Just pointing out potentially unfair decisions provides little redressal to impacted applicants. Operationalizing contestability mechanisms and grievance processes poses an added challenge.
Monitoring models in production – Data drifts, new financial products etc. can degrade model performance over time. But detecting accuracy deterioration or bias amplification relies wholly on observed decisions.
Regulations still lag practice – Guidelines on transparency for applicants and oversight bodies to inspect inner working are still largely absent or not enforced for private sector models. Global regulatory harmonization also appears distant.
So while explainability, stakeholder participation and lifecycle validation offer a strong starting point, continued progress requires converging on precise fairness measures, recourse systems, and adaptive governance to respond to emerging issues in a fast-changing industry. The tools exist but practical realization depends wholly on institutional accountability and responsibility.
Conclusion
In conclusion, as AI-based credit models deeply embed within lending decisions and infrastructure, ensuring their fairness, transparency and accountability is imperative. Validation cannot be a one-time paperwork exercise but rather an ongoing governance process spanning the entire model development lifecycle and even post-production.
Hopefully this comprehensive exploration has provided a solid grounding on the key considerations, established methodologies like A/B testing and global explanations as well as practical case studies. Fair lending is a complex multidimensional challenge, but step-by-step progress is certainly viable through collaboration between finance institutions, policy makers and affected communities.
The frameworks and principles referenced above offer guideposts for the journey ahead, even as technology and applications continue maturing. Overall by making values of inclusion and equity central to decisions, AI systems can expand credit access for underserved groups rather than excluding them from opportunities. The mission now is to prudently steward these emerging capabilities towards democratization not marginalization.
