Unintended Consequences

Unintended consequences of AI

While AI and advanced analytics offer many positive benefits, they can lead to significant unintended (or maliciously intended) consequences for individuals, organizations, and society.

Individuals

Physical safety

Digital safety

Financial health

Equity and fair treatment

Organizations

Financial performance

Nonfinancial performance

Legal and compliance

Reputational integrity

Society

National security

Economic stability

Political stability

Labor-market stability

Infrastructure integrity

1/13

Individuals

Physical safety

• Autonomous-vehicle malfunction leads to injury or death • Overreliance on inadequate equipment predictive- maintenance decisions leads to worker injury • Machine-learning models misdiagnose medical conditions

3/13

Digital safety

• Distortion of individual data/information leads to digital libel or defamation

4/13

Financial health

• Poor financial recommendations result in mismanagement of consumer or employee funds • Machine-driven, sophisticated phishing steals and exploits financial information

5/13

Equity and fair treatment

• Underwriting model inadvertently discriminates based on race, rejecting minority customers from acquiring mortgages • Lending algorithm takes into account social-media connections, giving better rates to people who have perceived “higher quality” networks and penalizing those who don’t

2/13

Privacy and reputation

• Private data used without consumers’ consent • Personally identifiable information (PII) data are not securely stored, resulting in data breach and downstream individual implications

6/13

Financial performance

• Trading algorithms unable to correctly adapt to new circumstances (eg, similar to a flash crash) lead to sudden financial losses • Organization makes adverse pricing decisions that materially misgauge consumer price elasticity, leading to poor production decisions

Organizations

7/13

Nonfinancial performance

• Hiring and promotion use complex algorithms that unintentionally lead to nondiverse workforce or unintended behavior • Suboptimal estimates of funds and resources required during different natural disasters/emergencies result in inadequate preparation

8/13

Legal and compliance

• Unintended discrimination embedded into lending decisions results in litigation • Disclosure of protected consumer healthcare data

9/13

Reputational integrity

• Lack of clarity regarding consumer data-privacy settings causes social backlash • Advertising algorithm utilizing invasive PII (or other personal information) causes public to view company as intrusive/dishonest

10/13

National security

11/13

Economic stability

• Automated trading algorithms increase volatility in financial markets • Algorithms create instability in currency markets, resulting in decreased trade • Black-box financial instruments lead to unintended systemic risk

12/13

Political stability

• Manipulation of national institutional processes (eg, elections, appointments) through misrepresentation of information and false messaging

13/14

Labor-market stability

• Mass intelligent machine-led automation leads to instability of labor market • As medical diagnoses are automated, medical workforce de-skills (ie, health professionals lose expertise), creating problems when unforeseen circumstances create need for nonautomated doctors

13/13

Infrastructure integrity

• Risk concentration materially affects societal infrastructure as more processes and decisions become interconnected (eg, disabling power, water supplies, communications) • Intelligent systems lead to overuse/misuse of infrastructure (eg, GPS routes cars through side streets, causing unprecedented traffic in residential areas)

Individuals

Organizations

Society

• Actors with malicious intent coopt AI-enabled products (eg, weaponry, drones, cybertools) and use for illegal activity • Data breaches of sensitive data expose key military vulnerabilities/technical secrets

Privacy and reputation