Data Ethics: What It Means and What It Takes PDF

QuantumBlack, AI by McKinsey Data ethics: What it means and what it takes Every company must establish its own best practices for managing its data. Here are five pitfalls to avoid based on our conversations...

QuantumBlack, AI by McKinsey Data ethics: What it means and what it takes Every company must establish its own best practices for managing its data. Here are five pitfalls to avoid based on our conversations with experts and early adopters. by Alex Edquist, Liz Grennan, Sian Griffiths, and Kayvaun Rowshankish © Hiroshi Watanabe/Getty Images September 2022  Now more than ever, every company is a data governance models, and collaborating across company. By 2025, individuals and companies disciplines and organizations. This list of potential around the world will produce an estimated challenges and remedies is not exhaustive; our 463 exabytes of data each day,1 compared with research base was relatively small, and leaders less than three exabytes a decade ago.2 could face many other obstacles, beyond our discussion here, to the ethical use of data. But With that in mind, most businesses have begun what’s clear from our research is that data ethics to address the operational aspects of data needs both more and sustained attention from all management—for instance, determining how to members of the C-suite, including the CEO. build and maintain a data lake or how to integrate data scientists and other technology experts into existing teams. Fewer companies have Potential challenges for systematically considered and started to address business leaders the ethical aspects of data management, which There is a dynamic body of literature on data could have broad ramifications and responsibilities. ethics. Just as the methods companies use to If algorithms are trained with biased data sets collect, analyze, and access data are evolving, so or data sets are breached, sold without consent, will definitions of the term itself. In this article, or otherwise mishandled, for instance, companies we define data ethics as data-related practices can incur significant reputational and financial that seek to preserve the trust of users, patients, costs. Board members could even be held consumers, clients, employees, and partners. Most personally liable.3 of the business leaders we spoke to agreed broadly with that definition, but some have tailored it to the So how should companies begin to think about needs of their own sectors or organizations (see ethical data management? What measures can sidebar, “What is data ethics?”). Our conversations they put in place to ensure that they are using with these business leaders also revealed the consumer, patient, HR, facilities, and other forms unintended lapses in data ethics that can happen in of data appropriately across the value chain—from organizations. These include the following: collection to analytics to insights? Thinking that data ethics doesn’t apply to We began to explore these questions by speaking your organization with about a dozen global business leaders and data While privacy and ethical considerations are ethics experts. Through these conversations, we essential whenever companies use data (including learned about some common data management artificial-intelligence and machine-learning traps that leaders and organizations can fall into, applications), they often aren’t top of mind for some despite their best intentions. These traps include executives. In our experience, business leaders are thinking that data ethics does not apply to your not intentionally pushing these thoughts away; it’s organization, that legal and compliance have data often just easier for them to focus on things they ethics covered, and that data scientists have all the can “see”— the tools, technologies, and strategic answers—to say nothing of chasing short-term ROI objectives associated with data management—than at all costs and looking only at the data rather than on the seemingly invisible ways data management their sources. can go wrong. In this article, we explore these traps and suggest In a 2021 McKinsey Global Survey on the state some potential ways to avoid them, such as adopting of AI, for instance, only 27 percent of some 1,000 new standards for data management, rethinking respondents said that their data professionals 1 Jeff Desjardins, “How much data is generated each day?” World Economic Forum, April 17, 2019. 2 IBM Research Blog, “Dimitri Kanevsky translating big data,” blog entry by IBM Research Editorial Staff, March 5, 2013. 3 Leah Rizkallah, “Potential board liability for cybersecurity failures under Caremark law,” CPO Magazine, February 22, 2022. 2 Data ethics: What it means and what it takes What is data ethics? We spoke with about a dozen business be getting something” from a data-based don’t think I ever agreed to this,’ another leaders and data ethics experts. In their transaction, explained an executive at a company might say, ‘On page 41, down in eyes, these are some characteristics of large financial-services company. “If you’re the footnote in the four-point font, you did ethical data use: not solving a problem for a consumer, actually agree to this.’ We never want to be you’ve got to ask yourself why you’re that company.” It preserves data security and protects doing what you’re doing.” The benefit to customer information. The practitioners customers should be straightforward and It is in line with your company’s promises. we spoke with tend to view cybersecurity easy to summarize in a single sentence: In data management, organizations and data privacy as part and parcel of customers might, for instance, get greater must do what they say they will do— data ethics. They believe companies have speed, convenience, value, or savings. or risk losing the trust of customers an ethical responsibility (as well as legal and other key stakeholders. As one obligations) to protect customers’ data, It offers customers some measure of senior executive pointed out, keeping defend against breaches, and ensure that agency. “We don’t want consumers to faith with stakeholders may mean turning personal data are not compromised. be surprised,” one executive told us. “If a down certain contracts if they contradict customer receives an offer and says, ‘I think the organization’s stated data values It offers a clear benefit to both consumers I got this because of how you’re using my and commitments. and companies. “The consumer’s got to data, and that makes me uncomfortable. I actively check for skewed or biased data during line to the C-suite—will need to raise, respond to, data ingestion. Only 17 percent said that their and think through various ethical issues surrounding companies have a dedicated data governance data. Business unit leaders will need to vet their committee that includes risk and legal professionals. data strategies with legal and marketing teams, In that same survey, only 30 percent of respondents for example, to ensure that their strategic and said their companies recognized equity and fairness commercial objectives are in line with customers’ as relevant AI risks. AI-related data risks are only a expectations and with regulatory and legal subset of broader data ethics concerns, of course, requirements for data usage. but these numbers are striking. As executives navigate usage questions, they must Thinking in silos: Legal, compliance, or data acknowledge that although regulatory requirements scientists have data ethics covered and ethical obligations are related, adherence to Companies may believe that just by hiring a few data data ethics goes far beyond the question of what’s scientists, they’ve fulfilled their data management legal. Indeed, companies must often make decisions obligations. The truth is data ethics is everyone’s before the passage of relevant laws. The European domain, not just the province of data scientists or Union’s General Data Protection Regulation (GDPR) of legal and compliance teams. At different times, went into effect only in May 2018, the California employees across the organization—from the front Consumer Privacy Act has been in effect only since Data ethics: What it means and what it takes 3  January 2020, and federal privacy law is only now information? Such due diligence is key: one pending in the US Congress. Years before these alternative data provider was charged with and other statutes and regulations were put in place, securities fraud for misrepresenting to trading firms leaders had to set the terms for their organizations’ how its data were derived. In that case, companies use of data—just as they currently make decisions had provided confidential information about the about matters that will be regulated in years to come. performance of their apps to the data vendor, which did not aggregate and anonymize the data as Laws can show executives what they can do. But a promised. Ultimately, the vendor had to settle with comprehensive data ethics framework can guide the US Securities and Exchange Commission.4 executives on whether they should, say, pursue a certain commercial strategy and, if so, how they should go about it. One senior executive we spoke A few important building blocks with put the data management task for executives These data management challenges are common— plainly: “The bar here is not regulation. The bar here and they are by no means the only ones. As is setting an expectation with consumers and then organizations generate more data, adopt new tools meeting that expectation—and doing it in a way and technologies to collect and analyze data, and that’s additive to your brand.” find new ways to apply insights from data, new privacy and ethical challenges and complications Chasing short-term ROI will inevitably emerge. Organizations must Prompted by economic volatility, aggressive experiment with ways to build fault-tolerant data innovation in some industries, and other disruptive management programs. These seven data-related business trends, executives and other employees principles, drawn from our research, may provide a may be tempted to make unethical data choices— helpful starting point. for instance, inappropriately sharing confidential information because it is useful—to chase short- Set company-specific rules for data usage term profits. Boards increasingly want more Leaders in the business units, functional areas, and standards for the use of consumer and business legal and compliance teams must come together to data, but the short-term financial pressures remain. create a data usage framework for employees—a As one tech company president explained: “It’s framework that reflects a shared vision and mission tempting to collect as much data as possible and to for the company’s use of data. As a start, the CEO use as much data as possible. Because at the end and other C-suite leaders must also be involved of the day, my board cares about whether I deliver in defining data rules that give employees a clear growth and EBITDA.… If my chief marketing officer sense of the company’s threshold for risk and which can’t target users to create an efficient customer data-related ventures are OK to pursue and which acquisition channel, he will likely get fired at some are not. point—or at least he won’t make his bonus.” Such rules can improve and potentially speed up Looking only at the data, not at the sources individual and organizational decision making. Ethical lapses can occur when executives look They should be tailored to your specific industry, only at the fidelity and utility of discrete data even to the products and services your company sets and don’t consider the entire data pipeline. offers. They should be accessible to all employees, Where did the data come from? Can this vendor partners, and other critical stakeholders. And they ensure that the subjects of the data gave their should be grounded in a core principle—for example, informed consent for use by third parties? Do any “We do not use data in any way that we cannot link of the market data contain material nonpublic to a better outcome for our customers.” Business 4 “SEC charges App Annie and its founder with securities fraud,” US Securities and Exchange Commission, September 14, 2021. 4 Data ethics: What it means and what it takes Leaders must come together to create a data usage framework that reflects a shared vision and mission for the company’s use of data. leaders should plan to revisit and revise the rules for powerful joint action, such as the creation of periodically to account for shifts in the business and industry-wide data ethics standards. technology landscape. Build a diverse data-focused team Communicate your data values, both inside and A strong data ethics program won’t materialize out outside your organization of the blue. Organizations large and small need Once you’ve established common data usage rules, people who focus on ethics issues; it cannot be it’s important to communicate them effectively a side activity. The work should be assigned to a inside and outside the organization. That might specific team or attached to a particular role. Some mean featuring the company’s data values on larger technology and pharmaceutical companies employees’ screen savers, as the company of one have appointed chief ethics or chief trust officers of our interview subjects has done. Or it may be as in recent years. Others have set up interdisciplinary simple as tailoring discussions about data ethics to teams, sometimes referred to as data ethics various business units and functions and speaking boards, to define and uphold data ethics. Ideally, to their employees in language they understand. such boards would include representatives from, The messaging to the IT group and data scientists, for example, the business units, marketing and for instance, may be about creating ethical data sales, compliance and legal, audit, IT, and the algorithms or safe and robust data storage C-suite. These boards should also have a range of protocols. The messaging to marketing and sales genders, races, ethnicities, classes, and so on: an teams may focus on transparency and opt-in/opt- organization will be more likely to identify issues out protocols. early on (in algorithm-training data, for example) when people with a range of different backgrounds Organizations also need to earn the public’s and experiences sit around the table. trust. Posting a statement about data ethics on the corporate website worked for one financial- One multinational financial-services corporation services organization. As an executive explained: has developed an effective structure for its data “When you’re having a conversation with a ethics deliberations and decision making. It has government entity, it’s really helpful to be able to two main data ethics groups. The major decisions say, ‘Go to our website and click on Responsible are made by a group of senior stakeholders, Data Use, and you’ll see what we think.’ We’re on including the head of security and other senior record in a way that you can’t really walk back.” technology executives, the chief privacy officer, Indeed, publicizing your company’s data ethics the head of the consulting arm, the head of framework may help increase the momentum strategy, and the heads of brand, communications, Data ethics: What it means and what it takes 5  and digital advertising. These are the people most organization, put teeth into data rules, and support likely to use the data. the case for investment in data-related initiatives. Governance is the province of another group, Indeed, corporate boards and audit committees which is chaired by the chief privacy officer and can provide the checks needed to ensure that includes the global head of data, a senior risk data ethics are being upheld, regardless of executive, and the executive responsible for the conflicting incentives. The president of one tech company’s brand. Anything new concerning data company told us that its board had recently begun use gets referred to this council, and teams must asking for a data ethics report as part of the audit explain how proposed products comply with the committee’s agenda, which had previously focused company’s data use principles. As one senior more narrowly on privacy and security. “You have company executive explains, “It’s important that to provide enough of an incentive—a carrot or a both of these bodies be cross-functional because stick to make sure people take this seriously,” the in both cases you’re trying to make sure that you president said. have a fairly holistic perspective.” Consider the impact of your algorithms and As we’ve noted, compliance teams and legal overall data use counsel should not be the only people thinking Organizations should continually assess the effects about a company’s data ethics, but they do have of the algorithms and data they use—and test an important role to play in ensuring that data for bias throughout the value chain. That means ethics programs succeed. Legal experts are best thinking about the problems organizations might positioned to advise on how your company should create, even unwittingly, in building AI products. apply existing and emerging regulations. But teams For instance, who might be disadvantaged by may also want to bring in outside experts to navigate an algorithm or a particular use of data? One particularly difficult ethical challenges. For example, technologist we spoke with advises asking the hard a large tech company brought in an academic expert questions: “Start your meetings about AI by asking, on AI ethics to help it figure out how to navigate gray ‘Are the algorithms we are building sexist or racist?’” areas, such as the environmental impact of certain kinds of data use. That expert was a sitting but not Certain data applications require far greater scrutiny voting member of the group because the team “did and consideration. Security is one such area. A tech not want to outsource the decision making.” But the company executive recalled the extra measures his expert participated in every meeting and led the organization took to prevent its image and video team in the work that preceded the meetings. recognition products and services from being misused: “We would insist that if you were going to Engage champions in the C-suite use our technology for security purposes, we had Some practitioners and experts we spoke with who to get very involved in ensuring that you debiased had convened data ethics boards pointed to the the data set as much as possible so that particular importance of keeping the CEO and the corporate groups would not be unfairly singled out.” It’s board apprised of decisions and activities. A senior important to consider not only what types of data executive who chaired his organization’s data ethics are being used but also what they are being used group explained that while it did not involve the CEO for—and what they could potentially be used for directly in the decision-making process, it brought down the line. all data ethics conclusions to him “and made sure he agreed with the stance that we were taking.” Think globally All these practitioners and experts agreed that The ethical use of data requires organizations to having a champion or two in the C-suite can signal consider the interests of people who are not in the the importance of data ethics to the rest of the room. Anthropologist Mary Gray, the senior principal 6 Data ethics: What it means and what it takes  researcher at Microsoft Research, raises questions to real-world use cases for data ethics, such as about global reach in her 2019 book, Ghost Work. decisions on design processes or M&A. In some Among them: Who labeled the data? Who tagged cases, there will be obvious places to operationalize these images? Who kept violent videos off this data ethics—for instance, data operations teams, website? Who weighed in when the algorithm secure-development operations teams, and needed a steer? machine-learning operations teams. Trust-building frameworks for machine-learning operations can Today’s leaders need to ask these sorts of ensure that data ethics will be considered at every questions, along with others about how such step in the development of AI applications. tech work happens. Broadly, leaders must take a 10,000-foot view of their companies as players Regardless of which part of the organization the in the digital economy, the data ecosystem, leaders target first, they should identify KPIs and societies everywhere. There may be ways that can be used to monitor and measure its they can support policy initiatives or otherwise performance in realizing their data ethics objectives. help to bridge the digital divide, support the To ensure that the ethical use of data becomes part expansion of broadband infrastructure, and of everyone’s daily work, the leadership team also create pathways for diversity in the tech industry. should advocate, help to build, and facilitate formal Ultimately, data ethics requires leaders to reckon training programs on data ethics. with the ongoing rise in global inequality—and the increasing concentration of wealth and value both in geographical tech hubs and among AI-enabled organizations.5 Data ethics can‘t be put into practice overnight. As many business leaders know firsthand, building Embed your data principles in your operations teams, establishing practices, and changing It’s one thing to define what constitutes the ethical organizational culture are all easier said than done. use of data and to set data usage rules; it’s another What’s more, upholding your organization’s data to integrate those rules into operations across ethics principles may mean walking away from the organization. Data ethics boards, business potential partnerships and other opportunities to unit leaders, and C-suite champions should build generate short-term revenues. But the stakes for a common view (and a common language) about companies could not be higher. Organizations that how data usage rules should link up to both the fail to walk the walk on data ethics risk losing their company’s data and corporate strategies and customers’ trust and destroying value. Find more content like this on the McKinsey Insights App Alex Edquist is an alumna of McKinsey’s Atlanta office; Liz Grennan is an associate partner in the Stamford, Connecticut, office; Sian Griffiths is a partner in the Washington, DC, office; and Kayvaun Rowshankish is a senior partner in the New York office. The authors wish to thank Alyssa Bryan, Kasia Chmielinski, Ilona Logvinova, Keith Otis, Marc Singer, Naomi Sosner, and Eckart Windhagen for their contributions to this article. Designed by McKinsey Global Publishing Scan Download Personalize Copyright © 2022 McKinsey & Company. All rights reserved. 5 For more on the concentration of value among AI-enabled firms, see Marco Iansiti and Karim R. Lakhani, Competing in the Age of AI: Strategy and Leadership When Algorithms and Networks Run the World, Boston: Harvard Business Review Press, 2020. Data ethics: What it means and what it takes 7