Report

Interpretable machine learning

Modern machine learning (ML) systems are increasingly being used to inform decision making in a variety of applications. However, for some types of ML, such as ‘deep learning’, it may not be possible to explain completely how a system has reached its output. A further concern is that ML systems are susceptible to introducing or perpetuating discriminatory bias. Experts have warned that a lack of clarity on how ML decisions are made may make it unclear whether the systems are behaving fairly and reliably, and may be a barrier to wider ML adoption. 

In 2018, the Lords Committee on AI called for the development of AI systems that are “intelligible to developers, users and regulators”. It recommended that an AI system that could have a substantial impact on an individual’s life should not be used unless it can produce an explanation of its decisions. In a January 2020 review, the Committee on Standards in Public Life noted that explanations for decisions made using ML in the public sector are important for public accountability and recommended that government guidance on the public sector use of AI should be made easier to use. 

The UK Government has highlighted the importance of ethical ML and the risks of a lack of transparency in ML-assisted decision-making. In 2018 it established the Centre for Data Ethics and Innovation to provide independent advice on measures needed to ensure safe, ethical and innovative uses of AI.  

Key Points

  • ML is increasingly being used to inform decision making in a variety of applications. It has the potential to bring benefits such as increased labour productivity and improved services.  
  • ML relies on large datasets to train its underlying algorithms. Unrepresentative, inaccurate or incomplete training data can lead to risks such as algorithmic bias. 
  • The term ‘algorithmic bias’ is used to describe discrimination against certain groups on the basis of an ML system’s outputs. Bias can be introduced into a ML system in different ways including through a system’s training data or decisions made during development. There have been several recent high-profile examples of algorithmic bias.  
  • Experts have raised concerns about a lack of transparency in decisions made or informed by ML systems. This is a particular issue for certain complex types of ML, such as deep learning, where it may not be possible to explain completely how a decision has been reached. 
  • Complex ML systems where it is difficult or impossible to fully understand how a decision has been reached are often referred to as ‘black box’ ML.  
  • Terminology varies, but ‘interpretability’ is typically used to describe the ability to present or explain an ML system’s decision-making process in terms that can be understood by humans. 
  • Many stakeholders have highlighted that the extent to which ML needs to be interpretable is dependent on the audience and context in which it is used. 
  • Technical approaches to interpretable ML include designing systems using types of ML that are inherently easy to understand and using retrospective tools to probe complex ML systems and obtain a simplified overview of how they function. 
  • Some stakeholders have said that limiting applications to inherently interpretable ML types may limit the capability of ML technology. However, others argue that there is not always a trade-off between performance accuracy and interpretability, and in many cases complex ML can be substituted for a more interpretable method. 
  • Tools for interpreting complex ML retrospectively are in early stages of development and their use is not currently widespread. Some tools aim to interpret a specific ML decision, while others can be used to give a broad understanding of how an ML system behaves. 
  • The ICO and Alan Turing Institute have produced guidance for organisations to help them explain AI-based decisions to affected individuals. 
  • Benefits of interpretable ML including improved understanding of how a system functions and improved user trust in a system. 
  • However, there are also challenges with interpretability such as commercial sensitivities and the risk of gaming. 
  • In addition to technical approaches to interpretable ML, many stakeholders have called for wider accountability mechanisms to ensure that ML systems are designed and deployed in an ethical and responsible way. 
  • Some wider ML accountability mechanisms include detailed documentation of a ML system’s development process, algorithm impact assessments, algorithm audits and use of frameworks and standards.