AI in Washington, D.C. - AI Risk Management Framework
In January 2023, the U.S. Department of Commerce NIST (National Institute of Standards and Technology) released the Artificial Intelligence Risk Management Framework, also called "AI RMF". There is a companion AI RMF Playbook with suggested actions for achieving the outcomes described by the AI RMF.
AI systems were defined as an "engineered or machine-based system that can, for a given set of objectives, generate outputs such as predictions, recommendations, or decisions influencing real or virtual environments. AI systems are designed to operate with varying levels of autonomy."
Professional responsibility was defined as "an approach that aims to ensure that professionals who design, develop, or deploy AI systems and applications or AI-based products or systems recognize their unique position to exert influence on people, society, and the future of AI".
Risk management was defined as "coordinated activities to direct and control an organization with regard to risk".
Validation was defined as "confirmation, through the provision of objective evidence, that the requirements for a specific intended use or application have been fulfilled".
Reliability was defined as "ability of an item to perform as required, without failure, for a given time interval, under given conditions".
The framework recognizes that AI technology pose risks that can negatively impact individuals, groups, organizations, communities, society, the environment, and the planet. These risks can be classified as long or short term, high or low probability, systemic or localized, and high or low impact. Risk management is a key component of responsible development and use of AI systems. The core concepts of such responsible AI are human centricity, social responsibility, and sustainability. Those who design, develop, and deploy AI should think critically about context and potential for unexpected negative and positive impacts.
AI risk or failures must be well-defined and adequately understood in order to measure quantitatively or qualitatively.
The characteristics of trustworthy AI systems include valid and reliable, safe, secure and resilient, accountable and transparent, explainable and interpretable, privacy enhanced, and fair with harmful biases managed.
AI systems have risks that are not comprehensively addressed by current risk frameworks and approaches.
Four specific functions to address AI system risks:
Govern: A culture of risk management is cultivated and present Map: Context is recognized and risks related to context are identified Measure: Identified risks are assessed, analyzed, or tracked Manage: Risks are prioritized and acted upon based on a projected impact
Potential harm posed by AI systems:
- civil liberties or rights
- physical or psychological safety
- economic opportunity
- discrimination against a population sub-group
- democratic participation
- educational access
- business operations
- security breaches or monetary loss
- reputation
- interconnected/interdependent elements/resources of an ecosystem
- global financial system
- supply chain
- natural resources
- environment
Risk may be different at various stages of the AI lifecycle. Some risks may increase as AI systems adapt. Developers may have different perceptions or perspectives of risk.
Laboratory and real-world risks are different.
Risk tolerance is influenced by legal or regulatory requirements and is impacted by acceptance of risk to achieve an objective.
Risk prioritization must be realistic to avoid ineffective resource allocation.
AI risks should not be isolated from broader risks. For example, there are overlapping risks associated with privacy of the underlying data to train AI systems, or the environmental concerns of resource-heavy computing demands.
AI actors should represent a diversity of experience, expertise, and backgrounds and comprise demographically and disciplinary diverse teams.
Subject matter experts can assist in the evaluation of TEVV (test, evaluation, verification, and validation) findings and work with product and deployment teams to align TEVV parameters to requirements and deployment conditions.
AI systems should not endanger human life, health, property, or the environment. Safe AI system operation is achieved through (1) responsible design, development, and deployment practices; (2) clear information to deployers on responsible use of the system; (3) responsible decision-making by deployers and end users; and (4) explanations and documentation of risks based on empirical evidence of incidents.
An AI system is resilient if able to withstand unexpected adverse events or changes in their environment or use.
Trustworthiness of AI requires transparency of information about an AI system and its outputs available to individuals interacting with the system.
Transparency answers the question of "what happened" in the system. Explainability answers "how" a decision was made. Interpretability answers "why" a decision was made.
Privacy refers to the norms and practices that help safeguard human autonomy, identity, and dignity. This includes freedom from intrusion, limiting observation, or an individual's consent to disclosure or control of their identities.
Lifecycle Stages
Plan and Design: Articulate and document the system's concept and objectives, underlying assumptions, and context in light of legal and regulatory requirements and ethical considerations
Collect and Process Data: Gather, validate, and clean data and document the metadata and characteristics of the dataset
Build and Use Model: Create or select algorithms and train models.
Verify and Validate: Verify and validate, calibrate, and interpret model output
Deploy and Use: Pilot, check compatibility with legacy systems, verify regulatory compliance, manage organizational change, and evaluate user experience
Operate and Monitor: Operate the AI system and continuously assess its recommendations and impacts
Use or Impact: Use system or technology, monitor and assess impacts, seek mitigation of impacts, and advocate for rights
Govern Function
GOVERN is a cross-cutting function that is infused throughout AI risk management and enables the other functions of the process.
- Policies, processes, procedures, and practices across the organization related to the mapping, measuring, and managing of AI risks are in place, transparent, and implemented effectively.
- Accountability structures are in place so that the appropriate teams and individuals are empowered, responsible, and trained for mapping, measuring, and managing AI risks.
- Workforce diversity, equity, inclusion, and accessibility processes are prioritized in the mapping, measuring, and managing of AI risks throughout the lifecycle.
- Organizational teams are committed to a culture that considers and communicates AI risk.
- Processes are in place for robust engagement with relevant AI actors.
- Policies and procedures are in place to address AI risks and benefits arising from third-party software and data and other supply chain issues.
Map Function
The MAP function establishes the context to frame risks related to an AI system. The information gathered enables negative risk prevention and informs decisions for processes such as model management. Framework users should have sufficient contextual knowledge about AI system impacts to inform an initial go/no-go decision about whether to design, develop, or deploy an AI system.
- Context is established and understood.
- Categorization of the AI system is performed.
- AI capabilities, targeted usage, goals, and expected benefits and costs compared with appropriate benchmarks are understood.
- Risks and benefits are mapped for all components of the AI system including third-party software and data.
- Impacts to individuals, groups, communities, organizations, and society are characterized.
Measure Function
The MEASURE function employs quantitative, qualitative, or mixed-method tools, techniques, and methodologies to analyze, assess, benchmark, and monitor AI risk and related impacts. After completing the MEASURE function, objective, repeatable, or scalable test, evaluation, verification, and validation (TEVV) processes including metrics, methods, and methodologies are in place, followed, and documented.
- Appropriate methods and metrics are identified and applied.
- AI systems are evaluated for trustworthy characteristics.
- Mechanisms for tracking identified AI risks over time are in place.
- Feedback about efficacy of measurement is gathered and assessed.
Manage Function
The MANAGE function entails allocating risk resources to mapped and measured risks on a regular basis and as defined by the GOVERN function. After completing the MANAGE function, plans for prioritizing risk and regular monitoring and improvement will be in place.
- AI risks based on assessments and other analytical output from the MAP and MEASURE functions are prioritized, responded to, and managed.
- Strategies to maximize AI benefits and minimize negative impacts are planned, prepared, implemented, documented, and informed by input from relevant AI actors.
- AI risks and benefits from third-party entities are managed.
- Risk treatments, including response and recovery, and communication plans for the identified and measured AI risks are documented and monitored regularly.
AI Actors
AI Design: data scientists, domain experts, socio-cultural analysts, experts in the field of diversity, equity, inclusion, and accessibility, members of impacted communities, human factors experts (e.g., UX/UI design), governance experts, data engineers, data providers, system funders, product managers, third-party entities, evaluators, and legal and privacy governance.
AI Development: machine learning experts, data scientists, developers, third-party entities, legal and privacy governance experts, and experts in the socio-cultural and contextual factors associated with the deployment setting.
AI Deployment: system integrators, software developers, end users, operators and practitioners, evaluators, and domain experts with expertise in human factors, socio-cultural analysis, and governance.
Operation and Monitoring: system operators, domain experts, AI designers, users who interpret or incorporate the output of AI systems, product developers, evaluators and auditors, compliance experts, organizational management, and members of the research community.
AI Risks
Compared to traditional software, AI-specific risks that are new or increased include:
- The data used for building an AI system may not be a true or appropriate representation of the context or intended use of the AI system
- AI system dependency and reliance on data for training tasks, combined with increased volume and complexity typically associated with such data
- Intentional or unintentional changes during training may fundamentally alter AI system performance
- Datasets used to train AI systems may become detached from their original and intended context or may become stale or outdated relative to deployment context
- AI system scale and complexity (many systems contain billions or even trillions of decision points) housed within more traditional software applications
- Use of pre-trained models that can advance research and improve performance can also increase levels of statistical uncertainty and cause issues with bias management, scientific validity, and reproducibility
- Higher degree of difficulty in predicting failure modes for emergent properties of large-scale pre-trained models
- Privacy risk due to enhanced data aggregation capability for AI systems
- AI systems may require more frequent maintenance and triggers for conducting corrective maintenance due to data, model, or concept drift
- Increased opacity and concerns about reproducibility
- Underdeveloped software testing standards and inability to document AI-based practices to the standard expected of traditionally engineered software for all but the simplest of cases
- Difficulty in performing regular AI-based software testing, or determining what to test, since AI systems are not subject to the same controls as traditional code development
- Computational costs for developing AI systems and their impact on the environment and planet
- Inability to predict or detect the side effects of AI-based systems beyond statistical measures
Sources
AI Risk Management Framework - NIST. https://www.nist.gov/itl/ai-risk-management-framework [Accessed 12/17/2023]