Artificial intelligence (AI) procurement essentials

1. Review policy documents, including AI Ethics Policy, undertake self-assessment outlined in the AIAF and identify relevant agency policies and requirements.
2. Identify and engage responsible officers.
3. Risk assessment: Identify and assess AI-related risks and determine the controls to be implemented during the procurement lifecycle.
4. Contracting framework: Determine the appropriate contracting framework to manage risks.
5. Responsible officers: Document processes, controls, and responsibilities of responsible officers.

Expected outputs:

• AIAF triage
• a procurement strategy that incorporate AI considerations and selected contracting framework to use.

1. Draft RFx documentation with relevant solution and process requirements, and assessment questions to understand and assess available project and solution controls.
2. Assess risks and controls: Revisit the risk assessment from the plan stage to ensure relevant controls are embedded, including process and solution elements, review and reporting, and specific contractual clauses.
3. Contract development: Draft a contract that includes clauses to mitigate any residual risks, as well as the controls that have been decided upon. The contract also needs to include ability to manage future features and technology or system changes. This contract will govern the relationship between the supplier and the NSW Government agency.
4. Evaluate responses: Assess supplier and/or solution's ability to manage risks identified as part of the AIAF.
5. Negotiate and sign contract: Incorporate the contractual controls that best address residual risks, including ability to assess system’s performance, manage algorithm updates and failsafe conditions.

Expected outputs:

• AIAF self-assessment
• supplier solution assessment
• negotiated contracting framework documents
• completed statement of work, including levers to conduct ongoing assessments, and ability to manage failsafe conditions and deployment of future features in line with AI Ethics Principles.

1. Resource allocation: Allocate team members and other resources for implementation and ongoing management.
2. System design and development: Customise and integrate the AI system into the existing infrastructure.
3. Training and onboarding: Train staff on how to use the new system.
4. Pilot testing: Conduct a small-scale test to ensure everything is working as expected.
5. Full deployment: Roll out the AI system across the agency.
6. Monitoring and maintenance: Carry out ongoing monitoring and updates to ensure optimal performance.

Expected outputs:

• responsible officers RACI
• monitoring plans.

1. Performance review: Conduct periodic evaluations to measure the system's return on investment and effectiveness of controls.
2. Scalability assessment: Develop plans for potential scaling or further development.

The NSW Procurement Policy Framework sets out the policy and operating framework for all government procurement. The Procurement Policy Framework has the status of a ‘policy’ under the Public Works and Procurement Act 1912, which means that NSW Government buyers must comply with its mandatory requirements in every procurement activity. The Framework provides a comprehensive guide on how to apply different policies throughout the procurement process to achieve best value for money in supporting the delivery of government services.

NSW Government buyers must use the ICT Purchasing Framework for all ICT and digital procurement when entering into a contract with a supplier. This is mandated by the NSW Procurement Board Direction PBD 2021-02. The ICT Purchasing Framework comprises:

• Core& contracting framework for low-risk procurements up to $1 million (excluding GST)
• MICTA/ICTA contracting framework for procurements that are high risk or over $1 million (excluding GST).

NSW Government buyers are required to make a risk-based determination about which contracting framework is appropriate, using the guidelines for assessing risk in ICT/digital sourcing, in accordance with their own agency risk management policies and processes.

The Artificial Intelligence Ethics Policy sets out 5 overarching principles that are designed to ensure best practice use of AI, focusing on community benefit, fairness, privacy and security, transparency and accountability. These ethical principles are mandatory for NSW Government agencies for the use of AI.

The AI Ethics Policy provides that AI must be:

• the most appropriate solution for a service delivery or policy problem,
• used in such a way as to mitigate as much potential bias as possible,
• used safely, securely, and in line with existing privacy and information access requirements,
• a solution that is open and transparent so that NSW citizens have access to efficient review mechanisms, and
• a solution where the decisions are always subject to human review and intervention.

The AI Strategy (the Strategy) is focused on improved service delivery and government decision-making. The Strategy outlines the first steps in building a consistent approach across the sector.

Procurement is identified as one of the key areas of the Strategy and is identified as an area of opportunity and future growth. The Strategy also notes that guidance on the effective procurement of AI is crucial in sourcing the right solution and for maintaining consistent approaches across the sector.

The NSW AIAF supports NSW Government to innovate with AI technology, while ensuring we use them safely and appropriately, with clear accountability for the design and use. The AIAF is a self-assessment, intended to be applied during all phases of development, training and use of AI. The AIAF assists NSW Government agencies to procure AI enabled systems and solutions.

The AIAF is not a complete list of all requirements for AI projects. NSW Government buyers are also required to comply with their agency-specific AI processes, policy requirements and governance mechanisms.

Responsible officers are defined under the NSW AIAF. However, the responsibilities of the officers can extend beyond the Assessment Framework.

These include the officer who is responsible for: use of the AI insights/decisions; the outcomes from the project; the technical performance of the AI system; data governance.

Early engagement ensures that the AI procurement aligns with the AIAF from the outset. It also allows for proactive risk management and ethical considerations.

• Manages delivery of the procurement activities on behalf of the project lead and in accordance with the procurement's project plan.
• Conducts strategic planning, resource allocation, risk management, and ensuring project success.
• Collaborates with other key personnel to align procurement activities with project goals.

Engaging the project manager early allows for effective planning and resource allocation. It also helps in setting realistic timelines and managing expectations.

• Undertakes the procurement activities during the sourcing phase.
• Engages with suppliers, issues tender, evaluates proposals, and negotiate contracts.
• Ensures compliance with procurement policies and regulations.

Early engagement with the Procurement team allows for thorough market research and supplier evaluation. It also provides ample time for negotiation to secure the best terms.

• Manages the contract during post-award phase.
• Monitors supplier performance, manages contract variations, and resolves issues.
• Ensures value for money and compliance throughout the contract duration.

Early involvement of the contract manager ensures that the contract is clear, fair, and protects the agency's interests. It also allows for monitoring of the service provider's performance from the start. In the case where AI is introduced in-contract, early engagement of the contract manager ensures the resolution of issues and efficient negotiation and variation of any contracts where necessary.

• Collaborates closely with the Project Manager business unit that owns the product or service from the service provider.
• Manages budget allocations, tracks expenses, and ensures financial transparency.
• Crucial for cost-effective decision-making and setting up ongoing cost monitoring levers.

Engaging the finance officer early ensures that the financial aspects of the procurement are managed effectively. It allows for budget preparation, thorough cost-benefit analysis, and ensures value for money.

• Provides legal expertise.
• Reviews contracts, contract variations, assesses risks, and ensures compliance with legal requirements.
• Contractually safeguards the agency's interests.

Early engagement with legal representative ensures that the agency is legally protected. It allows for review of contracts, advice on intellectual property rights, and compliance with data protection laws and other regulations.

• Contributes subject matter expertise and contributes domain-specific knowledge.
• Assists in defining requirements, evaluating technical aspects, and assessing supplier capabilities.
• Vital for informed decision-making, and determining any changes to a product, service, or system during the life of a contract.

Engaging technical experts early allows for assessment of the technical feasibility of the AI solution. It also helps in defining technical requirements and planning for implementation and integration.

• Engages stakeholders, communicates progress, and manages expectations.
• Facilitates collaboration across departments.

Early engagement with the stakeholder engagement Lead ensures that all stakeholders are informed and involved in the process from the beginning. It helps in managing expectations and gathering valuable feedback.

• Provides oversight for the delivery of projects, the undertaking of business operations and/or strategic directions.
• Comprised of senior executives and board members it approves major decisions, reviews risks, and ensures strategic alignment.

Engaging governance groups early ensures that the procurement process aligns with the agency's governance framework. It allows for strategic decision-making and monitoring of the project's progress and outcomes.

Amplified risks exist for cybersecurity, privacy and system performance related to initial and ongoing data quality. Procurements involving high-risk AI systems should set clear expectations and controls for ensuring high-quality data are managed via security-by-design and privacy-by-design.

Action

Buying teams to ensure that both training and operational data sets used in AI systems are lawfully and appropriately used, while being fully secured.

Example controls

Plan: Assess available data for project and determine solution requirements to maintain accessibility, reproducibility and robustness of system that will use the data.

Source: Request from Suppliers and assess methodologies for identifying and minimising bias in data sets and outcomes.

Manage: Implement agreed monitoring and review process of data quality and its use by system.

AI systems can experience performance issues and ‘model drift’ as the data they were trained in becomes less representative over time. Iterative AI life cycle increases likelihood of post procurement emergence of risk and unintended consequences.

Action

Buying teams need to build into procurement and contracting processes levers to ensure they can continue ongoing performance assessments and ability restore system to required operation levels.

Example controls

Plan:  Accountability for monitoring the performance of the AI system and outcomes assigned to a responsible officer.

Source:  

Ensure service levels include:

• quality assurance and evaluation for primary (accuracy, precision, sensitivity, specificity)
• secondary characteristics (Fairness including robustness and reproducibility, transparency, usability, privacy and security, data use tracking).

Manage: Implement agreed monitoring and review processes required to monitor and calibrate performance measures and targets.

Many third party-provided AI systems will provide automatic updates and revisions to AI models and components after initial risk assessments are complete. This may introduce further amplified and emerging risks, requiring special controls to manage safely.

Action

• Introduce governance steps to manage any potential harms (especially for vulnerable groups) arising from changes to algorithms and features.
• Build into the contracting process needed clauses to ensure system continue to best serve our purposes and the citizens of NSW over time.

Example controls

Plan: Set stakeholder engagement plans and outcomes for the system or model, demonstrating adherence to diversity principles and consideration given to vulnerable groups.

Source: Assess and agree timeframes, triggers, and processes for assessing changes to algorithms and new features prior to implementation.

Manage: Implement processes and metrics for any required adaptation, maintenance, updates and retraining as a result of algorithm changes.

AI systems may be more brittle, more complex to debug and more challenging to secure than traditional IT systems. Ensuring system resilience, integrity, and continuity may therefore require additional controls.

Scalability of AI-based systems can quickly and exponentially cause significant and irreversible harms. Efficiencies gained through their deployment can cause over-reliance on its capabilities, compounded by removal of existing expertise and processes to optimize cost.

Action

Buying teams to ensure that, if critical errors or harms arise, they can shut off systems without jeopardising our core operations and services.

Example controls

Plan: Determine target performance metrics for system security, resilience, and integrity, define trigger events and failsafe conditions.

Source: Assess solution’s reliability measures and targets, and mechanisms to prevent unintended or unauthorised use. Set service levels and reporting cadence.

Assess solution’s trigger event measures for unacceptable outcomes, and include required measures in the contract (e.g., Service levels, warranties, ability to suspend systems/operations).

Manage:

Implement processes to review and manage:

• ongoing performance of the system,
• triggers and processes for out-of-cycle reviews, based on agreed unacceptable outcomes or system failures
• resulting contract implications.

Lack of transparency of the operational aspects of the system can impair the ability to understand whether it is performing effectively and/or in line with regulatory requirements.

Action

Buying teams to ensure that they have access to the information required to understand AI systems, verify supplier claims, and provide reasons for decisions. Buying teams to ensure system design and supplier contracts contain measures to maintain the required levels of transparency and explainability, including where appropriate third-party audit rights.

Example controls

Plan: Determine transparency and audit requirements of the system using AI Ethics principles and the AIAF.

Source: Assess the system's ability to notify users they are using AI.

Assess the supplier's ability to produce:

• documentation
• process logs for activity
• process logs for outcomes across the AI life cycle
• process logs during the term of the agreement.

Negotiate required audit provisions.

Manage:

Implement processes to:

• assess and respond to regulatory changes
• review logs and audit
• review training plans for both supplier and agency staff to ensure project goals and regulatory requirements are understood and able to be met.