2. Formal oversight mechanisms for government AI use, proportional to risk
2. Formal oversight mechanisms for government AI use, proportional to risk
We want to see how the government will make sure that their use of AI is making public services better and not worse. This information should be available to the public, and it should relate to the risks and benefits that New Zealanders care about.
When the government is using AI in a way that could have a big impact on New Zealand, we want to see how they plan to manage these risks. This may mean:
setting a date to check if the model is still working as expected,
deciding not to use the AI for certain tasks,
building in a safeguard to prevent harms in specific scenarios, or
only using the AI for low-risk parts of the work.
For example, ACC currently uses AI to automatically approve people's benefits if it's a straightforward ACC claim. The system has a very high accuracy rate and enables claim decisions to be approved in seconds. However, a human has to make the decision if a claim will be denied.
We also want to make sure that the plan is followed. Therefore, we're asking for a 'formal oversight mechanism' someone who can check that the plan has been made correctly, and that the system continues to be safe. If it's a low-risk use of AI, this could be the same agency that's using the AI. For medium-risk or high-risk uses of AI, it may be better to have a central oversight agency like the Public Service Commission or a new AI Safety Institute (or someone else) to check in and make sure everything is set up for safety.
If you're worried that assessing every way AI is used is too much paperwork... it doesn't have to be! Some types of AI aren't very risky. For example, many automatic phone menus use AI to understand what you have said and choose where to send your call. If a use of AI is clearly low-risk, it could be OK to report less information about it. But systems that have a material impact on the public, from welfare to housing to legal rights, should be able to receive the necessary public scrutiny.
At this stage, we don't know exactly how to define 'low-risk' and 'high-risk' uses of AI. That's why we're also asking for public input. We, the New Zealand public, should have time and space to figure out what types of AI use in government we are comfortable with.
Knowing that an AI system is being used is only half the battle; if we want to trust AI use in government, we must know that it's worthy of our trust. This means that we need to know the risks were fully explored before the AI system was deployed, and that there is a plan in place to ensure it is working correctly.
New Zealand currently relies on a patchwork of voluntary guidelines and internal agency policies to ensure that government AI systems are safe and fair. Most people in the public sector want to do this well. But when a machine is handed the power to influence life-altering decisions, good intentions are not as good as formal oversight.
We believe that oversight must be legally mandated where there is a material level of risk. We don't want to clog up the public service with red tape for low-risk administrative tools. But systems that can have a material impact on someone's life demand rigorous assessment and auditing. Formal oversight mechanisms are a chance for experts to test the system, make sure it's performing well, and stop any systems that aren't up to scratch before they cause further harm.
The call for formal oversight mechanisms has two parts: impact assessment while the systems are designed and developed; and ongoing governance after the systems have been deployed. This helps shift public perception from blind trust to earned confidence. This gives both the public and our public servants the reassurance that AI is being used safely, equitably, and in a way that actively upholds our rights and values.
If you're wondering what 'our rights and values' are, then have a look at our third idea, around public deliberation.
Canada has a Directive on Automated Decision-Making and have published their Algorithmic Impact Assessment Tool
Singapore’s Government Developer Portal includes a Responsible AI Playbook
The UK government’s guidance on algorithmic transparency includes specific guidance about impact assessments.
Australia has just introduced an AI Review Committee to provide advice on AI uses cases that are high-risk or highly sensitive, novel, or complex.
Our very own government has published the Algorithm Impact Assessment toolkit developed by Frith Tweedie. Using this toolkit is currently voluntary, and we haven't found any completed assessments that have been published proactively in New Zealand. Either no one is doing the assessments, or we aren't being told about it.
2020 August: The Algorithm Charter chapter in More Zeros and Ones* calls for the government to consider adopting an Algorithmic Impact Assessment tool like Canada's.
2023 December: Under the Algorithm Charter, StatsNZ published the Algorithm Impact Assessment. (See the section above this, where we note that this is voluntary, and we can't see any evidence online that any agencies have used it.)
2025: Te Kāhui Raraunga released the Māori AI Governance framework, recommending monitoring by an independent authority for both potential and actual data and algorithmic harm to Māori.
2026 January: The Public Service AI Work Programme has an item to deliver a "Public Service AI Assurance Model". This would likely apply to agencies under the Public Service Act 2020. However, there are not yet any publicly-available details about this model, and it's unclear if this will be anchored into the risks and benefits that the people of New Zealand care about.
2026 March: Johniel Bocacao summarises the current (and very complex) mechanisms for AI oversight in the NZ government in this report.
The details of what is included in the formal oversight will need to be debated before they reach their final form - that's why we have asked for public deliberation. But we have some starting ideas:
Before any tool is deployed, agencies could use a standardised risk matrix to determine if the level of risk is high enough to justify formal oversight. For an example of a risk matrix, see New Zealand's Algorithm Charter and Algorithm Impact Assessments. This should also consider Māori Algorithmic Sovereignty Principles for ensuring the protection of Māori and Māori taonga during the design and use of algorithms.
For systems that are low or medium-risk, the agency could be responsible for the oversight. For systems that are high-risk, we think there should be independent oversight from another central government agency (such as the Public Service Commission).
Agencies could be required to add their AI system to the public register, publish a plain-language Al Impact Assessment modelled off existing tools, and demonstrate that they have ongoing plans in place to measure performance and use human oversight.
Agency Chief Executives could be held legally accountable under the Public Service Act for ensuring that medium-risk and high-risk systems are appropriately disclosed and audited on an ongoing basis.
Ideally, a small oversight/governance unit (likely sitting within an existing Commission or agency) would be given the authority to issue compliance notices (or similar legal instrument) to agencies that have demonstrably run AI systems poorly.
Nobody wants to do more work for no reason. But there is a reason to do the extra analysis: it supports workers, executives, and public-service leaders to adopt AI with confidence. Brakes on a high-performance car don't exist to make the car slow - they exist so you can drive fast safely without crashing.
Risk-averse government agencies are slow to approve and adopt AI systems because it is not clear if they have done everything they need to do to be safe. Project teams spend months arguing internally about ethics, privacy, and public perception. Delays are expensive! Having clear processes for appropriate formal oversight can address that systemic anxiety and give agencies a clear, standarised path to safe deployment.
That being said, not all technology is equally risky. We would be excited about an impact assessment process that uses different categories risk. Detailed audits and oversight mechanisms would be reserved for the medium and high-risk systems that could have significant impact on our country or our people.
If the government publishes information about their formal oversight, this will also provide New Zealand businesses a blueprint for their own AI governance approaches. This will help businesses also push past systemic anxiety and into confident, trustworthy adoption. Further, government agencies engaging in the formal oversight process will build public-service capacity and capability within the government, enabling clearer advice and more useful guidance to businesses.