When Governments Buy AI: The Procurement Accountability Gap in the Caribbean

Every time a Caribbean government agency uses software to flag a tax return for audit, screen a social welfare application, or prioritise cases in a healthcare queue, it is deploying a system that makes judgements about individual citizens. Many of those systems now incorporate machine learning components or AI-powered scoring tools. The procurement processes through which those systems were acquired, almost universally, asked none of the questions that matter most: how was the system trained, what populations does it perform poorly on, what happens when it is wrong, and who is accountable when it causes harm?

My hypothesis is this: Caribbean governments are deploying AI systems in public services without adequate procurement frameworks, technical standards, or accountability mechanisms, and the absence of these structures creates a systematic exposure to undetected, unremedied harm to citizens who have no meaningful way to challenge algorithmic decisions made about them by the state. This is not a speculative future risk. It is a present condition.

This article examines the specific gaps in Caribbean government AI procurement, draws on international frameworks including the EU AI Act's provisions for public sector AI deployers, Canada's Directive on Automated Decision-Making, and ISO/IEC 42001 (published December 2023), and sets out a concrete Caribbean AI Government Procurement Framework that CARICOM member states can begin implementing immediately.

The Procurement Gap Is Structural, Not Accidental

Caribbean government procurement frameworks were designed to address different problems. Jamaica's Government of Jamaica Procurement Guidelines, which govern the purchase of goods and services across the public sector, prioritise value for money, competition, transparency in tendering, and prevention of corruption. These are legitimate and important objectives. But they address the process of acquisition, not the nature of what is being acquired.

When a government agency procures a fleet of vehicles, the relevant questions are about price, fuel efficiency, maintenance costs, and delivery schedule. The vehicle will perform predictably. When the same agency procures an AI-powered case management system, the relevant questions are fundamentally different: what training data underlies the system's recommendations, what is its accuracy rate across different demographic groups, what happens when its confidence score is low, can the system's outputs be explained to the citizens they affect, and who bears liability when the system produces an error that harms a citizen? Current procurement frameworks cannot ask these questions because they were not designed to ask them.

This structural gap is compounded by constrained government capacity. Most Caribbean government ministries do not employ data scientists or AI specialists in their technology or procurement teams. When an AI vendor presents a system, the government officials reviewing it are assessing it on the basis of its user interface, its price, and its vendor's reputation, not on the basis of its algorithmic architecture, its training data, or its documented failure modes. This information asymmetry between vendor and procurer is itself a governance failure that specific procurement standards can address.

Where AI Is Already in Caribbean Government

It would be easier to argue that this gap is less urgent if Caribbean governments were not already using AI-influenced systems in consequential contexts. The evidence suggests they are. Jamaica's Tax Administration Jamaica uses data analytics tools, some of which incorporate machine learning components, to flag compliance risks. Financial intelligence units across the region use automated systems for suspicious transaction reporting analysis. Social protection agencies in several CARICOM member states have implemented or piloted benefit eligibility systems that incorporate automated scoring.

The COVID-19 pandemic accelerated this trajectory. Emergency procurement of digital tools during 2020 and 2021 bypassed normal tendering processes in many jurisdictions, creating a cohort of AI-adjacent systems that were acquired without the scrutiny that even conventional procurement would have required. Some of those systems remain in operational use without having been formally reviewed.

Healthcare is a specific concern. Caribbean health ministries are exploring AI tools for patient triage, diagnostic support, and resource allocation. The EU AI Act classifies AI systems used in medical devices and healthcare as high-risk, subject to mandatory conformity assessments. Caribbean health authorities are not currently applying equivalent standards. An AI triage tool that is less accurate for certain patient populations could cause measurable harm before anyone detects that its performance is uneven.

Immigration and border management is another area of acute concern. Automated biometric systems, which the EU AI Act classifies as among the most sensitive applications of AI, are being deployed across the Caribbean at ports of entry. The accuracy of biometric systems across different demographic groups, particularly skin tones and facial structures, is well documented as uneven. Without pre-deployment bias assessment requirements, Caribbean governments deploying these systems have no assurance that they are not systematically misidentifying particular communities of travellers.

What International Frameworks Require of Governments

The EU AI Act imposes specific obligations on deployers of high-risk AI systems, including public sector deployers. Those obligations include: ensuring systems are used in accordance with the instructions of use provided by the provider; maintaining human oversight of high-risk AI outputs; ensuring that the natural persons assigned to oversee the system have the competence, authority, and means to exercise genuine oversight; and establishing procedures for monitoring the system's performance once deployed.

Canada's Directive on Automated Decision-Making, which came into effect in April 2019, provides perhaps the most directly applicable model for Caribbean governments. It requires federal institutions to classify automated decision systems by impact level, to conduct pre-deployment impact assessments for medium and high-impact systems, to ensure meaningful human review is available for automated decisions affecting individuals, and to publish plain-language explanations of how automated systems are used. It also requires peer review of systems above certain impact thresholds.

ISO/IEC 42001, published in December 2023, provides an AI management systems standard that organisations, including government agencies, can certify against. It addresses AI risk management, governance structures, impact assessment, and performance monitoring. Requiring vendors that supply AI systems to Caribbean governments to hold ISO/IEC 42001 certification, or to demonstrate equivalent practices, would immediately raise the accountability floor without requiring Caribbean governments to develop bespoke technical assessment capacity.

The UN Advisory Body on AI's final report of September 2024, "Governing AI for Humanity," specifically called on governments to lead by example, applying to their own AI use at least the same standards of transparency and accountability that they expect of private sector actors. Caribbean governments are not meeting this standard today.

The Public Register Principle

One of the most important accountability mechanisms that international frameworks have converged on is the public register of government AI systems. The idea is simple: if a government agency uses an AI system to make or influence decisions about citizens, those citizens should be able to find out, in accessible terms, that such a system exists, what it does, and how it works in general terms.

The Netherlands has published a government algorithm register since 2022, listing AI and algorithmic systems used across Dutch public services. The UK's Centre for Data Ethics and Innovation has developed guidance on algorithmic transparency for public sector bodies. New Zealand has published similar guidance as part of its Algorithm Charter. In each case, the principle is the same: public money, public accountability.

Caribbean governments have no equivalent mechanism. A Jamaican citizen whose social welfare application is affected by an algorithmic scoring system cannot, today, find any publicly accessible information that such a system is being used, what its decision criteria are, or how to challenge its output. This is inconsistent with the right to fair administrative action that Jamaican constitutional jurisprudence protects, and it is inconsistent with the accountability principles that Jamaica's Data Protection Act 2020 establishes for the private sector.

Establishing a public register of government AI systems is administratively achievable within existing institutional structures. The relevant ministry in each Caribbean government, most likely the ministry responsible for digital transformation or public administration, could maintain such a register with modest resource investment. The political and reputational benefits of doing so proactively, before an AI-related government scandal forces the issue, are considerable.

The Vendor Accountability Problem

Caribbean governments often purchase AI systems from international technology companies that are headquartered in the United States, Europe, or elsewhere. The terms on which those companies supply systems typically reflect the regulatory environment of their home jurisdiction, not the Caribbean's. If a system is being sold into a market that does not require bias audits, the vendor will not volunteer them. If a contract does not require incident reporting, vendors will not report incidents.

This asymmetry is addressable through procurement conditions. A Caribbean government that insists, as a condition of contract award, that the vendor provide: an AI system card documenting training data, known limitations, and accuracy metrics across demographic groups; independent third-party bias audit results; contractual obligations to report incidents where the system produces erroneous outputs affecting more than a defined number of citizens; and the right for the procuring government to conduct its own technical audit of the system upon reasonable notice, would be in a materially stronger position than governments that accept vendor terms without negotiation.

The negotiating power of individual Caribbean governments is limited. But a CARICOM-wide procurement standard, applied consistently across member states, represents a collective market signal that international vendors cannot ignore. This is the same logic that makes GDPR compliance a global de facto standard: the EU market is large enough that vendors prefer to build GDPR-compliant systems for everyone rather than maintain separate non-compliant versions. CARICOM is smaller, but the principle of collective demand is applicable.

Recommendations

1. Amend each member state's government procurement guidelines to include a mandatory AI Disclosure Declaration. Every technology procurement above a defined threshold value (suggested: US$50,000) should require the vendor to complete an AI Disclosure Declaration confirming whether the proposed system incorporates AI or automated decision-making components, and if so, providing specified information about training data, accuracy metrics, known limitations, and the existence of independent audit documentation.
2. Adopt an Algorithmic Impact Assessment requirement for public sector AI deployments. Based on Canada's Directive on Automated Decision-Making, each CARICOM member state should require government agencies to conduct an Algorithmic Impact Assessment before deploying any AI system that influences decisions about individuals. A standardised Caribbean AIA template should be developed by the CARICOM AI Governance Council.
3. Require ISO/IEC 42001 certification or equivalent as a condition of contract for AI vendors supplying Caribbean governments. Vendors that cannot demonstrate compliance with the international AI management systems standard should not be awarded contracts for AI-incorporating systems above high-risk thresholds.
4. Establish a public register of government AI systems in each CARICOM member state by the end of 2026. The register should include, for each listed system: the name and description, the agency using it, the types of decisions it influences, a plain-language explanation, the date of most recent bias audit, and contact information for citizen concerns.
5. Insert mandatory human review provisions into all AI procurement contracts for systems affecting individual citizens. Every AI system used in public service delivery that produces outputs affecting individual citizens must provide a genuine mechanism for human review of adverse automated outputs.
6. Conduct a retroactive audit of AI systems procured during the COVID-19 pandemic emergency procurement period. Systems acquired without standard tendering processes between 2020 and 2022 that remain in operational use should be subjected to the new procurement standards within twelve months of those standards' adoption.

Conclusion

Government AI procurement is not a technical backwater. It is the point at which the state's power over its citizens intersects with algorithmic systems that neither the procuring official nor the affected citizen fully understands. The international frameworks, including the EU AI Act's high-risk provisions, Canada's Directive on Automated Decision-Making, and ISO/IEC 42001, have already established the principles that effective government AI procurement requires. Caribbean governments do not need to invent these principles; they need to adopt, adapt, and implement them.

The takeaway is unambiguous: no Caribbean government should purchase another AI system for use in public services without an AI Disclosure Declaration, an Algorithmic Impact Assessment, and a mandatory human review provision in the contract. These are not aspirational standards for a future regulatory environment. They are the minimum that accountable governance requires today.

About the Author

Adrian Dunkley is a Caribbean AI governance expert with extensive experience in legal and regulatory framework analysis, legislative gap analysis, and policy reform recommendations in AI governance, digital technologies, data protection, and human rights law. He advises Caribbean government institutions and regional bodies on AI policy and has worked across Jamaica and the wider CARICOM region on digital economy development. Adrian is a co-founder of StarApple AI, the Caribbean's first AI company, and founder of AI Jamaica. Contact: insights@starapple.ai