Cohere's Model Vault beta runs inference in attested, encrypted H100 enclaves on Azure

Cohere locks Model Vault inference inside confidential hardware

Cohere has opened a public beta of confidential computing for Model Vault, its dedicated, fully managed model inference service. In a launch post dated 22 October 2026, the company said prompts, documents, responses and the model's own execution are now protected while inference is running, inside hardware-isolated environments on NVIDIA H100 GPUs in Microsoft Azure. Customers can check cryptographic evidence about that environment before any sensitive data leaves their own systems.

The pitch is aimed at work that companies have kept away from cloud AI: fraud investigations in financial services, privileged legal documents, sensitive public-sector information, patents and research, engineering design documents and proprietary source code. Cohere's argument is that a managed AI service should not need to see a customer's plaintext data in order to run a model on it.

Cohere's chief technology officer, Phil Blunsom, summed up the change in the post as moving from "trust us" to "verify it yourself". The post also carries statements from Vikas Bhatia, head of product for Azure confidential computing at Microsoft, and John Fanelli, vice president of enterprise software at NVIDIA, the two partners whose hardware and cloud the beta runs on.

Data in use is the gap it targets

Encryption has traditionally covered two states of data: stored on disk and moving across a network. The third state, data in use, is the hard one, because a processor normally needs plaintext to compute on it. That is the moment when a cloud operator, a compromised host or an over-privileged administrator could in principle read what a model is working on.

Confidential computing answers this with hardware isolation. In a technical companion post published the same day, Cohere explains that the model-serving stack runs inside a confidential virtual machine, so the operating system, drivers, inference server and their memory are walled off from the host. Its documentation names the CPU technologies as confidential VMs such as Intel TDX or AMD SEV-SNP.

The GPU is the part that makes this an AI story rather than a generic cloud one. Most of a model's computation runs on the accelerator, so protecting only the CPU would leave model state exposed the moment work moved to the GPU. With NVIDIA's confidential computing mode, Cohere says, GPU memory is protected and traffic between the CPU and GPU is encrypted, so the protected boundary covers the chip doing the inference. Cohere adds that the workload is restricted by policy so only approved software can run, and that operators have no interactive path into it.

How a request is protected on the way in

Protecting the machine is only half of the problem; the request still has to get there. Ordinary TLS encrypts the connection, but Cohere notes that this alone does not keep every intermediary in a managed service, such as load balancers and routing layers, out of the plaintext path.

Model Vault therefore encrypts the application payload itself using Oblivious HTTP, or OHTTP. Each request is encrypted to a public key tied to the attested confidential environment, travels through Cohere's infrastructure as ciphertext, and can only be decrypted by a gateway inside that environment. Responses are protected on the way back the same way. The routing layer can still do its job without being able to read what it routes.

For developers, the change is meant to be small. According to the Encrypted Vault docs, encrypted vaults are created through the same flow as standard ones and called with the same API. The one addition is a Cohere OHTTP proxy on the client side, available either as a standalone proxy in the customer's environment or as a Python package that wraps the client's transport layer. Either form checks the deployment and encrypts each request before it leaves the customer's machine, so application code is supposed to stay unchanged.

Attestation decides whether data is sent at all

The feature Cohere leans on hardest is remote attestation. The confidential environment produces signed evidence describing what is running, including measurements of the hardware, firmware, operating environment and application software. Customer-side software checks that evidence against an approved policy, which can include whether the GPU is actually in confidential mode.

What makes this more than an audit log is that it changes behaviour. If a measurement has changed or the GPU is not in the required mode, verification fails and the client refuses to send the protected request, so the data never leaves the customer's environment. Cohere says the evidence is renewed frequently and rechecked each time. The docs add that every inference response carries an attestation certificate that can be checked per request, and that a verification badge in the Model Vault app reflects the result.

Cohere says it intends to open-source the confidential inference stack so customers and researchers can inspect the code, understand the policies being enforced and reproduce the measurements used in verification. The current system builds on the open-source Kata Containers and Confidential Containers projects. Until the stack is published, customers are checking measurements against reference values whose underlying code they cannot yet read.

Which models and hardware are in the beta

The beta runs on NVIDIA H100 GPUs on Azure, and Cohere says it is working toward NVIDIA Blackwell hardware, including B200 GPUs, on the way to general availability. The supported models page lists Embed, Rerank, Parse, the Compass bundle of those three, North Mini Code, Cohere Transcribe and customised Cohere models. Cohere's large Command generative models are not on that list, and the page warns that availability can differ from standard vaults while support expands.

That list fits Cohere's focus on enterprise search. Its Rerank 5 retrieval model launched a day earlier, and retrieval, parsing and transcription are where sensitive documents most often enter an AI pipeline. Cohere has also been building out security work, including its plan to acquire the AI bug hunter Aisle and launch Cohere Cyber.

On price, the pricing page says encrypted vaults are priced separately from standard ones because of the confidential hardware, that the beta is free for design partners who contact Cohere to join, and that full pricing will come as the product matures. So although the launch post says customers can start deploying today, in practice access runs through Cohere's sales team for now.

What Cohere says it does not protect

Unusually for a launch, the technical post spends real space on limits. Cohere says confidential computing makes trust smaller and measurable rather than removing it. The hardware, firmware, measured software, reference measurements and the attestation service itself all remain trusted.

Some physical and side-channel attacks are outside the boundary, such as inferring information from timing or other shared hardware behaviour. An operator that cannot read a workload can still stop or delay it, so there is no availability guarantee. Prompt injection, insecure tool use, authorisation mistakes and other application bugs are not fixed by running inside protected hardware.

Metadata stays visible too. Request timing, message size, traffic volume and routing information can be observed, and in the current design the API credential and model identifier sit outside the encrypted payload so requests can be authenticated and routed. Cohere also says the feature is not a data-residency control, not a substitute for contracts and compliance work, and not homomorphic encryption: the model still computes on plaintext, just inside a protected box.

Where this fits in the confidential AI race

Cohere is not alone in selling verifiable privacy for AI. NVIDIA recently introduced confidential bare-metal servers that let AI labs lock their models on rented GPUs, which protects the model owner's weights from the cloud host. Cohere's beta points the same technology the other way, at the customer's data. On the consumer side, Brave has offered end-to-end encrypted chats in its Leo assistant backed by trusted execution environments.

For Cohere, the product supports a long-running strategy of winning regulated buyers that are wary of sending data to big model labs. Its North agent platform and private deployments target the same banks, insurers, governments and telecoms. Confidential computing gives those customers a middle path between running models entirely on their own hardware and trusting a cloud provider's word.

What to check before signing up

Security teams evaluating the beta should ask for the exact attestation policy, the reference measurements and how often they change, and what happens to running applications when a measurement is rotated. They should confirm which model they need is on the encrypted list, which Azure regions are available, and how audit logs and zero data retention are handled, all of which Cohere's docs route to its sales team or to separate pages.

It is also worth separating vendor claims from what has been independently verified. Every security property described here comes from Cohere's own posts and documentation, and there is no third-party audit or published source code yet. Until the promised open-source release, the strongest guarantee customers get is cryptographic proof that the expected hardware and measured software are running. That is real progress over a policy document, but it is not the same as being able to read the code.