There is a urgent need for AI agents in healthcare. Throughout the industry, overworked teams are overwhelmed by time-consuming tasks that occupy patients’ care. The clinician stretches thinly, the payer call center is overwhelmed, and the patient is waiting for an answer to immediately follow.
Artificial intelligence agents can expand coverage and availability of clinical and administrative staff and reduce burnout among healthy people and patients by filling large gaps. But before we do this, we need to have a big foundation to build trust in AI agents. This trust does not come from a warm tone or fluent conversation. It comes from engineering.
Even with interest in AI agents and headlines touting the promise of proxy AI, healthcare leaders (responsible to their patients and communities) are hesitant to deploy the technology at scale. Startups are touting agent capabilities that range from automated mundane tasks such as appointment schedules to communication and care for high-touch patients. However, most people have not proven that these involvement is safe.
Many of them will never.
The reality is that anyone can rotate a voice proxy powered by a large language model (LLM), giving it a compassionate tone, and the scripting script sounds convincing. In every industry, there are many such platforms that have agents in every industry. Their agents look and sound different, but all behave the same – easy to hallucinate, unable to verify key facts, and lack of mechanisms to ensure accountability.
This approach – building a too-weak wrapper around the base LLM – may work in industries such as retail or hospitality, but fail in healthcare. Base models are extraordinary tools, but they are largely universal. They do not specifically train on clinical programs, payer policies or regulatory standards. Even the most eloquent agents built on these models may flow into the realm of hallucinations, answering questions they shouldn’t, invent facts, or failing to recognize when humans need to be brought into the cycle.
The consequences of these actions are not theoretical. They can confuse patients, interfere with care and lead to expensive human rework. This is not an intellectual matter. This is an infrastructure issue.
To operate safely, effectively and reliably in healthcare, AI agents need not only have autonomous voices on the other end of the phone. They must be operated by systems dedicated to control, context and accountability. From my experience building these systems, this is what it looks like in practice.
Response control may make hallucinations absent
AI agents in healthcare will not only produce reasonable answers. They need to provide the right one every time. This requires a controllable “action space” – a mechanism that allows AI to understand and facilitate natural dialogue, but ensures that all possible responses are limited by predefined, approved logic.
With built-in response control parameters, the agent can only reference verified protocols, predefined operating procedures, and regulatory standards. The creativity of the model is to guide interactions rather than improvise facts. That’s the risk of healthcare leaders being able to ensure that hallucinations are completely eliminated – not by testing in a pilot or a single focus group, but by designing risks at the bottom.
Expertise graph ensures trusted communication
The background of each medical conversation is in-depth. Two people with type 2 diabetes may live in the same community and be suitable for the same risk profile. Their eligibility for a specific drug will vary based on their medical history, doctor’s treatment guidelines, insurance plans and formula rules.
An AI proxy not only needs to access this context, but also needs to be able to reason about it in real time. The expertise graph provides this feature. This is a structured way to represent information from multiple trusted sources, allowing agents to verify what they hear and make sure the information they give back is both accurate and personal. Agents without this layer may sound conscious, but they are really just following a strict workflow and filling in the blanks.
A powerful comment system that evaluates accuracy
Patients may be with and satisfied with the AI agent, but the agency’s work is far from over. Healthcare organizations need to ensure that agents can not only generate the right information, but understand and record the interaction. That is where the automated post-processing system is located.
A powerful audit system should evaluate every conversation with the same careful review as human supervisors at all times in the world. It should be able to determine whether the response is accurate, ensure that the correct information is captured, and whether follow-up is required. If something is not correct, the agent should be able to upgrade to human, but if everything is checked, you can check the task with confidence.
In addition to these three fundamental elements to be designed, each proxy AI infrastructure requires a strong security and compliance framework to protect patient data and ensure that the proxy operates within a regulated scope. The framework should include strict compliance with common industry standards such as SOC 2 and HIPAA, but should also be built in the process of bias testing, protected health information repair and data retention.
These security guarantees are more than just checking the compliance box. They form the backbone of a trusted system that ensures that each interaction is managed at the level expected by the patient and the provider and the provider expects.
The healthcare industry doesn’t need more AI hype. It requires a reliable AI infrastructure. In the case of proxying AI, trust will not be as much as it is designed.
