AI vendors and HIPAA: questions to ask before an AI tool touches patient data

Why AI raises old HIPAA questions in new places

From HIPAA's point of view, an AI model is software that receives, processes and sometimes stores data. A vendor that runs AI on your patients' information is creating, receiving, maintaining or transmitting PHI on your behalf, which makes it a business associate under 45 CFR 160.103.

What makes AI different is the supply chain. Many AI products are built on top of a large model operated by a separate company. Your schedule export, call transcripts or patient messages may pass from your vendor to that model provider and back. Every company in that path that handles PHI needs to be covered, and the uses each is allowed must be limited.

There is one AI-specific rule worth knowing. Under the Section 1557 nondiscrimination rule, 45 CFR 92.210, covered entities must make reasonable efforts to identify and mitigate discrimination risks from patient care decision support tools, with compliance required by May 2025 according to HHS. Scheduling and messaging tools usually are not clinical decision tools, but if an AI product influences clinical decisions or triage, ask the vendor how it addresses that rule.

Questions about contracts and the data chain

1. Will you sign a Business Associate Agreement before any patient data is shared, including in a pilot or trial? If the answer is no, or only on a higher plan, stop here.
2. Which outside AI model providers receive our data, and do you have a BAA or equivalent subcontractor agreement with each? Under 45 CFR 164.502(e), your vendor must get written assurances from its subcontractors. Ask for the list.
3. Is any of our data used to train, fine-tune or improve AI models, yours or anyone else's? A business associate may use PHI only as its BAA permits (45 CFR 164.502(a)(3)). If the vendor wants to use de-identified data for product development, HHS guidance indicates it may de-identify PHI only if the BAA allows it. Decide whether you want to allow that, and get the answer in writing.
4. Where is the data processed and stored? Ask whether processing, including by the model provider, happens in U.S. data centers.

Questions about what the AI actually sees

1. Which data fields reach the AI model? The minimum necessary standard, 45 CFR 164.502(b), applies to business associates. A tool that books appointments should not need diagnoses or clinical notes. Ask for a field-by-field answer.
2. Does every patient interaction go through the model, or only some? Well-designed systems handle routine replies, such as a patient texting 1 to confirm, with simple rules and reserve AI for what genuinely needs it. Fewer data trips mean less exposure.
3. Are prompts, transcripts and outputs retained, and for how long? Model providers sometimes keep inputs for abuse monitoring or debugging. Ask what is kept, where, for how long and who can read it.
4. Are call recordings or transcripts stored, and who can access them? Voice tools often generate recordings that contain far more PHI than the booking itself required.

Questions about safety and accountability

1. What will the AI never do? Look for clear limits, such as no medical advice, no discussion of results or diagnoses, and escalation to staff when a patient raises a clinical question or an emergency.
2. How are errors caught? AI systems can produce confident, wrong answers. Ask how the vendor prevents the tool from inventing appointment times, prices or policies, and how mistakes are surfaced to your staff.
3. What gets logged? The Security Rule requires audit controls that record and examine activity in systems containing electronic PHI (45 CFR 164.312(b)). You should be able to see what the AI did, when and for which patient.
4. What happens to our data when we leave? The BAA must require return or destruction of PHI at termination (45 CFR 164.504(e)(2)(ii)(J)). Ask whether that includes model provider logs and any derived data.

Reading the answers

Good answers are specific. "We use enterprise-grade security" tells you nothing. "Our model provider is covered by a BAA, does not retain inputs beyond 30 days and never trains on customer data" is the kind of statement you can verify and hold a vendor to. (That example is illustrative; ask each vendor for its own terms.)

Watch for these warning signs:

• The vendor's terms of service reserve the right to use "customer content" to improve its services, with no carve-out for PHI.
• The vendor cannot name its model providers or says the question is confidential.
• Patient data is sent to a consumer AI product or an API plan with no BAA.
• Staff are encouraged to paste patient details into a general-purpose chat assistant.
• The product markets itself as "HIPAA certified." HHS does not certify vendors.

Add AI to your risk analysis

The Security Rule requires every practice to conduct an accurate and thorough assessment of the risks to its electronic PHI (45 CFR 164.308(a)(1)(ii)(A)). New AI tools belong in that assessment. Document which tools you use, what data each one receives, which BAAs cover them and what controls limit their access.

Revisit the analysis when a vendor changes its model provider or adds new features. AI products change quickly, and a tool that only read appointment times last year may be reading call transcripts this year.