If you are building or thinking about using an application based on OpenAI’s ChatGPT or another large language model (LLM) which will collect or otherwise access health information, you may have to comply with some of the strictest data protection laws currently in force. If your business offers its services in the US, this Mobile Health App Interactive Tool can give you a first glimpse of the complexity of the legal landscape you’ll have to navigate. It is not feasible to cover all the compliance requirements in this blog post. Instead, we focus on themes that are particularly important in the context of healthcare apps built on LLMs such as ChatGPT.We first provide some examples of healthcare-related apps currently being developed based on LLMs and then address the following privacy concerns around health data:
- Transparency;
- Control over use; and
- Access.
We then look at the current privacy posture of OpenAI to determine whether and how these concerns can be mitigated.
Health Applications Based on ChatGPT
Research regarding AI applications assisting with image-based diagnosis, and big data analytics for public health purposes, for example, has been ongoing for over a decade. Considerable improvements have been made in terms of the accuracy of diagnoses and the technology even gave rise to “precision public health.” But what do LLMs such as ChatGPT bring to the table in this context? Arguably, the biggest change they will introduce is the way in which patients or healthcare service providers and this technology interact. Examples range from chatbots providing assistance with administrative tasks to medical diagnosis and treatment plans.Doximity’s DocsGPT uses OpenAI’s model to generate faxes to insurers, saving doctors time on repetitive administrative tasks. The company offers a HIPAA compliant environment in addition to the free digital fax service. Customers need to enter into a Business Associate Agreement with Doximity in order to gain access to that environment.Abridge has developed an AI-driven medical transcription service based on ChatGPT that listens to patient-doctor visits and summarizes the important points. It has already been deployed to over 1,500 physicians.Glass Health is developing an experimental feature that generates a differential diagnosis or clinical plan to be used by clinicians who would input a one-liner presenting a diagnostic problem.Google Health’s Med-PaLM2 is excelling at answering medical questions. Researchers are exploring how this model can be used to assist in research, draw insights from unstructured medical texts, and provide summaries of internal data sets.
Transparency
One concern with using LLMs to assist with healthcare services, particularly when diagnosis or treatment recommendations are concerned, lies in the “black box” problem. This term describes the fact that it can often not be known, neither by clinicians nor by the model developers, how an LLM arrived at the output it provided. From a data privacy perspective, this could be problematic because patients can have a right under privacy laws to obtain an explanation of an “automated decision” made on the basis of their personal data.Under the GDPR, for example, this right only applies to decisions made exclusively based on automated processing, that is, without human intervention. While you might hope that a physician would check the screening decision of a diagnostic AI tool, that is not necessarily required in all cases.In addition, under the proposed revisions to the Canadian private sector privacy act it would suffice for the decision to be aided by automated processing of personal information to give rise to this right. In the absence of clarity on how the decision regarding a treatment, for example, is arrived at, this right cannot fully be given effect.
Control Over Use
While lots of the research regarding the use of AI in healthcare is undertaken by academic institutions, the development of the AI is, for the most part, driven by large for-profits in partnership with public health institutions. Such partnerships are at risk of becoming lopsided with public health institutions becoming more and more dependent upon the technical expertise of AI companies. And with profit motives coming into play, privacy considerations are often weighed against the financial benefits that may result from using the data in ways that individuals have not been informed of or consented to. Given that LLMs require large amounts of data for training and in light of the sensitivity of health information, this issue is significantly amplified in the context of healthcare-related applications built on such LLMs.
Access
ChatGPT’s access to health data must be carefully monitored and controlled. Healthcare providers must be aware that the privacy compliance requirements imposed upon them under the law also apply to third-party service providers to whom healthcare data is disclosed, and that it is their responsibility to ensure compliance by the third party. If a tech company builds an app that provides healthcare services based on ChatGPT, for example, OpenAI would be considered a fourth-party service provider and would also have to be compliant with applicable laws and regulations.The GDPR as well as HIPAA prescribe contractual clauses that are non-negotiable and backed up by penalties that are likely painful enough to motivate compliance. Tech companies building ChatGPT-based applications can be required to enter into such contracts which also restrict under which circumstances the health data may be further transferred, e.g., to OpenAI. Among other things, strict security measures including access controls are required under such contracts.
HIPAA Compliance of OpenAI
An entity that performs certain functions or activities that involve the use or disclosure of protected health information (PHI) on behalf of, or provides services to, an entity to which HIPAA applies, is called a “business associate.” As mentioned, OpenAI recently amended its FAQs to state that it is willing and able to enter into a Business Associate Agreement (BAA), which is a requirement under HIPAA for a healthcare service provider covered under HIPAA to disclose any PHI to a business associate. Without such a contract in place, no PHI should be entered into ChatGPT. The only alternative which would allow disclosing information to ChatGPT is to comply with the HIPAA Safe Harbor rule, requiring that the PHI is deidentified by removing a prescribed list of identifiers. Read more about the Safe Harbor requirements here.The US Department of Health and Human Services advised that a covered entity is not liable for the actions of its business associates and is not required to monitor the business associate’s compliance with the contract. Only upon positive knowledge of a violation is the covered entity required to cure the breach, end the violation, and as a last resort, terminate the contract. If that is not possible because of a lack of viable business alternatives, the covered entity must report the issue to the Department of Health and Human Services Office for Civil Rights.The business associate, in turn, is directly liable for compliance with certain HIPAA provisions. So, if a tech company provides services to a covered entity and it does so using ChatGPT, the tech company is responsible for impermissible uses and disclosure of PHI, and to implement necessary safeguards, for example. The business associate must also enter into BAAs of their own with their subcontractors, which may be OpenAI.
Further Best Practices
In general, it is best to limit the amount of personal information transferred to what is absolutely necessary and, indeed, it is a requirement of the GDPR under its ‘data minimization’ principle. Limiting the transfer of personal information, including PHI, limits the vulnerability of the data and allows for more user control of their data – the fundamental purpose of privacy.
Conclusion
In conclusion, as service providers build applications based on ChatGPT, addressing healthcare compliance issues becomes paramount. Transparency, control of use, and access, along with adherence to HIPAA's Business Associate Agreements, are vital for ensuring ethical and responsible utilization of these AI tools. However, considering the current uncertainty surrounding OpenAI’s privacy compliance, it is be advisable to implement additional measures, such as utilizing PrivateGPT. This solution allows PII and PHI to be redacted before it’s shared with ChatGPT, adding an extra layer of protection. The response is then re-identified before coming back to the user for a seamless user experience without sacrificing privacy. By embracing such proactive measures, service providers can navigate compliance challenges, mitigate risks, and bolster patient trust in AI-powered healthcare applications.
