Introducing AI-powered Clinical Notes API for Telehealth

Clinical notes drafts in SOAP, DAP, and other formats, generated by a HIPAA-compliant Large Language Model

Authors: Kwindla Hultman Kramer, Nina Kuruvilla, Corey Hobbs

At Daily we provide global infrastructure and developer SDKs for high quality video and audio communications. We’ve served telehealth customers from our earliest days and are proud of our long list of industry firsts. We were the first WebRTC developer platform to offer a full suite of HIPAA-compliant APIs, the first to support Safari and iOS for one-click, no-download video calls, and the first platform to offer an integrated customer support dashboard with complete application logs and WebRTC metrics.

Today, we’re announcing another first: clinical notes generation seamlessly integrated into telehealth developer APIs, powered by today’s most advanced AI tools.

AI that gives healthcare providers time back, every day

At Daily, we’ve been working with healthcare developers since 2016. As telehealth growth has accelerated over the past few years, we’ve focused on creating building blocks that our customers can use to make clinical care more accessible, reliable, and human-centered.

Telehealth plays a crucial role in care delivery today. Yet telehealth faces a challenge shared across healthcare: clinical documentation is massively time-consuming. The amount of time required to write clinical notes takes time away from providing care to patients and contributes to clinician burnout, staff shortages, and impacted patient care.

Today’s new AI Large Language Models (LLM) can help change this.

Daily now gives telehealth developers the ability to:

Generate a transcript from a recording of any telehealth session built on Daily
Generate AI-powered SOAP notes with a single HIPAA-compliant REST API call
Deploy a polished, standalone “notes dashboard” UX, to let providers interact with SOAP notes
Rapidly explore and ship custom LLM capabilities and structured medical data workflows on HIPAA-compliant infrastructure

We’ve never seen more excitement about a new feature than we’ve seen for the early prototypes of AI-powered clinical notes generation that we’ve been sharing with early customers. These APIs are available today to developers building video experiences on Daily.

Below we’ll discuss how we built the infrastructure enabling these features, what’s in today’s release, and what’s ahead on our roadmap. We’ll close with resources.

Data privacy, accuracy, and human-centered design

We built these new clinical notes APIs as extensions of Daily’s existing HIPAA-compliant, privacy-first architecture. All PHI is stored, transmitted, and processed either within Daily’s own cloud infrastructure or by partners that are also HIPAA-compliant. Daily signs BAAs and takes patient privacy extremely seriously. Data geo-fencing is available, and we have the capacity to work with product engineering teams to provide additional advice and architecture reviews of security and data privacy where that support is helpful.

In addition to data privacy, we also strongly emphasize accuracy, reliability, and ease of use in all of our infrastructure and product work at Daily.

Because we manage the audio streams end-to-end on infrastructure built from the ground up to deliver high-quality video and audio to any device anywhere in the world, we can provide the most accurate possible transcription of a session. We have had transcription APIs at Daily since 2020 and we continue to track the state of the art as transcription technologies advance. Recent industry improvements in word error rate and multilingual capabilities are impressive — and important. Accurate transcription is the “input” to automated drafting of clinical notes. The better the transcription, the better the “output.”

To deliver the best possible output, we’ve tested a wide range of AI models and supporting tools. Today, we’re producing industry-leading clinical notes text by combining Deepgram’s highly capable transcription models, ScienceIO’s healthcare-specific AI for processing medical data, and the Microsoft Azure OpenAI GPT-4 large language model. We’ve worked closely with all three teams to bring this new capability to market and to ensure HIPAA-compliance and data privacy.

Finally, in the context of clinical notes, we believe that designing workflows around the principle of having a “human in the loop” is critical. We’ve designed our APIs to generate a first draft of clinical notes, in a format that is easy to cross-check against a transcription. In many cases, this draft will require little or no revision. But the primary care provider always owns the workflow and makes the decisions.

While the promise of AI is real, the scope of things that AI can do today in a primary care context is quite limited. However, today’s cutting edge AI tools can turn an unstructured transcript into a structured, formatted summary, with key medical terms highlighted, and with patient history and action items laid out clearly. Relying on AI to create a draft of clinical notes — for approval or editing by the provider — can save a truly significant amount of time each day.

A complete set of telehealth building blocks

Daily’s AI-Powered Clinical Notes are part of our HIPAA offering, and like all of our APIs, support several ways to build telehealth solutions. Developers can:

Use our HIPAA-compliant Video SDKs in Web, iOS, and Android applications to build custom user interfaces and video experiences,
Get started quickly with our React Components Library and native mobile sample apps (iOS/Android), or
Add HIPAA-compliant, secure, one-click video to any web application with just a few lines of code using Daily Prebuilt

Expanding our telehealth offerings with AI functionality, today we are making generally available:

REST APIs to generate a transcript from a recording of any Daily telehealth session
REST APIs to generate clinical notes, in SOAP format
An optional, standalone “notes dashboard” UX for care providers, as shown below.

Our goal is to allow our customers to deploy AI-generated clinical notes in a variety of ways: fully integrated into an existing product workflow via our REST APIs; or as a new hosted component, with a polished UX, that requires minimal engineering work to roll out.

Below is an example of a transcript and the associated clinical notes draft.

The optional, hosted clinical notes dashboard gives providers access to SOAP notes. With just a few lines of code, developers can customize the dashboard UI and integrate it into any app workflow so that:

Users can authenticate via SSO
Users can review SOAP notes alongside the associated transcript
Users can copy the SOAP notes text for pasting into an EHR/EMR
Developers can customize the dashboard UI

Source code for the dashboard app is available.

Where this technology is headed

Our early test users have provided both very positive feedback and offered a long list of suggestions for improvements and additional features. Here are some of the things we’re working on.

Customizable output formats. We started with SOAP notes because a critical mass of our user base requested the format. We can output a variety of text formats, including DAP notes, BIRP notes, and completely custom structures. In fact, the architecture that underpins our AI clinical notes is designed to provide workflow flexibility. We are working with early adopter customers on specific use cases that benefit from adding enrichment steps, using different LLMs and prompts, and incorporating post-processing tooling that modifies the output from the LLM.

Support for languages other than English. Multilingual transcription is available today, and we’re seeing promising early results in our testing of LLM-enabled translation into English during clinical notes generation.

Instant availability and streaming data. We’re reducing output time down below one minute, and also working on making the output of our AI workflows available as streaming data during a session. Daily can help you implement streaming LLM input/output today with semi-custom development and infrastructure. But live interactions with LLMs are useful enough for a wide variety of clinical use cases that we expect to ship standard, simple APIs that give you access to healthcare-specific LLM enrichments during a video session.

If any of these are particularly important for your use case, please let us know. We like to work closely with customers to bring new capabilities to market. We can often provide early access to features that we are testing, plus help integrate these features into your application.

Partnering with developers to leverage AI

At Daily, our customers span a wide range of industries and use cases. In addition to healthcare, we power education apps, events platforms, games, social video experiences, sports broadcasting, live commerce, and much more. Across almost our entire customer base, there’s both excitement and uncertainty about the capabilities that new generative AI technologies offer.

We bring our unique expertise in video, audio, and real-time infrastructure to this new world of AI. As the possibilities of these new tools come into focus, we’re delivering APIs that make it possible to leverage them, along with guard rails that help you use them in safe, principled ways and in compliance with data privacy regulations.

We’re here to help, so if you’re interested in learning more, or have thoughts to share, please reach out. Contact sales for a demo or to discuss your enterprise needs. Developers can view our HIPAA guide, sign up for a free developer account, or visit our peerConnection forum.

Finally, we also are thrilled to partner this week with the American Telemedicine Association as a sponsor of its Telehealth Awareness Week. Sit in on a demo session with Daily Director of Product Corey Hobbs as he looks at this new functionality. Our CEO Kwindla Hultman Kramer also is continuing the conversation with our roundtable, AI-powered Automation Technology in Telehealth. We’re looking forward to keep building and supporting our healthcare customers.