ChatOCT: embedded clinical decision support systems for optical coherence tomography in offline and resource-limited settings

Large Language Models (LLMs) are revolutionizing various aspects of our lives. Integrating LLMs into Optical Coherence Tomography (OCT) devices could significantly assist doctors by offering diagnostic and treatment recommendations, particularly in situations where medical expertise is scarce. Unlike artificial intelligence (AI) models that provide diagnostic conclusions opaquely, LLMs enhance the safety and efficacy of AI-assisted diagnosis and treatment through interactive human-computer dialogue. However, due to the imperative to protect patient privacy and ensure data security, OCT devices in healthcare settings are commonly operated offline. This limitation prevents the implementation of LLMs in an online environment, contrasting with platforms like ChatGPT or Kimi. Additionally, the operation of current LLMs demands substantial graphics memory, posing a significant challenge for OCT devices that already require considerable graphics memory for tasks such as data acquisition, rendering, and real-time display. In this paper, we introduce ChatOCT, an embedded clinical decision support system with specialized knowledge in OCT and related medical fields, designed to operate offline and with minimal computational resources. We propose a framework for its development, encompassing OCT knowledge injection, Q&A-based clinical instruction tuning, and model compression techniques. The superiority of our models, both the original and a 79% compressed version, has been independently confirmed by ChatGPT (GPT-4) and two clinical ophthalmologists. We envision that ChatOCT could significantly elevate the intelligence and adoption of OCT technology.