AI tips for device developers from Medtronic partner Cosmo Intelligent Medical Devices

When Medtronic needed artificial intelligence expertise to develop the GI Genius system, it went to Cosmo Intelligent Medical Devices.

The Cosmo Pharmaceuticals division worked with the world’s largest medical device company to bring the GI Genius intelligent endoscopy module to market, helping physicians spot polyps that they might have otherwise missed during colonoscopies. Cosmo recently announced FDA clearance for the latest version.

In a Medical Design & Outsourcing interview, Cosmo Intelligent Medical Devices President Nhan Ngo Dinh and Science, AI and Data SVP Andrea Cherubini detailed how Cosmo cranked up the cancer-catching AI behind Medtronic’s GI Genius ColonPro and offered lessons about innovating with speed and a startup mentality.

Now, in the third and final part of this interview with Ngo Dinh and Cherubini, they share AI tips and advice that could be useful for other device developers of all sizes.

(Editor’s note: While MDO prefers live interviews to offer readers medtech expertise without the filter of marketing or public relations staff, we conducted this Q&A by email through a Medtronic representative because Cherubini and Ngo Dinh live and work in Italy. The following has been lightly edited for space and clarity.)

MDO: What are some best practices for consistent labeling when expanding data sets?

Cherubini: “To ensure consistent labeling in our data sets, it is important to adopt several best practices. We developed in-house data annotation expertise and never outsourced this vital task, maintaining control over quality and consistency. Furthermore, we developed our own software to annotate images and videos, which allows for tailored solutions and better integration with our systems. Clear and detailed labeling instructions are fundamental, providing annotators with explicit criteria and illustrative examples for each category. Regular training sessions and workshops help annotators adhere to these guidelines. Close collaboration with a team of gastroenterologists with various levels of expertise is crucial, facilitated by regular meetings between physicians, data annotation specialists, and AI engineers to resolve doubts and share different needs and expertise. A robust quality control system, including inter-annotator agreement metrics and random audits, detects and corrects inconsistencies. Finally, we maintain an internal knowledge base and guidelines that are continuously updated to reflect best practices.”

MDO: What is labeling drift, and how can you prevent it?

Cherubini: “Labeling drift is a phenomenon where the criteria and consistency of data labeling change over time, potentially degrading the performance of AI models. The best practices for consistent labeling we previously described are crucial in preventing labeling drift. These include establishing detailed and explicit annotation guidelines, supported by regular training for all annotators. Employing a multi-tiered quality control system, including cross-validation and random audits, helps maintain high standards of labeling accuracy. In-house development of data annotation software ensures that the tools used are perfectly suited to the task, further reducing the risk of inconsistencies. Regular collaboration and feedback loops involving data annotation specialists, AI engineers, and gastroenterologists ensure that any potential drift is identified and addressed promptly.”

MDO: Do you have any advice for software developers who want to work with the Cosmos Innovation Center on apps for the GI Genius OS?

Ngo Dinh: “Developers looking to collaborate with the AI Access Program should be aware that the onboarding process is highly structured, aimed at reducing the time-to-market for software as a medical device (SaMD). The process starts with a detailed questionnaire that evaluates your expertise across various domains, including science, software, data, compliance, regulatory, and hardware. Focus on key aspects such as the safety and effectiveness of your AI algorithm, benefit-risk ratio, data privacy measures, and regulatory compliance.

“Perform a gap analysis to pinpoint areas where your capabilities may be insufficient and work on enhancing these aspects. Engage in regular communication with the AI Access Program team and healthcare professionals to gather insights and feedback. Leverage the technical documentation and Sandbox testing environment provided by the AI Access Program to test and optimize your application, ensuring it meets the clinical and technical requirements of the GI Genius OS. This strategy will help you develop robust, compliant, and effective solutions that meet the high standards expected in medical software.”

Related: New features of updated Medtronic GI Genius include room for third-party SaMD developers

MDO: What are some technical tips you can share with other device developers that want to use neural networks/machine learning/deep learning to improve patient care?

Cherubini: “When developing AI-enhanced medical devices, integrating domain expertise is crucial. Collaborate closely with medical professionals to understand clinical nuances and tailor your models accordingly. Ensure your datasets are comprehensive, well-labeled, and representative of the patient population. Invest in robust data preprocessing methodologies to clean and normalize the data and incorporate cross-validation and rigorous testing to prevent overfitting and ensure generalizability.

“Prioritize patient data privacy and security, ensuring compliance with regulations like HIPAA and GDPR. Make sure your models are explainable, providing clear rationale for their predictions to support clinical decision-making and maintain transparency and trust among clinicians. Develop intuitive user interfaces that allow clinicians to interact with and understand the AI’s suggestions easily and effectively.

“Implement robust validation frameworks, including external validation with diverse datasets, to confirm model performance across different populations. Use real-world feedback to iteratively refine your models, ensuring continuous monitoring and updating to adapt to new data and evolving clinical guidelines. Finally, engage directly with the AI scientific community to stay updated with the latest advancements in AI research and integrate these innovations to keep your solutions at the forefront of technology.”

MDO: Is there anything you wished we would have discussed but haven’t in this interview?

Ngo Dinh and Cherubini: “One major lesson learned during our journey is the importance of transparency and accountability. Developing AI for healthcare isn’t just about making something that works; it’s about ensuring it’s safe, effective, and built on solid, ethical foundations. This means being open about how the AI is developed, rigorously testing it, and following guidelines to make sure it does more good than harm.

“Another key point is the critical role of communication between AI developers and healthcare professionals. For AI to be truly useful, doctors need to understand it and trust it. This involves a lot of education and collaboration, making sure the technology fits seamlessly into their workflows and addresses their real-world needs. It’s not just about throwing tech at a problem, but about creating a tool that clinicians are comfortable using and that genuinely enhances their ability to care for patients. Indeed, it is fascinating how AI can complement human skills, making the combination of a doctor’s expertise and AI’s analytical power greater than either alone. For this to work, the AI needs to communicate clearly, helping doctors understand its predictions and limitations so they can make the best decisions.

“Looking ahead, AI can handle complex medical information and assist in ways we’re only beginning to imagine, potentially transforming everything from diagnostics to patient care. The key is to balance innovation with responsibility, ensuring clear communication and strong collaboration between all parties involved. This way, AI can truly become a valuable partner in medicine, enhancing both the efficiency and quality of patient care.”

Read more from our interview with Cosmo’s Nhan Ngo Dinh and Andrea Cherubini:

• How Cosmo cranked up the cancer-catching AI behind Medtronic’s GI Genius ColonPro
• Innovating with speed and a startup mentality: Cosmo leaders offer lessons from their Medtronic AI project