“Why isn’t Google a medical device and why might Claude be one?”

With that question, Dr Bilal Mateen, Chief AI Officer at PATH, opened his keynote at The Conduit’s recent Co-Lab on AI and the Future of Mental Health. Over two hours, speakers and attendees wrestled with how to balance AI’s potential to offer new forms of support for mental health with the need to ensure that innovation remains safe, equitable, and ethical.

AI already sits in people’s pockets, offering guidance at 3 a.m. when anxiety peaks and the impulse to reach for a judgment-free listener feels irresistible. For many, the appeal lies in this anonymity. But as AI becomes more deeply embedded in how people seek care, regulation must be seen not as a constraint on innovation but as protection for those who are most vulnerable. As Mateen put it, “Regulation exists to protect those who are vulnerable and we have a real problem because not enough people understand this distinction.”

Regulation Exists, But It’s Misunderstood

Mateen explained that the question of what counts as a medical device is not as ambiguous as it might seem. The distinction lies in two criteria: purpose and functionality.

If a digital tool is intended to support or treat a medical condition, it becomes a medical device regardless of how it is marketed. A wellbeing app that helps users relax or sleep can fall outside regulation. However, the moment it claims to treat sleep issues as a result of a medical condition like anxiety or depression, it crosses into medical territory.

He pointed to Sleepio, developed by Professor Colin Espie at the University of Oxford, as a clear example. The program functions as a wellbeing tool for sleep support, but it also holds medical device approval when prescribed to patients whose poor sleep is linked to clinical depression. That dual use underscores how regulators draw the line between wellness and healthcare. 

Regulators can also consider how users engage with a tool, not just what companies disclose. If people are using an AI system as a therapist then it effectively operates as one.

Yet many apps that claim to cure depression or anxiety still insist they are not subject to medical device regulation. That gap between functionality and accountability leaves millions of users exposed to tools that are effectively unregulated, with no assurance of quality, efficacy, or safety.

Three Gaps Holding Back Safe Innovation

Mateen outlined three urgent challenges that health systems and regulators must address to ensure that AI innovations do more good than harm:

1. Logging and Post-Market Oversight
   There is currently no systematic way for health systems in the UK to log which AI tools clinicians recommend or prescribe. Without this visibility, regulators cannot track how technologies perform in practice or learn from emerging risks and benefits.
2. Real-Time Evaluation
   The absence of real-world monitoring means regulators often learn about problems only after harm occurs. Mateen cited a simple but haunting example: a digital assistant telling a distressed person where the nearest bridge is, rather than directing them to crisis support. These “near misses” highlight the need for systems that can detect risk in real time.
3. Adaptive and Personalized Interventions
   Unlike medicines or traditional devices, AI evolves continuously. This adaptability, while powerful, breaks the static models that regulators rely on for approval and oversight. Current frameworks weren’t built for technologies that learn, update, and change with every user interaction.

Together, these challenges illustrate how innovation is outpacing the infrastructure designed to keep it safe. This is why regulatory frameworks must evolve as quickly as the tools they oversee.

Balancing Innovation, Trust, and Equity

Across the room, discussion turned to how to balance progress with public trust. Participants agreed that confidence in AI for mental health depends less on algorithms and more on the institutions and people who stand behind them.

AI’s immediacy and anonymity make it accessible, especially for those hesitant to seek therapy, but emotional safety depends on transparency and accountability. Ethical adoption cannot rest with developers alone, it requires collaboration across healthcare, academia, policy, and finance.

If safety is the baseline, participants agreed, equity must be the benchmark for progress. Many current systems still perform better on certain demographics, reflecting the biases in their data. Unequal access between public and private sectors risks deepening inequality in care.