California Senate Bill 903, currently advancing through the state legislature, would prohibit chatbot developers from advertising their products as therapy and impose new disclosure requirements on AI tools deployed in formal mental health settings. The proximate cause is straightforward: chatbot usage for mental health advice has grown dramatically over the past two years, and regulators are responding to documented cases where users treated AI responses as clinical guidance. The bill is, on its face, a consumer protection measure. But the governance logic embedded in SB 903 reveals something more interesting, and more troubling, than its sponsors likely intended.

The Competence Assumption in Regulatory Design

Every governance framework rests on assumptions about what users already know. Classical consumer protection law assumes an informed party who can, in principle, distinguish a product claim from a clinical diagnosis. What SB 903 implicitly acknowledges is that this assumption has collapsed in AI-mediated communication contexts. Users interacting with conversational AI in moments of psychological distress are not operating as informed consumers evaluating product claims. They are engaging with what Hancock, Naaman, and Levy (2020) called AI-mediated communication, where the interface layer actively shapes the user's perception of the relationship itself. The chatbot does not just deliver information. It simulates a communicative role, and users respond to that simulation with the schemas they have for that role, not the schemas they have for software products.

This is precisely what makes SB 903 both necessary and insufficient. The bill targets advertising claims, which is the topography of the problem. The topology, the underlying structural shape of the issue, is that users lack schemas for distinguishing conversational competence from clinical competence. Banning the word "therapy" in a marketing headline does not install that schema. It removes one misleading signal without addressing the signal-processing deficit that made the misleading signal dangerous in the first place.

Awareness Without Capability

The research literature on algorithmic literacy has documented a consistent and uncomfortable finding: awareness of how a system works does not produce improved outcomes when interacting with that system. Gagrain, Naab, and Grub (2024) formalize this as the awareness-capability gap. Users can be told, and can accurately report, that an AI chatbot is not a licensed therapist. That declarative knowledge does not reliably change how they process the chatbot's responses during an emotionally activated interaction. Sundar (2020) provides a complementary mechanism here, noting that machine agency cues, specifically the experience of a system that responds, adapts, and appears to understand, trigger social processing heuristics that override analytical evaluation. The disclosure that SB 903 requires will be read cognitively at sign-up and ignored affectively during use.

This is not an argument against disclosure requirements. It is an argument that disclosure requirements address the wrong layer of the problem. The governance intervention that would actually move outcomes is schema induction, training users to recognize the structural difference between a system optimized for response fluency and a system accountable for clinical accuracy. These are not the same competence, and the former can mimic the latter with high fidelity. Gentner's (1983) structure-mapping theory predicts that surface similarity, in this case, conversational warmth and apparent understanding, will dominate structural dissimilarity, the absence of licensure, training, and accountability, when users lack schemas that organize around the structural dimension.

What This Reveals About Platform Governance More Broadly

The California legislature is doing what legislatures do: responding to visible harm with visible intervention. I do not fault the pragmatics of this. But SB 903 is a case study in what happens when governance frameworks inherit the competence assumptions of classical communication regulation and apply them to algorithmically-mediated environments where those assumptions no longer hold. Kellogg, Valentine, and Christin (2020) argued that algorithmic work environments generate forms of dependence that existing regulatory categories were not designed to address. AI therapy chatbots are an extreme instance of this: the dependence is not economic but epistemic, and the harm is not labor precarity but clinical substitution.

The more productive regulatory question is not "what can companies claim?" but "what do users need to understand structurally about this class of system before they encounter it in a vulnerable state?" That is a schema-induction problem, not a disclosure problem. California has identified the right domain. The bill as written operates at the wrong level of the governance stack.

References

Gagrain, A., Naab, T., & Grub, J. (2024). Algorithmic media use and algorithm literacy. New Media & Society.

Gentner, D. (1983). Structure-mapping: A theoretical framework for analogy. Cognitive Science, 7(2), 155-170.

Hancock, J. T., Naaman, M., & Levy, K. (2020). AI-mediated communication: Definition, research agenda, and ethical considerations. Journal of Computer-Mediated Communication, 25(1), 89-100.

Kellogg, K. C., Valentine, M. A., & Christin, A. (2020). Algorithms at work: The new contested terrain of control. Academy of Management Annals, 14(1), 366-410.

Sundar, S. S. (2020). Rise of machine agency: A framework for studying the psychology of human-AI interaction. Journal of Computer-Mediated Communication, 25(1), 74-88.