The Specific Event

On a recent weekday, Google launched an AI-powered image generation feature inside Google Earth. Within hours, the company pulled it. The reason: users immediately began generating manipulated aerial views depicting fake bombings, riots, and large-scale destruction. The Washington Post documented how the feature was exploited to produce what appeared to be photorealistic scenes of violence and crisis layered onto real geographic coordinates. Google's response was swift, but the damage to the question of AI deployment governance was already visible. The gap between what the feature was designed to do and what users actually did with it was not a narrow gap. It was structural.

This Is Not a Moderation Story

The instinct when reading this story is to frame it as a content moderation failure. That framing misses something more important. Content moderation is a downstream response to a governance design problem. What Google Earth's rapid rollback actually reveals is an organizational failure to anticipate the schema mismatch between how engineers and product designers understood the feature and how users encountered it. The engineers likely held an accurate structural model of what the tool could produce. The users who immediately exploited it held a different, equally accurate structural model of the same tool. Both groups understood the topology. Only one group had institutional authority to act on it in advance.

Awareness Without Governance Structure

This distinction maps directly onto what algorithmic literacy research describes as the awareness-capability gap (Kellogg, Valentine, and Christin, 2020). In the platform labor literature, this gap refers to workers who know algorithms govern their outcomes but cannot translate that awareness into improved performance. The governance analog is organizations that know AI systems can be misused but cannot translate that knowledge into pre-deployment structural constraints. Awareness of misuse potential is not equivalent to institutional capacity to prevent it. Google's internal teams almost certainly understood that generative AI layered onto satellite imagery of real-world locations carried misuse risk. The rollback happened in hours, which suggests the harm was not surprising in retrospect. The surprise was that no structural gate existed to catch it before release.

Folk Theories at the Organizational Level

Gentner's (1983) structure-mapping theory distinguishes between surface-level feature matching and deep relational structure. When organizations reason about AI risk, they often operate from what I would call organizational folk theories: intuitive, surface-level impressions of how a feature will be used rather than structurally grounded models of the relational space of possible uses. A structural schema for this Google Earth feature would have mapped the relational logic connecting generative AI, geocoded real-world imagery, and the informational authority that satellite images carry as apparent evidence. That combination produces a specific misuse topology that is distinct from, say, generative AI on a blank canvas. The misuse that occurred was not random. It was predictable from the structural relationships between the tool's components, not from surface observation of its intended function.

The Organizational Competence Inversion Problem

Hatano and Inagaki (1986) draw a line between routine expertise, which involves executing established procedures, and adaptive expertise, which involves constructing new responses when established procedures do not apply. AI deployment governance currently operates mostly through routine expertise. Organizations apply existing trust and safety checklists, red-team exercises borrowed from prior product categories, and post-hoc moderation pipelines. These are procedural tools built for known risk topographies. Generative AI layered onto authoritative data sources like satellite imagery represents a novel risk topology. Routine expertise fails here precisely because the structural relationships that generate harm are new, not because the organization lacks experience with AI risk in general. Rahman (2021) notes that algorithmic systems create invisible constraints that workers and users navigate without full visibility into the governing logic. The governance challenge is the inverse: organizations must develop visibility into constraint structures before deployment, not after harm occurs.

What This Means for Organizational Theory

The Google Earth case is a useful data point for organizational theorists working on AI governance not because the rollback was embarrassing but because it was fast. The speed of the rollback suggests that institutional recognition of the problem occurred quickly once harm was observable. The gap was not in organizational response capacity. It was in pre-deployment schema induction: the organizational process of building accurate structural models of novel AI capabilities before they reach users. Hancock, Naaman, and Levy (2020) argue that AI-mediated communication requires new frameworks for understanding agency and accountability. I would extend that claim to organizational governance: AI-mediated deployment requires new frameworks for structural anticipation, not just reactive moderation. Until organizations treat pre-deployment schema construction as a formal competency rather than an informal checklist exercise, rapid rollbacks will remain the primary evidence that governance happened at all.

References

Gentner, D. (1983). Structure-mapping: A theoretical framework for analogy. Cognitive Science, 7(2), 155-170.

Hancock, J. T., Naaman, M., and Levy, K. (2020). AI-mediated communication: Definition, research agenda, and ethical considerations. Journal of Computer-Mediated Communication, 25(1), 89-100.

Hatano, G., and Inagaki, K. (1986). Two courses of expertise. In H. Stevenson, H. Azuma, and K. Hakuta (Eds.), Child development and education in Japan (pp. 262-272). Freeman.

Kellogg, K. C., Valentine, M. A., and Christin, A. (2020). Algorithms at work: The new contested terrain of control. Academy of Management Annals, 14(1), 366-410.

Rahman, H. A. (2021). The invisible cage: Workers' reactivity to opaque algorithmic evaluations. Administrative Science Quarterly, 66(4), 945-988.