The Specific Event

A notable shift is underway in corporate HR practice. Companies are actively revising performance review frameworks to include explicit evaluation criteria around employees' AI use. This is not a fringe experiment at a handful of tech firms. According to recent reporting, organizations across sectors are scrambling to operationalize what "good AI use" looks like in a formal appraisal context. The scramble itself is the signal worth analyzing. When organizations cannot define the competence they are trying to measure, the measurement instrument reveals more about institutional confusion than about employee performance.

The Measurement Problem Is a Schema Problem

The core difficulty is that most organizations are attempting to evaluate AI competence through behavioral proxies: Did the employee use the tool? How frequently? Did output volume increase? These are topographic measures. They describe the surface of the behavior without capturing the structural logic underneath it. This distinction, between topology and topography, is one I have written about in relation to platform work, but it applies here with equal force. An employee who uses an AI assistant daily to generate first drafts of reports is doing something categorically different from an employee who understands why a particular prompt structure reduces hallucination rates in a specific model architecture. Both show up identically in a usage log.

This maps directly onto what Hatano and Inagaki (1986) identified as the distinction between routine and adaptive expertise. Routine expertise produces consistent performance under stable conditions. Adaptive expertise produces diagnostic flexibility when conditions change. Performance reviews built around frequency metrics are, at best, measuring routine competence. They are calibrated to reward employees who have learned the topography of AI tools, not those who have developed transferable structural understanding of how these systems behave across contexts.

Why Organizations Cannot Easily Fix This

The reason organizations are struggling is not primarily technical. It is theoretical. Most HR frameworks were built on an implicit assumption that competence is largely stable and observable prior to task performance. You hire for demonstrated skills, then deploy those skills. Platforms and AI systems invert this. As I argue in my dissertation research on the Algorithmic Literacy Coordination framework, these environments generate competence endogenously through participation. You cannot fully assess AI competence before the employee has worked with the specific combination of tools, organizational data structures, and task contexts that define their role. The competence emerges from the interaction, not from prior training alone.

Kellogg, Valentine, and Christin (2020) document this dynamic in their review of algorithmic work arrangements, noting that workers develop what amount to folk theories about how systems behave, theories that are often partially accurate but structurally incomplete. The same phenomenon will appear in corporate AI adoption. Employees will develop impressionistic accounts of when AI helps and when it does not, and those folk theories will be mistaken for genuine competence during performance reviews because the evaluator often holds the same folk theory.

The Awareness-Capability Gap, Reproduced at Organizational Scale

Research on algorithmic literacy consistently demonstrates that awareness of a system's existence does not translate into improved outcomes (Gagrain, Naab, & Grub, 2024). Knowing that an algorithm governs content distribution does not tell you how to respond to it effectively. The same structural gap is now appearing at the organizational level. Companies are aware that AI use matters. They are aware that variation in AI competence produces variation in output quality. But awareness of this variance does not translate into the organizational capacity to measure it accurately or develop it systematically.

Hancock, Naaman, and Levy (2020) raise a related concern in their treatment of AI-mediated communication: when AI is embedded in consequential human processes, the legibility of human agency becomes contested. Performance reviews are exactly that kind of consequential process. If an employee produces a high-quality strategic memo with significant AI assistance, the review system has to answer a question it was never designed to answer: what exactly is being evaluated, and at what layer of the work?

What This Actually Requires

Organizations that want to build valid AI competence metrics need to move away from usage tracking and toward schema-based assessment. The relevant question is not whether an employee used an AI tool but whether that employee can transfer their understanding of AI behavior to a novel task context they have not encountered before. Gentner's (1983) structure-mapping theory provides a useful anchor here: genuine competence is demonstrated when structural relations, not surface features, transfer across domains. A performance review framework built on this principle would look less like a usage audit and more like a structured diagnostic interview. Most organizations are not close to that yet, and the scramble visible in current reporting suggests they may not know what they are actually trying to build.

References

Gagrain, A., Naab, T. K., & Grub, J. (2024). Algorithmic media use and algorithm literacy. New Media & Society.

Gentner, D. (1983). Structure-mapping: A theoretical framework for analogy. Cognitive Science, 7(2), 155-170.

Hancock, J. T., Naaman, M., & Levy, K. (2020). AI-mediated communication: Definition, research agenda, and ethical considerations. Journal of Computer-Mediated Communication, 25(1), 89-100.

Hatano, G., & Inagaki, K. (1986). Two courses of expertise. In H. Stevenson, H. Azuma, & K. Hakuta (Eds.), Child development and education in Japan (pp. 262-272). Freeman.

Kellogg, K. C., Valentine, M. A., & Christin, A. (2020). Algorithms at work: The new contested terrain of control. Academy of Management Annals, 14(1), 366-410.