Valence's 31% performance odds need a selection-and-confounding test

Preserve the claim's exact estimand

A buyer should first obtain the analysis population, enrollment and observation dates, eligible and excluded employees, organization levels and functions, baseline and follow-up review cycles, performance-band definitions, transition counts, missing outcomes, and the model used to calculate the 31% higher odds. Ask for absolute probabilities by group and uncertainty intervals; an odds ratio cannot be read as a probability-point change, and a move between company-specific bands may not translate into another employer's outcome. Confirm whether the reported comparison is power users versus all others, another engagement band, or a modeled contrast, and whether people could move down, remain, leave, or lack a rating. Preserve the provider, employer, and analyst roles and any publication or commercial interest.

Audit the exposure before interpreting it

The AI Power User Index combines frequency, breadth, and depth, so the buyer needs definitions, feature construction, thresholds, weights, time window, validation, sensitivity, and the distribution of each component. Depth based on how much of one's thinking appears in an exchange may encode writing style, job type, language, seniority, workload, access, or comfort disclosing context rather than coaching dose alone. Determine whether exposure was calculated before the outcome, whether review-related sessions were included, how inactive or partial users were handled, and whether people could become power users because they were already motivated, supported, or performing well. Inspect privacy-preserving aggregate evidence; do not request individual coaching content merely to validate engagement. The metric must be reproducible without turning confidential conversations into performance surveillance.

Test selection and plausible confounding

Power use was not described as randomly assigned. Ask how the study addressed prior performance and trajectory, manager status, team, function, level, tenure, location, workload, development opportunity, access timing, manager encouragement, concurrent talent programs, organizational change, rating calibration, attrition, and missing data. The provider's finding that managers and teams with power-user managers were more likely to be power users makes managerial and team context especially important. Request adjusted and unadjusted estimates, balance diagnostics, sensitivity analyses, alternative exposure thresholds, negative controls where appropriate, and results across material subgroups. Even a stable adjusted association may reflect unmeasured motivation or opportunity. Use causal language only if the design and assumptions support it; otherwise describe the observed relationship.

Design prospective buyer evidence

For a buyer pilot, pre-register the coaching purpose, eligible population, assignment or rollout method, baseline, primary and harm outcomes, exposure measure, privacy boundary, analysis, subgroup protections, and stop criteria before results are visible. Separate adoption from coaching quality and from employment evaluation; managers and decision-makers should not gain access to private session content. Measure access, engagement, self-efficacy, behavior evidence, manager and peer observations where appropriate, work outcomes, rating-process changes, adverse experiences, differential participation, attrition, and total cost. A stepped or randomized design may be possible, but only with appropriate workforce and ethics review. Valence's study is a relevant provider signal that moves the question beyond logins. It is not a substitute for method disclosure, independent scrutiny, or evidence that the buyer's configured system caused a fair and useful outcome.

Turn this source into a reviewable decision

For AI Coaching Platforms for Leadership Development, use this briefing as a dated decision record rather than a substitute for the source. Preserve Does AI coaching move performance?, the exact URL, the August 27, 2026 review date, the supported facts above, the editorial interpretation, the limitations, and any buyer-specific evidence. Link that record to the decisions most directly affected: Coaching method and content provenance; Data flow and confidentiality; Validation and outcome evidence; Organizational configuration without surveillance. State whether the source changes the scope, evidence requirement, control, sequence, or only the language used to describe the decision.

Before action, name the accountable owner, affected population and workflow, exact offering or configuration, source data and rights, human decision point, exception and appeal path, complete cost, expected benefit, failure and stop conditions, retained evidence, and next review date. Keep official facts, provider statements, buyer observations, representative tests, measured outcomes, editorial inferences, and unknowns visibly separate. Reopen the record when the source, offer, model, integration, data, policy, population, responsible person, or measured result changes.

Limitations and unknowns

Valence is the provider and study source. Its August 18, 2026 article reports a study of 11,000 employees at one large technology company, an AI Power User Index based on frequency, breadth, and depth, and 31% higher odds of moving up a performance band among power users. It does not publish in the article the complete protocol, eligibility and exclusions, group sizes, absolute transition probabilities, uncertainty, model specification, index thresholds and validation, missingness and attrition, adjustment set, balance and sensitivity analyses, subgroup estimates, independent replication, or evidence that engagement caused performance movement. Current full methods and aggregate results, privacy and employment-use boundaries, independent statistical review, prospective buyer design, participant evidence, and qualified coaching, people analytics, statistics, employment, accessibility, privacy, security, procurement, ethics, and legal review control.