Sample
The study included 139 employee-supervisor dyads recruited from multiple organizations in the United States. Employees’ ages ranged from 19 to 69 (M = 39.21, SD = 11.66), and 53% were male. Ethnic representation included 89 White, 25 Black, 12 Mixed, 11 Asian, and 2 Other. Supervisory experience with the employee varied: more than 2 years (41%), 1–2 years (23%), 6–12 months (19%), 3–6 months (9%), and less than 3 months (7%).
Instruments
The Problem Solving test is an online multiple-choice assessment designed to measure an individual’s ability to identify patterns, abstract rules, and solve novel problems. It is conceptually inspired by Raven’s Progressive Matrices and assesses fluid reasoning, inductive reasoning, and eductive ability. The test consists of 8 items, each with four answer options and scored as correct/incorrect. Job performance outcomes were assessed using supervisor-rated Likert scales, including overall performance, quality of work, adaptability, learning new tools, and promotion potential.
Example item
Study
Employees completed the Problem Solving test online, with a time limit of 8 minutes, and then provided a link to their supervisor to complete the performance rating survey. Supervisors were blinded to employee test scores. Only dyads with verified supervisor responses were included. Analyses were conducted in Python.
Results
Criterion validity was evaluated via correlations between test scores and supervisor-rated outcomes. Correlations ranged from r = 0.203 to r = 0.276, with the strongest relationship observed for ability to solve unfamiliar problems (r = 0.276). Ceiling and floor effects were minimal (0.3% and 2.7%, respectively), indicating the test effectively differentiates performance across a broad ability range.
