What is Skill Assessment?
TroubleshootingSkill Assessment continuously measures the AI Collaboration Skills of your engineers from the usage logs of the AI tools they already use in their daily work, and visualizes those results relative to the rest of the organization.
Unlike Skill Interview, which asks candidates to take a test in a dedicated environment, Skill Assessment measures engineers in their normal working environment. Engineers do not need to move to a special editor or exam environment; they simply keep using the AI tools they already use.
How Skill Assessment Differs from Skill Interview
| Skill Interview | Skill Assessment | |
|---|---|---|
| Subject | Job candidates | Engineers on your team |
| Where measurement happens | HireRoo exam environment | The engineer's normal working environment |
| Unit of measurement | A snapshot from a single test | Continuous measurement and trends over time |
| Primary use | Assessment during hiring | Understanding and developing AI use across the organization |
Both use the same “AI Collaboration Skills” axes from SEI (Software Engineering Index), so the indicators you review during hiring connect directly to the indicators you review after the engineer joins.
The Five AI Collaboration Skills Axes
Skill Assessment evaluates the same five axes used in AI Collaboration format reports.
| Evaluation Axis | Behaviors Evaluated |
|---|---|
| Environment Engineering | Prepare tools and context so that AI can work appropriately, and establish rules for safe execution |
| Intent Specification | Frame the problem in the engineer's own words and communicate constraints and completion criteria clearly |
| Co-Reasoning | Elicit alternatives from AI and decide whether to adopt them based on rationale and trade-offs |
| Execution Control | Adjust the scope delegated to AI, steer work during execution, and guide convergence after failures |
| Output Quality Assurance | Design verification methods, critically assess AI output, and further improve quality |
For detailed definitions of each axis, see AI Collaboration Skills | SEI. For the thinking behind the evaluation criteria, see also July 2026 | Updated AI Collaboration Skills Evaluation Criteria to Better Assess the Depth of AI Use.
What Is and Is Not Used for Evaluation
Skill Assessment evaluates only the prompts an engineer writes and the tool calls made by the AI agent.
AI responses and tool execution results are not used for evaluation. This prevents scores from rising simply because AI models become more capable. By limiting evaluation to the human side of the interaction, changes in a score reflect changes in how the engineer actually works.
What You Can See
Skill Assessment provides views for administrators and a separate view for the engineers being measured.
For Administrators
| Screen | What you can review |
|---|---|
| Dashboard | Organization average score, active rate, AI tool spend, token usage, AI usage (request count, token consumption, average prompt length), usage categories, languages, working hours, rank distribution, and members whose scores are growing fastest |
| Member list | Score, rank, week-over-week change, primary AI tool, and last log time per member. Filter by rank, score, status, tool, name, or email |
| Member detail | Overall score and rank within the organization, a five-axis radar chart comparing the member against the organization average, tool usage ratio, usage categories, and activity |
For Engineers Being Measured
Each engineer can review their own overall score, rank, and position within the organization, along with a breakdown of the five axes compared against the organization average. They can also review their own request count, token consumption, average prompt length, usage categories, languages, and working hours, each shown relative to the organization average.
Individual scores of other members are not shown to engineers. They see only their own score and the organization average as a point of comparison.
Getting Started
1. An administrator invites the members to be measured
From the Skill Assessment screen, invite the members you want to measure by email address. Employees already registered with HireRoo can be assigned directly without sending an invitation.
2. Members install the hireroo CLI
Each invited member installs the hireroo CLI on their machine and runs the setup command. Installation takes a single command, and setup finishes with a browser sign-in. There is no need to copy or paste a token manually.
Setup takes about five minutes.
3. Measurement begins
After setup, measurement starts as soon as the member uses their AI tools as usual. Progress is shown on the member's own page in three steps.
- Install the CLI — setup is complete
- Confirm connectivity — the first usage log has reached HireRoo
- Wait for measurement to start — the first evaluation result has been generated
Once the third step is complete, the score appears on the dashboard. Scores are then updated as usage accumulates.
Supported Environments
| Support | |
|---|---|
| OS | macOS / Linux / Windows |
| AI tools | Claude Code / Codex CLI |
The type of AI tool credentials or subscription (API key, subscription, Amazon Bedrock, or Google Vertex AI) does not matter. Measurement works with your existing contracts as they are.
How Data Is Handled
AI Tool Behavior Is Not Affected
Skill Assessment does not sit between AI tools and AI providers. Only usage logs are sent to HireRoo; requests to AI are sent directly to the AI provider as before.
As a result, an incident on the HireRoo side does not stop your engineers from working. The only impact is that usage logs for that period are not recorded.
Prompt Contents Are Not Stored
Prompt contents and the contents of tool calls are held in temporary storage only for as long as needed to calculate scores, and are deleted as soon as calculation finishes. They are never stored long-term in a database.
The only data retained long-term is information that does not include any content (token counts, cost, timestamps, the tool used, prompt length, and similar) along with the calculated scores. Personal information such as email addresses is removed as soon as logs are received.
Deployment Options
Skill Assessment is currently available in the form where HireRoo hosts the measurement infrastructure (the cloud option). For organizations that cannot allow logs to leave their own environment, we plan to offer an option in which the measurement infrastructure is installed inside your own cloud environment. With that option, prompt contents never leave your environment, and only calculated scores and aggregated values are sent to HireRoo.
Points to Note
- Measurement depends on configuration on the engineer's machine. If a member changes their AI tool configuration or uninstalls the CLI, measurement stops. Because AI tools continue to work normally in that case, the member is unlikely to notice, so members whose logs have not arrived for a period of time can be identified from the administration screen.
- The same email address cannot be both an administrator and a measured member. To measure an administrator's own AI Collaboration Skills, invite a separate account as a measured member.
- Scores are not updated during periods with no AI tool usage. Scores update once a certain volume of usage logs has accumulated, so members with light usage will see longer intervals between updates.
About the Beta
Skill Assessment is currently offered as a beta. We will continue to refine the definition and weighting of the evaluation axes and the score calculation based on customer data.
If you would like to adopt Skill Assessment, please contact your customer success representative or reach out through our contact form.