August 2026 | Released Skill Assessment (Beta), Which Measures the AI Collaboration Skills of Your Engineers

Account

We have released Skill Assessment as a beta. It continuously measures the AI Collaboration Skills of your engineers from the usage logs of the AI tools they already use in their daily work.

Until now, HireRoo evaluated AI Collaboration Skills only from tests taken by job candidates in a dedicated exam environment. With Skill Assessment, your engineers can be measured in their normal working environment. There is no need to move to a special editor or exam environment; they simply keep using the AI tools they already use.

For details, see What is Skill Assessment?.

Background

AI tool adoption is progressing, but there has been no way to measure how much an organization's AI use has actually improved as a result. License counts and usage counts do not tell you whether that usage is the kind that leads to outcomes.

In addition, the approaches used by engineers who work well with AI remained known only to those individuals, with no route for sharing them across the organization. Because it was unclear who was using AI effectively and how, it was also difficult to decide where to invest in development.

Skill Assessment was built to address both of these with the same measurement mechanism.

What's Included

Measurement in the Normal Working Environment

Once an engineer installs the hireroo CLI on their machine, usage logs from their AI tools (Claude Code / Codex CLI) are sent to HireRoo. Installation takes a single command and setup finishes with a browser sign-in, taking about five minutes in total.

The type of AI tool credentials or subscription does not matter. Because the request path to AI is left unchanged, measurement works with your existing contracts and configuration as they are.

Continuous Measurement on the Same Axes Used in Hiring

Skill Assessment uses the same five “AI Collaboration Skills” axes from SEI (Software Engineering Index) that appear in AI Collaboration format reports: Environment Engineering, Intent Specification, Co-Reasoning, Execution Control, and Output Quality Assurance. The indicators you review during hiring connect directly to the indicators you review after the engineer joins.

For detailed definitions of each axis, see AI Collaboration Skills | SEI.

Visibility Across the Organization and for Each Individual

Administrators can review the organization average score, rank distribution, AI tool spend, usage categories, languages, and working hours on a single screen, and compare scores, trends, and primary tools across members from the member list. The member detail screen shows a five-axis radar chart placing the member against the organization average.

Engineers being measured can also review their own score and five-axis breakdown compared against the organization average. Individual scores of other members are not shown to them.

How Data Is Handled

  • Prompt contents are not stored long-term. They are held in temporary storage only for as long as needed to calculate scores, then deleted.
  • Personal information such as email addresses is removed as soon as logs are received.
  • AI responses and tool execution results are not used for evaluation and are not stored. This prevents scores from rising simply because AI models become more capable, and limits evaluation to the human side of the interaction.
  • AI tool behavior is not affected. An incident on the HireRoo side does not stop your engineers from working; the only impact is that usage logs for that period are not recorded.

For organizations that cannot allow logs to leave their own environment, we plan to offer an option in which the measurement infrastructure is installed inside your own cloud environment.

Notes on the Beta

  • We will continue to refine the definition and weighting of the evaluation axes and the score calculation based on customer data. During the beta, we recommend focusing on relative comparison within your organization and trends over time rather than absolute score values.
  • Suggested improvement actions based on scores are not included in this beta release. We plan to support this in a future update.
  • Supported AI tools are Claude Code and Codex CLI. We will expand tool support over time.

Availability

If you would like to adopt Skill Assessment, please contact your customer success representative or reach out through our contact form.