May 2026 | AI Collaboration Format Released

Account

Background of the Release

To solve the challenge of "wanting to hire engineers who can leverage AI, but not knowing how to evaluate them," HireRoo has released a new evaluation format: the AI Collaboration Format.

While traditional coding tests allow you to review the final product, it was previously impossible to see how candidates utilized AI. This feature makes it possible to clearly evaluate whether a candidate is simply offloading tasks to AI or collaborating with it while thinking for themselves.

What is the AI Collaboration Format?

The AI Collaboration Format is a new problem type that records and visualizes the entire process of a candidate working on a task using an AI agent. This allows you to evaluate a candidate's AI proficiency.

By adding the interaction process with the AI agent to the evaluation criteria—not just the final output—you can make hiring decisions that align with modern engineering environments.

Three New Experiences

  1. Browser-based AI Development Environment: Candidates are provided with a VSCode (code-server) environment that runs directly on HireRoo. Pre-configured with tools like Claude Code and Codex, candidates can focus on development with AI agents immediately without any local setup. Employers can also operate the tests without the support costs associated with environment configuration.

  2. Automatic Recording of AI Interactions: Candidates' prompts, AI responses, tool usage, and iteration counts are all automatically traced without the candidate having to do anything extra. After the test, recruiters and engineering managers can use these process logs directly for evaluation.

  3. Revisit the Thought Process with Playback: The newly established playback feature allows you to review the interaction between the candidate and the AI in chronological order. You can grasp whether they "let the AI do everything or used it while thinking for themselves" just by looking at the screen. The trackable items include:

  • Prompt Content: What instructions were given to the AI
  • Agent Responses: How the AI responded to those instructions
  • Tools Used: Which tools (e.g., file operations, command execution) the AI used
  • Token Count: Total tokens used up to submission
  • Turn Count: How many turns it took to complete the task
  • Cost: Estimated cost derived from token and turn counts
  • Strengthen Hiring: For companies looking to bolster development teams that assume AI utilization.
  • Process Evaluation: For teams that want to see the thinking and decision-making process, not just the quality of the final product.
  • Standardize Evaluation Criteria: For companies where engineering managers and recruiters want to discuss candidates using a common set of metrics.

For more detailed information about this feature and to see actual screen images, please check the following pages:

  • Overview of the AI Collaboration Format: To learn about the big picture and basic mechanisms of this feature, see the AI Collaboration Format Overview Page.
  • About Employer Reports: For details on how to view and utilize the Reports available to the company side, see the Report Explanation Page.
  • About the Candidate Experience: To see the test screens and operation flow that candidates actually experience, see the Candidate Experience Page.

Available Plans

Supported Plans: Regarding plans that support the AI Collaboration Format or consultations for implementation, please contact your dedicated Customer Success representative.