AI Collaboration Format Reports

Troubleshooting

Key Report Features

  • Score
  • Playback
  • Submitted Code Review
  • Deep-Dive Question Generation

Score

Scores for the AI Collaboration format consist of two axes: Deliverable Evaluation and AI Score.

The score for individual problems that impact the total test score is reflected directly by the Deliverable Evaluation. The AI Score does not affect the total score; it is provided as reference information to understand the candidate's ability to utilize AI.

Deliverable Evaluation

Evaluation is based on the deliverables submitted by the candidate.

Requirement Fulfillment Evaluates how well the candidate addressed the requirements presented in the problem. Details can be expanded via an accordion menu for each test case, allowing you to check how accurately each requirement—such as request parameter specifications, expected response formats, and calculation logic—was implemented.

AI Score (β)

This score is an automated evaluation by AI of how the candidate utilized AI tools. The evaluation axes vary depending on the problem format.

  • Interactive Problems: Evaluated on three axes: "Communication Ability," "Problem-Solving Ability," and "AI Collaboration Ability." These can be visually reviewed via a radar chart alongside the overall score.
  • Non-Interactive Problems: Evaluated solely on "AI Collaboration Ability."

Each axis is further divided into detailed evaluation items. You can see which turn's prompt resulted in points added or deducted, along with the specific reasons for the evaluation.

For more details, please visit the AI Score Explanation Page.

Playback

This feature allows you to review the logs of the candidate's interactions with the AI agent during the exam. Click the "Open Playback" button to launch the dedicated viewer.

The viewer consists of the following elements:

Top Header Displays the total number of turns, token usage, and cost.

Left Panel Lists the prompts the candidate sent to the AI agent by turn. It shows the points added/deducted per turn, token usage, cost, and the number of tools used. Clicking a turn will navigate to its details. For interactive problems, the chat history with AI stakeholders can also be viewed in the "Chat" tab.

Turn Details / Tool Calls (Center) Review the candidate's prompt and the AI agent's response for the selected turn. The "Turn Details" tab shows the content of the interaction, while the "Tool Calls" tab shows details of the tools executed by the AI during that turn.

Evaluation Perspectives (Right Panel) Displays which AI score evaluation perspectives apply to the selected turn. Reasons for both positive and negative evaluations are provided, allowing you to review not just the final output but the process of how the candidate mastered the AI.

How to Review Submitted Code

You can view the code submitted by the candidate in HireRoo by clicking the "Open Submitted Code" button on the report page.

You can also download the complete submission from the report page.

Interview Preparation

Automated Deep-Dive Question Generation

Based on the candidate's submission, the AI automatically generates deep-dive questions that can be used during interviews. Because these questions are based on each candidate's specific answers and design decisions, they can be used directly for interview preparation.

How to View

You can view the generated deep-dive questions in the HireRoo report. They are displayed alongside the relevant lines of submitted code, so you can identify points to explore in the interview while reviewing the candidate's implementation and decisions.