AI Collaboration Format Report

Troubleshooting

Report Screen Layout

Scores and Skill Tags

The right side of the report displays automated scoring results generated by AI. In the AI Collaboration Format specifically, "AI Collaboration Skills" and "Implementation Skills" are evaluated with a focus on the collaborative process with the AI.

# Item Description
1 AI Collaboration The ability to effectively leverage LLMs and AI tools to increase development speed and quality through collaboration.
2 Implementation The ability to translate designs into functional code while ensuring quality through dialogue with AI.

The AI Collaboration score is automatically calculated based on several criteria, such as the setup of the agent environment, the quality of instructions given to the AI, iterative improvements, and verification of outputs. For details on the meaning and interpretation of each evaluation item, please refer to About Evaluation Items for AI Collaboration Skills.

Below the scores, Skill Tags extracted from the code (e.g., Abstraction, Deduplication, Edge Case Handling, Error Propagation, Readability, etc.) are displayed. Each tag is assigned a level (Lv. 1–5).

Reviewing Submitted Code

Open Submitted Code

By clicking the "Open Submitted Code" button, you can review the repository worked on by the candidate using VS Code directly in your browser. You can select files from the file tree to inspect their contents.

Open Playback

By clicking the "Open Playback" button, you can review the interaction between the candidate and the AI (such as Claude Code) in chronological order.

How to read the Playback screen:

  • Turns: The number of interactions between the candidate and the AI.
  • Token Usage: Total tokens processed by the AI.
  • Cost: The cost incurred for AI usage.

Clicking on the Turn List on the left will display the details for each turn.

  • Turn Details Tab: View the candidate's input and the AI's response.
  • Tool Calls Tab: View which tools the AI used (e.g., file read/write, command execution).

The right side displays Evaluation Perspectives, where automated comments are added regarding "Tool Utilization," "AI Collaboration Style," and more.

💡 Key Checkpoints

  • Is the candidate delegating everything to the AI, or are they thinking for themselves while using it?
  • Are they performing verifications (behavioral checks, type checks, etc.) themselves?
  • How did they manage the process up to the creation of the Pull Request (PR)?

Reviewing Code on GitHub

Open PR

Click the "Open PR (GitHub)" button to view the Pull Request created by the candidate directly on GitHub.

On GitHub, you can check:

  • Commits: Number and content of commits.
  • Files changed: Diffs of the modified files.
  • Conversation: Automated review comments by hireroo-system.

Automated Review by hireroo-system

Automated review comments from the hireroo-system account are posted to the PR. These become visible once you click the "Generate Deep-Dive Questions" button. Comments are provided in both Japanese and English and include content such as:

  • Overall assessment of requirement fulfillment.
  • Evaluation of code design and separation of concerns.
  • Questions to encourage improvement (e.g., "Why did you define this component inline?").

Comments are also assigned priority labels (e.g., P2 medium).

Repository Access Permissions

The GitHub repositories are private. If you have a specific GitHub account you wish to use for viewing, you can apply for access via the "Request Access" button. Accounts that have been granted access will be listed under "Accounts with Access."

Interview Preparation

Generate Deep-Dive Questions

Click the "Generate Deep-Dive Questions" button to have the AI automatically generate questions for the interview based on the submitted code. Generation takes approximately 3–4 minutes.

Generated questions will appear in the "Perspectives for Interview Deep-Dive" section. These can be used as material to ask candidates directly about issues found during code review or their design decisions. Additionally, this action allows you to check the hireroo-system automated review comments at the same time.

Final Evaluation

You can record your pass/fail decision for the candidate via the "Final Evaluation" button in the top right of the report. You can also leave technical feedback via the "Technical Review" button.

Evaluation Workflow Summary

Candidate List → Open Candidate Detail Report
  ↓
Review Scores and Skill Tags
  ↓
Open Submitted Code (VS Code View)
Open Playback (Review AI usage)
  ↓
Review PR on GitHub (Diffs, Automated Review Comments)
  ↓
Generate Deep-Dive Questions (Prepare for interview, view review comments)
  ↓
Record Final Evaluation