What is the AI Collaboration Format?

Troubleshooting

The AI Collaboration Format is a practical test format where candidates progress through development by collaborating with AI (primarily large language models). In an environment that closely mirrors real-world development, this format evaluates a candidate's ability to utilize AI tools and work alongside AI to solve problems and implement features.

Assessment Flow

  1. A dedicated test URL is sent to the candidate.
  2. In an environment where AI (e.g., Claude, Codex, etc.) is installed, the candidate proceeds with implementation according to specifications while utilizing the AI.
  3. After implementation, the candidate creates a Pull Request (PR) and submits it.

Report Screen Overview

After submission, you can review evaluations such as "Scores" and "Skill Tags" on the dedicated report screen.

  • The AI-generated scores are displayed on the right side, evaluated primarily from the following perspectives:
    • AI Collaboration Skills: Ability in prompt engineering, instructing AI, iterative improvement, and verification/integration of outputs.
    • Implementation Skills: Comprehensive ability in design, robustness, debugging, and testing.
  • Skill Tags extracted from the code (e.g., Edge Case Handling, Readability, Abstraction, etc.) are displayed with their respective levels.

Detailed meanings for each evaluation item and skill tag can be confirmed via the on-screen guides or help documentation.

Features of AI Evaluation

  • Quantitative Evaluation: AI automatically calculates scores for various criteria.
  • Qualitative Evaluation: AI analyzes the implementation and provides comments on design intent, readability, and maintainability.
  • Playback Function: You can review the chronological history of interactions, instructions, and discussions between the candidate and the AI.

Reviewing Code and Submissions

  • The "Open Submitted Code" button allows you to view the candidate's repository content via a Web IDE (VS Code interface).
  • Diffs are listed with color coding: added (green), modified (orange), and deleted (red).
  • From "Open PR," you can check the Pull Request diffs and automated review comments on GitHub.

Interview Preparation and Final Evaluation

  • Using the "Generate Deep-Dive Questions" button on the report screen, the AI will automatically generate suggested questions for the candidate.
  • There is also an input field for final evaluation, where you can record technical comments and the pass/fail decision.

If you have any questions, please feel free to contact us via chat support or reach out to your account representative.