Evaluation Metrics and Feature Details for Coding Format Exams

In coding format exams, candidates' submitted code is automatically and quantitatively evaluated. This article explains how to interpret each key metric in the evaluation report, as well as various features such as behavior detection, playback, and skill interviews.

How to Interpret Key Metrics

The evaluation report displays "Score," "Correctness," "Performance," and "Readability" metrics.

Score

Represents the "Overall Score." It is the average value weighted for each metric as follows:

Correctness: 0.7
Performance: 0.2
Readability: 0.1

Correctness

An indicator of "code correctness." It shows the percentage of how well the submitted code performs as expected against multiple pre-prepared test cases.

Test Content: Includes normal inputs, as well as abnormal inputs within the problem constraints.
Verification Method: Candidates can also run test cases in their exam environment for verification (however, there may be hidden test cases used only during scoring).

Performance

An indicator of "code execution speed." It compares the execution speed of the candidate's code (solid line) with the execution speed of the optimal solution (dashed line) for large inputs.

Calculation Method: Calculates the percentage of test cases where the execution speed falls within 3σ (standard deviation) of the optimal solution.
Comparison Function: Depending on the problem, it is possible to compare graphs with "brute-force solutions" and "suboptimal solutions."
Note: If the correctness is 0, the performance value will also be 0 (to exclude speed evaluation of incorrect code).

Readability

An indicator of "code readability." It calculates the "cyclomatic complexity" of the candidate's code and evaluates it from 0% to 100%.

Evaluation Criteria: Compares the minimum value of multiple model answers with the "maximum value × 1.5" as the comparison standard.
Trend: The more complex the code, the higher the cyclomatic complexity, and the lower the score.

Relative Evaluation and Answer Details

Relative Evaluation

Compares the candidate's relative position (rank and deviation) with other candidates who solved the same problem. For more details, please refer to the "About Relative Evaluation" page.

Answer Details

You can check detailed data regarding the candidate's work. Hovering over the icon next to each item allows you to compare it with the average value of all respondents (including candidates from other companies).

Selected Language: The programming language selected at the time of submission.
Elapsed Time: The time the candidate spent on the problem.
Number of Executions: The number of times tests were run before submission.
Hints Used: The number of hints used before submission.
Estimated Computational Complexity: The estimated algorithmic complexity predicted by the system.
Performance: The average execution speed when calculating performance.
Memory: The average memory usage when calculating performance.

Behavior Detection Function

This function records and analyzes the candidate's behavior history during the exam.

Number of Exits: The number of times the candidate left the answer screen by switching tabs or operating other applications.
Number of Pastes: The number of times code was pasted into the editor.
Paste Detection: Highlights pasted sections in red.
IP Address Detection: Displays the respondent's IP address and the number of detections.
Similar Code: If the code is similar to code found on the web, it highlights the number of occurrences and the relevant code (only for algorithm and class problems).

Confirming Submitted Code

Submitted Code

This is the final code the candidate submitted by clicking the "Submit" button. If multiple submissions were made, the selected submission will be displayed.

Description

A natural language memo that candidates can input during the exam. It will be displayed if the candidate has written about their approach to the problem, thought process, etc.

Playback Function

This function allows you to play back the coding process (thought process) like a video, not just the submitted code. It can be accessed via the "Playback" button.

Basic Operations

Playback Controls: Play/pause and skip between events can be done using the timeline at the bottom of the screen.
External Tool Usage History: You can check web search and ChatGPT usage history on the right side of the screen (if permitted during test creation).
Display Toggle: You can toggle the display of web search / ChatGPT using the gear icon on the timeline.

Playback Details

Pressing the enlarge button will display a more detailed analysis screen.

Event Timeline: The elapsed time of each event is displayed on the left, and clicking on it allows you to view the editor state at that point.
Numerical Data: The following aggregated information can be viewed from the buttons at the top of the editor:

Item	Content
Time Allocation	Time spent on coding, web search, ChatGPT, and page exits
Activity History	Shows "which operations were performed how many times" per elapsed time
Code Execution Time	The timing of code execution events (not program processing time)
Search Queries	Search terms used during web searches
Page Exit Time	Elapsed time and duration at the time of exit
Hint Usage Frequency	Elapsed time at the point a hint was used

Important

The usage/execution time for each event refers to the period from "the time the event occurred" to "the start of the next event." This also includes the candidate's thinking time.

Skill Interview Function

This is an interview support function specialized for engineering recruitment. It provides tools to facilitate interviews based on submitted code and design content.

Main Features

Automatic Question Generation It analyzes submitted code and design documents to automatically generate questions that should be explored in depth during the interview. This reduces the time spent on preparing questions beforehand.
Live Coding Live coding can be conducted during the interview, where candidates write code on the spot while referring to their submission.
Seamless Interview Experience Connecting the submission results with the interview enables a consistent evaluation.