500 free credits, no card. Try Smalt AI →

People

What should go in an interview scorecard for a new hire

Pick four to six job-related competencies, ask every candidate the same questions, and rate on a labelled scale with written examples of each level.

Smalt AI team ·

An interview scorecard needs four to six competencies taken from the job, the same questions for every candidate in the same order, and one labelled rating scale with a written example of what each level sounds like. Add space for notes on what the candidate actually said or did. Rate straight after the interview, and sign and date the form.

How to build and use an interview scorecard

Why a scorecard is worth the effort

The US Office of Personnel Management (OPM) says unstructured interviews show low reliability and low-to-moderate validity, and are more open to legal challenge. Its guide to structured interviews defines “structured” as the same questions in the same order, a common rating scale, and interviewers agreeing in advance on what an acceptable answer is. The scorecard is the written form of that agreement.

A 2022 meta-analysis in the Journal of Applied Psychology by Sackett, Zhang, Berry and Lievens, as summarised by HumRRO, put the revised mean operational validity of the structured interview at r = .42. That was the highest of the predictors studied, against .31 for cognitive ability. HumRRO notes considerable variation around that average, so it promises nothing for any single process.

A Harvard Business Review piece describes the gap: managers tend to prefer unstructured interviews and rate them most effective, while studies find them among the weakest predictors.

Build the scorecard before the first interview

OPM’s development steps run in this order: job analysis, competencies, questions, rating scales, probes, a pilot test, an interviewer’s guide, and documentation.

  1. Choose the competencies. OPM says interviews typically assess four to six. List what the new hire will actually do in the first year, then name the skills and behaviours that separate someone who does it well from someone who does not.
  2. Write the questions. Use one or two per competency, tied to the work. Google’s re:Work guide lists vetted, role-relevant questions as the first of four components of structured interviewing.
  3. Add probes. These are the follow-ups an interviewer may use when an answer is thin, such as “what did you do yourself?” or “what happened next?”.
  4. Define the scale. OPM recommends one proficiency range for all competencies, at least three levels and ideally five to seven, each labelled. Its own five levels are Expert, Advanced, Intermediate, Basic and Awareness, with behavioural examples per level written by subject-matter experts. Google’s rubrics use four labels: poor, borderline, solid and outstanding, each with example answers.
  5. Pilot it. Have two colleagues rate the same mock answer. If they differ, the level descriptions are too vague.

Here is an illustrative row for a people-operations hire. It is a sample to adapt, not a standard.

CompetencyQuestionPoorBorderlineSolidOutstanding
Handling a sensitive caseTell me about a time an employee brought you something confidential. What did you do?No clear steps, or shared it widelyFollowed a process but missed a stepFollowed the process and documented it, with access limited to those who needed itDid all of that and fixed the gap that let it happen

Run the interview and score it

Ask the questions in the same order every time. OPM advises against consulting resumes during the interview, because they can bias the rating.

Take notes as the candidate speaks. OPM says notes should contain actual behavioural examples, and that each rating should be tied to those notes. Google likewise lists detailed note-taking as a core component. “Seemed confident” is an impression; “described moving the payroll cutover back two days after finding 14 wrong tax codes” is a note.

Rate straight after the candidate leaves, before the next interview blurs the two together. OPM asks the interviewer to sign and date the form. It also names two common rating errors: similarity bias, where you favour people like yourself, and rating everyone all-high or all-low.

If you use a panel, OPM’s guidance for panels of two or three is that each member scores independently, the panel discusses ratings after each candidate, and the final score is a panel consensus.

What to keep on file

OPM’s retention list: the date, time and length of the interview, the interviewer, the questions asked, the scores, the notes and the scales used. How long you must keep them depends on where you hire, so ask your counsel.

Calibration matters too. Google names interviewer training and calibration as the fourth component. Have interviewers score the same sample answer and compare. Where they differ by more than a level, rewrite the description.

Google also reports that pre-made questions, guides and rubrics save about 40 minutes per interview, and that rejected candidates who had structured interviews reported 35% higher satisfaction. Those are Google’s own figures and the page gives no methodology, so read them as indicative.

Doing it with an AI coworker

You can ask Smalt to draft the scorecard as a Word document on your own template, starting from the job description and the competencies you give it. After the interviews, put each interviewer’s completed scorecards in a .zip and ask for one Excel workbook with a row per candidate and a column per competency, so you can compare ratings side by side before the debrief. It can also save the interview guide to Notion for the panel. If you ask it to send interview invites, they wait for your approval before anything goes out.

You still choose the competencies, write the level descriptions and make the hiring decision.

What to check before you rely on it

A scorecard standardises the process. It does not make interviewers unbiased, and it only measures what you wrote into it.

  • Vague levels. If two interviewers cannot explain the difference between “solid” and “outstanding”, the labels are decoration. Add a concrete example answer to each.
  • Averages hiding a gap. A high average can conceal a poor rating on the one competency the role cannot do without. Mark any must-have as a minimum.
  • Interviewer drift. Compare ratings by interviewer over time. One who rates everyone high or low is making the error OPM describes.
  • Local rules. Rules on records, permitted questions and automated tools differ by country and state. Check with employment counsel.

This is not legal advice.

Smalt is an AI coworker that takes jobs like this end to end and hands back the finished file.