How do I screen a stack of resumes against a scorecard without reading every one
Write the scorecard from the job analysis first, score every resume the same way, and have a person review borderline cases, rejections and group-level results.
Smalt AI team ·
Write the scorecard before you open any resume: three to six job-related criteria, a fixed scale and fixed weights. Score every resume the same way, let software do a first pass if you like, and have a person review borderline cases and every rejection. Then compare selection rates by group so an unintended pattern gets caught.
Build, score and check a resume scorecard
Build the scorecard from the job, not from the resumes
The US Office of Personnel Management calls this a rating schedule, or crediting plan. It scores experience, education and training against factors taken from a job analysis, and HR sets the benchmarks that turn what an applicant reports into a score. Setting them before you see any applicant keeps the benchmarks from drifting toward whoever you read first.
- List the tasks the role does and the knowledge those tasks need.
- Turn that into three to six criteria.
- Pick one scale for all of them (0 to 3 works) and write one sentence per level saying what earns it.
- Set the weights now. If one criterion is a must-have, mark it as a gate instead of a heavy weight.
Here is an illustrative example for a payroll and HR generalist role. It is a sample to adapt, not a recommended standard.
| Criterion | Weight | 0 | 1 | 2 | 3 |
|---|---|---|---|---|---|
| Payroll | 30% | No payroll work | Supported a cycle | Ran a cycle end to end | Ran several, including corrections |
| Hiring | 30% | No hiring work | Scheduled interviews | Owned 5 to 19 open roles | Owned 20 or more |
| HR systems | 20% | None named | Used one | Built reports in one | Configured one and trained others |
| Employment law | 20% | None | Followed set procedures | Handled leave or terminations | Advised managers on cases |
OPM’s guidance says content validity for this kind of evaluation is typically high, while criterion-related validity (how well the score predicts later performance) is typically low. Read that as a reason to keep the scorecard narrow and tied to the work. OPM also lists length and recency of education, academic achievement and extracurricular activities among the weakest predictors, so do not weight them heavily.
Score the same way every time
Apply the identical criteria, scale and weights to every resume. If you change a definition halfway through the stack, rescore the earlier ones. Record the score and a one-line reason per criterion, so a second reader can see why a resume landed where it did.
Resumes are self-reported, and OPM notes that self-reported information is open to inflation. Its advice is to tell applicants their answers will be verified, and then verify them.
Let software do the first pass, and a person do the second
A first pass by software, or by a colleague with a spreadsheet, is a reasonable way to avoid reading every resume cold. A person should then read the borderline band (say, within a point of your cut-off) and every resume that would be rejected, at least until you have seen how the scoring behaves on your own roles.
Blinding names, photos and graduation years before scoring is common practitioner advice. We found no quantified study to cite for it, so treat it as a sensible habit and not a proven fix.
Track selection rates by group
The US Uniform Guidelines on Employee Selection Procedures say that a selection rate for any group below four-fifths (80%) of the rate for the highest group is generally treated as evidence of adverse impact. Smaller gaps can still count if they are statistically and practically significant, and large gaps on small numbers may not. The guidelines also expect you to keep records of the effects on sex and race or ethnic groups, and they allow sampling when volumes are large.
For example, if 40 of 100 applicants in one group advance and 25 of 100 in another do, the second rate is 62.5% of the first, below the 80% line and worth investigating. This is a rule of thumb for spotting a problem, not a safe harbour: passing it does not show the process is fair.
Doing it with an AI coworker
To skip building the first-pass sheet by hand, put the resumes in a .zip, attach your written scorecard and ask Smalt something like “Summarise these 40 resumes against the scorecard.” It reads the whole batch and gives you one Excel workbook, with a row per candidate and a column per criterion. It can also put the criteria and level definitions into a Word document for the hiring panel. What comes back is a set of summaries for you to review, not a decision. You still write the scorecard and read the borderline rows and rejections. Before you use the rows, run the name-swap test below on them.
What to check before you rely on it
Test any AI tool before trusting it. A University of Washington study ran three open-source embedding models over more than 550 real resumes and more than 500 job listings across nine occupations. It swapped 120 names and made over 3 million comparisons. The share of comparisons each name group was favoured:
- White-associated names: about 85%; Black-associated names: about 9%.
- Male-associated names: about 52%; female-associated names: about 11%.
These were research models, not commercial products, and the study looked only at race and gender, with names as a proxy. It does not tell you how any particular tool behaves, which is why you test your own.
The test is simple. Take a sample of resumes, swap the names and nothing else, run both versions through the same scoring and compare. A large, consistent difference means stop and fix it before the tool touches a real candidate.
A few other limits:
- A scorecard only measures what you wrote into it. If the job analysis is thin, the scores will be precise and still not very useful.
- Scanned or unusual-format resumes may be read less accurately, so spot-check a few by hand.
- Rules depend on where you hire. New York City’s Local Law 144 bars using an automated employment decision tool for hiring or promotion unless it has had a bias audit within the past year (with a public summary) and candidates were given notice, which the city’s slides put at least 10 business days. Enforcement began on 5 July 2023.
- That law applies where a tool substantially assists or replaces discretionary decisions. A software or AI first pass is the route most likely to be covered, while a manual scorecard is probably outside it. Confirm which applies to you with counsel.
This is not legal advice. Check local law, and your employment counsel, before you put any screening method into use.
Smalt is an AI coworker that takes jobs like this end to end and hands back the finished file.