resume-mirror-eval

Files (34)

• README.md
• SKILL.md
• README.md · mock-data/
• agent-notes.json · mock-data/example-output/
• results-detail.csv · mock-data/example-output/
• results.csv · mock-data/example-output/
• results.json · mock-data/example-output/
• results.xlsx · mock-data/example-output/
• job-description.md · mock-data/
• labels.csv · mock-data/
• andre_williams.md · mock-data/resumes/
• arjun_singh.md · mock-data/resumes/
• dana_whitfield.md · mock-data/resumes/
• hanna_mueller.md · mock-data/resumes/
• jordan_harris.md · mock-data/resumes/
• linh_nguyen.md · mock-data/resumes/
• marcus_chen.docx · mock-data/resumes/
• ngozi_okafor.md · mock-data/resumes/
• petr_ivanov.md · mock-data/resumes/
• riya_patel.pdf · mock-data/resumes/
• sofia_garcia.md · mock-data/resumes/
• soojin_kim.md · mock-data/resumes/
• tyler_brooks.md · mock-data/resumes/
• methodology.md · references/
• analyze.py · scripts/
• bootstrap.py · scripts/
• evidence.py · scripts/
• extract.py · scripts/
• jev.py · scripts/
• lexical.py · scripts/
• llm_notes.py · scripts/
• questions.py · scripts/
• report.py · scripts/
• scoring.py · scripts/

SKILL.md (first lines)

# Resume Mirror Eval

> **Research proof of concept.** This skill measures how closely a resume's
> *wording* tracks a job description. It does not determine whether a person
> used AI and it makes no decision about any candidate. Its output is a list of
> resumes a human should read first; the **Needs human review** column exists
> for exactly that purpose. It is a tool for directing human review, not a
> substitute for it.
>
> Any use must comply with the laws, regulations and policies that apply to
> the user and their applicants; automated tools in hiring are regulated in
> many places, and the user should consult their legal team before running
> this on real applicants. Provided as is, for research and education.
>
> Repeat the short form of this to the user at the end of every run.

## What it does

1. Extracts text from every resume (.pdf, .docx, .txt, .md) in the folders or
   files the user points at.
2. Computes text statistics in code: verbatim 4-word phrase overlap with the
   JD, longest shared word run, whether matched terms appear in the JD's order,
   TF-IDF cosine within the batch, keyword and acronym coverage, and a z-score
   against the rest of the batch.
3. Sends `{job_description, resume}` to TypeSafe's Jev with six typed
   questions in one request per resume (phrasing mirror, requirement echo,
   concrete specifics, generic template, posting-language leak, career
   consistency). Roughly 2,000 tokens and a tenth of a cent per resume; a batch
   of 100 finishes in seconds.
4. Turns Jev's answers into a 0-100 **Mirror score (Jev)**; the statistics
   into **Text match analysis (code)**, template sentences a reviewer can check
   against the two documents; and both into a **Needs human review** flag. Nothing in the output rates the candidate's fit or
   qualifications; the tool judges the file's relationship to the posting only.
5. Writes `results.xlsx` sorted by mirror score: a five-column Results sheet
   (review flag, file, Jev score, text match analysis, AI analysis), a Details sheet with every
   statistic and Jev answer, the JD, and the disclaimer. Also `results.csv`,
   `results-detail.csv` and `results.json`.
6. Optionally adds **AI analysis (agent)**: terse bullets per flagged resume,
   written by you (the agent running this skill) or by an API model. Three
   methods, three columns; none feeds another.

Every question, weight and threshold lives in `scripts/questions.py`. If the
user disagrees with how something scored, that file is where to look first.

## Workflow

`SKILL_DIR` below means the folder containing this SKILL.md.

### 1. Locate the sources and stage them

The script reads local files. Where those files come from is your job. The
user may point you at anything you can reach: a folder, a synced drive, a
connector, an ATS, a URL, an email thread, or text in the conversation. Use
whatever tools you have to fetch the documents, land copies in a temp staging
folder, then run the script on that folder. Do not ask the user to download
things you can download yourself.

| Source | How to stage it |
|---|---|
| Local folder or files | Pass the paths directly; nothing to stage. |
| Synced drive (Google Drive, Dropbox, OneDrive, SharePoint sync) | It is a local path. Pass it directly. |
| Cloud drive via a connector (Google Drive, SharePoint, Box MCP...) | Download each file into the staging folder, keeping the original file name. |
| ATS via an MCP server (Greenhouse, Lever, Ashby, Workable, iCIMS...) | Fetch the requisition's job description to `staging/jd.md` and each application's resume attachment to `staging/resumes/`. Name files `<lastname>_<firstname>_<application_id>.<ext>` and write `staging/manifest.csv` with `file,application_id,candidate,source_url` so those ids appear in the spreadsheet. |
| URL to a posting | Fetch it and save the text as `staging/jd.md`. |
| Pasted text | Save it to `staging/jd.md`. |
| Email or ticket attachments | Save each attachment into `staging/resumes/`. |

The staging folder is a **temp location**, never the user's working directory
or the source folder: use your scratchpad if you have one, otherwise
`$TMPDIR/resume-mirror-eval/<timestamp>/`. The script's own default `--out` is
under the system temp dir for the same reason. Resumes are personal data; the
copies exist only for the run. When the run is done, tell the user where the
staging and output folders are and offer to delete the staged copies (keep the
spreadsheet, or copy it to wherever they asked for it).

Ask for whatever is missing. Two things are required: one job description and
one or more resumes. If a source yields several postings, confirm which one
before pulling resumes.

Never upload resumes anywhere other than the TypeSafe API (and the LLM provider
if the user opts into API notes). Say so if asked.

### 2. Check prerequisites

```bash
python3 --version   # 3.10 or newer; `python` on Windows
test -n "$TYPESAFE_API_KEY" || test -f ~/.config/typesafe/env || echo "need a TypeSafe key"
```

Nothing else. On first run the script creates a private virtual environment
under the system temp directory and installs its two pure-Python dependencies
there (a few seconds); later runs reuse it. If the user has `uv`, `uv run`
works too and skips that step.

The script reads `TYPESAFE_API_KEY` from the environment, or from a line
`TYPESAFE_API_KEY=...` in `~/.config/typesafe/env`. Keys come from
https://console.typesafe.ai/keys. If the user has neither, stop and ask; do
not guess or put a key on the command line.

### 3. Run the analysis

```bash
python3 "$SKILL_DIR/scripts/analyze.py" \
  --jd "$STAGING/jd.md" \
  --resumes "$STAGING/resumes" \
  --out "$STAGING/out" \
  --llm-notes agent
```

Add `--manifest "$STAGING/manifest.csv"` when you staged from an ATS or any
source with ids worth carrying into the sheet; every column but `file` is
copied in after the File column.

`--llm-notes agent` makes the script write `notes-request.json` for the rows it
flagged, so you can write the notes yourself in step 4 without any other API
key. Leave the flag off if the user only wants scores.

Other flags: `--llm-top N` (annotate only the top N by score), `--labels
labels.csv` (score the run against known `human` / `ai_tailored` labels, for

From SecurityMindedSolutions/ai-skills, MIT.

More from SecurityMindedSolutions/ai-skills