How this library reaches a judgment
This methodology evaluates an AI recruiting system in the context of one defined workflow. It separates vendor claims from observed evidence, tests both successful and failed cases, and records what a source cannot establish before anyone assigns a score.
Last substantive review: August 7, 2026 · General evaluation guidance, not legal advice.
On this page
Start with context
The unit of evaluation is not a logo, model name, profile count, or polished demo. It is a named system performing a defined task for a particular role, employer, population, jurisdiction, and period. A sourcing assistant that proposes names has a different evidence burden from a system that rejects applicants or sends messages without per-action review.
Write the decision in one sentence: who will use which output to make or support what hiring decision? Then record the role family, seniority, locations, languages, data sources, review budget, external actions, and human owners. This context prevents evidence from one workflow being recycled as proof for another. It also makes a future change visible: replacing a model, data source, rubric, or approval path creates a new evaluated configuration.
The NIST AI RMF treats context mapping as part of risk management. The UK recruitment guide similarly begins with purpose, functionality, resources, governance, and applicant impact. Neither source supplies a universal score; both support making the use case explicit before measuring it.
Evidence before score
A capability statement, a scripted demo, a benchmark, and a real-role outcome answer different questions. The library records the strongest available level for each claim and keeps lower-level evidence visible rather than silently promoting it. A demo can show that a control exists in one path; it cannot establish that the control is used consistently. A benchmark can compare models under a protocol; it cannot show the buyer's operating burden or candidate experience.
Evidence also needs provenance. Record who produced it, when, with which inputs, under what access, and whether unsuccessful cases were available. Vendor-supplied material remains useful when labeled first-party. The error is not using first-party evidence; it is presenting it as independent validation.
| Level | What it can support | What remains unresolved |
|---|---|---|
| 0: assertion | The vendor or reviewer states that a capability exists. | Whether it exists, works, or applies to the evaluated configuration. |
| 1: artifact | Documentation, a screenshot, policy, or model card describes intended behavior. | Whether behavior matches the artifact in practice. |
| 2: demonstration | A reviewer observes the workflow on prepared or buyer-provided examples. | Repeatability, selection effects, and real operating conditions. |
| 3: controlled test | A pre-specified test includes successes, failures, and relevant comparison points. | Longitudinal behavior and downstream hiring outcomes. |
| 4: operating evidence | The system produces repeatable results on real roles with auditable labor, errors, and outcomes. | Generalization to new roles, populations, jurisdictions, or changed components. |
Make claims testable
Break broad promises such as "better candidates," "bias-free screening," or "fully automated recruiting" into observable claims. For each claim, record the owner, population, metric, comparison, time window, required artifact, known limitation, and decision that depends on it. If a claim cannot name a population or decision, it is probably too vague to score.
Request denominators and exclusions. Ten accepted candidates says little without knowing how many were retrieved, reviewed, contacted, interested, or rejected. A response rate needs sender, channel, audience, time window, bounce handling, follow-up rule, and definition of response. An accuracy number needs the task, label source, sample, threshold, and error distribution. This is especially important where automated output influences selection and the employer remains responsible for how it is used.
Negative evidence belongs in the same register. Document unsupported inferences, stale records, missing groups, accessibility failures, inconsistent explanations, duplicate messages, and cases where the human reviewer overrode the system. A method that captures only showcase cases measures presentation quality, not operational reliability.
Avoid false precision
This library uses ordinal scores to organize discussion, not to manufacture a universal ranking. A difference between 2 and 3 means the evidence moved from a limited pilot toward repeatable and auditable operation. It does not mean the system became exactly one unit safer, fairer, or more valuable. Weighting also belongs to the buyer: outreach control may be decisive for an agent that sends messages and irrelevant for an offline search benchmark.
Use gates before totals. An inaccessible candidate path, an unowned external action, missing selection-validity evidence, or inability to pause the system can stop deployment even if other dimensions score highly. Then compare alternatives using the same role and evidence request. Do not compensate for a critical zero by adding points from dashboard polish or feature breadth.
Decision language should remain conditional: stop; investigate; run a narrow reversible pilot; continue under named controls; or expand after new evidence. "Approved," "compliant," and "safe" require authorities and scopes that a general scorecard does not possess.
Disclose incentives
OpenJobs AI is now Metix AI, so this archive has a relationship that readers should know. The site does not claim institutional independence. Instead, it publishes its method, links to primary sources, labels Metix material first-party, and states the limits of every cited artifact. The same questions about dataset scope, comparison, failures, operating labor, and external validation apply to Metix and to any other vendor.
The library does not accept paid rankings, sell score improvements, hide sponsorship in editorial copy, or place vendors on a league table without comparable evidence. It does not use the absence of public documentation as proof of poor performance; it records the evidence as unavailable. Corrections change the record, not the standard applied to the subject.
Contextual outbound links are chosen because they help evaluate the claim on the page. Official sources explain requirements or practices. First-party research illustrates a method or product architecture. Link placement is not an endorsement, and inclusion in the ledger is not a certification.
Maintain the record
Regulations, regulator guidance, products, models, and integrations change on different schedules. Every time-sensitive source in this library has a last-checked date. A date means the source was reviewed for the stated use; it does not mean every linked page changed that day. Material updates change the page's substantive-review date and the machine-readable evidence register.
Re-evaluate when intended purpose, decision authority, model, retrieval source, candidate population, language, jurisdiction, approval path, or integration changes. Also re-evaluate after an incident, a material error trend, a new accommodation issue, or unexplained movement in quality or override rates. Do not refresh only the date while leaving stale conclusions untouched.
Corrections should identify the affected claim, prior wording, new evidence, date, and downstream pages or assets updated. This makes the library citable as a maintained resource and gives agents enough context to avoid blending older requirements with current guidance.
Reusable files
Evidence register
A lifecycle structure for governing, mapping, measuring, and managing AI risk and trustworthiness characteristics.
Evidence limit: Use of the voluntary framework does not establish legal compliance, product quality, or fitness for a particular hiring process.Suggested actions for applying the AI RMF functions across design, deployment, evaluation, and operation.
Evidence limit: The suggested actions are optional and use-case agnostic; they are not a certification checklist or employment-law opinion.Technical assistance on job-related selection procedures, discriminatory impact, validation, and employer responsibility.
Evidence limit: The page describes federal considerations but does not determine whether a specific tool, employer, or use is lawful.Procurement and deployment questions covering purpose, governance, accessibility, assurance, testing, pilots, transparency, and monitoring.
Evidence limit: The guide expressly does not provide legal assurance and its examples are not universal deployment instructions.Observed privacy and information-rights issues in recruitment sourcing, screening, and selection tools, plus remediation themes.
Evidence limit: Consensual audits of selected providers do not establish prevalence, legal status, or performance of another product.Metix describes a component, trajectory, and outcome evaluation model that can be tested against this library's evidence ladder.
Evidence limit: This is first-party engineering research, not independent assurance or proof of results in a buyer's workflow.Relationship disclosure: OpenJobs AI is now Metix AI. Metix material is labeled first-party and is not treated as independent validation.
Questions teams ask
No. The score describes the maturity of evidence reviewed under a defined workflow. Legal obligations depend on facts, jurisdiction, role, system use, and accountable professional advice.
Yes, when it is labeled first-party and its method, scope, date, comparison, and limitations are visible. It should not be presented as independent validation.
Products perform different tasks and expose different decision risks. A single ranking would hide role context, buyer priorities, missing evidence, and material workflow differences.
Repeat it after material changes to purpose, models, data sources, integrations, candidate populations, jurisdictions, controls, or observed failure patterns.
Continue evaluating