A buyer question bank
Use this checklist before a contract or broad pilot. It asks vendors to demonstrate one real workflow, disclose evidence and limitations, expose candidate-impact controls, and account for the labor and systems the buyer must still operate.
Last substantive review: August 7, 2026 · General evaluation guidance, not legal advice.
On this page
Buyer preparation
Start with the hiring bottleneck and the decision owner, not a feature list. A team that needs more qualified passive candidates is evaluating a different system from one that needs structured applicant screening, scheduling, or end-to-end managed delivery. If the category remains vague, every vendor can appear complete while solving a different step.
Document the current workflow in enough detail to create a comparison: role approval, sourcing channels, people reviewed, outreach, replies, screening, interviews, recruiter time, manager rework, candidate complaints, and integrations. Note what the team will not delegate. This baseline becomes the anchor for demonstrations, reference calls, security review, pilot design, and total cost.
Invite talent acquisition, the hiring manager, people operations, IT or security, data or privacy owners, accessibility expertise, procurement, and counsel as appropriate. One person can coordinate the review, but a vendor should not be allowed to answer a security question with a recruiting metric or a validity question with a compliance badge.
Compare like with like
AI recruiting products often combine several layers: job-description assistance, search, ranking, enrichment, messaging, chat, assessment, scheduling, analytics, or human delivery. Ask the vendor to draw the production workflow, including third-party models, data providers, human reviewers, ATS writes, email or messaging channels, and manual exception handling. A box labeled "AI" is not an architecture.
Then identify the highest-impact output. A tool that drafts a Boolean query under review presents different risks from one that filters applicants, scores an interview, or sends candidate communications. The evaluation depth should follow the consequence and reversibility of the action. High volume does not automatically mean high risk, but it can multiply a small error quickly.
Ask which functions are generally available, optional, beta, region-limited, partner-delivered, or dependent on a particular plan. Record the evaluated version and configuration. Demonstrating an adjacent product or future roadmap item does not satisfy a requirement for the purchased workflow.
| System role | Primary evaluation question | Common hidden dependency |
|---|---|---|
| Sourcing and ranking | Does the system retrieve relevant people within the review budget without hiding exclusions? | Coverage, refresh, enrichment, and human labeling. |
| Screening and assessment | Is the evaluated construct job-related, consistently administered, accessible, and reviewable? | Job analysis, rubric design, accommodations, and employer use. |
| Engagement agent | Are sender, message, channel, timing, suppression, and approval controls enforceable? | Provider policy, identity, contact data, and reply handling. |
| Workflow automation | Can writes, tool calls, and handoffs be traced, reversed, and paused? | ATS permissions, integration behavior, and manual fallback. |
| Managed outcome service | What result is delivered, who performs hidden work, and how are misses remedied? | Human operations, service boundaries, and commercial terms. |
Evidence request
A useful answer identifies an artifact, owner, scope, and limitation. "We are accurate" is not an answer; a test report with task definition, labeled sample, threshold, group results, failure cases, and date may be. "We keep humans in the loop" is not an answer; a permissions screen, approval log, rejected action, and manual takeover can show how the control works.
Request examples that are difficult rather than merely diverse in appearance: borderline qualifications, title ambiguity, career gaps, international experience, nontraditional paths, stale records, contradictory sources, accessibility needs, and uncertain replies. Ask to inspect rejected or lower-ranked cases. Selecting only high-confidence successes inflates the apparent quality of every system.
Separate company-level assurances from workflow evidence. Security reports, privacy documentation, policies, and certifications may support governance questions; they do not establish candidate relevance, assessment validity, or hiring outcomes. A model card may document intended use and known limitations; it does not prove the buyer will use the system as intended.
Data questions
For sourcing, ask which sources are searched, licensed, inferred, enriched, or supplied by the buyer; how often they refresh; which geographies and populations are weak; and how a person can correct or remove data. A large profile count is inventory, not evidence of current coverage or permission for every downstream use.
For assessment, identify training, validation, configuration, and production data separately. Ask whether protected or sensitive characteristics are collected, inferred, proxied, or excluded; how labels were created; and what the system does when data is missing or contradictory. The ICO's recruitment audits show why purpose, minimization, transparency, retention, and inference deserve direct procurement questions.
Map every transfer and retention location, including model providers, subprocessors, analytics, logs, exports, support tools, and backups. Record customer controls for deletion, retention, model training, human review, and regional processing. Do not infer that a general privacy page describes the contracted configuration.
Candidate-facing risk
Ask what candidates are told about AI use, what information is evaluated, how they request an accommodation, whether an alternative path exists, how they correct relevant data, and how they reach a person. Notice is not meaningful when it arrives after a consequential action or uses language that the candidate cannot connect to the actual process.
Test the complete candidate journey with keyboard navigation, assistive technology where appropriate, mobile and low-bandwidth conditions, multiple languages, time limits, interruptions, and error recovery. WCAG can inform web testing, while ADA.gov emphasizes that hiring technology should measure job skills rather than disability and should support reasonable accommodations. Neither is replaced by a vendor saying its interface is "accessible."
Inspect how the system treats uncertain or incomplete data and how human reviewers see confidence. Ask whether candidates can be rejected solely from automated output, whether reviewers can access the underlying evidence, and whether an appeal or reassessment changes the record. A human click does not create meaningful oversight if the person lacks time, authority, or explanation.
Operational control
Ask the vendor to demonstrate the exact ATS, CRM, calendar, identity, and communication flows the buyer will use. Inspect field mappings, duplicate handling, retries, idempotency, permissions, error states, audit logs, and what happens when the downstream system is unavailable. A connector logo does not prove production depth or data fidelity.
For agents that communicate externally, test sender authorization, message approval, channel restrictions, timing, suppression, opt-out handling, uncertain replies, follow-up limits, and kill switches. Distinguish a recommendation from an executed action in logs and user interfaces. Require a manual path that can complete the hiring workflow without losing context when automation pauses.
Apply least privilege. A search agent does not need permission to reject an applicant; a scheduling tool may not need access to an entire candidate record. Ask how tool permissions are provisioned, reviewed, revoked, and changed by plan or feature updates. Include support access and human service teams in the same map.
Cost beyond license
Build total cost around the evaluated workflow: subscription or usage, implementation, integration, data, model or communication overages, security review, training, change management, recruiter operation, manager review, quality assurance, candidate support, compliance work, and switching. Ask which labor is performed by the vendor, the buyer, or a partner and whether that boundary changes by plan.
Compare remedy terms to the promised outcome. A credit for a failed search, a rerun, a replacement, service support, and a refund are different. Record definitions for a role, candidate, introduction, contact, screen, interview, usage unit, and expiration. Do not turn a marketing phrase into a contractual commitment unless the governing order form or terms say the same thing.
Evaluate viability without pretending to predict the company. Review support coverage, incident communication, roadmap dependency, data portability, exit assistance, subcontractors, and business continuity. Reference calls should ask what required manual work, what broke, how long remediation took, and what the customer would scope differently. Whether they "like the AI" is less useful.
| Category | Measure | Common omission |
|---|---|---|
| Commercial | Fixed, variable, minimum, overage, renewal, and remedy terms. | Credits or services excluded from headline price. |
| Implementation | Internal and vendor hours to configure, integrate, test, and train. | Hiring-manager and security-review time. |
| Operation | Weekly recruiter, reviewer, QA, support, and exception-handling labor. | Manual cleanup hidden behind an automated interface. |
| Outcome | Cost per qualified review, interested candidate, completed screen, or interview under agreed definitions. | Counting activity instead of delivered progress. |
| Exit | Export, transition, retraining, contract overlap, and lost workflow context. | Assuming data portability equals operational portability. |
Close procurement
The output of evaluation is not a completed spreadsheet. It is a decision record naming the selected scope, evidence reviewed, unresolved questions, rejected alternatives, accountable owners, required controls, commercial assumptions, and conditions for stopping or expanding. Missing evidence can be an explicit risk acceptance, a pilot question, or a reason not to proceed; it should not disappear into an average score.
Translate the remaining uncertainty into a narrow pilot. Choose one or a small number of representative roles, preserve a current baseline, predefine quality, sample both selected and rejected cases, limit candidate exposure, measure human labor, and schedule review checkpoints. The pilot design guide gives a protocol instead of a generic "try it and see."
Download the question bank and adapt weights only after the gating questions are answered. The template is intentionally vendor-neutral and contains no pre-filled vendor scores. Evidence links and notes remain with the buyer rather than being submitted to this site.
Reusable files
Evidence register
A lifecycle structure for governing, mapping, measuring, and managing AI risk and trustworthiness characteristics.
Evidence limit: Use of the voluntary framework does not establish legal compliance, product quality, or fitness for a particular hiring process.Technical assistance on job-related selection procedures, discriminatory impact, validation, and employer responsibility.
Evidence limit: The page describes federal considerations but does not determine whether a specific tool, employer, or use is lawful.Guidance on disability-related screening risk, accommodations, accessibility, notice, and measuring job skills rather than disability.
Evidence limit: The informal guidance is not a final agency action and cannot decide whether a particular process complies with the ADA.Procurement and deployment questions covering purpose, governance, accessibility, assurance, testing, pilots, transparency, and monitoring.
Evidence limit: The guide expressly does not provide legal assurance and its examples are not universal deployment instructions.Observed privacy and information-rights issues in recruitment sourcing, screening, and selection tools, plus remediation themes.
Evidence limit: Consensual audits of selected providers do not establish prevalence, legal status, or performance of another product.Testable web-content accessibility criteria across perceivability, operability, understandability, and robustness.
Evidence limit: WCAG conformance covers web content and does not by itself prove that an end-to-end hiring process is accessible or lawful.Official access to Local Law 144 materials on covered AEDT use, bias audits, public summaries, and candidate or employee notices.
Evidence limit: The overview does not determine whether a system or use falls within the law's definitions or satisfies its requirements.Current Commission explanations of scope, risk classification, employment use cases, obligations, and implementation timing.
Evidence limit: The FAQ is explanatory, timing can change, and classification depends on intended purpose and the facts of a deployment.Metix offers a first-party view of an outcome-delivery model that buyers can subject to the same workflow, labor, remedy, and pilot questions in this checklist.
Evidence limit: This is a vendor perspective, not an independent comparison or proof of return on investment.Relationship disclosure: OpenJobs AI is now Metix AI. Metix material is labeled first-party and is not treated as independent validation.
Questions teams ask
Compare only enough vendors to test materially different approaches and a credible status-quo option. A large list adds work without improving evidence if products solve different stages.
No. Use gates for non-negotiable controls and evidence, ordinal scores for maturity, and narrative notes for unresolved transfer or jurisdiction questions.
No. It may support a security-control question, but it does not establish selection validity, accessibility, candidate experience, sourcing quality, or operating outcomes.
Define the claim, request the relevant artifact, test it on a buyer-controlled case, inspect failures and exclusions, and record what the result still cannot establish.
Continue evaluating