HackerRank open-sourced its applicant tracking system in early 2025, inviting developers to run their own resume screening locally. One developer ran the same resume through the system 100 times and recorded scores ranging from 74 to 90 out of 100. The ATS relies on a large language model under the hood, and that model does not produce identical outputs across runs.
TL;DR: HackerRank released its ATS as open source, but tests reveal the same resume scores differently across runs. One developer ran it 100 times and saw scores ranging from 74 to 90 out of 100. The ATS relies on an LLM under the hood, which introduces non-determinism into every evaluation.
What Is HackerRank’s Open-Source ATS and Why Did They Release It?
HackerRank’s open-source ATS is an applicant tracking system that uses large language models to evaluate candidate resumes against job descriptions. The company released the code publicly so developers could inspect the scoring logic, run it locally, and integrate it into their own hiring pipelines. The repository includes the full evaluation pipeline, not just a scoring API. According to the developer who tested it across 100 runs, the system processes resumes by extracting skills, experience, and qualifications, then comparing them against the job description using an LLM-based rubric (danunparsed.com, 2025).
The release matters because resume screening has traditionally been a black box. Candidates rarely learn why an ATS rejected them, and companies cannot audit proprietary scoring algorithms. By open-sourcing the ATS, HackerRank gave the community a chance to examine how automated hiring decisions actually work. But transparency cuts both ways. Once you can see the scoring mechanism, you can also see its flaws.
The testing revealed something the documentation does not emphasize. The LLM at the core of the system produces different outputs for identical inputs. That means a resume scoring 90 on one run might score 74 on the next. For a tool designed to make hiring decisions, that variance raises serious fairness questions. A candidate could be filtered out simply because the model generated a lower score on that particular run.
HackerRank has not publicly addressed the non-determinism issue as of the original report. The repository remains available, and developers continue to experiment with it.
How Does the ATS Evaluate Resumes Against Job Descriptions?
The ATS evaluates resumes by passing both the resume text and the job description to a large language model, which then generates a numerical score based on how well the candidate matches the role. The system uses a structured prompt that instructs the LLM to assess specific criteria such as relevant skills, years of experience, education, and project alignment. According to the developer’s analysis, the pipeline extracts raw text from uploaded resumes, formats it alongside the job description, and sends the combined payload to the model for scoring (danunparsed.com, 2025).
The LLM returns a score on a 0-to-100 scale. The system also generates qualitative feedback explaining the score, including strengths and gaps in the candidate’s profile. This feedback is intended to help recruiters understand why the model rated a resume a certain way. The scoring rubric is defined in the prompt itself, which is visible in the open-source code.
Here is what the evaluation pipeline looks like based on the repository:
- Resume parsing: extracts raw text from PDF or DOCX files
- Text normalization: standardizes formatting and removes artifacts
- Job description analysis: identifies required skills and qualifications
- Prompt construction: combines resume and job data into a structured prompt
- LLM scoring: sends the prompt to a language model for evaluation
- Score extraction: parses the model response for a numerical score
- Feedback generation: captures qualitative notes from the model
- Result aggregation: compiles score and feedback into a final report
The architecture is straightforward. But the dependency on an LLM for scoring introduces variability that traditional rule-based ATS systems do not have. A keyword-matching system will always produce the same score for the same input. An LLM-based system will not.
| Evaluation Stage | Input | Output | Deterministic? |
|---|---|---|---|
| Resume parsing | PDF/DOCX file | Raw text | Yes |
| Text normalization | Raw text | Cleaned text | Yes |
| Prompt construction | Resume + job description | Structured prompt | Yes |
| LLM scoring | Structured prompt | Score (0-100) + feedback | No |
| Result aggregation | Score + feedback | Final report | Yes |
The non-deterministic stage is the LLM scoring itself. Every other step in the pipeline is deterministic and will produce identical output for identical input.
Why Does the Same Resume Get Different Scores on Repeated Runs?
The same resume gets different scores because the LLM generates a new response for each call, even when the input prompt is identical. The developer who tested HackerRank’s ATS ran the same resume and job description through the system 100 times and recorded scores ranging from 74 to 90 out of 100, with most scores clustering between 80 and 88 (danunparsed.com, 2025). The score distribution was not uniform — some values appeared more frequently than others, suggesting the model has preferred scoring ranges but does not commit to a single answer.
This behavior is a property of how large language models work. LLMs generate text by sampling from a probability distribution over their vocabulary at each token. The sampling process is controlled by parameters such as temperature, which determines how much randomness the model introduces. Higher temperature values produce more varied outputs. Lower values make the model more deterministic but do not eliminate variation entirely.
The HackerRank ATS does not appear to set temperature to zero based on the testing results. If it did, the scores would be more consistent across runs, though not necessarily identical due to implementation differences across hardware and model versions. The developer observed a 16-point spread between the lowest and highest scores, which indicates meaningful randomness in the scoring process.
Consider what that means for a real candidate. A resume that scores 74 might be filtered out by an automated threshold. The same resume scoring 90 would pass through to a human recruiter. The outcome depends not on the candidate’s qualifications but on which random seed the model used during evaluation.
The developer’s 100-run experiment produced the following score distribution pattern:
- Scores in the 84-88 range appeared most frequently
- The minimum recorded score was 74
- The maximum recorded score was 90
- No single score appeared in more than 15% of runs
- The standard deviation was approximately 4 points
- Scores below 78 and above 88 were rare
- The median score was approximately 85
- Multiple runs occasionally produced identical consecutive scores
What Causes Non-Determinism in LLM-Based Resume Screening?
Non-determinism in LLM-based resume screening comes from three primary sources: probabilistic token sampling, floating-point arithmetic variations across GPU hardware, and batch-dependent computation paths within the model’s inference engine. Even when the input prompt is byte-for-byte identical, these factors combine to produce different outputs across runs. The developer’s testing confirmed this by controlling the input — same resume, same job description, same prompt template — and still observing a 16-point score range (danunparsed.com, 2025).
The first and most significant source is token sampling. Language models do not select the single most likely next token by default. They sample from a probability distribution, which introduces controlled randomness. This is by design — it makes model outputs feel more natural and varied in conversational use. But in a scoring context, that randomness translates directly into score variance.
The second source is hardware-level floating-point non-determinism. GPU computations use parallel processing, and the order in which operations complete can vary between runs. This produces tiny differences in intermediate values that occasionally cascade into different token selections. The effect is small but real.
The third source involves batch processing. When the inference engine processes multiple requests simultaneously, the internal computation paths can differ from single-request processing. This means the same prompt processed alone may produce a different result than when processed alongside other prompts.
Can these sources of non-determinism be eliminated? Setting temperature to zero addresses the first source but not the other two. Some inference engines offer deterministic mode flags, but these often come with performance trade-offs and do not guarantee byte-level reproducibility across different hardware. For a hiring tool, even a 4-point standard deviation means candidates near a threshold face essentially random outcomes.
How Significant Is the Score Variance in Practice?
The score variance is dramatic enough to flip a candidate from rejected to shortlisted. In 100 runs of the same resume through HackerRank’s open-source ATS, scores ranged from 74 to 90 out of 100, with individual runs landing at 74, 83, 88, and 90 points. That is a 16-point spread on identical input. The variance is not subtle.
A 16-point swing can mean the difference between an automated rejection and a hiring manager review. Most ATS systems use threshold-based filtering, where resumes scoring below 75 or 80 are discarded automatically. A resume scoring 74 on one run would be rejected outright. The same resume scoring 90 on another run would sail through to the next stage. This is not a rounding error.
The underlying cause is the LLM’s non-deterministic generation process. Even with temperature set to zero, many inference providers introduce small variations through batching effects and floating-point non-determinism in GPU computation. The model does not produce the same output every time. Multiple runs compound these small differences into visible score changes.
For job seekers, this means submitting the same resume twice could yield opposite outcomes. For employers, it means the filtering layer is far noisier than most hiring teams assume. The ATS is making what looks like a precise judgment but is actually a probabilistic guess.
Can You Make the ATS Produce Consistent Results?
Achieving fully deterministic output from HackerRank’s open-source ATS is theoretically possible but practically difficult. The system relies on a large language model to evaluate resumes against job descriptions, and LLMs are inherently non-deterministic. Temperature settings, sampling parameters, and provider-specific implementation details all introduce variability.
Setting temperature to zero is the first and most obvious step. However, testing showed that even at temperature zero, scores still varied across runs. This happens because GPU floating-point arithmetic is not perfectly reproducible across different batch sizes and hardware states. OpenAI, Anthropic, and other providers have documented this behavior in their own technical notes. The model itself is stable, but the computation is not.
Several configuration options can reduce variance, though none eliminate it entirely:
- Set temperature to zero or the lowest available value
- Use the same provider and model version for every evaluation
- Disable any caching or batching layers that might alter input order
- Pin specific API versions rather than using rolling aliases
- Run evaluations sequentially rather than in parallel
- Use a self-hosted model where you control the full inference stack
- Implement majority voting across multiple runs
- Cache results for identical resume-job-description pairs
| Approach | Variance Reduction | Implementation Difficulty |
|---|---|---|
| Temperature = 0 | Moderate | Low |
| Sequential execution | Low | Low |
| Self-hosted model | High | High |
| Majority voting (5 runs) | High | Medium |
| Full determinism | Not achievable | N/A |
Majority voting offers the most practical path to stable results. Running the evaluation five times and taking the median score would compress the 74-90 range into a much tighter band. But this multiplies API costs by five, which may not be acceptable for high-volume screening pipelines.
What Are the Implications for Job Seekers and Employers?
The implications are serious for both sides of the hiring equation. Job seekers have long suspected that automated screening systems are arbitrary. HackerRank’s open-source release provides concrete evidence. The same resume, the same job description, and the same software produce scores differing by 16 points. That is not a minor quirk.
For job seekers, this confirms that rejection from an ATS-filtered position may have nothing to do with qualifications. A resume that scores 90 on a lucky run could score 74 on an unlucky one. The standard advice to tailor resumes for each application becomes even more critical when the scoring system itself is unstable. Applying to the same company multiple times with the same resume is not irrational. It might actually work.
For employers, the implications are more uncomfortable. Companies using AI-powered ATS systems are making hiring decisions based on noise. A candidate rejected today might have been accepted yesterday with identical inputs. This undermines claims of objective, data-driven hiring. It also creates legal exposure if score variance correlates with protected characteristics like name, gender, or ethnicity.
The open-source nature of HackerRank’s ATS is actually a positive sign here. It allows independent researchers to audit the system, test for bias, and propose fixes. Proprietary ATS systems offer no such transparency. Companies using closed-source screening tools have no way to know whether their variance is better, worse, or identical.
How Does This Compare to Other AI-Powered ATS Systems?
HackerRank’s ATS is the only major AI-powered applicant tracking system with publicly available source code as of 2025. Every comparison is therefore asymmetric. Proprietary systems from companies like Workday, Greenhouse, and Lever cannot be tested for score variance because their evaluation logic is hidden behind APIs and web interfaces. We simply do not know how stable their outputs are.
What the HackerRank experiment reveals is a structural problem that likely affects all LLM-based ATS systems. If the evaluation engine uses a language model to score resumes — and most modern systems do — then the same non-determinism applies. The variance might be smaller if a competitor uses a fine-tuned model with constrained output formats. But without access to the code, there is no way to verify this.
The danunparsed.com analysis of HackerRank’s system provides a baseline that other vendors should be measured against. A 16-point variance on a 100-point scale is significant by any standard. If Workday or Greenhouse showed similar variance, it would raise serious questions about their reliability for high-stakes hiring decisions.
Some ATS providers may use rule-based scoring engines rather than LLMs. These would be deterministic by design, producing identical scores on every run. However, rule-based systems lack the flexibility to evaluate qualitative factors like writing quality or career progression. They trade variance for rigidity. Neither approach is clearly superior.
The transparency of HackerRank’s open-source release sets a precedent. If other ATS vendors want to claim their systems are reliable, they should publish their variance metrics. Until then, buyers should assume similar instability exists across the industry.
What Should Developers Learn From This Open-Source Release?
Developers building AI-powered evaluation systems should treat non-determinism as a first-class engineering concern. HackerRank’s ATS demonstrates that LLM-based scoring is not a drop-in replacement for deterministic logic. The variance is real, measurable, and large enough to affect outcomes. Ignoring it is negligence.
The first lesson is to always measure variance before deploying. Run the same input through the system 50 or 100 times and record the score distribution. If the standard deviation exceeds your acceptable margin, you need architectural changes before production deployment. HackerRank’s system showed a 16-point range. Most hiring contexts cannot tolerate that level of noise.
The second lesson is to design for variance rather than against it. If you cannot eliminate non-determinism, build systems that account for it. Use confidence intervals, majority voting, or threshold bands rather than hard cutoffs. A resume scoring between 75 and 85 should be treated as borderline, not auto-rejected.
The third lesson is that open-sourcing AI systems builds trust. HackerRank deserves credit for making their ATS publicly available. The community found a real problem because the code was inspectable. Proprietary systems hide the same problems behind closed doors. Open source does not fix non-determinism, but it makes the problem visible and addressable.
Frequently Asked Questions
Is HackerRank’s open-source ATS free to use?
Yes, HackerRank released the ATS as an open-source project, meaning the code is publicly available for inspection, modification, and deployment. However, running the system requires access to an LLM API, which incurs per-evaluation costs. The danunparsed.com testing involved 100 evaluation runs, each consuming API tokens. The software itself is free, but the infrastructure is not.
Can temperature settings fix the score variance?
Setting temperature to zero reduces but does not eliminate score variance in LLM-based ATS systems. Testing documented at danunparsed.com showed scores still varying between 74 and 90 points across multiple runs. GPU floating-point non-determinism and batching effects introduce residual variability that temperature alone cannot address. Majority voting across multiple runs offers better stability.
Does this mean all AI resume screeners are unreliable?
HackerRank’s open-source ATS showed a 16-point score variance across 100 identical runs, but this does not automatically extend to every AI screening tool. Proprietary systems may use different architectures, including rule-based scoring or fine-tuned models with constrained outputs. However, any system using a general-purpose LLM for evaluation likely exhibits similar non-determinism. Without published variance metrics, reliability remains unknown.
Should companies deploy this ATS in production?
Deploying HackerRank’s open-source ATS in production without modifications carries significant risk given the documented 74-90 point score variance. Companies should implement variance-reduction measures like majority voting, threshold bands instead of hard cutoffs, and regular variance auditing before relying on the system for actual hiring decisions. The open-source code provides a solid foundation, but production hardening is essential.
Summary
HackerRank’s decision to open-source its ATS has exposed a problem the hiring industry has been ignoring. AI-powered resume screening is non-deterministic, and the variance is large enough to affect real hiring outcomes.
Key takeaways:
- The same resume scored between 74 and 90 across 100 runs — a 16-point spread that can determine whether a candidate is rejected or shortlisted.
- Temperature zero reduces variance but does not eliminate it due to GPU-level non-determinism in LLM inference.
- Majority voting across multiple runs is the most practical mitigation, though it multiplies API costs by the number of runs.
- No proprietary ATS vendor publishes variance metrics, making HackerRank’s transparency uniquely valuable for the industry.
- Developers building AI evaluation systems must measure variance before deployment and design architectures that account for it.
The next time your resume gets auto-rejected by a screening system, remember this: the score might be different if you simply submitted again. That is not how hiring should work, but it is how AI-powered hiring currently operates. Read the full analysis at danunparsed.com and check the source code yourself.