From Confusion to Confidence: How AI Transforms Employee Benefits Knowledge from a D+ to an A-

We partnered with a large U.S. manufacturer to explore how AI impacts benefits education. The results were astounding: benefits understanding soared from D+ to an A-.

Every year, companies pour millions into helping employees understand their benefits—through call centers, portals, lengthy guides, and HR teams answering the same questions over and over. And yet, these resources don’t work. Call centers are offshored, HR teams are stretched thin, and benefits guides? Too long, too complicated.

So, what if there were a faster, smarter way to get the right answers? Enter AI.

The potential is game-changing. If an AI agent can answer an employee’s question in real time, it would be like each employee having a dedicated benefits expert ready to help when they need it most.

Yet, there’s been plenty of times when a new technology is over promised and under delivered. As exciting as this vision is, we wanted to test if AI really could deliver. So we had our Chief Medical Officer, Dan Dworkis, MD PhD, design a study for us to run with a large U.S. manufacturer. The study tested if AI could answer employee benefit questions better than existing resources, like benefit guides and intranets.

We’ll just say it: the results were astounding. By introducing an AI agent that employees could text message questions to, we saw employees’ scores on a benefits quiz soar from a D+ to A-. They also reported experiencing 74% less frustration compared to employees using traditional tools. 

The key takeaway: despite all the money spent, the current system doesn’t work. But AI can change that—dramatically improving how employees understand and use their benefits, while freeing up HR teams to focus on bigger-picture initiatives. 

Now, let’s dive into the details of the study.

The Question We Asked

Will employees be able to correctly answer questions about the specific benefits they have access to at their own company? And if they use an AI-based tool, does it improve both accuracy and satisfaction compared to traditional methods?

The Hypothesis

We believed that providing real-time, personalized answers via a text-based AI tool would help employees achieve better quiz scores and report lower frustration than those using guides, microsites, or call centers.

Expected Outcomes

1. Higher quiz scores about company-specific benefits.
2. Lower levels of frustration than employees relying on traditional methods.

The Study Method

We recruited 35 employees for this study: 19 used existing resources (like benefits guides, portals, or call centers) while 16 used an AI-powered tool developed by Airbo. Each participant answered 13 multiple-choice questions about their company’s specific benefits. 

The analysis was blinded—meaning the researchers didn’t know who used the AI tool when evaluating—and measured both quiz scores (accuracy) and frustration levels (rated on a 1–10 scale). All findings were verified as statistically significant.

Key Findings

From D+ to A– Students

On average, employees using traditional resources scored 69%, while the AI (Interactive Search) group reached 92%. This jump from “D+” to “A–” shows how a guided, interactive approach can dramatically improve employees’ grasp of complex benefits.

Perfect Scores

No employees in the baseline group achieved a perfect score, but 44% of those using the AI tool got every question correct. 

74% Reduction in Frustration

Employees also reported their frustration levels after each question. The AI group experienced a 74% drop in frustration, suggesting that a user-friendly, immediate feedback loop can significantly reduce stress and confusion.

A Note on Limitations

While promising and statistically significant, these results are based on a relatively small group of 35 participants. Larger-scale implementations could confirm—or refine—how effectively an AI tool improves benefits understanding in different organizational settings.

Conclusion

These findings show how AI can quickly elevate employees from “D+” to “A–” in their benefits knowledge while reducing frustration by 74%. It also frees HR teams from repetitive inquiries, allowing them to focus on bigger-picture initiatives.

Free AI Tool for HR

If you’d like to see how this approach can work in your organization, sign up for Airbo’s HR Assistant below. This free AI tool saves time on answering employee questions and routine HR tasks. 

→ Sign up for HR Assistant