TELUS Digital’s Michael Freenor on Why ASR Distributions Are the Next Big Step in LLM Security Testing

We sat down with Michael Freenor, Manager of Applied AI at TELUS Digital, to discuss why traditional single-point testing falls short in assessing large language model (LLM) security. He explains how attack success rate (ASR) distributions uncover hidden vulnerabilities and offer a more realistic risk profile. Freenor also discusses the potential for ASR metrics to become an industry standard and what CISOs should prioritize when building AI security programs.

How does moving from single-point attack testing to ASR distributions give security leaders a more accurate picture of how reliably their LLMs can be compromised?

Single-point testing only offers a binary view, showing whether an attack succeeds or fails, but it misses the variability in how LLMs respond. Using attack success rate (ASR) distributions, where the same attack is repeated multiple times, gives security leaders a more complete view of how discoverable vulnerabilities in their system are. It also highlights vulnerabilities that might otherwise seem minor and provides a clearer, more realistic assessment of risk. Trying attacks multiple times may turn up vulnerabilities that a single try may miss either due to randomness or bad luck.

What should CISOs take away from your findings when prioritizing investments in LLM security controls and incident response planning?

CISOs should understand that vulnerabilities in LLMs are often subtle and can shift over time. This is why continuous, iterative testing is needed, rather than one-off evaluations. Investments in security programs should prioritize tools and methodologies that support repeated and continuous test coverage. Just because a test has been run before doesn't mean it shouldn't be repeated multiple times for every new iteration of a product.

Could ASR distribution metrics become a standardized way for organizations—or even regulators—to assess AI model safety across the industry?

Absolutely. ASR distributions offer a statistically robust way to evaluate LLM safety by capturing both the likelihood and variability of successful attacks. Calculating an ASR for a specific attack is similar to measuring a discoverability score, showing how likely it is that the attack would succeed if tested in the real world.  A metric like this could feed into broader risk models, allowing organizations to adjust their risk outlook based on how likely it is that a vulnerability would be exploited, once an attack becomes known.

Do you believe the OPRO methodology and ASR-delta insights will generalize across different vendors’ LLMs, or will each require bespoke testing?

The OPRO methodology and ASR-delta insights have general applicability, but their effectiveness can vary based on the specific architecture and implementation of an LLM. While the underlying principles remain consistent, achieving optimal results may require tailored adjustments for each vendor's models and use-cases.

How practical is it for enterprises with limited budgets or staff to implement this level of rigorous, repeated red teaming against their AI systems?

Rigorous and repeated red teaming for AI and LLMs is complex, resource-intensive and challenging to scale cost-effectively. For many enterprises, especially those with limited budget or in-house expertise, it is more practical to outsource these services to specialized contractors, or to use third-party tools designed to automate red-teaming and reporting. The latter can often provide a more scalable and cost-effective solution.