In the fast-evolving world of machine learning (ML) and artificial intelligence (AI), vulnerabilities have become a critical concern, especially as the field scales up to more industries and applications. A recent research report by JFrog’s Security Research team highlights the increasing risk landscape associated with ML frameworks, uncovering dozens of software vulnerabilities in popular open-source ML platforms. The report shines a light on the relative immaturity of the ML field and its implications for security, a reminder that as machine learning continues to advance, so too do the threats surrounding its infrastructure.
The team at JFrog has discovered 22 unique vulnerabilities across 15 ML projects, encompassing widely used frameworks and tools. This extensive investigation is part of JFrog's effort to fortify the open-source ecosystem and comes as part of a larger two-part series focusing on both server-side and client-side vulnerabilities. JFrog’s research includes shocking findings, such as privilege escalation vulnerabilities in Weights & Biases (WANDB) and ZenML, exposing the extent to which malicious actors could manipulate ML tools in unexpected ways.
Server-Side Vulnerabilities: The Risks of ML Framework Manipulation
Server-side vulnerabilities often provide entry points for malicious actors to penetrate enterprise systems and, potentially, hijack key ML assets like model registries and data pipelines. For instance, the research highlights a directory traversal vulnerability within WANDB’s Weave server (CVE-2024-7340), allowing attackers to read any file on the system and potentially escalate their privileges to an administrator role. This vulnerability, fixed in version 0.50.8, underscores how unchecked access to sensitive files can allow attackers to target low-level resources within an organization.
In another example, JFrog disclosed a critical improper access control vulnerability in ZenML Cloud, which allowed attackers to escalate their privileges within ZenML’s tenant-based structure. By manipulating role permissions, attackers could gain admin access and extract sensitive information, such as credentials from the platform's Secret Store. These findings echo the importance of rigorous access control within complex cloud environments, particularly those hosting sensitive ML workloads.
Shachar Menashe, JFrog’s Senior Director of Security Research, explains, “This collection of ML vulnerabilities proves that the Machine Learning software world is still very immature and not yet built with a security mindset, which is why organizations should use an abundance of caution when deploying AI/ML tools. Companies would be best served to frequently poll for available updates to their AI/ML tools, as they are more likely to receive security patches in the coming years compared to more established software categories.”
Database Exploits and Prompt Injection Risks
The vulnerabilities don’t end with server-side ML frameworks; ML database frameworks and natural language processing tools also suffer from critical weaknesses. Deep Lake, a database optimized for AI, was found to have a command injection vulnerability (CVE-2024-6507), allowing attackers to execute OS-level commands via API misuse. Such vulnerabilities are particularly dangerous in ML environments, where data integrity is paramount, and database exploits can lead to data poisoning or model backdooring.
The threat posed by prompt injection attacks is particularly significant in ML applications where LLMs (large language models) interact directly with systems. Vanna.AI, a tool that translates natural language queries into SQL, was found to be vulnerable to such prompt injection attacks (CVE-2024-5565), with the potential for remote code execution. The implication here is that attackers could gain access to backend systems through compromised SQL queries, allowing them to manipulate data and wreak havoc within data-driven environments.
What the Future Holds: Balancing ML Innovation with Security
ML software remains vulnerable to a wide range of threats, with attackers often able to target both the backend infrastructure (such as databases and model registries) and front-end ML tools. These risks aren’t unique to lesser-known tools; even widely adopted platforms like Weights & Biases and ZenML are susceptible. JFrog’s findings reveal a systemic challenge: the immaturity of the ML field and the tendency of its practitioners to prioritize performance over security.
To safeguard against these vulnerabilities, JFrog recommends that organizations implement stringent access controls and prioritize regular security patches for their ML tools. Furthermore, companies should take a proactive approach to vulnerability management, regularly scanning ML environments for threats and ensuring the secure configuration of any third-party ML libraries or tools they rely upon.
The Path Forward: Security-First in MLOps
As companies embrace the potential of machine learning, securing the infrastructure that supports it has become non-negotiable. The report by JFrog's Security Research team serves as a sobering reminder that the same powerful models capable of transforming business operations also present attractive targets for bad actors. The real challenge is finding a way to innovate without compromising on security.
In an era where digital risks are ever-evolving, these vulnerabilities in ML frameworks emphasize the need for a security-first mindset across the entire MLOps lifecycle. From rigorous code reviews to automated threat detection, organizations can reduce their exposure by treating ML systems with the same caution they apply to traditional enterprise software.