Cybersecurity product reviews have put a bad taste in the mouths of many leaders across the industry in recent years, especially CSOs and security teams shopping for solutions. Years back, product reviews used to be highly sought after. Major security publications such as CSO Online, SC Media and eWeek used to do in-depth product reviews where there was actual product testing occurring in lab environments. Due to budget constraints, many of these formal review programs were either disbanded, made pay-to-play or significantly watered down. This left a major hole in the industry's unbiased review landscape. Gartner's MQ, Peer Insights program, and 'Cool Vendor' recognition become the only gold to hold in town for many security vendors. And too bad if you didn't fit into one of their categories. Furthermore, this recognition is constrained by the lack of a real product review or real-world testing process. Words on a page describing facts and figures and cherry picked customer testimonials are making or breaking company rankings. And Gartner Peer Insights are a lopsided view of a customer use base, as only positive reviews are encouraged and incentivised by marketing and analyst relations teams.
This has left many smaller vendors without a third-party validator and with the burden of having to fork over the funds to conduct product testing or produce product whitepapers via smaller analyst houses, such as ESG or 451 Research. Some startup vendors have taken testing and product comparisons into their own hands in an attempt to show the market that they are in fact either challenging the leader or flat out dethroning them using superior technology. This should be fair game, given the landscape, as long as the result of the review or comparison is re-creatable, publicly demonstrable and without overt caveats.
This letter from Orca Security Co-founder and CEO Avi Shua interested us when we read it. It demonstrates just how lopsided the cybersecurity product review landscape is, and how large vendors can throw their bank accounts and legal teams around in order to keep their products dirt-free from imperfect comparisons or reviews. Here is the letter in its entirety, originally appearing on the Orca Security blog:
The Cybersecurity Community Demands Transparency, Not Legal Threats
Abstract: A few weeks ago, Orca Security published a comparison between the Orca Cloud Security Platform and a few other cloud security tools—including a comparison with Palo Alto Networks Prisma. In response, Palo Alto Networks sent a cease and desist letter, demanding the comparison be removed immediately. Here is my response. I urge you to see the videos in question and if you, like me, believe the cybersecurity community deserves transparency and vendors shouldn’t be allowed to prevent publishing reviews or benchmarks via legal threats, then please share this post. You can also leave your own comments down below.
To: Palo Alto Networks
CC: The cybersecurity community
Subject: The Cybersecurity community demands transparency, not legal threats
Security has always been about transparency. The concept of security by obscurity was frowned upon as early as 1851—even before the invention of electricity—when Alfred Hobbs, a Massachusetts-based locksmith, demonstrated how then state-of-the-art locks could be picked. He explained that exposing the information would make the public more secure, as rogues already knew the deficiencies. The public needed to be educated, and he’d pursue better locks. Today’s locks are more advanced, but the principle is the same.
The cybersecurity community preaches about many products. All come with their own advantages and disadvantages, capabilities, and limitations. I believe that the only way practitioners can choose the tools that fit their environments best is by viewing factual evidence—not by relying solely on marketing materials. This is why we launched our Cloud Security Punch-Out! Series, where we deploy a few tools—including Orca Security—on the exact same environment and share the results with viewers who deserve to see them. I urge you to take a look at the one we did with Palo Alto Networks; as you’ll see we don’t hide those areas where Palo Alto Networks shines.
Unfortunately, Palo Alto Networks is now trying to use legal threats to prevent us from publishing these video reviews. In its letter, Palo Alto Networks does not point to any factual inaccuracies in the reviews of its products’ performance. Instead, it premises its threats on flimsy, boilerplate contract terms that prohibit reviews and comparisons of its products and hollow trademark allegations purporting that Palo Alto Networks is sponsoring the videos.
It’s outrageous that the world’s largest cybersecurity vendor (its products being used by over 65,000 organizations according to its website), believes that its users aren’t entitled to share any benchmark or performance comparison of its products. According to its boilerplate contract terms that prohibit “disclosing, publishing, or otherwise making publicly available any benchmark, performance, or comparison tests” of its products, you’re in violation even if you publish the results of an internal comparison of Palo Alto Networks against other products as part of your procurement process. The same goes for the hundreds of Palo Alto Networks reviews on various sites that include G2 Crowd, Capterra, and Gartner Peer Insights. It means that only benchmarks approved by Palo Alto Networks can be published.
Palo Alto Networks appears oblivious to the fact that the New York Attorney General’s office sued and won an injunction against McAfee from enforcing its contractual restrictions against publishing reviews or comparisons of its products without its consent more than 17 years ago. In enacting the Consumer Review Fairness Act, Congress has also prohibited businesses from including contract terms that prohibit consumers from reviewing products or services they purchase.
Palo Alto Networks, do you think your products are flawless or that the bad guys will follow along, not openly talking about products’ deficiencies? If the answer is no to both, then why resort to legal threats to remove such benchmarks and comparisons? I refuse to accept a world where any vendor believes it has the right to prevent the free flow of information, and control which product reviews are made publicly available.
I urge you to make your products better and focus your marketing efforts on demonstrating that, rather than throwing away money on ill-conceived gag efforts. Such action doesn’t benefit anyone. If you believe we missed something in our test, then tell us so we can make adjustments—we’ll happily integrate your comments and suggestions.
We could contract an objective third party to conduct additional tests. You could conduct your own tests with Palo Alto Networks and Orca Security’s products, then let the audience see and decide for themselves. All such actions would be far more beneficial to the industry, permitting both companies to learn and improve our products for the sake of customers.
As we all recently learned too well, sunlight is the best disinfectant. The cybersecurity community deserves better than a vendor’s lack of transparency while wielding dubious legal methods. Palo Alto Networks is the worlds’ largest cybersecurity vendor; with great power comes great responsibility. Your products are great—but nothing is perfect, and the public should have free access to all of the facts.
Yours faithfully, Avi Shua, CEO and Co-Founder Orca Security