How much do independent test results affect your security purchases?

Working in security, it always interests me when I speak to people who have chosen company X or Y based on promises from the vendor themselves. Every single vendor claims to be the best at what they do. I can't imagine there is a company out there aiming to be any less than a leader. But there are huge gaps in the quality of Threat Prevention solutions on the market at the moment. 

Many independent tests on hardware solutions pick the same handful of leaders. As an example I believe Miercom, Gartner and SS Labs put Check Point, Cisco and Fortinet on a pedestal above alternatives and ranked them pretty much in the order I wrote. For price, they're all willing to compete and will bend over backwards to add value (in my experience of selling them) and with the right partner you generally wont have major issues because you've got an expert on hand to set everything up and make it work. 

Do we need more independent testers? Better proof of independence? Sites like this obviously aim to bring that evidence to the user from the user. But I see reviews for Endpoint solutions that I know are factually ineffective at catching threats ranking in the top 1-5 which is surprising. Threat protection/prevention should be under constant scrutiny and nothing less than 100% should be the aim (regardless of how impossible 100% is.)

it_user428178 - PeerSpot reviewer
Technical Account Manager at a security firm with 51-200 employees
  • 13
  • 75
PeerSpot user
11 Answers
it_user297039 - PeerSpot reviewer
Network Engineer at a tech consulting company with 501-1,000 employees
Apr 20, 2016

I ripped off this line from a good friend a college a few years back because he put it so perfectly. Security is not a product, its a process. With that said I don't rely on any independent tests what so ever because the hardware and product can only go so far. For example, there are free and open source firewalls you can implement that utilize identical technology to the new Cisco Fire Power and the Fortigate FortiOS for threat protection and intrusion prevention. and you will need to have the same expert to set it up and maintain it. You will have equally as good and in some cases better community support for the day to day tasks and the performance is a dependent on the hardware and what you buy. But at the end of the day security breaches are going to happen. Your greatest defense is going to be an educated end user which is something that the "Independent Tests" wont tell you. nor will the sales reps. Don't get me wrong. YOU DO NEED SECURITY APPLIANCES. But you should do an internal analysis of your infrastructure to determine what features you are going to need and base your decisions off of that. Most of your claimed industry leaders are going to offer pretty much the same thing. They just have different Prices, Look, and Performance. They will also have different strong points over their competitors.

Search for a product comparison in ATP (Advanced Threat Protection)
Apr 20, 2016

One of the things that I see is that once a brand is stuck in someone's head, it stays there. I remember when Juniper was all the rage. However, I just removed the last Juniper firewall in our our organization and have replaced it with a Cisco ASA because of reliability.

Another issue that I have is that the publications that I receive which highlight these security solutions are actually sponsored by the big names (though if you don't do your research, you wouldn't know this). So you're thinking that this publication is singing its praises, it has to be good.

Unfortunately, while I do believe the independent testers would be very valuable, it needs to be made public so that it's seen by those who need to see it. I find more value in attending cyber security conferences, speaking with the vendors, and performing demos of their product on my network. Don't sell me "It can do this, this, and this on any network". Any network is not my network, and my network is the one that matters.

Give me 30 days to test the product on my network -- let THAT speak as the value of the product. There was one vendor that had some really cool technology, and I would go all in on the technology. However, after demoing the product on my network -- I wasn't seeing the value. As much as I loved the concept, it just was not worth the money to invest what they were asking.

I hope this answers your question

it_user428178 - PeerSpot reviewer
Technical Account Manager at a security firm with 51-200 employees
Jul 16, 2016

The response about Palo Alto surprises me greatly because vendors consistently beat them and it's usually always PAN that has unpatched vulnerabilities or major issues in the security press. In independent tests they're commonly known as 'vendor a' because they refuse to allow their name to appear on tests as they always fall short. There are some major flaws to the underlying tech that they've had to try and work on in recent times. To say they're the best seems crazy to me.

That said as somebody else said it is all about how you deploy a piece of hardware and in some circumstances they may well be the best solution. In other instances (see the Unknown 300) they're a risk. I once had a customer who asked me what questions they should ask specific vendors about vulnerabilities because they couldn't decide between PAN, Check Point and Fortinet.

The PAN salesman took the questions, read through them, stood up and left. I won't say who they ultimately picked but the other two vendors held their own and it was down to management capabilities and scope of power.

it_user285771 - PeerSpot reviewer
Director / CTO at a tech services company with 51-200 employees
Apr 20, 2016

If you are looking for the best protection you need to be looking at Palo-Alto or FireEye. These are the only two brands that can deliver what they do..

PA are by far the best product I’ve ever used from a security standpoint.. I’ve been doing IT security for 25 years and I won’t go past them..

We still support Cisco, Sonicwall, Cyberoam and Fortinet but they are all seriously lacking..

PA is not cheap. But it’s the best…

it_user306159 - PeerSpot reviewer
CEO at Makros SPA
Apr 20, 2016

This is not an exact science, I'm a VAR for several solutions, and from my point of view the first thing that determinates what solutions we will represent its the kind of support that we as partners will receive when we face any kind of situation. Besides that we are specialized in IT security so we also know a little bit of what`s in the wild when it comes to "bad things". Most of the solutions that are in the IT security market are very good, some of them with different approaches and other with different level of coverage, the important thing here's is that IT security must be a "system" that works in harmony with other solutions, and you have to work with certified partners and If you can, with specialized partners in order to get fastest response when it comes to troubles. My first experience in this market was selling a Korean antivirus in Chile, we succeed with that because we support that solution, and not because of what the testers say. Of course you will have brand buyers and tech lovers buyers, and this market requires better and more committed services.

it_user375090 - PeerSpot reviewer
Senior Cloud Security Architect at Rackspace, the #1 managed cloud company
Apr 20, 2016

Cisco is well known to have unpatched vulnerabilities on their ASA platform that are exploited in the blackhat community. No one in their right mind thinks of them as a reliable security partner. Even after the SourceFire acquisition. Fortinet and Checkpoint are good, but Palo Alto is far and away the best next-gen edge device. My buddy ran the largest Windows Active Directory domain in the world (US Air Force)...after deployment of Palo Alto devices they saw a 100x reduction in malware attacks and general network noise. I even deploy them at home and they have saved my MAcs more than once from 0day vulnerabilites that Apple had not yet released a patch. IMHO once they (eventually) acquire Imperva, they will further separate themselves from the pack.

That being said. Palto Alto, Checkpoint, Fortinet, etc. are nothing more than noise filters. You still need good security polices, properly hardened infrastructure, and executive buy-in that a good security posture is good risk management for the business. Also personnel that are able to discern the threats that get through your noise filters and hardened infrastructure!

Learn what your peers think about ESET Endpoint Protection Platform. Get advice and tips from experienced pros sharing their opinions. Updated: September 2023.
733,828 professionals have used our research since 2012.
it_user337209 - PeerSpot reviewer
User at Capgemini America
Real User
Apr 20, 2016

My guess is that the PoV (Proof of Value) with vendors is the best way.

Professioan Services Engineer at A10 Networks
Real User
Apr 20, 2016

I agree that the weakest chain in security is end user but at the same time not all companies have skilled people ( penetration tester ) to evaluate their security in place. You cannot easily test your current security product on every single layer of OSI ( layer 3 - layer 7 ) this is why people such NSS LAB to do the job for us at the first step when choosing which product to go with, consider this as base-line when evaluating product(s).

it_user296700 - PeerSpot reviewer
Security Systems Sales at a tech company with 10,001+ employees
Apr 20, 2016

Sure, the industry would benefit from more independent testing. Who's
going to do it? Everybody has a bottom line.

Professioan Services Engineer at A10 Networks
Real User
Apr 20, 2016

I agree with about the statement that every single vendor is the leader at what they do but the real question is how to measure the effectiveness of their solution(s)?

Personally, I don’t trust information available on Gartner report due to my understanding about how Gartner collecting such information by simply interviewing people. So, the report is totally based on personal experience / opinion with X or Y product(s) which is not what we want when deciding which ATP or any security product(s) to go with. On the other hand, I do trust NSS LAB as independent validation for many products in the market. NSS LAB published their testing methodologies to public for review by everyone. Add to this, NSS LAB do in-depth security product test reports, research, and analyst services.

What we need from NSS LAB is to cover more and more security products in the market to make it easy for professional / companies to choose trusted vendor in the market.

it_user391749 - PeerSpot reviewer
Co Owner and Managing Director at R3INVENTATE GROUP sprl at R3 CyberSecurity
Apr 20, 2016

Sure! Only independent tests could assure in some way, the accuracy or not of vendor figures.

Anyway, for these subject, I always try to use my common sense, taking into account that it is the less common of the senses....

First: If vendor figures don't cover my needs, no way. (I think this is normal for everybody...)

Second: if vendor figures double my needs, I will check....

Yes, it is true that maybe my way of work is not so funny, but for fun, I dive.
In my over 20 years of experience, there is something in the functionalities document, that you always understand in a different way and makes the different between being happy with the purchase or having a nice paperweight in your desktop....

So, after choosing 3 vendors with figures doubling my actual requirements, I will try them to be 100% sure. This way, our new Managed Security Service is working perfectly and our customers are 100% satisfied.

And regarding external testers, Gartner I don't trust, sorry, but the leader product is not valid in every market segment, so, either they split the Gartner quadrant for every single market segment, or their information will be valid for only big companies, having a really "always-the-same" situation... Here I have to say that finally we selected one of the leaders as Gartner says, but not the first one although I tried... (still has to return the test device).
And also, while working for a Cloud company, this company paid to be in the Gartner quadrant...

Anyway, yes, we need independent testers, but really independent and not paid by vendors in any way...

Related Questions
EhabAli - PeerSpot reviewer
Sr. Cybersecurity Solutions Architect at BMB
Jun 27, 2023
Hello peers,  I am a Sr. Cybersecurity Solutions Architect at a medium-sized computer software company.  What are the main evaluation criteria for you when choosing the right vendor for brand protection services? What are you usually expecting it to do? Thank you for your help.
2 out of 4 answers
May 17, 2023
Hi Ehab, I'm Karim Sabry from Giza Systems. It all boils down to how many channels can be monitored and where are they authorized to initiate a takedown and the time to take down (SLA). Channels can be Fake Internet Domains, Social Media various channels (LinkedIn, Facebook, Instagram, etc..), Dark Web chatter around brand and IP mentions, attack plans, and adversary motivations.
AJITHH G - PeerSpot reviewer
Solution Engineer at AppSmart
May 18, 2023
Sometimes you need to look for tools based on your business and compliance requirements.Otherwise, you will end up having too many tools for the same things, which doesn't make any sense.Evaluate the business and identify the requirements.Gather the dependency and check the skills of your resources to manage.
Cloud Security Architect at Kyndryl
Oct 26, 2021
Hi community members, I'm working as a Cloud Security Architect at a Tech Services Company with 10000+ employees. I'm looking for a security solution to detect and prevent APT attacks.  Can anyone suggest a good and cost-effective solution? Please explain why would you choose this particular tool or solution. Thank you!
2 out of 5 answers
Shibu Babuchandran - PeerSpot reviewer
Regional Manager/ Service Delivery Manager at a tech services company with 201-500 employees
Oct 25, 2021
Hi @Satish Singh, No single solution will 100% protect the environment. You need multiple layers of security working together, all the time, in addition to constant network monitoring. With that said, there are multiple ways to protect against advanced persistent threats. Install a Firewall Choosing a firewall is an essential first layer of defense against APT attacks. Software firewalls, hardware firewalls, and cloud firewalls are the 3 most common types of firewalls used – any of which will help you prevent advanced persistent threats. Enable a Web Application Firewall A web application firewall is a useful tool for defeating APT attacks because it can detect and prevent attacks coming from web applications by inspecting HTTP traffic. Install an Antivirus Up-to-date antivirus programs can detect and prevent a wide range of malware, trojans, and viruses, which APT hackers will use to exploit your system. Make sure that your antivirus can access real-time data and detect the newest threats, instead of only being able to recognize well-known malware. Implement Intrusion Prevention Systems Intrusion prevention systems (IPS) are an essential IT security service that monitors your network for any strange behavior or malicious code and alerts you if any is found. This is a powerful tool for recognizing network compromises before they can be exploited. Create a Sandboxing Environment A sandbox is a secure, virtual environment that allows you to open and run untrusted programs or codes without risking harm to your operating system. If a file is found to be infected, you isolate it, remove it, and prevent future infections. Install a VPN Remote access risks such as an insecure WiFi hotspot, present an easy opportunity for APT hackers to gain initial access to your company’s network. A virtual private network (VPN) provides an encrypted “tunnel” that you and your employees can use to access your network without cybercriminals snooping on your activity or gathering your data. Enable Email Protection Email is one of the most-used and most-effective forms of infiltration. Advanced persistent threat protection relies on good software as much as it does on good end-user behavior. Enable spam and malware protection for your email applications, and educate your employees on how to identify potentially malicious emails.
Sr.Customer Engineer- Projects at a tech services company with 201-500 employees
Oct 25, 2021
When you are considering cost-effectiveness Hardening perimeter defenses such as firewalls and antivirus are pivot points of preventing APT malware from being installed on your computer systems. Not sharing account details, recognizing phishing attempts at the first stage, safe web browsing at work. As per me, no clear-cut solution is effective... it's a mixture of solutions / tools you may use when you are tackling the aftermath... There are solutions like Trend Micro XDR which can trace back but not so cost-effective.  APT attacks use cutting-edge technology and hacking methods to sneak into a company’s system, So the best thing is Prevention...
Related Articles
Editor at a tech company with 11-50 employees
Sep 27, 2021
Small and big organizations often face targeted attacks. APT groups improve the quality of their operations, causing more serious damage. Timely detection and response, training of personnel, advanced training of information security department employees help reduce the risks associated with targeted attacks. The growth dynamics of APT (Advanced Persistent Threat) attacks has been declinin...
Related Articles
Editor at a tech company with 11-50 employees
Sep 27, 2021
More on Targeted Attacks and How to Protect Against Them
Small and big organizations often face targeted attacks. APT groups improve the quality of th...
Download Free Report
Download our free ESET Endpoint Protection Platform Report and get advice and tips from experienced pros sharing their opinions. Updated: September 2023.
733,828 professionals have used our research since 2012.