Working in security, it always interests me when I speak to people who have chosen company X or Y based on promises from the vendor themselves. Every single vendor claims to be the best at what they do. I can't imagine there is a company out there aiming to be any less than a leader. But there are huge gaps in the quality of Threat Prevention solutions on the market at the moment.
Many independent tests on hardware solutions pick the same handful of leaders. As an example I believe Miercom, Gartner and SS Labs put Check Point, Cisco and Fortinet on a pedestal above alternatives and ranked them pretty much in the order I wrote. For price, they're all willing to compete and will bend over backwards to add value (in my experience of selling them) and with the right partner you generally wont have major issues because you've got an expert on hand to set everything up and make it work.
Do we need more independent testers? Better proof of independence? Sites like this obviously aim to bring that evidence to the user from the user. But I see reviews for Endpoint solutions that I know are factually ineffective at catching threats ranking in the top 1-5 which is surprising. Threat protection/prevention should be under constant scrutiny and nothing less than 100% should be the aim (regardless of how impossible 100% is.)
I ripped off this line from a good friend a college a few years back because he put it so perfectly. Security is not a product, its a process. With that said I don't rely on any independent tests what so ever because the hardware and product can only go so far. For example, there are free and open source firewalls you can implement that utilize identical technology to the new Cisco Fire Power and the Fortigate FortiOS for threat protection and intrusion prevention. and you will need to have the same expert to set it up and maintain it. You will have equally as good and in some cases better community support for the day to day tasks and the performance is a dependent on the hardware and what you buy. But at the end of the day security breaches are going to happen. Your greatest defense is going to be an educated end user which is something that the "Independent Tests" wont tell you. nor will the sales reps. Don't get me wrong. YOU DO NEED SECURITY APPLIANCES. But you should do an internal analysis of your infrastructure to determine what features you are going to need and base your decisions off of that. Most of your claimed industry leaders are going to offer pretty much the same thing. They just have different Prices, Look, and Performance. They will also have different strong points over their competitors.
One of the things that I see is that once a brand is stuck in someone's head, it stays there. I remember when Juniper was all the rage. However, I just removed the last Juniper firewall in our our organization and have replaced it with a Cisco ASA because of reliability.
Another issue that I have is that the publications that I receive which highlight these security solutions are actually sponsored by the big names (though if you don't do your research, you wouldn't know this). So you're thinking that this publication is singing its praises, it has to be good.
Unfortunately, while I do believe the independent testers would be very valuable, it needs to be made public so that it's seen by those who need to see it. I find more value in attending cyber security conferences, speaking with the vendors, and performing demos of their product on my network. Don't sell me "It can do this, this, and this on any network". Any network is not my network, and my network is the one that matters.
Give me 30 days to test the product on my network -- let THAT speak as the value of the product. There was one vendor that had some really cool technology, and I would go all in on the technology. However, after demoing the product on my network -- I wasn't seeing the value. As much as I loved the concept, it just was not worth the money to invest what they were asking.
I hope this answers your question
The response about Palo Alto surprises me greatly because vendors consistently beat them and it's usually always PAN that has unpatched vulnerabilities or major issues in the security press. In independent tests they're commonly known as 'vendor a' because they refuse to allow their name to appear on tests as they always fall short. There are some major flaws to the underlying tech that they've had to try and work on in recent times. To say they're the best seems crazy to me.
That said as somebody else said it is all about how you deploy a piece of hardware and in some circumstances they may well be the best solution. In other instances (see the Unknown 300) they're a risk. I once had a customer who asked me what questions they should ask specific vendors about vulnerabilities because they couldn't decide between PAN, Check Point and Fortinet.
The PAN salesman took the questions, read through them, stood up and left. I won't say who they ultimately picked but the other two vendors held their own and it was down to management capabilities and scope of power.
If you are looking for the best protection you need to be looking at Palo-Alto or FireEye. These are the only two brands that can deliver what they do..
PA are by far the best product I’ve ever used from a security standpoint.. I’ve been doing IT security for 25 years and I won’t go past them..
We still support Cisco, Sonicwall, Cyberoam and Fortinet but they are all seriously lacking..
PA is not cheap. But it’s the best…
This is not an exact science, I'm a VAR for several solutions, and from my point of view the first thing that determinates what solutions we will represent its the kind of support that we as partners will receive when we face any kind of situation. Besides that we are specialized in IT security so we also know a little bit of what`s in the wild when it comes to "bad things". Most of the solutions that are in the IT security market are very good, some of them with different approaches and other with different level of coverage, the important thing here's is that IT security must be a "system" that works in harmony with other solutions, and you have to work with certified partners and If you can, with specialized partners in order to get fastest response when it comes to troubles. My first experience in this market was selling a Korean antivirus in Chile, we succeed with that because we support that solution, and not because of what the testers say. Of course you will have brand buyers and tech lovers buyers, and this market requires better and more committed services.
Cisco is well known to have unpatched vulnerabilities on their ASA platform that are exploited in the blackhat community. No one in their right mind thinks of them as a reliable security partner. Even after the SourceFire acquisition. Fortinet and Checkpoint are good, but Palo Alto is far and away the best next-gen edge device. My buddy ran the largest Windows Active Directory domain in the world (US Air Force)...after deployment of Palo Alto devices they saw a 100x reduction in malware attacks and general network noise. I even deploy them at home and they have saved my MAcs more than once from 0day vulnerabilites that Apple had not yet released a patch. IMHO once they (eventually) acquire Imperva, they will further separate themselves from the pack.
That being said. Palto Alto, Checkpoint, Fortinet, etc. are nothing more than noise filters. You still need good security polices, properly hardened infrastructure, and executive buy-in that a good security posture is good risk management for the business. Also personnel that are able to discern the threats that get through your noise filters and hardened infrastructure!
My guess is that the PoV (Proof of Value) with vendors is the best way.
I agree that the weakest chain in security is end user but at the same time not all companies have skilled people ( penetration tester ) to evaluate their security in place. You cannot easily test your current security product on every single layer of OSI ( layer 3 - layer 7 ) this is why people such NSS LAB to do the job for us at the first step when choosing which product to go with, consider this as base-line when evaluating product(s).
Sure, the industry would benefit from more independent testing. Who's
going to do it? Everybody has a bottom line.
I agree with about the statement that every single vendor is the leader at what they do but the real question is how to measure the effectiveness of their solution(s)?
Personally, I don’t trust information available on Gartner report due to my understanding about how Gartner collecting such information by simply interviewing people. So, the report is totally based on personal experience / opinion with X or Y product(s) which is not what we want when deciding which ATP or any security product(s) to go with. On the other hand, I do trust NSS LAB as independent validation for many products in the market. NSS LAB published their testing methodologies to public for review by everyone. Add to this, NSS LAB do in-depth security product test reports, research, and analyst services.
What we need from NSS LAB is to cover more and more security products in the market to make it easy for professional / companies to choose trusted vendor in the market.
Sure! Only independent tests could assure in some way, the accuracy or not of vendor figures.
Anyway, for these subject, I always try to use my common sense, taking into account that it is the less common of the senses....
First: If vendor figures don't cover my needs, no way. (I think this is normal for everybody...)
Second: if vendor figures double my needs, I will check....
Yes, it is true that maybe my way of work is not so funny, but for fun, I dive.
In my over 20 years of experience, there is something in the functionalities document, that you always understand in a different way and makes the different between being happy with the purchase or having a nice paperweight in your desktop....
So, after choosing 3 vendors with figures doubling my actual requirements, I will try them to be 100% sure. This way, our new Managed Security Service is working perfectly and our customers are 100% satisfied.
And regarding external testers, Gartner I don't trust, sorry, but the leader product is not valid in every market segment, so, either they split the Gartner quadrant for every single market segment, or their information will be valid for only big companies, having a really "always-the-same" situation... Here I have to say that finally we selected one of the leaders as Gartner says, but not the first one although I tried... (still has to return the test device).
And also, while working for a Cloud company, this company paid to be in the Gartner quadrant...
Anyway, yes, we need independent testers, but really independent and not paid by vendors in any way...