CRITICALSTART Reviews

We were looking for a third-party managed detection response provider for our integrations with Cylance and Carbon Black. We had to deploy the Cylance and Carbon Black agents after we received them from CRITICALSTART.

Types of challenges that we were looking to address:

• 24/7 monitoring
• Reducing alerts.
• Getting Level 0 and 1 taken care of, along with that first triage of alerts. Those are taken care of before our team has to look at it.

The way that the user interface presents data enables our team to be able to make decisions significantly quicker, rather than have to dig into the details or go back to the original tools.

The transparency of data in the platform is perfect. The way they built it out, you are seeing everything as they are seeing it. There is not a black box; it's not the magic sauce happening behind the curtain. You have the ability to see everything that they do right there in the console.

The service has significantly increased our analysts’ efficiency to the point that they can focus on other areas of the business. We went from triaging an email inbox and a few other tools to being able to manage the queue appropriately at regular intervals. We also have begun looking for other tasks or items to further advance some of the analysts' careers.

Services have been fully delivered on time, on budget, and on spec. Whether it be for implementations, go-lives, or enhancements for anything that we want to add to the platform, they have always been consistent, ready, and willing to help out, build out, and troubleshoot should there be any issues.

Their Zero Trust Analytics Platform (ZTAP) engine, which is kind of their correlation engine, is by far and away one of the best in the business. We can filter and utilize different lists to build out different alerts, such as, what to alert on and when not to alert. This engine helps reduce our number of alerts and false positives.

The service's Trusted Behavior Registry helps the provider solve every alert. The way that they have it built out is very intelligent. The way every alert comes in, it gets triaged one direction or another. If it is already a false positive, then it is still getting addressed and reviewed on a regular cadence. Also, true positive alerts get escalated to the appropriate personnel.

Its mobile app is great. The ability just to be able to quick reference and see what's coming in when you're on the move or go. You don't always need to have your computer or laptop handy, because you can operate it just from the mobile app. It can communicate with analysts, which is great.

The mobile app is great at affecting the efficiency of our security operations. Those guys are using it throughout the day, whether that be at the office, home, or off hours. Typically, they triage from the mobile app. Then, if an escalation needs to be done on a computer, they will pull out a computer.

We were on the original UI for a few years, so the updated UI has been a refreshing change. It has significantly more ability to filter and translate data, then load that data. It is rather intuitive to click through for some of our junior analysts or interns, especially as we are starting to onboard and teach them different aspects of the security operations team.

The biggest room for improvement is not necessarily in their service or offering, but in the products that they support. I would like them to further their knowledge and ability to integrate with those tools. They have base integrations with everything, and we haven't come across anything. They should just continue to build on that API interface between their applications and other third-party consoles.

We started using it in 2017.

We have about 15 to 20 users. That is a mix of the security team, sysadmin server administrators, and the network operations group.

Our team members talk regularly with CRITICALSTART's analysts. They go back and forth with them regularly on individual incidents or investigations as well as support calls or conversations around monthly trends.

The number one value their service, as a whole, provides is the people. They hire the right guys and train them. We can then leverage their knowledge of looking at the greater picture. They are able to see all of their different clients, then translate what they are seeing there to our individual instance.

Whether it be alerts that they have already given us, or if we want to do some different threat hunting, have different ideas that we're trying to dig into, or we need assistance with an investigation, they are always a phone call away. They have analysts ready and willing to dive into a specific issue, even if it's not related to something their service has provided or alerted us to.

We didn't have a third-party provider previously.

The primary reason that we went for a service like CRITICALSTART was just the need to lift the burden off of a small team. When we started with CRITICALSTART, there were four of us. Now, we are a team of 15 or 16, so our team has grown. However, being able to have that first layer with a first set of eyes on alerts, incidents, and investigations as they came in, it was a big point for us, rather than getting stuck in our backlog and trying to keep up.

We entered into an agreement to use CRITICALSTART's service, then it took us two months before we went live.

There was nothing significant that we had to do in addition to the initial setup. When we do firewall changes, we just do it through our agents and communicate back to CRITICALSTART appropriately. This took four to six weeks of our setup time.

Four people from our organization were involved in the setup: 

• Our security operations manager
• Our internal IT manager
• Our network operations team
• Myself, as I manage the security engineering team.

Monthly, we are looking at 10 to 12 million alerts that the Trusted Behavior Registry sees. Of that, about 250 to 300 get escalated to our team.

CRITICALSTART takes care of the Tier 1 and Tier 2 triage for us. We only escalate up when there is a true positive that needs to be investigated. On a weekly basis, this saves us close to 50 to 60 hours.

The pricing has always been competitive. They have always been good to us. They will make it a fight. They don't try to hide anything; it's always been fully transparent and well-worth what we pay for it.

There are SLAs within our contract regarding the different alert tiers. This was a big factor in our decision to go with this service. They are willing to stand behind their product and team, then put that in a contract. It is evident that they are doing the right thing for their clients. They have not missed any SLAs so far.

We also looked at CrowdStrike. Their service just wasn't quite as mature. They only integrated with their only product. 

We looked at Arctic Wolf, who is not local. Critical Start is just down the street from us. Being able to build that relationship locally was a big selling point as well.

Trust the CRITICALSTART team. For the products that they resell and support, they know them very well. As you go down that path, you have a good heap of knowledge to rely on. Do not try to build it out or figure it out yourself.

We have since transitioned Cylance and Carbon Black over to CrowdStrike. We still use them for that service and also use them for our SIEM, because they host and manage Splunk for us. That all integrates into ZTAP. Using that and any new products that we bring in-house, we work with CRITICALSTART to see if they have already gotten an integration connector built. Typically, we'll use theirs. If there's already something built, or they have the appetite to build it, we'll use that service as we onboard it internally as well as into CRITICALSTART.

The biggest lesson is transitioning from alert overload to being at a point where we do have eyes on alerts, where every alert is truly possible. It's something that a lot of people sell and not a lot of people do very well. Being able to come into this relationship, then where we're at today, it kind of opened my eyes to: There is the opportunity and the possibility to do this. Stuff is not going to get dropped or missed by our operations group.

I would give them a nine (out of 10). They are right there at the edge, probably a leader in the market. That's kind of why we chose them. Of course, there is always room to improve, but they're doing a lot of things right. We appreciate their team.

What I was looking to achieve with this service was to have less work on my plate, and to leverage people. Usually, when you buy a big product like an antivirus or endpoint protection, if it's a big solution and you have a big company, you need another person to just manage it or things like it. We didn't have those resources. We got the antivirus product, but we didn't have another person to add to it, so I needed someone to help me manage it.

CRIICALSTART is helping me manage this solution because I don't have time to manage it.

Originally, they were managing CylancePROTECT for us. Now, they manage CylancePROTECT, Carbon Black Defense, and Palo Alto Cortex XDR for us.

They take work off my plate and that frees me up to work on other things. The fact that I have time to do more of my job isn't game-changing for my company, but for me it's a huge deal. Otherwise, I'd be spread so thin. What would have happened if we didn't CRITICALSTART is that I would either have been getting thousands of alerts a day and having to ignore everything else, or we would have used a different security product that is less noisy but also less secure. And then, maybe, we would have been compromised and not even know it.

Our expectations have been met in terms of services delivered on time, on budget, and on spec. When you sign up with them, they tell you they're going to cut your alerts down by 99 percent, and they did that. They did that with Carbon Black Defense and they did that with XDR. That's all I could really hope for.

The most valuable feature of their service is their tuning. All the service really does is get things to the point where we get fewer alerts sent to us. If we were getting 1,000 alerts a day without them, they tune it until they know what to do for 999 of them, and one will make it through to us per day. That tuning is the most valuable part of their solution.

When we had Carbon Black, we were getting at least one escalated alert a day, maybe more, because it wasn't able to be tuned the same way that other services can be, or maybe Carbon Black itself alerts that much more. With Cortex XDR, we're only getting about one escalated alert a week, or one a month. It's much less.

They just did a user interface overhaul to the website portal that you use for troubleshooting tickets. The old one was fine. The new one is not intuitive and I hate it.

It's an information overload issue. When you go there, there is a bunch of stuff to look at. I had to get a walkthrough last week because I didn't know how to get to the one screen that I'm looking for when I use it, the one that shows the tickets that I have and the tickets that I don't have. I couldn't figure out how to get to that. In the middle of the main screen there's a little button that'll take you there. And at the top there's a search bar and a filter that helps you find tickets that are assigned to your organization or their organization, tickets that are open, tickets that are closed. But it's not intuitive.

I have been using CRITICALSTART for one-and-a-half years.

If they expanded the scope of what they can ingest and did so at good pricing for managing other services and remediating other issues, I would definitely look into expanding our usage. At this point, I don't know what else they take in, other than endpoint protection.

From a project management standpoint they have performed very well. They're very organized. They're very reliable and responsive. Their customer support is a 10 out of 10. I'm always happy to hear from them and see them.

I haven't had any problems since they've been managing XDR, but back with Carbon Black I had a lot of problems trying to understand why something was being alerted this way and why this or that was being blocked. They helped me troubleshoot all of that stuff as well. And they do it within their SLA. It's nice to have that insurance that they should be responding within an hour.

This is the first time I've used a managed service provider for managing anything like endpoint protection.

There was an initial setup required at our end to use their service and they helped me take care of that. It was very straightforward. There were a few settings for me to change and there were a lot of settings for them to change, and they just remoted into my machine and helped me do it. Either way it was not rocket science for me.

We've used this service with three different products. For the first one, CylancePROTECT, there wasn't a portal for me to log into. That was all behind the scenes. We didn't get to know what was happening. They just took care of everything. 

When we had Carbon Black Defense, we had the old portal, but that was a year-and-a half-ago and I don't remember how long it took to get set up. It hooked in pretty quickly. 

With Palo Alto Cortex XDR, we were either their first or one of their first customers to use that service, so it took a little bit longer to get everything set up correctly, even though we were already connected to them through the old service. We were in the system immediately, but we weren't in full-on production mode for about four-and-a-half months. That's not that bad because they were actively managing it until then.

I looked at Arctic Wolf. There were some others as well. But the pricing of other services was so insane that they weren't even an option. And they don't do exactly the same thing. CRITICALSTART has a narrow scope that fit our requirements. I had a problem and CRITICALSTART specifically works with that thing. I don't know if they do other stuff now, but when we started working together, pretty much all they covered was antivirus.

If you have people who already do this at your company, and they're paid well and they know what they're doing, and you have multiple products like this that they can manage, then you don't really need CRITICALSTART. But if you are a small group of IT people trying to support an entire company and you have a crazy, complex product like CylancePROTECT or Carbon Black defense or Palo Alto Cortex XDR, or anything like that, then it's probably better to leverage an expert company like CRITICALSTART.

The only data source we are using them to manage is our antivirus and they integrate with that. I don't know if they would have been able to integrate with our other data sources. We didn't try that.

I have used CRITICALSTART's mobile app but I haven't used it lately because we get so few alerts that I don't really need it. A lot of people use the mobile app for when they're home on the weekends and they need to get stuff remediated quickly. We don't have people working on the weekends, usually, so it's not a huge issue for us. If my company is working, I'm at my office and at my computer already so I don't need the mobile app for that.

The mobile app has the basic features that you need to use their service. I don't remember if it lets you link to the service they're managing; for example, I don't think there's a link to the Cortex XDR app from CRITICALSTART's mobile app. So you can't really dig deep into anything on there, but that's not their fault. It's just because you can't do that, period. But for quick remediation or quick alerting, it's perfect.

I haven't spoken to CRITICALSTART's analysts lately. During implementation, we had weekly meetings. Usually I only talk to them when things aren't going well, so the fact that I haven't talked to them in a while means we're good. But they were always available when I needed them. If I needed them quickly, they could join a meeting within a day.

Out of all the service providers I've had to work with over the years—I've been here six years—CRITICALSTART is my favorite to work with. I see them at almost every convention that I go to, no matter what city I'm in. I'm always happy to see them and they always recognize me. I feel like that's worth something when you're looking for someone to work with. They have a personal touch.

I have a very small team and anytime I can maximize efficiencies within the work I'm trying to do with Kirby, it's a good thing. That's what I was trying to do by using CRITICALSTART.

The most valuable part of the service is the time saved. CRITICALSTART helps with so many of these alerts that my team and I don't get alert fatigue. It saves us time to concentrate on the more important things. It probably saves us a day or two, 10 to 15 hours, a week.

I also talk to CRITICALSTART analysts and the value in that is immense. I just talked to my Board of Directors about that this morning. The value from it is what I'm spending on the service versus what I would have to spend to build a team like that internally. It's at least one-fifth of the cost. There's value in that for me. And their availability is generally pretty quick. I've never really had to wait very long for anything. The availability of the analysts where they will say, "Hey, I know we sent an alert on this, but you should really take a closer look at it," via a phone call or a message, is just phenomenal.

In a given quarter, I get 589,000 security events and 584,000 of those get reduced by the service before they even get to me. The alerts that actually come through to me end up being about 1,400 in that quarter, which is a 99.7 percent efficiency rate.

The Trusted Behavior Registry helps resolve alerts in the sense that CRITICALSTART is doing a lot of that initial triage for me. Out of a given 500,000 events and alerts, for example, that come through, they're taking out 495,000 of them. That only leaves me with a subset of that to actually have to triage, and that's where it benefits us. They take care of Tier-1 and Tier-2 triage.

And the new mobile app is awesome. It is one of the best I've ever seen. It's much better than its predecessor. It's more intuitive, a whole lot easier to navigate and get where you need to go. It's less repetitive and just generally easier to use. It allows me to not have to be sitting at my computer all the time. I can be on my phone or tablet or wherever I'm at. It makes it a lot easier to answer tickets and do that kind of thing.

Also, the intuitiveness of the updated user interface for the service is spot-on. It is much easier to navigate, and know where to navigate, in the newer interface. I've never had an issue with responsiveness. It's very quick and doesn't sit there and chug on anything. It's fast, it's efficient. It has enabled our SecOps team to take action faster because if you have multiple ways of connecting to it and actually getting your alerts answered and taking care of things fast, it is extremely helpful.

All the information that you need to make a determination is usually in the alert itself that comes through the Zero-Trust Analytics Platform (ZTAP). I don't find myself going back to the app itself very often. That still happens, but not as often. The ability to flow the information forward, from the alert standpoint, helps me because it saves me from running back to get the information. It's improved my efficiency.

Finally, there haven't been any data sources that the service wasn't able to integrate with.

The only thing I can think of that I would like to see, and I'm sure they could work this into a service pretty easily, is not only alerts on issues that are affecting my company, but some threat intelligence of a general nature on what's out there in the environment. That might be a nice add-in.

I have been using CRITICALSTART for about seven years now.

If I have issues, all I have to do is either send a message or a ticket over and ZTAP will pick up the phone and call somebody. It's pretty easy.

We were all internal prior to using CRITICALSTART for this. We didn't use a third-party external service to look at any of this data. We were actually doing it ourselves.

From the time we entered into an agreement to use this service until we could start using it, it was pretty quick. They jumped right on it from a project management standpoint. On a scale of one to 10, the project management aspect was a 10. Their performance was spot-on. I was actually using it, even though we were still tuning, within a week or so.

In terms of initial setup, you have to start pointing all your sources to the app to have them adjusted. Once you start doing that, you can start getting some data out of it. Within that week I started seeing events start coming through.

The initial setup is always straightforward. The complexity comes in the tuning, because then you have to say, "Is this normal? Is this not normal? Does this only happen once a year?" That's where the complexity comes in. The fine-tuning took a couple of months. But that was more on my side then it was on CRITICALSTART's side.

I was the only one involved in the setup from our company, and I'm the only user. Our entire domain reports into it from a SIEM perspective, and every node that we have is reporting in from an endpoint protection standpoint. That's 5,000 to 6,000 user nodes and probably another 1,000 servers. It's a 100 percent adoption rate. They don't get a choice.

Overall, for what I'm paying for it, and the benefit I'm getting out of it, it is right where it needs to be, if not a little bit in my favor. For what it costs me to actually have this service, I could afford one internal person to do that job, but now I have a team of 10 or more who are doing that job, and they don't sleep because they work shifts.

Licensing is always one of those things that you can have some degree of negotiation on. There are hard costs associated with the service because they're paying salaries. I always look for opportunities to improve from a pricing standpoint, but I've not been displeased, so far, with it.

I knew of a few other options. Alert Logic is one of them, and there was another one called Fulcrum that has a service now around it, but it's nowhere near the maturity of what CRITICALSTART has.

I also had an existing relationship with CRITICALSTART. We did have an issue and they stepped in and helped us with that issue and really went to bat for us. That helped build that relationship from a trust standpoint.

There wasn't any kind of bake-off. It's a close-knit community, so I didn't really have to go to that level. I knew I didn't want certain other ones.

The main difference between the other options and this one is the quality of the personnel within the SOC. It's their knowledge and depth and the way they handle customers. Their guiding principles fit really well to get you the best service that you can possibly get.

I would suggest using a phased approach, instead of dumping everything in from the beginning and then trying to sort it out, triage-wise. If you add types of sources or tools to it one at a time, instead of "everybody into the pool" right away, that really helps you. That way it allows you to get your handle on the smaller piece of the pie first and then work your way forward.

As for what to start with, it depends on what you're pushing to them. I didn't start necessarily right away with the MDR, but I did have my endpoint protection being looked at by them, at least. Then I added in my SIEM, which added to the overall complexity level. Unfortunately, I didn't have one completely finished before I added the next and that slowed me down a little bit. That was too much for one person to try to handle all by himself.

The biggest lesson is that even if you have a small team and limited resources, you can actually be effective as a company, from a security program standpoint, by using their service.

My expectations have been more than met in terms of service delivered on time, on budget, and on spec from CRITICALSTART.

We are using it to try and improve our cybersecurity overall. We are also using it to reflect on our business growth whether we need to invest in more cybersecurity.

We started as a small, family-owned  business which was purchased by a U.S. company under the same umbrella. That company wanted to have all their portfolios have a higher level of security. This was an initiative taken by the parent company. This came at the right time because we started to get more phishing attacks as we started to manage more users. There has also been more requirements on the IT department to keep us secure along with more focus in today's world on IT security. Previously, we didn't really pay as much attention because we always thought we were a small company, and thought, "Who would want to hack us?" I guess that is no longer the case.

The service for endpoint protection needs to have an agent installed on the endpoint, and that is pretty much it. There is no specialized hardware required to use their service.

It removed a huge task from my shoulders onto someone who it's their profession to do this because I'm not from a security background. It definitely makes my life a lot easier. In terms of company, we have invested in something sophisticated and management knows that we have access to a 24/7 service. It makes them feel happier as well, especially these days when you hear about attacks, etc. For them, knowing we have a service like this in place is a good thing.

I receive probably less than five alerts per week. Most of them are caused by OpenDNS, which means there is not much they can do. These happen when our workstation is trying to reach a destination with IP addresses, then it will raise an alert because it suspects someone is trying to bypass the DNS security to go directly to a certain destination. With that kind of alert, the only thing we can do if we don't think it's safe is block it in the firewall. With the service from CRITICALSTART, they don't have the capability to actually block individual IP addresses. That's why those alerts keep coming in whenever there's a new IP. Our regular processes, like our ERP software, are mostly filtered and no longer come up as alerts. This has being cut down by probably more than 80 percent compared to day one.

On whatever CRITICALSTART does, it will show up and be logged. If there is an alert, and someone made a comment or did something, it will all show up in one place. That has sort of a paper trail of what people did. Because we have agents installed on endpoints, I don't know exactly all the details of information that are sent to CRITICALSTART. I assume since this is Zero Trust, they probably be sending everything because we keep thousands of processes with a playbook and a whitelist of filters. So, I never go in and actually check exactly what's being sent over. As far as I can see, if they done anything, like putting something to a whitelist or triggering/disabling a filter, it all shows up.

Now, all I need to do is just go in. Luckily, we're relatively small. With most of the alerts, I'm able to address them right away because I know exactly what they are and they have done most of the leg work, then I ask the team if they will take care of the rest. That definitely saves a lot of time on my side. I can't really make a comparison between now and before last November, because we didn't do much because we weren't equipped to do much then. There might have been something going on, but we didn't know because we didn't have the resources for this kind of service.

We now have the tools and the support to actually have a clear view of what's going on. Before, it was just the traditional antivirus installed on the computer. Whatever it did, it was done without us because we couldn't really do much except block something or whitelist it. There were no humans involved. I'm not spending too much time on this because most of the jobs are done by the team from CRITICALSTART. All I do is just help them confirm whether the alert is legitimate or another regular process that we haven't playbooked yet.

The 24/7 SOC security: There is a team of people who monitor our traffic and processes 24/7, so if anything raises a flag or alert, it will escalate back to me right away. That's the most incredible part: Humans working behind the scenes 24/7 to monitor our networks.

The intuitiveness and responsiveness of the updated service's user interface is pretty cool, especially the dark theme, which I like. It is easy on the eyes. It's not like a traditional portal. It looks very futuristic, but I think it's more accessible and less crowded. The new interface is definitely an improvement. 

I am a one-man team. Everything is done by just me. I did find that it is easy to find things on the UI. I think it's an improvement from the one we had when I started.

Our infrastructure is very simple. The service covers almost all the endpoints, except that a service we use doesn't have a function that can control portable storage. It does scan everything, including whatever you have on a USB plugged into your computer. My suspicion is it will get there, but not right away. It doesn't have a special function to control the portable devices, and that's one thing I see lacking because sometimes we do have users who need this.

In terms of responsiveness, when I open up an alert, sometimes it takes a bit of time to load. However, it only happened once or twice. Most of time, it take just one click, then I'm there.

The dark theme might not be everybody's favorite. When I built the app for our users with a dark theme, everybody kept complaining. However, it's perfect for me and I like it a lot.

We have not been using it for very long. We started using it sometime around last November.

We do have every single endpoint covered. That's how extensive it is. One more thing we can do is have company issued mobile device coverage. We haven't done that. It's just that we don't have that many company issued mobile devices. Other than that, we have everything else covered.

As long as we are growing, we can probably stay with a service like this.

If I have a question, I do talk to the service provider's analyst, though not very often. This is partly because we're relatively small and don't have as many processes going on a daily basis compared to some of the bigger companies. If there is an alert that I don't quite get, then I will reach out. I think the best part about CRITICALSTART is that you have access to real human beings, and usually, their response is very timely.

It's too early to see but definitely there is value to their service. Because of the size of our business and also it's not a very complex business, we have had maybe two or three incidents that were close to real threats, but not even a threat. They were just some user transfer files, which were questionable, but not malware. However, they are not something we want to have in our system. We have only been working with CRITICALSTART for less than a year, but I do see value in terms of having a team of professionals that we can access anytime that we want to provide more peace of mind. It's like an insurance. You don't have to use it, but having it in place definitely makes us feel better. Given how many phishing emails we receive every day, their service will be become more valuable down the road. Right now, they haven't had the chance to prevent a real attack or threat yet.

In terms of support, they're really good. They have quarterly meetings where we actually talk to an engineer and their support to just go through what has gone on in the past quarter. They will give some tips on how to respond to their tickets. This makes you feel like they have your back all the time. The service side of things is really great. When they see a concern, they reach out and help just to make sure that I actually know what I'm doing.

We were able to start using it almost right away, mainly because this was an initiative taken by our parent company. We got top priority. From the day we signed the contract to the day we started the tuning process, which was during Christmas, it was maybe two to three weeks max because there are things that I had to do on my side. I had to install all the agents on the endpoint. That was the only requirement. But if I remember correctly, it was pretty quick.

Most of the service is very straightforward. We did have a little problem removing it from endpoint, and I had to select that change in the portal. That was the only challenge we had. Part of the service does require us to set up a DNS forwarder onsite, and that took a bit longer than the rest. Overall, everything is very straightforward. Also, when this problem came up, the support was very efficient.

It was a bit worse initially because there would be some Zero Trust; it didn't trust anything. We did have to spend a few months of time building a playbook to whitelist all our common processes and the software that we use. But, as time goes on, all these rapid program were playbooked, then we started to see real behaviors that might cause problems. I think this is a very good approach. It's definitely labor-intensive, but mostly on their side, because that's the service that they provide.

Once they created the playbook, we saw less alerts on a daily basis. I will still see some alerts that were caused by some of our less used programs, which maybe just start triggering alerts. Also, we can start seeing things that look more like real threats, but this stopped a long time ago because of the Zero Trust policy. So, anything new to them will raise a flag, and we will work together to add a filter or block it.

From a project management standpoint, the service provider is pretty good. The onboarding process is very smooth despite the fact that it was Christmas season. Right after we signed the contract, I went on a vacation so they were able to speed up things and make sure that we had this thing up and running before I left for vacation.

If you consider sleeping better at night as a return, there is definitely a return in that. It is a comfort to know that there is a team of professionals backing you up, especially in an area that you don't feel 100 percent comfortable. Because we never had an incident in the past, we can't really see whether the service has earned every penny that they charge. Sometimes, I still wonder if I had just gone with Sophos, would we have gotten the same result?

Our expectations have been met in terms of service delivered on time and on spec. It's just the time limits of the response and friendliness of their support. You don't see that in every service provider.

It costs a lot for what we felt comfortable to spend.

We just decided to bite the bullet because we have to do something as a requirement first, and we have to have all our areas covered. In terms of pricing, we probably got a good deal because we are part of a bigger organization now, so we got a discount. But in this case, I guess you get what you pay for. For security, there's a balance somewhere regarding how much money you can spend in relation to how much value it's generating every year. There must be some sort of guideline out there to say what the percentage of IT spending is acceptable. I think it really depends on each company. In my experience with CRITICALSTART, I think if you have the resources to use the service, go for it. Definitely, I think it's worth it.

Before we committed to CRITICALSTART, we did shop around. We saw two approaches:

1. Having real humans to go through every single process and help create playbooks. 
2. Using some sort of artificial intelligence, but still trying to do the same thing. 

I definitely prefer to have a real team working on this rather than AI, because AI is still not as smart as we would hope it to be. However, it definitely costs more when real people do the job. If a resource is not a problem, I would definitely recommend this type of solution.

We had a few meetings with the guys from Sophos because they came in highly recommended by our teams in the same industry. At that time, we were still in some sort of transition from the family owned business to the larger business. Therefore, we thought Sophos would fit our bill better, as they are cheaper. They have good service. They also have hardware appliances they we were interested in buying. We thought it would be a good fit to our business because we weren't budgeted as much to use a service like CRITICALSTART. We had quite a few meetings. We even had those meetings with the person from our parent company who took the initiative to talk to all their portfolios to push a corporate-wide solution so that we could get better discounts.

We ended up not going with Sophos because:

1. As a service, Sophos was all new for us. We had never used them before. 
2. CRITICALSTART Zero-Trust platform is somewhat more attractive to our non-technical management. It sounds like a lot better idea not to trust anything. 

At the end of the day, CRITICALSTART was recommended by a consultant company, which was used by our parent company. So, we thought if Sophos was new to us, it's probably safer to go with what they recommended just in case something happens. That's why we went with CRITICALSTART. Initially, we just felt like it was a huge jump from what we used to have. We were a little bit uncomfortable at first. Once we get used to it, it was a good service and I think we can afford it.

So far, I'm very happy with the service. However, we have no comparison. This is the first ever MDR service that we have used. We have not had enough time to really verify the protection that the service offers is enough because we haven't suffered any attacks. We don't know whether we're lucky or if the service really does work. 

You can never do enough to stay safe. It has helped me to see a lot of things going on with our network that I didn't see before. We were just not equipped with the right tools to really have a clear view of our network, and now we do. 

For smaller companies, in order for them to grow, they have to trust the professionals. Sometimes, we tend to save every dollar possible and do everything on our own, either by reading a book or taking a course. It's a good thing to learn new things but I learned that no one can cover every aspect of a company's IT needs. When the time is ready, you need to leave certain things to the people who are really good in that area, freeing up yourself to do things that you are really good at.

I would give it nine out of 10 because of the pricing. So far, that's the only downside that I can see.

We're a small shop on the security side and our goal with CRITICALSTART was to alleviate some of the constant looking at our phones 24/7 and allowing somebody who is actually sitting in front of a computer 24/7 to handle the front end alerts that come through our automated services or systems. As those come in, we wanted them to be able to escalate to us as seen fit. We were looking to weed out the lower priority, false-positive portion of the alerts.

Due to our size limitations, we needed assistance with the lower level alerts so that we could focus with the real, priority alerts. Because of the use cases that they've built up in some of the logging systems that we already had they were able to amplify the type of alerts that we were getting in a way that gave us better and more visibility than we were receiving beforehand.

All of the hardware and software that we were already utilizing was already in place. We were able to offload the management of our Splunk environment. CRITICALSTART began to manage this for us. That alleviated a good portion of one of my analyst's time, to where they didn't have to manage that them self by allowing CRITICALSTART to manage it. We have it 24/7 so if something was to go wrong, they can look into it.

Sometimes the hardest part of showing a ROI with Security in general, is the fact that when you do not have an incident there.  Then comparing that to when a peer has an incident, and how much you are saving because of the tools in place to prevent specific types of attacks. 

My impression of the transparency of the data is that it has good detail. It allows you to see how many events have come in, how many of those events have made it down to their analysts to review, and then how many have been escalated to us. It's a good metric that we can share with my management. They see the value of what the SOC is bringing on top of what my team is already doing.

CRITICALSTART does not take care of the tier one and tier two triage on the Splunk integration that we utilize. If we move to the endpoint integration, then it could by providing the ability to lock down a system and providing additional contexts that Splunk logs are not capable of. But as of right now, no. That's just because of what we decided to go with on the first round to give them a shot by doing the Splunk integration.

They haven't missed a one hour SLA to resolve an escalated alert as of yet. We haven't had to enact on their commitment to pay a penalty. It's supposed to be that they will deduct it from our renewal rate.

This type of SLA commitment was reassuring. But I was already well established with CRITICALSTART long before deciding to go with them on the MDR. My relationship with them was what really drove this inner engagement. I used them for other services such as with Pen Testing, architect,  and  purchasing equipment.

The ability to review and close out tickets or alerts through our mobile phone and being able to interact with engineers on their side via the app are the most valuable features. That's been one of the more beneficial components.

So far, the mobile app has been great. We've been able to reply and interact with them through it. The collaboration is very cohesive.

Nothing will help me solve every alert by any means. We don't do a lot of remediation through CRITICALSTART. We're doing more detection because we do Splunk integration. Whereas, if we were doing the endpoint integration like Microsoft Silence or SentinelOne they would have the ability to lock down a computer-based on that and probably get more insight than what they get right now. The trusted behavior registry does give us the ability based on the alert logging we have with Splunk to dig in a little bit deeper and to even know that something that was an anomaly even occurred. Whereas, before we didn't have that dataset.

In terms of how many escalated alerts we receive in a week or a month, we would always get them during tuning. I would say that we would probably get about a couple of hundred alerts during any normal month, if not a week. It just depends. However, when we moved to CRITICALSTART, we found that we could turn anything on and give them a little bit more information. Of course, until that gets tuned down and we find out what's normal versus not normal in our environment, it is a little chatty. For example, we turned on a certain logging type for our command line and our alerts increased by around double, if not two and a half times but it ended up being a false positive. We just had to go in and ended up tweaking it or filtering it out. It helps decide those alerts but for the most part, it's dropped our alerts down by around 50-75% and we're able to focus on the more important things.

We have decreased a lot of these other alerts that we're able to filter out through CRITICALSTART. With their integration into Splunk, they've been able to add new alerts that we never had set up prior, so it's increased. Whereas, one area might have decreased and another area has increased. Now, we have the visibility of seeing when people don't change, they're having a hard time changing their password or if someone's being added to a local administrator group, things like that. We're getting more visibility than we had before. For those types of things, there's nothing CRITICALSTART can do. So we have those sent right over to us. And we'll work it out on our end and investigate it because they don't know who was added to what. It makes no sense for them to be able to try to work those.

For the most part, using the mobile app to talk to service providers has been pretty responsive, they usually respond within a couple of hours. I would say that before they respond, they typically will do their own homework, which is why it probably takes that long to get their response to investigate. If we escalate it back to them to do filtering, they're pretty quick about getting it.

It definitely alleviates workload because now if we filter out something, for example, if we find that a CID is based on a security group for something that's allowed to be put anywhere in our environment, like an elevated group or a privileged group, then we don't ever need to see that. It doesn't need to come to us so we filter it out. Now, if we've been getting 10 or 15 tickets because of that, we filter it out and we don't see it anymore. Or if there's some change in the way that Microsoft operating system works and it initiates a lot of command-line processes that are alerting because of the way that they're being handled by the operating system, but we know it's not a true positive, but a false positive, we'll filter that out and that'll drop our alerts dramatically.

In terms of the intuitiveness, they are still working out the bugs in the new version because we're testing that out. When I say bugs, I mean that there is still a little bit of slowness. But because they're still working on it, I'm giving a little grace in that aspect. Overall, intuitiveness is great. I've noticed that over the last week or so the response time has actually increased, so that's good. They're still working on it. It's not primetime but the intuitiveness is pretty slick. Responsiveness is a work in progress. 

The updated UI allows us to respond to escalations. It's able to close out tickets or review them a lot quicker because it's all within one interface. If we received the email alert, we already know exactly that this ticket just needs to be closed out because we've already investigated it. Before, we'd have to go click in it, go into that specific alert, and close it out. There were three or four steps. They've removed those two or three extra steps and made it to where you can do all of that from the initial page.

We've been able to integrate everything that we need to integrate. 

They could dig a little bit deeper into the Splunk alerts when they feel like they need to be escalated to us. For example, if a locked account shows up, they could do a little extra digging to verify that the locked account was due to a bad password on the local system. They could just do a little extra digging within the Splunk environment instead of pushing it onto us to go do that extra little digging. We actually created dashboards for our help desk group to be able to hunt down locked down accounts. We've asked CRITICALSTART to start using that as a means of validating the lockdown accounts before they just start escalating them to us.

If we go down the endpoint protection route, then I could probably have other input after I've used that for a while.

I have been using CRITICALSTART for almost a year.

I would say technical support is an A+. They respond quickly and they're quick to help find a resolution, especially in the Splunk environment. 

Sometimes I'll ask them why a certain alert went off and they'll tell me it was a false positive and that they're cleaning it up. They're a little vague in their responses because they have to be generic in how they respond to it. It's not the answer I always want. But I understand why they're giving that answer, as not every environment will be the same nor is every situation.

We did everything in-house.

From the time we entered into an agreement to use the service it took a couple of months until we were able to start to fully utilize it. But I don't know if it's fair to say that's their same practice now. I say that because I feel like their project management for the MDR is night and day different from what it was when we first started, versus what it is now. Their leadership has changed, their structure has changed. So they've got a better handle on their project management onboarding side than it used to be.

We ended up migrating all of our Splunk environment out to their infrastructure for them to take over and manage. That took about a month to be able to get everything onboarded properly. Therefore, for a period of time, we were utilizing two Splunk dashboards. One on-prem and one in the cloud. We had a couple of hiccups along the way, but it wasn't people skills but technical issues. We've been able to get those migrated over and now it works great.

There were three of us involved in the setup, along with assistance of CRITICALSTART.

Splunk has a lot of integration with different toolsets by the way that it ingests logs. We've got several of our toolsets that integrate directly with Splunk, which then create those use cases that they take and ingest into their toolset.

Their new leader over the project management group came in. We were one of the first projects that he took over and started running. Shortly after he completed ours, he became their director over the project management group for the MDR. He's done a really good job. He did a good job for us and he understood our frustrations and made sure to clean it up and listened. 

We have seen ROI with CRITICALSTART. Being able to alleviate a lot of the alerts has allowed my team to have somewhat of a life. It's not the same kind of ROI that we would see for the organization. It's more of an ROI for the livelihood of my team. And that sometimes is more important, especially when we have such a small team.

I think pricing is fair. They're fairly priced. I don't think that they're over. I don't think that they're undercutting other people. I think that they find that they do it at a value that is equal to what they do.

I went off of peer reviews. The other vendors didn't meet my expectations or my criteria but that doesn't mean that they don't meet somebody else's.

There were two or three other ones that I spoke with and talked to, but after speaking with them, or consulting with other peers in my arena, they just weren't there. They didn't have a consistent way of doing things, because they were too willing to bend over backwards for every one of their clients.

It causes their playbooks to be out of whack way too much. What happens is because engineer A is consistently bending over backwards to do whatever it is that you want, but then he gets sick or leaves, and then engineer B comes in and doesn't know the playbooks or know how they handle things. And next thing you know, you're getting frustrated because of that. It would be almost like having to train a whole new teammate.

I love the fact that they were local to the DFW area because I know them and they know me. When I've had to have some heart-to-heart conversations, it's simple enough to have a face-to-face meeting with their leadership, break bread, and have some pretty direct conversation.

And they listen. They express why they handle things a certain way, but they are willing to listen and see how they can integrate, modify, and change, not to just accommodate the customer, but also to make it consistent amongst all of their customers. That's the other thing that I'm very big on a proponent is, if I'm doing something, I don't want to do it just for me. I want to make it better for all the other customers that use that product.

After a year of using the service, our expectations have been met in terms of services delivered on time, on budget, and on spec. I'm ready to take it to the next level. I'm ready to do the endpoint protection integration. Unfortunately, that costs more money so I've got to get that approved.

My advice would be to make sure that you know what it is that you really want done. Understand what your use cases are as an organization before you get a jump in with anybody. Ask very direct and hard questions to those that you're meeting with. Take it beyond the sales engineer or the sales guy. Ask for meetings with the leadership of the MDR Service, they're willing to meet with people, to have those good conversations about what the services are.

When I first went into it, I thought it was machine learning that was handling Splunk integration. I found out after the fact that it wasn't. It was use case build-outs that they built as alerts within Splunk that did correlation. And then based on those correlations, or use cases as they call them, they are ingested into Z-TAP, and Z-TAP then looks at filters. If it doesn't meet a filter, then it gets populated down to an analyst. If the analyst finds that it needs to be further investigated by the client, then they escalate it down to us.

Whereas, with an endpoint integration, that is machine learning. I think that was the misconception in the way that it was described and explained. That was one of the direct conversations that I had with them. Was that going into it, I thought that Splunk was machine learning as well but then I found out after we integrated it and asking some very direct and hard questions to their implementation people, that it wasn't. They explained to me why it can't be or why they're not there yet. Needless to say, that was one thing that I wish was better explained and articulated, and they now know that.

Unfortunately, machine learning is the future for this type of service. The way that technology is progressing and the more and more the bad guys are utilizing machine learning themselves on how to build out malware and attack situations, if you're not using machine learning in certain aspects, you're behind the game or you're doing it the old school way. Which is not saying that the old school way is bad, it's just slower.

I would rate CRITICALSTART an eight out of ten.

We needed a company with expert solutions in the security field. We needed to secure our internal network, external users. CRITICALSTART has resources and know-how in those specific areas. The second part was that we needed assistance with security, hardware support, and implementation of Palo Alto firewalls, and they are the experts in that too.

There are additional features on the Palo Alto firewalls, security on the level of the apps. The users cannot go to certain places. There's a service that gets set up so we don't have to manage it; there is an automatic shield on those firewalls. Software-wise, we use CRITICALSTART to manage the ZTAP (Zero-Trust Analytics Platform). They manage an antivirus solution for us by Cylance and another protection level is Cisco Umbrella. They manage and monitor our systems with their MDR solution.

For example, alerts come in from the Cylance antivirus to their systems and the CRITICALSTART team informs us and helps us combine the white lists, the black lists, what's allowed, which machines are behaving abnormally, and they monitor various aspects.

It is deployed to over 100 people within our company. That is the user base.

In terms of the MDR, if we didn't use CRITICALSTART, we would have to hire a full-time person to sit and do that job. It frees up resources. It's far less expensive for the company to hire CRITICALSTART instead. And CRITICALSTART has a large knowledge base in the field, whereas we would have to learn within our company how things work. With CRITICALSTART, we tap into the knowledge of all the companies that they manage. It's definitely a win for us.

There was the initial adjustment period, as every environment is different. Initially, they came in and looked at our stuff, our alerts. We tweaked things a little bit, but then we could tell that out of thousands, or even hundreds of thousands of alerts, we were only getting, say, 10 tickets per week from CRITICALSTART, if that. The rest of the things they handle automatically, or their system handles them automatically. It really frees up our time quite a bit.

It allows us to free up our resources. We don't have to get into the super-deep details of the alerts if something is happening. They bring a vast knowledge of the threats to the table. We don't have to research them ourselves so it frees up our time.

And they've previously seen the resources we use for the Palo Alto designs, and they know our environment because we have a person that deals with us directly. It's so much easier to work this way, versus if we were to hire somebody from a large consultant like CDW or Softchoice. With a third-party like that there's always a learning curve — you have to invest so many hours first — before you get to the problem. With CRITICALSTART, we can engage them right away with problem solving. There's no onboarding every time. They already know what's going on.

We have a SCADA system which is something that our field team operates 24/7, all year round. It's a pipeline. We have the Cylance umbrella solution on those critical machines and if something gets blocked by an error we get an alert right away on the mobile phone. We respond and CRITICALSTART comes in and makes live changes. That prevents us from having any downtime due to a blocked file on some system. If it's a bad file, it will get blocked, obviously. That's great. But if it's a false positive, we are able to get CRITICALSTART, using the mobile app, to respond right away and prevent downtime of the SCADA system.

There are two parts of CRITICALSTART's services that are most valuable to us

• The MDR solution where they monitor our computers, laptops, and users across the board. 
• Their knowledge of Palo Alto firewalls.

And their mobile app is actually our preferred method of interacting with them. We get notifications and can reply to tickets on-the-go. I don't think there's any other solution that offers such a thing. It's super-useful. Everybody's got a web portal, but this mobile app is quite something. It's pretty cool.

The mobile app is self-explanatory. You have a ticket or you get a notification and you can chat or submit information. You can talk to their team on-the-go. It's very convenient. If you go farther, you can look up tickets and you can look at the assigned statuses. There's more to it; it's a full-blown app. Maybe there are a couple of features that are easier to use in a web browser with a larger window, but I think it's pretty full-featured. You can change tickets, you can assign the queues, you can post a reply. You can look at the details. The whole thing is there. For us, the main thing is that when there is an alert we can act on it right then.

We also talk with CRITICALSTART analysts, two folks in particular. Their response time is very quick. If they cannot talk to us, we get a reply from them anyway. We don't have to wait around. The response time is very good in comparison to larger companies. CRITICALSTART is fairly large, but there are larger companies where you send a ticket, request support, and you're not sure who's going to get the ticket, who's going to respond; you're not sure when that is going to happen. It's always a waiting game. With CRITICALSTART, it doesn't look that way. They give you a personal approach. Their folks are always available. That makes us more likely to do business with them.

When it comes to the transparency of data in the platform, everything is there if we want to look at it. We really don't get too much into it, but if you want to look at it, it's all available. They show the details; they show how they do it. If you want to know if they're lying to you or not, you can look at the details and the facts they base their decisions on when blocking certain things or monitoring certain stuff. It's pretty transparent. It's very trustworthy. It gives us confidence in the decision-making process, because we see how things are done. It gives us peace of mind.

There is room for improvement with the new UI, and that's about it. I would like to see a more intuitive design.

We have been using CRITICALSTART for two years.

We don't have plans to increase usage for now. We're happy with it and we renewed for another two years.

From a project management standpoint their performance has been very satisfactory. We deployed seven sites. Those were new sites due to expansion that we went through and CRITICALSTART was on each one of them. We involved them and we had success every time.

The customer support is great. Our expectations have been met in terms of service being delivered. We have met all deadlines so far.

The main thing would be the roll-out of those sites. We could schedule something at fairly short notice, like only three weeks ahead, and we were able to book them. They were available to fly with us for the site deployment, if needed. They were also able to deliver hardware in that short period of time. Three weeks is super-fast for obtaining hardware and booking a person who is able to do a project.

We used in-house solutions and it was more involved. There was more time spent with longer project timelines. With CRITICALSTART, we were able to get delivery and get things done quickly.

From the time we entered into an agreement to use CRITICALSTART until we were able to start using it, things were wrapped up within a month. There wasn't any type of initial setup required at our end to use the service. It was just me involved in the setup, on our side.

We don't have any data sources that their service wasn't able to integrate with. They provide a full-blown spectrum of anything you want. Whatever you want, they can deliver.

We looked at other solutions that other folks provide and nobody came close. We had previous experience. We had acquired three other companies in a similar business line to ours, and those folks recommended it. So we had a meeting with CRITICALSTART and we discussed a few things, and it seemed like they were the ones to go with.

The main difference was the value you get for what you pay. You can't beat it. As far as the expense goes, it's very competitive pricing and the services you get are almost like you have a person on your team.

The new web portal they implemented is quite robust. It's very next-generation, but it does need small tweaks. You have to get used to it and learn a little bit about it. That's why I prefer the mobile app. The mobile app seems to be more straightforward. The new UI has more advanced features but you would have to click around and learn a little bit more. It's not as intuitive as the mobile app, but the functionality is there.

As for their contractually committing to paying a penalty if they miss a one-hour SLA to resolve an escalated alert, we have never run into that situation. They haven't missed an SLA in two years.

They offer a very personal, connected experience. I don't know of any other company that has that kind of a personal touch to either its services or its MDR solution. That was the decision-maker for us. 

This has been a positive experience and money well spent. If we had to do it again, we would gladly choose the solution that CRITICALSTART provides, versus going with other solutions or using something in-house where we would probably have to spend double what we are spending now.

We needed a SOC operation, and we weren't going to build it in-house, so we were looking for exactly what they offer. They're an MDR service, and we were looking for somebody that would manage the SIEM tool as well as the endpoint management tool and have the ability to take action, when necessary, on endpoints and function as a full, hands-on SOC. That is why we selected them.

The service doesn't require us to make use of any hardware. The software required is Splunk, as a SIEM tool, which provides options as to how it's managed. We opted to have CRITICALSTART fully manage it, so we're hands-off with the SIEM tool, and it's hosted in AWS. Then you have to have an endpoint endpoint detection tool that CRITICALSTART has approved. I don't know what their current selection is, but a year-and-a-half ago it was either Cylance or Carbon Black. We're using Cylance.

Our use of the service covers 100 percent of our endpoints. We're covering 1,100 endpoints.

We didn't have a security team before. If I were to say the service had improved our organization, it might lead you to think we were doing security a certain way before, but we weren't. I came into the company as the first security professional for them.

The service has increased efficiency for me to the point that I can focus on other areas of the business. Again, as a department of one, and not having to attempt a one-person SOC operation, I'm able to focus on the strategic security posture, the architecture, for the company, and focus on where our keys to the kingdom are. I can also pay attention to compliance, which is part of my role. I'm able to do my job because I have this outsourced SOC.

The most valuable part of the service is that they are 100 percent taking care of all first-line alerts. With eyes on glass, fingers on keyboard, they're doing the work. If they have a question, or they haven't seen something in our environment before, then they will escalate it to me. The service takes care of Tier-1 and Tier-2 triage. They actually provide a report that gives details on how much that saves us. I looked at it when we first started, and it was multiple FTEs, on an annual basis, that they're saving us.

I also use their mobile app. It's very easy to use and very convenient to be able to respond to alerts wherever you are. I love the app. You can respond and communicate, per ticket, with their SOC in near real-time. The response is very quick. I can close tickets, I can escalate them. I have very close to all of the capabilities that I have on my desktop. All the things that I need to do in a ticket, I can typically do them from the app. I am a one-man show. I'm the only security analyst for our organization. I couldn't really do my job without the app. I can't sit in front of a computer all the time, so it's critical for us.

I communicate with CRITICALSTART's security analysts. I haven't spoken with them over the phone, except for one time, in a year-and-a-half, but their accessibility is very high. I always receive quick responses to my escalated tickets. When I'm commenting, they're following up, and they're very fast.

I feel I have full transparency to their SOC. Anything I want to go look at, I can do so. I can see all of the comments and discussions that the SOC team has on behalf of us. I have full transparency.

In terms of CRITICALSTART contractually committing to paying a penalty if it misses a one-hour SLA to resolve an escalated alert, I honestly haven't looked at the contract in a year and a half, so I don't remember if it's monetary. I believe that it is. They're very proud of their SLA and not missing it, so I've not ever had an issue or concern or had to think about it. This high commitment to SLAs was our CIO's primary concern when we were looking at CRITICALSTART. After seeing their record, 18 months ago, of not missing a single SLA, it became a moot point. It was a concern at the time but they satisfied that concern.

The updated UI is actually pretty bad. Regarding the intuitiveness, it is fairly easy to use, but the responsiveness, on a scale of one to 10, is a one. It's really poor performance.

I have shared this next point with them already, but I would like to see a monthly report to talk about advancements or new alerts, anything to do with what we call IOCs — indicators of compromise. When there is anything that they have changed on behalf of their customers on the backend, they should say, "Hey, we have made these modifications. We're now looking at these types of alerts." It would give the customer a sense that they're actively looking for new IOCs. So I would like a monthly recap of what they have done, not specifically for me, but what they've done for all of their customers. That would be good.

I have been using CRITICALSTART for a year and a half.

I would rate the customer support, post-deployment, as highly as it can be rated. Their focus on doing the right thing for the customer is how you would hope that every company you deal with would respond to customers. They are 100 percent focused on doing the right thing for the customer, and they back it up. I've seen that multiple times.

In terms of project management, in the lifespan of managed detection and response companies, I'm an old customer now, at 18 months. Back then, the project management was poor and that was part of the reason our roll-out was delayed. CRITICALSTART took all of the necessary steps to revamp that department and correct their mistakes, and that's why we were compensated monetarily, as well. It was poor then, and I haven't had the experience of working with the revamped project management team, because I'm already established.

In terms of delivering services on time, on budget, and on spec, we're a little bit of a unique customer. I know that because we had some early growing pains. They did miss the scoping of our network, which did impact the budget. I brought it to their attention and they stepped up. From a monetary standpoint, they made it right, with no fight. They just recognized it. They have a great ability to put themselves in the customer's shoes and do the right thing on behalf of the customer without any friction.

Prior to CRITICALSTART, we were a customer of Arctic Wolf.

It's really not even fair to compare the two companies, because Arctic Wolf was not a 24/7 SOC operation, even though they sold themselves as that. It was more like a managed SIEM service. They used a proprietary SIEM. I cannot say anything positive about that company. Not a single thing. Right from the time for migration and sending the SIEM tools back to them, it was a very bad experience. They don't do what CRITICALSTART does. Even though they try to market themselves as an MDR, they're really not an MDR. They don't manage the endpoint tool, so it was really apples and oranges.

There wasn't really an initial setup required at our end to use this service. The implementation of the endpoint tool, in this case Cylance, was a requirement for us. That involved some GPOs and the Splunk forwarders that we implemented in our environment. But as far as man-hours on our side to do the setup, it was very low.

It was straightforward. Pushing out software is something we do. Creating GPOs to make sure that the correct data from servers was being pushed and directed to the Splunk forwarders was all typical, sysadmin-type work. Nothing was complicated.

There were no data sources that this service wasn't able to integrate with.

From the time we entered into an agreement to use them, it was about four to five months until we started using it, but a lot of that was dependent on our ability to get the product rolled out, and our activity for base-lining the system, or our environment. Some of that time span was us, and some of it was them, but they made monetary compensations for the delay that we had. While it didn't go as fast as we wanted, the end result was positive.

We are absolutely seeing return on our investment from CRITICALSTART's services. They're doing the job of a 24/7 SOC at a fraction of the price that it would cost me to run it myself.

You get what you pay for.

Compared to the competitors that we looked at, CRITICALSTART had a longer history, even though they were a young company. I liked that they were not using proprietary tools in the environment. That allowed us the freedom to move, if we wanted to, to another provider. They were just ahead of everybody else in terms of maturity.

In terms of advice, I don't feel that implementing this service is any different than implementing any other system into your environment. A lot relies on your project management skills.

I would attempt to test your MDR choices against a framework. The framework that comes to mind is the MITRE ATT&CK framework, which everybody is familiar with. Have realistic expectations about what vulnerabilities your MDR partner is really going to mitigate. That's the lesson I have learned.

In terms of CRITICALSTART's Trusted Behavior Registry and the way it resolves things that are known as trusted, so that the focus is on resolving unknown alerts, I'm obviously not looking at all of the alerts that they work on. But what they escalate to me, only the alerts that I'm seeing —which is a small percentage — if I were to rate them on a scale of one to 10, I'd rate this aspect at eight. There are a few things that slip through, things that they'll escalate that I know should not have been escalated, but it's a very small percentage of what they actually escalate. It's a very small percentage where I'll have to just say, "Hey, did you mean to do this one, because we've been through this before," or a virus total shows that it's 100 percent clean, so why did it get escalated? It's not common but it does happen.

The service missed a pen test, but I still have a high level of confidence with the data and the actions they take. We had hired a red team, so the situation was a red team test. Red teams are generally 100 percent successful, or very close to it. With them, you always expect to uncover the unknown. But I do have confidence in the tool and the data that they are looking at.

The number of escalated alerts we receive, compared to the number the service's Trusted Behavior Registry resolves, is probably less than 5 percent of the total.

The challenges we were looking to address were mainly around making sure that my team wasn't overloaded with alerts and that we could tune out things we don't care about or that aren't important to us at that particular time. That was really what I was trying to accomplish, since I knew I wasn't going to be able to build out a team large enough to be 24 by seven.

Before we really had anything in place, or when we didn't use them in a managed way, we felt overloaded with issues like: "How do we deal with all these alerts?" "Is this a problem? Is this not a problem?" "Are there other customers that are also experiencing this?" It was pretty easy for us to justify paying a little bit to get some help on those things and get the benefit of their experience with the other customers they have on their platform.

In terms of the transparency of data on the platform, what comes to mind is that I've asked them a few times, "Hey, we've got this weird alert that you've escalated to us and we don't really know what to think about it. Have you had any other customers that have experienced it?" Obviously they're not quick to say, "Oh, well, Company XYZ had the same experience," and for good reason. But when asked, they're usually pretty good about saying, "Yeah, we've had some other customers that found this, or we've worked with them to determine it was this or that." Some of that you get upfront, but there are times when you do have to prod to get more information about something. Once we learn more about it, it affects our security operations because we're pretty small. So if I know that a large organization has spent time on this and had other analysts looking at it, analysts who have determined it's this and that, I'm going to lean toward what they found. I often just don't have the resources to do that myself, or it may be because I have respect for the security organization of that company. It's definitely valuable.

Using CRITICALSTART has increased our analysts' efficiency to the point where they can focus on other areas of businesses. That's definitely been a benefit of the whole thing. Instead of worrying about every little alert coming in, we really only pay attention to the ones that we need to pay attention to, the ones that are escalated to us. Otherwise, we would just be thumbing through thousands of things that likely don't really matter that much.

We have different groups throughout our company that use the equipment that we give them in different ways. So we've reached out to CRITICALSTART to build out groups and we can update those groups ourselves with different peoples' usernames. That way we can say, "All right, Nmap for the engineering group is always allowed. Don't ever alert us about that," or perhaps we make it a low alert as opposed to high. But if it's any other group, or if a user falls outside of those groups, we want to know. And that's been really useful for us in bringing down the number of escalations to us, things that would pop up as "high" at 8:00 at night, because some guy's running Nmap or something similar.

CRITICALSTART also takes care of Tier-1 and Tier-2 triage. In terms of time saved, I've always assumed that if we did this ourselves, I'd have to have at least a minimum 24/7 staff, or at least a few shifts throughout the day to cover the amount of things that would have to be researched and looked at.

Outside of using the platform to manage alerts, the feature of the service that we get the most value from is being able to reach out to them and say, "Hey, we might go buy a SIEM," for example. They give us their overview of what's out there, what they've dealt with, what they integrate with, and what that looks like. That's been pretty powerful over the years for us.

And when it comes to the alerts, they get the number of them down and only alert us about what we really need to know about. We get about a dozen or so things escalated in a day. Most of those are low alerts.

We chat with CRITICALSTART's analysts back and forth with comments or when we escalate things back to them. Occasionally we'll open a support request for a feature or we'll have a question about something and we may converse with them over that. Their availability has always been pretty good, especially when it comes to escalating to the SOC directly. We get responses pretty quickly.

I've used the updated user interface about a half-a-dozen times. I felt like it was going to take a little bit of getting used to it, but it did seem like it was pretty quick. It had more of the data right in front of me that I usually want, as opposed to clicking around to go find it. So far I have nothing but positive things to say about it.

We've had a little bit of frustration with some of the alerts that we receive because they're not as high-priority for our type of organization, as we are very engineering-heavy. But I can understand from their perspective, if a bank were a customer, or some other organization that doesn't have a lot of heavy engineering folks who are in a command-line and running all kinds of tools, the service would be much more valuable to them. But that's one of the main frustrations we've had: Trying to find ways to tune that out so that we can say, "Look, for this group it's normal for them to run a ping or Nmap or the like, but if accounting does it that's a problem.

Also, it has frustrated us that they don't have a native Slack integration, because most things do now. That's something we've asked for, for years, and it just doesn't really seem like it's a priority. The workaround is that we just have it sent to an email and you can email into Slack. Of course, email through Slack is not very good, but that's our workaround. We set that up ourselves.

Where CRITICALSTART could potentially grow is on its internal compliance, and maybe how they disclose how they secure data. All of that could be a little stronger. I pushed them on that early on, and they did provide some information, but like I'm doing with us — we're ramping up our compliance efforts too — that's where I'm likely going to have to push them in the future to make sure that they're at least meeting the minimums that we have, because they are seeing data from our employees.

I've been using CRITICALSTART for four years now.

It's definitely not utilized as much as I would like because of other priorities that have come up. My team is pretty small, so we can only do so much. But we are ramping it up because some of those other priorities are no longer as much of a priority. We should have some more time to do it.

I want my team to get in there and just clean up a lot of the low alerts that are sitting out there, alerts that we looked at and just didn't care about or that weren't important to us. We just need to go in, close those out, and get them to update filters about that stuff so we don't get alerted in the future. There's a fair bit of that.

I lean towards evaluating their support as good. Occasionally, we have spoken with them about something we have had open for a while and have had to look for an update. But they're generally quick to respond, initially.

From a project management standpoint, I have always felt that CRITIALSTART was pretty good. When we first brought them on, and when we switched to different products, and even when we tried out some of their other products, they were pretty good on that score. We had weekly calls and it seemed like we were getting moving on things. I really don't have any issues or complaints there.

Their overall customer support is pretty good. I can only compare it to our company and the support we provide, which I feel works pretty decently. They're on par with our organization.

Prior to using CRITICALSTART we were just managing things ourselves completely, without any help. But we brought them on pretty early after the company's creation, so it wasn't too painful from that perspective.

From the time that we entered into an agreement to use CRITICALSTART until we were able to start actually using it — I don't remember it taking too terribly long. We used them for a different endpoint service for a little while. When we switched to the new one, I do remember thinking that it took a little bit longer than I would have liked, but when they came back and technically explained it, it made sense to me.

Initially there were some calls where they were just getting an understanding of the environment and the types of users we have. We voluntarily provided them usernames of folks who were more high-priority or the groups that we needed to really focus on.

But the setup was definitely straightforward. It was a couple months before we were really comfortable with the setup, from our perspective, and felt that it was complete.

There were four of us, from our organization, involved. My architect was leading the effort and then I and one or two other analysts were the ones who were looking at the alerts and providing feedback to them so they would know we didn't care about this or that issue and that they should filter it.

When I start thinking about if I were to try to light up a SOC, which I've done before and I have no interest in doing, it could be a million dollars a year or more to do that. For what I am paying them for the managed fees, it's a steal. What I can get from them costs less than one body that I would hire. I've always felt like that it's a really good deal.

I've told CRITICALSTART that I think the managed service they provide is cheaper than it should be. It's a really good deal. 

As far as using them to purchase software and other things that they don't necessarily manage for us, they seem to be pretty on-point with pricing. We've looked at them and put them up against Myriad or some others to see if we are getting good value, and they've always been pretty aggressive. In some cases, I feel they have been able to get us a bit more than another VAR would have been able to get us, because of the relationships they have. I feel pretty good about the value there.

Our expectations have been met when it comes to their services being delivered on time, on budget, and on spec.

We didn't evaluate other options. I have worked with the architect that I have for a long time now, and I know that he had evaluated options when he was at IBM. I didn't feel the need to, since he had just done it before he came on board with us.

The biggest lesson I've learned from using CRITICALSTART is that you don't necessarily need an internal SOC to make your customers happy. We get asked all the time on questionnaires, "Do you have a SOC?" We're able to say, "No, we use an external SOC to manage alerts for us." I've really only been pushed on that a couple of times. And at other times I've had companies that are larger than you would think come back and say, "Hey, we do the same thing." They may have an internal SOC too, but they still leverage a similar company to triage stuff before it even gets to their SOC.

I use CRITICALSTART's mobile app occasionally, although not as much as I did when I didn't have a dedicated person really looking through the alerts. It's mostly good. I don't have any major complaints about it. There are a few things here or there that need to be polished, but I think it's come a long way. The rest of the team is like me. They use it occasionally to pull up an incident that may be a higher risk, when they're running around doing things. But for the most part, we use the web browser.

On a daily basis there is only one person using CRITICALSTART. He's a security analyst for me. I'll occasionally jump in and my architect will as well, to help on the more advanced things or to adjust the filters and to do things that the analyst doesn't really do.

I would rate CRITICALSTART at eight out of 10. There's room for them to improve, but overall it's a good value and we're happy with them.

Our primary use case is to gain the ability to monitor our systems more thoroughly. We are looking for it to address the overload of information from security monitoring systems.

Everything is cloud-based and other than the security agents that are installed on those systems, we also use Cylance Protect, and Carbon Black Response.

The quick interaction between the agents is the most valuable feature. If we have questions, they're quick to answer. If we make a change to our system, they quickly make the changes that are necessary to filter the logs correctly.

They do trusted behavior registry. They filter out the unnecessary stuff and present us with the things that are interesting and let us determine the validity of that type of action in our environment.

We get probably 10 or 12 escalated alerts a week, and there are hundreds or thousands of transactions that would need to be filtered otherwise.

The mobile app is a nice way to get quick access to something when I don't have access to the full system. It's a good way of accessing all the data that I would need when I'm remote. The mobile app gives me more comfort in that I will be alerted if there is something going on, even when I'm remote.

CRITICALSTART makes us much more comfortable with knowing someone else is watching our data and our systems and knowing that professional security people are taking a look at any issues that do arise.

The new UI seems a little slower but some of the functionality is a little bit quicker to get to things in terms of navigation. It has made it easier to respond to escalations. The alerts are displayed in a way that makes it simpler to respond. The response dialogue is right on the screen.

In terms of transparency, it seems like all the data is available to us. It affects our security by allowing us to see what they are doing in terms of filtering and making sure that we agree with all the filters that they're adding.

CRITICALSTART has increased our analyst's efficiency to the point that they can focus on other areas of business. We implemented some of these tools at the same time we started with CRITICALSTART. Some of that wasn't being done before, but now it is being done and we still have the time to do other things.

It also takes care of the tier one and tier two triage. It saves my team around 10 hours a week. 

I think that the provider contractually committed to paying a penalty if it misses a one hour SLA to resolve an escalated alert. But it wasn't a huge deal for us. It wasn't a critical thing that we looked at. So far, they haven't missed such SLAs, as far as I know. It has yet to miss an attack. 

We chose not to integrate data sources due to the cost of our firewall logs. They would have been able to ingest them through a SIEM had we wanted to.

The UI has become slower but it's not something I would call them out on. 

I have been using CRITICALSTART since January of 2020.

We communicate with support mostly via the tools, via email and their security application. There is somebody available 24/7. They add a lot of value in terms of being there 24/7 and having access to the data and access to their knowledge base of issues.

Their support is fast, thorough, and easy to use.

We just had to get the security agents installed on the systems that we wanted to use it on.

The process was quite simple and straightforward. We were able to push out the agents with group policy and that made it simple to get everything installed.

Two of us were involved in the setup. I am the Director of IT and my colleague is a network administrator.

Three of us use this solution. The other one would be the chief product officer.

In terms of the size of our environment, it's on over 200 endpoints. We are adding a few machines, but it's close to a 100% adoption rate. 

The implementation was very straightforward. We didn't have any real problems with the product management side.

We have seen ROI but I can't explicitly say what. We've been able to easily manage the security information and alerts coming out of the products without having to deal with them on a day to day basis.

The price was less than I would have expected.

We did evaluate another solution but we like CRITICALSTART's pricing and we liked the people that we were working with.

Our expectations have been met in terms of services delivered on time, on budget, and on spec. The implementation went as expected. The pricing hasn't been an issue. Everything went as was decided at the beginning. Everything has gone through as I would expect.

I would rate CRITICALSTART a ten out of ten. 

We were looking for a managed service provider who could handle our endpoint alerts as well as our SIEM alerts. We were looking to address alert reduction, better correlation, and reduction in head count that would ultimately lead to a more secure environment.

We brought our own endpoint solution into the equation. We added a full functionality SIEM solution. There wasn't a whole lot of infrastructure. 

The transparency is extremely effective. The ability to maneuver through the GUI (the front-end) allows the team to be more effective and perform their job efficiently and effectively. They can bounce around, get in there, and know what they're looking at, which gives them the ability to really dive into the alerts. It's a user-friendly front-end.

From where we were prior to going into them, the service has increased our analysts’ efficiency to the point that they can focus on other areas of the business. It gives me the ability to allow analysts to do Level 3 and 4 work and stay out of the weeds of the alerts, where you tend to get alert fatigue. The service takes care of much of the Tier 1 and Tier 2 triage. It is more effective than what we had been used to, because it allows the filtering of Level 1 and Level 2 type alerts to be taken care of. This leaves less for us to handle, which is a good thing.

We remained within budget. They continue to work with us to add additional logging sources to effectively meet our budget requirements and are in line with our cost cutting efforts.

We benefit from alert reduction and the ability to cross-correlate multiple logs to achieve a more secure environment. CRITICALSTART consolidates alerts, creates reporting, and gives us a more holistic view of our security landscape.

They start with a Zero-Trust model and build from Zero up to Trusted. We found this to be extremely effective in filtering out alerts. So, we started from Zero-Trust, then we built the trust from there. This has became extremely effective for us in our environment. 

We have over 99 percent filter rate for the service’s Trusted Behavior Registry.

It is extremely effective for our team's utilization of the service. It is easy to maneuver and understand. If it ever requires any additional information or a deep dive, we reach out to CRITICALSTART to help understand an alert, why we're getting an alert that we think we shouldn't be getting, or fine tuning an alert.

It has enabled our SecOps and internal SOC managers to take action faster and respond to escalations more easily. Because the front-end is easily maneuverable, we have the ability to work through it, get into it, and understand it. This allows us to pivot back and forth between logs, log sources, and understand the alerting. It's not a convoluted front-end.

Our analysts do like getting into the console more than they like getting into the mobile app.

We have questioned them on the level of an alert and why alerts have come in lower than we would had anticipated them, e.g., it was maybe a medium instead of a high or medium instead of a critical.

We have a lot of homegrown applications, and we don't push a lot of those data sources to them. We are kind of a unique outfit in that way. So, there are some data sources that the service wasn’t able to integrate with. We're working on having the service be able to ingest them through a SIEM and provide us access right now. They will be storing some of those logs for us.

From a project management standpoint, better communication was needed with the customer during the setup/project phase. I have expressed that, and they have understood this. They have tried to make corrective actions.

18 months.

We have already adopted it 100 percent in our company.

I occasionally talk with the service provider’s analyst. Mostly, my manager is in touch with him two to three times a week. They are available for us at a moment's notice. We have desk phones, email, and cellphones, and you can get a hold of any of them at any time of the day. From that perspective, it is an excellent, trusted relationship. It allows us to effectively troubleshoot any issues that we may have. They're very responsive.

Customer service and their response are phenomenal. I would give their customer support a nine point five (out of 10). Our easy access to their SOC analyst, sales team, and leadership team instills confidence in me that they are there for us 24/7.

We weren't able to have a comprehensive, overarching view of our environment because we couldn't get all our log sources into the previously managed service SIEM solution. It was one of those situations where we had to pivot.

During the six-month integration and rollout, there were some bumpy roads along the way. There were communication breakdowns between the project manager, CRITICALSTART leadership, and us (as the customer). I expressed my displeasure during the integration in their inability to effectively communicate when there were holdups or issues. They were going through some growing pains at that time, but they have been right there for us ever since.

We were able to use the service early on, but never fully used it until it was completely integrated after six months. It took about us six months to onboard it and get it integrated into our environment, then get the log sources to it.

It took about six months to go from Zero-Trust to Trusted Behavior. While it is an ongoing process, the install and tuning was about a six-month process.

For the most part, security operations, security engineering, and the infrastructure folks were involved in the initial setup. It could have touched two dozen people over six months. These are the same teams who are currently using the service.

We have seen ROI. There has definitely been time savings. From a logging solution, we are effectively getting all our logs into one solution, which gives us a better holistic view.

Their pricing was very competitive with other vendors. Their ability to be creative struck me as being very customer service friendly. Their creativity in pricing and working with the customer to achieve their financial restraint or goals was very creative.

There are contractual penalties if their SLAs are not met. This commitment was very important in our decision to go with this service, because not having downtime is extremely important to us. The providers has not missed an SLA in the 18 months that I have worked with them.

We looked at two other big companies. 

My main thought process was I wanted to go with a smaller, more boutique firm where I felt I would get personal, undivided attention. It was also important for me for them to be local.

Do your homework. Compare the big boys, the larger managed service solutions, with some of the more boutique ones, like CRITICALSTART, and ask yourself: What is it that you want? Do you want to be a small fish in a big pond or a big fish in a small pond?

You always need more logging space than you actually think you need.

They monitor our endpoints.

I would definitely give them a nine (out of 10). They are extremely effective in combating alert fatigue. They're creative in the way they do business. They are also very approachable and very customer service-oriented.