We use Splunk Enterprise Security to ensure the security of our endpoints, including corporate workstations, tracking proxy logs, and all of the other benefits that Splunk Enterprise Security brings, including observability and visibility into the environment.
We run Splunk Enterprise Security on a single search head, and it talks to about nine separate clusters. It's a hybrid environment of on-prem and AWS. Ideally, we will migrate that to a search head cluster for Enterprise Security for high availability. Then, in the upgrade process, we generally have about two hours of downtime when we upgrade Enterprise Security. Ideally, moving to the cluster environment will allow us to mitigate that entirely. So, we did some assessments earlier in the year. We've gotta do some finalized testing, but we're hoping that will eliminate almost the entire two hours of downtime for our customers when upgrading. Then, it's two hours from start to finish to get the search head back up, and that does not include backfill time or anything like that. It could be a good full workday. So getting that workday back is going to be very important for us, and that's where I think we're gonna end up evolving for the Enterprise Security environment.
One benefit we have seen using Splunk Enterprise Security is keeping it all integrated, so no jumping between tools during investigations is the biggest benefit from the analyst's perspective. When we're setting up an investigation, it allows them to use one tool versus having to compartmentalize all the tools together, link it together, document it, and ultimately end up in one spot. Using Enterprise Security as it allows for integrated tracking for the investigations.
It's very important that Splunk Enterprise Security provides end-to-end visibility into our environment because not seeing something is a potential risk to the business. Having that visibility also assures the business, all the way up to the C Suite level, that there is coverage. And if not, we at least have that identified as an uncovered portion.
As long as we can point the data into Splunk Enterprise Security, it is easy to identify security events across cloud, on-premise, and hybrid environments. Getting it into Splunk is typically the challenge because it needs to be in a usable format. So once I've got it properly shaped and tagged, the rest trickles down. Generally, there are a lot of good TAs for getting data into Splunk around the cloud providers. So we don't have to customize it as much. It's just about getting it implemented, going through the checklist, and doing our due diligence to make sure we have the coverage we need. We will see events as long as they're flowing into Splunk. Once it gets into the data models in Enterprise Security, it will show up.
As far as ingesting data, Splunk Enterprise Security specifically hasn't helped. We shape and normalize our data to meet Enterprise Security's needs. So, we did that as a preemptive during our initial assessment. What does it come in as? What do we want it to look like? How can Enterprise Security more optimally use it? Will it hit the data models? Will it show up? Things like that. So, a lot of that is already there before Enterprise Security, but then using the data is where Enterprise Security shines. It makes the data more usable across all data sources. We don't have to know what to look for in each data type. We could go to the data model and view it.
We've increased our alert volume a little bit, not in a bad way, but getting new detections. The risk-based alerting has decreased. So what is happening elsewhere in the environment correlates with that event, and those risks are bubbling up to the top, whereas somebody getting locked out isn't as important as an account takeover. It's hard to portray that image with one event, but a series of events on the timeline makes it a little easier.
Splunk Enterprise Security lets us know who owns what hardware, who should access it, and who shouldn't, more specifically, during an investigation or escalation path. So we know there's a problem. Who do we talk to next to start that process and up the chain? We have a lot of that in there as well, which helps.
Splunk Enterprise Security has generally helped reduce our meantime to resolve. How much is hard to say because it depends on the investigation's scope and scale. It does help the analysts get a clearer picture of what's happening everywhere in the environment.
Enterprise Security will automatically correlate those events for us. When an analyst gets assigned to that investigation, it becomes looking at the picture and putting the puzzle together versus having to go through a threat hunt or find those indicators and then identify the account lockouts and takeovers. It's already in one pane of glass, and then that gets us to the meantime to resolution quicker.
It has decreased our mean time to detection, especially for the high critical alerts. When we leverage that risk-based alerting, we can say, alright, multiple events have now happened to propagate this into a larger event instead of trying to correlate that as an individual or a team of analysts. Ad hoc is going to always be slower than automatic. Doing it in the back end means my analysts get there and get the job done quicker.
Splunk Enterprise Security has helped with our organization's resilience. We generally use observability metrics to determine the state of the hardware and the status of the environment at the time, so that has been a good point. It's definitely made us more resilient to figure out what happened post-incident and on what time scale and then go back and try to either remediate or mitigate that wherever possible. The historical context is just as valuable as their live real-time learning context.
The most valuable feature of Splunk Enterprise Security is the threat intelligence integration because essentially having to go out and correlate all the data on our own becomes convoluted. We don't have the resources, so having that included in the product makes it easier for us.
For us, the area that Splunk Enterprise Security can improve is performance optimization. Enterprise Security is so critical that right now, we're working on getting it to a clustered state to have high availability. The challenge there is hardware procurement and utilization. It's very resource-intensive. A type of performance optimization would generally be a huge improvement.
I have been using Splunk Enterprise Security for six years.
Splunk Enterprise Security seems stable to me. I haven't seen many issues, so I'm looking to try and test the latest version.
Scalability is a mixed bag. So, when we first started Enterprise Security, they told us not to cluster it. Now they're recommending we cluster it. We haven't gone down that road yet. I am looking forward to it. But if they say it can scale, they have customers that have done it. We gotta go through the growing pains of implementing it, rolling it out, and making sure it's ready to go. I think it's possible, but I have no formal experience yet. I am looking forward to it.
We started in Splunk, used it historically, and saw the product's value. It becomes the other data that would not be allowed for business reasons. How can we leverage that to provide value for the business? I know a lot about searchability this year, such as trace logs and metrics. These are generally good, but some trace stacks can be a lot of ingestion against our license. If we could put that in somewhere, that would not be as cost-effective, ideally. The trade-off is performance. Splunk is very performant. It does its job well. It's just a little pricey for the non-business critical logs.
The deployment is generally good. We must stand up the search heads, get them ready, tie them into the index clusters, and then deploy. Generally, we don't expose anything to the customers until it's production-ready. So deploying it was just getting it out there and built, doing some finalized testing to make sure it's ready to be used by the end customer.
We implemented Splunk Enterprise Security ourselves. Through Splunk, we've engaged some professional services to ensure that our plan of attack is moving in the right direction. Professional services have also provided a lot of guidance.
We have seen a return on investment with Splunk Enterprise Security. Getting that holistic view. Splunk gives us a better picture of what's going on in our environment. Without it, we would have to go hunt for it. It's like Google searching for logs. It's easy, and everybody uses Google. So it's time-tested in the market. It's just about how much data we can get in, how we're storing it, retention, pulling it back, and what goes with that associated.
While Splunk offers generous developer licenses and obtaining annual licenses is straightforward, the cost is a major consideration. As open-source competitors become more sophisticated, Splunk will need to address this pricing issue in the future.
We have not used other SIEM tools in the past, but we are evaluating other tools. We don't want to migrate away from Splunk. We want to replicate it at a larger scale for non-security-based data, such as application and developer data. Anything they want to throw in and search is fine. But at Splunk's current cost, it is generally very expensive to do non-business-critical logs in that environment.
I would rate Splunk Enterprise Security eight out of ten. Things that could be better would be further integrations into other security tools. I know a series of threat intelligence feeds can be integrated, and I'm sure they are slated. It's just a matter of getting the resources to integrate them. Splunk Enterprise Security is a solid product. I run it in my home lab as well. It's generally one of the better Splunk apps.