Splunk On-Call Reviews

The on-call scheduling and the alerting. It completely changed our organization as far as recognizing system-generated alerts and responding to them.

We have a hundred licenses. I have 93 users in the system right now. All of our teams that have on-call resources are using VictorOps with rotations, schedules and escalations set up. This way, our monitoring systems can use route keys to route alerts to the proper groups for escalations alerting. Typically most alerts go to my Tier Two SysOps team for initial triage, and then if they need to escalate to a Tier Three enterprise resource provider, the database, Windows OS, Linux OS, or network people can then reroute the VictorOps ticket to them. The value of that is that obviously once that reroute happens the alert starts paging the on-call guys within the appropriate group. This has helped us to drastically reduce our acknowledgement time.

My VP of Operations is ecstatic about the VictorOps product. He has visibility into what's going on, and he can tell that folks are responding to alerts. It drives a level of responsibility. We used to have a problem of folks who would be an hour from their computer when they were on-call. We don't have that problem anymore, because our escalation schedules work, you have the on-call user, the next user in the rotation, their manager, and if they don't all acknowledge, it goes to the VP of operations, and you really don't want him to get your page.

It's driven a level of accountability. When folks are on call, they're responsive, they know that the mean time to acknowledgement timeframe is being looked at and monitored. That's a hugely important aspect of the product. It drives accountability and visibility of mean time to acknowledgement, which is the first piece. The sooner you acknowledgement you have a problem, the sooner you get to the mean time to repair part.

In reporting, being able to group by alerts and group by products would be good. They should put this in the titles of the alerts that come in, and make the report a little longer or a little more granular. With some of the reporting areas, we'd like to see some improvement with incident trends. Incident frequency kind of gets in that direction, but with incident frequency, you can only can move it around and it needs work. Also, being able to retain data for longer.

The other area is in the Timeline, being able to back up through the Timeline a little further and easier would be helpful. Of course, you can always do a post-mortem report to get there.

Generally, me being operations manager, I'm always interested in getting data out of the system as far as reporting goes, because I have VPs and CIOs that are asking for data, so I might be a little biased towards reporting improvements.

The other input I would have is in the users' area, when you're in Settings. The ability to search for a user by name would be handy, because you get a list of folks there, and sometimes it's hard to sort through that, so if you had a search box for users, that would be awful handy, especially when you get out to a hundred users, trying to scroll through a list of a hundred users, trying to find the one you're looking for. If it was a text sensitive text box for searches, type in "AR" and it starts filtering it out for you, that would help.

We went into production on the first of October, and we had been working with VictorOps probably 45 days before that. That would mean we've used it in total since roughly the middle of August of 2015, so about ten months.

It's absolutely been a stable product.

We have scaled from our initial deployment of 40 users and 22 applications being monitored by two different monitoring products to having SolarWinds, AppDynamics, Nagios, select servers, Splunk, along with Oracle OAM and IBM's Sterling Gateway, and a couple other systems that are generating alerts into VictorOps. We've scaled the amount of systems that are pushing alerts into VictorOps and have scaled all the way out to almost a hundred users.

The guys at VictorOps have been outstanding as far as support goes. They've bent over backwards to work with us and help us along, when in fact you can just go to training. I had 40 users here a month ago or so, so the support is outstanding, and the reality of it is, I looked at two other competitive products to VictorOps before I chose to go with them, and a colleague of mine who is kind of doing the same job but at a very large security company pointed me in the direction of VictorOps. I got onto WebEx and did an online demo. The price is attractive, it does what we need it to do, and it's been a great choice.

We were previously using one of the big house APM suites. We were using HP OpenView, so Operations Manager would have been the single pane of glass, so to speak, and we chose to move away from the HP product as far as our monitoring goes and dealing with system-generated alerts, simply because it took too much time and effort to manage the APM platform. It wasn't stable, it was complicated, it wasn't user-friendly, so we restructured our whole APM program platform, and VictorOps became one of the pieces of that.

I would advise you that it was a good choice for us to use VictorOps over a couple of the competitive products that we looked at. The price is very attractive, and the VictorOps folks have been outstanding as far as support goes. You don't feel like a number like with some of the larger companies. The support folks know me by name and the spirit and the culture of the VictorOps company is outstanding. Their customer service is good and the product speaks for itself. It does the job that it's designed to do. The support of the product by the VictorOps folks is outstanding. I would definitely recommend it.

• On-call tool
• Transmogrifier
• Third Party integration

It's provided real time incident management and alert paging via email, SMS, or call. It has also enabled duty rotation and operations management.

More support is needed for customization in the Transmogrifier for fetching dynamic content from URL.

I've used this solution for one year.

There were no issues with the deployment.

We had no issues with the performance.

It's been able to scale for our needs.

It's excellent.

We used SCOM before.

I implemented it myself.

We evaluated PagerDuty, but VictOps has a better overview of incidents as it has an incident timeline and Transmogifier tool.

Its the best solution of its type out there, you should go for it.