Datadog vs ELK: which one is good in terms of performance, cost and efficiency?
Dear community members,
I've been exploring Datadog vs ELK and I need your opinion about both of them in terms of performance, cost, and efficiency? Which one would you recommend?
With Datadog, we have near-live visibility across our entire platform. We have seen APM metrics impacted several times lately using the dashboards we have created with Datadog; they are very good core indicators of the health of our system. You can build very useful dashboards right out of the box using this solution. Our developers are able to see how code is running in production, and having custom built-in metrics gives us a wealth of knowledge and historical data that helps keep our system running smoothly.
The pricing with Datadog can be very high. We feel there should be a more tiered approach to give users the opportunity to buy a more tailored product specific to their organizational needs. Datadog could also be more user-friendly. We would like to see deeper application-level insight and better incident management.
Using ELK, you can gather authentication information from service providers and determine which identity provider is not performing properly. ELK is very flexible; you can have a number of scenarios and get logs from all of them. ELK is a very cost-effective solution.
But it can be a complex solution to use. Better integration with 3rd party APMs would be very helpful. Currently, upgrades with ELK are released as stacks. Plugins or extensions would save removal and reinstallation and make the process move seamlessly.
Conclusion
We researched both of these solutions and concluded Datadog gave us the best visibility, better integration, and helpful, timely support. The logs and error reporting are extremely useful to conduct analysis and root cause analysis. The setup, ease of use, and flexibility with dashboard creation and reporting are just some of the things that our team liked about Datadog.
Datadog:Unify logs, metrics, and traces from across your distributed infrastructure. Datadog is the leading service for cloud-scale monitoring. It is used by IT, operations, and development teams who build and operate applications that run on dynamic or hybrid cloud infrastructure. Start monitoring in minutes with Datadog!
Datadog features offered are:
200+ turn-key integrations for data aggregation Clean graphs of StatsD and other integrations
Elasticsearch:Open Source, Distributed, RESTful Search Engine. Elasticsearch is a distributed, RESTful search and analytics engine capable of storing data and searching it in near real time. Elasticsearch, Kibana, Beats and Logstash are the Elastic Stack (sometimes called the ELK Stack).
Elasticsearch provides the following key features:
Distributed and Highly Available Search Engine. Multi Tenant with Multi Types. Various set of APIs including RESTful
IT Technical Architect at a insurance company with 5,001-10,000 employees
Real User
2021-10-21T12:30:56Z
Oct 21, 2021
Where do you want to spend your money, on people or licenses?
ELK requires a long-term investment in engineering resources to manage the system and to provide the capability.
Datadog provides capabilities for you so you only need some administrators. What are the capabilities? Some critical ones include availability, scalability, consuming log files, platform upgrades, ...
If you are consuming smaller data sets (100's of GB) with shorter retention, the size and scaling are much easier making ELK easier.
Do you have admins or engineers? If your team doesn't have dedicated time & skills to spend developing solutions like elastic-alert you should look for a vendor to provide capabilities.
I expect some capabilities in Datadog you will not be able to replicate in ELK.... so that answer makes this obvious.
We are going to evaluate the same for our org. We do about 10 TB a day consumption in ELK and are looking to see if we can shift $$$ from engineers and infra to SaaS.
I have used both ELK and Datadog, and there are lots of variables to consider here. The three important points that I looked at are:
- Cost. In addition to service costs, you have to consider egress and ingress costs as well.
- Real-time observability that you need during development vs long-term Observability. Keep in mind, when you export data over the internet, it comes with the same reliability issues as any other service on the internet. Regardless of how Datadog classifies its service as real-time, it is not real-time, IMO. It very much depends on your definition of real-time.
- Deployment and maintenance complexity. When your ELK cluster grows it has some pain points you need to be aware of.
My general approach is to deploy ELK for development, tune the data, and then pivot toward commercial solutions if I need to. This gives you insight into your data and what you should be preserving and that way you are not paying high costs, when or if you do decide to take advantage of a commercial solution.
It depends on your requirement. If you are looking for a SIEM/log management solution ELK would be a better option.
But if you are looking for more of a monitoring solution Datadog would be better. Also, Datadog provides out-of-the-box integrations with a lot of cloud applications. ELK could be cost-effective but a bit challenging to configure & finetune.
Cyber Security Consultant at a tech services company with 11-50 employees
Real User
2021-06-21T19:57:40Z
Jun 21, 2021
Dear,
Unfortunately, I can't say much about Datadog but I have used ELK for a short period.
And I can tell you not everything works the way it should. For example, I noticed heavy CPU usage for a Windows client on MS AD servers. I advise you to consider this if it's important to you.
Senior Software Engineer at a tech services company with 501-1,000 employees
Real User
Jun 23, 2021
@it_user860751 I want to store application logs, metrics , kafka queue stats topic level, in short looking for best APM solution in terms of cost, efficiency , scalibility. Nearly 6GB- 7GB data daily basis i need to logs as part of app logs.
I find that Datadog is a solution that offers me flexibility when I am trying to decide what plan I should choose. Once I have established what I need, I can select the payment plan that I want from the three basic options that they offer. These plans are the following:
Free - This is the most basic of the three options. While it costs nothing to use, it also affords me the fewest capabilities. This only allows me to handle a maximum of five hosts at a time. Additionally, it only preserves logged metrics for one day before deleting them. It provides me with access to Datadog’s core collection and visualization features.
Pro - The Pro option costs $15 per month per host and is billed annually. This plan enables me to centralize my monitoring of systems, services, and serverless functions. Datadog Pro is an option that offers many more features than the free plan. For example, it makes it possible for me to retain metric data for 15 months.
Datadog Enterprise - At $23 monthly per host billed annually with a 100 host minimum, this is the most comprehensive option. This plan includes a variety of advanced features, such as an alert system that is powered by a machine learning engine.
The pricing of this product is good on a theoretical level. If all I need is the most basic plan, then it is easy for me to control the cost to my company. However, the other two options require me to be very careful with how many resources I use. My ability to plan out my resource expenditure is what ultimately affects what I pay at the end of the year. All of this being said, the way that this solution is priced can be said to be much more reasonable than many other competing products.
There are areas where, for the price that they charge for the more expensive plans, I feel Datadog could afford to improve. Areas of potential improvement include:
Security - Some of the aspects of its security suite could use improvement. One example of this is an aspect of its monitoring capabilities. Its Windows and Linux security monitoring capabilities leave a lot to be desired. This is an area where the cost should warrant more effective monitoring of these two major platforms.
Supporting documentation - It could provide documentation that breaks down various aspects of the product in a clearer way than it does currently. There is a presumption that we already know the service expertly. This makes it harder for us because instead of helping us with the basics, the documentation mainly discusses advanced use cases.
Ease of use - Datadog is not always the most intuitive product to use. It can have a learning curve to it. When they are compared to a solution like Tenable Nessus, their usability appears to be less attractive. There is definitely room for growth as far as its usability is concerned.
There are areas that I feel justify Datadog’s potentially high cost, though. These include:
Integration - Datadog’s integrations are invaluable. They enable me to connect to a vast array of service providers that add to the capabilities that I have at my disposal.
Single pane of glass - The solution’s dashboard gathers all of metrics and data I need in a centralized location. It also centralizes my solution controls so that I can exercise control from this one place instead of jumping to multiple different screens to perform critical operations.
Visibility - Datadog enables me to gain a level of visibility that shows me all of the critical aspects of my cloud environment and operations. Everything that could aid or harm my business operations is placed so that I can easily view them and use them to make critical decisions.
Datadog is a cloud monitoring solution that is used by companies across a wide variety of professional fields. My organization and I cannot help but recommend it, as it is a tool that offers us many valuable benefits and an immense level of value. This solution empowers us to prevent digital threats from harming our organization’s network as well as to ensure that everything is running in the best possible way. We gain a wealth of features and capabilities that make it so much easier for our administrators, IT, and compliance to do their jobs. The richness of the features suite gives us everything that we need to keep our organization’s network totally secure. They enable us to maximize the way that we use our resources.
Some of these benefits my organization sees from using Datadog include:
Our ability to customize our network by connecting and integrating with hundreds of turn-key platforms. Datadog enables us to integrate our network with more than 500 different platforms. This flexibility provides us with the ability to customize our network monitoring capabilities so that our needs are met in the best way possible.
High level of scalability. Datadog gives us the ability to add as many users as we wish to the log management portion of the solution. We can easily expand our log management team when necessary..
Flexible pricing plan. We can use this solution without tying ourselves down to an expensive commitment. Datadog allows us to pay by the month and control the costs that we incur. This keeps our costs down without compromising our security capabilities.
Datadog also offers us a rich suite of features that we can employ, including:
Customizable and prefabricated monitoring dashboards. Our administrators are supplied with two different types of dashboards that they can choose from when we are setting Datadog up. We can customize the dashboards to fit any specialized monitoring need that our organization might have. Additionally, we can choose to use prefabricated dashboards that come with the solution.
Easy to use search and filtering capability. Datadog has an easy-to-use built-in filter and search tool. We can search for and find anything that we need without having to devote a significant amount of time learning how to use this tool.
Vulnerability scanning tool. We can keep ahead of threats to networks by employing Datadog’s vulnerability scanning feature. This tool scans the entirety of our network and warns us if a vulnerability is detected. We can then move to patch these holes in our security before the threat to our network can escalate into something far worse.
Collaboration tools. Datadog gives us access to collaboration tools that enable our various teams to be united in their efforts to safeguard our network. We can all work in tandem and ensure that our efforts do not interfere with what other teams are doing.
Overall, I feel as though Datadog is a very solid product that has a great deal to offer. If I had to rate it out of ten, I would give it an eight or a nine.
Buying a cloud monitoring software solution can be a daunting process. You will need software that can be automated for any cloud infrastructure, provide visibility within critical business applications, and meet your organization’s budgetary restrictions.
What is cloud monitoring software exactly?
Cloud monitoring software is a network management platform that monitors cloud security proced...
Users consider Datadog to be one of the top cloud monitoring and security software products on the market. It allows you to see inside any stack, any app, at any scale, anywhere.
Datadog’s SaaS-based infrastructure monitoring provides:
Metrics
Visualizations
Alerting
to ensure your engineering teams can maintain and optimize your cloud or hybrid environments.
Some of the main reas...
PeerSpot’s crowdsourced user review platform helps technology decision-makers around the world to better connect with peers and other independent experts who provide advice without vendor bias.
Our users have ranked these solutions according to their valuable features, and discuss which features they like most and why.
You can read user reviews for the Top 8 Network Monitoring Software Tools ...
PeerSpot’s crowdsourced user review platform helps technology decision-makers around the world to better connect with peers and other independent experts who provide advice without vendor bias.
Our users have ranked these solutions according to their valuable features, and discuss which features they like most and why.
You can read user reviews for the Top 8 Log Management Tools to help you d...
PeerSpot’s crowdsourced user review platform helps technology decision-makers around the world to better connect with peers and other independent experts who provide advice without vendor bias.
Our users have ranked these solutions according to their valuable features, and discuss which features they like most and why.
You can read user reviews for the Top 8 Cloud Monitoring Software Tools to...
With Datadog, we have near-live visibility across our entire platform. We have seen APM metrics impacted several times lately using the dashboards we have created with Datadog; they are very good core indicators of the health of our system. You can build very useful dashboards right out of the box using this solution. Our developers are able to see how code is running in production, and having custom built-in metrics gives us a wealth of knowledge and historical data that helps keep our system running smoothly.
The pricing with Datadog can be very high. We feel there should be a more tiered approach to give users the opportunity to buy a more tailored product specific to their organizational needs. Datadog could also be more user-friendly. We would like to see deeper application-level insight and better incident management.
Using ELK, you can gather authentication information from service providers and determine which identity provider is not performing properly. ELK is very flexible; you can have a number of scenarios and get logs from all of them. ELK is a very cost-effective solution.
But it can be a complex solution to use. Better integration with 3rd party APMs would be very helpful. Currently, upgrades with ELK are released as stacks. Plugins or extensions would save removal and reinstallation and make the process move seamlessly.
Conclusion
We researched both of these solutions and concluded Datadog gave us the best visibility, better integration, and helpful, timely support. The logs and error reporting are extremely useful to conduct analysis and root cause analysis. The setup, ease of use, and flexibility with dashboard creation and reporting are just some of the things that our team liked about Datadog.
Datadog: Unify logs, metrics, and traces from across your distributed infrastructure. Datadog is the leading service for cloud-scale monitoring. It is used by IT, operations, and development teams who build and operate applications that run on dynamic or hybrid cloud infrastructure. Start monitoring in minutes with Datadog!
Datadog features offered are:
200+ turn-key integrations for data aggregation
Clean graphs of StatsD and other integrations
Elasticsearch: Open Source, Distributed, RESTful Search Engine. Elasticsearch is a distributed, RESTful search and analytics engine capable of storing data and searching it in near real time. Elasticsearch, Kibana, Beats and Logstash are the Elastic Stack (sometimes called the ELK Stack).
Elasticsearch provides the following key features:
Distributed and Highly Available Search Engine.
Multi Tenant with Multi Types.
Various set of APIs including RESTful
@Shibu Babuchandran thanks @Shibu for your valuable comments.
Where do you want to spend your money, on people or licenses?
ELK requires a long-term investment in engineering resources to manage the system and to provide the capability.
Datadog provides capabilities for you so you only need some administrators. What are the capabilities? Some critical ones include availability, scalability, consuming log files, platform upgrades, ...
If you are consuming smaller data sets (100's of GB) with shorter retention, the size and scaling are much easier making ELK easier.
Do you have admins or engineers? If your team doesn't have dedicated time & skills to spend developing solutions like elastic-alert you should look for a vendor to provide capabilities.
I expect some capabilities in Datadog you will not be able to replicate in ELK.... so that answer makes this obvious.
We are going to evaluate the same for our org. We do about 10 TB a day consumption in ELK and are looking to see if we can shift $$$ from engineers and infra to SaaS.
I have used both ELK and Datadog, and there are lots of variables to consider here. The three important points that I looked at are:
- Cost. In addition to service costs, you have to consider egress and ingress costs as well.
- Real-time observability that you need during development vs long-term Observability. Keep in mind, when you export data over the internet, it comes with the same reliability issues as any other service on the internet. Regardless of how Datadog classifies its service as real-time, it is not real-time, IMO. It very much depends on your definition of real-time.
- Deployment and maintenance complexity. When your ELK cluster grows it has some pain points you need to be aware of.
My general approach is to deploy ELK for development, tune the data, and then pivot toward commercial solutions if I need to. This gives you insight into your data and what you should be preserving and that way you are not paying high costs, when or if you do decide to take advantage of a commercial solution.
It depends on your requirement. If you are looking for a SIEM/log management solution ELK would be a better option.
But if you are looking for more of a monitoring solution Datadog would be better. Also, Datadog provides out-of-the-box integrations with a lot of cloud applications. ELK could be cost-effective but a bit challenging to configure & finetune.
Dear,
Unfortunately, I can't say much about Datadog but I have used ELK for a short period.
And I can tell you not everything works the way it should. For example, I noticed heavy CPU usage for a Windows client on MS AD servers. I advise you to consider this if it's important to you.
Good luck!
Can you tell me what you actually want to do so that I can help you?
@it_user860751 I want to store application logs, metrics , kafka queue stats topic level, in short looking for best APM solution in terms of cost, efficiency , scalibility. Nearly 6GB- 7GB data daily basis i need to logs as part of app logs.