What is our primary use case?
I work as an Elastic consultant, so I am interested in learning about other tools.
The ability to deploy Elastic Stack both in the cloud and on-premises meets my organization's infrastructure needs and compliance requirements completely. Elastic holds most of the compliance requirements that customers would ever need, including being HIPAA certified and possessing various security-based certifications. Many government organizations use Elastic Stack, so it aligns with compliance standards.
How has it helped my organization?
The centralized logging capabilities of Elastic Stack have helped me streamline my logging processes significantly because there are many open-source tools available, such as Filebeat and Logstash, to collect the logs. Some people are using Vector these days, which is built on Rust, making it very fast. Since it is open-source, I can push logs from anywhere, and most of the client tools available that can push logs are open-source.
In terms of centralizing, this tool is mostly built for that purpose. People who are using traditional bases face a challenging issue with the schema; without a proper understanding of the schema, they cannot search. However, with Elastic Stack and other Lucene-based or indexing-based tools, everything is converted to JSON, parsed, and ingested. I can do full-text search within seconds because it creates inverted trees at the time of indexing itself.
What is most valuable?
The best features of the Elastic Stack are that it is mostly out of the box. Predominantly it is built as an indexing tool to store logs and gives me the ability to search quicker. Over time, they have built many frameworks on top of it, such as the security feature and centralized agent monitoring.
I can scale the cluster depending on the costing. I have all these data tiers, such as hot, cold, and warm, so based on the customer's use case, I can decide if they only need two days or three days' worth of hot data, and the remaining can be saved in a frozen tier, where everything is saved in an S3 bucket but still accessible for searching.
The most important feature now is the Elastic Agent, which comes with out-of-the-box integrations for most firewalls, Windows, Linux, and many other systems, and is improving. All I need to do is deploy the agent, deploy the connector, and they have the out-of-the-box parsers. Moreover, the detections in terms of security come out of the box as well. In terms of go-to-market, I can very quickly deploy a system for any customer without taking much time at all.
What needs improvement?
There are improvements needed for Elastic Stack. It is mostly based on Lucene, and the heart of Elastic Stack is Lucene, which has some limitations. Anything built on top of Lucene often feels an add-on, and that includes vector databases. Elastic Stack can store vector embeddings as well and perform AI and machine learning tasks out of the box without excessive configuration.
The main improvements involve increasing speed and compression capabilities; I have seen databases that claim to achieve significantly better compression. While Elastic Stack can manage vast amounts of data, if the mapping is not specified correctly, the indexing time can be slow, especially with many events per second. Improper mapping usually means that every document received gets indexed for all fields, which is not desired. Elastic consultants typically optimize this, but out of the box, as data volume increases, scaling becomes necessary. They are working on these improvements in new versions.
For how long have I used the solution?
I have been dealing with Elastic Stack for five-plus years.
What do I think about the scalability of the solution?
Elastic Stack supports high availability in my systems because it is a clustered system. I have at least two or three master nodes and multiple data nodes, which can be segregated for disk tiering based on disk performance. My hot tier runs on NVMe drives, while the cold tier may use slower disks or even EBS storage. High availability is configurable; by default, anything I ingest has a replica saved elsewhere on a different node, so no primary and replicas reside on the same machine. If a machine goes down, I can bring in another node, and it recovers on its own.
How are customer service and support?
I would rate their technical support a seven. The support personnel I work with are knowledgeable and helpful. However, the quality depends on the service agreement I have with Elastic Stack; not every customer has an enterprise subscription. If I do have an enterprise subscription, they are typically available within 30 minutes for a Sev1 issue, but lower-tier licensing may lead to slightly delayed responses. Resolution is much quicker if they are on the call. I often find that support is similar across most tools, which is why I would rate them a seven.
How would you rate customer service and support?
Which solution did I use previously and why did I switch?
I am not working with any new solution, but out of curiosity, I am comparing a few others.
How was the initial setup?
The initial setup of Elastic Stack is very straightforward. Setting it up is easy; to go to the cloud, I just need to visit elastic.cloud, select my preferences, and I do not even need to choose specific hardware. I select the region, such as AWS or Azure, and enter my anticipated data retention needs, such as 120 GB of storage. Based on my expected ingestion, I key in that size, and the system automatically selects the appropriate hardware and shows me the costs per hour or month. It allows me to create an account quickly, making it simple.
What's my experience with pricing, setup cost, and licensing?
My experience with Elastic Stack pricing indicates that it is node-based. While I do not have complete pricing details, they are available online. If I choose Elastic Cloud, it includes licensing and data transfer costs. To start with a bare minimum cluster in Elastic Cloud, such as a two-node cluster, the cost is reasonably low, around $5 to $6, for a setup that can store about 120 GB of data with all features enabled.
While starting, I can monitor external endpoints without needing an agent, but eventually, there are charges for API calls. For smaller usage scenarios, such as 5,000 to 10,000 events per second, it is relatively affordable compared to Splunk. However, costs can escalate for higher volumes of events, such as 100,000 per second.
What other advice do I have?
The flexibility of Elastic Stack when handling diverse data types is remarkable. It works as a log management tool, so anything I ingest into Elastic Stack undergoes a full-text scan, extracting tokens and creating an inverted index behind the scenes. This allows me to search for specific words or groups of words without any parsing, pinpointing documents effortlessly.
Users do not always just search; they also want to create dashboards and aggregations. In such cases, I can write parsers to break down messages and create fields such as text, keyword, or numeric fields. It is very flexible; I can ingest everything in its raw format and then refine it later, giving developers time to devise the proper schema and mappings.
X-Pack features have had a significant impact on my systems because X-Pack used to be a licensed feature for the open-source version of Elastic Stack, but now it is more integrated with different licensing tiers such as platinum and gold. Some customers who primarily operate in Kubernetes have deployed those as well.
I use the cost-to-performance ratio as the primary metric to measure the impact of the centralized logging capabilities, as most of the customers I deal with are really concerned about the cost. Performance metrics typically center around events per second, which are crucial for capturing performance. When building a system, it is not just one tool; it requires additional brokers such as Kafka. Even those can be monitored through Elastic Stack.
Security is one of the use cases, and observability has a separate connector. They also have out-of-the-box integrations for Kafka, RabbitMQ, and all the other systems, so I just need to deploy an agent and configure JMS or Kafka. All the logs and metrics are collected, and they come with their own dashboards. If someone wants to start a monitoring solution and later extend it to a security solution, it is seamless. They can start very small and pay very little, then explore additional features over time. That is how I perceive Elastic Stack.
I would rate the overall product as a nine.
Which deployment model are you using for this solution?
Hybrid Cloud
*Disclosure: My company has a business relationship with this vendor other than being a customer. Partner