Elastic Search Reviews

I have been using Elastic Search for the last five years.

I have a couple of use cases. First, I use it for logging purposes and observability logging of our product. In Azure, Elastic Search has good support. Whenever I deploy any application, it automatically detects the application and tags the elastic log with it. This provides proper logging and observability to our application. That is my main use case. Another use case is making AI agents. In AI agents, I use it for vector search. Vector search means whenever I am searching anything in Elastic Search, which is a database, I can perform vector search on whatever I store in the database. Vector search is similarity search. For example, if I ask what are the petrol prices today, it will try to find similar items such as petrol, diesel, or similar things. If I ask about petrol, it will not only search for petrol but can also search for diesel because they are both liquid forms. Elastic Search has this search capability. I take the similarity search and after that add some of my algorithms to create the AI agent using that.

In traditional search, I get some log file and have to manually find information in it. For example, with text search, I type some keyword and manually have to open it in Notepad++ or any other similar tool. With Elastic Search, it is much better. I can search based on date ranges. For example, if I want to check the last one hour of data, I give the time frame and my application data appears there. If I want to search history, such as what happened one week ago with this application, and some customer provided some issue saying that one week back they received this issue, I can search the logs from one week back and go through those logs. Elastic Search has more search criteria. With different search criteria I can search it. I can also search based on context, where if I select the search in that time frame, it will search just before and after some context for me. That is also available in Elastic Search.

Hybrid search can be used programmatically as well. In Elastic Search, there is one user interface where I can provide a lot of things. That is one part of search. Hybrid search means if I want to search programmatically, I can search and get some data from Elastic Search and use it in my application. For example, if I am developing one agent, I definitely have to write some code and search some data using my program in Elastic Search. In that way, hybrid search is very useful. I can directly connect with Elastic Search database where I store all the data and get the data and use it in my application, wherever I want to use it. For example, if I am developing the AI agent, that is fine. If I want to just apply similarity search, I can also use it in my application.

Observability is one part when I am deploying my application. When I deploy my application on the server in Azure, observability comes into the picture. Whenever I deploy my application, I need the log. Logging means observability, how my application is going on, whether I am getting any issues or whether I am getting any exception in the backend. That comes into the observability bucket. That is one use case of observability. The second is whenever I am developing RAG or AI agent. Whenever I am working on RAG, hybrid search comes into the picture, vector search, hybrid search. For security purposes, whenever it is deployed on Azure, it automatically handles security. I have worked with the cloud only, so I cannot tell much about security on this.

Regarding how I use Elastic Search in generative AI, I mostly use it for observability and RAG. Whenever I am deploying or creating the AI agent, I use RAG. Vector similarity search has been very helpful for me. I have different search criteria based on KNN or cosine similarity that I can use to search on Elastic Search database. The second is observability, which is also very good because most people are using Elastic Search because it is easy to use. As I explained before, I can give criteria by providing a date and time, and I can also see the graphs as well. Whenever I deploy the application, I can see usability graphs. It also shows the flow of data. Flow of data means if much data or some more operations are performed in this time frame, that graph will show as darker. I can easily see this because of small user interface presentations that are very good. I find it very useful in observability, log observability, and RAG development and AI agent development.

Hybrid search will be valuable.

Elastic Search is easy to use in Azure cloud. Mostly, my full company uses Azure cloud, so it is easy to use. Cost-wise, my company found Elastic Search is good. Cost matters. Based on cost and use cases, I found Elastic Search is good. Even compared to Splunk, Elastic Search has good easy-to-use user interface. Even non-technical people can easily search and easily observe the logs and easily track the applications. With Splunk, I found I have to be a little more technical in that area. There are key-based searches and some criteria that I have to remember. I found that difference between Splunk and Elastic Search.

Support-wise, it is good because I did not get much support work. Mostly my DevOps team handles it, but one or two times I did get support. There is a ticket creation option. Within the available time zone, somebody will be there to support me. Within two to three hours, somebody can help and try to resolve the issue.

Elastic Search is not specifically being used for certain purposes. I deploy Elastic Search database on the cloud and use cloud services so that nobody can attack. However, I do not use Elastic Search to resolve attack issues.

The basic main purpose of Elastic Search, as of now, I feel it can do more in the AI area. Sometime I saw that when I am developing RAG and have to generate the embeddings, which I call metadata, sometimes it tries to fail. That durability or issue handling should be improved, but apart from that, I did not find anything as of now. As per my use case, whatever I am using seems pretty good. Apart from that, some definitely improvement will be there. One improvement is that it should be faster. Whenever I am searching any logs, it takes much time. For example, if I open my log in Notepad or a similar tool, I can search the text within a second. With Elastic Search, it takes a little bit of time, ten to fifteen seconds. That can be improved. Sometimes, engineers take time to assign when I create a ticket.

Till now, I did not face any issue with the stability and availability of Elastic Search. It is not that the server is down. I faced issues such as some slowness. Whenever heavyweight logging will be there or heavyweight operations are performed, at that time, it will be a little slow. That sometimes also depends on cloud connectivity. Sometimes the cloud is only down, so it is very hard to perform my application better. I did not face any issue related to availability and other things. It is pretty good till now. The slowness is the one part, otherwise it is good.

Definitely, because I have very big applications in my company. It auto-scales up. Whenever I am deploying multiple instances of my application on a server, as I told, no need to give any configurations. For example, if I have five instances of my application I am deploying, automatically it will configure the five Elastic Search logs. Automatically it will create five Elastic Search configurations. Every application will have their own Elastic Search log. Auto-scaling wise, it is pretty good.

Support-wise, it is good because I did not get much support work. Mostly my DevOps team handles it, but one or two times I did get support. There is a ticket creation option. Within the available time zone, somebody will be there to support me. Within two to three hours, somebody can help and try to resolve the issue.

Sometimes, engineers take time to assign when I create a ticket.

I used Splunk. I have Splunk. Kibana, I think, merged with Elastic Search. I used Splunk and Kibana before. I am using pure Elastic Search now. For the last four to five years, I have been using pure Elastic Search. Before that, I was using Kibana and Splunk.

I am not aware of licensing and cost because I am not from the DevOps team. From a usability point of view, it is very easy to use and easy to plug with my application. I do not need extra configuration. Whenever I deploy my application on the server, I have to give the path of any observability tool such as Splunk or Kibana. Initially, I have to provide some extra configuration so that my log will appear on Elastic Search or Splunk. But nowadays, whenever I deploy my application, whatever logging I am doing is it will automatically connect with Elastic Search because Elastic Search has the capability to track. Whatever logging I am doing, whether it is SLF logging in Java, or in Python, whatever logging I am doing, basic logging is easily tracked by Elastic Search. No extra configuration is needed. It is just easy to plugin. I just deploy my application, and that is it. Automatically Elastic Search will track my log. No extra configuration is needed. I just have to make sure that I have Elastic Search services in my cloud and it should be enabled. That is all. Otherwise, it is easy to plugin.

Elastic Search is easy to use in Azure cloud. Mostly, my full company uses Azure cloud, so it is easy to use. Cost-wise, my company found Elastic Search is good. Cost matters. Based on cost and use cases, I found Elastic Search is good.

Elastic Search is easy to use in Azure cloud. Mostly, my full company uses Azure cloud, so it is easy to use. Cost-wise, my company found Elastic Search is good. Cost matters. Based on cost and use cases, I found Elastic Search is good. Even compared to Splunk, Elastic Search has a good easy-to-use user interface. Even non-technical people can easily search and easily observe the logs and easily track the applications. With Splunk, I found I have to be a little more technical in that area. There are key-based searches and some criteria that I have to remember. I found that difference between Splunk and Elastic Search.

Stack discovery is something I did not use till now. Whenever I am deploying my application on the cloud, and any attacks happen, I have some monitoring services in the cloud. Whenever something happens, if any attack happens to my Elastic Search database, it can happen through log injection. Something attackers can do a direct attack on my Elastic Search database and change some logs. This kind of scenario can come into the picture. I have some monitoring services deployed on the cloud. Whenever outside my company, outside of my company IP is trying to access my database or my data, that time automatically that monitoring alerts will be triggered and it will go to whoever is tagged into the mail. It will go to my higher manager and that mail will go to them. Regarding generative AI and how it will protect, nowadays, what is happening is that if I want to monitor this kind of attack, for that also, cloud is providing GenAI solutions. If this kind of attack comes, how automatically this GenAI resolves my problem, or how it suggests me to resolve the problem. That kind of solution I have already deployed on cloud.

I did not see much or connect with the support people much, but based on my experience, I would rate customer service as a four out of ten.

My overall rating for Elastic Search is eight out of ten.

I am using Elastic Search not only for search purposes but for rendering on maps as well.

I have not searched any vectors so far, so I cannot provide you with the exact output of that.

I was not using vectors in Elastic Search because I was using a vector database. As I mentioned, I use other databases for that. I have not explored it because when it comes to the data, Elastic Search will become expensive. In that case, what I suggest to my clients is to go with PostgreSQL, a vector database, or any other vector database. They are a startup, which is the problem.

We are using streams.

My favorite feature is always aggregations and aggregators. You do not have to do multiple queries and it is always optimized for me.

I always got the perfect results because I am using full text search with aliases and keyword search, everything I am performing it. It always performs out of the box.

It is easy because I have been doing it for years. The last version I remember is 3.5 or 3.1 that I used. Since then, I have been following Elastic Search and the changes they do. For configuration, I have never seen any problem.

Elastic Search consumes lots of memory. You have to provide the heap size a lot if you want the best out of it. The major problem is when a company wants to use Elastic Search but it is at a startup stage. At a startup stage, there is a lot of funds to consider. However, their use case is that they have to use a pretty significant amount of data. For that, it is very expensive. For example, if you take OLTP-based databases in the current scenario, such as ClickHouse or Iceberg, you can do it on 4GB RAM also. Elastic Search is for analytical records. You have to do the analytics on it. According to me, as far as I have seen, people will start moving from Elastic Search sooner or later. Why? Because it is expensive. Another thing is that there is an open source available for that, such as ClickHouse. Around 2014 and 2012, there was only one competitor at that time, which was Solr. But now, not only is Solr there, but you can take ClickHouse and you have Iceberg also. How are we going to compete with them? There is also a fork of Elastic Search that is OpenSearch. As far as I have seen in lots of articles I am reading, users are using it as the ELK stack for logs and analyzing logs. That is not the exact use case. It can do more than that if used correctly. But as it involves lots of cost, people are shifting from Elastic Search to other sources.

When I am talking about pricing, it is not only the server pricing. It is the amount of memory it is using. The pricing is basically the heap Java, which is taking memory. That is the major problem happening here. If we have to run an MVP, a client comes to me and says, "Anurag, we need to do a proof of concept. Can we do it if I can pay a 4GB or 16GB expense?" How can I suggest to them that a minimum of 16GB is needed for Elastic Search so that your proof of concept will be proved? In that case, what I have to suggest from the beginning is to go with Cassandra or at the initial stage, go with PostgreSQL. The problem is the memory it is taking. That is the only thing.

I have been using Elastic Search since around 2012.

I have never seen any instabilities, even from the initial state.

I have checked it for a petabyte of records. It is scalable.

One person can do it, but when it comes to DevOps, we need a team always. Only if we have to manage Elastic Search, one person is fine.

Neutral

I have used Solr and MongoDB as direct alternatives. According to the situation, it basically happens based on what the client wants. Sometimes they want Cassandra in place of Elastic Search. Our thing is only to suggest them. When it comes to the server costing, they are always asking, "Can we move to another server?" For example, I was working with a lower attorney's application and we implemented Elastic Search. For AWS only, we had to take two instances of 32GB for Elastic Search. After a few months only, the client asked, "Anurag, is it possible if we can go to another source if the latency is reduced or if some concurrency will reduce?" In that case, we had to move to Cassandra. Alternatives, I do use them.

Elastic Search is working fine with streaming. I do not have any problem with that. I do not feel any problem with it because the library works well for the solution I am providing in Go. The libraries are healthy over there and it has worked well. I am satisfied with that. If there are some lags, I manage that. I have not used it. My review rating for Elastic Search is 9.5 out of 10.

My use case has evolved over time with Elastic Search. Initially, we started with it as a searching solution. Before Elastic Search, our primary source of truth was SQL databases, the traditional RDBMS. We thought about taking the data from the traditional RDBMS because they were not able to cater to the scale that we wanted to achieve, so we migrated the data from MySQL, keeping it as the primary source of truth, but for the searching mechanism and wildcard searches, we migrated to Elastic Search.

My experience with the relevancy of search results in Elastic Search includes both traditional keywords and full-text search. In the supply chain industry, with millions of orders and customers such as CMA CGM, Maersk, or Kuehne+Nagel, filtering out those orders was essential, using a shipment number, transportation order number, or an origin or destination number. In the gaming industry at FDJ United, full-text searches make more sense to understand gaming intent. For example, when a user searches for 'I really want to play action games', we break down that full-text query, use custom text analyzers, and derive the intent behind the user's query in combination with a vector database alongside Elastic Search.

My assessment of the effectiveness of hybrid search, combining vector and text searches, shows that Elastic Search is remarkable for text-based searches. I have explored other solutions, but none can beat Elastic Search in that area. When I combine hybrid searches with vector databases, they store the mathematical representation of the data. For instance, to find the top 10 closest proximity based on a query, the vector database uses cosine similarity on the available data and suggests the top 10 results while Elastic Search can keep the metadata, enabling quick access to the entire database based on derived intent.

I have utilized trusted GenAI experiences related to semantic search and text-based search in my current project using Elastic Search. My go-to solution for text-based searches will always be Elastic Search, but for semantic search, I am trying to build a solution that emphasizes system-level understanding agents. For example, if a new engineer queries the agent for a system explanation, it scans all the relevant data and provides a comprehensive analysis of the service, contextualizing inputs to reduce hallucination, controlled temperatures for the LLM model, and reducing nucleus sampling. As for knowledge preservation, I use a vector database to store significant outputs generated by the LLM, depending on user preferences regarding the gravity of the analyses performed.

The best features of Elastic Search that I appreciate include its capability for eventual consistent systems where you do not need hard consistency, and it scales very smoothly. For wildcard searches and regex patterns, it really scales massively. It offers ILM, indexation lifecycle management, which allows you to enable a search for a span of six months for the data fed into the system while moving the rest to a new cluster. The structure of the inverted index document facilitates its core features, and I find how Elastic Search understands, indexes, and creates mappings for your data to be remarkable.

While Elastic Search is a good product, I see areas for improvement, particularly regarding the misconception that any amount of data can simply be dumped into Elastic Search. When creating an index, careful consideration of data massaging is essential. Elastic Search stores mappings for various data types, which must remain below a certain threshold to maintain functionality. Users need to throttle the number of fields for searching to avoid overloading the system and ensure that the design of the document is efficient for the Elastic Search index. Additionally, I suggest utilizing ILM periodically throughout the year to manage data shuffling between clusters, preventing hotspots in the distribution of requests across nodes.

I have been using Elastic Search for more than six years.

In terms of stability, I would rate it eight out of ten regarding downtime, bugs, and glitches.

For scalability, I assign it a ten out of ten.

I would rate Elastic Search's technical support as nine out of ten.

Positive

Before Elastic Search, our primary source of truth was SQL databases, the traditional RDBMS.

Estimating the return on investment from Elastic Search is nuanced; however, I can share that initially, search times from traditional RDBMS were around two to three seconds, and with Elastic Search, we reduced that to 50 milliseconds, indicating a significant improvement.

Assessing the complexity of deploying Elastic Search, I have a gray area because a separate DevOps team handles that aspect, but from my experience writing code and utilizing its features, I find it not complex at all.

Estimating the return on investment from Elastic Search is nuanced; however, I can share that initially, search times from traditional RDBMS were around two to three seconds, and with Elastic Search, we reduced that to 50 milliseconds, indicating a significant improvement.

On the subject of pricing, Elastic Search is very cost-efficient. You can host it on-premises, which would incur zero cost, or take it as a SaaS-based service, where the expenses remain minimal.

When comparing Elastic Search to other vendors and products, I have recently explored Algolia, which is also a fully managed service. Elastic Search offers a choice between hosting on-premises or as a fully managed service, which has been beneficial compared to other solutions.

In my company's relationship with the vendor, I have always worked in product-based companies using Elastic Search, often as part of solutions from companies such as Manhattan Associates and in the gaming sector. For B2B industries, they sold to large clients such as Maersk and CMA CGM while my current company, Agoda, operates in the B2C space.

Elastic Search does require some maintenance, especially when considering features such as ILM if you want to enjoy its capabilities. Maintenance tasks depend on the established data pipeline and may introduce some friction.

Currently, we are not using Elastic streams for log ingestion; previously, we utilized the ELK and EFK stacks with Logstash for log ingestion and Kibana for visualization. I also observe a trend where companies migrate to Grafana Loki instead of ELK.

Regarding integration aspects, Elastic Search has exposed REST APIs for all its services, making it easy to integrate with third-party models or endpoints regardless of the underlying infrastructure, as any modern development language can interact with these REST services.

I have not used the attack discovery feature.

My deployment of Elastic Search is on-premises.

At Agoda, we handle over 1.2 billion searches daily, facilitated by Elastic Search.

While I have been at my current company for four months, I am still getting to know my colleagues; however, I know there is a dedicated team focused on Elastic Search. This team exposes a service that acts as an intermediary for communication between Elastic Search and other services.

In my department, there are more than 100 people, whereas the overall organization consists of thousands, exceeding 10,000.

I would rate this review overall as a nine out of ten.

The main use cases are for logging, centralized logging system, and security purposes. We also use it for application monitoring and APM to monitor all the applications that run in our environment.

Applications developed by some of our users are monitored using APM, which is one of our primary implementations. For security purposes, we centralize logging for all 6,000 servers using Elastic Search. With more than 12,000 servers in our infrastructure, we need to track which server requires attention and receive alerts. For example, if we need to update all servers, some may be missed, but the system will trigger an alert to notify us. Monitoring and logging are the main functions we use in our current systems.

We are using Elastic Search for log ingestion only.

I chose Elastic Search because it has high search capabilities and setting up the cluster and maintaining it is very easy. Due to this, I found it very user-friendly. High availability and shards allocation are significant advantages that led us to shift to Elastic Search.

I particularly appreciate the sharding concepts because data has high availability. The semantic search feature and the new logsDB feature are valuable additions. These are things I appreciate most about the platform.

Semantic search is a very advanced feature that has proven useful for our data in current systems. I am working with Aadhaar, which is a Unique Identification Authentication firm. When we search for name-related terms, the semantic search provides relevant results. I have also implemented semantic features with hospital data, and it has been very useful for multiple cases.

Elastic Search Hybrid Search is an advanced feature that functions as a future vector database. Vectors are the main component of the database. In current systems, it shows only similar data, but with a vector database, we can store all types of data using vectors. Everything in the future will revolve around vectors. All systems are moving from CPUs to GPUs. This is very useful because comparing vector databases will be a more efficient way to store and retrieve data compared to traditional methods.

Pricing is very high compared to other solutions, but given the features they provide, the pricing is acceptable. The licensing part is also decent compared to other features. I have no issues with this because the features they provide are excellent and position us for next-level future capabilities.

Many banks are moving to Elastic Search, and many identification systems are adopting it because the search capability is significantly higher compared to other solutions, and data retrieval is also very efficient. Many industries are transitioning from old solutions like Splunk to Elastic Search. Banking sectors and healthcare sectors are leading this adoption. Many applications use Elastic Search as their backend, such as Zama. Industries are thinking about and adopting Elastic Search technology because of the features it provides.

There are several areas that need improvement. First, while storing data, there are many mapping issues and mapping conflicts that cause Elastic Search to reject the data. We have to develop solutions or significantly change our processes to address mapping conflicts. This is one of the issues that needs to be fixed.

Second, building semantic search requires significant setup and configuration work. If Elastic Search could provide a one-shot, easy-to-use semantic search implementation, many more users would adopt it. Currently, only a few users are using semantic search, but if they brought it with one-shot ease of use, many people could use it easily and create alerts.

Third, Elastic Search Vector Database needs more attention in the market. We need to bring more features about the vector database to make it easier to set up and use. The use cases also need to be brought to market. Additionally, building dashboards in Kibana is challenging. Compared to Grafana, Kibana has very few features and chart options. We need to enhance Kibana to allow very customized dashboards to be built. Kibana needs significant enhancement in this area.

I have been using Elastic Search for five years.

Elastic Search is stable and reliable until you build the cluster for one terabyte. If data reaches one terabyte, it functions well. However, if data exceeds that or reaches a bottleneck, it becomes unstable. If data is at eighty hundred gigabytes or seven hundred gigabytes, which represents seventy to seventy-five percent of the built cluster capacity, it is very stable and reliable. Search latency is very low compared to other solutions like ClickHouse. Stability and reliability are completely dependent on the data volume.

From the scaling perspective, horizontal scaling by adding extra nodes works well when data increases. We can easily add nodes into the cluster and scale horizontally. Vertical scaling is also straightforward where we can increase the size. We can add new nodes and new components very easily.

I have raised ticket sizes with them many times. I feel very supported by their customer service. For P1 tickets, they provide very immediate quick responses and join calls to support and troubleshoot the issue accordingly. They provide solutions very efficiently. Their service is very good.

I have used Splunk and Dynatrace previously.

I have worked with ClickHouse, and there were many issues with indexing while storing data. The approach is different with ClickHouse. I have also worked with Splunk, and it functioned adequately. However, when storing large setups or large amounts of data, Elastic Search capability is superior and is really useful for the end user.

I believe the initial setup for this solution is complex for new members. However, if you are technically strong and understand how Elastic Search systems work, it is very easy. With five years of experience, I have set up many clusters for banking sectors and healthcare sectors. I have built fourteen clusters in production environments with large-scale systems exceeding five terabytes. This will be typical for those who have technical knowledge and can build easily. Those starting without experience can use Elastic Cloud, which offers very easy one-click deployment. They can deploy an Elastic Search cluster with single clicks. Those with technical knowledge can build the cluster themselves, but those without experience can use Elastic Cloud. This is not an issue.

Correlation alerts is a feature I did not get the opportunity to work on. I have only theoretical knowledge but not practical knowledge.

We can use agentless approaches with a script in addition to agent-based approaches. We are building both agentless and agent-based solutions. Both are good. Agent-based approaches for fetching data work well. Both are functioning well.

Discovery is a feature we are using, and it works well. Attack is a feature I did not get the opportunity to try.

Elastic Search is very user-friendly, and we can easily integrate it with third-party models and other AWS S3 buckets. It is very user-friendly for integrating with other third-party tools.

My overall review rating for this solution is ten out of ten.

I can describe a few use cases for Elastic Search because in my previous company, we had a message database and needed to implement a search system. We first used Postgres full text search, but it did not work well, so we had to migrate everything into Elastic Search. Elastic Search could better index the data and we could search every document in instant time.

The key differences between Elastic Search and Postgres search, including both pros and cons, are primarily related to indexing speed. In Postgres, the full text search speed is quite noticeable if you have a message document. In Elastic Search, I am not quite certain about when comparing to normal data, but for our use case of searching through message documents, the speed difference is noticeable in Postgres because our documents are very large. Since Elastic Search is primarily built for search, I think it can better search through the document. Our documents were sometimes really large, ranging from 100 megabytes to 200 megabytes per document, so I think Elastic Search handles this much better than Postgres.

What I appreciate about Elastic Search is that the best features include the ability to search through very big documents and index and search through them really fast. This is the one thing I value most about Elastic Search.

Regarding stability, I have not had any crashes, downtimes, or performance issues with it. We did have one incident, but it was not from Elastic Search. I think it was an AWS service outage. The downtime was an AWS error, not from Elastic Search.

Concerning scalability, I find it scalable because it is quite scalable right now. We currently have a terabyte of compliance data, and the client can search through that very effectively. We have not experienced any scalability errors so far. I think our compliance data amounts to approximately five or six terabytes of data, which is very large. We can search through that document quite easily, sometimes in 7 milliseconds, sometimes one or two milliseconds. It was quite fast.

Apart from the good things, what I would like to see improved or enhanced in Elastic Search is the storage cost. I think the main problem with Elastic Search is that sometimes the storage was quite expensive. We also have a file system in addition to compliance. We have an FDS on our server, and we sometimes want to attach something on top of the FDS and search through every file without having to create a search index dedicatedly.

The missing features or functionalities in Elastic Search that I would like to see included in the future or some functionality that requires enhancement would be the ability to attach to our file system, such as network file system or NFS, or maybe our on-premise NAS server, and then search through everything, whether it is a document, text, or some information from those documents. That may be our primary use case right now, but we do not have that capability. Additionally, I would like to see a better search system so we can locally embed and find through everything.

I have been working with Elastic Search for approximately one or two years.

Regarding stability, I have not had any crashes, downtimes, or performance issues with it. We did have one incident, but it was not from Elastic Search. I think it was an AWS service outage, not from Elastic Search. The error was an AWS error.

Concerning scalability, I find it scalable because it is quite scalable right now. We currently have a terabyte of compliance data, and the client can search through that very effectively. We do not have any scalability errors so far. I think our compliance data amounts to approximately five or six terabytes of data, which is very large. We can search through that document quite easily, sometimes in 7 milliseconds, sometimes one or two milliseconds. It was quite fast.

I do not know anything about the tech support because I have not escalated any questions to the technical support or customer service teams. We have not talked to anyone.

I previously used a different solution for search. The solution I used for the search previously was Postgres full text search.

The initial setup process of Elastic Search was straightforward. I did not face many challenges or complexities except for the fact that we had to extract every document and build a search index. Aside from that, we did not experience much complexity during that time.

When it comes to pricing, I think we had to pay AWS approximately 1,000 to 1,200 per month for the overall stack. I am not quite certain about how much Elastic Search costs specifically because I was not in charge of pricing. The overall system cost was approximately 1,200 to 1,500 per month.

I do not find it cost-effective. I am not quite certain. Maybe the client might complain, but I am not certain. We just built out the system.

Before choosing Elastic Search, I evaluated other options. At first, we tried to go with Redis search because we really needed fast retrieval, but Redis search was closed source at that time, so we could not go with Redis search. We had to try Elastic Search and it performed quite surprisingly well.

Given my experience with Elastic Search, a piece of advice or recommendation I may share with other organizations considering it is that if you are looking for a simple search, I am not certain whether I would recommend Elastic Search. However, if you are handling message data with a massive amount of data and you need sub-millisecond search time, I think in that scenario Elastic Search outperforms everything. I would give this product a rating of eight out of ten. Especially if you are using SQL to search through the data, Elastic Search really outperforms SQL when you have to search through massive data.

We utilize Elastic Search to bring a bunch of data sources together into a large search corpus, which is used to power our core research platform.

We don't generally do a lot of full-text search with Elastic Search. We do a lot of keyword-based searching and a lot of faceted search, and it works really well. We've also had to build custom relevance algorithms based on data that's being stored in the search index. This is more about the algorithm being less about text matching and more about feature matching and relevance on a number of different scales. It's generally worked out really well.

The best feature of Elastic Search is it does exactly what it says. It's really easy to get set up and running and have search running very quickly with basic, out-of-the-box features. It scales very well, and we can do a whole lot with the core feature set before having to move to more advanced concepts. Even then, it performs very well, whether we need to expand into vector databases or decide that the Elastic Search Query DSL doesn't solve our needs anymore and have to go with ESQL or something. It expands and scales really well.

The hosted solution means Elastic Search takes care of the maintenance, which is one of the reasons we chose it. There's been very little maintenance from a data perspective on our side. As we make changes to our database structure, we've had to mirror them into Elastic Search.

We haven't had the opportunity to use the hybrid search with Elastic Search yet. I think there's a place for it in our long-term solution, but we're not quite there yet.

We haven't yet used any AI features built into Elastic Search.

To do what we want to do with Elastic Search, the queries can get complex and require a fuller understanding of the DSL. Once we start to build that understanding, it's another muscle we have, so it's not a bad thing, but it just takes a while to get up and running with expertise for our engineers.

It's not hard to learn how to use more complex things in Elastic Search; it's just a challenge we're going to face.

In my career, I've been using Elastic Search for three or four companies, probably on and off for 10 years.

We've had various very small blips with Elastic Search, but it's never been an issue that was concerning. We have limited infrastructure, so we could go further in terms of our hosted deployment to ensure that some of those things didn't happen. We've simply accepted the level of risk we have.

Thus far, everything seems really good in terms of scalability for Elastic Search. We don't have the largest data set in the world; we have millions of records, single-digit millions, so two or three million records. I feel confident knowing that we could times that by 10 or 100, maybe, and it would still work. The cost would obviously scale, the number of nodes would scale, but Elastic Search would be able to handle that level of scale.

Before I was using Elastic Search and actually before Elastic Search even existed, I previously used Apache Solr and Lucene in my career. The release of Elastic Search way back when was a boon because it was out of the box and did what it said. We've also worked with Pinecone, Amazon's OpenSearch, and essentially Postgres trying to do vector search in Postgres. All of those tools have their place, but if we're doing straight search, Elastic Search is just really the right answer.

The initial deployment of Elastic Search was really straightforward because we used the hosted solution.

We had Elastic Search live and our first initial searches running in our staging environment within a week. We moved into production with our full data set within six weeks.

We had one engineer working on this implementation. That's why it took six weeks.

Elastic Search's pricing is affordable when using the hosted solution through Elastic Search. The pay-as-you-go monthly approach has been nice, and if we scale as a company grows, we'll probably switch to a prepaid model, which will be an even bigger benefit. Having the hosted solution and not having to pay for essentially a DevOps person on staff to manage makes it affordable. We haven't really looked into serverless, which has its own benefits. I think serverless still had some challenges early on, and I wanted to go with something I had previously worked with. The hosted solution pricing fits, but the pricing for serverless also looks really interesting. The self-managed solution is nice from a pricing perspective, but we need the right staff to support it, and we don't have that staff.

We don't use Elastic Search for log ingestion, though I think they have a feature for this.

We haven't worked with anything in terms of Elastic Search integration process for third-party models with interference endpoints.

I'm not using the Attack Discovery feature because we're not using Elastic Search for our observability approach.

We have no partnerships or anything with Elastic Search. I would rate this review as a 9.

Elastic Search is normally used for full-text search where users are fully depending on it for searching by name, address, and similar fields, and we need to gather the data with good latency, so we normally prefer to save it into Elastic Search.

Elastic Search helps for full-text search because we normally use it for keywords and other related terms. If there are keywords and searching requires numerical data and other elements, we prefer RDS over Elastic Search. However, if it is regarding complete full-text search in which we cannot do any kind of indexing and it is very difficult, we prefer Elastic Search.

Elastic Search's best feature is that it is very convenient to save, plus it is schema-less, and it has very good latency and also provides us with different kinds of mapping strategies which allow us to optimize many things according to the data structure. It is a kind of non-structured and structured mix.

The benefits of using Elastic Search are mostly for two to three purposes. For logging, it is very easy to insert the logs into Elastic Search and start searching it using Kibana, and it is very easy to make visualizations over there. The second purpose is that we normally use it for views. If we have searches from the front end with a specific structure, it is very difficult to go to a different table and create the query in the database, so what we do is sync our database with Elastic Search and create a view on Elastic Search which will give us the result in milliseconds. This is how we are currently utilizing it.

Elastic Search has an annoying limitation regarding page size. It has a specific limit for queries on Elastic Search, and the default is ten thousand, and we can increase it. However, after increasing, it can slow down. Pagination in Elastic Search is very slow. If I need to parse one million records saved into Elastic Search, it becomes a nightmare because I need to do the pagination, and it is very problematic in that regard. I need to do ten thousand records and then go to the other page, and when going to the other page, it currently takes much more time than RDS. For specific cases, if we need to do full-text search and searching for one specific word returns less than ten thousand records, it works very well. However, if we go for more than ten thousand, then it creates an issue for us.

It has been almost ten years since using Elastic Search.

Elastic Search receives a stability rating of nine point five; we rely on it.

In terms of scalability, for the managed service, it is very easy, but the scalability aspect is a bit tricky. Scaling up Elastic Search cluster requires a bit of time because of sharding and replications. It takes more time since it needs to copy the data. For example, if we are working on three nodes and adding a fourth node, the synchronization process will occur in the middle, and the higher the data volume, the more time it will take. Scalability is rated around five to six.

Elastic Search's technical support receives a rating of eight.

Positive

Previously, we were using the AWS managed cluster on the cloud, but now we have created our own. On the same cloud, we have deployed Elastic Search on our EC2 machines, so it is self-managed, not on-premises. On-premises would be if we give the solution to somebody else, then we would deploy Elastic Search on their specific cloud, but we only deployed it in our system.

I did not go into the deployment part of Elastic Search because it is a DevOps matter. I was in a senior role, so I sent the request and we received it. Normally, it does not take a lot of time if the person deploying is capable; it does not take more than two to three days.

We have about twelve specialists.

I cannot say much about the return on investment part because we normally work on a use case basis. If we find some kind of issue in our database which is currently taking time, then we need to shift to Elastic Search, and it will start giving us very good results. On the cost-saving side, rather than increasing our RDS from a less cluster to a big cluster, we can create a specific separate Elastic Search cluster, and it saves our money on our basic structure while giving us much more performance. I cannot tell you the exact part on how much was saved with the calculation, and I cannot provide the numbers, but it saves our time on the debugging side. Using it on the logs and creating visualization is very convenient for us to search the log and identify the issue as soon as possible. This saves our time, saves the customer's time, and decreases the time to respond and resolve.

Elastic Search's pricing totally depends on the server. Managed services from AWS are used, and we have worked on a self-managed Elastic Search cluster. On the AWS side, it is very expensive because they charge based on query basis or how much data is transferred in and out, making it very expensive. That is why we moved to the self-managed option. In self-managed, it is very easy to handle. We do not think any kind of proprietary Elastic Search solution is required.

Elastic Cloud Serverless is not being used. The GenAI experience with features like agentic AI, RAG, or semantic search is not currently being used. Kafka Streams is being used for log instigation.

Elastic Search has many pros, but the cons of it are that it is not structured, and we need to put all the things which are connected into a single index. Therefore, we cannot use it for our base structure database, but we always use it for supporting purposes.

While part of Careem, there were hundreds of thousands of customers using the solution, and now that in a startup, the clients are no more than one hundred.

Elastic Search requires maintenance. We need to keep it updated because Elastic Search normally launches new features and versions on both Kibana and Elastic Search sides. We need to keep updated ourselves, and also, we need to do maintenance on the storage side. Normally, we use Elastic Search for timelines, saving all the data from beginning to end, so normally the storage maintenance is an issue, and we have to increase the storage time to time, but it is not related to Elastic Search; it is actually related to our use case.

There is lots of support for Elastic Search in different tools like Logstash which we normally use for integration, and there are other tools as well, but it is very easy and not a big issue for that.

The Attack Discovery feature is not being used. Big businesses cannot survive without Elastic Search because it gives us very good visibility and handles our use cases very well. If we need something reliable and trustworthy as a solution, then Elastic Search is the way to go, as it is an integral part of big solutions. The overall review rating for Elastic Search is eight point five.

Elastic Search is being used for two main streams. The first use case is an internal analytics engine for the usage of our services, which is based on logs that are put into Elastic Search indices to build different dashboards for key executives and developers, providing different levels of information. This is essential to provide statistics as a nonprofit organization funded by the Department of Energy and other infrastructures. The main focus is on web access to the Protein Data Bank for scientists and bioinformaticians with a publicly facing service supporting roughly 15 million users and an average load of about 700 requests per second. There are two data centers, one on the East Coast and another on the West Coast, serving the same publicly available interface. Logs from these services are monitored and collected, then put into Elastic Search database, from which different perspectives are provided for various stakeholders.

The second use case is Application Performance Monitoring, where Elastic Search APM stack is used to collect application performance metrics, primarily using Java, with a bit of Python and Node.js. Those three agents are used along with a standard infrastructure with the APM server that injects everything into Elastic Search indices for incident recovery and finding performance bottlenecks. As a nonprofit organization using an open-source license, there have been no problems with Elastic Search trying to change the license. Since no commercialized services are provided, the organization remains out of the scope of those issues and continues using open-source licenses. Recently, integration with an internal Keycloak instance was completed to provide role-based access to the Kibana application, which was a bit non-trivial but was managed successfully.

The experience regarding the relevancy of search results with Elastic Search is positive since it is used for providing search features for end-users of the Protein Data Bank. During ETL processes, information is collected from different data sources regarding proteins, including protein annotations and structures, which are transformed and loaded into the internal database. One part of that database includes Elastic Search indices. For search capabilities, full-text search is performed for end-users while keyword search is used primarily for internal needs, and no complaints have been heard about either of them.

The main focus is on web access to the Protein Data Bank for scientists and bioinformaticians with a publicly facing service supporting roughly 15 million users and an average load of about 700 requests per second. There are two data centers, one on the East Coast and another on the West Coast, serving the same publicly available interface. Logs from these services are monitored and collected, then put into Elastic Search database, from which different perspectives are provided for various stakeholders.

There are a couple of improvements that would definitely save a lot of headache with Elastic Search. One would be if the open-source license had multi-user access to Kibana, which exists in the paid tier license. There were also some difficult times with parallel and point-in-time interfaces, so better documentation could help, particularly more example-driven content. The provided documentation tends to have some common words but lacks real applicable examples. More specific examples, such as step-by-step guides, would be ideal. From a technical point of view, there are no significant issues recalled as Elastic Search has been absolutely awesome for this use case and covers 100% of the needs.

Elastic Search has been used for roughly five years.

Regarding stability, there are no major incidents recalled with Elastic Search. While not part of the DevOps team, nothing significant has ever exploded to affect the whole organization. If there were issues, the DevOps team was able to fix them quickly. Problems have been experienced with other services, but not with Elastic Search.

In terms of scalability, Elastic Search is good for this organization. A standard three-node setup with multiple clusters is being used for internal and public needs, resulting in six nodes per database across the data centers.

There has been no need to contact customer tech support for Elastic Search. It has been sufficient to visit conferences such as SCALE in Southern California Linux Expo, where Elastic Search has a booth to talk to their staff. The organization often relies on publicly available resources such as forums, issue trackers, and an internal knowledge base. Once, a ticket was created on GitHub concerning a Kibana issue with Application Performance Monitoring, but that was essentially the extent of it. The main sources of support are conferences and documentation.

No alternatives similar to Elastic Search have been tried. When the discussion about the open-source license started, OpenSearch was briefly looked into but the decision was made not to move forward because the organization felt secure in the current usage without commercialization.

Elastic Search AI, RAG, and semantic search have not been explored yet, as those opportunities for integration are just beginning. Nothing has been moved into production, so further comment cannot be provided. Standard agents from APM are being used to collect telemetry metrics and send them to the Application Performance Monitoring server, which are different from AI agents.

It is difficult to assess the current pricing of Elastic Search because the organization is in a specific niche as a nonprofit organization. On-premises instances are managed internally and a managed option had been considered, but that did not pass the board's approval. Open-source licensing has worked well, and there have been no ceilings where payment options for additional services needed to be considered. Users are quite satisfied with what is provided, and the organization is happy with what is received from Elastic Search.

The learning curve with Elastic Search was very easy. With a strong background in Java and software engineering, and having a great tutor in the organization who showed how to perform ingestion pipelines with Grok and how to use the development environment within the stack, the process was manageable. While it might be difficult for middle-level and junior developers, having someone experienced in the organization makes it manageable to share knowledge.

Elastic Search mostly requires maintenance during upgrades. While it is running in standard mode, there have been no major incidents from memory, so it has quite low maintenance requirements.

There are no official partnerships with Elastic Search; the organization is just a user utilizing the open-source license. Overall, this review has been given a rating of 9.

The main use cases for Elastic Search are index building and retrieving information using Elastic Search vector, vector search, and related functionalities. Search is the primary use case.

Computation is very good. The scalability is very good because we have a huge customer database that is searching lots of products, and auto-scaling or load balancing are the prominent features we are using in this.

If we look at the impact on operational efficiency, we can see that decision-making has become much faster due to real-time data and quick responses. We have also implemented many automations, which enhance our processes. For example, when we optimize certain fields to improve search functionality, it yields great results.

I have not explored Elastic Search at the most. Searching from vector DB is available in Elastic Search, and there is one more concept of graph searching or graph database searching. I have not explored it, but if it is not there, that would be an improvement area where Elastic Search can improve.

I have been working with Elastic Search for more than two years.

It is very reliable, and it has no downtime.

I believe it is scalable. Every day, we have thousands of users continuously utilizing the search feature. We haven't encountered any problems so far, and there is the potential for auto-scaling. It is currently a scalable solution.

We have not contacted them yet. So far, we haven't had any need.

Positive

The initial setup is straightforward.

We have a team of developers, so it is internally managed.

We have not calculated the ROI for Elastic Search, but we are a consumer platform where numerous searches are happening, and we are getting very good results from the current infrastructure of Elastic Search. Though the exact numbers or ROI were never calculated, the performance has been beneficial.

It is average compared to other platforms. There isn’t anything particularly special about the pricing. However, the pay-as-you-go model is advantageous for the organization, as we only pay for what we utilize.

We are using AWS for our solutions. In AWS, we are heavily using Redshift and Glue. We focus more on vector searches and boosting the keywords, and all those features we are using heavily. In search, the key parameter that we boost up during indexing is essential.

We self-implement Elastic Search in our e-commerce application. We are not currently doing a regex setup for RAG Playground, but there is a plan to do that. We are more into vector searches when it comes to how effectively the hybrid search capability meets our needs for combining traditional keyword and vector searches.

Regarding the workflow, we are using the API for real-time inference because lots of data is being loaded at real-time on the application, and it is working well for us. 

I can definitely recommend Elastic Search to be used wherever you have consumer search capabilities needed in a large or scalable manner because it is very effective. 

I would rate Elastic Search an eight out of ten.

I have completed two different Elastic Search implementations, and in both cases, the goal was to speed up very slow Postgres databases. As a platform PM, I am typically responsible for user management and company management. These areas are quite heavy depending on how many users, customers, or companies exist. Before Elastic Search, when we relied solely on Postgres, there were significant delays to user list pages and company list pages. In the other company, there was a lot of data displayed for particular list pages for admins. We combined Postgres with Elastic Search to speed this up, and it certainly does speed it up. We have used it throughout my current job and previous job.

From the customer side, Elastic Search is super fast and very efficient, delivering results quickly. We recently tuned a series of compliance results in the CMS where we would specify that certain results should come up higher by adding keywords and other factors. However, the results were not as good as when we restarted and used Elastic Search out-of-the-box search results. We actually got better results that were more logical.

The most significant issue I find with Elastic Search is that it gets out of sync, and this has happened in both cases where I have implemented it. When Elastic Search gets out of sync, for instance, if I create a company or user, it gets created in Postgres and then sometimes there is a delay for it to appear in Elastic Search. This could be a 15-minute delay depending on how it was implemented. If other significant processes are running on the platform where you are touching a lot of records, such as a million records, that will take a hit on Elastic Search. We have seen differences of 800 records between Postgres and Elastic Search. Proactive tools that would find and adjust any mismatches would be beneficial.

Occasionally, Elastic Search has failed, and when that happens, search results do not come up at all. This has been a rare occurrence, and I am not certain Elastic Search is entirely to blame, as it could have been platform storage or other factors. For the most part, the most common problem is the out-of-sync issue.

I have used Elastic Search since 2019 at the last two companies where I have worked.

I would say Elastic Search is pretty scalable. We have had good results.

Earlier, in the 2019 and 2020 range, we were having a lot of trouble with syncing, and we tried to see if consulting was available. At the time, we could not find what we needed from the knowledge bases, and we could not really get support. There was not a technical support option at that time. That may have changed. Currently, I do not think we have gone to Elastic Search to ask for any significant help.

Negative

Google Appliance was a search engine that Google had for a while, and we used that and were pretty happy with it, but then they deprecated it and it is no longer available. After that, I do not remember what we used, something else.

Although I am not an engineer, it seemed easy to medium to set up. It was not complicated.

For the initial rollout, I would say it was maybe two or three people for the initial implementation, and then I have a team of 40 or 50 engineers with somebody always working on updates and other tasks.

We have not yet used anything in combination with Elastic Search, but that is on the roadmap.

I would say Elastic Search's relevancy is okay, and if I were to give it a score, I would give it a B. Elastic Search works best out of the box as much as possible. When you start to overtune or put in other factors that will increase the priority of specific results you want to come up, it gets really complicated and then you do not necessarily get the best results.

Elastic Search works best when used out of the box without excessive tuning. My overall review rating for Elastic Search is seven out of ten.