StreamSets Reviews

We were receiving data from hospitals or any kind of healthcare service providers in the country. We were dominantly operating in the US. When we received that data, we had to classify it into different repositories or different datasets. This data was sent to different vendors, and for that, the data needed to get processed in different ways. We needed to bifurcate data at many steps with different kinds of filters. For that, we used StreamSets.

We could bifurcate the datasets that we received from different hospitals. We could bifurcate it on the basis of the medical requirements of the hospitals, and sometimes, on the basis of the schedule or purpose. We were obtaining data that we could then supply to some consulting firms or other sources.

StreamSets saved us time. The accuracy was pretty good, and it was definitely better than what we were using previously. Earlier, we had hired two people who were doing the job manually, and we were also using some other platform. We had to pay for them. Overall, we have saved a lot of time, and the accuracy has improved as well. We didn't calculate the time savings, but I believe we saved about three days in a week, so there were about 30% to 40% time savings.

StreamSets reduced the workload. There was a 10% to 15% reduction in the workload.

StreamSets helped us to scale our data operations. The limit at which we purchased this solution was incredible. We were never able to reach the limit that we purchased, but it helped us to increase or scale our operation. Especially in months when we received a higher number of entries, we were able to perform our work on time.

The ability to have a good bifurcation rate and fewer mistakes is valuable. In the scenario we had, when we had to bifurcate the data, we did not completely cut the data. We made a different route for one set of data, which went into a different operating system. There was also a complete set of data along with the original data that got cut, which once again went through the filtration process, and in this way, it kept on happening. Different solutions that were in place were not providing this feasibility. With the other solutions that we were using earlier, we had to reuse the data again and again from the start. It was a time-taking process.

Their support system was pretty good. When we were setting up the bifurcation protocols that we wanted to set up, we had a few support calls with them, and those were really helpful.

The design or the way they have set up the protocol is pretty good. One thing that I would like to add is the ability to manually enter data. The way the solution currently works is we don't have the option to manually change the data at any point in time. Being able to do that will allow us to do everything that we want to do with our data. Sometimes, we need to manually manipulate the data to make it more accurate in case our prior bifurcation filters are not good. If we have the option to manually enter the data or make the exact iterations on the data set, that would be a good thing. It does not have that feature. None of the solutions provides this feature, but this is the feature that we are looking for. If we could bifurcate the data or do manual manipulation of data at any point in time, it would be a game changer. 

Its initial setup could also be a bit easier.

I used this solution for about a year.

It's a stable product. We used it for about a year, and we hardly had to shut it down.

We are a medium enterprise. We only have three departments in our company, and only one of the departments is using it. Salespeople don't use it. The development people don't use it. We are the ones using it, and our job is to process the information, so only one department is using the solution. We have about 18 people in the department.

Up to medium enterprises, it's a good choice. You can scale between one million to ten million data files. I don't believe they offer the service for a hundred million or one billion datasets. It isn't too scalable for large enterprises, but for small and medium enterprises, it's good.

I'd rate them an eight out of ten. The only reason for not giving them a ten out of ten is that if you're doing very important work and you need to get the solution the same day, it's a bit tough to have the team support you in a very short period of time. They usually give you appointments about a day or two days later. Other than that, everything is good.

Positive

We were using another solution previously. The major reason for switching to StreamSets was that we needed to scale our operations. Our prior solution could have been scaled, but the cost of scaling was a bit higher. We would have had to hire one more person to be able to scale, but we did not want to hire more people, so we decided to use a completely automated solution for this part so that it could be handled by only one of our team members. That was the primary requirement. The cost-benefit analysis was done by one of our peers. His proposal was pretty good, and everyone agreed to it.

Its initial setup is a bit tough. You need to have the technical expertise to do that. The support team is good. They help you around, but if they could make it a bit easier, it would be better.

I believe it operates only from the cloud. We also received the data from our associations on the cloud. We processed it on the cloud, and everything happened on the cloud.

The initial setup was complex because we were not able to directly link the data we were receiving with the StreamSets solution. Linking it required us to fill in or enter some information in StreamSets, but we were not able to figure out what to enter. For that part, we needed their help.

We spent about a week. For the first three days, our team members were trying their best to do it, but then we had to schedule a meeting with them. In terms of the number of people, only one person was working with our team, and there were three people working with the product. I was also involved in the product as a product manager, but I was not directly operating that system.

It didn't require any maintenance as such. Any maintenance activities were related to our side of things. There were mistakes on our end. When we were entering different data, we had to do different configurations in the system.

We did the cost-benefit analysis before buying the solution, and it performed even better than that. We were able to replace two of our staff members who were doing this work. The cost that we paid for this solution was pretty less as compared to their salaries, so on the cost-benefit side of things, it was a good deal. We saved about two persons' manual wage, which is about $6,000 a month, and we also saved 15% of a week's time. These two were the biggest returns on the investment. The accuracy was also a bit higher.

Its pricing is pretty much up to the mark. For smaller enterprises, it could be a big price to pay at the initial stage of operations, but the moment you have the Seed B or Seed C funding and you want to scale up your operations and aren't much worried about the funds, at that point in time, you would need a solution that could be scaled. Simultaneously, you need a solution that you don't want to use on a very long-term basis. This solution could not be applied if we were operating with all the hospital chains in the US. We were operating just with one hospital. That's why it worked pretty well, so for medium enterprises, I believe it's very good.

To those evaluating StreamSets, I'd advise doing a cost-benefit analysis because the way of using StreamSets differs from person to person. Someone else might have a very different use case, and they may not run into profit using the solution. For us, it was a good solution because we were hiring people for this work. People were doing the job manually. We saved both time and money, so doing a cost-benefit analysis would be the best thing.

If you are looking to expand your domain or range of operations, StreamSets is very helpful. If you are just looking for a better data analytics tool that can do bifurcation on data, I believe there are other tools or services available in the market that do not focus on the expansion of operations. They focus on doing better and more complex bifurcations. 

StreamSets enables you to build data pipelines without knowing how to code. After generating a few responses, you have to enter some basic syntax or code, but generally, one can do a lot of no-code stuff, which was not an important aspect for us because we were operating in the IT space, and our entire team was capable of entering all the syntaxes that were required. It was not an issue for us at any point in time. In fact, in the operations that we were performing, we only used code. When we were testing out our initial datasets, we used some no-code features that were there, but at the later stage, we used only syntaxes.

We did not connect to the messaging systems, but we connected some enterprise databases. We were operating with a set of hospitals in the US, and we had to connect with them only the first time. Afterward, it was the data that was passing through the pipeline. Initially, for a completely new user, it's a bit tricky. Some technical expertise is required. It's a bit tough, but because the support team is there, one would be able to do it.

Overall, I would rate StreamSets an eight out of ten.

We use it for building a data lake in our content. We have sales multiple times during the day, and a sale is the trigger. Sales use the lake as a landing zone. We also use it for various types of data transformation.

It enables us to create data streams and pipelines that our team can use to identify areas for improvement. Our marketing team can read the data generated on sales to understand how we can integrate our product and focus on the areas in which we need more improvement. By the end of the day, we have an improved solution.

The lack of coding makes work easier and faster, and after creating a template you can immediately transform any source. It saves a lot of time and makes things efficient. You complete things on time.

The impact that it has had on my company is that when we have a variety of data that we want to convert or transform, StreamSets is helpful. We can store a maximum amount of data, and transfer various data from different departments and use the analysis to understand how to improve our business.

And because it's a service, it's very helpful to me as a CEO. It's serverless and secure.

In addition, the data drift resilience has reduced the time it takes to fix data drift breakages by 35 percent. Overall, StreamSets, as a solution, saves me about 45 percent of time, and has reduced workload by 25 percent. It also saves me about $500 a month.

Another benefit is that breaking down sums of data gives you the ability to create graphical reports and present them to any team, and they will be understood.

One of the things I like is the data pipelines. They have a very good design. Implementing pipelines is very straightforward. It doesn't require any technical skill.

We have also integrated it with Kafka messaging and it is not complex to do. It is really so easy to connect or integrate with data interfaces. And moving data into analytics platforms using StreamSets is easy. It doesn't require any coding, meaning your can transfer or move data into data payloads without coding skills. It's a good move, for someone in the beginning, who doesn't have any knowledge because it's quite easy.

Sometimes, it is not clear at first how to set up nodes. A site with an explanation of how each node works would be very helpful. 

Also, it doesn't provide a very good user experience.

I have been using StreamSets for three years.

It is stable. I've never seen any negative downtime.

Their technical support is very supportive. They really know what to do, and they are very good people, very friendly.

Positive

It took me three days to deploy it. I did it on my own. We use it in two departments in one location and there are four users.

There is no maintenance of the solution on our side.

Since I implemented StreamSets, we have more generated sales, on the order of 50 percent.

The pricing is affordable for any business.

The transformation logic is a bit complex when you begin and you may need to read the documentation. When you create logic, you have to be sure of the scenarios in the logic.

Any company that is looking for data engineering should use StreamSets because the pricing is quite favorable. I would recommend it.

In our department, we use StreamSets to design data pipelines that load all data from various RD and VMS sources to the cloud, such as Azure. We also use the data set for data analysts to generate panels for our organization, as well as for real-time use cases for monitoring and consuming other streaming data. Additionally, we are able to customize StreamSets to suit our needs and budget.

Using StreamSets to create pipelines for batch streaming or ETL is easy and straightforward. However, if one is new to StreamSets, it may not be so simple and may require a lot of documentation for assistance.

We utilize StreamSets' ability to connect to enterprise data stores, making it easy to begin trading instantly without needing to be technically skilled. We use StreamSets to move data into analytics platforms. In my experience, it is initially quite easy to move data back if we have a clear understanding of data transit, importation, and exporting from external sources.

This solution enables us to build data pipelines without knowing how to code. The solution includes templates that guide us and help us customize our data easily. It is essential that StreamSets does not necessitate coding, as this saves a considerable amount of time that would otherwise be spent writing code, as well as resources that would be required to hire experts.

Transformer for Snowflake can help with both simple and complex transformation logic. For example, creating a plan to perform EPL and machine learning operations is easy and fast. However, if the same operations are performed on-site, it can be difficult to troubleshoot events due to limited visibility into the results. StreamSets' Transformer for Snowflake is important to us because it saves us a lot of time and enables us to complete a task remotely with only two or three people.

It is important that Transformer for Snowflake is a serverless engine embedded within the platform. We have the capability of creating a data operations platform, so we don't have to worry or even be aware of what we are doing at the moment. We can simply create a device and use it in the pipeline we want it to be in.

The solution improved the way we work, benefiting both our customers and our development and retainer teams. StreamSets helps us develop a platform manually, with a lot of teamwork, either remotely or on-site, depending on which option we use. This has had a significant impact on our organization in terms of how we process and transform data.

I would say that it is very easy for us to update the template so that we can have real, actual data in APL claims and in the supply chain. StreamSets' data drift resilience is very effective and can run in the data grid. The data drift resilience has reduced the time it takes us to fix data drift breakages by approximately 25 percent.

StreamSets helped us break down data silos within our organization. The ability to break down data silos helps StreamSets to gain quick insights. In general, it is a great feature that ensures we have activities or processes in place. We know precisely what to prevent and what to implement.

StreamSets saved us around 30 percent of our time, meaning that a task that would take five hours to complete manually can now be done in around three and a half hours.

The reusable assets are reducing workload by 35 percent by allowing different people to use a single platform or resource, regardless of whether they have a similar SKU or a different SKU. This feature can help an organization simplify, implement, and transmit more easily.

It is not only the cost of one packet that we paid for, but now we are implementing a strategy using different people within the company. It would be very expensive if we had to hire a new person to manage that task and it would also take a lot of time. StreamSets is not only saving us money, but it is also ensuring that we complete strategies on time.

StreamSets as well helped us scale our operations, which has had a significant impact on our business. We now have a better understanding of how to secure data and provide reliable security for the transmission of data from internal servers to external services, as well as meeting our client's application needs.

The best feature that I really like is the integration. The software can be integrated with Azure Keyvault or AWS Secrets Manager, as well as scheduling. It is very easy to schedule an event, which is much easier than I expected through StreamSets. The solution is also fast at determining pipelines. Additionally, I like that StreamSets has many components, such as sources, processes, execution, and other useful elements that I need to plan.

There should be a concept of creating double variables because it's still missing.

The loading machine mechanism needs to be simplified. Currently, it takes some time to get familiar with and understand that. 

Visualization and monitoring need to be improved and refined. For example, it is difficult to monitor a job to see what happened in the past seven days when a transfer occurred.

The licensing model also has room for improvement. The solution is currently expensive.

I have been using the solution for five years.

The solution is stable.

The solution is scalable. We currently have four people using StreamSets in our organization.

The technical support is good and they prioritize issues based on their severity, so sometimes we have to wait a while for a response.

Neutral

The initial setup is a bit complex for first-time people. There is a lot of documentation that needs to be reviewed before deploying. The deployment takes around one month.

The implementation is completed in-house.

StreamSets simplified our data ingestion and integration process without the need for the large financial investment that would be required if we were to use other, cheaper solutions. This is due to StreamSets' security and safety in supporting various heterogeneous sources such as RDZMS, and Salesforce. StreamSets ensures that we have a secure and easy way to launch any integration tool, resulting in increased profits. StreamSets is very stable, secure, and compliant, and has yielded a return on investment of around 30 percent.

I believe the pricing is not equitable. Different businesses operate in various models and ways, so I wish StreamSets would be able to adjust their pricing depending on the intended use of the software. This would be beneficial to businesses with limited budgets. Currently, the cost of StreamSets is the same regardless of the amount of backup, which is costly.

I give the solution an eight out of ten. StreamSets still needs to improve the monitoring and visualization before the solution can be a ten out of ten.

Since StreamSets is deployed in the cloud, we don't have any maintenance requirements or costs.

I highly recommend StreamSets; it is an excellent tool with both batch and streaming capabilities. StreamSets is a great option for anyone to try, though it does require an organization to have the budget to use it.

My primary use case with StreamSets is to integrate large data sets from multiple sources into a destination. We also use it as a platform to ingest data and deliver data for database analytics.

One major benefit that we have realized with StreamSets is that we are now able to run pipelines that scale horizontally, instead of using a static service to host the service. This has improved efficiency and reduced our workload by around 85 percent. Initially, we started out with around 40 users. Now, there are 100 users. We have definitely scaled up, in terms of usage, with StreamSets.

The fact that it is a single centralized platform saves us a lot of time. It's very intuitive and very effective, saving us a lot of resources with its built-in capabilities. No manual intervention is needed, and nobody needs to oversee it. It's an "all-in-one" deal for us. We are able to save 15 to 18 hours per week. Tasks that required three people can be done with StreamSets itself.

And with its ability to integrate large data sets, we are now able to pull thousands of records instantly, thereby reducing the need to do some complex coding for this asset. That has also been a very big plus for us.

We also use it to connect our Apache Kafka with data lakes and, as a result, this connection has gotten much more efficient and quicker for us. The overall efficiency has also drastically improved for us with this. Connecting these enterprise systems using StreamSets is pretty easy. The StreamSets platform is very straightforward. There is no major coding required, so any non-technical person can also do it.

Without the need for any complex coding at all, we are able to pull records. The records are vast and very large and pulling them usually requires coding, but the fact that there is literally no coding required is a very big plus for us. Once you start to code, there is a lot of time involved and a lot of QA involved, but all of that is eliminated here.

And it has definitely helped us break down data silos. With our large amount of data, we have different data formats, and as a result, there are data silos that are present by default. With StreamSets, we were able to completely eliminate that because StreamSets has become a centralized system for us to accommodate everything. We have been able to get a single, centralized view of all our data.

We have a lot of different data formats, and transforming them manually without any tool or system is a cumbersome and frustrating process. We use StreamSets to do that. It has made that process much more elegant and efficient for us.

For me, the most valuable features in StreamSets have to be the Data Collector and Control Hub, but especially the Data Collector. That feature is very elegant and seamlessly works with numerous source systems. 

Also, the intuitive canvas for designing all the streams in the pipeline, along with the simplicity of the entire product are very big pluses for me. The software is very simple and straightforward. That is something that is needed right now. 

Apart from that, the user interface of StreamSets is very good. It's very user-friendly and very appealing. Moving data into modern analytics platforms is a very straightforward procedure. There is no difficulty involved in it.

In addition, the ETL capabilities of StreamSets are also very useful for us. We are able to extract and transform data from multiple data sources into a single, consistent data store that is loaded into our target system.

In terms of the product, I don't think there is any room for improvement because it is very good. One small area of improvement that is very much needed is on the knowledge base side. Sometimes, it is not very clear how to set up a certain process or a certain node for a person who's using the platform for the first time.

Some visual explanation or some visually appealing knowledge-based content would be very good. That is something that I could have done with, once I started using it, because I found it very difficult.

I have been using StreamSets for about a year.

It is definitely a stable product. In fact, it is one of the top products in the market in that particular category. We have not faced any stability issues so far, in terms of server speed, latency, or deployment.

It's a scalable product. In our company, the platform is used across seven teams in our organization.

A couple of more teams are evaluating StreamSets in our organization. They're running things and asking for some feedback from our side as well. There are plans to expand our use of it.

I have been in contact with their technical support and I would rate them very highly. They're very knowledgeable and patient. That is something that I like very much. For a very new user, it's not very easy to understand and we contact the support team over email.

We do have a relationship manager as well, who acts as the central point of contact for us. They're very prompt, knowledgeable, and friendly.

Positive

This was one of the first products we used.

Within about three months we were able to see benefits from the system. We saw a lot of time being saved, and about a 30 percent increase in our overall efficiency.

Apart from reducing our workload and improving our efficiency, we saw a 12 percent increase in our revenue last year after we implemented StreamSets. I know people will definitely see a return investment on their money from it.

From what I hear from my team, I believe it's moderately priced because they're happy with the pricing.

Server update maintenance is required, but that is minimal. Any product would require that type of maintenance. I don't think we are investing a lot of time and money in maintenance. The maintenance is just another cost for us. We have only two guys working on the maintenance part of the software.

It's a very intuitive product, modern, and very user-friendly in terms of the UI. Almost all our requirements have been met by StreamSets and we don't have any complaints so far.

I would recommend starting to use it as soon as possible. No tool is perfect. You have to choose the best of the lot. I certainly believe StreamSets is at the top of the ladder when it comes to similar software.

My biggest lesson from using StreamSets is that data integration can be done much more easily now. I only knew that after starting to use StreamSets. When it comes to data integration from multiple sources, and having multiple destinations, people always assume it's a time-consuming, cumbersome project. But once we started using StreamSets, all those assumptions were broken. It's very straightforward and elegant software.

We are using StreamSets to migrate our on-premise data to the cloud.

I really appreciate the numerous ready connectors available on both the source and target sides, the support for various media file formats, and the ease of configuring and managing pipelines centrally. It's like a plug-and-play setup.

StreamSets should provide a mechanism to be able to perform data quality assessment when the data is being moved from one source to the target. So the ability to validate the data against various data rules. Then, based on the failure of data quality assessment, be able to send alerts or information to help people understand the data validation issues.

I have been using StreamSets for a year and a half. 

It's reasonably stable.

It's reasonably easy to scale. Around 25 to 30 end users are using this solution in our organization.

Customer service and support are good. 

Positive

It's reasonably easy to deploy. However, since it is used at an enterprise level, it requires maintenance. So we had a maintenance contract. 

In the financial industry, we have very strict regulations around deploying something in the cloud. So, it requires a lot of permission and other processes.

Just one person is enough for the maintenance. 

The pricing was reasonably economical and easy for us to afford when we engaged with StreamSets. It was not part of Software AG at that time.

It's a very good tool. Overall, I would rate the solution an eight out of ten. 

Our major use case with StreamSets is to build data pipelines from multiple sources to multiple destinations. We mainly use the StreamSets Data Collector Engine for seamless streaming from any source to any destination.

We also use it to deliver continuous data for database operations and modern analytics.

One great thing is that now, with the implementation of StreamSets, we have been able to eliminate about 80 percent of our break/fix costs and maintenance time. It is very easy to connect with streaming platforms and streaming services.

Also, we can integrate and stream databases by connecting with multiple streaming services. Before StreamSets, data transfer from source to destination took about three hours of time and it was prone to errors. Now, with the introduction of StreamSets, we primarily use the Data Collector and this has enabled us to complete the same job in less than 30 minutes. We save that much time per day or about 15 hours per week.

Another definite benefit is that it has helped us to break down data silos within our organization. We are able to work together, with the interaction of StreamSets. Previously, the data silos were extremely perilous because data would come from multiple, scattered sources. We were not able to consolidate it on time and we were not able to exactly pinpoint errors. But StreamSets has helped us streamline the use of multiple sources and destinations, completely eliminating the silos. That saves us a lot of time and we have reduced the number of errors by a lot.

The most valuable features of StreamSets, for me, are the Data Collector and the Control Hub platform. They are both very straightforward to use and user-friendly. And with the Data Collector and Control Hub, we get canvas selection for designing all our pipelines, which is very intuitive and useful for us.

In fact, the entire user interface is very simple and the simplicity of creating pipelines is something that I like very much about it. The design experience is very smooth. A great thing about StreamSets is that it is a single, centralized platform. All our design-pattern requirements are met with a single design experience through StreamSets. 

We can also easily build pipelines with minimal coding and minimal technical knowledge. It is very easy to start and very easy to scale as well. That is very important to me, personally, because I'm from a non-technical background. One of the most important criteria was for me to be able to use this platform efficiently.

Also, moving data to modern analytics platforms is very straightforward. That is why StreamSets is one of the top players in the market right now.

And one of the major advantages for us is the built-in functionality. StreamSets has a plethora of features that combine well with ETL.

In terms of features, I don't have any complaints so far. But one area for improvement could be the cloud storage server speed, as we have faced some latency issues here and there.

I have been using StreamSets for about eight months.

It is stable. It's a cloud-based solution, so there is a little bit of latency, some server speed issues, but apart from that, there is no question about the stability of the solution.

The platform is definitely scalable.

Maybe in the future we will increase our usage of StreamSets, but I don't see any immediate scalability requirements for us.

I have not contacted their customer support, but my team contacts them. From what I understand they have a pretty healthy conversation with the StreamSets customer support. All of our queries are sent via email and they get them sorted out. They also join Google Meet sessions or calls, if required, to sort out our queries. It has been a very smooth journey so far. I don't have any complaints with regard to their customer service.

Positive

StreamSets is the first solution that we are using in this space.

I was not fully involved in the initial implementation, but we did the implementation in phases. We wanted to get it on board as soon as possible, so instead of doing a complete implementation, we did it in phases and it didn't take a lot of time. We were able to get on with the work as soon as possible with this model.

The initial setup was simple. We didn't require any additional training or third-party vendors. We were able to do it along with the StreamSets team, so it was smooth for us.

We have 15 people using StreamSets, all at one location. They are developers and users.

Because it is a cloud platform there isn't much maintenance required other than server updates, but that is expected with any cloud platform. No extensive maintenance is required. We have a team of two people who maintain it and handle updates and all the latest releases.

Tasks that took three hours can now be done in less than 30 minutes. This is one of the prime data points in terms of ROI for this product.

In terms of money saved, we still haven't seen any direct results from StreamSets. With its automation, we are able to focus on other tasks because StreamSets is taking care of the operations side. Theoretically, it should save us some money but it hasn't until now. We still have the same number of employees.

We are moving in a positive direction. Hopefully, this trend continues. We were able to see the time savings and reduced errors within three months of deployment.

There are two editions, Professional and Enterprise, and there is a free trial. We're using the Professional edition and it is competitively priced. I wouldn't say it's cheap or moderate, but it's also not a high price.

We have been experimenting with Hadoop, but apart from that, we do not use it to establish a connection with other services. As an organization, we have not faced any issues with connectivity using StreamSets. The platform is very stable.

Overall, StreamSets is very efficient and effective. It has helped us save a lot of time and also reduced errors a lot. I would definitely rate it very highly. The major reason is that it gives us a single, centralized platform for all our design-pattern requirements and we are able to produce results efficiently. With StreamSets, we are able to transfer or stream data from any source to any destination. It has increased the overall efficiency of our organization.

Software AG is constantly improving and evolving the product, and that is something that I like: using a product that is ever-evolving and being upgraded.

After deploying StreamSets, I learned a lot about how data planning works and how easy it is to stream from multiple sources to multiple destinations. That is one of my major takeaways. I thought it would be a very complex task, but that myth was broken by StreamSets. The complexity was made very simple for me.

My advice is to try the free edition. It's a very user-friendly and intuitive product as well. Try it to get a grasp of what's happening inside the product. Once you try the free edition, you'll definitely go for the Professional edition. I don't have any doubt about that. The product itself will lure you. That is the power of the product.

I was working on an integration project where I was using the StreamSets platform. I was looking at both their data collector and their transformer. The idea was to integrate it with AWS SageMaker Canvas. Both of them are what they call no-code options. StreamSets is for data pipelining, managing your data flow, and transforming your data. SageMaker is AWS, and Canvas is basically their no-code option for machine learning.

I was trying to connect it to a data object repository. For AWS, that's a specific managed service called S3. I wasn't trying to run it with a data warehouse.

It's still in the trial stage. I don't get a 30-day trial period or anything like that. I just got to write about what's involved and then see if that's something that justifies the use case for going ahead and purchasing the license for it.

It enables you to build data pipelines without knowing how to code. It abstracts away the need for Spark or anything like that. This ability is highly important because it reduces development time.

It saves time because you don't have to write code. 

It saves money by not having to hire people with specialized skills. You don't need Spark or anything like that for doing the same thing.

It helps to scale your data operations. You can get to the execution engine and provision bigger machines or bigger clusters. You can scale out to however much data you need to scale out to.

The most valuable would be the GUI platform that I saw. I first saw it at a special session that StreamSets provided towards the end of the summer. I saw the way you set it up and how you have different processes going on with your data. The design experience seemed to be pretty straightforward to me in terms of how you drag and drop these nodes and connect them with arrows.

The execution engine could be improved. When I was at their session, they were using some obscure platform to run. There is a controller, which controls what happens on that, but you should be able to easily do this at any of the cloud services, such as Google Cloud. You shouldn't have any issues in terms of how to run it with their online development platform or design platform, basically their execution engine. There are issues with that.

It can break down data silos within the organization. One person can do the whole thing with StreamSets and SageMaker Canvas, but it hasn't yet had any effect on our operations or business because it's one of those situations where you can either get a demo from them or you basically have to go to one of these sessions and they give you temporary credentials and try to work with your use case. Personally, I would change their model a bit and give a two-week trial license for a cloud platform at the very least. You can then try to get something to work or call up their technical department and say, "Look, I've been evaluating this thing for the last few days. I don't know exactly how to resolve this issue."

I started using it in June of this year. 

The whole issue of the execution engine needs to be better resolved. If you pick a cloud, why isn't it working with this cloud? Or what do I need to do to get it to work with one specific cloud service if it can be deployed across multiple clouds?

It seems pretty highly scalable to me. That's not going to be an issue. Just the administration of it could be an issue.

It's currently being used in a dev department for machine learning. It's being used by the business analyst team.

I haven't contacted their support.

AWS has native solutions. There are AWS Data Wrangler and others that come bundled with their services, like AWS Glue. We haven't yet switched to StreamSets. It's still in the evaluation stage, but the no-code and the drag-and-drop option with a GUI are some of the things that seem to resonate with people. 

I was involved in its setup. I was the one who basically had to try to get it to run with whatever process or custom processor I developed. 

It was complex to set up. I had to go to the sessions. On a couple of occasions, I was doing it directly from the cloud platform, and apparently, that wasn't the way to do it. You have to go through their universal designer platform first. 

In terms of maintenance, once you're deployed from the cloud, that's all handled for you. It's managed for you directly from the cloud service. So, you don't have to worry about that. They maintain their design platform.

I didn't use any consultant.

I didn't get into that with the StreamSets representative. It seems to be pay-as-you-go, but I don't know exactly how they do it.

Alteryx is another option. It's a similar tool, and it looks almost the same as StreamSets. Alteryx is something that's available for any cloud. It doesn't matter which cloud. You go on the various clouds, and you look and see what they have.

To those evaluating this solution, I would advise looking into how it integrates with the cloud service that they're going to try it with. Does it naturally integrate better with AWS or Azure? It's one of those situations.

I used StreamSets' ability to move data into a modern analytics platform. That's what the AWS SageMaker Canvas is. It's like predictive analytics. In terms of ease of moving data into this analytics platform, doing the design on the StreamSets platform is one thing, but having the execution engine and getting that provision is a totally different ball game. Basically, that's where its limitation comes in.

Overall, I would rate it a seven out of ten. The issue that was never resolved for me was if you're running a compute or execution engine on AWS versus Azure versus GCP, how does that integration work because that has got nothing to do with StreamSets? That is outside of StreamSets. You're now dealing with the cloud service, and there's a good reason for that.

We use the whole Data Collector application.

We now consume many more hundreds of terabytes of data than we used to before we had StreamSets. It has definitely enabled us to do things a lot faster, and be a lot more agile, with a lot more data consumption and a lot more reporting.

Another benefit is that it has helped us to break down data silos. We now consume data across different silos and then we aggregate it together so that we can do reporting that is not just for that one silo of people but for a number of different people across the entire organization. That has had a positive effect, enabling us to save money, spend money more effectively, and have more up-to-date data in reports, as well as in auditing. Our safety processes are better too.

One way we have saved money is thanks to how the solution streamlines the data that we pull in, data that we weren't pulling in before.

StreamSets allows more people to know what's going on. It helps us with better allocation of resources, better allocation of staff, and right-sizing. We're in oil and gas and, in our case, it allows us to optimize what we're pulling out of the ground and then what we're selling.

It has helped to scale our data operations and as a result, in addition to saving money and right-sizing, it's helped our field operations and provided us with more management reporting.

Also, the data drift resilience reduces the time it takes to fix data drift breakages.

The most valuable feature is the pipelines because they enable us to pull in and push out data from different sources and to manipulate and clean things up within them.

We use StreamSets to connect to enterprise data stores, including OLTP databases and  Hadoop. Connecting to them is pretty easy. It's the data manipulation and the data streaming that are the harder parts behind that, just because of the way the tool is written.

The design experience is the bane of our existence because their documentation is not the best. Even when they update their software, they don't publish the best information on how to update and change your pipeline configuration to make it conform to current best practices.

We don't pay for the added support. We use the "freeware version." The user community, as well as the documentation they provide for the standard user, are difficult, at best.

However, we have a couple of people in-house here who are experts in data analysis and they have figured out how to use this tool. We have to have people who are extremely skilled to go in and write the pipelines for this software because it's so complicated. The software works great for us, but there is an extremely steep learning curve because they don't provide a lot of information outside of paying their ridiculous support costs. Their support starts at $50,000 a year and up.

Also, the built-in data drift resilience for ETL operations requires a bunch of custom code development to be able to handle that. It's somewhat difficult because you have to customize it a fair amount.

I also would like a more user-friendly interface and better error-trap handling.

We have been using StreamSets for about four years.

We just patched ourselves up to the latest release about a month ago, so it's actually pretty stable at this point. It used to be quite buggy, going back over the last little while, but it's pretty stable now.

This software is very scalable.

We did not have a previous solution.

The initial setup was somewhere between straightforward and complex. It was pretty straightforward to start with, but then it started ramping up to be more difficult as we wanted to add more stuff in.

The difficulty depends upon your data sources. If you have just one data source and you want to consume a lot of different types of data from that one source, it's pretty straightforward. But when you have 20 or 25 different data sources, and you need to pipeline all that data into a couple of data warehouses so that you can use advanced data analytics software to do reporting, analysis, and notifications, it's a lot more complicated. With every data source, it becomes exponentially more complicated to manage.

We spent a significant amount of time doing it, but otherwise, it was seamless because it was our own staff. We didn't have to worry about trying to find money or resource time or do any of the prep work needed to get external resources.

Ours is a single deployment, but it is used across our entire staff base of 200-plus people. We need three people for deployment and maintenance, whose responsibilities include software management, application management, and data analysis and management.

The ROI we have seen is in savings of time and money.

We use the free version. It's great for a public, free release. Our stance is that the paid support model is too expensive to get into. They should honestly reevaluate that.

We tried to go and get them to look at their licensing and support model and they said they were not interested in reevaluating that in any way.

We tried to use another freeware ETL tool. It's fairly well-known. We ran it for a couple of months but it was going to be even more difficult than StreamSets, so we chose that in the end.

The ease of using StreamSet to move data into modern analytics platforms, on a scale of one to 10, is about a five.

The solution enables you to build data pipelines without knowing how to code if it's the latest, state-of-the-art cloud connecting stuff. If it's for anything structured for Oracle and SQL Server and other data sources, it's difficult. Without knowing how to write code, some of it's easy and some of it is not.

My advice to someone who is considering this software is to be very aware that their integrator and data analysis people will need a very specific skill set.