For people who are starting out, the simple advice is to first try out the cloud login of StreamSets. It is freely available for everyone these days. StreamSets has released its online practice platform to design and create pipelines. Someone simply needs to go to cloud.login.streamsets.com, which is StreamSets official website. It is there that people who are starting out can log into StreamSets cloud and spin up their StreamSets Data Collector machines. Then, they can choose their execution mode. It is all in a Docker-containerized fashion. You don't need to do anything. You simply need to have your laptop ready and step-by-step instructions are given. You just simply spin up your Data Collector, the execution mode, and then you are ready with the canvas. You can design your pipeline, practice, and test there. So, if you want to evaluate StreamSets in basic mode, you can take a look online. This is the easiest way to evaluate StreamSets. It is a drag-and-drop, UI-based approach with a canvas, where you design the pipeline. It is pretty easy to follow. So, once your team feels confident, then they can purchase the StreamSets add-ons, which will provide them end-to-end solutions and vendor support. The best way is to log into their cloud practice platform and create some pipelines. In my current project, there is a requirement to integrate with Snowflake, but I don't have Snowflake experience. I have not integrated Snowflake with StreamSets yet. I personally love working on StreamSets. It is part of my day-to-day activities. I do a lot of work on StreamSets, so I would rate them pretty well as nine out of 10.
Senior Data Engineer at a energy/utilities company with 1,001-5,000 employees
Jun 9, 2022
Every tool in the market at the moment has some major gaps, especially for large enterprises. It could be the way that the data or pipeline is secured. At present, StreamSets looks like the market leader and is trying to fill that gap. For anyone going through a proof of concept for various tools, StreamSets is almost at the top. I don't think that they need to look any further. We are working only with API, a relational database management system, and our enterprise warehouses at the moment. We are not using any streaming sort of ingestion at the moment. We are not using Snowflake Transformer yet. It just got released. We are using a traditional Snowflake destination stage because our enterprise is huge. We have our own Snowflake architecture. We load the security in the data into our own databases using the destination stage, not Transformer yet. I would rate the solution as 7.5 out of 10.
So many products in the market these days for building, deploying & monitoring Data Pipelines for Cloud based analytics.
Currently, I'm assessing Matillion ETL and StreamSets, which appear on par (on the surface). I'm curious what 1st hand experiences folks have.
Thanks for the help!