Amazon Kinesis Reviews

I work as a senior software engineer in eCommerce analytics company, we have to process a huge amount of data.

Only a few people within our organization use Kinesis. My team, which includes three backend developers, simply wanted to test out different approaches.

We are now in the middle of migrating our existing databases in MySQL and Postgres, to Snowflake. We use Kinesis Firehose to ingest data in Snowflake at the same time that we ingest data in MySQL, without it impacting any performance.

If you ingest two databases in a synchronous way, then the performance is very slow. We wanted to avoid that so we came up with this solution to ingest the data in the stream.

We use Kinesis Firehose to send the data to the stream, which then buffers the data for roughly two minutes. Afterwards, it places the files in an S3 bucket, which is then loaded automatically, via an integration with Snowflake that's called Snowpipe. Snowpipe reads and ingests every message and every file that's in the S3 bucket. This stage doesn't bother us because we don't need to wait for it. We just stream the data — fire and forget. Sometimes, if the record is not ingested successfully, we have to retry. Apart from that, it's great because we don't need to wait and the performance is great.

There are some caveats there, but overall, the performance and the reality of it all has been great. This year, 100% of the time when there was an issue in production, it was due to a bug in our code rather than a bug in Kinesis.

We save a lot of time with Kinesis, but it's difficult to measure just how much. We actually have something similar regarding some other processes. We have developed somewhere else a tool that takes note of the contents of the stream, places them into a file, manually uploads them to the S3, and copies the files into Snowflake. That could be done with Kinesis, but it could take two weeks or 1 month less to get it production-ready.

The first would be the one found in the AWS SDK using the asynchronous client: put Record batch function which allows you to put a list of records in one put record request, which saves time and it's more efficient. Also, by using the asynchronous client, the records are sent in the background using an internal thread pool that can be configurable for your needs. In our performance testing, we came across this setting was the fastest solution. It didn't impact anything in the performance of the system process.

The second one would be the ability to link the stream to other places other than S3 via configuration of the stream and without changing a line of code.

Lastly, you can also link a lambda function to the stream to transform the data as it arrives in before writing it in S3, which is great to perform some aggregations or enrich the data with other data sources.

The default limit that they have, which at the moment is 5,000 records per second (I'm talking about Kinesis Firehose which is a specialized form of the Amazon Kinesis service) seems too low. Actually, on the first week that we deployed it into production, we had to roll it back and ask Amazon to increase the default limits.

It's mentioned in the documentation, but I think the default settings are far too low. The first week it was extremely slow because the records were not properly ingested in the stream, so we had to try it again. This happened the first week that we deployed it into production, but after talking with Amazon, they increased their throttling limits up to 10,000 records. Now it works fine.

We've been using this solution since September 2019.

The stability is great. I'd say that maybe we have it running 99% of the time, and nothing stops it.

Amazon Kinesis is definitely scalable. We have huge spikes of data that get processed around midnight and Kinesis handles it fine.

It automatically scales up and down, We don't need to compute it for that. It's great.

The only time that we needed to contact Amazon was to ask them to increase the throttling limit. They replied to us very quickly and did what we asked.

Initially, we were evaluating Kafka. I think Kafka is faster, but it's less reliable in terms of maintenance; however, when Kafka works, and you have it properly configured, it's much better than Kinesis, to be honest.

On the other hand, Kinesis provides us with better maintenance. Our DevOps team is already oversaturated, so we didn't want to increase the maintenance cost of the production environment. That's why we decided to go with Kinesis; because performance-wise, it's easy to configure and maintain.

I found this solution to be really easy to configure. The essential parts of the configuration include naming the stream and also configuring the buffering time that it takes for a record to get ingested into S3 (how long it will be in the stream until it's put into an S3). You also need to link the Amazon S3 buckets with the Amazon Kinesis stream. After you've completed these configurations, it's pretty much production-ready. It's very, very easy. That's a huge advantage of using this service.

Deployment took a few minutes.

You don't need a deployment plan or an implementation strategy because once you configure it, you can just use a stream. It's not an obligatory version that needs a library, etc. This stream is completely abstract in that way. You only need to configure it once, that's it.

We have seen a return on our investment with Amazon Kinesis. We are able to process data without any issue. It's our solution for ingesting data in other databases, such as Snowflake. 

Developing the stream process manually or using Kafka

If you want to use a stream solution you need to evaluate your needs. If your needs are really performance-based, maybe you should go with Kafka, but for near, real-time performance, I would recommend Amazon Kinesis.

If you need more than one destination for the data that you are ingesting in the stream, you will need to use Amazon Kinesis Data Streams rather than Firehose. If you only want to integrate from one point to another, then Kinesis Firehose is a considerably cheaper option and is much easier to configure. 

From using Kinesis, I have learned a lot about the synchronous way of processing data. We always had a more sequential way of doing things. 

On a scale from one to ten, I would give this solution a rating of eight.

I work in a gaming company that builds games for the global market. We use Amazon Kinesis to stream events. 

I have worked in companies that build tools in-house. They face scaling challenges. 

Amazon Kinesis should improve its limits. 

I have been using the product for a month. 

I rate the tool's stability a ten out of ten. 

My company has two to three users for Amazon Kinesis. 

I rate the tool's deployment a nine out of ten. Deployment takes one day to complete. 

The tool's entry price is cheap. However, pricing increases with data volume. 

I rate Amazon Kinesis a seven out of ten. 

We use it for:

• Security 
• DDoS attacks
• Server application firewall
• Kinesis makes it easy to collect the process 
• Real-time analysis
• Streaming data

We can get insight and react quickly to new information.

Kinesis is in real-time. It enables you to process stream data in real-time. You can drive it in seconds or minutes instead of hours or days.

Kinesis is a fully managed program streaming application. You can manage any infrastructure. It is also scalable. Kinesis can handle any amount of data streaming and process data from hundreds to thousands of processes in every source with very low latency.

Kinesis is good for Amazon Cloud but not as suitable for other cloud vendors. 

I have been using Amazon Kinesis for two years. 

We have used their support before. They are quick to help. Support is very good. They can resolve an issue within ten minutes. 

The initial setup was straightforward. The deployment took two months. 

There are 15 employees who work on it in my company. Some roles include myself as the team lead, support, and solution architects. 

I have seen ROI. 

Kinesis has the best of Amazon: data streaming, building processes, data analytics, data in real-time are very good. The output and monitoring are easy. It has good performance.

I would rate it a nine out of ten. 

We do data acquisition based on what is pumped from the remote data and process it centrally so that we may present to our customers meaningful reports, charts, additional layers of support, or alerts. 

At the moment, I am not using Amazon Kinesis, but Azure Event Hub, which I have found to be more meaningful and easier to use. 

I like the event bubbling feature of Amazon Kinesis, although I ultimately switched to Azure Event Hub. Both solutions have similar features, but the latter offers us certain operational advantages. 

Amazon Kinesis is not a bad product, but Azure Event Hub provides us with certain operational advantages, as our focus is on Microsoft related coding. This is why .NET is what we use at the backend. While we can use both Azure Event Hub and Amazon Kinesis towards this end, I feel the latter to be less customized or developed for use in connection with the server-less programming.

Amazon Kinesis has a less meaningful and easy use than Azure Event Hub. 

Amazon Kinesis involved a more complex setup and configuration than Azure Event Hub. 

I have been using Amazon Kinesis for the past year, although I have since switched to Azure Event Hub. 

The scalability is pretty good. One can have any number of nodes spawned or replicated on the primary. Any load can be handled, perhaps a few terabytes with ease in around 15 seconds. One can scale up to this. 

While we have not had occasion to contact Amazon tech support concerning the solution, we have in relation to other matters. We felt it to be good. 

The initial setup and configuration of Amazon Kinesis was more involved than that of Azure Event Hub. 

The solution's pricing is fair. The trick lies in Amazon's pricing. They charge according to the different layers of or types of data that is transfered.

In addition to Azure Event Hub, we also have experience with Apache Kafka, which I feel to offer greater power but more complex configuration. This solution has more features for a variety of purposes. 

The question of whether I would recommend Amazon Kinesis over Azure Event Hub is tricky. While both have their advantages and I consider them to be almost equal, we feel the latter to be better suited to our environment, which is why we went with it. The data transferring policies and associated costs of Amazon were the deciding factors for me.

I rate Amazon Kinesis as an eight or nine out of ten. 

We use this solution for quite large environments.

We use it to capture and process a lot of data. We use it, for example for data analytics and query and analyze a stream's data.

We are a sizable organization and as such, have a lot of data. The solution allows for real-time analysis and you can use a scaler to handle data flows. 

The solution is very flexible and allows for a lot of configuration. It just offers up a lot of possibilities.

I'm using Amazon S3 and Redshift using Amazon server. I can make large configurations and update in near real-time, so that we have real-time use for batch intervals. 

The solution is great for scanning in order to handle environmental data.

The data stream feature on offer is excellent. We use it quite extensively.

The solution works well in rather sizable environments. We deal with a lot of data and it handles it very well.

The solution has a very good alerts system to allow us to respond in real-time.

The dashboards are excellent.

The solution offers very good data capture and integrates well with Power BI and Tableau, for example.

The product makes it very easy to create jobs.

The automation could be better. The solution needs to be better at information capture.

Some jobs have limitations which can make the process a bit challenging.

In order to do a successful setup, the person handling the implementation needs to know the solution very well. You can't just come into it blind and with little to no experience.

I've used the solution for six or seven years or so.

We work with very large environments and haven't had any issues with feeling constricted by the solution.

Personally, based on my past experience and my long history with the solution, the initial setup was not complex. It was pretty straightforward. I find it very easy to use these tools.

A user will need to understand how to create analytics using processing a large amount of information. There may be legacy solutions in the mix as well. A new user will need to understand the environment and all of the requirements before really digging in.

What I will need, basically, is a data map, where I can find any legacy data. From there I can do the setup and it goes pretty smoothly.

I handle the implementation myself.

You can compare this solution to Data Factory and Hadoop. They have a few overlapping characteristics. However, for my industry, Hadoop, for example, wouldn't work as it was lacking some characteristics and parameters and some understanding of the industry itself.

I have a lot of experience in Kinesis and data analytics including in networking in the Amazon AWS environment. My experience is as a big data architect. I draw all environments in AWS. 

On a scale from one to ten, I would rate the solution at a six. It's pretty good, and great for big environments, however, you do need to be well versed in the product to set it up.