Amazon Kinesis Reviews

The solution provides real-time event streaming. 

From my experience, one of the most valuable features is the ability to track silent events on endpoints. Previously, these events might have gone unnoticed, but now we can access them within the product range. For example, if a customer reports that their calls are not reaching the portal files, we can use this feature to troubleshoot and optimize the system.

I suggest integrating additional features, such as incorporating Amazon Pinpoint or Amazon Connect as bundled offerings, rather than deploying them as separate services.

I have been using the product for a year. 

My company has four to five customers. The scalability of Amazon Kinesis has improved various data processing capabilities.

I am happy with the tool's customer support. 

Amazon Kinesis' deployment is easy. 

The tool's pricing is cheap. 

If you're considering using Amazon Kinesis for the first time, I would advise exploring their services. It can be useful for capturing real-time events, greatly benefiting your solution. I rate it a seven out of ten. 

We have utilized the solution for ingesting the data from different applications. For example, when people use a web server they send their weblogs, and clickstreams and the service provider wants to know how many users are currently using the site and what their areas of interest are, we use Amazon Kinesis which has the capability to enable the analytics and provide the information to them.

We also use Amazon Kinesis Data Firehose for putting the IoT data on the Kinesis Data Streams because the data has to be brought from on-prem to the cloud in order to perform the analysis.

The solution has improved our organization by saving time. Before launching the solution, the only thing we have to do is know the volume of the data and what is the frequency of the data we are going to receive. Based on that, we can configure the capacity of the Kinesis Data Streams, so it divides our requirements in terms of the shards or partitions. Unlike Kafka which works in partitions, Amazon Kinesis works in shards. This means the solution can handle thousands of requests per second for writing and reading. The writing capacity is one KB per request and the reading capacity is four KB per request.

The solution has the capacity to store the data anywhere from one day to a week and provides limitless storage for us. The data is time-stamped, and the data is in sequence, so we don't need to maintain the sequence or order of the data, and multiple consumers can reference the data in the Kinesis Data Streams simultaneously. Using the same data one application can perform Task A, and another application can perform Task B. 

The Kinesis Data Stream is integrated with Amazon CloudWatch where we can monitor all the requests of who wants to read information, what kind of APIs are being requested, what errors there are, who is producing them, and who are the data consumers. All the information is logged in one spot and this allows us to identify the problematic points easily.

The solution allows us to apply the security so the consumers can have access, based on the subscription they have.

Amazon Kinesis has a fan-out feature that allows us to increase the throughput when the number of consumers increases, instead of having to pull the data from the consumer side the information is posted by the solution itself.

The services which are described in the documentation could use some visual presentation because for someone who is new to the solution the documentation is not easy to follow or beginner friendly and can leave a person feeling helpless.

I have been using the solution for five years.

The stability is good as long as no configuration mistakes are made and the data source works properly.

The scalability is good but if for example, we only created one stream, then wanted to scale it further, we would have to continuously monitor using CloudWatch to ensure the shards are not getting overloaded, otherwise, we have to split the shards. We would have to merge the shards if it is under load. The implementation is only required once with some monitoring and after that, it is very easy to scale. The scalability has some limitations but there is also an on-demand option with a script so when we need to scale up the capacity we can and if we need to decrease the capacity, we can. We can also stop the service completely by scheduling it based on event triggers.

The technical support was not able to resolve the one issue we had.

Positive

The initial setup is straightforward and I give it an eight out of ten. The deployment takes around five minutes. 

We have to create our service using CLI Streams, or we can create using the console because we have to provide the configuration, including, what throughput we want, and how many reads and writes we want to be supported. Once we launch, we'll be charged monthly. Setup is very easy, we have to specify whether we want encryption of the data or not if it is accessible to everyone or not, and what kind of services are going to interact with the solution. In AWS, we have to specify, and then we have to provide the roles and policies based on the consumers.

The implementation is completed in-house.

The solution is worth the money because it is easier to set up, and since it is integrated with all the AWS services, we can manage the security, the monitoring requirements, and manage the audit information, much easier. 

The solution is costly. There is a fee to activate the service and even if it is not being used there is a monthly fee because they continue to maintain the service. If we want to retain the data for longer durations then we are charged the equivalent of Amazon S2 or S3 services. The fee is based on the number of hours the service is running. 

I give the solution a nine out of ten.

No maintenance is required for the solution because it is cloud-based.

The solution is available in all three zones, Amazon Kinesis is a good solution if a person wants scalability, availability, and durability of data. I recommend the solution to anyone already using AWS Amazon-managed Kafka service.

I use the solution in my company for streaming purposes, considering that my company has an AI-based camera for streaming.

The most valuable features of the solution are data streaming and the real-time data of our screen, which helps provide the analytics our company needs.

There are certain shortcomings in the machine learning capacity offered by the product, making it an area where improvements are required. There is a need to introduce something more into the machine learning area because it helps users learn and get newer things in their day-to-day lives. I think Amazon Kinesis should update machine learning and be up to the mark.

I have been using Amazon Kinesis for a year. I am a customer of the product.

Stability-wise, I rate the solution a nine out of ten.

Scalability-wise, I rate the solution a ten out of ten.

There are around three users of the product in my company.

The solution's technical support is flawless. I rate the technical support a ten out of ten.

Positive

One of the plus points of the product that makes it so much better is that it is very easy to set up, especially since it is managed by AWS, making it a smooth process.

The product falls on a bit of an expensive side.

There is a separate team in my company that looks after the real-time data analytics in our organization, which I don't know much about. As a DevOps engineer, I take care of the cloud, and because of this, I know why my company uses the product.

The most valuable feature of the product for our company's data processing needs stems from the fact that it operates in real-time. The simplicity of the services offered by the product and the way I can use them is very smooth and easy to understand. The tool also provides good storage and an increase in on-demand capacity, which I think is the best for our company.

Integrating Amazon Kinesis with other AWS services has smoothly helped our analytics workflow. It is also very easy to integrate with other AWS services. The analytics part is also very good with Amazon Kinesis.

I recommend the product to those who plan to use it. Those who are new to data streaming and want to start with a new product can go for Amazon Kinesis, as it is very easy to set up, especially the installation part, which is very easy to handle. Amazon Kinesis is the first option others should consider since it is easy to set it up.

I rate the tool an eight out of ten.

We had real-time streaming of data and a very large volume of user activity. We applied machine learning to the data streams. So, Kinesis basically made sure that we got the data, and we didn't lose the data.

I'm not using it at my current organization. I used it at the last company. Kinesis replaced a whole tier of servers. So, we didn't need to have a server to catch the data and then send the data somewhere else. Kinesis was the input port for very large amounts of data.

What turns out to be most valuable is its integration with Lambda functions because you can process the data as it comes in. As soon as data comes, you'll fire a Lambda function to process a trench of data.

One thing that would be nice would be a policy for increasing the number of Kinesis streams because that's the one thing that's constant. You can change it in real time, but somebody has to change it, or you have to set some kind of meter. So, auto-scaling of adding and removing streams would be nice.

I'd like to see the size of a Kinesis message go to at least one megabyte per message. That would be nice, but that's an extreme case.

I have been using it since it came out in 2014.

It is stable.

It is scalable. In the previous company, it was customer-facing. So, there were hundreds of thousands of users. There were very large data volumes.

In this company, we'll probably use Kinesis. We haven't used it yet. We have some more projects that'll come along, and then we'll need it.

I was there when it was beta. They were pretty good then, and they're still pretty good. So, they were five out of five. They were right on top of it.

Positive

We didn't use any other solution for streaming data. 

It was pretty straightforward. I had one person who did all DevOps. It did not need a dedicated person.

We had seen an ROI. We wouldn't have been able to build the product without it.

It was actually a fairly high volume we were spending. We were spending about 150 a month.

There were some that we considered, but it was a little cruder.

To someone who would like to implement it, I would simply tell not to shove giant bricks in. Data has to be reasonably sized. A single Kinesis message is measured in K, not megabytes. It's not meant for gigantic things. There is a different strategy for streaming data.

I'd rate it at least a nine out of ten. It was very close to perfect.

Amazon Kinesis is a queuing or buffering system that we use as a central place to buffer the incoming data we receive from the source. The actual destination is open-faced. Amazon Kinesis is used as a buffer in between to decouple the workload.

Amazon Kinesis's main purpose is to provide near real-time data streaming at a consistent 2Mbps rate, which is really impressive.

The solution currently provides an option to retrieve data in the stream or the queue, but it's not that helpful. We have to write some custom scripts to fetch data from there. An option to search for data in the queue can really help us in our day-to-day operations.

Since the solution is a buffer system, you write to it and read from it. The readers are called consumers. If you want to run multiple consumers reading from the queue, you have to enable the enhanced fan-out feature on Amazon Kinesis. This enhanced fan-out feature is quite costly.

There was a point when we had a huge budget increase in one week just because of the enhanced fan-out feature. This feature does not provide any special out-of-the-box functionality. Hence, we struggle to optimize multiple consumers reading from a single queue. We were charged high costs for the solution’s enhanced fan-out feature.

I have been using Amazon Kinesis for more than two years.

The solution is pretty good in terms of scaling. Amazon Kinesis has shards, which are the instances or units that the solution spins up for you. Depending upon your account quota, you can spin up as many shards as you want. You can even raise a request to increase that quota, which will be done sooner. Overall, Amazon Kinesis is a really scalable solution.

Our team, consisting of four to five people, uses the solution extensively in our organization.

We really struggle to get better support for Amazon Kinesis.

Amazon Kinesis is an expensive solution.

Amazon Kinesis is an AWS-managed service, just like S3 or EC. We don't have to deploy it; it is just there, and we spin it up. You must go to AWS' service page and click on Kinesis. Then, you can create it by clicking on Create and entering the name.

I would not recommend Amazon Kinesis to other users. Users can choose a cheaper alternative. They can use any other queuing system or in-house Kafka if they have a Kafka team. Amazon Kinesis provides near real-time read-and-write, but its cost is too high. Users can choose another option that provides the same functionality at less cost.

With Amazon Kinesis, you have to run a consumer who sees from Amazon Kinesis. AWS provides the Kinesis Client Library (KCL), which reads from the Kinesis stream. That library is also used in DynamoDB for data checkpointing. For example, if you have one day of data in Amazon Kinesis and started reading from 12 AM yesterday. The Kinesis Client Library (KCL) will check on the data in the DynamoDB. You get charged for the DynamoDB table out-of-the-box, along with Amazon Kinesis.

The DynamoDB table also costs a lot, which should not be the case. It is just read-and-write and is downloaded from the Kinesis Client Library (KCL). The DynamoDB table's cost should be very minimal, but that's not the case. The consumer is not optimal for efficient read-and-write, which further increases the cost. Both Amazon Kinesis and DynamoDB come into the picture.

Overall, I rate the solution a five or six out of ten.

We collect data from AWS IoT Core and then capture the stream in Amazon Kinesis. The data is then stored in S3 and shifted to Snowflake for analysis.

What I like about Amazon Kinesis is that it's very effective for small businesses. It's a well-managed solution with excellent reporting. Amazon Kinesis is also easy to use, and even a novice developer can work with it, versus Apache Kafka, which requires expertise.

My company found some Amazon Kinesis discrepancies, so it's looking forward to a more modernized solution from Apache Kafka.

One area for improvement in the solution is the file size limitation of 10 Mb. My company works with files with a larger file size.

The batch size and throughput also need improvement in Amazon Kinesis. The solution needs to be more open regarding the type of files for streaming and the streaming size. Amazon should not limit those aspects. It should be unlimited. If a company is ready to pay, why not make it unlimited?

What I want to add to Amazon Kinesis is modernization based on the container environment, where I can add containers and more workers. I also expect some human resources to be added and an SLA agreement with Amazon, if possible.

I've been using Amazon Kinesis for about one year, and I'm still using it.

Amazon Kinesis could be more stable. One of my clients rejected it, while some clients find it okay, stability-wise. I'd rate Amazon Kinesis stability as five out of ten.

I can rate Amazon Kinesis scalability according to the organization size and data load. For a small organization using the solution and Lambda with some transformation through AWS Glue, Amazon Kinesis is the best, scalability-wise. However, if you're dealing with a billion tuples, for example, the solution isn't as scalable, so I would go for Apache Spark or Apache Kafka to handle the load.

When I see that the processing takes longer than fifteen minutes with Lambda and the tenants fail, I use Apache Spark for processing, but that could take up to three or four days to be comparable to big data technologies.

I'd rate the scalability of Amazon Kinesis as four out of ten.

My company contacted some premium partners and technicians of Amazon Kinesis and found the technical support good, but with some limitations. I'd rate support a seven out of ten. Though it had limitations, the interaction with support was pleasant.

Neutral

In the future, my company plans to switch to Apache Kafka because it's very flexible and easier to manage. It's also easier to control and manage limits about topics. On the other hand, Amazon Kinesis has some limitations to its charts. It also has a 10 Mb limit to its file size, so if you have a 20 Mb file, you have to make it 10 Mb.

Amazon Kinesis is easy to set up, and it's a ten out of ten for me. Setting it up is a straightforward process.

My company set up Amazon Kinesis for the client.

If you ask a client about Amazon Kinesis pricing, the client usually says it's high. If you ask a business owner, the business owner would tell you that pricing for Amazon Kinesis is a little bit high. For each region, it's a little bit high.

There is a particular concern regarding Amazon Kinesis here in Pakistan because there's no zone in Pakistan. Amazon needs to develop zones here because Pakistan is the biggest country in the region after India. Amazon is losing a lot of business in Pakistan because there's no AWS zone here.

AWS also didn't accept my Pakistan credit card when I was trying to register with AWS. AWS should develop trust here in Pakistan and excellent AWS zones, so Pakistan businesses that want to purchase Amazon Kinesis won't need to depend on Singapore or India.

When I'm closing a deal with a new client, the client would ask, "Why do you need to sign up with a zone in India or Singapore to save data?" I don't have an answer to that question, so a workaround would be to develop on-premise environments for clients to save data.

Amazon Kinesis pricing is sometimes reasonable and sometimes could be better, depending on the planning, so it's a five out of ten for me.

Nowadays, my company works with AWS, Snowflake, Redshift, Amazon Kinesis, Firehose, Aurora, and Athena. In the future, my company plans to work with SAP HANA.

My rating for Amazon Kinesis is six out of ten.

My company is a user of Amazon Kinesis.

We are using Kinesis' third-party streaming engine. We are using the AWS cloud and are moving to Azure.

One of the best features of Amazon Kinesis is the multi-partition. 

Another valuable feature of Kinesis is that when it is down, and in the backup stage, the data is still available. 

Currently, Kinesis provides only seven days of retention support. It would be beneficial if this could be extended to upwards of 40 days or more. 

In the next future release, I would like to see a library that is Java-compliant. It would be beneficial if Amazon Kinesis provided document-based support on the internet to be able to read the data from the Kinesis site.

I have been using Amazon Kinesis for almost two years.

Amazon Kinesis is stable. I do not see any issues.

The solution is scalable. We have 20 team members using Kinesis.

We have not required support from Amazon.

The initial setup of Amazon Kinesis is easy. 

We have been looking for a streaming tool. We looked into Kafka, E-Hub, and Kinesis. Kafta is better than Kinesis as it has multiple cloud connectors. More features are available by default with Kafta.

Overall, I would rate Amazon Kinesis a seven out of ten.

I work as a senior software engineer in eCommerce analytics company, we have to process a huge amount of data.

Only a few people within our organization use Kinesis. My team, which includes three backend developers, simply wanted to test out different approaches.

We are now in the middle of migrating our existing databases in MySQL and Postgres, to Snowflake. We use Kinesis Firehose to ingest data in Snowflake at the same time that we ingest data in MySQL, without it impacting any performance.

If you ingest two databases in a synchronous way, then the performance is very slow. We wanted to avoid that so we came up with this solution to ingest the data in the stream.

We use Kinesis Firehose to send the data to the stream, which then buffers the data for roughly two minutes. Afterwards, it places the files in an S3 bucket, which is then loaded automatically, via an integration with Snowflake that's called Snowpipe. Snowpipe reads and ingests every message and every file that's in the S3 bucket. This stage doesn't bother us because we don't need to wait for it. We just stream the data — fire and forget. Sometimes, if the record is not ingested successfully, we have to retry. Apart from that, it's great because we don't need to wait and the performance is great.

There are some caveats there, but overall, the performance and the reality of it all has been great. This year, 100% of the time when there was an issue in production, it was due to a bug in our code rather than a bug in Kinesis.

We save a lot of time with Kinesis, but it's difficult to measure just how much. We actually have something similar regarding some other processes. We have developed somewhere else a tool that takes note of the contents of the stream, places them into a file, manually uploads them to the S3, and copies the files into Snowflake. That could be done with Kinesis, but it could take two weeks or 1 month less to get it production-ready.

The first would be the one found in the AWS SDK using the asynchronous client: put Record batch function which allows you to put a list of records in one put record request, which saves time and it's more efficient. Also, by using the asynchronous client, the records are sent in the background using an internal thread pool that can be configurable for your needs. In our performance testing, we came across this setting was the fastest solution. It didn't impact anything in the performance of the system process.

The second one would be the ability to link the stream to other places other than S3 via configuration of the stream and without changing a line of code.

Lastly, you can also link a lambda function to the stream to transform the data as it arrives in before writing it in S3, which is great to perform some aggregations or enrich the data with other data sources.

The default limit that they have, which at the moment is 5,000 records per second (I'm talking about Kinesis Firehose which is a specialized form of the Amazon Kinesis service) seems too low. Actually, on the first week that we deployed it into production, we had to roll it back and ask Amazon to increase the default limits.

It's mentioned in the documentation, but I think the default settings are far too low. The first week it was extremely slow because the records were not properly ingested in the stream, so we had to try it again. This happened the first week that we deployed it into production, but after talking with Amazon, they increased their throttling limits up to 10,000 records. Now it works fine.

We've been using this solution since September 2019.

The stability is great. I'd say that maybe we have it running 99% of the time, and nothing stops it.

Amazon Kinesis is definitely scalable. We have huge spikes of data that get processed around midnight and Kinesis handles it fine.

It automatically scales up and down, We don't need to compute it for that. It's great.

The only time that we needed to contact Amazon was to ask them to increase the throttling limit. They replied to us very quickly and did what we asked.

Initially, we were evaluating Kafka. I think Kafka is faster, but it's less reliable in terms of maintenance; however, when Kafka works, and you have it properly configured, it's much better than Kinesis, to be honest.

On the other hand, Kinesis provides us with better maintenance. Our DevOps team is already oversaturated, so we didn't want to increase the maintenance cost of the production environment. That's why we decided to go with Kinesis; because performance-wise, it's easy to configure and maintain.

I found this solution to be really easy to configure. The essential parts of the configuration include naming the stream and also configuring the buffering time that it takes for a record to get ingested into S3 (how long it will be in the stream until it's put into an S3). You also need to link the Amazon S3 buckets with the Amazon Kinesis stream. After you've completed these configurations, it's pretty much production-ready. It's very, very easy. That's a huge advantage of using this service.

Deployment took a few minutes.

You don't need a deployment plan or an implementation strategy because once you configure it, you can just use a stream. It's not an obligatory version that needs a library, etc. This stream is completely abstract in that way. You only need to configure it once, that's it.

We have seen a return on our investment with Amazon Kinesis. We are able to process data without any issue. It's our solution for ingesting data in other databases, such as Snowflake. 

Developing the stream process manually or using Kafka

If you want to use a stream solution you need to evaluate your needs. If your needs are really performance-based, maybe you should go with Kafka, but for near, real-time performance, I would recommend Amazon Kinesis.

If you need more than one destination for the data that you are ingesting in the stream, you will need to use Amazon Kinesis Data Streams rather than Firehose. If you only want to integrate from one point to another, then Kinesis Firehose is a considerably cheaper option and is much easier to configure. 

From using Kinesis, I have learned a lot about the synchronous way of processing data. We always had a more sequential way of doing things. 

On a scale from one to ten, I would give this solution a rating of eight.