2018-05-07T10:36:00Z
it_user434868 - PeerSpot reviewer
Senior Director of Delivery at a tech services company with 51-200 employees
  • 0
  • 53

What is your primary use case for Spring Cloud Data Flow?

How do you or your organization use this solution?

Please share with us so that your peers can learn from your experiences.

Thank you!

3
PeerSpot user
3 Answers
Mohammad Masudu Rahaman - PeerSpot reviewer
Founder at Talkingdeal.com LLC
Real User
2020-10-21T04:33:51Z
Oct 21, 2020

Mostly the use cases are related to building a data pipeline. There are multiple microservices that are working in the Spring Cloud Data Flow infrastructure, and we are building a data pipeline, mostly a step-by-step process processing data using Kafka. Most of the processor sync and sources are being developed based on the customers' business requirements or use cases. In the example of the bank we work with, we are actually building a document analysis pipeline. There are some defined sources where we get the documents. Later on, we extract some information united from the summary and we export the data to multiple destinations. We may export it to the POGI Database, and/or to Kafka Topic. For CoreLogic, we were actually doing data import to elastic. We had a BigQuery data source. And from there we did some transformation of the data then imported it in the elastic clusters. That was the ETL solution.

Search for a product comparison
Saket Puranik - PeerSpot reviewer
Senior Platform Associate L2 at a tech services company with 10,001+ employees
Real User
2020-10-19T09:33:41Z
Oct 19, 2020

In my last project, I worked on Spring Cloud Data Flow (SCDF). We created a stream using this product and we had a Spring Kafka Binder as well. The project included creating a data lake for our clients. The platform that we created maintained a data lake for an internet banking user and provided an out-of-the-box solution for integration with it. We used SCDF to gather the data, as well as our ETL (extract, transform, and load) pipelines.

Ken Ramirez - PeerSpot reviewer
Senior Digital Architect at Kubota USA Inc.
Real User
2018-05-07T10:36:00Z
May 7, 2018

The organization I’m currently consulting for is performing a lift-and-shift, moving its existing software from an on-prem platform and infrastructure to the cloud. They have chosen Azure as their cloud provider. As part of this process, they have orders to move away from expensive, monolithic, proprietary software platforms, and to replace them with open-source, publicly available software technologies. One area we’ll be replacing is their current middleware software which consists of IBM WebSphere Message Broker. While it is a fine tool for the most part, it’s also bulky and expensive to operate. The final solution we’re working towards will be much more cloud-native, will support scalability, be able to process messages much faster, and consist of several different technologies and vendors (not just a single vendor, as is the case with the current IBM solution). This new middleware platform will consist of: Apache Kafka for the delivery of messages, Spring Cloud Data Flow, and a handful of RESTful APIs.

Find out what your peers are saying about VMware, StreamSets, TIBCO and others in Data Integration Tools. Updated: November 2022.
653,757 professionals have used our research since 2012.
Related Questions
KS
User
Oct 4, 2021
Hello community members, Which solution (Spring Cloud Data Flow or TIBCO BusinessWorks) is better and why? Thanks in advance!
See 1 answer
Evgeny Belenky - PeerSpot reviewer
Director of Community at PeerSpot (formerly IT Central Station)
Oct 4, 2021
Hi @Mohammad Masudu Rahaman, @Saket Puranik, @MahmoudAbu-Ghali and @Fabio Ferri, Can you chime in here to share your experience and expertise?​
Miriam Tover - PeerSpot reviewer
Service Delivery Manager at PeerSpot (formerly IT Central Station)
Oct 21, 2022
Please share with the community what you think needs improvement with Spring Cloud Data Flow. What are its weaknesses? What would you like to see changed in a future version?
2 out of 3 answers
Saket Puranik - PeerSpot reviewer
Senior Platform Associate L2 at a tech services company with 10,001+ employees
Oct 19, 2020
Some of the features, like the monitoring tools, are not very mature and are still evolving. With some of the products we used, they did not integrate well and were hanging a lot. One of the advantages of using open-source is that if you don't like a particular tool then you can use another one. If you want to use Kubernetes then you have to optimize a lot in terms of resources. I had a 15 GB MacBook Pro, but initially, it wouldn't work because it would hang. There were also some weird shutdowns. We weren't able to figure out exactly why it happened but it was clearly due to having not enough system resources. When then needed to optimize and increase our heap memory.
Mohammad Masudu Rahaman - PeerSpot reviewer
Founder at Talkingdeal.com LLC
Oct 21, 2020
The configurations could be better. Some configurations are a little bit time-consuming in terms of trying to understand using the Spring Cloud documentation. The documentation on offer is not that good. Spring Cloud Data Flow documentation for the configurations is not exactly clear. Sometimes they provide some examples, which are not complete examples. Some parts are presented in the documentation, but not shown in an example code. When we try to implement multiple configurations, for example, when we integrated with PCF, Pivotal Cloud Foundry, with CDF, there were issues. It has workspace concept, however, in a CDF when we tried to implement the workspace some kind of boundary configuration was not integrating properly. Then we went to the documentation and tried to somehow customize it a little bit on the configuration level - not in the code level - to get the solution working. It is open source. Therefore, you need to work a little bit. You need to do some brainstorming on your own. There's no one to ask. We cannot call someone and ask what the problem is. It is an open-source project without technical support. It's up to us to figure out what the problem is.
Download Free Report
Download our free Data Integration Tools Report and find out what your peers are saying about VMware, StreamSets, TIBCO, and more! Updated: November 2022.
DOWNLOAD NOW
653,757 professionals have used our research since 2012.