What is our primary use case?
Qlik Compose is basically an integration tool, which has been acquired by Qlik from an Israeli IT company. So that Qlik can become leaders or can jump into the integration space.
So, there are two tools. One is Qlik Replicate, which replicates the whole data. And then after the replication is done, Qlik Compose is primarily designed because if users use Qlik Replicate, it will replicate the data. For example, if users have ERP data in entity relationships, users can offload it instead of building ETL jobs over the same system. Users can offload that kind of load by pulling the data from the Replicate system. Users are not building a data warehouse from that.
So, Compose will come into the picture from the Qlik stack point of view, which will help users automate it quickly. Users need to define the relationships between different tables of what is present in the OLTP system. Based on that, it will automatically design the dimension. It will automatically design the fact, and it will automatically design the relationships. And it will create the table just like what users do in Erwin. Erwin, it is like users define the relationships and an SQL query is generated. But here, it’s 60% automation, where users define the relationship, and then automatically, it will even identify the dimension. Then, after identifying the dimension attributes and what needs to be in that, it will also generate the call for it. That is the data modeling part. So, the advantage is that data modeling is automated.
And then users will have something like slowly changing dimensions, later having dimensions, similar pattern jobs in the dimension, and fact ETL process that users need to develop. That, again, is time-consuming. So even that is automated. Usually, that can be automated in Compose.
So overall, what we claim from Qlik Replicate is that 60% of the process can be automated. Users have the data modeling effort, where users manually define relationships and put in some effort, and then it's automated. Even in the ETL process, users manually define some connections, map the attributes together, and specify what they need. After that, the rest is automated. So, 60% of the time is spent there.
How has it helped my organization?
Qlik Compose plays a key role after users’ve purchased Qlik Replicate. Once replication is done from one system to another, but the data warehouse isn't in place, that's when users start using Qlik Compose. It pulls in tables from the source system and allows to define relationships between those tables. Once this is done, Compose will automatically create dimensions—this is part of the data modeling process.
After the tables are created, the next step is data integration. Data integration can involve developing jobs, and even data modeling is considered a part of data integration. Essentially, data integration is done from the replicated ERP system, which is an entity-relationship (ER) model. Once the data is replicated, users can decide how to compose the dimensions.
From there, users define the relationships, specify details, and set up ETL jobs. Users might deal with data-driven dimensions, slowly changing dimensions (SCD) types 1, 2, or 3. Users simply drag and drop the source and target tables, and Qlik Compose will automatically generate the required code and complete the integration process.
I haven't implemented this fully myself, but I’ve learned it from a pre-sales perspective and demonstrated it once or twice. If I had implemented it over a year, I would have hands-on experience with everything, but I understand the automation of data modeling and ETL jobs well enough to explain it to customers.
What is most valuable?
Qlik Compose is something that will automate user's overall data modernization. Here data modernization includes data modeling, ETL jobs, etc.
But the advantage is users can automate the overall process of data engineering and data modeling through Qlik Compose. I think that's useful when users are able to manage 60% of the workload automated. That will be very useful. That's fantastic.
Replicate does not have a great AI capability. AI capabilities are present in Qlik Sense.
Qlik Replicate is a very light tool. It is only meant to capture data from the log files, get the data, and transfer it, read that table structure, create the table structure, and transfer the data whenever there is a change. So, it basically integrates with the kernel of the operating system.
The way it works is that these replicate tools will integrate with the kernel of the operating system, and they will access the redo log files of the database. The redo log should have access to all the files of the structure of the schema, too. So, using that technique, they redo all the data structures, create a similar structure, and replicate the structure in the target schema, table, and database. After that is done, it will start tracing the instances that are happening.
For example, if data is inserted into the table, then an insert is fired on the statement on the table. So, that particular insert is captured. And based on that insert statement, it will pull the SQL query and say, "Okay, there is an insert. I need to get that data." It will get the data from the redo log itself rather than going to a database. Then, it will just pass that transaction into the target system, where it will just insert the data.
And this happens instantaneously, within a microsecond. So, if there is an insert, an update, or a delete, everything is transferred immediately. It is picked from the redo log because it comes to the redo log, and then the redo log sends it to Qlik Replicate and Replicate to the target system on which Replicate is installed.
What needs improvement?
The disadvantage is, I think, people are not going for this license because it is not marketed properly.
Qlik was not promoting it because Talend was acquired at the same time. So Talend has become their primary product. Compose was not being sold much. The other reason is there is a tool called Qlik Cloud Version, which is a combination of Compose and Qlik Replicate. So, Compose was not promoted much by Qlik. So, I started to concentrate on Talend more. This was an affair of around two or three months.
For how long have I used the solution?
I used it for a year, from 2023 to 2024. I practiced it, but I have not implemented it for any customer as a project.
What do I think about the stability of the solution?
I have not come across any latency concerns raised by any customer because I’ve not implemented it. But latency issues, I have not heard from any customer. And what I understand is it is pretty smooth.
Overall, the load and the volume it can handle were pretty high. Qlik Replicate and Compose have handled even a very large number of transactions within a few fractions of a second without any glitches. It was good. So, latency is never an issue.
What do I think about the scalability of the solution?
Qlik tools can be scaled based on the source load. What happens when the data transfer happens is that the data resides in the RAM of the tool. So, what users need to do is increase the RAM for the tool so that data does not reside in the RAM for more than a few seconds. It just moves fast.
Once it is on the disk, the performance will definitely be slow. If it is only on the cache and on the memory, the performance would be better. If it is on the database, it would be best. However, when data transfer happens, some movement of data has to be done through memory. But if that memory is not enough, then it will spill to the disk. So scaling out that memory is not a challenge.
So, scaling is possible. It requires some effort to scale the solution.
How are customer service and support?
There is a proper product support team. I heard that with the new tools like Qlik AutoML, there is a delay in the technical support from the product support team. We raised it with the product support team for some AutoML issues, which were not working as expected in Qlik Sense. The response was not so great, so we needed to escalate it to someone else to get it done.
The support is not that great when it comes to Qlik AutoML. I’m not sure about Replicate. My experience with the AutoML issue was not good. So, it is not as great as Informatica or Oracle in the way we get the support. It is not mature.
How would you rate customer service and support?
Which solution did I use previously and why did I switch?
I used to use Informatica, Databricks, data modeling, and other ETL tools while I was working as an architect and project manager. When I was playing these roles. And since the last year, I started to play a different role. I started to play the role of presales engineer, presales manager. So I did not use these tools, but I used data integration tools like Qlik Talend and Qlik, from a solution architect and presales manager point of view.
Qlik Compose, Qlik Replicate, Qlik Talend, all these tools I’ve worked with to demonstrate and present to customers and propose them for their modernization.
So, I used not only Qlik Compose, but also Qlik Sense, Qlik AutoML, Talend, and several components of Talend. I’ve worked with AWS and Azure, though not extensively. I learned enough to demonstrate how these tools work because developing a product end-to-end for a customer is a different experience.
For example, when doing a data integration with Compose, there might be issues with system readiness, data types, integration failures, and so on. These challenges provide a much deeper learning experience. However, in my demonstrations, I worked with pre-configured demo systems with sample data, so I didn’t encounter any errors. As a result, my learning from implementation is minimal. My focus has been on demonstrating how it works to customers.
How was the initial setup?
I’ve not implemented it, but I have used it to present to customers and provide solutions across landscapes from a presales point of view.
But for the trial version, we were able to deploy it easily. I think we were able to deploy it. I gave my DB to deploy it. He deployed it. But, I have not deployed it personally.
It would definitely need maintenance. For example, if I’m deploying a product in a customer environment where everything is automated and ready, then definitely, if Compose is going through upgrades, it needs upgrades.
Upgrades would definitely have to be made to the system wherever it is deployed. But from a normal maintenance point of view, I think that’s a very good point. It does not need maintenance. Everything has been automated. Unless your source system is changing, it doesn’t need any maintenance.
If your source system has changed, meaning your tables have changed or something has changed, I need to check this point.
If it can automatically recognize the tables, structures, or anything that has been changed, consume, absorb it, and then replicate that change into the target system from a data model, data integration point of view, and within data integration, even in the ETL process. So that’s a very good point you got.
From maintenance, I think there are two aspects. If the product itself has changed, the changes have to happen. But, again, if the product has not changed, but still, the source systems from where it is picking the data, the tables or anything has changed, how does it pick it up?
Replicate does it automatically. I remember that.
What's my experience with pricing, setup cost, and licensing?
Licensing is core-based for Qlik Replicate. Based on the cores that you have in the source system and the target, you’ll identify the license for it. If the source system has many cores, then the load that Replicate has to use to pull the data from the source system would be high. So, the licensing would be decided based on the cores.
Generally, the advantage of Compose or the purpose of Compose will come into the picture because, in the complete stack of Qlik integration, Replicate is doing just one job of replicating data from an existing OLTP system to another OLTP system. Whereas Compose has to create from this OLTP system an OLAP system, which is a data warehouse. So, that makes a complete stack of Qlik integration, which is advantageous for the customer. They see that this tool is taking away the load from my source system. It’s not loading my core banking system, which has hundreds or thousands of queries per second running on it. My transactions are not getting affected, and I don’t want my sales or banking to be impacted if my data warehouse has to be loaded in real-time. So, what I do is replicate the data using the log files of the system. I pull the data without affecting the database at all. After that, Compose creates a data warehouse automatically. This is very interesting to the customer and very advantageous or profitable.
Which other solutions did I evaluate?
Informatica is a totally different tool altogether. It is a data integration tool that can handle huge volumes of data and transform huge volumes of data.
Talend is similar to a tool that can handle huge volumes of data, but it is more of a multi-skilled, multi-layer tool that has not only RDBMS integration but also big data integration. It can be on-premise, multi-cloud, or hybrid. It has many advantages. It’s a totally different tool. So, Informatica can handle huge volumes of transformation. So, these two are huge integration tools.
Replicate is a very, very light tool that is only used to transfer data from one system to another. It cannot handle transformations. It can only do minor transformations, like addition or subtraction. A few of these calculations can be done on its machine and then transferred to the data. Or you can add a new column to it, put some data in that column, and then transfer it. But it is not used for transformation. Compose is used for transformation. That’s why Compose, Qlik will come into the picture. Whereas Qlik Replicate is very light. The purpose of the tool is different.
Informatica and Talend are integration tools. Qlik Compose, and Qlik Replicate are replication tools that will only replicate data from here to there to reduce the load on the core system that is running. Informatica will connect to the core system and increase the load by firing the queries on it. So, Replicate takes away the load by not firing the queries, and then it will go to the log and get the data. And it is a very light tool. It is an agentless tool. So we cannot compare both of them. In terms of support, I didn’t have that proper experience because Qlik Replicate and Qlik AutoML are all evolving tools.
What other advice do I have?
Overall, I would rate it an eight out of ten.