Talend has different modules. Talend has Talend Data integration (DI), Talend Data Quality (DQ), Talend MDM, and Talend Data Mapper (TDM). We have Talend DI, Talend DQ, and TDM. Our use cases span across these modules. We don't use Talend MDM because we have a different solution for MDM. Our EDF team is using an Informatica solution for that.
We have a platform that deals with MongoDB, Oracle, and SQL Server databases. We also have Teradata and Kafka. The first use case was to ensure that when the data traverses from one application to another, there is no data loss. This use case was more around data reconciliation, and it was also loosely tied to the data quality.
The second use case was related to data consistency. We wanted to make sure that the data is consistent across various applications. For example, we are a healthcare company. If I'm just validating the claim system, I need to see how do I inject the data into those systems without any issues.
The third use case was related to whether the data is matching the configurations. For example, in production, I want to see:
- If there is any data issue or duplicate data?
- Is the data coming from different states getting fed into the system and matching the configurations that have been set in our different engines, such as enrollment, billing, and all those things?
- Is it able to process this data with our configuration?
- Is it giving the right output?
The fourth use case was to see if I can virtually create data. For example, I want to test with some data that is not available in the current environment, or I'm trying to create some EDA files, which are 834 and 837 transaction files. These are the enrollment and claims processing files that come from different providers. If I want to test these files, do I have the right information within my systems, and who can give me that information.
The fifth use case was related to masking the information so that in your environment, people don't have access to certain data. For example, across the industry, people pull the data from production and then just push it into the lower environment and test, but because this is healthcare data, we have a lot of PHI and PII information. If you have your PHI and PII information in production and I am pulling that data, I have everything that is in production in the test environment. So, I know your address, and I know your residents. I can hack into your systems, and I can do anything. This is the main issue for us with HIPAA compliance. How do we mask that information so that in your environment, people don't have access to it?
These are different use cases on which we started our journey. Now, it is going more into the cloud, and we are using Talend to interact with various cloud environments in AWS. We are also interacting with Redshift and Snowflake by using Talend. So, it is expanding. We are using version 7.1, and we are migrating to version 7.3 very soon.