If you were talking to someone whose organization is considering erwin Data Intelligence (DI) for Data Governance, what would you say?
How would you rate it and why? Any other tips or advice?
We haven't integrated Data Catalog and Data Literacy yet. Our client is a little bit behind on being able to utilize these aspects that we've presented for additional value. My advice would be to partner with an integrator. erwin has quite a few of them. If you're going to jump into this in earnest, you're going to need to have that experience and support. The biggest lesson I have learned is that the only limitation is the imagination. Anything is possible. There's quite a strong capability with this product. I've seen what you can come up with as far as innovative flows, processes, automation, etc. It's got quite strong capabilities. The next lesson would be in regards to how automation fits within a company's framework and to embrace automation. There are some good quality points to continue with, certainly within the data cataloging, data governance, and so forth. There's quite a bit of good capability there. I rate erwin Data Intelligence for Data Governance a nine out of ten.
It is a different experience. Collaboration and communication are very important when you want to harvest the value from the humongous amount of data that you have in your organization. All these aspects are soft aspects, but are very important when it comes to getting value from data. Data pipelines are really important because of the kinds of data that are spread across different formats, in differing granularity. You need to have a pipeline which removes all the complexities and connects many types of sources, to bring data into any type of target. Irrespective of the kind of technology you use, your data platform should be adaptive enough to bring data in from any types of sources, at any intervals, in real-time. It should handle any volume of data, structured and unstructured. That kind of pipeline is very important for any analysis, because you need to bring in data from all types of sources. Only then you can do a proper analysis of data. A data pipeline is the heart of the analysis. Overall, erwin DI is not so costly and it brings a lot of unique features, like metadata hooks and metadata harvesters, along with the business glossaries, business to business mapping, and technology mapping. The product has so many nice features. For an organization that wants to realize value from the potential of its data, it is best to go with erwin and start the journey.
Erwin currently supports two implementations of this product: one on a SQL Server database and the other on an Oracle Database. It seems that the SQL Server database may have fewer complications than the Oracle Database. We chose to implement on an Oracle Database because we also had the erwin Data Modeler and Web Portal products in-house, which have been set up on Oracle Databases for many years. Sometimes the Oracle Database installation has caused some hiccups that wouldn't necessarily have been caused if we had used SQL Server. We are not currently using forward engineering capabilities of the Data Intelligence suite. We do use erwin Data Modeler for forward engineering the data definition language that is used to change the actual databases where the data resides. We are currently using the Informatica reverse smart connector so that we can understand what is in the Informatica jobs, jobs which may not have been designed with, or have, a source-to-target mapping document. That's as opposed to having a developer create data movement without any documentation to support it. We look forward to potentially using the capability to create Informatica jobs, or other types of jobs, based on the mapping work, so that we can automate our work more and decrease our delivery time and cost to deliver while increasing our accuracy of delivery. We've learned several lessons from using erwin Data Intelligence Suit. One lesson is around adoption. There will be better adoption through ease of use. We do have another product in-house and the largest complaint about that product is that it's extremely difficult to use. The ease of use with the Data Intelligence Suite has significantly improved our adoption rate. Also, having all of the information in one place has significantly improved our adoption and people's desire to use the tool, rather than looking here, there, and everywhere for their information. The automated data lineage and impact analysis being driven from the mapping documents are astounding in reducing the time to research impact analysis from six to 16 weeks down to minutes, because it's a couple of clicks with a mouse. Having all of the information in one place also improves our knowledge about where our data is and what it is so that we can use it in the best possible ways.
erwin is very good for companies who have a very structured, data governance process. It puts every possible tool around a company's data. This is very good for companies who are mature with their data. However, if a company is just looking for a tool to showcase their data in a data catalog, then I would advise companies to be careful because erwin is sometimes really complex to master and configure. Once it is set up, you have to put your hands in the gears of the software to model how your data works. It is more of a company process modeler than a directory of all data available you need and can access. Industrial companies over 30 to 40 years in age are struggling to find what data they may have and it may prove to be difficult for them to use erwin directly. What we have done with the lineage is valuable, but manual. For the IT dictionary, automation is possible. However, we have not installed the plugin that allows us to do this. Right now, all the data that we have configured for the lineage has been inputted by hand. I would rate this feature as average. We have not tested the automation. I would rate this solution as seven (out of 10) since we have not experienced all the functionalities of the product yet.
If you have the ability to pull a steering committee together to talk about how your data asset metadata needs to be used in different processes or how you can connect it into mission-critical business processes so you slowly change the culture, because erwin DI is just part of the processes, that probably would be a smoother transition than what I am trying to do. I'm sitting in an office by myself trying to push it out. If I had a steering committee to help market or move it into different processes, this would be easier. Along the same lines as setting up an erwin Workgroup environment, you need to be thoughtful about how you are going to name things. You can set up catalogs and collection points for all your physical data, for instance. We had to think about if we did it by server, then every time we moved a server name, we'd have to change everything. You have to be a little careful and thoughtful about how you want to do the collections because you don't want the collection names to change every time you're changing something physically. What we did is I set up a more logical collection, so crossing all the servers. The following going into different catalogs: * The analytics reporting data sets * The business-purchased applications * External data sets * The custom applications. I'm collecting the physical metadata, and they can change that and update it. However, the structure of how I am keeping the data available for people searching for it is more logically-focused. You can update it. However, once people get used to looking in a library using the Dewey Decimal System, they don't understand if all of a sudden you reorganize by author name. So, you have to think a bit down the road as to what is going to be stable into the future. Because the more people start to get accustomed to it being organized a certain way, they're not going to understand if all of a sudden you pull the rug out from under them. I'm going to give the solution an eight (out of 10) because I'm really happy with what I've been able to do so far. The more that the community uses this tool, the more feedback they will get, and the better it will become.
Our first goals were data literacy and data as an asset. Those were our two big, ultimate goals three years ago. Data literacy turned out to be 10 times more important than a data warehouse. We could look at existing data sets and, just by educating people, it gave them an advantage almost immediately. The fact that the data governance was able to put a framework around data literacy helped us focus on the right answer, even if it wasn't the first one given. In other words, sometimes we'd have the same answer three or four times, and it would shift until we nailed it. But without governance, we would never have done that and we would have stayed the same. The secret to the success of this project was that we had a vision and we stuck to it. Governance was important to us, no matter how other people might have thought about it. In my very first data steward meeting I was introducing everybody to these brand-new terms they'd never seen, and someone in our analytical group totally derailed the meeting. So be aware that it's not going to be easy, but have a vision. And make sure that governance is important or don't bother. It's not something that a lot of people add value to at all. When you say, "Oh, I want governance so we can have a data dictionary and you can go look at it," they'll say, "I don't want to look at it, just give me a report." But the ability for those who need to do that is huge. Have a vision and stick to it and be willing to take a step back, sometimes, to go two forward. The neat thing is that we've pretty much done all of this with two to three people, for our entire organization. We do have three data teams that are using the Modeler for development — ETL SSIS stuff — but we have a pretty serious "wash, rinse, repeat" standard. If anything is in doubt, we just go back to the business rules and see what our rules are. What are our principles, and are we meeting them? As far as automating the changes through the environments, it has helped, but not a lot. It's not like it was a silver bullet. We need help there, because there's so much. There's the model, but once you promote that in different environments, sometimes you miss it because you only get three or four days to get out of QA to get it into stage. Obviously, you mitigate risks with automation. It does have an impact. As a company, we just haven't been able to take full advantage of it now, but that's our hope. We're only into it for about a year-and-a-half, even though we have run with the suite for almost three years. We're still immature. I wish I had everything at the push of one button, the "Easy" button. Some of it's over our heads. We could use some new training and we could use some additional support. erwin has been great with us, but it's also a matter of the appetite and the resources. The biggest issue is that I don't have a team of people doing what a team of people need to do to accomplish what we would like to. It's done by a small number of people on a consistent basis, and not full-time. The solution's generation of production code through automated code engineering would reduce the time it takes to go from initial concept to implementation, but we're a Microsoft shop and most of all that is done inside TFS or Visual Studio. That's how we manage all our codebase, including release management. That's all done separately and is automated. We're trying to create some interfaces between the two. We just haven't gotten there. In my three years using erwin, besides actually getting approval for the money to purchase the software, I don't think I've had a struggle with it. They've been great. When we first got on and we had some questions, they got me to the development team in England and set it all up with us without question, no extras. They just tried to make sure it worked. I would rate erwin DI for Data Governance at eight out of 10. I never give a 10 because I have yet to see perfection. It has some gaps, but I definitely think it's in the top third. As far as rates go, I don't have a lot to compare it with. It's easy now but it took going through a learning curve, but that's the case with any software. Does it need to mature a little? Possibly. But that would be it. With their roadmap, they're buying companies, and changing things, and doing things. I've been pleased.
I learned how to automate in the data area and how this is very different from any CI/CD development platforms that I was working on before. I learned that we need totally different things to automate properly in the data area. We need very accurate metadata. We need precise mappings reviewed by different data stakeholders. I would rate this product as an eight (out of 10). I can imagine some capabilities for this product that would make it even better.
What do you like most about erwin Data Intelligence (DI) for Data Governance?
Thanks for sharing your thoughts with the community!