Databricks migration

BobAmy - PeerSpot reviewer

Project Description

Migrate our data lake house from Cloudera to Databricks.  My role is to extract the metadata and merge in the existing enriched metadata (manually entered business descriptions and categorization.  I am also involved in the DQ migration/improvement efforts, and setting up the support and ongoing maintenance policies and procedures.


Lessons Learned

Start sooner LOL
Never move to Cloudera in the first place.
I spend a lot of time trying to get ahead of the teams and put in processes to do it right the first time.  The hard part is what is right when you are embarking on a new technology.  Can you find experts that will teach and not do.


Highlights

Support from colleagues

Difficulties

Steep learning curve
Large no. of people impacted
Hard to meet schedule