GoDaddy Inc. is an American publicly traded Internet domain registrar and web hosting company headquartered in Scottsdale, Arizona and incorporated in Delaware. As of August 2020, GoDaddy has approximately 20 million customers and over 7,000 employees worldwide. GoDaddy is empowering everyday entrepreneurs around the world by providing all of the help and tools needed to succeed online. With 20 million customers worldwide, GoDaddy is the place people go to name their idea, build a professional website, attract customers and manage their work. GoDaddy’s mission is to give their customers the tools, insights and the people to transform their ideas and personal initiatives into success.
GoDaddy utilizes an 800-node Apache Hadoop cluster to hold over 2.5 petabytes of customer-related activity and behavior data. This on-premises data lake is critical for guiding business operations and determining the company’s investment strategies. The system is in operation 24x7. It can generate peak loads of more than 100,000 file system events per second, with sustained 12 hour periods processing an average of over 21,000 change operations every second.
The challenge for GoDaddy was how to migrate petabytes of actively changing, “live” data when the business depends on the continued operation of applications in the cluster and access to its data. Any disruption to business operations would be unacceptable and may have prevented a migration from even being attempted
GoDaddy used WANdisco’s LiveData Migrator to migrate data from their actively used cluster to AWS S3. LiveData Migrator performs a single scan of the source datasets and processes the ongoing changes that occur to achieve a complete and continuous data migration. It does not impose any cluster downtime or disruption to production applications and requires no changes to cluster operation or application behavior. LiveData Migrator enabled GoDaddy to perform their migration without disrupting business operation, and ensured that datasets were transferred completely, even while under active change in a very large and busy Hadoop environment.
GoDaddy, being a technically oriented company with deep software development skills, often builds their own solutions. As such, they investigated building their own custom migration solution leveraging open source tools. However, it was deemed that performing the initial migration and ongoing synchronization manually is a complex, error-prone task, and not the core competency on which they wanted their highly skilled engineers to spend their time. Instead, following a quick demonstration of a 2TB migration, and a subsequent 10TB proof-of-concept GoDaddy selected WANdisco LiveData Migrator to automate the migration. LiveData Migrator combines a single scan of the source datasets with processing of the ongoing changes that occur to achieve a complete and continuous data migration. It does not impose any cluster downtime or disruption, and requires no changes to cluster operation or application behavior
“At GoDaddy, deep technical knowledge is in our DNA, and we often build applications in-house to support growth. In the use case of a Hadoop to Amazon S3 data migration and replication, we found WANdisco’s LiveData Migrator to be the optimal approach to deliver the best time to value, rather than running a more time-consuming and costly manual migration project internally.”
Wayne Peacock, Chief Data and Analytics Officer, GoDaddy
The project resulted in a successful migration with the following outcomes:
Successful migration of initial 500TB data subset with zero business disruptionWith LiveData Migrator GoDaddy was able to complete the 500TB migration, while maintaining business continuity and ensuring all data and changes were migrated successfully.
Reduced cost and risk of custom data migration development.
The automated data migration reduced the cost and risks associated with manual and custom data migration initiatives, and enabled GoDaddy’s engineers to focus on high value tasks, such as analytics, AI and machine learning development.
Faster time-to-value for AWS services
LiveData Migrator enabled GoDaddy to establish the new AWS environment much more quickly than would have otherwise been possible. This allows GoDaddy to focus on the new cloud solution and ensure they meet their objectives of strengthening their platform, increasing their pace of experimentation, and accelerating delivery of their product to provide increased value to customers and financial outcomes to their shareholders