Cloud Data Migration

Since 2010 we have seen Hadoop become an essential part of the data management landscape. Over the decade the majority of organizations have adopted Hadoop to build out their data lake infrastructure. However, while Hadoop offered a cost-effective way to store petabytes of data across a distributed environment, it introduced many complexities. The systems required specialized IT skills and the on-premises environments lacked the flexibility to easily scale the systems up and down as usage demands changed.

The management complexity and flexibility challenges associated with on-premises Hadoop environments are much more optimally addressed in the cloud. Not only do cloud object stores provide enhanced scalability, elasticity, and manageability at lower costs, they provide many other features that are automatically inherited by their other service offerings. For example, Amazon S3’s built-in encryption and automatic storage of data in three availability zones, and Microsoft’s Azure Machine Learning Services, which runs models directly in ADLS instead of pipelining data. These capabilities are making cloud based object storage the most widely deployed technology as identified in a recent Gartner survey.