Stage 2: Warming Up to the Technology Stack

Next up, bring in the technology stack for people to familiarize themselves with. Build a playground or a dirty development environment where developers and analysts can experiment and innovate without the fear of bringing down the business. This will allow the Data Modelers and DBAs to build the most optimal warehouse as well as enable the ETL developers to learn the pit falls. This will ensure they build the best practices before heading into full-fledged project work.

Stage 3: Converting the Old to New

A key element of Hadoop Adoption is running an efficient conversion of the old to new. Identify early on the dark or missing data elements, build coding standards and optimization techniques, automate as much as possible to reduce the conversion errors and validate against the legacy the correctness of the conversion. Bitwise recommends a Proof -> Pilot -> Production path to conversion where we nibble away at the legacy applications and build a repeatable framework before biting a big chunk of business requirement.

Stage 4: Maintenance and Support

Once in production, Hadoop needs what every production system in the world needs – maintenance and support. Things breakdown and undergo upgradation or deprecation. What is needed is a dedicated team to keep track of the Hadoop ecosystem. Besides the regular application production, a support team structure is required to ensure availability and reliability of the environment.

Backed by extensive experience and having worked with Fortune 500 companies, we at Bitwise have ensured a walk in the park for our clients adopting Hadoop and have unraveled effective usage of Hadoop to meet their ELT and Analytics needs. Have a look at our Excellerators and get to know how organizations worldwide are unlocking the real value of Hadoop with our proven methodology.

A variety of data sources and objectives are supported by ETL tools.

Databases, cloud storage, APIs, and files are just a few examples of the numerous data sources and objectives available to modern businesses. ETL solutions may readily connect to these systems using standardized protocols and APIs because they are made to function with a wide variety of data sources and targets. Data from various sources can be more easily integrated since ETL systems also offer the necessary transformations to change the data’s format.

ETL software offers a complete data integration solution.

Data extraction, data transformation, and data loading are all handled by ETL technologies, which offer a comprehensive solution for data integration. Additionally, these solutions provide several features for resolving errors, validating data, and managing data quality.

ETL solutions are therefore an all-in-one data integration solution, making them perfect for large-scale data integration projects.

In some situations, ETL tools perform better than ELT.

In a more recent method of data integration called ELT (Extract, Load, Transform), the data is first loaded into the target system before being transformed. Even though ELT has grown in acceptance recently, ETL is still preferred in some circumstances. For instance, ETL can offer greater performance if the data source is big since it can filter, combine, and transform the data at the source. As a result, processing times are sped up because fewer data needs to be put into the target system.

Other data integration methods, such as data streaming and data virtualization, can also be integrated with ETL tools. ETL tools, for instance, can be used to load data from a legacy system into a data warehouse. At the same time, real-time data from the same system can be obtained using data streaming and integrated with the data warehouse using ETL. This enables businesses to employ the most effective data integration method for each scenario.

2) How to derive the most value?

There was a classic comment on the Windows XP migration blog, “If you are thinking about moving your enterprise off Windows XP NOW- you are already too late.” This was April 2013 when there was a year still left for MS to support XP.

Very often, licensing agreements run till the end of the calendar year which coincides with the reporting and real-time information needs of the business. If the operations or technology people cannot guarantee the up-time of the warehouse and the relevance of its data, the business folks cannot depend on its predictive analysis and risk assessment of marketing or sales strategies.

Timing your migration with a view and in cooperation with the business team is critical to ensuring the success of your migration.

Moving from one ETL platform to another is a time-consuming process that needs thorough planning and execution. At Bitwise, some of the critical migrations we have been involved with have ranged from 18 man-months to 240 man-months. What can help reduce the effort is building platform reassessment into the life cycles of the applications using the tools and planning moving across vendors based on these reassessments.

Adding value by cutting costs (and not corners) is the way forward for today’s lean enterprises. This is challenging the set paradigms for design and architecture to offer solutions that don’t necessarily reinvent the wheel – but do away with it altogether.

It is forcing stakeholders like analysts, modelers, and design and development teams to leave the silos they work in and come up with an integrated approach. Like the separation of critical business SLAs on proprietary products and leaving non-critical functions on open source. The integration of the two then takes on its own criticality and needs to be treated with the same respect the SLA-linked business functions get.

3) How Do I Curb My Costs?

Before jumping on any new platform it is essential to run a scaled-down version of your enterprise application in a Proof of Concept.

Furthermore, it would be a good idea to break down the application into segments that need to be migrated vs. elements that can easily or seamlessly be ported over to the new platform. Scripts that are used by the Enterprise-authorized batch scheduler and XML Metadata that can be parsed to generate Source to Target mappings are a few of the examples of elements that are easily ported. However, custom scripts that are specific to administration or managing the execution of the product in your environment do not port easily and are better redone from scratch.

Metadata management and Data Lineage tracking is a key elements of any enterprise Data Integration/ETL solution and the most appropriate platform for your enterprise should be including this as a focused offering. Besides the help it provides the Analysts, having effective data lineage in place reduces the time required for setting up test data for the migration team.

As is with every critical system, budget for extensive testing and data validation while also allowing parallel execution of the old and the new systems with the same input data sets for Production validation.

To efficiently get this done, it may be a good idea to seek out tools/services that can automate such migration and associated testing from source to The target platform. Having a data validation suite in place that can be used by both the migration team as well as the end business users helps reduce turnaround time for the UAT.This would save the efforts of manual conversion and bring in efficiencies both from a cost and Time-to-market perspectives. Bitwise’s ETL Converter is a good candidate for such an automated migration service offering which can serve as your starting point.

And, with a view of assessing the platform you are migrating to in a couple of years down the line, ensure the thought process behind a particular course of action is as well documented as the action itself.