Case Study: Migration of Legacy ETL to Cloud

As part of its organizational strategy to move critical processes to Cloud, a Fortune 500 Banking company required a flexible, cost effective solution for migrating its Human Resources ETL process from an on-premise enterprise data management environment to Cloud.

In this session, Bitwise will discuss how we partnered with the customer’s team to quickly and efficiently migrate legacy ETLs to a tool that is compatible with leading Cloud platforms and that provides a rapid development environment while delivering robust capabilities and performance.

Bitwise Booth: Your One-Stop Shop for All Things EDW and Big Data

At our booth, Bitwise will be demonstrating how we provide a ‘one-stop shop for all things EDW and Big Data’ for our customers with innovations for common pain areas such as ETL testing and test data management, PL/SQL and ETL conversion, and ETL on Hadoop.

Introducing Hadoop Adaptor for Mainframe Data

A common challenge that we see across our customers is a need for an easy, cost effective option for converting mainframe data to a Hadoop-friendly data format. At TDWI Savannah, Bitwise plans to introduce our Hadoop Adaptor for Mainframe Data that converts any mainframe data to any Hadoop format, including ASCII, Avro and Parquet. The solution makes it easy to bring mainframe EBCDIC data into the data lake, making it available for advanced analytics and being able to combine with any other data in the data lake.

Stay tuned for more updates on our Mainframe Modernization solutions…

Data Democracy – The Foundation for Big Data

Use cases involving Analytics or Data Mining with an integrated Social Media component are being thrown at enterprise executives. These use cases appear “cool” and compelling upfront but upon thorough analysis reveal that they are missing some necessary considerations such as Data/Info Security, Privacy regulations, Data Lineage from an implementation perspective, and in addition fail to build a compelling ROI case.

One needs to realize that for any “cool” use case to generate eventual ROI, it is very important to focus on Big Data Integration (i.e. Access, Preparation, and Availability of the data – see firms must not overlook importance of big data integration). Doing so essentially will empower the enterprises to implement ANY use case that makes the most sense to their particular business.

“Data Democracy” should be the focus. This focus also helps address the technology challenge of handling ever-growing enterprise data efficiently and leverage the scalable and cost-effective nature of these technologies – and an instant ROI!

Concept to Realization – Real Issues

Once this is understood, the next step is to figure out a way to introduce the use of these new technologies to achieve the above goals and doing so in the least disruptive and most cost-effective way. In fact, enterprises are looking at ETL as a standard use case for Big Data technologies like Hadoop. Using Hadoop as a Data Integration or ETL platform requires developing Data Integration applications using programming languages such as Map Reduce. This presents a new challenge in combining of Java skillsets with the expertise of ETL design and implementation. Most ETL designers do not have Java skills as they are used to working in a tool environment and most Java developers do not have experience in handling large volumes of data resulting in massive overheads of training, maintaining and “firefighting“ coding issues. This can cause massive delays and soak up valuable resources for organizations to solve half the problem.

Moreover, while making the investments in the form of hardware and skillsets like Map Reduce, when the underlying technology platforms inevitably would advance, development teams would be forced to rewrite the application to leverage these advancements.

Concept to Realization – a Possibility?

Yes it is. One of the key criteria for any data integration development environment on Hadoop is code abstraction to allow users to specify the data integration logic as a series of transformations chained together in a directed acyclic graph that models how users think about data movement making it significantly simpler to comprehend and change than a series of Map Reduce scripts.

Another important feature to look out for is technology insulation – provisions in the design to change the run-time environments such as Hadoop with any future technologies prevalent at that time.

A variety of data sources and objectives are supported by ETL tools.

Databases, cloud storage, APIs, and files are just a few examples of the numerous data sources and objectives available to modern businesses. ETL solutions may readily connect to these systems using standardized protocols and APIs because they are made to function with a wide variety of data sources and targets. Data from various sources can be more easily integrated since ETL systems also offer the necessary transformations to change the data’s format.

ETL software offers a complete data integration solution.

Data extraction, data transformation, and data loading are all handled by ETL technologies, which offer a comprehensive solution for data integration. Additionally, these solutions provide several features for resolving errors, validating data, and managing data quality.

ETL solutions are therefore an all-in-one data integration solution, making them perfect for large-scale data integration projects.

In some situations, ETL tools perform better than ELT.

In a more recent method of data integration called ELT (Extract, Load, Transform), the data is first loaded into the target system before being transformed. Even though ELT has grown in acceptance recently, ETL is still preferred in some circumstances. For instance, ETL can offer greater performance if the data source is big since it can filter, combine, and transform the data at the source. As a result, processing times are sped up because fewer data needs to be put into the target system.

Other data integration methods, such as data streaming and data virtualization, can also be integrated with ETL tools. ETL tools, for instance, can be used to load data from a legacy system into a data warehouse. At the same time, real-time data from the same system can be obtained using data streaming and integrated with the data warehouse using ETL. This enables businesses to employ the most effective data integration method for each scenario.