At YACG, we consider big data integration & ETL like building a highway, or to be precise planning the road system of your city. The arterial road system that allows smooth flow of data between your applications and scales up as demand grows.
We have expertise in the latest ETL tools that visualize data integration in real time and allow us to rapidly build high quality jobs significantly reducing the build and maintenance costs for our clients. We leverage the efficiencies of an Integration Competency Center (ICC) to rapidly build using standardized data integration patterns.
At YACG we understand the fundamental challenges of data integration include:
- Converting data from one model to another
- Integrating data from heterogeneous data sources
- Identification of Data Quality issues, Data cleanup and exception reporting
We start by understanding the information needs, profiling available data to understand its meaning, and relationship between different data elements. We verify data availability and chart out a road map for a data integration project. Then we map the source data to its target, create a logical data integration flow that can be quickly converted into a ETL job.
From this point forward, we use an iterative approach to build ETL jobs. This involves, creating the ETL job, loading the data, testing the data on the target system with target applications, cleanup data, and build de-duplication logic to match the grain of target data model, and create a robust exception reporting process that gives us a clear understanding of what data makes it and what falls out.
Subsequently we deploy these ETL jobs in production and continue to provide support.
All the ETL jobs we design are designed to be re-startable without corrupting the data in the target system. They also have an option to rollback in case of data corruption. Robust data quality checks and data cleanup are part and parcel of what we build.
This approach applies to variety of Data integration jobs, Extract Transform and Load (ETL) processes, Data Migration projects, data conversion and consolidation projects.
At YACG, we use an integration competency center. Specifically our competency center focuses following tools:
- Ab Initio
- Ibm Data Stage – Ascential
- Clover ETL
By using a competency center that focuses on Data Integration, we are able to not only develop highly efficient data integration jobs but we are also able to inject data integration best practices and data integration patterns into our clients. We conduct training programs for both developers and analysts, and we install, administrator and maintain development, test, and production environments.