Includes structured data stored in a database

Unstructured text data stored in documents or files

and other types of unstructured data, such as audio, video, streaming data, etc.

1. ETL goal: enter the data warehouse with clear goals

2. Structured data: enter the data warehouse

3. Data warehouse: The ultimate goal is BI

Can be executed as a regularly scheduled event (batch processing)

Can be executed when new data or data is updated (real-time or event-driven)

Includes selecting the required data and extracting it from its source data

The extracted data is then stored in a physical data repository on disk or in memory.

Is to make the selected data compatible with the structure of the target database

Format changes

structural changes

Semantic changes

Eliminate duplicates

rearrange

Can be executed in batches or in real time

Or store the conversion results in the cache area in the physical state

Or store the converted data in memory in a virtual state

until moving to the loading step

Physically store or present the conversion results on the target system

Overview

If the target system has stronger conversion capabilities than the source system or intermediate application system, the data processing sequence can be switched to ELT---extraction, loading, conversion

ELT allows data to be loaded into the target system before being converted

ELT allows source data to be instantiated on the target system in the form of raw data, which is beneficial to other processes

Loading into the data lake using ELT, which is common in big data environments

is a synonym for transformation, both the process of building a search matrix from a source structure to a target structure and the result of that process

Defines the source data to be extracted and the identification rules for the extracted data, the identification rules for the target to be loaded and the target rows to be updated, and the transformation or calculation rules to be applied.

Refers to the time difference between when data is generated by the source system and when the data is available on the target system.

Different data processing methods will lead to different degrees of data delay

Data moves between applications and organizations in batches of files, either based on manual requests from data consumers or automatically triggered on a periodic basis. This type of interaction is called batch processing or ETL

Data moved in batch mode will represent all data at a given point in time

This set of data is called an increment, and the data at a certain moment is called a snapshot

For batch data integration solutions, there is often a significant delay between data changes in the source and data updates in the target, resulting in high latency from

Micro-batching

opportunity

Is a method of reducing transmission bandwidth requirements by adding filtering to only include data that has changed within a specific time range

Change data capture monitors changes (insertions, changes, deletions) to a data set and then communicates these changes (deltas) to other data sets, applications, and organizations that consume the data

As part of the change data capture process, data can also be tagged with identifiers such as tags or timestamps.

Change data capture can be data-based or log-based

Based on data

Based on logs

Most data integration solutions that do not take a batch approach use a near-real-time or event-driven approach

Data is processed in smaller sets within a specific schedule, or as events occur, such as data updates

Near real-time processing has lower latency compared to batch processing

And because the work is distributed over time, the system load is lower

However, it is generally slower than synchronous data integration

Near real-time data integration solutions are often implemented using an enterprise service bus

In an asynchronous data flow, the system providing the data does not wait for the receiving system to confirm the update before continuing processing.

Asynchronous means that the sending or receiving system may be offline for a period of time while the other system is functioning normally

Asynchronous data integration does not block the source system application from continuing to execute, nor does it cause the source application to be unavailable if any target application is unavailable

Since data updates to the application in an asynchronous configuration are not timely, it is called quasi-real-time integration

There are situations where time delays or other differences between source and target data are not allowed

When data from one dataset must be perfectly synchronized with data from another dataset, a real-time synchronization solution must be used

In a synchronous integration solution, execution waits for confirmation from other applications or processes before executing the next activity or transaction.

Because time must be spent waiting for confirmation of data synchronization, this means that the solution can only handle fewer transactions

If any application that needs to update data is in an unavailable state, transactions within the application cannot be completed.

Low-latency data integration solution designed to reduce incident response times

Use solid state disk

Asynchronous solution

Massive multiprocessor or parallel processing

Typically monitor the change log of a dataset rather than the dataset itself

Because they do not compete with applications for access to data sets, they minimize the impact on any operational application.

Only data from the change log is transferred between replicas

Standard replication solutions are near real-time

The vast majority of interactions between shared data systems are "point-to-point", where they pass data directly to each other

This model works in the context of a small set of systems

But when many systems require the same data from the same source, it becomes inefficient and increases organizational risk

Impact processing

Management interface

potential inconsistencies

It consolidates shared data (physical or virtual) into a central data center that applications can use

All systems that want to exchange data do so through a central common data control, rather than directly with other systems (point-to-point).

Data warehouses, data marts, operational data stores, and master data management centers are all examples of data centers.

Data center provides consistent view of data with limited impact on source system performance

Adding systems to the mix requires only building interfaces to the data center

Enterprise Service Bus (ESB) is a data integration solution for sharing data in near real-time between multiple systems. Its data center is a virtual concept that represents a standard and standardized format for data sharing in an organization.

Some hub-and-spoke models have unacceptable latency or performance issues

The data center itself has the creation overhead in a hub-and-spoke architecture

The publish-and-subscribe model involves systems that promote (publish) data and other systems that accept (subscribe) data.

Systems that push data are listed in the catalog of data services, and systems that wish to use the data subscribe to these services

When publishing data, the data will automatically be sent to subscribing users

Coupling describes the degree to which two systems are intertwined

Tightly coupled

loose coupling

Arrange

process control

In the enterprise application integration model EAI, software modules interact only through well-defined interface calls (application programming interface API)

Data storage can only be updated through its own software module. Other software cannot directly access the data in the application, only through the defined API.

EAI is based on object-oriented concepts, which emphasizes the ability to reuse and replace any module without affecting any other module

It acts as an intermediary between systems, delivering messages between them

Applications can encapsulate sent and received messages or files through the existing capabilities of the ESB

As an example of loose coupling, the ESB acts as a service between two applications

Push data or update data can be provided through well-defined service calls between applications

Applications do not have to interact directly with other applications or understand the inner workings of other applications

Supports application independence and the organization's ability to replace systems without making significant changes to the systems they interact with

The goal of SOA is to define well-defined interactions between independent software modules

Each module can be used by other software modules or individual consumers to perform functions (provide functionality)

The key concept of SOA is that an independent service is provided: the service has no prior knowledge of the calling application, and the implementation of the service is a black box for the calling application

SOA can be implemented through various technologies such as Web services, messaging, and APIs.

Event processing is a method of tracking and analyzing (processing) the flow of information about an occurring event, and drawing conclusions from it

Complex event processing refers to merging data from multiple sources, identifying meaningful events, setting rules for these events to guide event processing and routing, and then predicting behaviors or activities, and automatically triggering real-time responses based on the predicted results.

Such as sales opportunities, web clicks, orders, and customer calls, etc.

Complex event processing requires an environment that can integrate various types of data

Since predictions often involve large amounts of data of various types, complex event processing is often associated with big data

Complex event processing often requires the use of technologies that support ultra-low latency, such as processing real-time streaming data and in-memory databases

When data exists in different data repositories, it can also be aggregated by means other than physical integration

Data federation provides access to a combination of independent data repositories regardless of their respective structures

Data virtualization enables distributed databases as well as multiple heterogeneous data stores to be accessed and viewed as a single database

Software as a ServiceSaaS

Data as a ServiceDaaS

Before cloud computing, integration could be divided into internal integration and inter-enterprise integration B2B

Internal integration

inter-enterprise integration

Cloud integration

Finding and eliminating these inefficiencies or ineffective configurations can greatly aid project success and improve an organization's overall ability to use its data.

Hub-and-spoke, point-to-point, publish-subscribe