Advanced Database Systems (mid-1980s-present)

Advanced Data Analysis (late 1980s-present)

composition

A relational database is a collection of tables, each table is given a unique name

Each tuple in the relational table represents an object, is identified by a unique keyword, and is described by a set of attribute values.

Each table contains a set of attributes (columns or fields) and usually holds a large number of tuples (records or rows)

Semantic data models are usually built for relational databases, such as the entity-relationship (ER) data model

A data warehouse is an information repository that collects information from multiple data sources, stores it in a consistent schema, and typically resides on a single site. Data warehouse is constructed through data cleansing, data transformation, data integration, data loading and periodic data refresh.

Generally, each record in the transaction database represents a transaction, such as a customer's purchase or a flight booking. A transaction contains a unique transaction identification number (TransID), and a list of items (such as purchased items) that make up the transaction. The transaction database may have some additional tables related to it that contain other information about the transaction, such as product descriptions.

Time-related or sequence data (historical records, time series data), data streams (video surveillance, they play continuously), spatial data (maps), engineering design data (building data, integrated circuits), hypertext and multimedia data (text, images), graph and network data (such as social information networks), the World Wide Web, special semantics (sequence, audio and video content, connectivity) and mining patterns with rich structure and semantics

Data characterization: generally summarize data for the class under study (target class)

Data differentiation: Compare the target class with one or more comparison classes (contrast classes)

frequent itemsets

Frequent subsequences (sequence patterns)

frequent substructure

Single-dimensional association rules: association rules containing a single predicate

Multidimensional association rules: associations involving multiple attributes or predicates

Classification

Regression: Used to predict missing or hard-to-obtain numerical data values, and also includes the identification of distribution trends based on available data.

Correlation analysis is performed before classification and regression and it attempts to identify attributes that are significantly related to the classification and regression processes.

Concept: Objects are clustered or grouped according to the principle of maximizing intra-class similarity and minimizing inter-class similarity. Clusters of objects are formed in such a way that objects in the same cluster are highly similar but very dissimilar to objects in other clusters. Each cluster formed can be viewed as an object class from which rules can be derived. Clustering also facilitates the formation of classifications, that is, organizing observations into a class hierarchy and grouping similar events together.

Concept: Find data objects in a dataset that are inconsistent with the general behavior or model of the data

Supervised learning: Similar to classification, the supervision in learning comes from labeled instances in the training data set

Unsupervised learning: similar to clustering, the input instances are not labeled

Semi-supervised learning: using labeled and unlabeled instances when learning a model

Active learning: Let users play an active role in the learning process

Discover new types of knowledge

Mining knowledge in multi-dimensional space

Data Mining—An Interdisciplinary Endeavor

Improve discovery capabilities in network environments

Dealing with uncertain data, noisy or incomplete data

Pattern evaluation and mining of pattern constraint guidance

interactive mining

Combine background knowledge

Specific data mining and data mining query languages

Representation and visualization of data mining results

Effectiveness and scalability of data mining algorithms

Parallel distributed and incremental mining algorithms

Handle complex data types

Mining dynamic, networked, global databases

The social impact of data mining

Privacy-preserving data mining

invisible data mining

Nominal attribute: The value is the name of some symbol or thing. Each value represents some category, encoding or status, so nominal attributes are considered categorical

Binary attribute: It is a nominal attribute with only two status categories: 0 or 1. 0 means that the attribute does not appear, and 1 means that the attribute appears.

Ordinal property: one whose possible values ​​have a meaningful order or rank, but the difference between successive values ​​is unknown.

Numeric properties

Ignore tuples

Fill in missing values ​​manually

Use a global variable to fill in missing values

Fill missing values ​​using a central measure of the attribute, such as the mean or median

Use the attribute mean or median of all samples in which a given tuple belongs to the same class

Fill missing values ​​using the most likely value

binning

return

Outlier analysis (clustering)

Concept: The original value of the numerical attribute is replaced with an interval label or a concept label

Methods: binning, histogram analysis, cluster analysis, decision tree analysis, correlation analysis

Concept: Define a mapping sequence that maps low-level concepts to higher-level, more general concepts

method

Dimensional protocol

quantity specification

data compression

Gathers all information on a subject, spans the entire enterprise, provides a full range of data integration, usually from one or more operational database systems or external information providers, and is versatile, containing both detailed and summary data

Concept: Contains a subset of enterprise-wide data that is useful to a specific user group. The scope is limited to selected topics. The data is usually aggregated.

Independent data mart: Data typically comes from one or more operational database systems or external information providers, or from data generated by a specific department or local area

Dependent data mart: directly from the enterprise database

Is a collection of views on the operating database. In order to process queries efficiently, only some possible summary views are materialized.

Description of the data warehouse structure: warehouse schema, views, dimensions, hierarchies, definition of exported data, location and content of data marts

Operational metadata: data lineage, data circulation, management information

Algorithms used for aggregation: measure and dimension definition algorithms, granularity at which the data is located, partitioning, subject areas, aggregation, summary, predefined queries and reports

Mapping from operating environment to data warehouse: source databases and their contents, gateway descriptions, data extraction, cleansing, transformation rules and default values, data refresh and cleansing rules, security (user authorization and access control)

Data on system performance: In addition to timed scheduling rules for refresh, update, and copy cycles, including indexes and summaries to improve access and retrieval performance

Business Metadata: Business Terms and Definitions, Data Owners and Charging Policies

(1) Metadata is used as a directory to help decision support system analysts locate the content of the data warehouse.

(2) As a guide for data mapping when data is converted from an operating environment to a data warehouse environment

(3) For the summary algorithm, the current detailed data is summarized into slightly comprehensive data, or the slightly comprehensive data is summarized into highly comprehensive data.

(4) Metadata should be stored and managed persistently (i.e. stored on the hard disk)

It consists of a lattice of cubes, each cube corresponding to a different level of aggregation of a given multidimensional data. Allows data to be modeled and observed in multiple dimensions, defined by dimensions and facts.

Top-down view: allows you to select relevant information needed for the data warehouse

Data source view: reveals the information collected, stored and managed by the operating database system

Data warehouse view: includes fact tables and dimension tables. They provide information stored in the data warehouse, including precomputed totals and counts, as well as information about sources, dates, and times that provide historical context.

Business Query View: Perspective on the data in the data warehouse from the end user's perspective

Select the business process to be modeled

Select business processing granularity

Select the dimensions to use for each fact table record

Select the measure to place in each fact table record

Information Processing: Supports queries and basic statistical analysis and reporting using crosstabs, tables, charts or graphs

Analytical processing: Supports basic OLAP operations, including slicing and dicing, drill-down, roll-up, and pivoting, generally operating on summary and detailed historical data

Data mining: supports knowledge discovery, including finding hidden patterns and associations, constructing analytical models, performing classification and prediction, and using visualization tools to provide mining results.

High quality of data in data warehouse

Information processing infrastructure surrounding databases

Multidimensional data exploration based on OLAP

Online selection of data mining functions

This will involve converting select, projection, roll-up (grouping) and drill-down operations in the query into corresponding SQL/OLAP operations

Involves finding all materialized cubes for which a possible user answers the query, using knowledge of "dominance" connections between cubes, performing pruning, evaluating the cost of using multiplication and division of the materialized cubes, and selecting the cube with the minimum cost.

branch topic

A collection of items. A set containing K items is called a K itemset. The frequency of occurrence of an item set is the number of transactions containing the item set, which is referred to as the frequency, support count, and count of the item set.

If the support of itemset I meets the predefined minimum support threshold (that is, the absolute support of I meets the corresponding minimum support count threshold), then I is a frequent itemset, and the set of frequent itemsets is recorded as Lk

If there is no true superitemset Y such that Y and X have the same support count in D, then the itemset X is closed in the data set D.

If X is closed and frequent in D, then the itemset X is a closed frequent itemset in D

If X is frequent, and there is no super itemset Y such that X belongs to Y, and Y is frequent in D, then the itemset

By definition, each of these itemsets appears frequently at least as often as the predefined minimum support count min_sup

By definition, these rules must satisfy minimum support and minimum confidence

Is a form of data analysis that extracts models that describe classes of data. A classifier or classification model predicts class labels (classes), and numerical predictions model continuous function values. Classification and numerical prediction are two major types of prediction problems.

The class label of each training tuple is provided, and the learning of the classifier is supervised by being told "which class" each training tuple belongs to.

The class label of each training tuple is unknown, and the number or set of classes to be learned may not be known in advance.

(1) Learning stage (building a classifier model)

(2) Classification stage (use the model to predict the class label of the given data)

(1) The construction of the decision tree classifier does not require any domain knowledge or parameter settings, so it is suitable for exploratory knowledge discovery.

(2) Decision trees can handle high-dimensional data

(3) The acquired knowledge is intuitive and easy to understand in the form of tree representation.

(4) The inductive learning and classification steps of decision trees are simple and fast

(5) Generally speaking, decision tree classifiers have good accuracy

(6) Decision trees are the basis of many business induction systems

Disadvantages: Successful use may depend on the data at hand

is a heuristic that selects the splitting criterion that best divides the data partition D of training tuples labeled by a given class into separate classes.

If D is divided into smaller partitions based on the output of the splitting criterion, ideally each partition should be pure (i.e. all tuples falling into a given partition belong to the same class)

Attribute selection metrics are also called splitting criteria because they determine how to split on a given tuple. Attribute selection metrics provide a rank rating for each attribute that describes a given training tuple. The attribute with the best metric score is chosen. The splitting property of the given tuple.

If the splitting attribute is continuous, or we are limited to constructing binary trees, then a splitting point or a splitting subset must also be returned as part of the splitting criterion. The tree nodes created for partition D are marked with the splitting criterion, from each output of the criterion. Branches are grown and tuples are divided accordingly.

information gain

Gain rate

Gini index

When a decision tree is created, due to noise and outliers in the data, many branches reflect anomalies in the training data. The pruning method deals with this overfitting problem.

The tree is pruned by stopping tree construction early (e.g. by deciding not to split at a given node or by dividing a subset of training tuples). Once stopped, the nodes become leaves. The leaf can hold the most frequent classes among a subset of tuples, or a probability distribution over these tuples.

It prunes subtrees from fully grown ones. Prune the subtree at a given node by removing its branch and replacing it with a leaf. The class label of the leaf is the most frequent class in the subtree.

Combining K learned models (or base classifiers) M1, M2, M3······MK together to create an improved composite classification model M*. Use the given data set to create K training sets D1, D2, D3...DK, where Di (1<=i<=k) is used to create the classifier Mi, given a new data to be classified A tuple that each base classifier votes by returning a class prediction.

bagging

promote

random forest

The difference between bagging and lifting

Given two classes of data, if the main class of interest (positive class) is represented by only a small number of tuples, while the vast majority of tuples represent the negative class, the data is class-imbalanced.

For multi-class imbalanced data, the data distribution of each class is significantly different, where tuples of the main class or the class of interest are rare. The class imbalance problem is closely related to cost-sensitive learning, where the cost of errors is not equal for each class.

Oversampling and undersampling

threshold shift