Operating procedures

A lot

Diverse

high speed

value

A computing process that uses methods such as artificial intelligence, machine learning, and statistics to extract useful, previously unknown patterns or knowledge from massive amounts of data.

The amount of data has expanded dramatically, giving rise to new research directions: database-based knowledge discovery, and research on corresponding data mining theories and technologies.

The next technology hotspot after the Internet

Explosive data but poor knowledge

The evolution from business data to business information

Data preprocessing

data mining

model evaluation

process

data

information

Knowledge

wisdom

Association rule mining

supervised machine learning

Unsupervised machine learning—clustering (similarity algorithm)

return

supervised learning

unsupervised learning

Enterprise data applications

semi-supervised learning

reinforcement learning

Predictive Mining

descriptive mining

Summary summary of the data set

Data association rules

Classification and prediction

clustering

Heterogeneous detection

time series model

Composed of data objects

A data object represents an entity

Known as: sample, example, instance, data point, object, tuple

a characteristic of a data object

the term

Classification

mean, median, mode

Range, quartile, quartile range

Five-number summary, box plots and outliers

Variance, standard deviation

Quantile plot

Quantile - Quantile plot

Histogram

Scatter plot

Express data effectively through graphics

Boxplot (boxplot)

Histogram

Scatter plot

A simple way to visualize one-dimensional values ​​is to use pixels, whose color reflects the value of that dimension

Suitable for one-dimensional values, not suitable for distribution of multi-dimensional spatial data

Help users discover projections of multidimensional data. The primary challenge of geometric projection technology is to figure out how to visualize high-dimensional space in two dimensions.

For two-dimensional data points, a Cartesian coordinate system scatter plot is usually used. Different colors or shapes can be used in the scatter plot as the third dimension of the data.

(Used for three-dimensional data sets) Scatter plots, scatter plot matrices, and parallel coordinate visualization (when the number of dimensions is large)

Represent multidimensional data values ​​with a small number of icons

Two commonly used icon methods

Divide all dimensions into subsets (i.e. subspaces) and visualize these subspaces hierarchically

Two commonly used hierarchical visualization methods

Tag Cloud

Similarity

Dissimilarity

Proximity

Data Matrix (Object - Attribute Matrix)

Dissimilarity Matrix (Object - Object Matrix)

Several common methods for calculating distance measures for the dissimilarity of numerical attribute objects

Each type of attributes is divided into a group, and data mining analysis (such as cluster analysis) is performed on each type. If these analyzes get the same results, the method works, but in practical applications it is difficult to get the same results for each attribute type classification.

A better approach: just do a single analysis, combine the different attributes in a single dissimilarity matrix, and transform the attributes into a common interval [0.0,0.1]

example

Text retrieval, biological information mining

Document vector, word frequency vector

Find the optimal hyperplane

height, weight

Unordered categorical variable

Orderly classification

Classification, ordering, distance, ratio

The learner selects the most likely hypothesis h from the candidate hypothesis set H when given the data D. h is called the maximum posterior hypothesis.

It is usually assumed that each attribute is independently and identically distributed

Attributes can be discrete or continuous

Solid mathematical foundation and stable classification efficiency

Not sensitive to missingness, noisy data, and outliers

If the attributes are not relevant, the classification effect is very good

Association rule mining is used to mine the correlation between item sets in the transaction database and mine all association rules that meet the minimum threshold requirements of support and confidence.

Association rules are used to find potentially useful dependencies between data items in large amounts of data.

frequent itemsets

Rules that meet or exceed minimum support and confidence

In the item set of big data, find the occurrence number ≥ frequent item set

From the frequent itemsets obtained above, establish association rules that meet the minimum support and credibility conditions.

consistency measurement

correlation measurement

Discrimination ability measurement

information measurement

Select forward step by step

step by step backward selection

Is the study of the relationship between a single dependent variable and one or more independent variables

Prediction refers to using observed variables to predict dependent variables

Causal analysis treats the independent variable as the cause of the dependent variable.

Model data that does not have linear dependencies

Use polynomial regression modeling method, and then perform variable transformation to convert the nonlinear model into a linear model, and then solve it using the least squares method

The combined variable is the principal component

The first step in the big data life cycle

Compared with traditional data, big data data is massive, diverse, and heterogeneous.

From collection to processing, big data needs to weigh consistency, availability, and partition fault tolerance.

Log collection of distributed systems

Network data collection

DPI Deep Packet Inspection

DFI Depth/Dynamic Flow Inspection

Specific system interface data collection

Improve data quality

Data cleaning

data integration

Data transformation (such as normalization)

1. Missing value cleaning

2. Cleaning of outliers (outliers, wild values)

3. Format content cleaning

4. Logic error cleaning

5. Non-required data cleaning

6.Relevance verification

Data integration in the traditional sense

Data integration in a general sense

Importance of models

Detect redundant attributes

Scenarios that require dimensionality reduction

Typical dimensionality reduction method—PCA principal component analysis

Reduce the size of the data by reducing its quality, such as pixels

Smoothness, attribute construction, aggregation, normalization, discretization, concept layering

Transform data through normalization

discretization by binning

Discretization by histogram binning

Discretization through clustering, decision trees and correlation analysis

Conceptual stratification of nominal data

equal width method

Equal frequency method

clustering method