What is big data

Problems solved by big data

Characteristics of big data

(1) The data scale is large and has exceeded the processing capabilities of a single machine or even a small computing cluster. However, current software and tools are not efficient, and new ideas need to be explored to solve this problem.

(2) In the process of data acquisition, analysis and processing, data quality problems are prone to occur, and special attention must be paid to data uncertainty.

(3) Data changes rapidly and dynamically and often exists in the form of streaming data. There is a need to find real-time analysis and visualization methods for streaming data.

(4) Facing complex and high-dimensional data, the current software system is mainly based on statistics and basic analysis, and its analysis capabilities are insufficient.

(5) Multi-source data have different types and structures, and existing methods are difficult to meet the processing needs of unstructured and heterogeneous data.

(1) Close integration of visualization technology and data mining technology. (2) Close integration of visualization technology and human-computer interaction technology. (3) Visualization technology is widely used in the processing and analysis of large-scale, high-dimensional, structured data. (4) Elastic changes in processing data jumping capabilities.

Internal data processing (buried points)

External data processing (web crawlers)

Data preprocessing

data mining

ladder diagram

line chart

Fitted curve plot

Scatter plot

Column chart

stacked column chart

pie chart

donut chart

stack

Rectangular tree diagram

stacked area chart

Scatter plot, scatter plot matrix, bubble plot

Histogram, density map, heat map

Personnel management, home appliances, public services, government decision-making, e-commerce

Word cloud, document collection, time series text visualization

(1) Easy to understand the central idea (2) Easy to classify, compare and summarize (3) Text visualization improves people’s ability to think relatedly

concept

Scatterplot, Scatterplot Matrix, Table Lens, Parallel Coordinates

Challenges faced

Heterogeneous data

large scale data

complex data