Structural data self-describing

The complexity of data structure descriptions

Dynamic nature of data structure description

Extraction of information content from web pages

Structural processing (including text lexical segmentation, part-of-speech analysis, and ambiguity processing)

Semantic processing (entity extraction, vocabulary correlation analysis, sentence correlation analysis)

Text construction (including vector space model, topic model)

Facebook’s open source log collection system

composition

composition

It is a distributed, reliable and highly available massive log collection, aggregation and transmission system provided by Cloudera.

Acquisition module

Data processing module

data module

depth first traversal

breadth-first traversal

Backlink Count Strategy

OPIC strategy

Big site priority strategy

storage layer

basic management

application interface layer

access layer

High reliability

Efficiency

High scalability

High fault tolerance

low cost

What does it run on linux platform?

Support multiple programming languages

Hadoop Distributed File System (HDFS)

Hadoop YARN (Yet Another Resource Negotiator) is Hadoop's resource management and job scheduling framework

MapReduce is the computing model of Hadoop, used to process parallel computing of large-scale data sets.

Hive is a data warehouse infrastructure that provides a SQL-like query language (HiveQL) to process and analyze data stored in Hadoop

Pig is a platform for large-scale data processing and provides a scripting language called Pig Latin. Pig Latin language is a data flow language that can be used to describe data transformation operations and analysis processes.

HBase is a distributed column-oriented NoSQL database built on Hadoop's HDFS. It provides real-time read and write access to large-scale structured data with high scalability and reliability

Spark is a fast, general-purpose big data processing engine that can perform parallel computing of large-scale data sets in memory.

Sqoop is a tool for data transfer between Hadoop and relational databases. It can import data from relational databases into Hadoop for analysis, and export result data from Hadoop to relational databases.

Flume is a reliable, scalable big data collection system for collecting, aggregating and moving data from various data sources (such as log files, sensor data, etc.) into Hadoop.

System log analysis

User habit analysis

integrity

fault tolerance

Ease of use

Free and open source

Support multiple languages

Topology

spout (data source)

Bolt (processing component)

Stream

Nimbus (master node)

Supervisor (worker node)

ZooKeeper (distributed coordination service)

information flow processing

continuous calculation

Distributed remote procedure call

Rapidity

Elastic scalability

Various computing models

Multi-language support

Comprehensive

Driver is the main component of Spark application. The driver program runs in an independent process and is responsible for the control and coordination of the entire Spark application.

Cluster Manager is responsible for managing the resource scheduling and task allocation of Spark applications on the cluster.

Executor is a process running on the worker nodes in the cluster and is responsible for performing specific tasks.

RDD is the core data abstraction of Spark. It is an immutable data collection that can be partitioned and operated in parallel.

The DAG scheduler is responsible for converting operations in Spark applications into directed acyclic graphs (DAGs), optimizing and scheduling them.

Task Scheduler is responsible for allocating tasks in the Stage to available Executors for execution.

In a Spark application, when data reshuffling (Shuffle) operation is required, the data will be network transferred and redistributed between different Executors.

Spark SQL is Spark's structured query module, providing a high-level API and query language for processing structured data.

Spark Streaming is Spark’s stream processing module for real-time processing and analysis of data streams.

MLlib is Spark's machine learning library, which provides a series of machine learning algorithms and tools for data mining, predictive analysis and model training.

GraphX ​​is Spark’s graph computing library for processing large-scale graph data and graph algorithms.

SparkR is the R language interface of Spark, allowing R language users to use Spark for large-scale data processing and analysis.

Application scenarios for operating specific data sets multiple times

Coarse-grained update status application

The amount of data is not particularly large, but it is suitable for applications requiring real-time statistical analysis.

Hadoop is suitable for offline large-scale data processing tasks, mainly used for batch data storage and analysis.

Spark supports multiple data processing models such as batch processing, real-time stream processing, interactive query, and machine learning.

Storm is a real-time stream processing framework for processing continuous data streams

Since Hadoop uses disk storage and MapReduce batch processing model, it is suitable for processing large-scale offline data, but it may be slower for scenarios with high real-time requirements.

Spark uses in-memory computing and RDD-based data abstraction, which can keep data in memory for iterative calculations and is suitable for data processing tasks that require higher performance and interactivity.

Storm focuses on real-time stream processing, has the characteristics of low latency and high throughput, and is suitable for rapid processing and analysis of real-time data.

Hadoop provides a reliable distributed file system (HDFS) and a scalable MapReduce computing model, which is suitable for the storage and batch processing of massive data. It has good fault tolerance and data reliability

Spark provides richer data processing capabilities and supports multiple models such as batch processing, real-time stream processing, interactive query, and machine learning. It also provides high-level APIs and libraries (such as Spark SQL, Spark Streaming, and MLlib) to simplify the development of big data processing and analysis

Storm focuses on real-time stream processing and provides reliable message passing and stream topology processing models. It can process large-scale data streams in real time and supports complex stream processing logic

Hadoop has an extensive ecosystem and a large number of tools and components, such as Hive, Pig, and HBase, for higher-level data processing and analysis. It has a mature community and extensive support

Spark also has an active open source community and a rich ecosystem that supports a variety of data processing and machine learning tasks. It is tightly integrated with the Hadoop ecosystem and can work seamlessly with HDFS, Hive and other Hadoop components

Storm's ecosystem is relatively small and mainly focuses on the field of real-time stream processing. It provides some plugins to integrate with other tools and systems such as Kafka and Cassandra

Cloud computing is a dynamically expanded computing model that provides users with network virtualized resources as services.

Hyperscale

Virtualization

High reliability

Versatility

High scalability

on demand services

extremely cheap

Public cloud is cloud computing infrastructure built and managed by third-party service providers (such as Amazon AWS, Microsoft Azure and Google Cloud Platform)

A private cloud is a cloud computing infrastructure built and managed by an organization itself to support its internal business needs

Hybrid cloud is a combination of public cloud and private cloud, providing more flexible and diverse solutions by connecting and integrating these two cloud environments. In a hybrid cloud, organizations can deploy workloads and applications into public or private clouds depending on their needs

IaaS is the most basic service model in cloud computing, which provides virtualized computing resources, storage, network and other infrastructure

Common IaaS providers include Amazon AWS’s EC2, Microsoft Azure’s virtual machine service, and Google Cloud Platform’s Compute Engine.

PaaS provides a platform to develop, run and manage applications in a cloud environment

Common PaaS providers include Microsoft Azure’s App Service, Google Cloud Platform’s App Engine and Heroku, etc.

SaaS is the highest level service model in cloud computing. It provides fully managed applications that users can directly access and use through the Internet.

Common SaaS applications include email services (such as Gmail), online office suites (such as Microsoft 365 and Google Workspace), and customer relationship management (CRM) systems (such as Salesforce)

Virtualization technology can realize server virtualization, storage virtualization, network virtualization, etc., enabling cloud computing platforms to achieve elastic allocation and management of resources.

Middleware technology plays a role in connecting and coordinating different components and services in cloud computing. It provides a set of software tools and services for managing and scheduling the deployment, configuration and execution of applications

Middleware technology also includes load balancing, container technology, message queues and service orchestration, etc., used to provide high availability, scalability and flexibility in cloud computing environments.

Cloud storage technology is a technology used to store and manage large-scale data

Decision support is a method that combines information technology and management science to provide decision-makers with the information, tools and models needed for decision-making.

It helps decision-makers make decisions by analyzing and interpreting data, providing decision-making models and algorithms, and providing visualization and interactive interfaces.

structured decision-making

unstructured decision making

semi-structured decision making

Identify problems and formulate decision-making goals

Use probability to quantitatively describe the possibility of various outcomes produced by each plan

Decision makers quantitatively evaluate various outcomes

Comprehensive analysis of all aspects of information

Data Management and Integration: Collect, integrate, and manage data relevant to decision-making.

Model and algorithm support: Provides various decision-making models and algorithms for analysis and prediction.

Visualization and interactive interface: Help decision makers understand and operate data through visual display and interactive interface.

Scenario simulation and optimization: Supports the simulation and optimization of different decision-making options and evaluates their potential effects.

Collaboration and sharing: Support the collaboration and information sharing of decision-making teams and promote the collective decision-making process.

Innovation models refer to methods and strategies used to innovate and change existing business models. It focuses on how to provide new value propositions to the market and gain competitive advantage through the creative use of resources, technology, market insights and business logic.

Provide brand-new products and services and create new industrial fields

Its business model differs from other companies in at least several elements

Have good performance

Change revenue model

Change business model

Change the industry model

changing technology paradigm

strategic positioning innovation

Resource capability innovation

Business ecological environment innovation

Hybrid business model innovation