Wondershare EdrawMind
EdrawMindAlibaba Cloud Message Queue
MindMap Gallery Alibaba Cloud Message Queue
- 22
Alibaba Cloud Message Queue
A detailed summary of Alibaba Cloud Message Queue RocketMQ, Cloud Message RocketMQ, Cloud Message Queue RabbitMQ, Cloud Message Queue Kafka, and Cloud Message Queue MQTT.
Edited at 2024-01-20 09:44:44- bacteria
This is a mind map about bacteria, and its main contents include: overview, morphology, types, structure, reproduction, distribution, application, and expansion. The summary is comprehensive and meticulous, suitable as review materials.
- Plant asexual reproduction
This is a mind map about plant asexual reproduction, and its main contents include: concept, spore reproduction, vegetative reproduction, tissue culture, and buds. The summary is comprehensive and meticulous, suitable as review materials.
- Reproductive development of animals
This is a mind map about the reproductive development of animals, and its main contents include: insects, frogs, birds, sexual reproduction, and asexual reproduction. The summary is comprehensive and meticulous, suitable as review materials.
Alibaba Cloud Message Queue
- bacteria
This is a mind map about bacteria, and its main contents include: overview, morphology, types, structure, reproduction, distribution, application, and expansion. The summary is comprehensive and meticulous, suitable as review materials.
- Plant asexual reproduction
This is a mind map about plant asexual reproduction, and its main contents include: concept, spore reproduction, vegetative reproduction, tissue culture, and buds. The summary is comprehensive and meticulous, suitable as review materials.
- Reproductive development of animals
This is a mind map about the reproductive development of animals, and its main contents include: insects, frogs, birds, sexual reproduction, and asexual reproduction. The summary is comprehensive and meticulous, suitable as review materials.
- Recommended to you
- Outline
Alibaba Cloud Message Queue
Cloud message queue RocketMQ
Product Positioning
Cloud Message Queue RocketMQ version is a typical middleware product under a distributed architecture, using asynchronous communication and a publish-subscribe message transmission model.
The product has the advantages of asynchronous communication, simple system topology, and weak upstream and downstream coupling. It is mainly used in asynchronous decoupling, traffic peak-shaving and valley-filling scenarios.
For synchronous links that require real-time return of call results, it is recommended to use the RPC call solution.
Message queue RocketMQ version domain model
As shown in the figure, the life cycle of messages in Cloud Message Queue RocketMQ version is mainly divided into three parts: message production, message storage, and message consumption.
message production
Producer is a running entity used to generate messages in Cloud Message Queue RocketMQ version. It is generally integrated upstream of the business call link. Producers are lightweight, anonymous and identityless.
Message storage
Topic: A grouping container for message transmission and storage in Cloud Message Queue RocketMQ. The topic is internally composed of multiple queues. The storage and horizontal expansion of messages are actually implemented through queues within the topic.
Queue (MessageQueue): The actual unit container for message transmission and storage in RocketMQ Cloud Message Queue, which is analogous to partitions in Kafka. Cloud Message Queue RocketMQ uses an infinite queue structure with streaming characteristics to store messages, and messages have sequential storage characteristics in the queue.
Message: The smallest transmission unit of RocketMQ version of Cloud Message Queue. Messages are immutable and become immutable after initial sending and completion of storage.
Message consumption
ConsumerGroup: An independent consumer identity group defined in the publish-subscribe model of Cloud Message Queue RocketMQ, used to uniformly manage multiple consumers (Consumers) running at the bottom. Multiple consumers of the same consumer group must keep their consumption logic and configuration consistent, and jointly share the messages subscribed by the consumer group to achieve horizontal expansion of consumption capabilities.
Consumer: The running entity of RocketMQ version of Cloud Message Queue that consumes messages. It is generally integrated downstream of the business call link. Consumers must be assigned to a consumer group.
Subscription: Rule configuration for message filtering, retry, and consumption progress in the publish and subscribe model of Cloud Message Queue RocketMQ. Subscription relationships are managed at the granularity of consumer groups. By defining subscription relationships, consumer groups control how consumers under the specified consumer group implement message filtering, consumption retry, consumption progress recovery, etc.
Introduction to communication methods
Under the idea of distributed system architecture, complex systems are split into multiple independent sub-modules, such as microservice modules. At this time, it is necessary to consider the remote communication between sub-modules. The typical communication modes are divided into the following two types, one is synchronous RPC remote call; the other is asynchronous communication method based on middleware agent.
Synchronous RPC call model
Under the synchronous RPC call model, call communication is conducted directly between different systems. Each request is sent directly from the caller to the callee, and then the callee is required to immediately return the response result to the caller to determine whether the call result is successful.
Asynchronous communication model
In the asynchronous message communication mode, the subsystems are directly connected without strong coupling. The caller only needs to convert the request into an asynchronous event (message) and send it to the intermediate agent. If the transmission is successful, the asynchronous link call is considered completed, and the remaining The working intermediate agent will be responsible for reliably notifying the event to the downstream calling system to ensure that the task execution is completed. The intermediate agent is generally a message middleware.
The advantages of asynchronous communication are as follows:
System topology is simple
Since the caller and the callee communicate with the intermediate agent in a unified manner, the system has a star structure and is easy to maintain and manage.
Weak upstream and downstream coupling
The upstream and downstream systems are weakly coupled, the structure is more flexible, and the intermediate agent is responsible for buffering and asynchronous recovery. Upstream and downstream systems can be upgraded and changed independently without affecting each other.
Capacity peak shaving and valley filling
Message-based intermediate proxies often have strong traffic buffering and shaping capabilities, so they will not overwhelm the downstream when business traffic peaks arrive.
Introduction to message transmission model
The mainstream message middleware transmission models are mainly point-to-point model and publish-subscribe model.
peer-to-peer model
The point-to-point model is also called the queue model and has the following characteristics:
Consumption anonymity: The only identity for upstream and downstream communication of messages is the queue. Downstream consumers cannot declare independent identities when obtaining messages from the queue.
One-to-one communication: Based on the characteristics of consumption anonymity, even if there are multiple downstream consumers, they do not have their own independent identities. Therefore, each message in the shared queue will only be processed by only one consumer. Therefore, the point-to-point model can only achieve one-to-one communication.
publish-subscribe model
The publish-subscribe model has the following characteristics:
Consumption independence: Compared with the anonymous consumption method of the queue model, all consumers in the publish-subscribe model have an identity, which is generally called a subscription group (subscription relationship). Different subscription groups are independent of each other and do not affect each other.
One-to-many communication: Based on the independent identity design, messages within the same topic can be processed by multiple subscription groups, and each subscription group can get the full amount of messages. Therefore, the publish-subscribe model can achieve one-to-many communication.
Transmission model comparison
The point-to-point model and the publish-subscribe model each have their own advantages. The point-to-point model is simpler, while the publish-subscribe model is more scalable.
The transmission model used by Cloud Message Queue RocketMQ is a publish-subscribe model, so it also has the characteristics of a publish-subscribe model.
Cloud message queue Kafka
Product advantages
Provide fully managed services for open source Apache Kafka to solve the pain points of open source products. With Cloud Message Queue Kafka version, you only need to focus on business development without deployment and operation and maintenance. Compared with open source Apache Kafka, Cloud Message Queue Kafka version is lower cost, more elastic, and more reliable.
Application ecology
It has a rich application ecosystem, mainly including the following aspects:
Big data field: website behavior analysis, log aggregation, application monitoring, streaming data processing, online and offline data analysis and other fields.
Data integration: Import messages into offline data warehouses such as MaxCompute, OSS, RDS, Hadoop, and HBase.
Data processing integration: Integrate with StreamCompute, E-MapReduce, Spark, Storm and other engines.
product architecture
Cloud Message Queue Kafka version
system structure
Producer
Send messages to Kafka Broker of Cloud Message Queue Kafka version through push mode. The messages sent can be website page visits, server logs, or system resource information related to CPU and memory.
Kafka Broker
The server used to store messages. Kafka Broker supports horizontal expansion. The greater the number of Kafka Broker nodes, the higher the throughput rate of the Cloud Message Queue Kafka cluster.
Group
Subscribe and consume messages from the Cloud Message Queue Kafka version Broker through pull mode.
Zookeeper
Manage the configuration of the cluster, elect leader partitions, and perform load balancing when the Group changes.
Cloud Message Queue Kafka version
The publish/subscribe model of Cloud Message Queue Kafka version
The corresponding relationship between Group and Topic is N:N, that is, a Group can subscribe to multiple Topics at the same time, and a Topic can also be subscribed by multiple Groups at the same time.
Although a Topic can be subscribed by multiple Groups at the same time, the Topic's messages can only be consumed by any Consumer in the same Group.
Product advantages
Ready out of the box
Compatible with open source:
Cloud Message Queue Kafka Edition is 100% compatible with the open source Apache Kafka. You can directly use the open source Apache Kafka client to communicate with Cloud Message Queue Kafka Edition. Cloud Message Queue Kafka version currently supports open source versions 0.10.x~2.6.x.
Seamless migration:
Cloud Message Queue Kafka version is based on the existing open source Apache Kafka ecosystem, and you can migrate to the cloud without any code modification.
No operation and maintenance, easy to observe
HouseKeeping (health inspection component): used for runtime inspection of core links of Cloud Message Queue Kafka version. It will perform a comprehensive scan and diagnosis of the cluster every minute, and can issue alarms for unhealthy status. At the same time, an overall inspection report will be issued to Alibaba Cloud operation and maintenance personnel every day to facilitate monitoring of the health status of the Cloud Message Queue Kafka version system.
Business monitoring and alarming: Each Consumer Group can set monitoring and alarming for the accumulation of messages to help you find problems in time.
OpenAPI: Cloud Message Queue Kafka version provides you with a complete set of management and control OpenAPI to implement a series of resource management and operation and maintenance functions.
Rich data ecosystem
Provides fully managed, operation-free data processing components, supports flexible Function programming, and processes data easily and flexibly.
Support Connector to connect massive cloud products and self-built big data ecological products to easily realize data integration and calculation.
Available in extra large format
Deeply optimize the kernel to solve the performance bottleneck of thousand-level partitions of the open source version and support the performance of ten thousand-level partitions without loss.
Supports second-level expansion.
Supports writing of up to 2 GB traffic.
High availability
Alibaba Cloud's messaging product R&D and performance optimization team has further optimized the pain points of open source products to provide you with better services.
Data persistence: A professional team ensures higher availability, and message persistence is placed in the message queue. Data reliability is high and service availability is high.
High throughput capability: Even when massive messages accumulate, it can always maintain
Cloud Message Queue Kafka version cluster's high throughput capabilities.
Tens of thousands of topics: supports high concurrent reading and writing of tens of thousands of topics, always maintaining
High SLA guarantee
Supports write optimization guarantees in extreme scenarios such as grayscale upgrade, upgrade/cold data, etc.
The fully automatic inspection and operation and maintenance system guarantees service availability of 99.9% and data reliability of 99.999999%.
Data Security
Provides functions such as authentication and authorization mechanisms, user decentralization, and enterprise-level security protection.
Permission management: Fully supports Alibaba Cloud accounts and RAM users, black and white lists, STS and other functions, and realizes authorization services for Alibaba Cloud accounts and RAM users as well as cross-accounts between enterprises.
Access security: Based on the Alibaba Cloud account system, the SASL mechanism is used to authenticate user identities, and SSL is used to encrypt the channel for transmission to ensure that data is not stolen or tampered with during the transmission process, ensuring the security of your data.
Alibaba Cloud VPC: In addition to public network access, Cloud Message Queue Kafka version also supports private network VPC. You can fully control your own private network, such as selecting IP address ranges, configuring routing tables and gateways, etc. You can use Alibaba Cloud resources such as cloud servers, cloud database RDS versions, and load balancing in your own defined private network.
Open source engine optimization
Compared with the open source Apache Kafka, the powerful processing engine supports tens of thousands of topics with high concurrent reading and writing, second-level scaling and reliable cloud storage, providing a very cost-effective choice; it also provides health inspection components, business monitoring and alarm functions. , helping you discover and deal with problems in a timely and effective manner.
Application scenarios
Website activity tracking
Cloud Message Queue Kafka version of the publish/subscribe model allows you to collect website activity data (such as registration, login, recharge, payment, purchase) in real time, publish messages to different topics according to business data types, and then use the real-time delivery of subscription messages , use message flow for real-time processing, real-time monitoring, or load it to offline data warehouse systems such as Hadoop and MaxCompute for offline processing.
Cloud Message Queue Kafka version has the following advantages for website activity tracking:
High throughput: The behavioral information generated by website users is relatively large and requires high throughput to support it.
Elastic expansion: Website activity leads to a surge in behavioral data, and the cloud platform can quickly expand on demand.
Big data analysis: It can be connected to real-time data processing engines such as Storm and Spark, as well as offline data warehouse systems such as Hadoop.
Log aggregation
Cloud Message Queue Kafka version can achieve stronger data persistence and shorter end-to-end response time while having high performance. This characteristic of Cloud Message Queue Kafka makes it suitable as a log collection center. Cloud Message Queue Kafka Edition ignores the details of files, and can abstract the log data of multiple hosts or applications into message streams of logs or events, and send them asynchronously to the Cloud Message Queue Kafka Edition cluster, thereby achieving very low RT. Cloud Message Queue Kafka version client can submit messages in batches and compress messages, with almost no performance overhead for producers. Consumers can use offline warehouse storage such as Hadoop and MaxCompute and real-time online analysis systems such as Strom and Spark to perform statistical analysis on logs.
Cloud Message Queue Kafka version has the following advantages for data aggregation:
Decoupling application and analysis: Build a bridge between the application system and the analysis system and decouple the association between them.
High scalability: It has high scalability, that is, when the amount of data increases, it can quickly expand horizontally by adding nodes.
Online or offline analysis system: supports real-time online analysis system and Hadoop-like offline analysis system.
data processing
In many fields, such as stock market trend analysis, meteorological data measurement and control, and website user behavior analysis, due to the rapid, real-time and large volume of data generation, it is difficult to collect these data in a unified manner and store them in a database for processing. As a result, traditional data processing architecture cannot meet the needs. Unlike traditional architecture,
The emergence of cloud message queue Kafka version and data processing engines such as Storm, Samza, and Spark is to better solve the problems encountered in the processing of this type of data. The data processing model can realize the processing of data during the data flow process. Capture and process in real time, perform calculation and analysis according to business requirements, and finally save or distribute the results to the required components.
Cloud Message Queue Kafka version has the following advantages for data processing:
Flowing data: Capture and process data in real time during data flow, and perform calculation and analysis based on business needs.
High scalability: Due to the fast speed of data generation and the large amount of data, high scalability is required.
Data processing engine: Can be connected to open source Storm, Samza, Spark, EMR, Blink, StreamCompute and other Alibaba Cloud products.
Data transfer hub
In the past 10 years or so, special systems such as KV storage (HBase), search (Elasticsearch), streaming processing (Storm, Spark, Samza), and time series databases (OpenTSDB) have emerged. These systems were created with a single goal in mind, and their simplicity makes it easier and more cost-effective to build distributed systems on commodity hardware. Often, the same data set needs to be injected into multiple specialized systems. For example, when application logs are used for offline log analysis, searching for individual log records is also indispensable. It is obviously impractical to build independent workflows to collect each type of data and then import them into their own dedicated systems. Using cloud messaging The queue Kafka version serves as a data transfer hub, and the same data can be imported into different dedicated systems.
Cloud Message Queue Kafka version has the following advantages as a data transfer hub:
High-capacity storage: Able to store high-capacity data on commercial hardware to achieve a horizontally scalable distributed system.
One-to-many consumption model: publish/subscribe model, which supports the same data set to be consumed multiple times at the same time.
Supports both real-time and batch processing: supports local data persistence and Page Cache, and can simultaneously transmit messages to real-time and batch consumers without performance loss.
Cloud message queue RabbitMQ
Open source comparison
Comparing Cloud Message Queue RabbitMQ Edition and open source RabbitMQ from three aspects: performance, stability, and functionality, Cloud Message Queue RabbitMQ Edition has more advantages.
Product Features
Compatible with open source RabbitMQ, provides complete operation and maintenance support, and supports multiple message types.
Usage restrictions
There are restrictions on clusters, interface calls, and characters. When using Cloud Message Queuing RabbitMQ version, be careful not to exceed the corresponding limits to avoid program exceptions.
Product advantages
Flexible and easy to use
Ready out of the box
Fully compatible with standard protocols, fully compatible with the RabbitMQ open source community, and quickly migrated to the cloud.
Fully managed service
Cloud Message Queue RabbitMQ version service is a cloud messaging service that requires no deployment and no operation and maintenance. Relying on Alibaba Cloud's professional automated operation and maintenance team, multi-dimensional inspections of core links are performed at the second and minute level, comprehensive health diagnosis reports are provided, and alarms and timely recovery are performed for unhealthy states to ensure service availability and business continuity. .
High availability
Provide a clear SLA to ensure high availability in multiple availability zones. Even if the entire computer room is unavailable, messaging services can still be provided normally.
Cluster distributed deployment, service nodes are stateless and have no single point. Even if a single node is unavailable, request Failover to other nodes to ensure the normal provision of message services.
Function enhancement
Delayed message
Supports delayed messages, achieving second-level accuracy and no first-in, first-out restrictions.
It is simple and easy to use. You only need to set one parameter in the code to solve the pain points of the open source RabbitMQ non-delay queue.
High-precision support, the shortest can be accurate to the second level, and the longest can last for 1 day.
High performance, performance is consistent with ordinary messages.
Multiple copies of data redundancy ensure message recovery.
message retry
Retries for one minute, up to 16 retries.
Messages that exceed the maximum number of retries enter the dead letter queue. Dead letter messages can be queried and exported.
Dead LetterExchange
Enhanced dead-letter Exchange not only supports messages with negative responses, but also supports messages that fail to be retried to automatically enter the dead-letter Exchange to ensure that messages are not lost.
Super performance
Platform performance upper limit
Supports millions of queues, horizontal expansion brings linear growth in performance, and there is no concurrency limit on performance.
Single queue expansion
Solve the performance bottleneck of RabbitMQ single queue, support horizontal expansion of single queue, and have no concurrency limit on performance.
Elastic scaling
It has strong scalability, linear performance enhancement, elastic expansion and contraction according to business needs, and is transparent to users.
Massive accumulation
In the case of massive message accumulation, high performance is always maintained without affecting the normal services of the cluster.
The producers and consumers of messages are isolated to meet the large concurrency of producers and stable consumption by consumers.
Safe and reliable
Permission granularity
Fine-grained permission control for instances, Vhosts, Queues and Exchange, and secure access control for each message request.
Master and sub-accounts
Comprehensive integration with Alibaba Cloud RAM master and sub-accounts, black and white lists, STS and other functions.
Encrypted transmission
Supports TLS transmission encryption protocol.
Application scenarios
Asynchronous decoupling
It can be used for communication between different microservices after a single application is disassembled into microservices. The benefit of application decoupling is that iterations of different applications no longer depend on each other, while the benefit of asynchronous communication is that data no longer needs to be processed immediately. Asynchronous decoupling can effectively shorten the data link length and improve data processing efficiency.
Peak shaving and valley filling
When large-scale events bring high traffic pulses, failure to protect accordingly can easily lead to system overload or even crash. Excessive restrictions can lead to a large number of failed requests, affecting user experience.
Cloud Message Queue RabbitMQ version can cut peaks and fill valleys. Its high-performance message processing capabilities can handle traffic pulses without being overwhelmed, ensuring system availability while improving user experience through fast and effective request response technology. Its massive message accumulation capability ensures that downstream businesses run smoothly and stably within a safe level and avoids the impact of traffic peaks.
Distributed cache synchronization
A large number of concurrent accesses to the database can result in long page response times. Build a distributed cache through Cloud Message Queue RabbitMQ version to support real-time notification of data changes, effectively reduce page response time, and meet large access requirements for changes.
Cloud Message Queue MQTT
It is a lightweight message middleware launched by Alibaba Cloud for the mobile Internet and Internet of Things fields. If traditional message queue middleware is generally used between microservices, then the Cloud Message Queue MQTT version suitable for the Internet of Things realizes message passing between the end and the cloud and the true interconnection of everything. This article introduces the message sending and receiving model, product advantages and application scenarios of Cloud Message Queue MQTT version.
Core idea
Topic
: Message topic, first-level message type, to which the producer sends messages.
producer
: Also known as the message publisher, responsible for producing and sending messages to Topic.
consumer
: Also called a message subscriber, responsible for receiving and consuming messages from Topic.
information
: The data sent by the producer to the Topic and finally transmitted to the consumer.
rule
Cloud Message Queue MQTT version is a resource that enables data interoperability with other Alibaba Cloud products.
Messaging model
It mainly includes the following two message sending and receiving models:
Terminal and cloud service interaction model
Cloud Message Queue MQTT version connects the terminal and the cloud to achieve two-way communication between the device and the cloud. The device can directly communicate with cloud business applications through the Cloud Message Queue MQTT version, and can also achieve cross-product interoperability of message data with other Alibaba Cloud products.
Typical application scenarios of this model are reporting status data of smart devices or issuing instructions for cloud control applications.
Terminal and terminal interaction model
This model is suitable for data communication between mobile apps or devices. Typical scenarios are two users directly chatting messages in the IM communication scenario, and the App controlling smart devices in the smart device scenario. In this model, message producers and consumers are distributed on terminal devices and connected to the cloud message queue MQTT version product through the MQTT protocol.
Based on the above two message sending and receiving models, developers using Cloud Message Queue MQTT version can be divided into two categories: terminal and cloud.
Product advantages
Seamless migration
Compatible with any SDK that supports MQTT 3.1.1 protocol, supports WebSocket protocol, and covers most mobile development platforms and languages.
high performance
It supports online connections of tens of millions of devices, millions of concurrent messages, trillions of transfers, and millisecond-level push; distributed architecture design, no single point bottleneck, and unlimited horizontal expansion between components.
Safe and reliable
Supports device-level permission control, temporary token service, and TLS 1.2 version secure transmission protocol to ensure user data is safe and reliable.
Natural interoperability
The message exchange between Cloud Message Queue MQTT version and Cloud Message Queue RocketMQ version enables two-way communication between the device and the cloud, making it more efficient and reliable.
Application scenarios
It has multi-protocol, multi-language and multi-platform support capabilities, and is widely used in the fields of mobile Internet and Internet of Things, covering a variety of application scenarios such as mobile live broadcast, Internet of Vehicles, financial payment, smart catering, and instant chat.
Messaging service MNS
Message Service MNS (Message Service) is an efficient, reliable, secure, convenient and elastically scalable distributed messaging service. Messaging service MNS can help developers freely transfer data and notification messages between distributed components of the application, thereby building a loosely coupled system. Messaging service MNS supports queue model and topic model.
queue model
The queue model provides a high-reliability, high-concurrency one-to-one consumption model, that is, each message in the queue can only be consumed by a certain consumer.
The queue is like a revolving sushi restaurant. There are multiple sushi chefs (producers) in the sushi restaurant making exquisite sushi. Each piece of sushi is unique. Customers (consumers) can take the sushi they like from the conveyor belt for eating (consumption).
Product advantages
Simple and easy to use
No operation and maintenance required, no need to build messaging service by yourself.
Easy access, standard HTTP RESTful interface.
Multiple language SDK support, including C, Java, C#, Python, PHP, Go, etc., and continues to be enriched.
Stable and reliable
Three copies of the message, high reliability.
High service availability.
Unique mechanism ensures Always Writable.
Security
Multi-level security protection and prevention of DDoS attacks.
Multi-user isolation mechanism, each user is equipped with an independent namespace.
Supports granting different access permissions to different RAM users.
Supports HTTPS and VPC access.
Large-scale and high scalability performance
The number of queues and queue storage capacity are highly scalable.
The service scale is automatically expanded and transparent to users.
Provide services in multiple regions around the world.
Fast and stable
Fast and direct, it only takes 3 seconds to connect to users.
Triple play exclusive channel.
Dedicated line computer room supports large capacity and high concurrency.
The arrival rate is high and it is connected to the number portability platform of the Ministry of Industry and Information Technology in real time.
Carrier-grade operation and maintenance guarantee, real-time monitoring and automatic switching.
Intimate technical service
Continuously enrich the sharing of best practice cases.
24/7 work order support.
Customizable on-site support available.