RabbitMQ Concepts

• Module 4 discusses message brokers and their architecture

Agenda

• Will cover general messaging concepts.
• Will discuss message brokers and SOA architecture.
• Will Cover messaging systems, including JMS, AMQP, and Big Data message brokers.

What is Messaging?

• Messaging a method of communication between software components or applications.
• Messages contain arbitrary data exchanged between processes.
• A broker facilitates message exchange.
• Messaging systems operate on a peer-to-peer basis
• Clients send and receive messages via a messaging server.
• Messages can be in various formats, commonly JSON and have headers and body

Distributed Systems – Synchronous Programming

• LPC (Local Procedure Call) occurs within the same process, possibly between threads.
• RCP(Remote Procedure Call) occurs between processes and can be deployed on different servers.
• RPC is not LPC simplified by a longer wire
• Today, most applications are distributed, thinking on cloud native
• Synchronous programming relies on connecting applications, often using RPC.
• RPC is unreliable.

Loosely and Tightly Coupled Applications

• Messaging enables distributed communication that is loosely coupled.
• Tightly coupled applications require N * (N-1)/2 connections.

Enterprise Messaging Systems

• Message-oriented middleware (MOM) enables loosely coupled applications.
• MOM provides asynchronous data delivery between applications and assures message delivery.
• It relieves application programmers from dealing with RPC and networking details

Messaging Benefits and Problems

• Benefits includes scale, failure tolerance, retries for missed messages
• Problems include disconnected systems, difficult correlation, and complex response handling

Publish/Subscribe Domain

• A messaging paradigm where a message from a publisher can have multiple consumers.
• The destination in this domain is called a Topic.
• Clients can receive messages after subscribing to topics.
• Subject to Quality of Service, Connectivity and Selection

Point-to-Point Domain

• Each message is addressed to a specific queue, where only one client can obtain the message sequentially and in FIFO order.
• Queues holds information as the destination in the Point-to-Point (PTP) domain.
• Queues retain messages until consumed or expired.

JMS Message Types

• TextMessage: Includes standard java.lang.String.
• MapMessage: Includes name/value pairs with String names and primitive types; entries are accessed sequentially or randomly.
• BytesMessage: includes a stream of uninterpreted bytes.
• StreamMessage: includes a stream of primitive values that are filled and read sequentially.
• ObjectMessage: includes Serializable Java object.
• Message: Empty, has no body, holds only header fields and properties.

Consuming Messages in Queue and Topic

• Asynchronously, a message listener is registered with a consumer.
• Synchronously, a message is explicitly retrieved from a destination, which blocks until a message is available.

JMS Transactions effects on Producers

• commit() sends staged messages.
• rollback() disposes staged messages.

JMS Transactions effects on Consumers

• commit() disposes of staged messages.
• rollback() resends staged messages.

Service Oriented Architecture - SOA

• A architecture focuses on discrete services instead of monolithic design.
• Evolves distributed computing using request-reply paradigm for synchronous and asynchronous applications.
• Application business logic and functions are modularized and presented as services.
• It allows a flexible method of structuring and managing heterogeneous component-based systems

Characteristics of SOA Services

• Services are loosely-coupled.
• Uses platform-independent, self describing interfaces (XML).
• Messages are formally defined (XSD).
• can be discovered (UDDI)
• QoS (Quality of Service) is defined in policies (security, authentication, authorization, reliability).
• can be provided on any platform.

Anatomy of a Service

• Includes a service consumer interacting through an interface proxy with new, wrapped legacy, or composite services

SOA Evolution (1) - Plate of Spaghetti

• Characterized by heterogeneous systems with proprietary interfaces.

SOA Evolution (2) – Service Based Integration

• Uses heterogeneous systems sharing data and functionality via a standard message format.
• Legacy systems made accessible by a service.
• At the time SOA was common, SOAP was the protocol that was used.
• Includes SOAP for messaging, WSDL for description, and UDDI for discovery.

SOA Evolution (3)-Service Oriented Architecture

• Dynamic discovery, security, and governance.
• Use of service brokers/ Enterprise Service Bus (ESB).

SOA Evolution (4)-Business Oriented SOA

• Focuses on Business Process Management, just-in-time integration, and B2B on demand.
• Utilizes service brokers, and adapts through Business Rules, flows, and adaptors

Message Broker Types

• JMS Implementations - Common API but without a common protocol – examples are WebSphere, JBoss, and ActiveMQ
• AMQP - Advanced Message Queueing Protocol with common protocol but no common API – example is RabbitMQ
• Big Data Message Brokers - Supports the 3 V's (Volume, Velocity, Variety) – example is Kafka

Rabbit MQ

• It's open source.
• written in Erlang, provides high scalability and robustness.
• supports AMQP, the standard messaging protocol

Rabbit MQ – Message Broker

• Built around exchange, binding, and queue primary items
• Publishers send messages to exchanges Exchanges are bound to queues.
• Consumers consume from queues.

Rabbit Exchanges

• There are 4 different exchange types
• Direct - Messages are routed to queues whose binding key matches the message key
• Fan out - Messages are transmitted to queues bound to it.
• Topic - Matches the routing key and the routing pattern specified in the binding.
• Headers - Uses message header attributes for routing.

Rabbit Queues & Bindings

• Queues can have Name, Durability, Exclusivity, Delete semantics, Other arguments
• queues are bound to exchanges via a routing key.
• Exchanges are used by publishers, consumers don't care about exchanges
• Bindings require the exchange to bind to, the queue, and routing key
• Different exchange types process bindings differently

Rabbit API

• ConnectionFactory creates connections.
• Connection manages single connection.
• Channel is a logical connection to Rabbit.
• exchangeDeclare declares a new exchange.
• queueDeclare declares a new queue (with or without name).
• queueBind binds a queue to an exchange.
• basicPublish publishes a message.
• basicConsume consumes a message and uses a callback.

Direct Exchange

• Messages go straight to a queue
• Exchange is given a routing key that tells the message to which queue to go.
• Default - RK the queue name
• The routing matches one single queue.
• There can be no wild cards

Message ack and durability

• Acknowledgement is sent by the consumer
• The default timeout is 30 minutes from consumers
• There is also manual acknowledgement.

Message Durability

• Messages can be lost if Broker down
• Broker can persist message, which means need to mark the queue and message as durable
• Set up queue declare in both producer and consumer

Publishing Durable Messages with Ack

• Need to set the properties as pika.BasicProperties and delivery_mode=pika.spec.PERSISTENT_DELIVERY_MODE
• Use queue for a routing key

Consuming Durable Messages with Ack

• If the connection is closed, the broker will re-queue it for other consumers.
• channel.basic_qos and prefetch_count=1 required

RabbitMQ vs. JMS Providers

• JMS supports queue - 1 : 1, with publish/subscribe – 1:many
• RabbitMQ supports the AMQP model which has 4 models with direct, fanout, topic, headers.
• JMS's data types has more differences
• JMS sends to queue and topic directly, but RabbitMQ first sendds to broker

Fan out Exchange – Deliver to multiple consumers

• The Fanout exchange broadcasts all the messages it receives to queues that are known
• All consumers get the message, there is no routing
• Need to bind any queue to the exchange

Distributed System

• A group of resources/independent components on machines, meant to meet a certain goal
• Usually consists of multiple workers or nodes, like Kafka
• Nodes reqauire coordination to ensure progress
• Coordination also requires communicaiton

Controller

• The top of the controller of supervisor
• A worker elected among peers
• Has critical responsibilities for workers and their progress

Controller in cluster

• If a task us unavalable the controller will assign another one
• Team becomes a designated controller

Choosing leader

• Task given to 1, the worker peers will fail
• Kafka has high risk
• Controllers promote worker
• Workers peer for the promotion

Kafka

• High Volume, Velocity and Variety
• Is a publish and subscribe messaginf retooled as commit log
• Operational monitoring
• Supports storm

Messaging System

• Logical and spans the entire cluster of brokers
• Need to know each topic
• Represented as log
• Stream that orders it sequentially
• Stored persistent and immutable
• Payload Consumers can affect other consumers and producers

Explore the fundamental concepts of RabbitMQ, including exchanges, queues, bindings, and message routing. Understand how RabbitMQ handles message distribution, redundancy, and task management among workers. Learn about key components and functions within the RabbitMQ architecture.