Zabbix HA Mode Configuration PDF

STARTING ZABBIX IN HA MODE To start Zabbix server in HA mode, new configuration parameters are added Y P HANodeName specifies the name of the node Must be unique for each node Without HANodeName specified, Zabbix server will start in a standalone mode ## Option: HANodeName # The high availability...

STARTING ZABBIX IN HA MODE To start Zabbix server in HA mode, new configuration parameters are added Y P HANodeName specifies the name of the node Must be unique for each node Without HANodeName specified, Zabbix server will start in a standalone mode ## Option: HANodeName # The high availability cluster node name. # When empty server is working in standalone mode. HANodeName=zbx-node-01 T N O C E D NodeAddress must match IP or FQDN name of the Zabbix server node This parameter will be used by Zabbix frontend to connect to active node Without this parameter frontend will be unable to display status, queue etc. U T ## Option: NodeAddress # IP or hostname to define how frontend should connect to the server. # Format: <address>[:port] NodeAddress=zabbix-node-01.example.com S 6.0 Certified Professional ● Day 1 © 2023 by Zabbix. All rights reserved Theory 41 DETECTING ACTIVE NODE Zabbix frontend will autodetect the active node: Y P Zabbix server address and port must be undefined in the frontend configuration // Uncomment and set to desired values to override Zabbix hostname/IP and port. // $ZBX_SERVER = ''; // $ZBX_SERVER_PORT = ''; O C Frontend reads settings from the nodes table in a Zabbix database T N The node address of the active node is used as the Zabbix server address +------------------+-------------+----------+-------+------------+--------+--------------------------+ | ha_nodeid | name | address | port | lastaccess | status | ha_sessionid | +------------------+-------------+----------+-------+------------+--------+--------------------------+ | ckzftwydt0001f6b | zbx-node-01 | 10.0.0.1 | 10051 | 1644427911 | 3 | ckzftwyd00 00f6qrqnm7319 | | awfytsxre0023c95 | zbx-node-02 | 10.0.0.2 | 10051 | 1644427905 | 0 | awfytsxre003c95qtdfr5463 | +------------------+-------------+----------+-------+------------+--------+--------------------------+ U T E D Multiple frontend nodes may be safely deployed S A session is stored on the user side in cookie and signed A cookie is validated to contain valid session id for each session request 6.0 Certified Professional ● Day 1 © 2023 by Zabbix. All rights reserved Theory 42 HA SCHEMATICS Y P DBHost=zabbix-db.example.com HANodeName=zbx-node-01 NodeAddress=10.0.0.1 E D U T S T N O C DBHost=zabbix-db.example.com HANodeName=zbx-node-02 NodeAddress=10.0.0.2 $DB['SERVER'] = zabbix-db.example.com; SELECT address FROM ha_node WHERE status=3 +--------------------------+-------------+----------+-------+------------+--------+ | ha_nodeid | name | address | port | lastaccess | status | +--------------------------+-------------+----------+-------+------------+--------+ | cl3k5nae20001gxtyq7qf5o | zbx-node-01 | 10.0.0.1 | 10051 | 1644427911 | 3 | | awfytsxre0023c95ct38df4 | zbx-node-02 | 10.0.0.2 | 10051 | 1644427905 | 0 | +--------------------------+-------------+----------+-------+------------+--------+ 6.0 Certified Professional ● Day 1 © 2023 by Zabbix. All rights reserved Theory 43 FRONTEND NODES All the frontend server nodes will connect to: Y P The same database server The active Zabbix server node only O C All the Zabbix server nodes will connect to a database to send heartbeat T N Only active node will detect problems and perform writes to the database Standby nodes will watch the status of the active node U T E D S 6.0 Certified Professional ● Day 1 Z © 2023 by Zabbix. All rights reserved Z Z Theory 44 ZABBIX HA NODE TYPES Zabbix has four statuses for an HA node: Y P Active 3 only one node can be active at a time Standby 0 multiple nodes can be in a standby mode Stopped 1 a node was previously detected, but it is shutdown now Unavailable 2 a node was previously detected, but was lost without a shutdown E D U T S 6.0 Certified Professional ● Day 1 T N O C © 2023 by Zabbix. All rights reserved Theory 45 On every node a special process called HA Manager is started: HA MANAGER Y P This is the only active Zabbix process on standby nodes that is started after the main process O C # ps ax | grep zabbix_server 12284 ? S 0:00 /usr/sbin/zabbix_server -c /etc/zabbix/zabbix_server.conf 12294 ? S 0:00 /usr/sbin/zabbix_server: ha manager T N All the other Zabbix processes will be started when failover happens E D # ps ax | grep zabbix_server 12284 ? S 0:06 /usr/sbin/zabbix_server -c /etc/zabbix/zabbix_server.conf 12294 ? S 0:11 /usr/sbin/zabbix_server: ha manager 12307 ? S 0:09 /usr/sbin/zabbix_server: service manager #1 12308 ? S 6:37 /usr/sbin/zabbix_server: configuration syncer 12315 ? S 0:10 /usr/sbin/zabbix_server: alert manager #1 12316 ? S 0:00 /usr/sbin/zabbix_server: alerter #1 started 12317 ? S 0:00 /usr/sbin/zabbix_server: alerter #2 started 12326 ? S 0:05 /usr/sbin/zabbix_server: timer #1 12341 ? S 0:07 /usr/sbin/zabbix_server: self-monitoring 12342 ? S 0:04 /usr/sbin/zabbix_server: task manager 12343 ? S 1:28 /usr/sbin/zabbix_server: poller #1 U T S 6.0 Certified Professional ● Day 1 © 2023 by Zabbix. All rights reserved Theory 46 ZABBIX AGENT CONFIGURATION Zabbix agent can communicate with an HA cluster: Y P Zabbix passive agent mode requires all the nodes to be written in the Server parameter Nodes are specified on a comma-separated list O C ### Option: Server # List of comma delimited IP addresses or DNS names of Zabbix # Incoming connections will be accepted only from the hosts listed here. Server=zbx-node01.example.com,zbx-node02.example.com T N Zabbix active agent requires cluster nodes to be specified in the ServerActive parameter E D Cluster nodes need to be separated by a semicolon Supported only starting from Zabbix agent 6.0 U T ### Option: ServerActive # List of comma delimited IP addresses or DNS names pairs of clusters of Zabbix # servers for active checks. # Cluster nodes need to be separated by semicolon. ServerActive=zbx-node01.example.com;zbx-node02.example.com S ! HA options are fully supported on Zabbix agent 6.0 or later 6.0 Certified Professional ● Day 1 © 2023 by Zabbix. All rights reserved Theory 47 ZABBIX PROXY CONFIGURATION HA mode is supported by Zabbix proxies: Y P Zabbix proxy in passive mode must accept connections from all the Zabbix cluster nodes O C ### Option: Server # List of comma delimited IP addresses or DNS names of Zabbix server. # Incoming connections will be accepted only from the addresses listed here. Server=zbx-node01.example.com,zbx-node02.example.com T N Zabbix proxy in active mode must connect to all the Zabbix server cluster nodes A semicolon must be used to separate node names E D ### Option: Server # IP address or DNS name (address:port) or cluster (address:port;address2:port) of # Zabbix server to get configuration data from and send data to. # Cluster nodes need to be separated by semicolon. Server=zbx-node01.example.com;zbx-node02.example.com U T S 6.0 Certified Professional ● Day 1 © 2023 by Zabbix. All rights reserved Theory 48 HA FAILOVER Zabbix HA supports automatic failover only: Y P Zabbix server restart will initiate automatic failover to another node All the nodes report their status every 5 seconds O C When a node shuts down, the standby node, which first detects the lost node, will take over T N Z Z check ha_node table Set status = 1 Z E D No active nodes detected; I will take over If the node is lost and does not respond in time U T The clock will keep on ticking until it reaches the failover delay (1 minute by default) When the failover delay is over, one of the standby nodes will take over S 6.0 Certified Professional ● Day 1 Z Z check ha_node table © 2023 by Zabbix. All rights reserved Z lastaccess is not updated; I will wait 1 minute and take over Theory 49 ZABBIX CLUSTER CONFIGURATION It is possible to adjust cluster parameters using runtime commands: Y P Runtime commands can be executed on an active node only Adjust failover delay using ha_set_failover_delay runtime command (10s to 15 min) # zabbix_server -R ha_set_failover_delay=5m HA failover delay set to 300 seconds T N O C Nodes can be removed using ha_remove_node runtime command Use node id or node name Only stopped or unreachable nodes can be removed E D # zabbix_server -R ha_remove_node=zbx-node3 Removed node "zbx-node3" with ID "ckye7ty900001sjqtfcw1tvkm" U T Display the current HA status with ha_status command # zabbix_server -R ha_status Failover delay: 300 seconds Cluster status: # ID 1. ckvciyod80001oaqr5o6eh5uu 2. ckvcj8gfr0001nmqsrewumcti 3. ckye7ty900001sjqtfcw1tvkm S 6.0 Certified Professional ● Day 1 Name zbx-node1 zbx-node2 zbx-node3 Address zbx-node01.example.com:10051 zbx-node02.example.com:10051 zbx-node03.example.com:10051 © 2023 by Zabbix. All rights reserved Status active standby standby Last Access 4s 5s 3s Theory 50 SETUP HA MODE How to setup Zabbix in an HA mode: Y P Start Zabbix server in an HA mode on all the nodes (specify the node name and address) Make sure $ZBX_SERVER and $ZBX_SERVER_PORT parameters are commented in frontend configuration Specify Server parameters for all the Zabbix proxies T N O C Specify Server and ServerActive parameters for all the Zabbix agents Configure other network devices for an HA environment U T E D S 6.0 Certified Professional ● Day 1 © 2023 by Zabbix. All rights reserved Theory 51 PRACTICAL SETUP Y P 1) Switch Zabbix server to High availability mode: Define HANodeName: Define NodeAddress zabbix-node-01 127.0.0.1:10051 2) Install Podman container engine: T N O C Create a new pod, and name it "zabbix" with three mapped ports (10055,10056 and 10061 to 10051) Start and run two Zabbix server 6.0 nodes in Podman in HA mode Check the HA cluster status E D 3) Manage the HA cluster: Stop Zabbix server node 3 Remove Zabbix server node 3 from the Zabbix HA cluster Stop Zabbix server node 1 and look at the HA cluster status Start Zabbix server node 1 and look at the HA cluster status Restart Zabbix server node 2 and look at the HA cluster status U T S 4) Stop Zabbix server node 2 6.0 Certified Professional ● Day 1 © 2023 by Zabbix. All rights reserved 30 minutes Practical task No: 3 52 Y P E D Proxies U T S 6.0 Certified Professional ● Day 1 T N O C © 2023 by Zabbix. All rights reserved 30 minutes 53 ZABBIX PROXY Zabbix proxy collects data on behalf of Zabbix server: Y P Proxies will receive monitoring configuration from Zabbix server automatically Collected data is sent to Zabbix server every second by default O C All the data is stored into a local proxy database for a limited amount of time The data is still collected by the proxy in case of network issues between the server and proxy T N Proxies preprocess data (if required) to offload Zabbix server resources U T E D S 6.0 Certified Professional ● Day 1 Zabbix proxy © 2023 by Zabbix. All rights reserved Theory 54 DISTRIBUTED MONITORING PROXY OVERVIEW Zabbix proxies are a reliable way of monitoring a distributed IT infrastructure: Y P Monitor behind firewalls or in DMZ Collect data in case of network issues O C Remotely run custom scripts on monitored hosts Preprocess data on proxies to offload Zabbix server E D U T i S T N One firewall rule Multiple firewall rules With Zabbix proxy Without Zabbix proxy https://www.zabbix.com/documentation/current/manual/distributed_monitoring 6.0 Certified Professional ● Day 1 © 2023 by Zabbix. All rights reserved Theory 55 DISTRIBUTED MONITORING OVERVIEW Zabbix has centralized management of all the proxies from Zabbix frontend: Y P Proxy collects performance and availability data on behalf of the Zabbix server Zabbix server controls monitoring configuration of all proxies O C Zabbix proxy supports the same OS platforms and DB engines as Zabbix server: It is possible to use different database engines on the server and proxies Proxies additionally support auto created SQLite3 database E D U T S ! T N SQLite3 PostgreSQL MySQL Don't use the same database and schema name for a proxy & server 6.0 Certified Professional ● Day 1 © 2023 by Zabbix. All rights reserved Theory 56 PROXY BUFFERS Proxy will use a database as a buffer in case of communication problems: Y P 1 hour of data is saved by default Up to 30 days of data can be saved O C ### Option: ProxyOfflineBuffer # Proxy will keep data for N hours in case if no connectivity with Zabbix Server. # Older data will be lost. # Range: 1-720 # Default: 1 ProxyOfflineBuffer=24 T N E D It is possible to store the data locally even if the data have been synced with the server: U T Not stored by default S ### Option: ProxyLocalBuffer # Proxy will keep data locally for N hours, even if the data have been synced with the server. # This parameter may be used if local data will be used by third party applications. # Range: 0-720 ProxyLocalBuffer=0 6.0 Certified Professional ● Day 1 © 2023 by Zabbix. All rights reserved Theory 57 PROXY SETUP STEPS Follow these steps to add a new proxy to your Zabbix instance: Y P Install proxy (using packages, docker, compile binary, etc.) Create a proxy database: Create DB schema in a similar way as for Zabbix server For SQLite3 specify path to the database file Update proxy configuration file T N Options depend on the proxy mode Start the proxy O C E D Add proxy in the frontend (Administration > Proxies) Configure hosts to be monitored by the proxy in the frontend U T Configure Zabbix agents to work with the proxy: For passive agents Server parameter must accept connections from proxy For active agents ServerActive must point to the proxy address S 6.0 Certified Professional ● Day 1 © 2023 by Zabbix. All rights reserved Theory 58 ACTIVE PROXY An active proxy will connect to Zabbix server and request the configuration data: Y P Data sender process connects to a free trapper on Zabbix server Up to 2 trappers per proxy may be utilized simultaneously One trapper for data upload One trapper for configuration download E D U T Zabbix proxy active S Data collectors 6.0 Certified Professional ● Day 1 T N Data sender © 2023 by Zabbix. All rights reserved O C Zabbix server Zabbix trapper Theory 59 ACTIVE PROXY CONFIGURATION For active proxies only Zabbix proxy configuration file is used: Y P Configuration file: /etc/zabbix/zabbix_proxy.conf O C ProxyMode 0 (active) Server IP address or DNS name of Zabbix server Hostname Proxy name as configured in the frontend ProxyOfflineBuffer Controls for how long data is kept locally if proxy can't contact the server ProxyLocalBuffer How long to preserve data in the proxy database after it is sent ConfigFrequency Controls how often proxy requests configuration information from Zabbix server DataSenderFrequency Controls how often data is sent to Zabbix server HeartbeatFrequency Makes proxy contact Zabbix server even if there is no new data to transmit E D U T S T N 6.0 Certified Professional ● Day 1 © 2023 by Zabbix. All rights reserved Theory 60