ZCP_day_2_slides.pdf

Y P T N E D O C U T CERTIFIED PROFESSIONAL TRAINING S 6.0 DAY 2 6.0 Certified Professional ● Day 2 © 2023 by Zabbix. All rights reserved COPYRIGHT NOTICE Rules Y P It is prohibited to make any video and/or audio recordings during the whole period of this course. This course is intended only for the officially enrolled student. Subject to the Copyright Notice below, the student is not allowed to share his credentials for attending this course, to allow others to join and take part, or otherwise make use of these Materials. Copyright notice © Zabbix, 2023. All rights reserved. T N O C Unless otherwise indicated, Zabbix owns the copyright and other intellectual property rights in the text, graphics, information, designs, data, verbal/audio/video presentations and files, comments, drawings, exam questions and exam answers, and other training content, lab manuals and practical tasks, and training courses themselves (further – Materials). E D The Materials are protected by watermarks, copyright statements, and other means. It is prohibited to remove any of watermarks and copyright statements, or in any other way to amend or change the content or appearance of the Materials. Any unauthorized reprint, publication, reproduction, sharing, or use of the Materials is prohibited. No part of the Materials may be reproduced, transmitted, or published in any form or by any means, electronic or mechanical, including photocopying, recording, or by any information storage and retrieval system without the express signed written permission from Zabbix. U T All course Materials made available to the student during the course of the training may be used solely by the student enrolled in the relevant course for personal and educational purposes only. Materials provided to the student should be treated as confidential information shared with the student only for the purpose of the student performing Zabbix Certified training. S The student acknowledges that damages alone would not be an adequate remedy for the breach of this copyright and the student shall be entitled to the granting of equitable relief concerning any threatened or actual breach of any of the provisions of this Copyright notice. 6.0 Certified Professional ● Day 2 © 2023 by Zabbix. All rights reserved Theory 2 AGENDA Advanced preprocessing ODBC Monitoring U T S LLD with dependent items 6.0 Certified Professional ● Day 2 T N O C LLD Prototypes LLD Filters and Overrides LLD Using SQL SNMP LLD E D LLD Internals Y P LLD basics © 2023 by Zabbix. All rights reserved 3 Y P E D ODBC monitoring U T S 6.0 Certified Professional ● Day 2 T N O C © 2023 by Zabbix. All rights reserved 20 minutes 4 Open Database Connectivity (ODBC) GET DATA FROM EXTERNAL DATABASE Y P Standard Programming Interface (API) for accessing database management systems This API is independent from any of the DBMS, or an operating system O C Zabbix may query any database, which is supported by ODBC: T N A unixODBC database driver should be installed for the database, which will be monitored Zabbix documentation has recommended settings for MySQL, PostgreSQL and MS SQL Server E D Database monitor item type must be used U T The Timeout parameter is used as the ODBC login timeout S Zabbix does not limit the query execution time 6.0 Certified Professional ● Day 2 © 2023 by Zabbix. All rights reserved Theory 5 DATABASE MONITORING - DATA FLOW Database monitoring begins with Zabbix making a call to unixODBC Driver Manager Y P The Driver Manager makes a call to the ODBC Driver The ODBC Driver accesses the database over the network using the database API Zabbix Server E D U T S Remote database server 6.0 Certified Professional ● Day 2 T N O C UNIXODBC ODBC Driver MySQL ODBC Driver ORACLE ODBC Driver MSSQL Database API Database API Database API MySQL Database © 2023 by Zabbix. All rights reserved Oracle Database MSSQL Database Theory 6 ODBC POLLERS ODBC monitoring is performed by ODBC poller processes Y P At least one ODBC poller must be started if ODBC checks are done ## Option: StartODBCPollers # Number of pre-forked ODBC poller instances. # # Mandatory: no # Range: 0-1000 # Default: StartODBCPollers=5 T N O C E D ODBC polling loop will work as follows: Retrieve ODBC items for an ODBC poller U T Prepare them (expand macros) S Requeue ODBC items, calculate next checks Clean the ODBC item cache 6.0 Certified Professional ● Day 2 © 2023 by Zabbix. All rights reserved Theory 7 DATABASE MONITORING - CONFIGURATION 1. Installing unixODBC driver on Zabbix server Y P To install MySQL database driver on RedHat based systems run: # dnf install unixODBC Out-of-the-box drivers are available for: MySQL MariaDB PostgreSQL Free Sybase & MS SQL Driver T N O C U T E D Default ODBC drivers might not work with the most recent database versions: Install the ODBC driver that matches DB version S Multiple ODBC drivers for different DB versions can be installed 6.0 Certified Professional ● Day 2 © 2023 by Zabbix. All rights reserved Theory 8 DATABASE MONITORING - CONFIGURATION Configuring unixODBC: Y P ODBC configuration is done by editing the /etc/odbcinst.ini and /etc/odbc.ini files odbcinst.ini is used to list the installed ODBC drivers Database driver name [MySQL] Description = ODBC for MySQL Driver = /usr/lib/libmyodbc5.so T N O C odbc.ini is used to define data sources/databases (create, if necessary) Data Source Name (DSN) E D [mysql01] Description = mysql01.zabbix.com database U T Driver name (odbcinst.ini) S Driver = MySQL Server = 10.132.11.56 It is possible to specify many additional parameters; the list depends on the driver used Database, User, Password, Socket etc. 6.0 Certified Professional ● Day 2 © 2023 by Zabbix. All rights reserved Theory 9 DATABASE MONITORING - CONFIGURATION Verify ODBC connection (-v for debug): Y P # isql -v <DSN> <MySQL user> <Password> Example: # isql mysql01 zabbix zabbix +------------------------------------+ | Connected! | | | | sql-statement | | help [tablename] | | quit | | | +------------------------------------+ SQL> E D U T S T N O C Test SQL queries: SQL> show tables 6.0 Certified Professional ● Day 2 © 2023 by Zabbix. All rights reserved Theory 10 DATABASE MONITORING - CONFIGURATION Add an item in Zabbix frontend: Y P Key: db.odbc.select[<unique short description>,<dsn>,<connection string>] Returns one value (the first column of the first row of the SQL query result) If a query returns more than one column, only the first column is read If a query returns more than one line, only the first line is read The query must return a value (empty result = not supported item) E D U T S 6.0 Certified Professional ● Day 2 T N © 2023 by Zabbix. All rights reserved O C DSN from odbc.ini Theory 11 DATABASE MONITORING - CONFIGURATION Key: db.odbc.get[<unique short description>,<dsn>,<connection string>] Y P Returns multiple rows/columns in JSON format May be used as a master item that collects all data in a single system call JSONPath preprocessing must be used in dependent items to extract individual values E D U T S T N O C Get all data in master item Extract data using preprocessing in dependent item 6.0 Certified Professional ● Day 2 © 2023 by Zabbix. All rights reserved Theory 12 DATABASE MONITORING - CONFIGURATION Alternatively, ODBC item keys can use a "connection string" instead of the DSN: db.odbc.select[<unique short description>,<dsn>,<connection string>] db.odbc.get[<unique short description>,<dsn>,<connection string>] Syntax: Y P O C "Driver=<DriverName or </path/to/driver.so>;Database=<DB NAME>;Server=<IP/DNS>;Port=3306" T N If fields "Username" and "Password" are set, then: E D string: "UID=<user>;PWD=<password>" will be added to the "connection string" automatically U T Examples: S "Driver=MariaDB;Database=zabbix;Server={HOST.CONN};Port=3306" "Driver=/usr/lib64/libmaodbc.so;Database=zabbix;Server={HOST.CONN};Port=3306" "Driver=MariaDB;Database=zabbix;Server={HOST.CONN};Port=3306;UID=<username>;PWD=<pass>" 6.0 Certified Professional ● Day 2 © 2023 by Zabbix. All rights reserved Theory 13 PRACTICAL SETUP Y P 1) Modify zbx_monitor database user: grant SELECT on zabbix.* to zbx_monitor@localhost; 2) Install "mariadb-connector-odbc" O C 3) Create the new template "Training ODBC monitoring" T N Item: Number of hosts in Zabbix Key: db.odbc.select[zabbixhosts,,"Driver=/usr/lib64/libmaodbc.so;Database=zabbix; Server={HOST.CONN};Port=3306"] SQL query: SELECT count(*) FROM zabbix.hosts WHERE status IN (0,1) AND flags IN (0,4) 4) Check the results U T E D S 6.0 Certified Professional ● Day 2 © 2023 by Zabbix. All rights reserved 20 minutes Practical task No: 9 14 Y P E DPreprocessing U T S 6.0 Certified Professional ● Day 2 T N O C © 2023 by Zabbix. All rights reserved 20 minutes 15 ITEM- PREPROCESSING Transformations are executed in the order in which they are defined Y P The order can be changed by dragging and dropping All the preprocessing is done by Zabbix server and proxies If any of preprocessing steps fails, then: O C it is possible to react to errors and introduce recovery options. Otherwise , the item becomes unsupported E D U T i S T N https://www.zabbix.com/documentation/6.0/manual/config/items/preprocessing 6.0 Certified Professional ● Day 2 © 2023 by Zabbix. All rights reserved Theory 16 PREPROCESSING - HOW IT WORKS Every collected value goes through the preprocessing: Y P Even the values without preprocessing steps are sent to the preprocessing manager The values collected by proxies are processed on proxies and skip queue on a server O C Data are passed to the preprocessing manager and placed into the preprocessing queue: T N The preprocessing worker takes the value and performs all steps (if any exists) E D After all the steps are done, the value is put back into the queue as processed All the processed values are flushed into a history cache U T S 6.0 Certified Professional ● Day 2 © 2023 by Zabbix. All rights reserved Theory 17 PREPROCESSING WORKERS Multiple preprocessing workers can process values in parallel: Y P ### Option: StartPreprocessors # Number of pre-forked instances of preprocessing workers. # The preprocessing manager process is automatically started when preprocessor worker is started. # Range: 1-1000 # Default: 3 StartPreprocessors=10 T N O C Watch Zabbix internal metrics to monitor the preprocessing performance: E D The items waiting in preprocessing queue How busy are the preprocessing workers U T S Optimal number of preprocessing workers can be determined by many factors: The number of items that require any preprocessing steps Heavy operations like parsing of large XML / JSON chunks may require more preprocessors. 6.0 Certified Professional ● Day 2 © 2023 by Zabbix. All rights reserved Theory 18 THROTTLING "Throttling" preprocessing step discards a value, if it has not changed: Y P Only one throttling option can be specified for an item Discard unchanged Discard unchanged with heartbeat T N O C If a value is discarded, history syncer has no idea that this value was collected No trigger expressions based on this value will be evaluated nodata() function will detect a false positive If history cache does not contain values for an hour, trends cannot be calculated E D U T S 6.0 Certified Professional ● Day 2 /dev/null © 2023 by Zabbix. All rights reserved Theory 19 CHECK FOR NOT SUPPORTED VALUE "Check for not supported value" is a very specific preprocessing step: Y P It is always placed as a first step (even if defined initially as the second, third, etc.) Custom on fail option is always enabled O C It can only fix problems detected during data collection time (wrong key, SNMP OID, etc.) Cannot fix problems with the wrong data type written into a database U T E D Collect data S 6.0 Certified Professional ● Day 2 T N Check not supported © 2023 by Zabbix. All rights reserved Write into database Theory 20 Y P E DPreprocessing U T S 6.0 Certified Professional ● Day 2 T N O C XML XPath © 2023 by Zabbix. All rights reserved 10 minutes 21 PREPROCESSING - XML PATH LANGUAGE (XPATH) XPath (XML Path Language) is a syntax for defining parts of an XML document Y P Navigate through elements and attributes in an XML document Extract a value or a fragment from XML data XML documents are treated as trees of nodes T N The node is selected by following a path or steps E D Zabbix supports XPath 1.0 version Example: U T S Path to a price is /bookstore/book/price i O C <?xml version="1.0" encoding="UTF-8"?> <bookstore> <book> <title lang="en">Harry Potter</title> <author>J K. Rowling</author> <year>2005</year> <price>29.99</price> </book> </bookstore> https://blog.zabbix.com/zabbix-xpath-preprocessing/7936/ 6.0 Certified Professional ● Day 2 © 2023 by Zabbix. All rights reserved Theory 22 XML XPATH FUNCTIONS XPath functions have the following four types: Y P Node functions, Numeric functions, String functions, Boolean functions <?xml version="1.0" encoding="UTF-8"?> <bookstore> string(/bookstore/book[3]/author) <book category="cooking"> <title lang="en">Everyday Italian</title> <author>Giada De Laurentiis</author> <year>2005</year> <price>30.00</price> </book> T N U T <book category="web"> <title lang="en">Learning XML</title> <author>Erik T. Ray</author> <year>2003</year> <price>39.95</price> </book> sum(/bookstore/book/price) 99.94 Count the number of books count(/bookstore/book) 3 Select the price node with a price that is less than 30 number(/bookstore/book[price<30]/price) 29.99 </bookstore> 6.0 Certified Professional ● Day 2 Erik T. Ray Select the sum of all prices E D <book category="children"> <title lang="en">Harry Potter</title> <author>J K. Rowling</author> <year>2005</year> <price>29.99</price> </book> S O C Select the author of the 3rd book © 2023 by Zabbix. All rights reserved Theory 23 PREPROCESSING - XML XPATH Y P <?xml version="1.0" encoding="UTF-8"?> <bookstore> <book category="cooking"> <title lang="en">Everyday Italian</title> <author>Giada De Laurentiis</author> <year>2005</year> <price>30.00</price> </book> U T <book category="web"> <title lang="en">Learning XML</title> <author>Erik T. Ray</author> <year>2003</year> <price>39.95</price> </book> S </bookstore> 6.0 Certified Professional ● Day 2 T N E D <book category="children"> <title lang="en">Harry Potter</title> <author>J K. Rowling</author> <year>2005</year> <price>29.99</price> </book> O C Select the entire XML part with the category "web" /bookstore/book[@category="web"] <book category="web"> <title lang="en">Learning XML</title> <author>Erik T. Ray</author> <year>2003</year> <price>39.95</price> </book> © 2023 by Zabbix. All rights reserved Theory 24 Y P E DPreprocessing U T S 6.0 Certified Professional ● Day 2 T N O C JSON Path © 2023 by Zabbix. All rights reserved 10 minutes 25 JSON FORMAT JSON stands for JavaScript Object Notation Y P Lightweight format for storing and transporting data Zabbix uses JSON format for communication and LLD JSON data is written as Name/Value pairs "firstName":"John" T N Objects can contain multiple Name/Value pairs: O C E D {"firstName":"John","lastName":"Doe"} JSON arrays are written inside square brackets; an array can contain objects: U T S { "employees":[ {"firstName":"John", "lastName":"Doe"}, {"firstName":"Anna", "lastName":"Smith"}, {"firstName":"Peter", "lastName":"Jones"} ] } 6.0 Certified Professional ● Day 2 © 2023 by Zabbix. All rights reserved Theory 26 PREPROCESSING - JSONPATH JSONPath is a query language for JSON Y P Features are similar to XPath Used for selecting and extracting JSON document’s property values O C A JSONPath expression begins with the dollar sign ($): Refers to the root element of a query $.object.category T N The dollar sign is followed by a sequence of child elements, which are separated: Using dot notation $.object.name E D Using square brackets $["object"]["name"] Filters are logical expressions used to filter arrays. U T @ represents the current array item or object being processed. S $.employees[?(@.firstName == "John")] JSONPATH supports the wildcard * operator: Returns all objects or elements regardless of their name. 6.0 Certified Professional ● Day 2 © 2023 by Zabbix. All rights reserved Theory 27 PREPROCESSING - JSONPATH JSONPath functionality is available as item value preprocessing step Y P Operators: Functions: E D Path $.object.name $["book"]["name"] U T $.object.history.length() i T N - + / * == != < <= > >= =~ ! || && avg min max sum length first S O C Result book.name contents number of object.history array elements $[?(@.name == "SomeBook")].price.first() price field of the first object with the name 'SomeBook' $[?(@.name == "SomeBook")].history.first().length() number of history array elements of the first object with the name 'SomeBook' https://www.zabbix.com/documentation/../preprocessing/jsonpath_functionality 6.0 Certified Professional ● Day 2 © 2023 by Zabbix. All rights reserved Theory 28 PREPROCESSING - JSONPATH EXAMPLES { "store": { "book": [ { "category": "reference", "author": "Nigel Rees", "title": "Sayings of the Century", "price": 8.95 }, { "category": "fiction", "author": "Evelyn Waugh", "title": "Sword of Honor", "price": 12.99 }, { "category": "fiction", "author": "J. R. R. Tolkien", "title": "The Lord of the Rings", "isbn": "0-395-19395-8", "price": 22.99 } ], "bicycle": { "color": "red", "price": 19.95 } } } S 6.0 Certified Professional ● Day 2 $.store.book[0].price O C 8.95 Select the price of the book with the name "Sword of Honor" T N $.store.book.[?(@.title == "Sword of Honor")].price E D U T Y P Select the price of the 1st book [12.99] $.store.book.[?(@.title == "Sword of Honor")].price.first() 12.99 Select all the elements of the "bicycle" array. $.store.bicycle { } © 2023 by Zabbix. All rights reserved "color": "red", "price": 19.95 Theory 29 Y P T N O C E D Preprocessing Prometheus U data T S 6.0 Certified Professional ● Day 2 © 2023 by Zabbix. All rights reserved 10 minutes 30 PROMETHEUS DATA COLLECTION Prometheus exporter is a free software application used for data collection: Y P It can report metrics from third-party systems such as Prometheus metrics Listens on some ports as a web server O C Zabbix can collect Prometheus data using HTTP item type T N Creates dependent items with Prometheus pattern and retrieves the specific metrics U T E D i S Prometheus format Extracted metrics Dependent item HTTP item Dependent item Dependent item https://github.com/prometheus/prometheus/wiki/Default-port-allocations 6.0 Certified Professional ● Day 2 © 2023 by Zabbix. All rights reserved Theory 31 PROMETHEUS FORMAT Prometheus format has a name combined with a series of labels with values <metric name>{<label name>=<label value>, ...} Y P O C The metric output is typically preceded by metadata lines: # HELP and # TYPE The # HELP string identifies the metric name and gives a brief description of it T N The # TYPE string identifies the type of metric Everything else that starts with a # is parsed as a comment. E D # HELP node_network_transmit_packets_total Network device statistic transmit_packets # TYPE node_network_transmit_packets_total counter node_network_transmit_packets_total{device="eth0"} 2.966359e+06 node_network_transmit_packets_total{device="eth1"} 5.703921e+06 # HELP node_network_receive_packets_total Network device statistic receive_packets. # TYPE node_network_receive_packets_total counter node_network_receive_packets_total{device="eth0"} 3.008294e+06 node_network_receive_packets_total{device="eth1"} 7.833472e+06 U T S 6.0 Certified Professional ● Day 2 © 2023 by Zabbix. All rights reserved Theory 32 PROMETHEUS PATTERN Prometheus query language is used to extract data: Y P Pattern <metric_name>{<label name>="<label value>"} - selects by a metric name and label <metric_name>{<label name>=~"<regex>"} - selects by matching a regular expression Function value, label Aggregation functions (sum, min, max, avg, count) Output T N E D O C The label name is returned (used only with the "label function") Pattern can match multiple rows only if aggregation functions are used U T S 6.0 Certified Professional ● Day 2 Pattern © 2023 by Zabbix. All rights reserved Function Output Theory 33 PRACTICAL SETUP Y P 1) Open HTTP on training.lan to see Prometheus data http://training.lan-zbxtr-YYYY.zabbix.training:9100/metrics Setup HTTP item to gather Prometheus data 2) Create dependent items T N Get the CPU seconds spent on iowait Get the CPU seconds spent by system Get the total CPU seconds E D U T S 6.0 Certified Professional ● Day 2 O C © 2023 by Zabbix. All rights reserved 20 minutes Practical task No: 10 34 Y P T N O C E D Conversion to JSON U T S 6.0 Certified Professional ● Day 2 © 2023 by Zabbix. All rights reserved 10 minutes 35 TRANSFORMATION TO JSON Zabbix has multiple preprocessing steps, which can transform data to JSON Y P CSV to JSON XML to JSON Prometheus to JSON T N O C U T E D The automatic data transformation is very useful when performing LLD: S Zabbix can use some items to collect data in CSV, XML or Prometheus format Preprocessing step will automatically convert this to JSON object Low level discovery rule can create items, triggers and graphs based on JSON data 6.0 Certified Professional ● Day 2 © 2023 by Zabbix. All rights reserved Theory 36 CSV TO JSON CSV to JSON preprocessing step converts CSV format data into JSON format: Y P The first parameter allows to set a custom delimiter The second optional parameter allows to set a quotation symbol O C The checkbox "With header row" defines how the values of the first line will be interpreted Checked: as column names Not checked: as regular data 6.0 Certified Professional ● Day 2 Quotation symbol S Delimiter U T E D T N © 2023 by Zabbix. All rights reserved Theory 37 CSV TO JSON EXAMPLES "Nr","Item name","Key","Qty" "1","active agent item","agent.hostname","33" "2","passive agent item","agent.version","44" "3","active,passive agent items","agent.ping","55" [{ "Nr":"1", "Item name":"active agent item", "Key":"agent.hostname", "Qty":"33" }, { "Nr":"2", "Item name":"passive agent item", "Key":"agent.version", "Qty":"44" }, { "Nr":"3", "Item name":"active,passive agent items", "Key":"agent.ping", "Qty":"55" }] E D U T S 6.0 Certified Professional ● Day 2 T N © 2023 by Zabbix. All rights reserved Y P O C [{ "1":"Nr", "2":"Item name", "3":"Key" "4":"Qty" },{ "1":"1", "2":"active agent item", "3":"agent.hostname" "4":"33" },{ "1":"2", "2":"passive agent item", "3":"agent.version" "4":"44" }] Theory 38 XML TO JSON XML to JSON preprocessing step converts XML format data into JSON format: Y P Conversion will be processed according to the serialization rules: XML attributes will be converted to keys that have their names prepended with '@' Self-closing elements (<foo/>) will be converted as having 'null' value Empty attributes (with "" value) will be converted as having empty string ('') value More rules can be found in documentation { <xml> <foo/> </xml> E D "xml": { "foo": null } } U T <xml> <foo>BAR</foo> <foo>BAZ</foo> <foo>QUX</foo> </xml> i S T N O C { <xml> <foo>BAZ</foo> </xml> "xml": { "foo": "BAZ" } } { "xml": { "foo": ["BAR", "BAZ", "QUX"] } } https://www.zabbix.com/documentation/6.0/en/manual/config/items/preprocessing/javascript/javascript_objects 6.0 Certified Professional ● Day 2 © 2023 by Zabbix. All rights reserved Theory 39 PROMETHEUS TO JSON When converting Prometheus metrics, following attributes are returned: Y P Metric name Metric value [ Help, type (if present) Labels (if present) E D # HELP wmi_logical_disk_free_bytes Free space in bytes # TYPE wmi_logical_disk_free_bytes gauge wmi_logical_disk_free_bytes{volume="C:"} 3.5180249088e+11 wmi_logical_disk_free_bytes{volume="D:"} 2.627731456e+09 S 6.0 Certified Professional ● Day 2 }, { ] } O C "name": "wmi_logical_disk_free_bytes", "help": "Free space in bytes", "type": "gauge", "labels": {"volume": "C:"}, "value": "3.5180249088e+11", "line_raw": "wmi_logical_disk_free_bytes{volume=\"C:\"} 3.5180249088e+11" T N Raw line U T { "name": "wmi_logical_disk_free_bytes", "help": "Free space in bytes (LogicalDisk.PercentFreeSpace)", "type": "gauge", "labels": {"volume": "D:"}, "value": "2.627731456e+09", "line_raw": "wmi_logical_disk_free_bytes{volume=\"D:\"} 2.627731456e+09" © 2023 by Zabbix. All rights reserved Theory 40 Y P T N O C E D Low-level Discovery U Basics T S 6.0 Certified Professional ● Day 2 © 2023 by Zabbix. All rights reserved 20 minutes 41 LOW LEVEL DISCOVERY Low-level discovery (LLD) provides a way to automatically create following entities: Y P Items Triggers Graphs T N Hosts O C U T E D LLD rules are very useful when monitoring file systems, network interfaces and similar dynamic things: S Entities are created automatically from their prototypes Unneeded entities can be removed automatically 6.0 Certified Professional ● Day 2 © 2023 by Zabbix. All rights reserved Theory 42 LOW-LEVEL DISCOVERY (LLD) Workflow: Y P Create discovery item/rule in Configuration > Templates > Discovery Create prototypes of items, triggers and graphs that should be created by the rule LLD Rule T N Item prototypes Trigger prototypes E D Graph prototypes U T S Discovered entity 1 Discovered entity 2 6.0 Certified Professional ● Day 2 O C Created entities Items as per prototypes Triggers as per prototypes Graphs as per prototypes Items as per prototypes Triggers as per prototypes Graphs as per prototypes © 2023 by Zabbix. All rights reserved Theory 43 LOW-LEVEL DISCOVERY(LLD) - EXAMPLE Example of the network interface discovery: Item prototype: Traffic in Trigger prototype: High traffic in LLD rule T N Graph prototype: traffic on interface E D Discovered eth0 U T S Discovered eth1 6.0 Certified Professional ● Day 2 Y P Item prototype: Traffic out O C Trigger prototype: High traffic out Item: Traffic in Item: Traffic out Trigger: High traffic in Trigger: High traffic out Graph: traffic on interface Item: Traffic in Item: Traffic out Trigger: High traffic in Trigger: High traffic out Graph: traffic on interface © 2023 by Zabbix. All rights reserved Theory 44 LOW-LEVEL DISCOVERY COMPONENTS LLD rule consists of the following entity prototypes: Y P Item prototypes Trigger prototypes Graph prototypes Host prototypes T N Prototypes are like blueprints for the entities: U T E D S O C Prototypes If a prototype is changed, the entities will be updated: Next time the discovery rule is executed based on the update interval For passive checks it can be forced manually by using [Execute now] 6.0 Certified Professional ● Day 2 © 2023 by Zabbix. All rights reserved Theory 45 LOW-LEVEL DISCOVERY RULE PROPERTIES LLD rule properties are very similar to regular Zabbix item properties: Y P LLD rule must have a name and a unique key Recommended to use the word "discovery" in both. Any item type can be used for data collection O C Data may require preprocessing before it may be used for LLD E D U T S 6.0 Certified Professional ● Day 2 T N © 2023 by Zabbix. All rights reserved Theory 46 LLD INTERNALS Update interval defines how often LLD rule is executed: Y P Based on the received discovery data, the entities will be created, removed or updated Too frequent execution on proxies can degrade proxy performance O C Proxies send LLD data over the network to Zabbix server T N Low level discovery has multiple processes involved: LLD data is collected by data collectors on every update interval (pollers, trappers etc.) E D Data can be preprocessed (optionally) - the result must be in JSON format JSON format data is passed to LLD manager, which will pass it to a free LLD worker process U T Multiple LLD workers can be started on Zabbix server, if required: S ### Option: StartLLDProcessors # Number of pre-forked instances of low level discovery processors. # Range: 1-100 StartLLDProcessors=5 6.0 Certified Professional ● Day 2 © 2023 by Zabbix. All rights reserved Theory 47 LOW-LEVEL DISCOVERY(LLD) - ITEM PROTOTYPES Prototypes are created under the LLD rule like regular entities Item prototypes T N Y P O C All entities created by LLD will have a rule name displayed in orange color: E D LLD rule Items created from prototypes U T S 6.0 Certified Professional ● Day 2 © 2023 by Zabbix. All rights reserved Theory 48 LLD - LOST RESOURCES Entities, which do not exist anymore, are kept for Keep lost resources period: Y P They do not exist anymore They are filtered out by LLD filters or overrides (discussed later) O C Entities will be removed automatically after this period: T N Default period is 30 days (maximum is 25 years) If a period is set to 0, the lost entities are removed on the next LLD rule execution E D The lost entities are marked with a yellow warning sign U T S 6.0 Certified Professional ● Day 2 © 2023 by Zabbix. All rights reserved Theory 49 LLD - DISABLING LLD RULE If the LLD rule is disabled / not supported: Y P New entities are not created Lost entities will not be removed Changes to prototypes will take no effect Existing items and triggers will not be affected E D U T S ! T N O C Internal actions can be used to report broken LLD rules 6.0 Certified Professional ● Day 2 The LLD rule is disabled © 2023 by Zabbix. All rights reserved Items are still enabled Theory 50 LLD - CREATED ENTITIES Deleting rules/prototypes: Y P If a prototype is deleted, every entity created from this prototype is deleted immediately If a discovery rule is deleted, all the entities created by this rule will be deleted immediately E D U T S 6.0 Certified Professional ● Day 2 T N O C 4 items created by LLD rule Item prototype Outgoing traffic is deleted 2 Items left after removing prototype © 2023 by Zabbix. All rights reserved Theory 51 PRACTICAL SETUP Y P 1) Create a new template: Name: Training Low-level discovery Groups: Training/Templates O C 2) Create a new discovery rule on the "Training Low-level discovery" template: T N Name: Network interfaces discovery Use Zabbix built in network discovery key net.if.discovery Change update interval to 1 hour E D 3) Create two item prototypes for incoming and outgoing traffic 4) Link the "Training Low-level discovery" template to the "Training-VM-XX" host U T Manually execute the discovery rule and check the results 5) Add two new dummy network interfaces “eth2” and “eth3” S Manually execute the discovery rule and check the results 6) Remove the dummy network interface “eth2” and loopback interface from the item list Manually execute the discovery rule and check results 6.0 Certified Professional ● Day 2 © 2023 by Zabbix. All rights reserved 20 minutes Practical task No: 11 52 Y P E DLLD INTERNALS U T S 6.0 Certified Professional ● Day 2 T N O C © 2023 by Zabbix. All rights reserved 15 minutes 53 LOW-LEVEL DISCOVERY MACROS Low-level discovery rules return data in LLD macros Y P Syntax: {#MACRO.NAME} LLD macros: Are used for creating prototypes O C Return real values of the file system name, network interface, SNMP OID, etc. T N When discovering real file systems, network interfaces, etc.: Items are created based on item prototypes Macros are substituted with real values E D Item key prototype U T S net.if.in["{#IFNAME}"] 6.0 Certified Professional ● Day 2 Item keys created from a prototype net.if.in["eth0"] net.if.in["eth1"] net.if.in["eth2"] net.if.in["eth3"] © 2023 by Zabbix. All rights reserved Theory 54 LOW-LEVEL DISCOVERY BUILT-IN MACROS Built-in discovery items have their own set of predefined LLD macros Y P discovery key LLD macros Mounted filesystems vfs.fs.discovery {#FSNAME}, {#FSTYPE} Block devices vfs.dev.discovery {#DEVNAME}, {#DEVTYPE} Network interfaces net.if.discovery {#IFNAME} CPU system.cpu.discovery {#CPU.NUMBER}, {#CPU.STATUS} Windows services service.discovery {#SERVICE.NAME}, {#SERVICE.STATE}, etc. systemd.unit.discovery {#UNIT.NAME}, {#UNIT.ACTIVESTATE}, etc. Systemd services E D T N U T O C Built-in LLD keys return an array of LLD rows at the root of JSON document Zabbix will automatically extract a macro and a value S zabbix_agent2 -t net.if.discovery [s|[{"{#IFNAME}":"eth0"},{"{#IFNAME}":"eth1"},{"{#IFNAME}":"eth2"},{"{#IFNAME}":"eth3"}]] 6.0 Certified Professional ● Day 2 © 2023 by Zabbix. All rights reserved Theory 55 LOW-LEVEL DISCOVERY(LLD) - JSON FORMAT Discovery rule relies on JSON data block to generate LLD macros: Y P Zabbix JSON format can be used to generate them automatically Any valid JSON data block containing data array is supported by defining LLD mappings ## Example of built in JSON format # zabbix_agent2 -t [ s|[ { "{#FSNAME}": "{#FSTYPE}": }, { "{#FSNAME}": "{#FSTYPE}": } ] ] ! [ { "TABLE_NAME": "acknowledges", "DATA_LENGTH": "16384", "INDEX_LENGTH": "49152" }, { "TABLE_NAME": "actions", "DATA_LENGTH": "16384", "INDEX_LENGTH": "32768" } E D U T S T N vfs.fs.discovery "/sys", "sysfs" O C ## Example of Custom JSON format "/", "xfs" ] Agent Discovery data format from older versions is supported 6.0 Certified Professional ● Day 2 © 2023 by Zabbix. All rights reserved Theory 56 LLD – BUILT-IN JSON FORMAT Zabbix JSON format already contains built-in LLD macro names and values: Macro names are already defined { Macros are ready for usage in prototypes O C },{ T N } Y P "{#FSNAME}": "/sys", "{#FSTYPE}": "sysfs" "{#FSNAME}": "/var", "{#FSTYPE}": "xfs" For custom JSON format LLD macro names must be mapped to JSON path E D Mapping is defined in the LLD rule properties (LLD macros tab) U T { "fsname": "/sys", "fstype": "sysfs", "bytes": {"total": 941412352,"free": 941412352} }, { "fsname": "/var", "fstype": "xfs", "bytes": {"total": 53673439232,"free": 45409468416} } S 6.0 Certified Professional ● Day 2 © 2023 by Zabbix. All rights reserved Theory 57 LLD - PREPROCESSING An example of received data block: TABLE_NAME acknowledges actions Raw data separated by tabs Preprocessing steps: Replace \t (tabs) with ; (semicolons) T N Use built-in CSV to JSON functionality DATA_LENGTH 16384 16384 O C Y P INDEX_LENGTH 49152 32768 E D After preprocessing - JSON data block is ready for processing with LLD U T LLD macro {#TABLE.NAME} is mapped to JSON path $.TABLE_NAME [{ S "TABLE_NAME": "acknowledges", }, { "TABLE_NAME": "actions", }] 6.0 Certified Professional ● Day 2 © 2023 by Zabbix. All rights reserved Theory 58 Y P T N O C E D Low-level Discovery U T Prototypes S 6.0 Certified Professional ● Day 2 © 2023 by Zabbix. All rights reserved 15 minutes 59 LOW-LEVEL DISCOVERY (LLD) - ITEM PROTOTYPES Item prototypes have the same properties as regular items Mandatory requirements: LLD {#MACRO} in the item key Keys must be unique T N Use {#MACRO} as key’s[parameter] Y P O C E D Recommended: U T To use {#MACRO} in the item name Use {#MACRO} in tags i S https://www.zabbix.com/documentation/.../discovery/low_level_discovery#item_prototypes 6.0 Certified Professional ● Day 2 © 2023 by Zabbix. All rights reserved Theory 60 LLD - TRIGGER PROTOTYPES Trigger prototypes: Y P It is recommended to use LLD {#MACRO} in the trigger name LLD {#MACRO} can be used in tags Expression may contain: Item prototypes from the same LLD rule Regular items from any host/template E D U T i S T N O C https://.../documentation/.../low_level_discovery#trigger_prototypes 6.0 Certified Professional ● Day 2 © 2023 by Zabbix. All rights reserved Theory 61 LLD - TRIGGER PROTOTYPE DEPENDENCIES Trigger prototype dependencies: Y P Between trigger prototypes from the same LLD rule Between a trigger prototype and any host/template regular trigger E D U T S 6.0 Certified Professional ● Day 2 T N O C © 2023 by Zabbix. All rights reserved Theory 62 LLD - ADDITIONAL ATTRIBUTES The item / trigger prototypes have additional attributes: Description Create enabled If checked (default), the item/trigger will be added in an enabled state. If unchecked, the item/trigger will be added to a host, but in a disabled state. Discover If checked (default), the item / trigger will be added to a host. If unchecked, the item / trigger will not be added to a host S T N O C E D U T ! Y P Parameter Create enabled setting won't affect already existing entities! 6.0 Certified Professional ● Day 2 © 2023 by Zabbix. All rights reserved Theory 63 LLD - GRAPH PROTOTYPES Graph prototypes: Y P A name must be unique - use {#LLD.MACRO} Graphs can include: Same LLD rule item prototypes Any host/template regular item E D U T i S T N O C https://www.zabbix.com/documentation/6.0/manual/discovery/low_level_discovery 6.0 Certified Professional ● Day 2 © 2023 by Zabbix. All rights reserved Theory 64 Y P T N O C E D Low-level Discovery U Filters T S 6.0 Certified Professional ● Day 2 © 2023 by Zabbix. All rights reserved 15 minutes 65 LOW-LEVEL DISCOVERY(LLD) - FILTERS By default, discovery rules return every discovered object: Y P zabbix_agent2 -t vfs.fs.discovery [s|[{"{#FSNAME}":"/sys","{#FSTYPE}":"sysfs"},{"{#FSNAME}":"/dev","{#FSTYPE}":"devtmpfs"},{"{#FSNAME}" :"/dev/pts","{#FSTYPE}":"devpts"},{"{#FSNAME}":"/var","{#FSFSTYPE}":"xfs"},{"{#FSNAME}":"/dev/mqueue" ,"{#FSTYPE}":"mqueue"},{"{#FSNAME}":"/sys/fs/binfmt_misc","{#FSTYPE}":"autofs"},{"{#FSNAME}":"/sys/ke rnel/debug","{#FSTYPE}":"debugfs"},{"{#FSNAME}":"/log","{#FSTYPE}":"ext4"}]] T N O C Many of these objects may not be needed and can be filtered out Discovery rule E D Discovered objects U T vfs.fs.discovery S 6.0 Certified Professional ● Day 2 /var Discovery filter {#FSNAME}=/var,/log Items created vfs.fs.size[/var,free] /dev /log vfs.fs.size[/log,free] /sys © 2023 by Zabbix. All rights reserved Theory 66 LOW-LEVEL DISCOVERY (LLD) - FILTERS Filters tab contains filter definitions: Y P Regular expression (PCRE regex) User macro (Host, Template, Global) Global regular expression E D U T S ! T N O C Recommended to keep regex for filters in user macros on Template 6.0 Certified Professional ● Day 2 © 2023 by Zabbix. All rights reserved Theory 67 FILTER CALCULATION The following options for calculating filters are available: Y P And - all filters must be passed; Or - enough if one filter is passed; O C And/Or - uses "And" with different macro names and "Or" with the same macro name Custom expression - defines a custom calculation of filters T N Objects that are filtered out will be treated as not discovered at all: E D Filtering is applied every time discovery rule is executed Errors in filters may remove thousands of items by accident (if "keep lost resources" = 0) U T S The following filter condition operators are available: matches, does not match - expects PCRE regular expression exists, does not exist - based on the presence or absence of the specified LLD macro 6.0 Certified Professional ● Day 2 © 2023 by Zabbix. All rights reserved Theory 68 PRACTICAL SETUP Y P 1) Create a new filter on the "Training Low-level discovery" template Discover only eth* and loopback interfaces Apply user macro to define the LLD filter 2) Make sure that the filter works T N 3) Clean up lost items E D U T S 6.0 Certified Professional ● Day 2 O C © 2023 by Zabbix. All rights reserved 10 minutes Practical task No: 12 69 Y P T N O C E D Low-level Discovery U T Overrides S 6.0 Certified Professional ● Day 2 © 2023 by Zabbix. All rights reserved 15 minutes 70 LOW-LEVEL DISCOVERY (LLD) - OVERRIDES Overrides: Y P Allow to set rules to modify attributes for discovered objects that meet given criteria Are displayed on a reorderable drag-and-drop list O C Override rules are based on filters matching LLD macros E D U T i S T N https://www.zabbix.com/documentation/.../discovery/low_level_discovery#override 6.0 Certified Professional ● Day 2 © 2023 by Zabbix. All rights reserved Theory 71 OVERRIDE OPERATIONS Name field defines the override name Y P Filter field is used to check LLD macro: Against regular expression Checks if the macro is defined at all O C One or more operations will be executed, if the filter condition is matched E D U T S 6.0 Certified Professional ● Day 2 T N © 2023 by Zabbix. All rights reserved Theory 72 LOW-LEVEL DISCOVERY (LLD) - OVERRIDES Override operations: Y P Affect the entities created by the LLD rule: Items Triggers Graphs Hosts created from the host prototypes Condition - checks the entity name pattern E D Equals / Does not equal Contains / Does not contain Matches / Does not match U T S T N O C Change Attributes for discovered objects that meet given criteria: Create enabled (Yes /No) Discover (Yes / No) Other attributes specific to the entity (history storage period, trigger severity, etc.) 6.0 Certified Professional ● Day 2 © 2023 by Zabbix. All rights reserved Theory 73 LOW-LEVEL DISCOVERY (LLD) - OVERRIDES Overrides are executed in that order, in which they are defined: Y P If filter matches, an override is applied Depending on the "Continue overrides" setting (Yes/No), subsequent overrides will be processed E D Filter U T No match S Match Match 6.0 Certified Professional ● Day 2 T N O C Continue Overrides Stop Processing Override 1 Override 1 Override 2 Override 2 Override 3 Override 3 © 2023 by Zabbix. All rights reserved Theory 74 PRACTICAL SETUP Y P 1) Add a new dummy network interface “trunk” 2) Create a trigger prototype for the Network interface discovery rule Name: Severity: Expression: O C Heavy traffic on interface {#IFNAME} Warning Incoming or Outgoing traffic for 10 minutes > 1G T N 3) Manually execute the Network interfaces discovery rule E D 4) Create overrides: Triggers for trunk ports - severity changed to High Triggers for the loopback interface - do not create triggers U T 5) Manually execute Network interfaces discovery rule Check triggers on your student-XX host S 6.0 Certified Professional Day ● Day 2 2 © 2023 by Zabbix. All rights reserved 20m Practical task No: 13 75 Y P T N O C E D Discovery Using Dependent Items U T S 6.0 Certified Professional ● Day 2 © 2023 by Zabbix. All rights reserved 30 minutes 76 DISCOVERY USING DEPENDENT ITEMS An item can have a dependent discovery rule: Y P Discovery rule type: Dependent item The rule is updated every time the master item collects data E D U T S 6.0 Certified Professional ● Day 2 T N O C © 2023 by Zabbix. All rights reserved Theory 77 DISCOVERY USING DEPENDENT ITEMS A discovery rule can create dependent items, which may depend on: Y P A regular item in the same host/template An item created from a prototype by the same discovery rule E D U T S 6.0 Certified Professional ● Day 2 T N O C © 2023 by Zabbix. All rights reserved Theory 78 DISCOVERY USING DEPENDENT ITEMS Master item data can be used: Y P To discover entities (items, triggers, graphs) To fill discovered items with values Master item Data entry Data entry U T S Data entry T N E D Data entry Data entry Discovery data Data values O C Dependent LLD Rule Dependent item prototype Dependent item management Dependent items Item 1 Data entry Item 2 Data entry Item ... 6.0 Certified Professional ● Day 2 © 2023 by Zabbix. All rights reserved Theory 79 PREPROCESSING - DEPENDENT ITEMS If a history is set to [Do not keep history], the master item will populate only dependent items Y P T N O C In the item list dependent items are displayed with their master item name as a prefix U T E D Without preprocessing, the dependent item value will be exactly the same as the master item S It is possible to create a dependent item from another dependent item (up to 3 levels are supported) 6.0 Certified Professional ● Day 2 © 2023 by Zabbix. All rights reserved Theory 80 PRACTICAL SETUP Y P 1) Create a new item on the "Training Low-level discovery" template Collect the the raw data of the file system using "vfs.fs.get" item key 2) Create a dependent discovery rule Discover filesystems using data collected by the item T N 3) Create item prototypes to monitor Total space on all the filesystems Free space on all the filesystems O C E D 4) Exclude the /run/* and /tmp/* filesystems from monitoring Apply user macro to define the LLD filter Delete data that is filtered out U T S 6.0 Certified Professional Day ● Day 2 2 © 2023 by Zabbix. All rights reserved 20 minutes Practical task No: 14 81 Y P E D LLD Preprocessing U T S 6.0 Certified Professional ● Day 2 T N O C © 2023 by Zabbix. All rights reserved 10 minutes 82 DISCOVERY USING DEPENDENT ITEMS Add Discard unchanged with a heartbeat preprocessing step to the LLD rule Choose reasonable heartbeat interval such as 1 hour or even 1 day Master item Data entry Discovery data Discard unchanged with heartbeat E D Data entry Data entry U T S Data entry O C Dependent item prototype T N Data entry Data values Dependent item management Dependent items Item 1 Data entry Item 2 Data entry Item ... 6.0 Certified Professional ● Day 2 Y P Dependent LLD Rule © 2023 by Zabbix. All rights reserved Theory 83 DISCOVERY USING DEPENDENT ITEMS - JSONPATH Escaping special characters from LLD macro values in JSONPath Y P When Low-level Discovery macros are used in JSONPath preprocessing and their values are resolved, the following rules of escaping special characters are applied: O C Only backslash (\) and double quote (") characters are considered for escaping If resolved macro value contains these characters, each of them is escaped with a backslash If they are already escaped with a backslash: T N It is not considered as escaping Both the backslash and the following special characters are escaped once again E D JSONPath U T $.[?(@.value == "{#MACRO}")] i S LLD macro value After substitution special "value" $.[?(@.value == "special \"value\"")] c:\temp $.[?(@.value == "c:\\temp")] a\\b $.[?(@.value == "a\\\\b")] https://.../preprocessing/jsonpath_functionality/escaping_lld_macros 6.0 Certified Professional ● Day 2 © 2023 by Zabbix. All rights reserved Theory 84 DISCOVERY USING DEPENDENT ITEMS - JSONPATH If macros have special characters: Y P Use double quotes LLD macro value JSONPath $.[?(@.value == "{#MACRO}")] After substitution special "value" $.[?(@.value == {#MACRO})] O C $[?(@.value == "special \"value\"")] T N $[?(@.value == special \"value\")] Result OK Bad JSONPath expression If special characters are used in the path: E D They should be enclosed in square brackets and double quotes U T JSONPath $.["{#MACRO}"].value S $.{#MACRO}.value LLD macro value c:\temp 6.0 Certified Professional ● Day 2 After substitution Result $.["c:\\temp"].value OK $.c:\\temp.value Bad JSONPath expression © 2023 by Zabbix. All rights reserved Theory 85 Y P T N O C E D CONTEXT SUPPORT U T IN USER MACROS S 6.0 Certified Professional ● Day 2 © 2023 by Zabbix. All rights reserved 10 minutes 86 LLD - CONTEXT SUPPORT IN USER MACROS An optional context can be used in user macros: Y P Allows to override the default value with a context-specific one The context is appended to the macro name Regular expressions are supported How to create context-based macros: T N O C Define a user macro on a template, host or global level Extend the user macro with the context to match different conditions i E D FS Type Description User macro Value any User macro Any undefined FS {$LOW.SPACE} 1G /boot Context macro boot FS (small) {$LOW.SPACE:"/boot"} 100M U T S /home Context macro home FS (medium) {$LOW.SPACE:"/home"} 5G /var/log/* Context macro log FS 100G (large) {$LOW.SPACE:regex:"^/var/log/.*$"} https://www.zabbix.com/documentation/6.0/manual/config/macros/user_macros_context 6.0 Certified Professional ● Day 2 © 2023 by Zabbix. All rights reserved Theory 87 USING CONTEXT MACROS IN TRIGGERS Can be defined to accomplish more flexible thresholds in trigger expressions: Y P Are based on the values retrieved by Low-level Discovery After the discovery, different thresholds will apply Configuration example: T N A trigger prototype: O C last(/host/vfs.fs.size[{#FSNAME},free]) < {$LOW.SPACE:"{#FSNAME}"} E D The triggers created from prototype: last(/host/vfs.fs.size[/home,free]) < {$LOW.SPACE:"/home"} last(/host/vfs.fs.size[/boot,free]) < {$LOW.SPACE:"/boot"} last(/host/vfs.fs.size[/var/log/mysql,free]) < {$LOW.SPACE:"/var/log/mysql"} last(/host/vfs.fs.size[/data,free]) < {$LOW.SPACE:"/data"} U T S the {$LOW.SPACE:"/data"} macro is not defined and will be calculated using {$LOW.SPACE} i https://www.zabbix.com/documentation/6.0/manual/config/macros/user_macros_context 6.0 Certified Professional ● Day 2 © 2023 by Zabbix. All rights reserved Theory 88 PRACTICAL SETUP Y P 1) On the "Training Low-level discovery" template: Create a new macro {$LOW.DISK.SPACE} = 10G In the "File systems discovery" rule create a new trigger prototype to detect free space < 10G: O C 2) Execute File systems discovery rule on the Training-VM-XX host T N 3) Make sure that triggers are created, and the problem is detected 4) On the "Training Low-level discovery" template: E D Create a new context-based macro for all /dev* filesystems with value 300M Adjust the trigger prototype to use context-based macros 5) Check whether the context-based macro is used U T The problem must be resolved S 6.0 Certified Professional Day ● Day 2 2 © 2023 by Zabbix. All rights reserved 20 minutes Practical task No: 15 89 Y P E D Specific LLD Keys U T S 6.0 Certified Professional ● Day 2 T N O C © 2023 by Zabbix. All rights reserved 10 minutes 90 LLD OF LINUX SYSTEMD SERVICES Linux systemd services can be discovered only by using Zabbix agent 2: An LLD rule returns data in JSON format U T S 6.0 Certified Professional ● Day 2 Y P }, { ] } O C "{#UNIT.NAME}": "nginx.service", "{#UNIT.DESCRIPTION}": "The nginx HTTP and reverse proxy server", "{#UNIT.LOADSTATE}": "loaded", "{#UNIT.ACTIVESTATE}": "active", "{#UNIT.SUBSTATE}": "running", "{#UNIT.FOLLOWED}": "", "{#UNIT.PATH}": "/org/freedesktop/systemd1/unit/nginx_2eservice", "{#UNIT.JOBID}": 0, "{#UNIT.JOBTYPE}": "", "{#UNIT.JOBPATH}": "/" T N E D LLD macros are used in prototypes [ { "{#UNIT.NAME}": "mysqld.service", "{#UNIT.DESCRIPTION}": "mysql.service", "{#UNIT.LOADSTATE}": "not-found", "{#UNIT.ACTIVESTATE}": "inactive", "{#UNIT.SUBSTATE}": "dead", "{#UNIT.FOLLOWED}": "", "{#UNIT.PATH}": "/org/freedesktop/systemd1/unit/mysql_2eservice", "{#UNIT.JOBID}": 0, "{#UNIT.JOBTYPE}": "", "{#UNIT.JOBPATH}": "/" © 2023 by Zabbix. All rights reserved Theory 91 LLD OF LINUX SYSTEMD SERVICES Some systemd key examples: Y P Description and key The primary name of the unit systemd.unit.info[mysqld.service,Id] Service Description T N systemd.unit.info[mysqld.service,Description] O C Returned value mysqld.service MySQL Server One of states: active, reloading, inactive, failed, activating, deactivating. E D systemd.unit.info[mysqld.service,ActiveState,Unit] The last time a unit entered the active state U T systemd.unit.info[mysqld.service,ActiveEnterTimestamp] active 1562565036283903 The number of currently open connections on this socket S systemd.unit.info[dbus.socket,NConnections,Socket] i 0 https://www.zabbix.com/documentation/6.0/manual/discovery/low_level_discovery/systemd 6.0 Certified Professional ● Day 2 © 2023 by Zabbix. All rights reserved Theory 92 LLD FOR WINDOWS SERVICES Windows services can be discovered by Zabbix agent: Key: service.discovery Item key: LLD rule returns data in macros: • • • • • • • • • • {#SERVICE.NAME} {#SERVICE.DISPLAYNAME} {#SERVICE.DESCRIPTION} {#SERVICE.STATE} {#SERVICE.STATENAME} {#SERVICE.PATH} {#SERVICE.USER} {#SERVICE.STARTUP} {#SERVICE.STARTUPNAME} {#SERVICE.STARTUPTRIGGER} S O C service.info[service,<param>] Item prototype example: T N service.info[{#SERVICE.NAME},state] E D U T Y P Trigger prototype example: last(/host/service.info[{#SERVICE.NAME},state])<>0 i https://.../discovery/low_level_discovery/windows_services 6.0 Certified Professional ● Day Day 22 © 2023 by Zabbix. All rights reserved Theory 93 LLD FOR WMI QUERIES Done using WQL queries Y P Results are automatically transformed into a JSON object Discovery key: wmi.getall[<namespace>,<query>] Example: T N O C wmi.getall[root\cimv2,"select * from win32_perfformatteddata_perfdisk_physicaldisk"] U T E D S [ ] 6.0 Certified Professional ● Day 2 © 2023 by Zabbix. All rights reserved { "Name": "0 C:", "DiskReadBytesPersec": "0", "DiskReadsPersec": 0, "DiskWriteBytesPersec": "784167", "DiskWritesPersec": 11, "PercentIdleTime": "99", }, { "Name": "1 D:", "DiskReadBytesPersec": "6372437 "DiskReadsPersec": 14256 "DiskWriteBytesPersec": "4362378", "DiskWritesPersec": 36231, "PercentIdleTime": "65", } Theory 94 LLD OF PERFORMANCE COUNTERS It is possible to discover object instances for Windows performance counters Y P Useful for multi-instance performance counters Discovery will return all instances in the {#INSTANCE} macro Discovery keys: perf_instance.discovery[object] T N perf_instance_en.discovery[object] Example discovery key and item prototype: E D perf_instance.discovery[Processor] U T S 6.0 Certified Professional ● Day 2 O C perf_counter_en["\Processor({#INSTANCE})\% Processor Time"] [ ] © 2023 by Zabbix. All rights reserved {"{#INSTANCE}":"0"}, {"{#INSTANCE}":"1"}, {"{#INSTANCE}":"2"}, {"{#INSTANCE}":"3"}, {"{#INSTANCE}":"_Total"} Theory 95 Y P T N O C E D DIFFERENT LLD METHODS U T S 6.0 Certified Professional ● Day 2 © 2023 by Zabbix. All rights reserved 20 minutes 96 CLASSIC WAY VS DEPENDENT ITEMS There are two approaches how to design LLD rules and prototypes: Y P Classic way: Every metric is polled individually Utilizes more pollers, if passive checks are used T N Problems related to Zabbix agent: More frequent requests to the agent Use active agent to avoid these problems E D U T Dependent items: O C Metrics are collected in bulk S More load on preprocessors, recommended to use proxies to offload Zabbix server Can cause spikes in data processing (triggers, history writes etc.) 6.0 Certified Professional ● Day 2 © 2023 by Zabbix. All rights reserved Theory 97 DISCOVERY OF MOUNTED FILESYSTEMS By example, Zabbix agent offers two ways how to work with the filesystems: Classic: using a discovery rule with the agent key vfs.fs.discovery O C Discovery rule creates "normal" Zabbix items for every metric Active agent checks can offload Zabbix server T N A new one: vfs.fs.get as a master item Y P Discovery rule creates dependent items Everything is processed by Zabbix server using preprocessors Load can be distributed among Zabbix proxies U T E D S 6.0 Certified Professional ● Day 2 © 2023 by Zabbix. All rights reserved Theory 98 DISCOVERY OF MOUNTED FILESYSTEMS - CLASSIC WAY vfs.fs.discovery item key discovers a list of mounted filesystems and their types: Y P Discovery rule 6.0 Certified Professional ● Day 2 T N O C E D U T S Prototypes © 2023 by Zabbix. All rights reserved Theory 99 DISCOVERY OF MOUNTED FILESYSTEMS - NEW WAY Uses vfs.fs.get agent item key for the master item Metrics are received in bulk { Processes the collected data using: A dependent low-level discovery rule T N $.[?(@.fsname=='{#FSNAME}')].bytes.free.first() Usage of the dependent items increases CPU utilization: U T Dependent items are processed by preprocessors "fsname": "/", "fstype": "xfs", "bytes": { "total": 26831990784, "free": 21950971904, "used": 4881018880, "pfree": 81.808957, "pused": 18.191043 }, "inodes": { "total": 13106624, "free": 13064122, "used": 42502, "pfree": 99.675721, "pused": 0.324279 } O C Item prototypes for each metric using preprocessing - JSON path E D Y P JSON example Proxies can be used to offload Zabbix server i S https://.../documentation/.../discovery/low_level_discovery/mounted_filesystems 6.0 Certified Professional ● Day 2 © 2023 by Zabbix. All rights reserved Theory 100 DISCOVERY OF MOUNTED FILESYSTEMS - NEW WAY Master item: LLD macros using JSONPath: Y P T N O C Item prototype: Dependent discovery rule: U T E D S 6.0 Certified Professional ● Day 2 Preprocessing - JSON path: © 2023 by Zabbix. All rights reserved Theory 101 Y P E DLLD Using SQL U T S 6.0 Certified Professional ● Day 2 T N O C © 2023 by Zabbix. All rights reserved 10 minutes 102 LLD VIA SQL QUERIES LLD via SQL queries: Y P db.odbc.discovery[<description>,<dsn>,<connection string>] Example: db.odbc.discovery[databases,mssqlsrv_01] Results will be automatically transformed into JSON O C Column names become macro names and the selected rows become the values of these macros. T N Use column aliases to define the macro names: mysql> SELECT c.name, c.l0c# AS location FROM customers c E D The macro names: {#NAME} and {#LOCATION} Be aware that: U T The discovery rule will become not supported, if a macro name is not valid (e.g. {#L0C#}) S This is the "old way" to do SQL discovery ! db.odbc.get is much more effective compared to db.odbc.discovery 6.0 Certified Professional ● Day 2 © 2023 by Zabbix. All rights reserved Theory 103 LLD VIA SQL QUERIES Key: db.odbc.get[<unique short description>,<dsn>,<connection string>] Y P Returns multiple rows/columns in JSON format May be used as a master item that collects all the data in one system call JSONPath preprocessing must be used in dependent items to extract individual values O C More efficient way to get discovery data and all the values in one call T N LLD rule is executed together with data collection; keeps items up to date Recommended way for Zabbix 6.0 U T E D S 6.0 Certified Professional ● Day 2 JSON © 2023 by Zabbix. All rights reserved Theory 104 LLD VIA SQL QUERIES Master item Dependent Discovery Rule Y P E D U T S 6.0 Certified Professional ● Day 2 T N Item prototype O C © 2023 by Zabbix. All rights reserved Theory 105 PRACTICAL SETUP Y P Create on the "Training ODBC monitoring" template: 1) Master item: Name: Type: Key: SQL query: T N SELECT table_name,data_length,index_length FROM information_schema.partitions WHERE table_schema = 'zabbix' E D 2) Dependent discovery rule: Name: Key: O C Raw database table data Database monitor db.odbc.get[table.raw.discovery,,<set proper uri>]" U T Database Table discovery db.table.discovery 3) Three item prototypes: S Table data length Table index length Total table size 6.0 Certified Professional Day ● Day 2 2 dependent item dependent item calculated item © 2023 by Za

ZCP_day_2_slides.pdf

Related

Transcript

Tags

Upgrade to continue