Zabbix Preprocessing Steps for XML, JSON & CSV Data PDF

PREPROCESSING - XML XPATH Y P <?xml version="1.0" encoding="UTF-8"?> <bookstore> <book category="cooking"> <title lang="en">Everyday Italian</title> <author>Giada De Laurentiis</author> <year>2005</year> <price>30.00</price> </book> U T <book category="web"> <title lang="en">Learning XML</title>...

PREPROCESSING - XML XPATH Y P <?xml version="1.0" encoding="UTF-8"?> <bookstore> <book category="cooking"> <title lang="en">Everyday Italian</title> <author>Giada De Laurentiis</author> <year>2005</year> <price>30.00</price> </book> U T <book category="web"> <title lang="en">Learning XML</title> <author>Erik T. Ray</author> <year>2003</year> <price>39.95</price> </book> S </bookstore> 6.0 Certified Professional ● Day 2 T N E D <book category="children"> <title lang="en">Harry Potter</title> <author>J K. Rowling</author> <year>2005</year> <price>29.99</price> </book> O C Select the entire XML part with the category "web" /bookstore/book[@category="web"] <book category="web"> <title lang="en">Learning XML</title> <author>Erik T. Ray</author> <year>2003</year> <price>39.95</price> </book> © 2023 by Zabbix. All rights reserved Theory 24 Y P E DPreprocessing U T S 6.0 Certified Professional ● Day 2 T N O C JSON Path © 2023 by Zabbix. All rights reserved 10 minutes 25 JSON FORMAT JSON stands for JavaScript Object Notation Y P Lightweight format for storing and transporting data Zabbix uses JSON format for communication and LLD JSON data is written as Name/Value pairs "firstName":"John" T N Objects can contain multiple Name/Value pairs: O C E D {"firstName":"John","lastName":"Doe"} JSON arrays are written inside square brackets; an array can contain objects: U T S { "employees":[ {"firstName":"John", "lastName":"Doe"}, {"firstName":"Anna", "lastName":"Smith"}, {"firstName":"Peter", "lastName":"Jones"} ] } 6.0 Certified Professional ● Day 2 © 2023 by Zabbix. All rights reserved Theory 26 PREPROCESSING - JSONPATH JSONPath is a query language for JSON Y P Features are similar to XPath Used for selecting and extracting JSON document’s property values O C A JSONPath expression begins with the dollar sign ($): Refers to the root element of a query $.object.category T N The dollar sign is followed by a sequence of child elements, which are separated: Using dot notation $.object.name E D Using square brackets $["object"]["name"] Filters are logical expressions used to filter arrays. U T @ represents the current array item or object being processed. S $.employees[?(@.firstName == "John")] JSONPATH supports the wildcard * operator: Returns all objects or elements regardless of their name. 6.0 Certified Professional ● Day 2 © 2023 by Zabbix. All rights reserved Theory 27 PREPROCESSING - JSONPATH JSONPath functionality is available as item value preprocessing step Y P Operators: Functions: E D Path $.object.name $["book"]["name"] U T $.object.history.length() i T N - + / * == != < <= > >= =~ ! || && avg min max sum length first S O C Result book.name contents number of object.history array elements $[?(@.name == "SomeBook")].price.first() price field of the first object with the name 'SomeBook' $[?(@.name == "SomeBook")].history.first().length() number of history array elements of the first object with the name 'SomeBook' https://www.zabbix.com/documentation/../preprocessing/jsonpath_functionality 6.0 Certified Professional ● Day 2 © 2023 by Zabbix. All rights reserved Theory 28 PREPROCESSING - JSONPATH EXAMPLES { "store": { "book": [ { "category": "reference", "author": "Nigel Rees", "title": "Sayings of the Century", "price": 8.95 }, { "category": "fiction", "author": "Evelyn Waugh", "title": "Sword of Honor", "price": 12.99 }, { "category": "fiction", "author": "J. R. R. Tolkien", "title": "The Lord of the Rings", "isbn": "0-395-19395-8", "price": 22.99 } ], "bicycle": { "color": "red", "price": 19.95 } } } S 6.0 Certified Professional ● Day 2 $.store.book[0].price O C 8.95 Select the price of the book with the name "Sword of Honor" T N $.store.book.[?(@.title == "Sword of Honor")].price E D U T Y P Select the price of the 1st book [12.99] $.store.book.[?(@.title == "Sword of Honor")].price.first() 12.99 Select all the elements of the "bicycle" array. $.store.bicycle { } © 2023 by Zabbix. All rights reserved "color": "red", "price": 19.95 Theory 29 Y P T N O C E D Preprocessing Prometheus U data T S 6.0 Certified Professional ● Day 2 © 2023 by Zabbix. All rights reserved 10 minutes 30 PROMETHEUS DATA COLLECTION Prometheus exporter is a free software application used for data collection: Y P It can report metrics from third-party systems such as Prometheus metrics Listens on some ports as a web server O C Zabbix can collect Prometheus data using HTTP item type T N Creates dependent items with Prometheus pattern and retrieves the specific metrics U T E D i S Prometheus format Extracted metrics Dependent item HTTP item Dependent item Dependent item https://github.com/prometheus/prometheus/wiki/Default-port-allocations 6.0 Certified Professional ● Day 2 © 2023 by Zabbix. All rights reserved Theory 31 PROMETHEUS FORMAT Prometheus format has a name combined with a series of labels with values <metric name>{<label name>=<label value>, ...} Y P O C The metric output is typically preceded by metadata lines: # HELP and # TYPE The # HELP string identifies the metric name and gives a brief description of it T N The # TYPE string identifies the type of metric Everything else that starts with a # is parsed as a comment. E D # HELP node_network_transmit_packets_total Network device statistic transmit_packets # TYPE node_network_transmit_packets_total counter node_network_transmit_packets_total{device="eth0"} 2.966359e+06 node_network_transmit_packets_total{device="eth1"} 5.703921e+06 # HELP node_network_receive_packets_total Network device statistic receive_packets. # TYPE node_network_receive_packets_total counter node_network_receive_packets_total{device="eth0"} 3.008294e+06 node_network_receive_packets_total{device="eth1"} 7.833472e+06 U T S 6.0 Certified Professional ● Day 2 © 2023 by Zabbix. All rights reserved Theory 32 PROMETHEUS PATTERN Prometheus query language is used to extract data: Y P Pattern <metric_name>{<label name>="<label value>"} - selects by a metric name and label <metric_name>{<label name>=~"<regex>"} - selects by matching a regular expression Function value, label Aggregation functions (sum, min, max, avg, count) Output T N E D O C The label name is returned (used only with the "label function") Pattern can match multiple rows only if aggregation functions are used U T S 6.0 Certified Professional ● Day 2 Pattern © 2023 by Zabbix. All rights reserved Function Output Theory 33 PRACTICAL SETUP Y P 1) Open HTTP on training.lan to see Prometheus data http://training.lan-zbxtr-YYYY.zabbix.training:9100/metrics Setup HTTP item to gather Prometheus data 2) Create dependent items T N Get the CPU seconds spent on iowait Get the CPU seconds spent by system Get the total CPU seconds E D U T S 6.0 Certified Professional ● Day 2 O C © 2023 by Zabbix. All rights reserved 20 minutes Practical task No: 10 34 Y P T N O C E D Conversion to JSON U T S 6.0 Certified Professional ● Day 2 © 2023 by Zabbix. All rights reserved 10 minutes 35 TRANSFORMATION TO JSON Zabbix has multiple preprocessing steps, which can transform data to JSON Y P CSV to JSON XML to JSON Prometheus to JSON T N O C U T E D The automatic data transformation is very useful when performing LLD: S Zabbix can use some items to collect data in CSV, XML or Prometheus format Preprocessing step will automatically convert this to JSON object Low level discovery rule can create items, triggers and graphs based on JSON data 6.0 Certified Professional ● Day 2 © 2023 by Zabbix. All rights reserved Theory 36 CSV TO JSON CSV to JSON preprocessing step converts CSV format data into JSON format: Y P The first parameter allows to set a custom delimiter The second optional parameter allows to set a quotation symbol O C The checkbox "With header row" defines how the values of the first line will be interpreted Checked: as column names Not checked: as regular data 6.0 Certified Professional ● Day 2 Quotation symbol S Delimiter U T E D T N © 2023 by Zabbix. All rights reserved Theory 37 CSV TO JSON EXAMPLES "Nr","Item name","Key","Qty" "1","active agent item","agent.hostname","33" "2","passive agent item","agent.version","44" "3","active,passive agent items","agent.ping","55" [{ "Nr":"1", "Item name":"active agent item", "Key":"agent.hostname", "Qty":"33" }, { "Nr":"2", "Item name":"passive agent item", "Key":"agent.version", "Qty":"44" }, { "Nr":"3", "Item name":"active,passive agent items", "Key":"agent.ping", "Qty":"55" }] E D U T S 6.0 Certified Professional ● Day 2 T N © 2023 by Zabbix. All rights reserved Y P O C [{ "1":"Nr", "2":"Item name", "3":"Key" "4":"Qty" },{ "1":"1", "2":"active agent item", "3":"agent.hostname" "4":"33" },{ "1":"2", "2":"passive agent item", "3":"agent.version" "4":"44" }] Theory 38 XML TO JSON XML to JSON preprocessing step converts XML format data into JSON format: Y P Conversion will be processed according to the serialization rules: XML attributes will be converted to keys that have their names prepended with '@' Self-closing elements (<foo/>) will be converted as having 'null' value Empty attributes (with "" value) will be converted as having empty string ('') value More rules can be found in documentation { <xml> <foo/> </xml> E D "xml": { "foo": null } } U T <xml> <foo>BAR</foo> <foo>BAZ</foo> <foo>QUX</foo> </xml> i S T N O C { <xml> <foo>BAZ</foo> </xml> "xml": { "foo": "BAZ" } } { "xml": { "foo": ["BAR", "BAZ", "QUX"] } } https://www.zabbix.com/documentation/6.0/en/manual/config/items/preprocessing/javascript/javascript_objects 6.0 Certified Professional ● Day 2 © 2023 by Zabbix. All rights reserved Theory 39 PROMETHEUS TO JSON When converting Prometheus metrics, following attributes are returned: Y P Metric name Metric value [ Help, type (if present) Labels (if present) E D # HELP wmi_logical_disk_free_bytes Free space in bytes # TYPE wmi_logical_disk_free_bytes gauge wmi_logical_disk_free_bytes{volume="C:"} 3.5180249088e+11 wmi_logical_disk_free_bytes{volume="D:"} 2.627731456e+09 S 6.0 Certified Professional ● Day 2 }, { ] } O C "name": "wmi_logical_disk_free_bytes", "help": "Free space in bytes", "type": "gauge", "labels": {"volume": "C:"}, "value": "3.5180249088e+11", "line_raw": "wmi_logical_disk_free_bytes{volume=\"C:\"} 3.5180249088e+11" T N Raw line U T { "name": "wmi_logical_disk_free_bytes", "help": "Free space in bytes (LogicalDisk.PercentFreeSpace)", "type": "gauge", "labels": {"volume": "D:"}, "value": "2.627731456e+09", "line_raw": "wmi_logical_disk_free_bytes{volume=\"D:\"} 2.627731456e+09" © 2023 by Zabbix. All rights reserved Theory 40 Y P T N O C E D Low-level Discovery U Basics T S 6.0 Certified Professional ● Day 2 © 2023 by Zabbix. All rights reserved 20 minutes 41 LOW LEVEL DISCOVERY Low-level discovery (LLD) provides a way to automatically create following entities: Y P Items Triggers Graphs T N Hosts O C U T E D LLD rules are very useful when monitoring file systems, network interfaces and similar dynamic things: S Entities are created automatically from their prototypes Unneeded entities can be removed automatically 6.0 Certified Professional ● Day 2 © 2023 by Zabbix. All rights reserved Theory 42 LOW-LEVEL DISCOVERY (LLD) Workflow: Y P Create discovery item/rule in Configuration > Templates > Discovery Create prototypes of items, triggers and graphs that should be created by the rule LLD Rule T N Item prototypes Trigger prototypes E D Graph prototypes U T S Discovered entity 1 Discovered entity 2 6.0 Certified Professional ● Day 2 O C Created entities Items as per prototypes Triggers as per prototypes Graphs as per prototypes Items as per prototypes Triggers as per prototypes Graphs as per prototypes © 2023 by Zabbix. All rights reserved Theory 43