ZCP_day_3_slides_Part4.pdf

TIME SHIFT WITH ABSOLUTE TIME PERIODS Absolute time periods have different syntax: Time shift is the first argument (1d, 1w, etc) followed by a colon : After that absolute time period starts with now specifying the current time Followed by any number of time operations: /<time unit> Time unit define...

TIME SHIFT WITH ABSOLUTE TIME PERIODS Absolute time periods have different syntax: Time shift is the first argument (1d, 1w, etc) followed by a colon : After that absolute time period starts with now specifying the current time Followed by any number of time operations: /<time unit> Time unit defines the end of the time period for the time interval used in function 1d:now/d 1d:now/d+1d Yesterday Today 1d = time period, now/d = 00:00 of current day 1d = time period, now/d+1d = 00:00 of the next day 1d:now/d+1d 1d:now/d time Yesterday 2021-10-20 00:00:00 6.0 Certified Professional ● Day 3 Today 2021-10-21 00:00:00 Now 2021-10-21 11:32:14 © 2023 by Zabbix. All rights reserved Tomorrow 2021-10-22 00:00:00 Theory 61 MORE ABSOLUTE TIME PERIODS More absolute time shift examples: 2d:now/d+1d 1w:now/w+1w 1M:now/M+1M 1d:now/M+1d 1d:now/M-1M+1d 1w:now/M+1w Yesterday and today This week This month First day of this month First day of previous month First week of this month Absolute time shift is required to use baseline monitoring and other functions 1d:now/d-1d 2d:now/d+1d time Day before yesterday 2021-10-19 00:00:00 6.0 Certified Professional ● Day 3 Yesterday 2021-10-20 00:00:00 Today 2021-10-21 00:00:00 Now 2021-10-21 11:32:14 © 2023 by Zabbix. All rights reserved Tomorrow 2021-10-22 00:00:00 Theory 62 MACRO FUNCTIONS The output of the macro can be modified using macro function: fmtnum (<digits>) digits - the number of digits after decimal point Number is formatted using the round() function This function can be applied to the following macros: Built-in macros {ITEM.VALUE}, {ITEM.LASTVALUE} Expression macros {?MACRO} Syntax of the macro function: {{?MACRO}.fmtnum(#)} Macro syntax i Macro value Output {{?avg(vm.memory.size[free],1h)}.fmtnum(4)} 24.8413523 24.8414 {{?avg(vm.memory.size[free],1h)}.fmtnum(2)} 24.8413523 24.84 {{ITEM.LASTVALUE}.fmtnum(0)} 24.8413523 25 https://www.zabbix.com/documentation/6.0/en/manual/config/macros/macro_functions 6.0 Certified Professional ● Day 3 © 2023 by Zabbix. All rights reserved Theory 63 TIME FORMATTING Event name field supports the {TIME} macro, which returns current date and time strftime format codes can be used to change the output, for example: %a %A %d %j %U An abbreviated textual representation of the day Sun through Sat A full textual representation of the day Sunday through Saturday Two-digit day of the month (with leading zeros) 01 to 31 Day of the year, 3 digits with leading zeros 001 to 366 Week number of the given year, starting with the first Sunday as the first week Time operations are allowed (+1d, -1M etc.) Macro syntax i Output {TIME} 14:10:24 {{TIME}.fmttime(%A)} Monday {{TIME}.fmttime(%F %X)} 2022-03-14 14:10:24 {{TIME}.fmttime(%B %Y)} March 2022 {{TIME}.fmttime(%B,-1M)} February https://www.php.net/manual/en/function.strftime.php 6.0 Certified Professional ● Day 3 © 2023 by Zabbix. All rights reserved Theory 64 EVENT NAME EXAMPLES Lack of available memory {ITEM.VALUE} compared to yesterday {?avg(/{HOST.HOST}/vm.memory.size[available],1h:now-1d)} Raw value Lack of available memory {ITEM.VALUE} compared to yesterday {?avg(/{HOST.HOST}/vm.memory.size[available],1h:now-1d)/1073741824} MB Value in MB Lack of available memory {ITEM.VALUE} compared to yesterday {{?avg(/{HOST.HOST}/vm.memory.size[available],1h:now-1d)/1073741824}.fmtnum(2)} MB Formatted value Lack of available memory {ITEM.VALUE} compared to {{TIME}.fmttime(%A,-1d)} {{?avg(/{HOST.HOST}/vm.memory.size[available],1h:now-1d)/1073741824}.fmtnum(2)} MB Formatted date 6.0 Certified Professional ● Day 3 © 2023 by Zabbix. All rights reserved Theory 65 TREND FUNCTIONS Zabbix has multiple trend* functions for the data analysis of a long period: trendfunction(period,period_shift) period - the time period (minimum '1h'), defined as <N><time unit>: N - number of time units time unit - h (hour), d (day), w (week), M (month) or y (year) period_shift - the absolute time period trendavg(1M,now/M-1y) monthly average a year ago All the trend functions use the data from the trends table: Data in trends table must exist for previous periods Current hour is not included in statistics and has no values until the hour finishes TrendFunctionCache is used to calculate results ! TrendCache is used for trends calculation only, these functions use TrendFunctionCache! 6.0 Certified Professional ● Day 3 © 2023 by Zabbix. All rights reserved Theory 66 TREND FUNCTIONS The complete list of trend functions: trendcount (period,period) Number of successfully retrieved trend values trenddelta (period,period) Difference between the maximum and minimum values trendmax (period,period) The maximum in trend values trendmin (period,period) The minimum in trend values trendavg (period,period) Average of trend values trendsum (period,period) Sum of trend values trendstl (/host/key,period,detection period,season,<deviations>,<devalg>,<s_window>) Anomaly rate 6.0 Certified Professional ● Day 3 © 2023 by Zabbix. All rights reserved Theory 67 STANDARD DEVIATION Deviation is a measure of data variability - how “far” the values are from an average: A standard deviation is a measure of how dispersed the data is in relation to the mean A standard deviation close to zero indicates that data points are close to the mean High or low standard deviation indicates that data points are above or below the mean Low deviation (1) 6.0 Certified Professional ● Day 3 © 2023 by Zabbix. All rights reserved Theory 68 ANOMALY DETECTION AND BASELINE MONITORING Anomaly detection in Zabbix is based on long term trends analytics: It works, if majority of the data is considered "normal" Zabbix uses STL decomposition (Seasonal and Trend decomposition using Loess) Baseline monitoring is based on periods within multiple seasons: Baseline is a value derived from an average over multiple periods of the same length for example, every Tuesday of the past 4 weeks Tuesday is a period week is a season 6.0 Certified Professional ● Day 3 © 2023 by Zabbix. All rights reserved Theory 69 BASELINEWMA baselinewma() function: baselinewma (/host/key,data period:time shift,season_unit,num_seasons) Returns baseline by averaging data periods in seasons Uses Weighted Moving Average algorithm (WMA) trendavg (/Web/nginx.req,1d:now/d) > baselinewma (/Web/nginx.req,1d:now/d,"w",4) * 2 The number of web requests yesterday was over twice as high as the WMA on the corresponding weekdays over the past 4 weeks baselinewma (/host/key,1d:now/d,"w",4) 6.0 Certified Professional ● Day 3 © 2023 by Zabbix. All rights reserved Theory 70 BASELINEDEV baselinedev() function: baselinedev (/host/key,data period,season_unit,num_seasons) Returns number of standard deviations Compares the last period to the periods within seasons baselinedev(/Production server/system.cpu.load,1h,d,10) > 3 Checks if a load for the last hour is more than 3 deviations away from mean Uses 10 one-hour periods over the last 10 days baselinedev (/host/key,1d:now/d,"w",4) 6.0 Certified Professional ● Day 3 © 2023 by Zabbix. All rights reserved Theory 71 ANOMALY DETECTION trendstl() function uses standard deviation to detect anomalies: The function compares a smaller detection period to a larger evaluation period A standard deviation means how far the values are from the average By default, the MAD (median absolute deviation) algorithm is used The number of deviations can be specified starting from 1 (default is 3) s_window is the span (in lags) of the loess window for seasonal extraction trendstl(/host/key,eval period,detection period,season,<deviations>,<devalg>,<s_window>) The function returns an anomaly rate - a decimal value between 0 and 1 The number of (anomaly values) / (total number of values) ! The function is usable only if data has seasonality! 6.0 Certified Professional ● Day 3 © 2023 by Zabbix. All rights reserved Theory 72 ANOMALY DETECTION EXAMPLE trendstl() function example: trendstl(/Production server/net.if.out[eth0],30d:now/d,7d,12h) > 0.1 Analyzing the last 30 days of trend data Find the anomalies rate for the last 7days of that period Expecting the periodicity to be 12h (traffic varies between day and nighttime) The number of deviations to count as anomaly equals 3 (default) MAD algorithm is used (default) Anomaly rate is larger than 0.1 (10% of all values) 6.0 Certified Professional ● Day 3 © 2023 by Zabbix. All rights reserved Theory 73 PREDICTIVE TRIGGER FUNCTIONS Zabbix has functions to predict the future behavior of the monitored system: Forecasting is not magic, it’s statistics based on historical data Trigger functions to use are forecast() and timeleft() Trigger will fire when the system is going to reach the problem state in less than “time to act” Some metrics unfortunately are unpredictable (for example, CPU) Functions will return -1 in these situations: Specified evaluation period contains no data Result of the mathematical operation is not defined: Exponential or power functions involves calculating logarithm of the item values Logarithm function is defined only for positive values Numerical complications: Range and precision of double-precision floating-point format may be insufficient ! Forecasting requires history data; trends data cannot be used! 6.0 Certified Professional ● Day 3 © 2023 by Zabbix. All rights reserved Theory 74 FUTURE DATA PREDICTION The system is expected to be in a problem state after a specific period of time: The forecast function displays the projected value for an item at a future point in time forecast(sec|#num,time,<fit>,<mode>) Parameters: sec | #num time <fit> <mode> time period or number of values forecasting horizon in seconds function used (linear, polynomialN, exponential, logarithmic, power) demanded output (value, max, min, delta, avg) forecast(/Production server/vfs.fs.size[/,free],14d,7d) < 100M 14 days of history 7 days of forecast Now 6.0 Certified Professional ● Day 3 © 2023 by Zabbix. All rights reserved Theory 75 FUTURE DATA PREDICTION The metric is going to reach a given value within a specified time frame: The timeleft function indicates the remaining time until the threshold is reached timeleft(sec|#num,threshold,<fit>) Parameters: sec | #num time period or number of values threshold value to reach <fit> function used (linear, polynomial(1-6), exponential, logarithmic, power) timeleft(/Production server/vfs.fs.size[/,free],14d,1G) < 1h 14 days of history X days until space less than 1G Now 6.0 Certified Professional ● Day 3 © 2023 by Zabbix. All rights reserved Theory 76 Calculated items can be created for a visualization purpose: CALCULATED ITEMS If the functions are used in trigger expressions the prediction data can not be seen The calculated items can be used to display the predicted values forecast(vfs.fs.size[/,free],1d,1h) timeleft(vfs.fs.size[/,free],1d,500M) 6.0 Certified Professional ● Day 3 © 2023 by Zabbix. All rights reserved Theory 77 TREND PREDICTION TIPS Some tips on prediction If you have no insights on how your monitored system behaves, start with the linear (default fit) If your data is not straight, but curved, you may want to try polynomial Forecasting with the default linear fit or polynomial(1-2) are cheap performance-wise Power fit may be useful when your data has "ups" and "downs" Exponential fit may be used for a peak detection take longer intervals with more data points to obtain more accurate data: Even minor jiggling in short data interval will result in huge oscillations in the forecast Forecasts based on longer intervals can be very slow to respond to the rapid change Long-term forecasts: A forecast for a year, based on the data of the last hour is a bad practice; take longer interval All values used for forecasting are stored in the Value cache i https://www.zabbix.com/documentation/6.0/manual/config/triggers/prediction 6.0 Certified Professional ● Day 3 © 2023 by Zabbix. All rights reserved Theory 78 PERCENTILE The percentile function finds the value at a given percentage point in a dataset: It indicates the percentage of data points that are equal to or below a given value It can be used to measure bandwidth level without random peaks or detect various anomalies Example: Percentile 80 is the value, which is greater than 80% of the measured values in given range 6.0 Certified Professional ● Day 3 © 2023 by Zabbix. All rights reserved Theory 79 PERCENTILE EXAMPLES More percentile examples in a 10 number set: The position of the 25th percentile (25/100) * 10 = 2.5 The position of the 50th percentile (50/100) * 10 = 5 The position of the 80th percentile (80/100) * 10 = 8 The 100th percentile refers to the maximum value in a data set = 50 10 values {3, 6, 7, 8, 8, 10, 13, 15, 35, 50} 25th percentile = 6.5 80th percentile = 15 50th percentile = 8 100th percentile = 50 6.0 Certified Professional ● Day 3 © 2023 by Zabbix. All rights reserved Theory 80