DAT320: Handling Missing Values

Podcast

Play an AI-generated podcast conversation about this lesson

Questions and Answers

What does MCAR stand for in the context of missing values?

Missing Can Avoid Randomness

Missing Conditioned on Randomness

Missing Completely at Random (correct)

Missing Consistently at Random

Which type of missing value means the probability of missingness relates to the process studied?

MNAR (correct)

MCAR

MAR

What is the primary function of the R package imputeTS?

To handle missing values (correct)

To perform statistical tests

To visualize time series data

To analyze data trends

Which method is used to replace missing values during upsampling?

Interpolation Signup and view all the answers

In downsampling, when is interpolation necessary?

When a non-integer factor is applied Signup and view all the answers

What does MAR indicate in the context of missing values?

Missing at Random Signup and view all the answers

Which of the following best describes a scenario of MCAR?

Data missing purely by chance Signup and view all the answers

What is the purpose of using interpolation in time series data?

To estimate values for missing data points Signup and view all the answers

What does the notation $w_i$ represent in the context of linearly weighted moving average calculations?

The weight assigned to time series values at different lags Signup and view all the answers

Which formula correctly defines the weighted average based on the information provided?

$ ext{µ}_w(x) = w^T x$ Signup and view all the answers

In the context of rolling statistics for missing value replacement, what value does ℓ represent?

The range within which to calculate the average Signup and view all the answers

What is the purpose of using exponentially weighted moving averages?

To reduce the impact of older observations on the average Signup and view all the answers

Which of the following correctly describes the weights for $i > ℓ$ in the exponentially weighted moving average?

$w_i = C imes (1 - eta)^{i - ℓ - 1}$ Signup and view all the answers

What happens if all observations in the current window are NA?

The window size is automatically increased until there are two non-NA values. Signup and view all the answers

What is the default weighting method used for imputation?

Exponential weighting Signup and view all the answers

What is the purpose of the maxgap parameter in na_* functions?

To limit the maximum size of gaps allowed for imputation. Signup and view all the answers

Which formula correctly interpolates a missing value xt given its preceding and following non-missing values?

xt = (t - s1)/(s2 - s1) * xs2 + (s2 - t)/(s2 - s1) * xs1 Signup and view all the answers

Which of the following statements regarding interpolation is true?

Missing values are filled using values from adjacent non-missing observations. Signup and view all the answers

What is a common issue when performing downsampling with a non-integer factor?

Creation of missing values Signup and view all the answers

Which of the following is NOT a method for replacing missing values?

Randomly deleting data points Signup and view all the answers

What does the Last Observation Carried Forward (LOCF) method involve?

Replacing missing data with the prior non-missing observation Signup and view all the answers

What is a major drawback of using global mean or median for missing value replacement?

It does not account for variations within time series Signup and view all the answers

What type of interpolation involves using non-missing neighboring values?

Linear interpolation Signup and view all the answers

Which of the following replacement methods considers the time sequence of data?

Spline interpolation Signup and view all the answers

What is a potential source of bias when using default values for missing data replacement?

It may be too high or low compared to actual values Signup and view all the answers

When would you most likely utilize carry-backward interpolation?

To fill gaps in historical data Signup and view all the answers

In time series analysis, why is simply removing missing values usually not an option?

It disrupts the regular time axis Signup and view all the answers

Which of the following best describes local missing value replacement techniques?

They rely on historical data rather than overall data trends Signup and view all the answers

What does the parameter ℓ represent in the context of rolling average imputation?

The length of the averaging window Signup and view all the answers

Which method of imputation can give more weight to recent observations?

Exponential weighted average Signup and view all the answers

In the context of the provided content, which statement about α is true?

α affects the weight assigned in the exponential weighted average Signup and view all the answers

What does the function na_ma in the package imputeTS primarily focus on?

Performing rolling average imputation Signup and view all the answers

What is the significance of the parameter k in the na_ma function?

It defines the range of rolling average calculations considered Signup and view all the answers

Which of these visual representations primarily shows imputed values over original data?

Density plot with separate lines for original and imputed values Signup and view all the answers

In a plain rolling average with ℓ = 2, how many data points are averaged for each observation?

Two observations, one on each side Signup and view all the answers

What kind of data alteration does missing value replacement via rolling averages perform?

It estimates and replaces missing values Signup and view all the answers

What impact does using a larger ℓ in rolling averages have on the resulting values?

It decreases the impact of fluctuations Signup and view all the answers

Which type of average is less sensitive to outliers in a dataset?

Linear weighted average Signup and view all the answers

What is the primary advantage of spline interpolation over linear interpolation?

It can use higher degree polynomials for more accuracy. Signup and view all the answers

What degree of polynomial is typically used in spline interpolation?

Cubic Signup and view all the answers

Which of the following is NOT a requirement for spline interpolation?

At least three data points Signup and view all the answers

In spline interpolation, what must be true about the polynomials at the knots?

They must meet specific derivative conditions. Signup and view all the answers

What is one of the main limitations of using high-degree polynomial interpolation?

It can lead to overfitting. Signup and view all the answers

What is the purpose of fitting polynomial parameters in spline interpolation?

To ensure the polynomial accurately intersects the data points. Signup and view all the answers

How does spline interpolation handle regions of missing data?

It utilizes multiple piecewise cubic polynomials. Signup and view all the answers

What does the term ‘knots’ refer to in spline interpolation?

The points where the polynomial changes form. Signup and view all the answers

Which of the following approaches is suitable when connecting more than two points in spline interpolation?

Utilize spline functions with cubic polynomials for segments. Signup and view all the answers

How does the derivative condition at the knots affect the spline function?

It guarantees that the splines smoothly connect each segment. Signup and view all the answers

Study Notes

Introduction to DAT320

Course: DAT320: Basics
Topic: Preprocessing: Missing Values, Imputation, and Interpolation
Institution: Norwegian University of Life Sciences
Semester: Autumn 2024
Instructor: Hans Ekkehard Plesser

Missing Values in Time Series

Missing values in time series data require careful handling.
Methods include global and local replacement, and interpolation.

Different Ways Missing Values Occur

Missing Completely At Random (MCAR): Missing values unrelated to the data.
Missing At Random (MAR): Missing values depend on other data.
Missing Not At Random (MNAR): Missing values depend on the variable itself.

Handling Missing Values

Removing Missing Values: Not generally useful for time series.
Replacing Missing Values: Substituting with a fixed value, or mean/median or interpolation, or using rolling mean/median/weighted mean, or linear/spline interpolation.
Interpolation: Using known values to predict missing values.

Handling Missing Values in Time Series

Consecutive Missing Values (sub-periods of missing values)
Single Missing Values
Using packages in R like imputeTS to handle missing values in time series

Upsampling and Downsampling

Upsampling (increasing resolution): Missing values need interpolation.
Downsampling (decreasing): Can require interpolation if non-integer factor used.

Global Missing Value Replacement

Default (often 0 or 1)
Global mean/median may be computed from the data
Using random data might not be suitable as it doesn't consider trends or seasonality.

Local Missing Value Replacement - Carry

Last Observation Carried Forward (LOCF): Replaces missing values with the last observed value.
Next Observation Carried Backward (NOCB): Replaces missing values with the next observed value.

Local Missing Value Replacement - Rolling Average

Rolling Average/Replacement (moving average): Replaces missing values with the mean from local neighbors.
Other methods include median, weighted average.
Linearly and Exponentially weighted moving average may be used.

Interpolation

Linear Interpolation: Missing values are interpolated linearly between previous and next non-missing values. The formula for calculating the missing values is provided.
Issues with linear interpolation: It is not always sufficient to capture the behavior of the time series.
Spline Interpolation: An alternative approach where it uses piecewise polynomial functions rather than connecting straight lines. The process of using spline interpolation is explained in detail.
Choosing an interpolation method: Considerations when selecting linear or spline methods for interpolation.

Spline Interpolation

Spline interpolation approach.
A method for fitting polynomials to data.
Polynomial degree considerations.
Function parameters.
Detail about the use of knot coordinates and polynomial degrees to predict values correctly.

Software details

Package (R package imputeTS): for missing value handling in time series data.
na_ma function and parameter k in a R package imputeTS for rolling average imputation.
maxgap parameter for preventing imputation of long gaps.

Studying That Suits You

Use AI to generate personalized quizzes and flashcards to suit your learning preferences.

Description

This quiz explores methods for preprocessing missing values in time series data, focusing on imputation and interpolation techniques. Understand the different scenarios in which missing values occur and how to effectively handle them in data analysis. Perfect for students of DAT320 at the Norwegian University of Life Sciences.