Basis Function Regression

• Linear models are not always sufficient for real-world phenomena because the relationships between inputs and outputs are not always linear, thus requiring nonlinear models.
• In nonlinear regression, choosing the right nonlinear function is important for the effectiveness of the regression.
• An ideal model form matches the underlying phenomenon, but in many cases, machine learning models like basis function regression, neural networks, and k-NN are used.
• The choice of objective function and underlying noise model is important, where extending least squares estimators can help models generalize better to unseen inputs

Basis Function Regression

• A common representation for a function f(x) is the basis function representation: y = f(x) = Σ w_k b_k(x)
• The functions b_k(x) are called basis functions, and the model can be expressed in vector form as y = f(x) = b(x)^T w, where b(x) = [b_1(x),...,b_M(x)]^T and w = [w_1, . . .,w_M]^T.
• Polynomials and Radial Basis Functions (RBF) are two common choices of basis functions.
• A simple basis for polynomials is the monomials.
• For monomials, the regression model is: f(x) = Σ w_k x^k
• Radial Basis Functions are defined as: b_k(x) = exp(-(x - c_k)^2 / 2σ^2), where c_k is the center and σ^2 determines the width.
• The resulting regression model is: f(x) = Σ w_k b_k(x) = Σ w_k exp(-(x - c_k)^2 / 2σ^2).
• The center and width of RBFs are parameters to be determined from training data.
• Other basis functions include sinusoidal functions and combinations of monomials and RBFs.
• Ideally, choose a family of basis functions that fits the data well with a small basis set.
• Least-squares regression is used to fit these models by minimizing the sum of squared residual error: E(w) = Σ (y_i - f(x_i))^2 = Σ (y_i - Σ w_k b_k(x))^2
• Minimizing E with respect to w has the same form as linear regression, and E is quadratic in the weight parameters w.
• Rewriting the objective function in matrix form yields: E(w) = ||y - Bw||^2, where B_ij = b_j(x_i).
• To pick the basis centers: Space the centers uniformly, place one center at each data point, or cluster the data and use one center for each cluster.
• To pick the width parameter: Manually try different values or use average squared distances to neighboring centers, scaled by a constant

Overfitting and Regularization

• Minimizing squared-error directly can lead to overfitting, where the model fits the training data very well but performs poorly on new test data.
• Overfitting can occur when the problem is not sufficiently constrained, when fitting noise, or when discarding uncertainty.
• Two solutions to overfitting are adding prior knowledge and handling uncertainty.
• A common assumption is that the underlying function is likely to be smooth, which reduces model complexity and makes estimation from small datasets easier.
• Smoothness can be added by parameterizing the model in a smooth way or by adding regularization terms to the learning objective function.
• Regularization adds extra terms to the learning objective function to prefer smooth models:
• E(w) = ||y - Bw||^2 + λ||w||^2, where the first term measures model fit and the second penalizes non-smoothness.
• The smoothness term (||w||) is called weight decay, which tends to make the weights smaller and implicitly leads to smoothness with RBF basis functions.
• The regularized least-squares objective function is still quadratic with respect to w and can be optimized in closed-form.
• Set the gradient of E(w) to zero to get the regularized LS estimate for w: w* = (B^T B + λI)^-1 B^T y.

Artificial Neural Networks

• The sigmoid function is a choice for a basis function
• Sigmoid: g(a) = 1 / (1 + e^-a)
• Sigmoids are used as the nonlinearity in Artificial Neural Networks (ANN).
• A sigmoid-based ANN has the form y = f(x) = Σ w_j^(1) g(Σ w_k,j^(2) x_k + b_j^(2)) + b^(1).
• This equation describes a process whereby a linear regressor (w^(2)) is applied to x, and passed to a nonlinear Sigmoid function, the outputs act as features to another linear regressor.
• The inner weights (w^(2)) are distinct parameters from the outer weights w^(1)
• The neural network is a linear combination of shifted (smoothed) step functions, linear ramps, and the bias term.
• Learning an ANN estimates its parameters (network weights).
• A regularized squared-error objective function with weight decay is E(w,b) = ||y - f(x)||^2 + λ||w||^2, where w comprises the weights at both levels.
• Since this is not optimized in closed-form, numerical optimization procedures are used.

k-Nearest Neighbors

• Learning procedures smooth the training data.
• k-Nearest Neighbors regression smooths data directly.
• Algorithm: Select k values, find values close to x in the set, base the Euclidian distance to the input, predict output.
• y = (1/k) Σ y_i
• Weighted: y = (Σ w(x_i)y_i) / (Σ w(x_i)), where w(x_i) = e^(-||x_i - x||^2 / 2σ^2)
• σ controls the smoothing degree.
• Does not compress well