Smoothing data with correlated errors

Suppose the dependent variable y is observed with error at a set of design points x on an interval, and that the mean of y is assumed to be a smooth function of x. Linear nearest neighbors, kernel regression estimators, and smoothing splines are all examples of techniques for estimating the mean function which depend on a single smoothing parameter, $\lambda$, and are linear functions of the data when $\lambda$ is fixed.

When the error process is weakly continuous, there is a non-zero lower bound on the variance of linear estimators of the mean as the sample size increases on a fixed interval. So the estimators cannot converge in any sense to a deterministic function, as they do when the errors are independent.

The standard techniques for selecting smoothing parameters, such as cross-validation and generalized cross-validation, perform very badly when the errors are correlated. If the sum of the correlations from zero to infinity is negative, the techniques favor over-smoothing; if the sum is positive, the techniques favor undersmoothing. However, the selection criteria can be adjusted to incorporate the known effects of the correlations or the residuals on which the criteria are based can be transformed to eliminate the effects of correlations.

Estimates of the correlation function based on residuals from a preliminary smooth are shown to be very biased. Oversmoothing leads to estimates of correlation which are too large, while undersmoothing leads to estimates which are too small. This leads to a negative feed-back effect which makes iterative techniques inadvisable.

In simulations, the standard selection criteria are shown to behave as predicted by the theory. The corrected criteria are shown to be very effective when the correlation function is known. Although the estimates of correlation based on the data are poor, they are shown to be sufficient for correcting the selection criteria, particularly if the signal to noise ratio is small.