2.3. Data Preprocessing and Calibration

Preprocessing is essential to reducing data volume, improving signal clarity, and preparing the data for analysis. It includes noise correction, band selection, and spectral normalisation.

Noise Removal

However, the sensitivity that makes HSI so powerful also introduces specific challenges. The signal can be susceptible to noise because it captures very narrow spectral bands. Variations introduced by the sensor’s electronics, stray light, or atmospheric interference can distort the data. Hyperspectral systems must maintain a high signal-to-noise ratio (SNR) to ensure accurate readings, meaning the actual surface spectral signal must be strong relative to the background noise. A low SNR can obscure essential features or even produce false readings.

Sensors may introduce noise through striping, sensor drift, or random interference. Common techniques include:

• Destriping filters: Correct systematic lines caused by sensor inconsistencies
• Bad band removal: Discard bands affected by water vapour absorption (~1400 nm, ~1900 nm)
• Smoothing filters: Apply techniques like Savitzky–Golay to reduce random spectral noise

Band Selection and Dimensionality Reduction

Hyperspectral images contain hundreds of spectral bands, but not all of them are  useful for analysis. Redundant bands add noise, increase computation time, and may degrade model performances. To address this, dimensionality reduction techniques are used to retain the most informative spectral data while disregarding the rest.

Dimensionality reduction transforms high-dimensional data into a lower-dimensional space, preserving the essential information. Two primary approaches are used in hyperspectral analysis: feature engineering and feature selection.

Feature engineering involves mathematically transforming the original spectral data into a new feature space that highlights relevant patterns. This approach does not select existing bands, but instead projects or combines features to create new ones.

Techniques include:

• Parametric linear methods like Principal Component Analysis (PCA) and Partial Least Squares (PLS).
• Non-parametric linear methods for datasets without assumed distributions.
• Non-linear methods leveraging machine learning models to capture complex relationships in the data.

These methods aim to compress spectral data into fewer components that still represent the majority of the original variability.

Feature selection involves choosing the most informative original spectral bands while discarding less useful ones. Unlike feature engineering, this method retains the selected bands, making results easier to interpret.

There are three main types of methods:

• Filter methods: Evaluate features using statistical tests (e.g., Chi-squared, correlation coefficients) without involving machine learning models.
• Wrapper methods: Use machine learning algorithms to assess combinations of features based on model performance.
• Embedded models: Perform feature selection during model training, often using regularisation techniques (e.g., LASSO, Random Forest).

Feature selection helps avoid overfitting, reduces model training times, and improves generalisation to new data.

By applying these dimensionality reduction techniques, analysts can narrow down from 100+ bands to a manageable set of 10-50 relevant features, streamlining workflows while retaining critical spectral information for analysis.

Spectral Normalisation Techniques

Hyperspectral data is sensitive to external factors such as varying illumination, atmospheric effects, surface shadows, and sensor viewing angles. These factors can distort the raw reflectance values, making direct comparisons between scenes, or even between different areas within a scene, unreliable.

Spectral normalisation techniques are essential to adjust for these inconsistencies, ensuring that the spectral signatures used in analysis represent true material properties rather than external distortions.

Different normalisation methods transform spectral data to reduce brightness and slope effects while preserving the material-specific absorption features. These methods fall into linear and nonlinear categories depending on how they adjust the spectral data. Some examples include:

• Maximum and Minimum Reflectance Method (Linear): Scales each spectrum between its minimum and maximum reflectance values to correct for brightness variations; simple to apply but doesn’t adjust for shape distortions caused by terrain or atmospheric effects.
• Full-Spectrum Reflectance Method (Linear): Divides all reflectance values by the spectrum’s maximum reflectance, reducing scene-wide brightness differences while preserving spectral shape; can be sensitive to noise if the peak reflectance occurs in a noisy band.
• Zero-Mean Full-Spectrum Method (Linear Transformation): Centers spectra by subtracting the mean reflectance value across all bands, enhancing absorption features by removing overall brightness offset; useful for statistical analyses but alters absolute reflectance levels.
• Uniform Scaling Approach (Nonlinear): Applies a nonlinear scaling function to align spectra with varying brightness levels, effective in complex terrains with variable illumination; requires empirical tuning and may introduce artifacts if misapplied.

Spectral normalisation ensures that spectral signatures remain consistent across varying acquisition conditions such as changes in illumination, sensor viewing angles, or atmospheric effects. By reducing brightness-induced variance, normalisation enhances the reliability of classification and target detection algorithms, especially those that rely on the shape of spectral curves rather than absolute reflectance values.

This makes normalisation an essential pre-processing step in hyperspectral workflows, ensuring that the spectral information reflects true material properties rather than external acquisition factors.

Summary

‍