Jan 2025 – Jun 2025

Last edited May 26, 2026

Deep Learning for Segmentation of Hyperspectral Satellite Images

This project trained convolutional neural networks for semantic segmentation of hyperspectral satellite imagery from the HYPSO-2 mission, classifying each pixel as sea, land, or cloud.

The pipeline addresses the challenge of training on imagery with hundreds of spectral bands per pixel, orders of magnitude richer than RGB but far more compute-intensive. NVIDIA GPU cluster acceleration was used to optimize training speed and enable rapid model iteration.

Affiliation

NTNU SmallSat Lab

Partners

Roger Birkeland (Advisor)

Report

Manuscript

Keywords

Hyperspectral Imaging
Semantic Segmentation
Convolutional Neural Networks
Lightweight Models
Earth Observation
Spectral Signatures
Python
PyTorch
CUDA
ENVI 5

▸ Deepdive

Introduction

This project, carried out for the NTNU SmallSat Lab in spring 2025, adapts the 1D-JustoLiuNet convolutional neural network, originally trained on hyperspectral imagery from the HYPSO-1 CubeSat, to imagery from its successor satellite, HYPSO-2. The downstream goal is on-board per-pixel segmentation of every image into one of three surface classes (sea, land, cloud), so that the spacecraft can prioritise which data to downlink over a limited communication window. The interesting result is not that the adapted network works, it does, on easy data, but that its accuracy collapses from $0.96$ on near-identical imagery to $0.37$ on geographically diverse imagery, and the writeup is mostly about diagnosing why.

Problem Definition

A HYPSO-2 image is a hyperspectral cube $X \in \mathbb{R}^{H \times W \times B}$ , where $H$ and $W$ are spatial dimensions and $B$ is the number of contiguous spectral bands spanning visible to near-infrared. Each pixel $x_{i,j} \in \mathbb{R}^B$ is its own reflectance spectrum; the task is to learn a classifier $f : \mathbb{R}^B \to \{\text{sea}, \text{land}, \text{cloud}\}$ that takes one pixel’s spectrum and emits its surface class, applied independently across all $H \times W$ pixels to produce a segmentation map. Critically, $f$ is a function of the spectrum only, no spatial neighbourhood, no texture, no context, which is the central architectural commitment that the project is testing.

Per-pixel ground truth is generated by semi-automatic labelling in ENVI (the standard tool for hyperspectral remote-sensing analysis), producing .dat label files aligned to BIP-formatted (Band-Interleaved-by-Pixel) .bip image files. Before any spectrum reaches the network it is normalised band-by-band per image,

x'_{i,j,b} \;=\; \frac{x_{i,j,b}, x^{(b)}_{\min}}{x^{(b)}_{\max}, x^{(b)}_{\min} + \epsilon}, \qquad \epsilon \ll 1,

mapping each band’s values into approximately $[0, 1]$ . Training minimises a softmax cross-entropy loss with label smoothing,

\mathcal{L} \;=\; -\sum_{c=1}^{3} \tilde{y}_c \log p_c, \qquad \tilde{y}_c = (1, \alpha)\, y_c + \frac{\alpha}{3}, \;\; \alpha = 0.1,

over per-pixel predictions, with AdamW as the optimiser ( $\mathrm{lr} = 10^{-3}$ , decoupled weight decay) and StepLR as the scheduler. Reported per-class metrics are precision, recall, $F_1$ , support, and a macro-/weighted-average roll-up plus a confusion matrix; the headline metric is weighted-average $F_1$ over the three classes.

Background

The spectral-signature assumption is the load-bearing idea behind every model in this family: different surface materials reflect different wavelengths with characteristically different intensities, and a high-resolution reflectance vector across $\sim 100$ contiguous bands carries enough information to identify the material even without spatial context.

Illustration of a spaceborne hyperspectral sensor imaging Earth's surface: a satellite captures a stack of co-registered narrow-band images across the visible and near-infrared spectrum, producing a 3D cube whose third dimension is wavelength; each pixel of the cube carries a full reflectance spectrum, and characteristic reflectance curves are shown for soil, water, and vegetation. — Spaceborne hyperspectral imaging in one picture. Every pixel of the captured cube is its own reflectance-vs-wavelength curve, and characteristically different surface materials, soil, water, vegetation, produce characteristically different spectra. Hyperspectral segmentation is the task of recovering the material label per pixel from that spectrum alone.

A neural network is the canonical learnable function for problems where the mapping from input to label is too complex to write down analytically. The base unit is a neuron that computes $z = \sum_i w_i x_i + b$ followed by a non-linear activation $\phi(z)$ ; stacking many such units across input, hidden, and output layers gives the network the capacity to represent arbitrarily complex decision surfaces, with the weights $w_i$ and biases $b$ learned end-to-end by gradient descent on a loss function.

Schematic of a fully-connected neural network with four input neurons, three hidden layers of five neurons each shown as light-purple circles, and three output neurons; every neuron in one layer is connected to every neuron in the next by a weighted edge, and the input/hidden/output role of each layer is annotated above the diagram. — Fully-connected feedforward neural network. Inputs propagate left-to-right through stacked layers of neurons; each connection carries a learnable weight, and the activations at the output layer become the class logits.

A convolutional neural network specialises this structure for inputs with spatial or sequential locality: instead of every output unit being connected to every input unit, each output unit looks at a small local window of the input through a shared learnable kernel that slides across the input. This is exactly the right inductive bias for image-like or spectrum-like inputs, because local structure (an edge, a narrow absorption band, a characteristic spectral slope) is more informative than any individual pixel or band on its own.

Example of a 2D convolution operation: a 6×6 input matrix I containing zeros and ones is convolved with a 3×3 kernel K (whose entries are 1/0/1 along the diagonals and zeros elsewhere); the kernel slides across the input computing a weighted sum at each position, producing a 4×4 output feature map I*K with values that highlight regions of the input matching the kernel pattern. — 2D convolution illustrated. The kernel $K$ slides across the input $I$ computing a weighted sum at each location; the resulting feature map highlights regions where the kernel pattern matches. In 1D-JustoLiuNet the same primitive is applied along the spectral axis with a 1D kernel, learning band-pass detectors over the reflectance spectrum.

The mathematical primitive at every convolutional layer is the same convolution sum,

(x * k)[t] \;=\; \sum_{\tau} x[\tau]\, k[t, \tau],

with the learnable kernel $k$ acting as a tunable band-pass detector. A stack of such convolutions, each followed by a ReLU non-linearity and a MaxPool over the spectral axis, builds an increasingly abstract spectral feature hierarchy; a final fully-connected layer maps that feature vector to three class logits.

1D-JustoLiuNet itself is a lightweight published architecture (Justo et al., 2025) that was designed for the on-board compute budget of a CubeSat: four sequential Conv1D + ReLU + MaxPool1D blocks, a flatten, and a single Linear projection to the class space. The forward pass is small enough to run pixel-by-pixel at the spacecraft’s clock rate, and the published accuracy on HYPSO-1 imagery is $93\,\%$ on a relatively homogeneous evaluation set, which is the number this project is trying to reach (or beat) on HYPSO-2.

Approach

End-to-end architecture. Solid blue is the per-pixel spectral path through the 1D-CNN; dashed orange is the ENVI supervision and the back-propagated optimiser step. The network sees a pixel’s spectrum only, never its spatial neighbourhood, which is the central assumption the diversity tests later stress.

The system decomposes into the hyperspectral input path, the 1D-JustoLiuNet model, and the training loop with its diagnostics. Each ### subsection below pulls one of these out.

Hyperspectral Input Path

The HYPSO-2 imager produces BIP-formatted hyperspectral cubes, where each pixel’s full spectral profile is laid out contiguously in memory, a layout that is exactly right for per-pixel spectral models, since a single sequential read pulls the entire input vector for one network call. Labels are produced semi-automatically in ENVI by combining spectral thresholds with manual cleanup; the resulting .dat files give one of three integer class IDs per pixel. The labels are not perfect, ENVI’s thresholding routinely mistakes sea pixels near clouds for land, and several training images carry visible mislabels (cf. Figure 21 in the report), which becomes a load-bearing problem in the results.

The min-max normalisation is applied per image, per band. The motivation is the standard one: bring all features onto a common scale so the network doesn’t have to learn the per-band dynamic range from scratch. The hidden cost, which this project ended up paying, is that per-image normalisation compresses the between-image spectral variability that the network actually needs to generalise. A sea pixel under Norwegian winter light and a sea pixel under tropical mid-day light have meaningfully different absolute reflectance spectra; per-image normalisation pushes both into roughly $[0, 1]$ and erases that signal.

1D-JustoLiuNet

The forward pass is the four-block convolutional stack followed by a flatten and a fully-connected output. Each block applies a 1D convolution along the spectral axis, a ReLU,

\mathrm{ReLU}(z) \;=\; \max(0, z),

and a 1D max-pool that strides over the spectral axis to compress the feature map. After four such blocks the activations are flattened into a single vector and projected to three logits by a Linear layer; an argmax over the logits gives the predicted class. The network has no batch normalisation, no dropout, and no skip connections, by design, for the on-board compute budget. The cost of those omissions shows up later: ReLU without dropout is prone to neuron death (units that get stuck outputting zero and never recover) and to overfitting, both of which the diversity tests expose.

Training and Evaluation Setup

Training runs on an NVIDIA RTX 3080 with CUDA through PyTorch. The hyperparameters are deliberately conventional, batch size $128$ , learning rate $10^{-3}$ , label smoothing $\alpha = 0.1$ , $10$ epochs, AdamW with decoupled weight decay, StepLR for staircase learning-rate decay, so that any pathological result is attributable to the architecture or the data, not to a fragile training recipe. MLflow tracks per-epoch train/eval accuracy, loss, the full confusion matrix, and per-class precision, recall, $F_1$ , and support, plus the macro and weighted averages.

The evaluation is structured around six controlled tests T01–T06 that vary in difficulty along two axes: how similar the evaluation images are to the training images (similarity index $\in \{0, 1\}$ ), and how visually homogeneous the dataset itself is. T01–T03 are sanity-check easy regimes; T04 is the medium similarity test; T05 and T06 are the hard diversity tests in which train and eval are drawn from a pool of geographically and environmentally varied imagery, the regime that mirrors actual operational deployment.

Test	Type	Difficulty	Images	Sim. index
T01	Sanity	Easy	6	1
T02	Sanity	Easy	8	0
T03	Sanity	Easy	25 (× 1 image)	1
T04	Similarity	Medium	9	1
T05	Diversity	Hard	12	0
T06	Diversity	Hard	27	0

Results

The headline observation across all six tests is a clean monotonic degradation as the dataset becomes more diverse:

Test	Accuracy	Weighted $F_1$	Macro $F_1$	Notable failure
T01 (sanity)	$0.81$	$0.80$	$0.54$	cloud $F_1 = 0.01$ , class collapse
T02 (sanity)	$0.76$	$0.77$	$0.75$	balanced, best macro- $F_1$
T03 (sanity, replicated image)	$0.96$	$0.95$	$0.40$	majority-class memorisation
T04 (similarity)	$0.63$	$0.61$	$0.41$	cloud $F_1 = 0$
T05 (diversity)	$0.42$	$0.39$	$0.35$	sea / land confused
T06 (diversity)	$0.37$	$0.37$	$0.36$	$\approx$ chance for three classes

Two of these numbers are worth dwelling on. T03 trains on $25$ copies of a single image and evaluates on the same image; accuracy looks great ( $0.96$ ) but macro- $F_1$ is $0.40$ because the model has simply memorised the majority class, the confusion matrix shows essentially all cloud pixels predicted as land. T06 is the realistic deployment regime, $27$ visually diverse images, low internal similarity, and weighted $F_1$ collapses to $0.37$ , which for a three-class problem is barely above the majority-class baseline.

Confusion matrix from the final epoch of T03 (the replicated-image sanity test), with rows real labels and columns predicted labels. Almost all cloud pixels (152 of 152) are predicted as land; all 627 512 land pixels are predicted as land; only 3 290 of 25 352 sea pixels are predicted as sea, with the rest predicted as land. The matrix is overwhelmingly concentrated in the land column. — T03 confusion matrix, class collapse on the replicated-image sanity test. Accuracy is $0.96$ because the land class dominates support, but every cloud pixel and almost every sea pixel is misclassified as land; macro- $F_1$ is only $0.40$ .

Confusion matrix from T05 (diversity test): cloud pixels are mostly predicted as land or sea (only 33 723 of 380 366 correct), land pixels are mostly correct (495 864 of 634 497) though many are predicted as sea, and sea pixels are heavily mis-assigned to land (528 992 of 944 185) with only 296 931 correctly classified. — T05 confusion matrix, diversity test. The model has lost the cloud class entirely and confuses sea with land at scale; the dominant prediction column is “land” regardless of the true class.

Confusion matrix from T06 (hardest diversity test): the prediction mass is spread roughly evenly across the three predicted columns for each true class, with no clear diagonal, cloud pixels are split across all three predictions, land pixels are misclassified more often than not, and sea pixels are nearly equally split between predicted sea and predicted land. — T06 confusion matrix, the hardest diversity test. Predictions are roughly uniform across the three columns regardless of the true label; the diagonal that defines a working classifier has effectively disappeared.

Train vs. eval accuracy curves for T05 over 10 epochs. The solid train-accuracy line climbs steadily from about 0.72 to 0.76, while the dashed eval-accuracy line stays roughly flat between 0.42 and 0.50 with no upward trend, a large and growing gap between the two. — T05 train (solid) vs. eval (dashed) accuracy per epoch. Train accuracy climbs to $\approx 0.76$ while eval accuracy stalls around $0.42$ – $0.50$ , the visual signature of the network memorising training spectra rather than learning generalising features.

Train vs. eval accuracy curves for T06 over 10 epochs. The solid train-accuracy line settles around 0.71, while the dashed eval-accuracy line oscillates between roughly 0.36 and 0.39 throughout training, never approaching the training curve. — T06, same gap, more extreme. Train accuracy converges around $0.71$ ; eval accuracy oscillates near $0.37$ for the full ten epochs.

The published 1D-JustoLiuNet paper reports $\sim 93\,\%$ evaluation accuracy on HYPSO-1 imagery, which is a $\approx 56$ -point gap to T06. Four mechanisms together account for this:

1 · Per-image min-max normalisation compresses inter-image variability. Because each image is independently rescaled to $[0, 1]$ , the natural spectral differences between scenes captured under different lighting, atmospheric, and seasonal conditions are flattened out. A sea pixel from one orbit and a sea pixel from another can land at similar normalised values even when their raw spectra are genuinely different, which makes it harder for the network to learn globally-consistent class boundaries.

2 · Label noise. ENVI’s semi-automatic labelling produces mislabels at sea / cloud boundaries (sea pixels near clouds frequently get marked as land). With training data this noisy, the network is being asked to fit signal and noise together; under cross-entropy loss the noisy gradient pulls the decision boundaries away from where they should be.

3 · Class imbalance. Every test exhibits the same fingerprint, the class with the largest support has the highest $F_1$ , and the rarest class is the one the network drops first. T01’s cloud $F_1$ of $0.01$ is the most extreme case. Cross-entropy loss is inherently biased toward majority classes unless explicitly reweighted, and the loss configuration here uses only label smoothing for regularisation, not class weights.

4 · ReLU without dropout. Standard ReLU is prone to neuron death, units that consistently receive negative inputs during training get stuck outputting zero and stop receiving gradient. Combined with the absence of dropout, the network is free to memorise the training set rather than develop generalising features. The train-vs-eval-accuracy gap on T05 and T06 is the visual signature of exactly this failure mode.

Top: original HYPSO-2 ocean scene over Ariake on 2025-02-11, showing predominantly dark open water with thin streaks of cloud and a small landmass. Bottom: the corresponding ENVI-generated label image with the same regions, where sea is coloured blue, land green, and cloud white. Several stretches of open water adjacent to cloud streaks have been labelled green (land) rather than blue (sea). — Original HYPSO-2 ocean scene (top) and its ENVI-generated label image (bottom). Several stretches of open sea adjacent to cloud streaks are mislabelled as land, the systematic supervision noise that propagates through cross-entropy training and biases the decision boundaries.

Future Work

The central finding of the project is that spectral signatures alone are not enough for a model deployed across truly diverse imagery, and the strongest available remedy is to bring spatial information back into the model. Moving from a 1D CNN over a single pixel’s spectrum to a 2D or 3D CNN that operates over a $k \times k$ spectral patch lets the model learn texture, edge, and neighbourhood cues, exactly the cues that distinguish a thin cloud over sea (which 1D-JustoLiuNet has no defence against) from a uniform stretch of land. The cost is non-trivial compute and memory on a CubeSat, but the architectural literature on hyperspectral classification has converged on this answer for a reason: even modest spatial context dramatically improves robustness to lighting and atmospheric variability.

A cheaper second avenue is rethinking the normalisation. Per-image min-max scaling is the standard for hyperspectral data, but in this project it is actively counter-productive because it strips the between-image variability the network needs. A global normalisation computed once across the training corpus, or a learnable per-band normalisation layer, would preserve scene-to-scene spectral differences while still keeping inputs on a stable scale.

The training recipe itself has obvious low-hanging fixes that the conservative hyperparameter choice in this project deliberately did not include. Swapping ReLU for Leaky ReLU keeps a small gradient flowing through “dead” units. Adding dropout (even $p = 0.1$ between blocks) forces the network to spread information across units rather than memorise. Class-reweighted cross-entropy or focal loss directly attacks the class-imbalance bias that is consistently the dominant failure mode in the confusion matrices. None of these change the on-board compute budget, and any one of them should noticeably narrow the train-vs-eval gap on the diversity tests.

Finally, the cleanest single experiment that would resolve the open question in this writeup is to run 1D-JustoLiuNet on the exact dataset the original paper used and compare numbers head-to-head. The current $\approx 56$ -point gap to the paper’s $93\,\%$ accuracy is partly architectural and partly distributional, the paper’s evaluation set was visibly easier (no snow, low environmental variation), and without running the model on the original corpus, the project can’t isolate which fraction of the gap belongs to which cause. That comparison is the right next step before committing to any of the more expensive architectural changes above.