Time-Frequency Features

While frequency-domain features reveal the spectral content of an EEG segment and time-domain features capture waveform morphology, time-frequency features jointly resolve signal properties in both time and frequency. This is crucial for affective computing because emotional states are dynamic: spectral properties shift over time as emotions unfold, and these transient changes are averaged away in static spectral estimates. Time-frequency representations make it possible to track how the spectrum evolves and to capture transient oscillatory events that are characteristic of emotional processing.

Figure 7.5: Time-frequency representation of EEG. STFT spectrogram aligned to an emotion induction procedure. Power shifts across bands illustrate the dynamic spectral changes that time-frequency features capture.

Short-Time Fourier Transform (STFT)

Definition

The STFT is the most straightforward time-frequency representation. It computes the Fourier transform over a sliding window:

$\text{STFT}(t, f) = \sum_{n=-\infty}^{\infty} x(n) \, w(n - t) \, e^{-j 2\pi f n}$

where $w(n - t)$ is a window function centered at time $t$. The squared magnitude $|\text{STFT}(t, f)|^2$ gives the spectrogram—the time-varying power spectrum.

Window Design

The choice of window length implements the fundamental time-frequency trade-off:

In affective EEG, a window of 1–2 seconds with 50% overlap is a common default that balances responsiveness and spectral reliability.

STFT Features for Emotion Recognition

Features derived from the spectrogram include:

• Time-varying band powers: Power in each frequency band as a function of time, capturing the spectral trajectory of emotional processing.
• Spectral centroid over time: A measure of the center of mass of the spectrum at each time point.
• Spectral flux: The rate of change of the spectrum between consecutive windows, reflecting the speed of spectral transitions.
• Spectral roll-off: The frequency below which a specified percentage of total power is contained, tracked over time.

These features can be vectorized (concatenated across time bins) to form a fixed-length feature vector, or fed into sequential models (RNNs, Transformers) that can exploit their temporal structure.

Wavelet Transform

Continuous Wavelet Transform (CWT)

The CWT decomposes a signal using scaled and translated versions of a mother wavelet $\psi(t)$:

$W(a, b) = \frac{1}{\sqrt{|a|}} \int_{-\infty}^{\infty} x(t) \, \psi^*\!\left(\frac{t - b}{a}\right) dt$

where $a$ is the scale (inversely related to frequency) and $b$ is the translation (time). Unlike the STFT with its fixed window, the CWT adapts its time-frequency resolution: high frequencies are resolved with better time resolution and lower frequency resolution, while low frequencies have better frequency resolution and lower time resolution. This multi-resolution property is well suited to EEG, where fast oscillations (gamma) are brief and slow oscillations (delta) are sustained.

Mother Wavelet Selection

The choice of mother wavelet affects the interpretability of the resulting features:

The Morlet wavelet is most widely used for EEG emotion analysis because its complex-valued output allows separate analysis of amplitude and phase, and its Gaussian envelope provides a natural balance between time and frequency localization.

Wavelet Features

Key features extracted from the wavelet transform include:

• Wavelet power: $|W(a, b)|^2$ — the time-frequency energy distribution.
• Wavelet coherence: A measure of phase consistency between two signals across time and frequency.
• Wavelet entropy: Entropy of the wavelet coefficients at each scale, reflecting the complexity of the signal at that frequency.
• Cross-wavelet power: Joint power between two signals, useful for studying functional coupling.

Discrete Wavelet Transform (DWT)

The DWT provides a computationally efficient, non-redundant decomposition using filter banks:

$x(t) = \sum_k c_{J,k} \, \phi_{J,k}(t) + \sum_{j=1}^{J} \sum_k d_{j,k} \, \psi_{j,k}(t)$

where $c_{J,k}$ are approximation coefficients (low-frequency content) and $d_{j,k}$ are detail coefficients at level $j$ (progressively higher frequencies). The sub-band energies $\sum_k d_{j,k}^2$ for each decomposition level provide a compact set of features that align roughly with the canonical EEG bands.

DWT features are popular because they are fast to compute, produce relatively few coefficients, and the decomposition levels can be chosen to match frequency bands of interest.

Hilbert-Huang Transform (HHT)

The HHT is an adaptive time-frequency method designed for non-stationary and nonlinear signals. It consists of two steps:

Empirical Mode Decomposition (EMD)

EMD decomposes the signal into a set of Intrinsic Mode Functions (IMFs) without assuming any basis functions:

$x(t) = \sum_{i=1}^{M} \text{IMF}_i(t) + r(t)$

Each IMF satisfies two conditions: (1) the number of extrema and zero-crossings differ by at most one, and (2) the mean of the upper and lower envelopes is zero at all points.

Hilbert Spectral Analysis

The Hilbert transform is applied to each IMF to obtain instantaneous amplitude $a_i(t)$ and instantaneous frequency $\omega_i(t)$:

$z_i(t) = \text{IMF}_i(t) + j \mathcal{H}\{\text{IMF}_i(t)\} = a_i(t) e^{j \theta_i(t)}, \quad \omega_i(t) = \frac{d\theta_i(t)}{dt}$

The Hilbert spectrum $H(\omega, t)$ provides a high-resolution time-frequency representation:

$H(\omega, t) = \sum_{i=1}^{M} a_i^2(t) \, \delta(\omega - \omega_i(t))$

HHT in Affective EEG

HHT-based features offer several advantages for emotion recognition:

• Adaptivity: No need to pre-specify frequency bands; the decomposition is data-driven.
• Nonlinearity: EMD naturally handles nonlinear oscillations.
• High resolution: The instantaneous frequency can resolve fine spectral changes.

Features such as the marginal Hilbert spectrum (time-averaged), instantaneous energy, and Hilbert weighted frequencies have been used for emotion classification with competitive results. However, HHT is more computationally intensive and can be sensitive to noise, and EMD can suffer from mode mixing.

Common Spatial Patterns (CSP)

CSP is a spatial filtering technique that finds linear combinations of electrode channels that maximize the variance difference between two conditions. While not a time-frequency method per se, CSP is often applied to band-pass filtered EEG, effectively creating a joint spatial-frequency feature space.

CSP Formulation

For two classes with covariance matrices $\Sigma_1$ and $\Sigma_2$, CSP finds spatial filters $\mathbf{w}$ that satisfy:

$\mathbf{w}^T \Sigma_1 \mathbf{w} = \lambda \, \mathbf{w}^T \Sigma_2 \mathbf{w}$

The filters with the largest and smallest eigenvalues maximize the variance ratio between the two classes. The log-variance of the CSP-filtered signals serves as features:

$f_k = \log\left(\text{var}(\mathbf{w}_k^T \mathbf{X})\right)$

CSP in Affective Computing

CSP is a staple of motor imagery BCI but has been adapted for affective computing by applying it to emotion-relevant frequency bands. Multi-class extensions (e.g., one-vs-rest CSP) and regularized variants (to handle the low sample sizes common in affective datasets) have been explored.

The Filter Bank CSP (FBCSP) extends CSP by applying it to multiple frequency sub-bands and selecting the most discriminative features, a strategy that aligns naturally with the spectral nature of affective EEG.

Figure 7.6: Time-frequency resolution trade-offs. Comparative time-frequency ridges for the same EEG segment. STFT uses a fixed window, CWT provides multi-resolution scaling, and HHT offers adaptive, data-driven ridges.

Comparison of Time-Frequency Methods

Practical Considerations

Feature Tokenization for Small EEG Datasets

Time-frequency representations are often converted directly into dense vectors or presented to a neural network. For small EEG datasets, a more data-efficient alternative is to build a finite library of recurring local patterns and represent each segment by the tokens that it contains. Here, a feature token is not a word and does not have a universal meaning. It is a learned prototype of a waveform, spectrum, or time-frequency patch, together with metadata such as its channel or sensor location, frequency range, time scale, and signal-quality statistics.

Bag-of-Waves as a Feature Token Library

The bag-of-waves (BOW) representation provides a useful example. Short EEG windows are normalized and matched, allowing temporal shifts when appropriate, to a learned dictionary of waveform prototypes. Each longer segment is then represented by an occurrence vector:

$h_k(s) = \sum_{t \in s} \mathbf{1}\left[q(x_t) = k\right]$

where $q$ assigns window $x_t$ to token $k$, and $h_k(s)$ counts how often token $k$ occurs in segment $s$. Counts can be converted to rates, TF-IDF-like weights, or normalized frequencies before fitting a linear model. The resulting representation is compact, compatible with regularized logistic regression, and interpretable: an important feature can be inspected by displaying the waveform prototype and its occurrence rate in each class.

The same idea can tokenize time-frequency features rather than raw waveforms. For example, a token library can be learned from short STFT patches, wavelet-coefficient patches, band-power trajectories, or concatenated channel-frequency patches. A token may therefore represent a transient alpha increase, a stable theta pattern, or a characteristic cross-channel spectral configuration. For an affective EEG segment, the feature vector can be a histogram of these local tokens, optionally augmented with coarse order information such as token transitions or counts in successive temporal bins.

The approach is especially attractive when the number of subjects is small but each recording contains many windows. It turns repeated local structure into evidence that a low-capacity classifier can use without learning a large neural encoder from scratch. Cano Achuri et al. provide a recent EEG example in which shift-invariant waveform dictionaries and occurrence counts form interpretable bag-of-waves biomarkers for mouse neurological disease models (Cano Achuri et al., 2026). The task and species differ from affective computing, so the paper should be read as a representation-learning example rather than as evidence that the same tokens transfer across datasets.

Design and Validation Constraints

Tokenization does not remove the small-sample problem. The dictionary, token frequencies, normalization parameters, and any token-selection rule must be learned inside each training fold. Splits should be grouped by subject, and usually by session, to prevent highly overlapping windows from placing the same participant in both training and test data. A class-conditional dictionary can improve discrimination but may also encode class-specific artifacts; a shared dictionary is often a more conservative starting point for affective EEG.

The histogram discards the order and precise timing of events. This is useful for stable phenotype or trial-level classification, but it may be inadequate when the emotional trajectory matters. Compare token histograms with time-binned histograms, transition features, or a small sequence model, and report performance at the subject or trial level rather than treating correlated windows as independent samples. Keep the library small enough to support repeated cross-validation, inspect token occupancy and stability across folds, and compare against simple band-power and raw-feature baselines.

This feature-library view connects naturally to the LLM-inspired discussion in Chapter 11, Section 10: LLM-Inspired Techniques for EEG. That section discusses neural tokenization, metadata, self-supervised learning, and the limits of importing language analogies. The present approach is a deliberately modest version of that idea: tokens are data-derived measurement patterns for statistical reuse, not semantic words, and their validity must be demonstrated for the target subjects, devices, references, and tasks.

Summary

Time-frequency features bridge the gap between static spectral features and raw time-domain signals, enabling the capture of dynamic spectral changes that accompany emotional processing. The STFT remains the most practical choice for many applications, while wavelet-based methods offer theoretical advantages in resolution, and HHT provides an adaptive alternative for nonlinear signals. For affective computing, the key is to match the time-frequency resolution to the expected time scale of emotional dynamics—typically on the order of seconds, not milliseconds.

References

• Cohen, M. X. (2014). Analyzing Neural Time Series Data: Theory and Practice. MIT Press.
• Mallat, S. (1999). A Wavelet Tour of Signal Processing (2nd ed.). Academic Press.
• Huang, N. E., Shen, Z., Long, S. R., et al. (1998). The empirical mode decomposition and the Hilbert spectrum for nonlinear and non-stationary time series analysis. Proceedings of the Royal Society A, 454(1971), 903–995. https://doi.org/10.1098/rspa.1998.0193
• Bruns, A. (2004). Fourier-, Hilbert- and wavelet-based signal analysis: Are they really different approaches? Journal of Neuroscience Methods, 137(2), 321–332. https://doi.org/10.1016/j.jneumeth.2004.03.002
• Cano Achuri, A., et al. (2026). Interpretable bag-of-waves biomarkers for EEG-based neurological disease models. Journal of Neural Engineering. https://doi.org/10.1088/1741-2552/ae4d8c