Transformers for forecasting: TFT, PatchTST, and iTransformer

Transformers changed language modelling, so it was natural to ask whether the same machinery could forecast electricity load, traffic, retail demand, or airline passengers. The answer turned out to be more interesting than a simple yes. Attention can be useful for time series, but only after several design decisions that have no counterpart in text: what counts as a token, how the model knows when each value happened, and how to stop it from overfitting a few hundred numbers.

This guide is the third in a forecasting series. The statistical guide builds the baselines, the neural guide covers MLPs, recurrent networks, N-BEATS and N-HiTS, and the foundation-model guide covers pretrained zero-shot models. If forecast origins, rolling backtests or MASE are new to you, read the mathematical intuition guide first. Here we go from a one-equation recap of attention, through Informer and Autoformer, to an honest critique, then spend most of our time on the Temporal Fusion Transformer, PatchTST and iTransformer, and finish by training all three on AirPassengers with Nixtla's neuralforecast.

Attention in one equation

The full derivation lives in the Attention Is All You Need walkthrough. The one equation we need is scaled dot-product attention:

Read it as a soft lookup table. Each of $L$ input positions is turned into a query vector (what am I looking for?), a key vector (what do I contain?) and a value vector (what do I hand over if selected?). Stacking them gives matrices $Q$ and $K$ of shape $L\times d_k$ and $V$ of shape $L\times d_v$. The product $QK^\top$ is an $L\times L$ table of similarity scores. Dividing by $\sqrt{d_k}$ keeps the scores from growing with the vector dimension, the softmax turns each row into weights that sum to one, and multiplying by $V$ returns a weighted average of values for every position.

A tiny example: suppose position 3 has scores $(2, 0, 1)$ against positions 1 to 3 after scaling. The softmax weights are about $(0.67, 0.09, 0.24)$, so the output for position 3 is mostly the value of position 1. If position 1 happens to be the same month one year earlier, attention has just learned to look back one season. That is the hope for time series: the model decides which parts of the past matter, rather than a fixed lag structure decided by us.

What changes when the sequence is a time series

Text arrives as discrete tokens from a vocabulary, and each token carries meaning on its own. A time series arrives as real numbers, and a single value such as 417 carries almost no meaning without its neighbours. Three consequences follow.

Point embeddings versus patch embeddings

The first transformer forecasters used point embeddings: every time step becomes a token. The scalar $y_t$ (plus any covariates at $t$) is mapped to a $d$-dimensional vector with a linear layer or a small convolution, $x_t = W y_t + b$. This is simple, but each token is semantically thin, and attention must compare individual noisy points.

Patch embeddings group $P$ consecutive values into one token, $x_n = W\,(y_{s_n}, \dots, y_{s_n+P-1}) + b$. A patch of twelve monthly values describes a local shape (a rise into summer, a dip in November), which is closer to what a word is in text. PatchTST, below, is built on this idea.

Telling the model when things happened

Self-attention is permutation-invariant: shuffle the input tokens and the set of outputs is shuffled in the same way, with no change in content. The model therefore needs explicit order information. Options include the sinusoidal or learned positional encodings from NLP, added to each token, and time-feature encodings that embed calendar facts such as month of year, day of week or hour of day. Informer, for example, adds a timestamp embedding built from such calendar fields. Calendar features are also known in advance for every future step, which makes them natural future covariates.

Quadratic cost in sequence length

The score table $QK^\top$ has $L^2$ entries per head per layer. With point embeddings and a lookback of $L=720$ hourly values, that is $518{,}400$ scores per head per layer. Memory and compute grow quadratically, which motivated a wave of "efficient attention" papers. Patching attacks the same problem differently: with stride $S$, the number of tokens drops from about $L$ to about $L/S$, so the score table shrinks by roughly $S^2$.

Informer and Autoformer in brief

These two models defined the "long-sequence time-series forecasting" benchmark era. You rarely need to use them today, but their ideas recur.

Informer (Zhou et al., 2021) makes three changes to a point-embedding transformer. First, ProbSparse self-attention: the authors observe that most queries produce nearly uniform attention, so only a few "active" queries matter. They score each query with a sparsity measure

the gap between its strongest score and its average score, and compute full attention only for the top $u = c\ln L$ queries (estimated from a sample of keys). The remaining queries receive the mean of the values. This reduces the cost to roughly $O(L\ln L)$. Second, self-attention distilling: between encoder layers, a convolution, an ELU activation and max-pooling halve the sequence length, so deeper layers see shorter, more summarized sequences. Third, a generative-style decoder produces the whole horizon in one forward pass instead of step by step, avoiding error accumulation from feeding predictions back in.

Autoformer (Wu et al., 2021) borrows from classical decomposition. Its series decomposition block splits any hidden sequence $X$ into a trend part and a seasonal part:

The block is used repeatedly inside the network, so trend is progressively separated out. Its second idea replaces dot-product attention with auto-correlation. Instead of comparing individual time steps, it computes the series' autocorrelation $R(\tau)$ for every lag $\tau$ (efficiently, via the fast Fourier transform), keeps the top $k$ lags, and aggregates copies of the series shifted by those lags, weighted by a softmax of their autocorrelations. For monthly data with yearly seasonality, the lag $\tau=12$ should get high weight, so the model averages "this point a year ago" rather than arbitrary points. Both models are available in neuralforecast as Informer and Autoformer.

An honest caution: the linear-baseline critique

In 2022 Zeng et al. asked a pointed question in Are Transformers Effective for Time Series Forecasting?. They compared the published transformer forecasters on the standard long-horizon benchmarks against embarrassingly simple models they called LTSF-Linear: a single linear layer mapping the lookback window directly to the horizon,

plus two variants: DLinear, which first splits the window into trend (a moving average) and remainder and fits one linear layer to each, and NLinear, which subtracts the last value before the linear layer and adds it back after. On their benchmarks these linear models beat the transformer-based ones in most settings, often by a wide margin according to the paper.

Their explanation was that self-attention is permutation-invariant, and positional encodings only partially restore the ordering information that matters most for forecasting. They also showed that several transformers did not improve when given a longer lookback, which suggests they were not extracting much temporal structure.

The lesson for us is not that transformers are useless. PatchTST, published shortly afterwards, reported beating DLinear on the same benchmarks by changing the tokenization. The lesson is that a transformer must earn its place against simple baselines: seasonal naive, ETS, and a linear model. neuralforecast ships DLinear and NLinear, so this check costs one extra line of code.

Temporal Fusion Transformer in depth

The Temporal Fusion Transformer (Lim et al., 2019, published 2021) is less a pure transformer than a carefully engineered hybrid. Its design goal is practical: handle many kinds of inputs, produce quantile forecasts, and offer some interpretability.

Three kinds of inputs

TFT distinguishes inputs by when they are known, which mirrors the information-boundary discipline from the intuition guide:

• Static covariates $s$: constant per series, such as store region or product category.
• Known future inputs $x_t$: values available for both past and future steps, such as calendar month, holidays, or a planned price.
• Observed inputs $z_t$: values only known up to the forecast origin, including the target itself and things like realized weather.

In neuralforecast these map to stat_exog_list, futr_exog_list and hist_exog_list. Putting a variable in the wrong bucket is the most common way to leak the future.

Gated residual networks

The basic building block is the gated residual network (GRN). Given a primary input $a$ and an optional context vector $c$:

with the gated linear unit

Here $\sigma$ is the logistic sigmoid and $\odot$ is element-wise multiplication. The intuition is in the gate. If the sigmoid outputs are near zero, the GLU output vanishes and the GRN reduces to $\operatorname{LayerNorm}(a)$: the block can switch itself off and pass its input through. On small datasets, where extra nonlinearity mostly adds variance, that escape hatch matters. For one hidden unit, if the gate value is $0.1$ and the linear branch is $5$, the GLU contributes only $0.5$ on top of the skip connection.

Variable selection networks

TFT embeds each input variable separately into a $d$-dimensional vector $\xi_t^{(j)}$ for $j = 1,\dots,m$. A variable selection network then learns how much each variable should count at each time step. It concatenates all embeddings into $\Xi_t$ and produces a weight vector with a softmax,

processes each variable with its own GRN, $\tilde{\xi}_t^{(j)}=\operatorname{GRN}_j(\xi_t^{(j)})$, and combines them:

Because the weights are non-negative and sum to one, you can average them across samples to get a rough ranking of variable importance. Separate selection networks exist for static, past and future inputs, and the static context $c_s$ conditions the selection, so a variable can matter for one kind of series and not another.

Static covariate encoders

Four separate GRNs turn the static embedding into four context vectors: $c_s$ for variable selection, $c_c$ and $c_h$ to initialize the LSTM cell and hidden states, and $c_e$ for static enrichment. Static information therefore enters the network at several points rather than being appended once.

Local processing with an LSTM sequence-to-sequence layer

Before any attention, the selected past features pass through an LSTM encoder and the selected future features through an LSTM decoder that continues from the encoder's state. This plays the role of positional encoding: a recurrent layer naturally knows order and captures local patterns such as "the last three months rose". A gated skip connection (GLU plus residual plus LayerNorm) wraps the LSTM so it too can be bypassed.

Static enrichment

Each temporal feature $\phi(t)$ from the LSTM is then passed through one more GRN using the static context, $\theta(t)=\operatorname{GRN}_\theta(\phi(t), c_e)$, so the same local pattern can be interpreted differently for a large city store and a small rural one.

Interpretable multi-head attention

Now comes the transformer part. Standard multi-head attention gives each head its own value projection, which makes it hard to read what the heads are doing. TFT shares one value projection $W_V$ across all heads and averages the heads instead of concatenating them:

Since every head attends over the same values, the averaged attention matrix is a single, meaningful map of which past time steps influenced each forecast step. A causal mask stops each position from attending to later positions. Peaks at lag 12 in that map are the kind of seasonal evidence the paper visualizes.

Quantile outputs and loss

The final layer outputs several quantiles for every horizon step at once. Training minimizes the quantile (pinball) loss summed over a set of quantile levels $\mathcal{Q}$, for example $0.1$, $0.5$, $0.9$:

For $q=0.9$, under-predicting by 10 costs $9$, while over-predicting by 10 costs only $1$, so the optimal prediction sits high: it is the 90th percentile. In neuralforecast, MQLoss(level=[80]) asks for the median and the 10th and 90th percentiles.

flowchart TD
  S["Static covariates"] --> SE["Static covariate encoders<br/>4 GRNs"]
  SE -.-> VS1
  SE -.-> VS2
  SE -.-> LSTM
  SE -.-> EN
  P["Past observed and known inputs"] --> VS1["Variable selection<br/>softmax weights over variables"]
  F["Known future inputs<br/>e.g. month of year"] --> VS2["Variable selection<br/>future"]
  VS1 --> LSTM["LSTM encoder to decoder<br/>local processing plus gated skip"]
  VS2 --> LSTM
  LSTM --> EN["Static enrichment GRN"]
  EN --> ATT["Interpretable multi-head attention<br/>shared values, causal mask"]
  ATT --> FF["Gated position-wise GRN"]
  FF --> Q["Quantile heads<br/>q10, q50, q90 for each horizon step"]

PatchTST: patches and channel independence

PatchTST (Nie et al., 2023) has two ideas, both simple.

Patching math

Take a lookback window of length $L$. Cut it into patches of length $P$ with stride $S$ (the step between patch starts). Without padding, the number of full patches is $\lfloor (L-P)/S \rfloor + 1$. The paper pads the end of the series with $S$ copies of the last value, which adds exactly one more patch:

The neuralforecast implementation does the same thing (a replication pad of width $S$ at the end, then one extra patch), so the formula above is what you get in code. A worked example with our tutorial settings: $L=36$ months, $P=12$, $S=6$. Then $\lfloor 24/6\rfloor + 2 = 6$ patches, starting at months 1, 7, 13, 19, 25 and 31, the last one using the six padded values. With the paper's common setting $L=336$, $P=16$, $S=8$, we get $\lfloor 320/8\rfloor+2=42$ tokens instead of 336, so the attention table shrinks from about $113{,}000$ entries to $1{,}764$.

Each patch vector in $\mathbb{R}^P$ is linearly projected to $\mathbb{R}^D$, a learnable positional encoding is added, and a standard transformer encoder processes the $N$ tokens. A flatten-and-linear head maps the $N\times D$ output to the $H$ forecast values.

Channel independence

For a multivariate dataset with $M$ series, PatchTST does not mix channels. Each series is fed separately through the same shared backbone, as if it were its own univariate problem. That sounds like throwing away information, but it multiplies the number of training examples, shares weights across series, and avoids learning spurious cross-series correlations. The paper reports that channel independence helped on its benchmarks.

Instance normalization (RevIN)

Real series drift in level and scale. PatchTST normalizes each input window by its own statistics before patching and undoes it afterwards, following RevIN (reversible instance normalization, Kim et al., 2022):

where $\mu$ and $\sigma$ are the mean and standard deviation of the current lookback window (RevIN optionally adds a learnable affine step). In neuralforecast this is revin=True, and by default it subtracts the last value rather than the mean (revin_subtract_last=True). For AirPassengers, whose level roughly quadruples, this matters: the network learns shapes, while the normalization carries the level.

flowchart LR
  X["Lookback window<br/>L = 36 values"] --> N["RevIN<br/>normalize per window"]
  N --> PAD["Pad end with S copies<br/>of the last value"]
  PAD --> PT["Cut into patches<br/>P = 12, stride S = 6<br/>N = 6 tokens"]
  PT --> EMB["Linear patch embedding<br/>plus learned position"]
  EMB --> ENC["Transformer encoder<br/>shared across channels"]
  ENC --> HEAD["Flatten and linear head<br/>to H = 12 values"]
  HEAD --> DN["RevIN inverse<br/>restore level and scale"]

iTransformer: inverting the dimensions

iTransformer (Liu et al., 2024) asks what happens if we swap the roles of time and variables. In a classic forecaster, each token is a time step containing all $N$ variables at that moment. In iTransformer, each token is one whole variate: its entire lookback history.

Write the input as a matrix $X\in\mathbb{R}^{L\times N}$, with $L$ time steps and $N$ series. The embedding takes each column (one series' history) through an MLP:

Self-attention then runs over the $N$ variate tokens, so the attention map is $N\times N$ and describes how series relate to one another. The feed-forward network is applied to each variate token separately, learning temporal features of that series, and layer normalization acts on each variate's representation, which reduces scale differences between series. A final linear projection maps each token to its $H$ forecasts. Cost now grows with the number of variables squared, not the lookback length squared, so long lookbacks are cheap.

An honest consequence for this guide: AirPassengers has a single series, so $N=1$ and attention over one token is trivial (the only weight is 1). On a univariate problem, iTransformer behaves like a normalized MLP from the lookback window to the horizon. It is shown below for completeness and API practice, but its real use case is datasets with many related series.

flowchart LR
  subgraph A["Classic: one token per time step"]
    T1["t1: all N variables"] --> AT1["Attention across time<br/>L by L"]
    T2["t2: all N variables"] --> AT1
    T3["tL: all N variables"] --> AT1
  end
  subgraph B["iTransformer: one token per variate"]
    V1["Series 1: whole lookback"] --> E1["MLP embed"]
    V2["Series N: whole lookback"] --> E2["MLP embed"]
    E1 --> AT2["Attention across variates<br/>N by N"]
    E2 --> AT2
    AT2 --> FFN["Per-token FFN<br/>temporal features"]
    FFN --> PR["Linear projection to H"]
  end

Implement it: TFT, PatchTST and iTransformer on AirPassengers

We use the common setup of this series: monthly AirPassengers (144 months, 1949 to 1960), the last 12 months held out, horizon $h=12$, season length 12. The code below was executed in this environment with Python 3.12, neuralforecast 3.3.0, statsforecast 2.1.1, PyTorch 2.15.0 and pandas 2.3.3, on CPU. All three model classes exist in that version. Training the four networks takes under two minutes.

python -m venv .venv && source .venv/bin/activate
pip install neuralforecast statsforecast

Data and a calendar feature

AirPassengersDF uses month-end timestamps, so the frequency must be "ME". With "MS" the forecast dates would not line up with the test dates, the merge would come back empty, and metrics would silently be NaN.

Month of year is a known future input: we know which month each future step falls in. Encoding it as a sine and cosine pair keeps December next to January.

import logging, warnings
warnings.filterwarnings("ignore")
logging.getLogger("pytorch_lightning").setLevel(logging.ERROR)

import numpy as np
import pandas as pd
from neuralforecast import NeuralForecast
from neuralforecast.models import TFT, PatchTST, iTransformer
from neuralforecast.losses.pytorch import MAE, MQLoss
from neuralforecast.utils import AirPassengersDF
from statsforecast import StatsForecast
from statsforecast.models import SeasonalNaive

H, M = 12, 12
df = AirPassengersDF.copy()
df["ds"] = pd.to_datetime(df["ds"])

def add_calendar(frame):
    month = frame["ds"].dt.month
    frame = frame.copy()
    frame["month_sin"] = np.sin(2 * np.pi * month / 12)
    frame["month_cos"] = np.cos(2 * np.pi * month / 12)
    return frame

df = add_calendar(df)
train, test = df.iloc[:-H], df.iloc[-H:]

# A toy static feature: one row per series. With a single series it is a
# constant, so it only demonstrates the API.
static_df = pd.DataFrame({"unique_id": [1.0], "is_international": [1.0]})

Four models, one call

Each model gets a fixed random_seed. TFT receives the calendar features through futr_exog_list and the static column through stat_exog_list. The second TFT uses MQLoss(level=[80]), which trains quantile outputs for the median and an 80% interval. PatchTST and iTransformer normalize internally (RevIN and use_norm), so we turn off the external scaler for them. PatchTST does not accept exogenous inputs in this version. iTransformer does accept future exogenous variables, but we give it only the target history so that it and PatchTST are compared like for like.

common = dict(h=H, input_size=36, max_steps=300, scaler_type="robust",
              accelerator="cpu", enable_progress_bar=False,
              enable_model_summary=False, logger=False)

models = [
    TFT(**common, hidden_size=32, n_head=4, loss=MAE(),
        futr_exog_list=["month_sin", "month_cos"],
        stat_exog_list=["is_international"],
        random_seed=1, alias="TFT"),
    TFT(**common, hidden_size=32, n_head=4, loss=MQLoss(level=[80]),
        futr_exog_list=["month_sin", "month_cos"],
        stat_exog_list=["is_international"],
        random_seed=1, alias="TFT_q"),
    PatchTST(**{**common, "scaler_type": "identity"}, patch_len=12, stride=6,
             encoder_layers=2, n_heads=4, hidden_size=32, linear_hidden_size=64,
             revin=True, learning_rate=1e-3, random_seed=1, alias="PatchTST"),
    iTransformer(**{**common, "scaler_type": "identity"}, n_series=1,
                 hidden_size=64, n_heads=4, e_layers=2, d_ff=128,
                 use_norm=True, random_seed=1, alias="iTransformer"),
]

nf = NeuralForecast(models=models, freq="ME")
nf.fit(df=train, static_df=static_df)

# Future covariates: build the 12 future dates, then add the same features.
futr = add_calendar(nf.make_future_dataframe())
fc = nf.predict(futr_df=futr)
print(fc.columns.tolist())
# ['unique_id', 'ds', 'TFT', 'TFT_q-median', 'TFT_q-lo-80', 'TFT_q-hi-80',
#  'PatchTST', 'iTransformer']

Notice that futr_df contains only calendar columns for future dates, never future values of y. If you had an observed-only covariate, it would go in hist_exog_list and would not appear in futr_df.

Baseline and metrics

sf = StatsForecast(models=[SeasonalNaive(season_length=M)], freq="ME")
snf = sf.forecast(df=train[["unique_id", "ds", "y"]], h=H)
fc = (fc.merge(snf, on=["unique_id", "ds"])
        .merge(test[["unique_id", "ds", "y"]], on=["unique_id", "ds"]))

y_tr = train["y"].to_numpy()
scale = np.mean(np.abs(y_tr[M:] - y_tr[:-M]))  # in-sample seasonal-naive MAE

rows = []
for col in ["SeasonalNaive", "TFT", "PatchTST", "iTransformer", "TFT_q-median"]:
    e = fc["y"] - fc[col]
    rows.append((col, np.mean(np.abs(e)), np.sqrt(np.mean(e**2)),
                 np.mean(np.abs(e)) / scale))
print(pd.DataFrame(rows, columns=["model", "MAE", "RMSE", "MASE"]).round(2))

lo, hi = fc["TFT_q-lo-80"], fc["TFT_q-hi-80"]
print("80% coverage:", np.mean((fc["y"] >= lo) & (fc["y"] <= hi)),
      "mean width:", round(float(np.mean(hi - lo)), 1))

Results on the 1960 holdout from this run (two consecutive runs gave identical numbers on this machine; expect different values with other seeds, library versions or hardware):

The quantile TFT's 80% interval covered only 2 of the 12 test months (coverage 0.17) with an average width of 28.3 passengers. The first rows looked like this:

A cheap rolling-origin check

One holdout year is a single draw. cross_validation gives more evidence. To keep the 1960 test year untouched, we run it on the training part only, with two 12-month windows. With the default refit=False, the network is trained once before the first window and then slides forward using the actual history as input.

cv_nf = NeuralForecast(models=[PatchTST(**{**common, "scaler_type": "identity"},
        patch_len=12, stride=6, encoder_layers=2, n_heads=4, hidden_size=32,
        linear_hidden_size=64, learning_rate=1e-3, random_seed=1,
        alias="PatchTST")], freq="ME")
cv = cv_nf.cross_validation(df=train[["unique_id", "ds", "y"]],
                            n_windows=2, step_size=12)
cv_sn = sf.cross_validation(df=train[["unique_id", "ds", "y"]], h=H,
                            n_windows=2, step_size=12)
cv = cv.merge(cv_sn[["unique_id", "ds", "cutoff", "SeasonalNaive"]],
              on=["unique_id", "ds", "cutoff"])
print(cv.groupby("cutoff").apply(lambda g: pd.Series(
    {c: np.mean(np.abs(g["y"] - g[c])) for c in ["SeasonalNaive", "PatchTST"]})))

How to read these results

Start with the baseline. Seasonal naive has MASE 1.57 on the holdout: its test error is larger than its own average in-sample error, because AirPassengers grows strongly and repeating last year's values misses that growth. That is a low bar. Every transformer clears it on 1960, and PatchTST does best here. But the rolling check tells a more cautious story: in 1958, a year when traffic barely grew, seasonal naive beat PatchTST comfortably. On a 144-point series, the ranking of models can flip from one year to the next, and two or three windows is still thin evidence.

Second, compare the two TFTs. They share architecture, data and seed, yet the median from the quantile-trained model is noticeably worse than the MAE-trained point forecast. Splitting a small training budget across three quantiles, and the fact that a 300-step run lands somewhere slightly different, both contribute. With this little data, differences of this size between configurations are normal and are not evidence that one loss is better in general.

Third, look at the interval. A nominal 80% interval that covers 17% of outcomes is badly overconfident. Most actual values sit above the upper quantile, which is the signature of a model underestimating the upward trend and having learned its spread from one series' history. Quantile heads are not automatically calibrated. Check coverage on backtests before trusting them, and consider conformal adjustments (available in neuralforecast through PredictionIntervals) or a model with explicit growth such as ETS from the statistical guide.

Finally, keep the iTransformer number in context: with one series it is effectively an MLP with normalization, so its score says nothing about the value of inverted attention.

Pitfalls

Too little data for the parameter count. Even our shrunken TFT has thousands of parameters fit to 132 months. Transformers were designed for datasets with many series or long histories. On a single short series, regularize hard, keep models small, and lean on backtests rather than one holdout.

Seed and step sensitivity. Change random_seed or max_steps and the table above will move, sometimes enough to reorder models. Report averages over several seeds and several windows before drawing conclusions.

Leaking the future through covariates. A variable belongs in futr_exog_list only if its future values are truly known at the forecast origin. Planned prices qualify; realized weather or actual promotion uplift does not. Inside a backtest, rebuild future covariates exactly as they would have existed at each origin.

Frequency and date mismatches. Month-end versus month-start timestamps produce empty merges, and NaN metrics do not raise errors. Assert the length of merged frames.

Scaling twice or not at all. PatchTST and iTransformer normalize per window internally; adding a global scaler on top can interact badly, and removing both can make training unstable on trending data. Know which normalization each model applies.

Reading attention as explanation. TFT's attention weights and variable-selection weights are useful diagnostics, but they describe what the network attended to, not causal effects. A highly weighted variable may be a proxy for something else.

Skipping linear and statistical baselines. Following Zeng et al., always include seasonal naive, ETS or AutoARIMA, and DLinear. If the transformer does not beat them across several windows, it has not earned its maintenance cost.

When to choose a transformer

Transformers make most sense when you have many related series (hundreds or more) so a shared model can learn cross-series patterns, long histories where patching makes long lookbacks affordable, rich covariates with different availability (TFT's sweet spot), or a need for quantile forecasts with some interpretability. They make least sense for a handful of short series, where ETS, ARIMA or Theta are faster, more stable and often more accurate. If you have little data but still want deep-learning-style forecasts, also consider a pretrained model from the foundation-model guide, or the simpler N-HiTS from the neural guide.

flowchart TD
  Q1{"Do you have many related series<br/>or a long history?"}
  Q1 -->|"No, a few short series"| STAT["Use seasonal naive, ETS, ARIMA, Theta<br/>or a pretrained foundation model"]
  Q1 -->|"Yes"| Q2{"Did DLinear or N-HiTS already<br/>beat baselines in backtests?"}
  Q2 -->|"Not tested yet"| BASE["Run baselines and linear models first"]
  Q2 -->|"Yes, and you want more"| Q3{"Rich covariates with different<br/>availability: static, known future, observed?"}
  Q3 -->|"Yes"| TFTN["Temporal Fusion Transformer<br/>with quantile loss"]
  Q3 -->|"No, mostly target history"| Q4{"Many variates with<br/>meaningful cross-series relations?"}
  Q4 -->|"Yes"| ITN["iTransformer"]
  Q4 -->|"No, channels are fairly independent"| PTN["PatchTST"]
  TFTN --> CHECK["Backtest over several windows and seeds<br/>check interval coverage"]
  ITN --> CHECK
  PTN --> CHECK

Practice exercises

1. Verify the patch formula. For $L=36$, $P=12$, $S=6$ compute $N$ by hand, then by slicing a padded NumPy array. Repeat for $L=48$, $P=16$, $S=8$. If you are curious, register a forward hook on the PatchTST backbone and confirm the token count.
2. Add the linear critique. Add DLinear(h=12, input_size=36, max_steps=300, random_seed=1) to the model list and rerun. Where does it land relative to PatchTST on the holdout and in the two CV windows?
3. Seed study. Rerun the whole table with seeds 1 to 5. Report the mean and range of MAE per model. Do any rankings flip?
4. Calendar ablation. Train TFT without futr_exog_list. Does the month encoding help on this series, or does the LSTM already find the seasonality from the target history?
5. Fix the intervals. Increase max_steps, try MQLoss(level=[80, 95]), and compare coverage with a conformal approach. Plot the intervals against the actual 1960 values.
6. Give iTransformer a real job. Use AirPassengersPanel from neuralforecast.utils (two series) or another multi-series dataset, set n_series accordingly, and compare it with PatchTST. Now attention has more than one token to look at.

Further reading

• Lim et al., Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecasting.
• Nie et al., A Time Series is Worth 64 Words: Long-term Forecasting with Transformers (PatchTST).
• Liu et al., iTransformer: Inverted Transformers Are Effective for Time Series Forecasting.
• Zeng et al., Are Transformers Effective for Time Series Forecasting?
• Zhou et al., Informer and Wu et al., Autoformer.
• The neuralforecast documentation for current model signatures and exogenous-variable handling.
• Hyndman and Athanasopoulos, Forecasting: Principles and Practice, for the baselines every transformer should be compared against.