Transfer Conformal Prediction Intervals

This tutorial evaluates the conformal prediction transfer methods available in mlforecast. We train a model on one domain (M4 Monthly Macro series) and then generate calibrated prediction intervals for a different domain (M4 Monthly Finance series) — without retraining the model.

Why transfer?

Standard conformal prediction intervals are calibrated on the same distribution as training data. When you apply a pretrained model to new, unseen series from a potentially different domain, the source conformity scores may be miscalibrated for the target domain. Transfer conformal methods attempt to correct for this shift using different strategies:

Method	Strategy	Needs CV on target?
recalibrate	Re-run cross-validation on target data	Yes
scale_aligned	Rescale source errors by target/source scale ratio (from y history)	No
error_scaled	Rescale source errors by target/source prediction error ratio	Yes
weighted_conformal	Reweight source errors via density-ratio estimation (covariate shift)	No
scale_aligned_weighted	Combine scale alignment with density-ratio weighting	No

We evaluate each method’s empirical coverage — the fraction of test observations that fall inside the predicted interval — and compare it to the nominal level.

Setup

import warnings
warnings.filterwarnings('ignore')

import lightgbm as lgb
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.preprocessing import FunctionTransformer

from mlforecast import MLForecast
from mlforecast.lag_transforms import (
    ExpandingMean,
    ExponentiallyWeightedMean,
    RollingMean,
    RollingStd,
    SeasonalRollingMean,
)
from mlforecast.target_transforms import Differences, GlobalSklearnTransformer
from mlforecast.utils import PredictionIntervals

Load M4 Monthly Data

The M4 Monthly dataset contains 48,000 monthly time series across 6 categories. We use it to create a cross-domain transfer scenario:

• Source domain: Macro category — macroeconomic time series
• Target domain: Finance category — financial time series

The forecast horizon for M4 Monthly is h = 18 months.

HORIZON = 18
DATA_DIR = '../../../data'

def read_and_melt(file):
    df = pd.read_csv(file)
    df.columns = ['unique_id'] + list(range(1, df.shape[1]))
    df = pd.melt(df, id_vars=['unique_id'], var_name='ds', value_name='y')
    df = df.dropna()
    df['ds'] = df['ds'].astype(int)
    return df

# Load train/test splits (M4 stores them separately)
m4_train = read_and_melt(f'{DATA_DIR}/m4/datasets/Monthly-train.csv')
m4_test = read_and_melt(f'{DATA_DIR}/m4/datasets/Monthly-test.csv')

# Adjust test ds so it continues from where train ends
last_train_ds = m4_train.groupby('unique_id')['ds'].max().reset_index()
last_train_ds.columns = ['unique_id', 'last_ds']
m4_test = m4_test.merge(last_train_ds, on='unique_id')
m4_test['ds'] = m4_test['ds'] + m4_test['last_ds']
m4_test = m4_test.drop(columns='last_ds')

print(f"Train: {m4_train.shape}, Test: {m4_test.shape}")
print(f"Training series count: {m4_train['unique_id'].nunique()}")
print(f"Test periods per series: {m4_test.groupby('unique_id')['ds'].count().unique().tolist()}")

Train: (10382411, 3), Test: (864000, 3)
Training series count: 48000
Test periods per series: [18]

# Load category labels from M4 info
m4_info = pd.read_csv(f'{DATA_DIR}/m4/datasets/M4-info.csv', usecols=['M4id', 'category'])
m4_info = m4_info[m4_info['M4id'].str.startswith('M')].rename(columns={'M4id': 'unique_id'})

print("M4 Monthly category counts:")
print(m4_info['category'].value_counts())

M4 Monthly category counts:
category
Finance        10987
Micro          10975
Industry       10017
Macro          10016
Demographic     5728
Other            277
Name: count, dtype: int64

Create Source and Target Domains

rng = np.random.default_rng(42)

# Source domain: sample from Macro category
macro_ids = m4_info[m4_info['category'] == 'Macro']['unique_id'].values
source_ids = rng.choice(macro_ids, size=800, replace=False)

# Target domain: sample from Finance category (disjoint from source)
finance_ids = m4_info[m4_info['category'] == 'Finance']['unique_id'].values
target_ids = rng.choice(finance_ids, size=200, replace=False)

# Source: use all training data (no test split needed for fitting)
source_train = m4_train[m4_train['unique_id'].isin(source_ids)].copy()

# Target: training data goes into new_df; test data is our evaluation ground truth
target_train = m4_train[m4_train['unique_id'].isin(target_ids)].copy()
target_test = m4_test[m4_test['unique_id'].isin(target_ids)].copy()

print(f"Source domain: {source_train['unique_id'].nunique()} Macro series")
print(f"  Train obs: {len(source_train):,}")
print(f"  Length range: {source_train.groupby('unique_id')['ds'].count().agg(['min','max']).to_dict()}")
print()
print(f"Target domain: {target_train['unique_id'].nunique()} Finance series")
print(f"  Train obs: {len(target_train):,}")
print(f"  Test obs: {len(target_test):,} ({HORIZON} steps per series)")

Source domain: 800 Macro series
  Train obs: 186,308
  Length range: {'min': 62, 'max': 1230}

Target domain: 200 Finance series
  Train obs: 38,794
  Test obs: 3,600 (18 steps per series)

Explore the Domains

Let’s visualize a few series from each domain to get a sense of the distribution shift.

fig, axes = plt.subplots(2, 3, figsize=(15, 6))

sample_source = source_ids[:3]
sample_target = target_ids[:3]

for ax, uid in zip(axes[0], sample_source):
    s = source_train[source_train['unique_id'] == uid]
    ax.plot(s['ds'], s['y'])
    ax.set_title(f'Source (Macro): {uid}')
    ax.set_xlabel('Period')

for ax, uid in zip(axes[1], sample_target):
    s = target_train[target_train['unique_id'] == uid]
    ax.plot(s['ds'], s['y'], color='orange')
    ax.set_title(f'Target (Finance): {uid}')
    ax.set_xlabel('Period')

plt.tight_layout()
plt.suptitle('Source vs Target Domain Series', y=1.02, fontsize=14)
plt.show()

Fit MLForecast on Source Domain

Feature engineering for cross-domain transfer

Because the model is trained on one domain and applied to another, the features must be scale-invariant. Tree models cannot extrapolate: if a Finance series has level changes larger than anything seen in the Macro training data, its predictions get clamped to the training range, producing large, skewed errors that no conformal correction can fully repair.

We therefore model log-returns instead of raw differences:

• GlobalSklearnTransformer(FunctionTransformer(np.log1p, np.expm1)) followed by Differences([1]) — the model sees relative changes, which live on a comparable scale in both domains. Back-transformed intervals scale multiplicatively with each series’ level.
• Volatility and trend features (RollingStd, ExponentiallyWeightedMean, SeasonalRollingMean) — these sharpen the point forecasts and, importantly, give the density-ratio estimator meaningful covariates: on the raw scale, lag features mostly encode the scale difference between domains rather than the dynamics.

Prediction interval configuration

We fit on Macro series using PredictionIntervals with: - method='weighted_conformal_error' — stores lag features in the conformity score dataframe, enabling density-ratio estimation (DRE) for the weighted_conformal and scale_aligned_weighted transfer methods. - scale_estimator='mad' — stores per-series scale estimates (MAD of first differences), enabling the scale_aligned and scale_aligned_weighted transfer methods.

Using this single fit configuration unlocks all five transfer methods.

mlf = MLForecast(
    models=lgb.LGBMRegressor(n_estimators=100, verbosity=-1, random_state=0),
    freq=1,
    lags=[1, 2, 3, 4, 6, 12],
    lag_transforms={
        1: [
            ExpandingMean(),
            RollingMean(window_size=3),
            RollingStd(window_size=12, min_samples=6),
            ExponentiallyWeightedMean(alpha=0.3),
        ],
        12: [SeasonalRollingMean(season_length=12, window_size=2, min_samples=1)],
    },
    target_transforms=[
        GlobalSklearnTransformer(FunctionTransformer(func=np.log1p, inverse_func=np.expm1)),
        Differences([1]),
    ],
    num_threads=1,
)

mlf.fit(
    source_train,
    prediction_intervals=PredictionIntervals(
        n_windows=2,
        h=HORIZON,
        method='weighted_conformal_error',
        scale_estimator='mad',
    ),
)

MLForecast(models=[LGBMRegressor], freq=1, lag_features=['lag1', 'lag2', 'lag3', 'lag4', 'lag6', 'lag12', 'expanding_mean_lag1', 'rolling_mean_lag1_window_size3', 'rolling_std_lag1_window_size12_min_samples6', 'exponentially_weighted_mean_lag1_alpha0.3', 'seasonal_rolling_mean_lag12_season_length12_window_size2_min_samples1'], date_features=[], num_threads=1)

Evaluate Transfer Methods

For each transfer method, we call mlf.predict() with: - new_df=target_train — the target domain training history (Finance series) - level=[80, 90, 95] — the requested coverage levels - transfer_conformal=method — which transfer strategy to use

We then merge predictions with target_test and compute the empirical coverage at each level.

LEVELS = [80, 90, 95]
MODEL = 'LGBMRegressor'

transfer_methods = [
    'recalibrate',
    'scale_aligned',
    'error_scaled',
    'weighted_conformal',
    'scale_aligned_weighted',
]

coverage_results = {}

for method in transfer_methods:
    print(f"Running '{method}'...", end=' ', flush=True)
    preds = mlf.predict(
        h=HORIZON,
        level=LEVELS,
        new_df=target_train,
        transfer_conformal=method,
    )
    merged = target_test.merge(preds, on=['unique_id', 'ds'])
    cov = {}
    for lv in LEVELS:
        lo_col = f'{MODEL}-lo-{lv}'
        hi_col = f'{MODEL}-hi-{lv}'
        covered = (merged['y'] >= merged[lo_col]) & (merged['y'] <= merged[hi_col])
        cov[lv] = float(covered.mean())
    coverage_results[method] = cov
    print(f"done. Coverage: { {k: f'{v:.1%}' for k,v in cov.items()} }")

print("\nAll methods evaluated.")

Running 'recalibrate'... done. Coverage: {80: '76.6%', 90: '85.9%', 95: '92.4%'}
Running 'scale_aligned'... done. Coverage: {80: '75.6%', 90: '87.2%', 95: '92.9%'}
Running 'error_scaled'... done. Coverage: {80: '76.3%', 90: '86.3%', 95: '91.7%'}
Running 'weighted_conformal'... done. Coverage: {80: '80.7%', 90: '89.4%', 95: '94.5%'}
Running 'scale_aligned_weighted'... done. Coverage: {80: '79.7%', 90: '90.5%', 95: '95.0%'}

All methods evaluated.

Results: Nominal vs Empirical Coverage

A well-calibrated method should have empirical coverage close to nominal. We show the results as a summary table and a bar chart.

# Build results dataframe
rows = []
for method, cov_dict in coverage_results.items():
    for lv, empirical in cov_dict.items():
        rows.append({
            'Method': method,
            'Nominal Level': f'{lv}%',
            'Nominal': lv / 100,
            'Empirical': empirical,
            'Gap': empirical - lv / 100,
        })

results_df = pd.DataFrame(rows)

# Pivot for display
pivot = results_df.pivot(index='Method', columns='Nominal Level', values='Empirical')
pivot = pivot[['80%', '90%', '95%']]
pivot.columns.name = 'Empirical Coverage @'
display_df = (pivot * 100).round(1).astype(str) + '%'

print("Empirical Coverage by Transfer Method (nominal levels: 80%, 90%, 95%)")
print("=" * 70)
print(display_df.to_string())

Empirical Coverage by Transfer Method (nominal levels: 80%, 90%, 95%)
======================================================================
Empirical Coverage @      80%    90%    95%
Method                                     
error_scaled            76.3%  86.3%  91.7%
recalibrate             76.6%  85.9%  92.4%
scale_aligned           75.6%  87.2%  92.9%
scale_aligned_weighted  79.7%  90.5%  95.0%
weighted_conformal      80.7%  89.4%  94.5%

# Bar chart: empirical vs nominal coverage
fig, axes = plt.subplots(1, 3, figsize=(15, 5), sharey=False)

method_labels = [
    'recalibrate',
    'scale_aligned',
    'error_scaled',
    'weighted\nconformal',
    'scale_aligned\nweighted',
]
x = np.arange(len(transfer_methods))

for ax, lv in zip(axes, LEVELS):
    empirical_vals = [coverage_results[m][lv] * 100 for m in transfer_methods]
    bars = ax.bar(x, empirical_vals, width=0.6, alpha=0.8, label='Empirical')
    ax.axhline(lv, color='red', linewidth=2, linestyle='--', label=f'Nominal {lv}%')
    ax.set_xticks(x)
    ax.set_xticklabels(method_labels, fontsize=9)
    ax.set_title(f'Coverage @ {lv}% Nominal', fontsize=12)
    ax.set_ylabel('Empirical Coverage (%)')
    ax.legend()
    # Add value labels on bars
    for bar, val in zip(bars, empirical_vals):
        ax.text(
            bar.get_x() + bar.get_width() / 2,
            bar.get_height() + 0.3,
            f'{val:.1f}%',
            ha='center', va='bottom', fontsize=8
        )
    y_min = min(min(empirical_vals), lv) - 5
    y_max = max(max(empirical_vals), lv) + 5
    ax.set_ylim(y_min, y_max)

plt.suptitle(
    'Transfer Conformal Coverage: Source=Macro, Target=Finance (M4 Monthly)',
    fontsize=13, y=1.02
)
plt.tight_layout()
plt.show()

Coverage Gap Analysis

# Signed gap: positive = over-coverage, negative = under-coverage
gap_pivot = results_df.pivot(index='Method', columns='Nominal Level', values='Gap')[['80%', '90%', '95%']]
gap_display = (gap_pivot * 100).round(2)

print("Coverage gap (Empirical − Nominal) in percentage points:")
print("  Positive = over-coverage (wider intervals than needed)")
print("  Negative = under-coverage (intervals too narrow)")
print()
print(gap_display.to_string())

# Best method: smallest mean absolute gap across all levels
mean_abs_gap = gap_pivot.abs().mean(axis=1).sort_values()
print("\nMethods ranked by mean |gap| across all levels:")
for method, gap in mean_abs_gap.items():
    print(f"  {method}: {gap*100:.2f} pp")

Coverage gap (Empirical − Nominal) in percentage points:
  Positive = over-coverage (wider intervals than needed)
  Negative = under-coverage (intervals too narrow)

Nominal Level            80%   90%   95%
Method                                  
error_scaled           -3.72 -3.72 -3.33
recalibrate            -3.36 -4.14 -2.61
scale_aligned          -4.44 -2.83 -2.06
scale_aligned_weighted -0.31  0.47  0.03
weighted_conformal      0.69 -0.64 -0.53

Methods ranked by mean |gap| across all levels:
  scale_aligned_weighted: 0.27 pp
  weighted_conformal: 0.62 pp
  scale_aligned: 3.11 pp
  recalibrate: 3.37 pp
  error_scaled: 3.59 pp

Interval Width Analysis

Beyond coverage, we also care about interval sharpness. Narrower intervals (lower width) are better, as long as coverage is maintained.

width_results = {}

for method in transfer_methods:
    preds = mlf.predict(
        h=HORIZON,
        level=LEVELS,
        new_df=target_train,
        transfer_conformal=method,
    )
    widths = {}
    for lv in LEVELS:
        lo_col = f'{MODEL}-lo-{lv}'
        hi_col = f'{MODEL}-hi-{lv}'
        width = (preds[hi_col] - preds[lo_col]).mean()
        widths[lv] = float(width)
    width_results[method] = widths

width_df = pd.DataFrame(width_results).T
width_df.columns = [f'{lv}%' for lv in LEVELS]
print("Mean interval width by method and level:")
print(width_df.round(2).to_string())

Mean interval width by method and level:
                            80%      90%       95%
recalibrate             1596.68  3038.93   5199.22
scale_aligned           3410.02  5759.49   9812.96
error_scaled            1469.70  2755.49   4186.24
weighted_conformal      1883.55  3512.86   5472.55
scale_aligned_weighted  4012.52  7362.65  13545.20

# Scatter plot: width vs coverage gap (ideal: small gap, narrow width)
fig, axes = plt.subplots(1, 3, figsize=(15, 4))

colors = plt.cm.tab10(np.linspace(0, 0.5, len(transfer_methods)))

for ax, lv in zip(axes, LEVELS):
    for i, method in enumerate(transfer_methods):
        gap = abs(coverage_results[method][lv] - lv / 100) * 100
        width = width_results[method][lv]
        ax.scatter(width, gap, color=colors[i], s=120, label=method, zorder=3)
        ax.annotate(
            method.replace('_', '\n'),
            (width, gap), fontsize=7, ha='center', va='bottom'
        )
    ax.set_xlabel('Mean Interval Width')
    ax.set_ylabel('|Coverage Gap| (pp)')
    ax.set_title(f'{lv}% Level: Width vs |Gap|')
    ax.set_ylim(bottom=0)
    ax.grid(True, alpha=0.3)

plt.suptitle('Sharpness vs Calibration Trade-off', fontsize=13, y=1.02)
plt.tight_layout()
plt.show()

Visual Inspection: Interval Examples

Let’s visually inspect the intervals produced by each method for a few target-domain series.

PLOT_LEVEL = 90
EXAMPLE_IDS = target_ids[:4].tolist()

# Collect predictions for all methods
all_preds = {}
for method in transfer_methods:
    preds = mlf.predict(
        h=HORIZON,
        level=[PLOT_LEVEL],
        new_df=target_train[target_train['unique_id'].isin(EXAMPLE_IDS)],
        transfer_conformal=method,
    )
    all_preds[method] = preds

fig, axes = plt.subplots(
    len(EXAMPLE_IDS), len(transfer_methods),
    figsize=(4 * len(transfer_methods), 3 * len(EXAMPLE_IDS)),
    sharex='row'
)

lo_col = f'{MODEL}-lo-{PLOT_LEVEL}'
hi_col = f'{MODEL}-hi-{PLOT_LEVEL}'

for row, uid in enumerate(EXAMPLE_IDS):
    hist = target_train[target_train['unique_id'] == uid].tail(36)
    test = target_test[target_test['unique_id'] == uid]
    for col, method in enumerate(transfer_methods):
        ax = axes[row][col]
        pred = all_preds[method][all_preds[method]['unique_id'] == uid]
        ax.plot(hist['ds'], hist['y'], color='black', linewidth=1)
        ax.plot(pred['ds'], pred[MODEL], color='blue', linewidth=1.5, label='Forecast')
        ax.fill_between(
            pred['ds'], pred[lo_col], pred[hi_col],
            alpha=0.3, color='blue', label=f'{PLOT_LEVEL}% PI'
        )
        ax.scatter(test['ds'], test['y'], color='red', s=20, zorder=5, label='Actuals')
        if row == 0:
            ax.set_title(method.replace('_', '\n'), fontsize=9)
        if col == 0:
            ax.set_ylabel(f'{uid}', fontsize=9)
        ax.tick_params(labelsize=7)

# Single legend at top
handles, labels = axes[0][0].get_legend_handles_labels()
fig.legend(handles, labels, loc='upper right', fontsize=9)

plt.suptitle(
    f'Prediction Intervals ({PLOT_LEVEL}% level) — Finance Target Series',
    fontsize=12, y=1.01
)
plt.tight_layout()
plt.show()

Summary

The table and charts above show how each transfer method calibrates prediction intervals when moving from a Macro source domain to a Finance target domain in M4 Monthly data.

Key takeaways:

• Feature engineering matters as much as the transfer method. Modeling log-returns (log1p + Differences([1])) instead of raw differences makes the features scale-invariant across domains, removes the systematic point-forecast bias, and is what allows the weighted methods to reach near-nominal coverage. With raw differences, every method under-covers by several percentage points.
• recalibrate runs cross-validation on the target data — it tends to be the most directly calibrated but requires running CV (computationally equivalent to retraining).
• scale_aligned uses the scale of the y signal (MAD of differences) to align source residuals — zero-shot, no CV needed.
• error_scaled runs CV on the target data to estimate prediction error magnitude — a middle ground between full recalibration and scale alignment.
• weighted_conformal uses density-ratio estimation to reweight source conformity scores — handles covariate shift without needing target labels during calibration.
• scale_aligned_weighted combines scale alignment with DRE weighting — the most sophisticated zero-shot method.
• The residual under-coverage of the non-weighted methods comes from pooling conformity scores across heterogeneous series: pooled intervals are too wide for calm series and too narrow for volatile ones. This is precisely the failure mode the weighted/scale-aligned variants are designed to mitigate.

The right method to use depends on your constraints: - If you can run CV on the target: recalibrate or error_scaled - If you need zero-shot transfer: scale_aligned or scale_aligned_weighted - If covariate shift is the main concern: weighted_conformal or scale_aligned_weighted