M4

M4 dataset

Other

Other(seasonality=1, horizon=8, freq='D', name='Other', n_ts=5000, included_groups=('Weekly', 'Daily', 'Hourly'))

Hourly

Hourly(seasonality=24, horizon=48, freq='H', name='Hourly', n_ts=414)

Daily

Daily(seasonality=1, horizon=14, freq='D', name='Daily', n_ts=4227)

Weekly

Weekly(seasonality=1, horizon=13, freq='W', name='Weekly', n_ts=359)

Monthly

Monthly(seasonality=12, horizon=18, freq='M', name='Monthly', n_ts=48000)

Quarterly

Quarterly(seasonality=4, horizon=8, freq='Q', name='Quarterly', n_ts=24000)

Yearly

Yearly(seasonality=1, horizon=6, freq='Y', name='Yearly', n_ts=23000)

Download data class

M4

M4(source_url='https://raw.githubusercontent.com/Mcompetitions/M4-methods/master/Dataset/', naive2_forecast_url='https://github.com/Nixtla/m4-forecasts/raw/master/forecasts/submission-Naive2.zip')

M4.async_download

async_download(directory, group=None)

Download M4 Dataset.

Parameters:

Name	Type	Description	Default
directory	str	Directory path to download dataset.	required

Example:

group = 'Hourly'
await M4.async_download('data', group=group)
df, *_ = M4.load(directory='data', group=group)
n_series = len(np.unique(df.unique_id.values))
display_str  = f'Group: {group} '
display_str += f'n_series: {n_series}'
print(display_str)

M4.download

download(directory, group=None)

Download M4 Dataset.

Parameters:

Name	Type	Description	Default
directory	str	Directory path to download dataset.	required
group	str	Name of the group to download. If None, downloads all. Defaults to None.	None

M4.load

load(directory, group, cache=True)

Downloads and loads M4 data.

Parameters:

Name	Type	Description	Default
directory	str	Directory where data will be downloaded.	required
group	str	Group name. Allowed groups: ‘Yearly’, ‘Quarterly’, ‘Monthly’, ‘Weekly’, ‘Daily’, ‘Hourly’.	required
cache	bool	If True saves and loads	True

Returns:

Type	Description
Tuple[DataFrame, Optional[DataFrame], Optional[DataFrame]]	Tuple[pd.DataFrame, Optional[pd.DataFrame], Optional[pd.DataFrame]]: Target time series with columns [‘unique_id’, ‘ds’, ‘y’], Static exogenous variables with columns [‘unique_id’, ‘ds’], and static variables.

Evaluation class

M4Evaluation

M4Evaluation.evaluate

evaluate(directory, group, y_hat)

Evaluates y_hat according to M4 methodology.

Parameters:

Name	Type	Description	Default
directory	str	Directory where data will be downloaded.	required
group	str	Group name. Allowed groups: ‘Yearly’, ‘Quarterly’, ‘Monthly’, ‘Weekly’, ‘Daily’, ‘Hourly’.	required
y_hat	Union[ndarray, str]	Group forecasts as numpy array or benchmark url from https://github.com/Nixtla/m4-forecasts/tree/master/forecasts.	required

Returns:

Type	Description
DataFrame	pd.DataFrame: DataFrame with columns OWA, SMAPE, MASE and group as index.

Examples:

esrnn_url = 'https://github.com/Nixtla/m4-forecasts/raw/master/forecasts/submission-118.zip'
esrnn_evaluation = M4Evaluation.evaluate('data', 'Hourly', esrnn_url)

fforma_url = 'https://github.com/Nixtla/m4-forecasts/raw/master/forecasts/submission-245.zip'
fforma_forecasts = M4Evaluation.load_benchmark('data', 'Hourly', fforma_url)
fforma_evaluation = M4Evaluation.evaluate('data', 'Hourly', fforma_forecasts)

M4Evaluation.load_benchmark

load_benchmark(directory, group, source_url=None)

Downloads and loads a bechmark forecasts.

Parameters:

Name	Type	Description	Default
directory	str	Directory where data will be downloaded.	required
group	str	Group name. Allowed groups: ‘Yearly’, ‘Quarterly’, ‘Monthly’, ‘Weekly’, ‘Daily’, ‘Hourly’.	required
source_url	str	Optional benchmark url obtained from https://github.com/Nixtla/m4-forecasts/tree/master/forecasts. If None returns Naive2.	None

Returns:

Type	Description
ndarray	np.ndarray: Numpy array of shape (n_series, horizon).

URL-based evaluation

The method evaluate from the class M4Evaluation can receive a url of a benchmark uploaded to the M4 competiton.

The results compared to the on-the-fly evaluation were obtained from the official evaluation.

import numpy as np

esrnn_url = 'https://github.com/Nixtla/m4-forecasts/raw/master/forecasts/submission-118.zip'
esrnn_evaluation = M4Evaluation.evaluate('data', 'Hourly', esrnn_url)
# Test of the same evaluation as the original one
assert np.isclose(esrnn_evaluation['SMAPE'].item(), 9.328, atol=1e-3)
assert np.isclose(esrnn_evaluation['MASE'].item(), 0.893, atol=1e-3)
assert np.isclose(esrnn_evaluation['OWA'].item(), 0.440, atol=1e-3)
esrnn_evaluation

Numpy-based evaluation

Also the method evaluate can recevie a numpy array of forecasts.

import numpy as np

fforma_url = 'https://github.com/Nixtla/m4-forecasts/raw/master/forecasts/submission-245.zip'
fforma_forecasts = M4Evaluation.load_benchmark('data', 'Hourly', fforma_url)
fforma_evaluation = M4Evaluation.evaluate('data', 'Hourly', fforma_forecasts)
# Test of the same evaluation as the original one
assert np.isclose(fforma_evaluation['SMAPE'].item(), 11.506, atol=1e-3)
assert np.isclose(fforma_evaluation['MASE'].item(), 0.819, atol=1e-3)
assert np.isclose(fforma_evaluation['OWA'].item(), 0.484, atol=1e-3)
fforma_evaluation