M5

M5 dataset

M5

M5(source_url='https://github.com/Nixtla/m5-forecasts/raw/main/datasets/m5.zip')

M5.download

download(directory)

Downloads M5 Competition Dataset.

Parameters:

Name	Type	Description	Default
directory	str	Directory path to download dataset.	required

M5.load

load(directory, cache=True)

Downloads and loads M5 data.

Parameters:

Name	Type	Description	Default
directory	str	Directory where data will be downloaded.	required
cache	bool	If True saves and loads.	True

Returns:

Type	Description
Tuple[DataFrame, DataFrame, DataFrame]	Tuple[pd.DataFrame, pd.DataFrame, pd.DataFrame]: Target time series with columns [‘unique_id’, ‘ds’, ‘y’], Exogenous time series with columns [‘unique_id’, ‘ds’, ‘y’], Static exogenous variables with columns [‘unique_id’, ‘ds’] and static variables.

M5.source_url

source_url: str = 'https://github.com/Nixtla/m5-forecasts/raw/main/datasets/m5.zip'

Evaluation class

M5Evaluation

M5Evaluation.aggregate_levels

aggregate_levels(y_hat, categories=None)

Aggregates the 30_480 series to get 42_840.

Parameters:

Name	Type	Description	Default
y_hat	DataFrame	Forecasts as wide pandas dataframe with columns [‘unique_id’].	required
categories	DataFrame	Categories of M5 dataset (not used). Defaults to None.	None

Returns:

Type	Description
DataFrame	pd.DataFrame: Aggregated forecasts as wide pandas dataframe with columns [‘unique_id’].

M5Evaluation.evaluate

evaluate(directory, y_hat, validation=False)

Evaluates y_hat according to M4 methodology.

Parameters:

Name	Type	Description	Default
directory	str	Directory where data will be downloaded.	required
validation	bool	Wheter perform validation evaluation. Default False, return test evaluation.	False
y_hat	Union[DataFrame, str]	Forecasts as wide pandas dataframe with columns [‘unique_id’] and forecasts or benchmark url from https://github.com/Nixtla/m5-forecasts/tree/main/forecasts.	required

Returns:

Type	Description
DataFrame	pd.DataFrame: DataFrame with columns OWA, SMAPE, MASE and group as index.

Examples:

m5_winner_url = 'https://github.com/Nixtla/m5-forecasts/raw/main/forecasts/0001 YJ_STU.zip'
winner_evaluation = M5Evaluation.evaluate('data', m5_winner_url)

m5_second_place_url = 'https://github.com/Nixtla/m5-forecasts/raw/main/forecasts/0002 Matthias.zip'
m5_second_place_forecasts = M5Evaluation.load_benchmark('data', m5_second_place_url)
second_place_evaluation = M5Evaluation.evaluate('data', m5_second_place_forecasts)

M5Evaluation.levels

levels: dict = dict(Level1=['total'], Level2=['state_id'], Level3=['store_id'], Level4=['cat_id'], Level5=['dept_id'], Level6=['state_id', 'cat_id'], Level7=['state_id', 'dept_id'], Level8=['store_id', 'cat_id'], Level9=['store_id', 'dept_id'], Level10=['item_id'], Level11=['state_id', 'item_id'], Level12=['item_id', 'store_id'])

M5Evaluation.load_benchmark

load_benchmark(directory, source_url=None, validation=False)

Downloads and loads a bechmark forecasts.

Parameters:

Name	Type	Description	Default
directory	str	Directory where data will be downloaded.	required
source_url	str	Optional benchmark url obtained from https://github.com/Nixtla/m5-forecasts/tree/master/forecasts. If None returns the M5 winner.	None
validation	bool	Wheter return validation forecasts. Default False, return test forecasts.	False

Returns:

Type	Description
ndarray	np.ndarray: Numpy array of shape (n_series, horizon).

Example:

winner_benchmark = M5Evaluation.load_benchmark('data')
winner_evaluation = M5Evaluation.evaluate('data', winner_benchmark)

URL-based evaluation

The method evaluate from the class M5Evaluation can receive a url of a submission to the M5 competiton.

The results compared to the on-the-fly evaluation were obtained from the official evaluation.

m5_winner_url = 'https://github.com/Nixtla/m5-forecasts/raw/main/forecasts/0001 YJ_STU.zip'
winner_evaluation = M5Evaluation.evaluate('data', m5_winner_url)
# Test of the same evaluation as the original one
test_close(winner_evaluation.loc['Total'].item(), 0.520, eps=1e-3)
winner_evaluation

Pandas-based evaluation

Also the method evaluate can recevie a pandas DataFrame of forecasts.

m5_second_place_url = 'https://github.com/Nixtla/m5-forecasts/raw/main/forecasts/0002 Matthias.zip'
m5_second_place_forecasts = M5Evaluation.load_benchmark('data', m5_second_place_url)
second_place_evaluation = M5Evaluation.evaluate('data', m5_second_place_forecasts)
# Test of the same evaluation as the original one
test_close(second_place_evaluation.loc['Total'].item(), 0.528, eps=1e-3)
second_place_evaluation

By default you can load the winner benchmark using the following.

winner_benchmark = M5Evaluation.load_benchmark('data')
winner_evaluation = M5Evaluation.evaluate('data', winner_benchmark)
# Test of the same evaluation as the original one
test_close(winner_evaluation.loc['Total'].item(), 0.520, eps=1e-3)
winner_evaluation

Validation evaluation

You can also evaluate the official validation set.

winner_benchmark_val = M5Evaluation.load_benchmark('data', validation=True)
winner_evaluation_val = M5Evaluation.evaluate('data', winner_benchmark_val, validation=True)
winner_evaluation_val

Kaggle-Competition-M5 References

The evaluation metric of the Favorita Kaggle competition was the normalized weighted root mean squared logarithmic error (NWRMSLE). Perishable items have a score weight of 1.25; otherwise, the weight is 1.0.

$NWRMSLE = \sqrt{\frac{\sum^{n}_{i=1} w_{i}\left(log(\hat{y}_{i}+1) - log(y_{i}+1)\right)^{2}}{\sum^{n}_{i=1} w_{i}}}$

Kaggle Competition Forecasting Methods	16D ahead NWRMSLE
LGBM [1]	0.5091
Seq2Seq WaveNet [2]	0.5129

1. Corporación Favorita. Corporación favorita grocery sales forecasting. Kaggle Competition Leaderboard, 2018.
2. Glib Kechyn, Lucius Yu, Yangguang Zang, and Svyatoslav Kechyn. Sales forecasting using wavenet within the framework of the Favorita Kaggle competition. Computing Research Repository, abs/1803.04037, 2018.