Fitting Size Impact

[Arsa-Nik]

Does the fitting size, specifically the number of unique time series in train_data when using
train_time_series_data = TimeSeriesDataFrame.from_data_frame(train_data, id_column='ID', timestamp_column='date')
impact the zero shot forecasts generated by the Chronos Large model for each individual series? For instance, does the forecasts differ when fitting on a single series compared to fitting on five series simultaneously?

[abdulfatir]

Chronos is a univariate model. Each time series is forecasted independently.

[abdulfatir]

I believe the issue here is the large uncertainty in the peak. Here's a MWE without autogluon:

import pandas as pd
import torch
from chronos import ChronosPipeline
import matplotlib.pyplot as plt
import numpy as np

pipeline = ChronosPipeline.from_pretrained(
    "amazon/chronos-t5-small", device_map="cuda", torch_dtype=torch.bfloat16
)

prediction_length = 12
num_samples = 100
df = pd.DataFrame(
    {
        "target": [
            1.0,
            0.0,
            0.0,
            0.0,
            0.0,
            0.0,
            0.0,
            0.0,
            2.0,
            4.0,
            46.0,
            395.0,
            0.0,
            0.0,
            5.0,
            2.0,
            2.0,
            0.0,
            0.0,
            3.0,
            7.0,
            23.0,
            84.0,
            494.0,
            177.0,
            387.0,
            8.0,
            2.0,
            5.0,
            1.0,
            0.0,
            5.0,
            10.0,
            33.0,
            86.0,
            2213.0,
            1331.0,
            141.0,
            19.0,
            5.0,
            1.0,
            5.0,
            7.0,
            6.0,
            7.0,
            17.0,
            202.0,
            3240.0,
            2178.0,
            8.0,
            2.0,
            1.0,
        ]
    }
)

forecast = pipeline.predict(
    context=torch.tensor(df["target"]),
    prediction_length=prediction_length,
    num_samples=num_samples,
)


forecast_index = range(len(df), len(df) + prediction_length)
low, median, high = np.quantile(forecast[0].numpy(), [0.1, 0.5, 0.9], axis=0)

plt.figure(figsize=(8, 4))
plt.plot(df["target"], color="royalblue", label="historical data")
plt.plot(forecast_index, median, color="tomato", label="median forecast")
plt.fill_between(
    forecast_index,
    low,
    high,
    color="tomato",
    alpha=0.3,
    label="80% prediction interval",
)
plt.legend()
plt.grid()
plt.show()

In this example, I have set num_samples=100 (note that autogluon uses 20 by default which may be too low for such data). Even with a larger number of samples, running the code multiple times will give you different higher quantiles suggesting that the model is uncertain about the peak. Here are are a few example plots:

[abdulfatir]

Just for fun, I ran with num_samples=1000 and I get fairly stable results. (cc @lostella)

[Arsa-Nik]

The size of seasonal peak is my follow up question :), However, before that, I'm curious why we obtain the exact same forecast (no variation even in higher quantiles) for series n whether we forecast it in isolation or alongside the second series when we rename the IDs from n and m to ID_1 and ID_2 in the code above. The only variation I observe with AutoGluon occurs when I use the original n and m IDs instead of ID_1 and ID_2. Even then, multiple runs do not change the result; the forecasts for series n alone and alongside series m remain different. Do you have any insights on this?

[abdulfatir]

@Arsa-Nik I looked deeper into this and my conclusion is still that it is related to the inherent uncertainty in the model for this specific time series. Here's what's going on:

AutoGluon sorts the time series dataframe by item ID, so when you use n and m, the n time series will be second in the batch vs. when you use ID_1 and ID_2, the ID_1 time series will stay at the first place. Note that there's nothing special about ID_1 and ID_2 and you can reproduce the same behavior with setting the index to m in the first case and m and n in the second.

Nevertheless, it is interesting that the randomness in the ID_x case is very similar. This is not always the case, however, as you can observe in the following code.

import pandas as pd
from autogluon.timeseries import TimeSeriesPredictor, TimeSeriesDataFrame
import transformers
import random
import torch
import numpy as np

np.random.seed(42)
random.seed(42)
torch.manual_seed(42)
transformers.set_seed(42)

# sample data
data_1 = {
    "ID": ["n"] * 52,
    "date": pd.date_range(start="2018-12-01", periods=52, freq="MS").tolist(),
    "target": [i for i in range(52)],
}

data_2 = {
    "ID": ["n"] * 52 + ["m"] * 52,
    "date": pd.date_range(start="2018-12-01", periods=52, freq="MS").tolist() * 2,
    "target": [i for i in range(52)] + [i + 10 for i in range(52)],
}

# Create DataFrames and ensure ID is of type string
train_data_1 = pd.DataFrame(data_1)
train_data_1["ID"] = train_data_1["ID"].astype(str)

train_data_2 = pd.DataFrame(data_2)
train_data_2["ID"] = train_data_2["ID"].astype(str)

# TimeSeriesDataFrame
train_time_series_data_1 = TimeSeriesDataFrame.from_data_frame(train_data_1, id_column="ID", timestamp_column="date")
train_time_series_data_2 = TimeSeriesDataFrame.from_data_frame(train_data_2, id_column="ID", timestamp_column="date")

# Initialize the predictor
predictor_1 = TimeSeriesPredictor(
    prediction_length=12,
    target="target",
    eval_metric="RMSE",
    freq="MS",
    verbosity=4,
)
predictor_2 = TimeSeriesPredictor(
    prediction_length=12,
    target="target",
    eval_metric="RMSE",
    freq="MS",
    verbosity=4,
)

# Fit the predictor to the training data
predictor_1.fit(
    train_time_series_data_1,
    presets="chronos_small",
    hyperparameters={
        "Chronos": {
            "model_path": "small",
            "batch_size": 32,
            "device": "cuda",
        }
    },
    skip_model_selection=True,
)

predictor_2.fit(
    train_time_series_data_2,
    presets="chronos_small",
    hyperparameters={
        "Chronos": {
            "model_path": "small",
            "batch_size": 32,
            "device": "cuda",
        }
    },
    skip_model_selection=True,
)

# predictions
forecast_1 = predictor_1.predict(train_time_series_data_1)
forecast_2 = predictor_2.predict(train_time_series_data_2)

Case 1 (n and n & m):

forecast_1:

	mean	0.1	0.2	0.3	0.4	0.5	0.6	0.7	0.8	0.9
('n', Timestamp('2023-04-01 00:00:00'))	52.0656	51.7852	51.9348	51.9721	51.9721	51.9721	52.1591	52.1591	52.1591	52.3647
('n', Timestamp('2023-05-01 00:00:00'))	53.1593	52.9069	52.9069	52.9069	53.0938	53.0938	53.1686	53.2808	53.4677	53.4864
('n', Timestamp('2023-06-01 00:00:00'))	54.2062	53.8229	53.8416	53.8416	54.066	54.2155	54.2155	54.4586	54.5894	54.6268
('n', Timestamp('2023-07-01 00:00:00'))	55.2251	54.7577	54.7764	54.9073	54.9633	55.1503	55.2251	55.7111	55.7111	55.7298
('n', Timestamp('2023-08-01 00:00:00'))	56.2533	55.6924	55.8607	55.8981	56.0103	56.085	56.1598	56.702	56.8328	57.0198

forecast_2:

	mean	0.1	0.2	0.3	0.4	0.5	0.6	0.7	0.8	0.9
('n', Timestamp('2023-04-01 00:00:00'))	52.1217	51.9534	51.9721	51.9721	52.0843	52.1591	52.1591	52.1591	52.346	52.346
('n', Timestamp('2023-05-01 00:00:00'))	53.2247	52.9069	53.0565	53.0938	53.0938	53.2808	53.2808	53.2808	53.4677	53.4864
('n', Timestamp('2023-06-01 00:00:00'))	54.309	54.0099	54.0286	54.0286	54.2155	54.2155	54.4025	54.5894	54.5894	54.6081
('n', Timestamp('2023-07-01 00:00:00'))	55.3746	54.9633	54.9633	54.9633	55.2625	55.3372	55.599	55.7111	55.7111	55.7298
('n', Timestamp('2023-08-01 00:00:00'))	56.4589	55.8981	55.8981	56.085	56.3842	56.5524	56.6459	56.8328	56.8328	57.0198

Case 2 (m and m & n):

forecast_1:

	mean	0.1	0.2	0.3	0.4	0.5	0.6	0.7	0.8	0.9
('m', Timestamp('2023-04-01 00:00:00'))	52.0656	51.7852	51.9348	51.9721	51.9721	51.9721	52.1591	52.1591	52.1591	52.3647
('m', Timestamp('2023-05-01 00:00:00'))	53.1593	52.9069	52.9069	52.9069	53.0938	53.0938	53.1686	53.2808	53.4677	53.4864
('m', Timestamp('2023-06-01 00:00:00'))	54.2062	53.8229	53.8416	53.8416	54.066	54.2155	54.2155	54.4586	54.5894	54.6268
('m', Timestamp('2023-07-01 00:00:00'))	55.2251	54.7577	54.7764	54.9073	54.9633	55.1503	55.2251	55.7111	55.7111	55.7298
('m', Timestamp('2023-08-01 00:00:00'))	56.2533	55.6924	55.8607	55.8981	56.0103	56.085	56.1598	56.702	56.8328	57.0198

forecast_2:

	mean	0.1	0.2	0.3	0.4	0.5	0.6	0.7	0.8	0.9
('m', Timestamp('2023-04-01 00:00:00'))	52.0656	51.7852	51.9348	51.9721	51.9721	51.9721	52.1591	52.1591	52.1591	52.3647
('m', Timestamp('2023-05-01 00:00:00'))	53.1406	52.8882	52.9069	52.9069	53.0938	53.0938	53.1686	53.2808	53.4677	53.4864
('m', Timestamp('2023-06-01 00:00:00'))	54.1875	53.6547	53.8416	53.8416	54.066	54.2155	54.2155	54.4586	54.5894	54.6268
('m', Timestamp('2023-07-01 00:00:00'))	55.197	54.5894	54.7764	54.9073	54.9633	55.1503	55.2251	55.7111	55.7111	55.7298
('m', Timestamp('2023-08-01 00:00:00'))	56.2253	55.5055	55.8607	55.8981	56.0103	56.085	56.272	56.702	56.8328	57.0198

As I mentioned before, one way to generate more stable predictions is by setting num_samples to a larger value.

Hope this helps.

cc @canerturkmen

[Arsa-Nik]

@abdulfatir Thank you for investigating this. My follow-up question relates to Chronos not capturing the seasonal peak. Could this be related to the "Overflow and Loss of Precision" section of your paper and the scaling issues associated with seasonal intermittent series? Do you have any recommendations for addressing this? Our data consists of demand data with a mixed range of series, from high-volume items (> 1,000 units per month) to intermittent series.