Transfer Learning- Changing learning rate after loading the model

[sinajahangir]

Description

Is it possible to change the learning rate when a model is loaded? I was not able to find any related documentation. For example, if I train a transformer-based model on thousands of data and want to fine-tune it (retrain) for a smaller, new series, using the same training configuration as the source model (e.g., learning rate) does not make sense.

Link

No response

[marcopeix]

Hey!
So, you can do this like so:

nf = NeuralForecast(models=models, freq='M')
nf.fit(df=Y_df)
nf.save()
loaded_model = nf.load()

Then, we can fit on another dataset with

loaded_model.models[0].learning_rate = 1e-4
loaded_model.fit(new_df)

I hope this helps!

[jenaparadies]

Hi,

I am facing the same issue and the proposed solution does not work, although I believe it worked at some point. Could somebody check, if it works or if it is just me?

nf = NeuralForecast.load(path=modelpath)
nf.models[0].max_steps = max_steps
nf.models[0].learning_rate = learning_rate
nf.fit(df=Y_df)

Any recommendations?
Some more information: I am building a NHITS model and I wanted to change the number of steps depending on how much idle time I have to spare for model training. Also, when the model is fully trained I want to use it to train on similar time series, and since it should converge faster, I need a different setting for the learning rate and max_steps.

[marcopeix]

Hi @jenaparadies. Not sure why the code is not working. What's the error? Feel free to reopen the issue if you're still experiencing errors.