Using Pretrained Models
Pretrained models are machine learning models trained previously that can be used as the starting point for your training tasks. Utilizing pretrained models can reduce training time and resource usage.
Configuration Parameters
To download and use models from previous training runs or external sources, use the pretrained-models
parameter in the training config. The keys in this parameter correspond to the training task kinds
capable of using pretrained models. This is currently train-teacher
and train-backwards
. See #515 for train-student
support.
experiment:
pretrained-models:
# Continue training a teacher model.
train-teacher:
urls:
- https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/task-id/artifacts/public/build
mode: continue
type: default
# Re-use an existing backwards model from a Google Cloud Storage bucket. This must
# be the original (non-quantized) student model.
train-backwards:
urls:
- https://storage.googleapis.com/releng-translations-dev/models/en-fi/opusmt/student/
mode: use
type: default
To find models from older training runs see the gs://releng-translations-dev/models
bucket.
For instance you can see the models available for the following commands:
gsutil ls gs://releng-translations-dev/models
And then use the URLs from:
gs://releng-translations-dev/models/en-fi/opusmt/student
This directory should contain the various .npz
and .decoder.yml
for the models, as well as the vocab.spm
. If the vocab.spm
is not present then run something like:
gsutil cp \
gs://releng-translations-dev/models/en-fi/opusmt/vocab/vocab.spm \
gs://releng-translations-dev/models/en-fi/opusmt/student/vocab.spm
The URLs Key
The urls
key is a list that specifies the locations from which the pretrained models are downloaded.
The Mode Key
Use Mode
In use
mode, the pipeline only downloads the model without further training. The tasks that depend on the training task will use the downloaded model artifacts as they are.
Continue Mode
In continue
mode the pipeline uses the downloaded model artifacts from the previous training run as a “checkpoint” and continues training. This is useful to continue
training a model on the same corpus.
Init Mode
In init
mode, the pipeline initializes model weights with the downloaded model using the --pretrained-model
flag in marian
. This is useful for fine-tuning an existing model on a different corpus.
The Type Key
default
is the npz
format that we are using for the model artifacts, this was added with opusmt
in mind.