Model Overview

This page gives an overview of the Transformer models currently supported by adapters. The table below further shows which model architectures support which adaptation methods and which features of adapters.

Note

Each supported model architecture X typically provides a class XAdapterModel for usage with AutoAdapterModel. Additionally, it is possible to use adapters with the model classes already shipped with Hugging Face Transformers. For these classes, initialize the model for adapters with adapters.init(model). E.g., for BERT, this means adapters provides a BertAdapterModel class, but you can also use BertModel, BertForSequenceClassification etc. together with adapters.

Model (Bottleneck)
Adapters
Prefix
Tuning
LoRA Compacter Adapter
Fusion
Invertible
Adapters
Parallel
block
Prompt
Tuning
ALBERT
BART
BEIT
BERT-Generation
BERT
CLIP
DeBERTa
DeBERTa-v2
DistilBERT
Electra
Encoder Decoder (*) (*) (*) (*) (*) (*)
GPT-2
GPT-J
Llama
MBart
MT5
RoBERTa
T5
ViT
XLM-RoBERTa
X-MOD

(*) If the used encoder and decoder model class are supported.

Missing a model architecture you’d like to use? adapters can be easily extended to new model architectures as described in Adding Adapters to a Model. Feel free to open an issue requesting support for a new architecture. We very much welcome pull requests adding new model implementations!