Auto Classes

Similar to the AutoModel classes built-in into HuggingFace Transformers, adapters provides an AutoAdapterModel class. As with other auto classes, the correct adapter model class is automatically instantiated based on the pre-trained model passed to the from_pretrained() method.

Note

If the model loaded with the from_pretrained(...) function has a head, this head gets loaded as well. However, this only works for non-sharded models. If you want to load a sharded model with a head, you first need to load the model and then the head separately.

AutoAdapterModel

class adapters.AutoAdapterModel(*args, **kwargs)

This is a generic model class that will be instantiated as one of the model classes of the library (with a adapters and flexible heads head) when created with the [~AutoAdapterModel.from_pretrained] class method or the [~AutoAdapterModel.from_config] class method.

This class cannot be instantiated directly using __init__() (throws an error).

classmethod from_config(**kwargs)

Instantiates one of the model classes of the library (with a adapters and flexible heads head) from a configuration.

Note

Loading a model from its configuration file does not load the model weights. It only affects the model’s configuration. Use [~AutoAdapterModel.from_pretrained] to load the model weights.

Parameters

  • config ([PretrainedConfig]) –

    The model class to instantiate is selected based on the configuration class:

    • [AlbertConfig] configuration class: [AlbertAdapterModel] (ALBERT model)

    • [BartConfig] configuration class: [BartAdapterModel] (BART model)

    • [BeitConfig] configuration class: [BeitAdapterModel] (BEiT model)

    • [BertConfig] configuration class: [BertAdapterModel] (BERT model)

    • [BertGenerationConfig] configuration class: [BertGenerationAdapterModel] (Bert Generation model)

    • [CLIPConfig] configuration class: [CLIPAdapterModel] (CLIP model)

    • [DebertaConfig] configuration class: [DebertaAdapterModel] (DeBERTa model)

    • [DebertaV2Config] configuration class: [DebertaV2AdapterModel] (DeBERTa-v2 model)

    • [DistilBertConfig] configuration class: [DistilBertAdapterModel] (DistilBERT model)

    • [ElectraConfig] configuration class: [ElectraAdapterModel] (ELECTRA model)

    • [GPT2Config] configuration class: [GPT2AdapterModel] (OpenAI GPT-2 model)

    • [GPTJConfig] configuration class: [GPTJAdapterModel] (GPT-J model)

    • [LlamaConfig] configuration class: [LlamaAdapterModel] (LLaMA model)

    • [MBartConfig] configuration class: [MBartAdapterModel] (mBART model)

    • [MT5Config] configuration class: [MT5AdapterModel] (MT5 model)

    • [MistralConfig] configuration class: [MistralAdapterModel] (Mistral model)

    • [PLBartConfig] configuration class: [PLBartAdapterModel] (PLBart model)

    • [RobertaConfig] configuration class: [RobertaAdapterModel] (RoBERTa model)

    • [T5Config] configuration class: [T5AdapterModel] (T5 model)

    • [ViTConfig] configuration class: [ViTAdapterModel] (ViT model)

    • [WhisperConfig] configuration class: [WhisperAdapterModel] (Whisper model)

    • [XLMRobertaConfig] configuration class: [XLMRobertaAdapterModel] (XLM-RoBERTa model)

    • [XmodConfig] configuration class: [XmodAdapterModel] (X-MOD model)

  • attn_implementation (str, optional) – The attention implementation to use in the model (if relevant). Can be any of “eager” (manual implementation of the attention), “sdpa” (using [F.scaled_dot_product_attention](https://pytorch.org/docs/master/generated/torch.nn.functional.scaled_dot_product_attention.html)), or “flash_attention_2” (using [Dao-AILab/flash-attention](https://github.com/Dao-AILab/flash-attention)). By default, if available, SDPA will be used for torch>=2.1.1. The default is otherwise the manual “eager” implementation.

Examples:

```python >>> from transformers import AutoConfig, AutoAdapterModel

>>> # Download configuration from huggingface.co and cache.
>>> config = AutoConfig.from_pretrained("google-bert/bert-base-cased")
>>> model = AutoAdapterModel.from_config(config)
```
classmethod from_pretrained(*model_args, **kwargs)

Instantiate one of the model classes of the library (with a adapters and flexible heads head) from a pretrained model.

The model class to instantiate is selected based on the model_type property of the config object (either passed as an argument or loaded from pretrained_model_name_or_path if possible), or when it’s missing, by falling back to using pattern matching on pretrained_model_name_or_path:

  • albert – [AlbertAdapterModel] (ALBERT model)

  • bart – [BartAdapterModel] (BART model)

  • beit – [BeitAdapterModel] (BEiT model)

  • bert – [BertAdapterModel] (BERT model)

  • bert-generation – [BertGenerationAdapterModel] (Bert Generation model)

  • clip – [CLIPAdapterModel] (CLIP model)

  • deberta – [DebertaAdapterModel] (DeBERTa model)

  • deberta-v2 – [DebertaV2AdapterModel] (DeBERTa-v2 model)

  • distilbert – [DistilBertAdapterModel] (DistilBERT model)

  • electra – [ElectraAdapterModel] (ELECTRA model)

  • gpt2 – [GPT2AdapterModel] (OpenAI GPT-2 model)

  • gptj – [GPTJAdapterModel] (GPT-J model)

  • llama – [LlamaAdapterModel] (LLaMA model)

  • mbart – [MBartAdapterModel] (mBART model)

  • mistral – [MistralAdapterModel] (Mistral model)

  • mt5 – [MT5AdapterModel] (MT5 model)

  • plbart – [PLBartAdapterModel] (PLBart model)

  • roberta – [RobertaAdapterModel] (RoBERTa model)

  • t5 – [T5AdapterModel] (T5 model)

  • vit – [ViTAdapterModel] (ViT model)

  • whisper – [WhisperAdapterModel] (Whisper model)

  • xlm-roberta – [XLMRobertaAdapterModel] (XLM-RoBERTa model)

  • xmod – [XmodAdapterModel] (X-MOD model)

The model is set in evaluation mode by default using model.eval() (so for instance, dropout modules are deactivated). To train the model, you should first set it back in training mode with model.train()

Parameters

  • pretrained_model_name_or_path (str or os.PathLike) –

    Can be either:

    • A string, the model id of a pretrained model hosted inside a model repo on huggingface.co.

    • A path to a directory containing model weights saved using [~PreTrainedModel.save_pretrained], e.g., ./my_model_directory/.

    • A path or url to a tensorflow index checkpoint file (e.g, ./tf_model/model.ckpt.index). In this case, from_tf should be set to True and a configuration object should be provided as config argument. This loading path is slower than converting the TensorFlow checkpoint in a PyTorch model using the provided conversion scripts and loading the PyTorch model afterwards.

  • model_args (additional positional arguments, optional) – Will be passed along to the underlying model __init__() method.

  • config ([PretrainedConfig], optional) –

    Configuration for the model to use instead of an automatically loaded configuration. Configuration can be automatically loaded when:

    • The model is a model provided by the library (loaded with the model id string of a pretrained model).

    • The model was saved using [~PreTrainedModel.save_pretrained] and is reloaded by supplying the save directory.

    • The model is loaded by supplying a local directory as pretrained_model_name_or_path and a configuration JSON file named config.json is found in the directory.

  • state_dict (Dict[str, torch.Tensor], optional) –

    A state dictionary to use instead of a state dictionary loaded from saved weights file.

    This option can be used if you want to create a model from a pretrained configuration but load your own weights. In this case though, you should check if using [~PreTrainedModel.save_pretrained] and [~PreTrainedModel.from_pretrained] is not a simpler option.

  • cache_dir (str or os.PathLike, optional) – Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used.

  • from_tf (bool, optional, defaults to False) – Load the model weights from a TensorFlow checkpoint save file (see docstring of pretrained_model_name_or_path argument).

  • force_download (bool, optional, defaults to False) – Whether or not to force the (re-)download of the model weights and configuration files, overriding the cached versions if they exist.

  • resume_download – Deprecated and ignored. All downloads are now resumed by default when possible. Will be removed in v5 of Transformers.

  • proxies (Dict[str, str], optional) – A dictionary of proxy servers to use by protocol or endpoint, e.g., {‘http’: ‘foo.bar:3128’, ‘http://hostname’: ‘foo.bar:4012’}. The proxies are used on each request.

  • output_loading_info (bool, optional, defaults to False) – Whether ot not to also return a dictionary containing missing keys, unexpected keys and error messages.

  • local_files_only (bool, optional, defaults to False) – Whether or not to only look at local files (e.g., not try downloading the model).

  • revision (str, optional, defaults to “main”) – The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.

  • trust_remote_code (bool, optional, defaults to False) – Whether or not to allow for custom models defined on the Hub in their own modeling files. This option should only be set to True for repositories you trust and in which you have read the code, as it will execute code present on the Hub on your local machine.

  • code_revision (str, optional, defaults to “main”) – The specific revision to use for the code on the Hub, if the code leaves in a different repository than the rest of the model. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.

  • kwargs (additional keyword arguments, optional) –

    Can be used to update the configuration object (after it being loaded) and initiate the model (e.g., output_attentions=True). Behaves differently depending on whether a config is provided or automatically loaded:

    • If a configuration is provided with config, **kwargs will be directly passed to the underlying model’s __init__ method (we assume all relevant updates to the configuration have already been done)

    • If a configuration is not provided, kwargs will be first passed to the configuration class initialization function ([~PretrainedConfig.from_pretrained]). Each key of kwargs that corresponds to a configuration attribute will be used to override said attribute with the supplied kwargs value. Remaining keys that do not correspond to any configuration attribute will be passed to the underlying model’s __init__ function.

Examples:

```python >>> from transformers import AutoConfig, AutoAdapterModel

>>> # Download model and configuration from huggingface.co and cache.
>>> model = AutoAdapterModel.from_pretrained("google-bert/bert-base-cased")

>>> # Update configuration during loading
>>> model = AutoAdapterModel.from_pretrained("google-bert/bert-base-cased", output_attentions=True)
>>> model.config.output_attentions
True

>>> # Loading from a TF checkpoint file instead of a PyTorch model (slower)
>>> config = AutoConfig.from_pretrained("./tf_model/bert_tf_model_config.json")
>>> model = AutoAdapterModel.from_pretrained(
...     "./tf_model/bert_tf_checkpoint.ckpt.index", from_tf=True, config=config
... )
```