AdapterHub Documentation

AdapterHub is a framework simplifying the integration, training and usage of adapter modules for Transformer-based language models. It integrates adapters for downstream tasks (Houlsby et al., 2019), adapters for cross-lingual transfer (Pfeiffer et al., 2020a) and AdapterFusion (Pfeiffer et al., 2020b).

The framework consists of two main components:

• adapter-transformers, an extension of Huggingface’s Transformers library that adds adapter components to transformer models

• The Hub, a central repository collecting pre-trained adapter modules

The adapter-transformers section documents the integration of adapters into the transformers library and how training adapters works.

The section on Adapter-Hub describes the fundamentals of the pre-trained adapter repository and how to contribute new adapters.

Currently, we support the PyTorch versions of all models listed in the Supported Models section.

adapter-transformers

• Installation
  • Using pip
  • From repository
• Quickstart
  • Introduction
  • Quick Tour: Using a pre-trained adapter for inference
  • Quick Tour: Adapter training
• Introduction to Adapters
  • Adapter types
  • Adapter architectures
• Adapter Activation and Composition
  • Stack
  • Fuse
  • Split
  • BatchSplit
  • Parallel
  • Nesting composition blocks
• Adapter Training
  • Train a Task Adapter
  • Train a Language Adapter
  • Train AdapterFusion
  • AdapterTrainer
• Prediction Heads
  • Models with flexible heads
  • Model with static heads (HuggingFace heads)
  • Automatic conversion
• Embeddings
  • Adding and Deleting Embeddings
  • Saving and Loading Embeddings
• Extending the Library
  • Integrating new Transformer models
  • Loading custom module weights
• Transitioning from v1 to v2
  • What’s new
  • What has changed
  • Internal changes

Adapter-Hub

• Loading Pre-Trained Adapters
  • Finding pre-trained adapters
  • Using pre-trained adapters in your code
  • How adapter resolving works
• Contributing to AdapterHub
  • Getting started
  • Add your pre-trained adapter
  • Add a new adapter architecture
  • Add a new task or subtask
• Integration with HuggingFace’s Model Hub
  • Downloading from the Hub
  • Uploading to the Hub

Adapter-Related Classes

• Adapter Configuration
  • Single Adapter
  • Adapter Fusion
• Model Adapters Config
• Adapter Modules
• AdapterLayerBaseMixin
• Model Mixins
  • InvertibleAdaptersMixin
  • ModelConfigAdaptersMixin
  • ModelAdaptersMixin
  • ModelWithHeadsAdaptersMixin
  • ModelWithFlexibleHeadsAdaptersMixin
  • PushAdapterToHubMixin
• Adapter Utilities
• Weights Loaders
  • WeightsLoader
  • AdapterLoader
  • AdapterFusionLoader
  • PredictionHeadLoader
  • WeightsLoaderHelper

Supported Models

• BART
• BERT
• DistilBERT
• Encoder Decoder Models
• OpenAI GPT2
• MBart
• RoBERTa
• T5
• XLM-RoBERTa

Citation

@inproceedings{pfeiffer2020AdapterHub,
   title={AdapterHub: A Framework for Adapting Transformers},
   author={Jonas Pfeiffer and
            Andreas R\"uckl\'{e} and
            Clifton Poth and
            Aishwarya Kamath and
            Ivan Vuli\'{c} and
            Sebastian Ruder and
            Kyunghyun Cho and
            Iryna Gurevych},
   booktitle={Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP 2020): Systems Demonstrations},
   year={2020},
   address = "Online",
   publisher = "Association for Computational Linguistics",
   url = "https://www.aclweb.org/anthology/2020.emnlp-demos.7",
   pages = "46--54",
}

Indices and tables

• Index

• Module Index