Adapter Model Interface
- class adapters.AdapterModelInterface(adapter_methods: List[str], model_embeddings: str, model_layers: str, layer_self_attn: str, layer_cross_attn: str, attn_o_proj: Optional[str], layer_intermediate_proj: str, layer_output_proj: str, attn_k_proj: Optional[str] = None, attn_q_proj: Optional[str] = None, attn_v_proj: Optional[str] = None, attn_qkv_proj: Optional[str] = None, layer_pre_self_attn: Optional[str] = None, layer_pre_cross_attn: Optional[str] = None, layer_pre_ffn: Optional[str] = None, layer_ln_1: Optional[str] = None, layer_ln_2: Optional[str] = None)
Defines the main interface for integrating adapter methods into a model class. This interface translates generic accessor names to model-specific attribute names.
- Parameters
adapter_methods (List[str]) – List of adapter types that are supported by the model. Subset of this list: [“bottleneck”, “lora”, “reft”, “prompt_tuning”, “invertible”]
model_embeddings (str) – Name of the model’s embedding layer.
model_layers (str) – Name of the model’s layer list.
layer_self_attn (str) – Name of the self-attention layer in a transformer layer.
layer_cross_attn (str) – Name of the cross-attention layer in a transformer layer.
attn_o_proj (str) – Name of the output projection layer in an attention layer.
layer_intermediate_proj (str) – Name of the intermediate projection layer in a transformer layer.
layer_output_proj (str) – Name of the output projection layer in a transformer layer.
specified (# Or this single attribute must be) –
attn_k_proj (Optional[str]) – Name of the key projection layer in an attention layer.
attn_q_proj (Optional[str]) – Name of the query projection layer in an attention layer.
attn_v_proj (Optional[str]) – Name of the value projection layer in an attention layer.
specified –
attn_qkv_proj (Optional[str]) – Name of the combined query-key-value projection layer (for models like GPT-2 or ModernBERT where QKV are in one tensor).
support (# Optional attributes for extended bottleneck adapter) –
layer_pre_self_attn (Optional[str]) – Hook point directly before the self attention layer. Used for extended bottleneck adapter support.
layer_pre_cross_attn (Optional[str]) – Hook point directly before the cross attention layer. Used for extended bottleneck adapter support.
layer_pre_ffn (Optional[str]) – Hook point directly before the feed forward layer. Used for extended bottleneck adapter support.
layer_ln_1 (Optional[str]) – Layer norm after the self-attention layer. Used for extended bottleneck adapter support.
layer_ln_2 (Optional[str]) – Layer norm after the feed forward layer. Used for extended bottleneck adapter support.
Note
You must specify either all three of the individual projection layers (attn_k_proj, attn_q_proj, attn_v_proj) OR the combined projection layer (attn_qkv_proj).
- class adapters.AdapterMethod
Enum of all supported adapter method types.
- bottleneck
Adapter methods using bottleneck layers.
- prefix_tuning
Adapters methods based on Prefix Tuning. Note that this is currently unsupported via AdapterModelInterface.
- lora
Adapter methods based on low-rank adaptation.
- prompt_tuning
Adapter methods based on Prompt Tuning.
- reft
Adapters methods based on Representation Fine-Tuning.
- invertible
Adapter methods using invertible modules.
- static get_from_config(config) List[str]
Get the adapter type from a given adapter config.
- Parameters
config – The adapter config.
- Returns
The adapter type.
- Return type
List[str]