Adapter Model Interface

class adapters.AdapterModelInterface(adapter_methods: List[str], model_embeddings: str, model_layers: str, layer_self_attn: str, layer_cross_attn: str, attn_o_proj: Optional[str], layer_intermediate_proj: str, layer_output_proj: str, attn_k_proj: Optional[str] = None, attn_q_proj: Optional[str] = None, attn_v_proj: Optional[str] = None, attn_qkv_proj: Optional[str] = None, layer_pre_self_attn: Optional[str] = None, layer_pre_cross_attn: Optional[str] = None, layer_pre_ffn: Optional[str] = None, layer_ln_1: Optional[str] = None, layer_ln_2: Optional[str] = None, base_model: Optional[str] = None)

Defines the main interface for integrating adapter methods into a model class. This interface translates generic accessor names to model-specific attribute names.

Parameters

  • adapter_methods (List[str]) – List of adapter types that are supported by the model. Subset of this list: [“bottleneck”, “lora”, “reft”, “prompt_tuning”, “invertible”]

  • model_embeddings (str) – Name of the model’s embedding layer.

  • model_layers (str) – Name of the model’s layer list.

  • layer_self_attn (str) – Name of the self-attention layer in a transformer layer.

  • layer_cross_attn (str) – Name of the cross-attention layer in a transformer layer.

  • attn_o_proj (str) – Name of the output projection layer in an attention layer.

  • layer_intermediate_proj (str) – Name of the intermediate projection layer in a transformer layer.

  • layer_output_proj (str) – Name of the output projection layer in a transformer layer.

  • specified (# Or this single attribute must be) –

  • attn_k_proj (Optional[str]) – Name of the key projection layer in an attention layer.

  • attn_q_proj (Optional[str]) – Name of the query projection layer in an attention layer.

  • attn_v_proj (Optional[str]) – Name of the value projection layer in an attention layer.

  • specified –

  • attn_qkv_proj (Optional[str]) – Name of the combined query-key-value projection layer (for models like GPT-2 or ModernBERT where QKV are in one tensor).

  • support (# Optional attributes for extended bottleneck adapter) –

  • layer_pre_self_attn (Optional[str]) – Hook point directly before the self attention layer. Used for extended bottleneck adapter support.

  • layer_pre_cross_attn (Optional[str]) – Hook point directly before the cross attention layer. Used for extended bottleneck adapter support.

  • layer_pre_ffn (Optional[str]) – Hook point directly before the feed forward layer. Used for extended bottleneck adapter support.

  • layer_ln_1 (Optional[str]) – Layer norm after the self-attention layer. Used for extended bottleneck adapter support.

  • layer_ln_2 (Optional[str]) – Layer norm after the feed forward layer. Used for extended bottleneck adapter support.

  • base_model (Optional[str]) – Name of the base transformers model holding the layer modules. By default, this uses the model class’ base_model_prefix attribute.

Note

You must specify either all three of the individual projection layers (attn_k_proj, attn_q_proj, attn_v_proj) OR the combined projection layer (attn_qkv_proj).

class adapters.AdapterMethod

Enum of all supported adapter method types.

bottleneck

Adapter methods using bottleneck layers.

prefix_tuning

Adapters methods based on Prefix Tuning. Note that this is currently unsupported via AdapterModelInterface.

lora

Adapter methods based on low-rank adaptation.

prompt_tuning

Adapter methods based on Prompt Tuning.

reft

Adapters methods based on Representation Fine-Tuning.

invertible

Adapter methods using invertible modules.

static get_from_config(config) → List[str]

Get the adapter type from a given adapter config.

Parameters

config – The adapter config.

Returns

The adapter type.

Return type

List[str]