cvnets.text_encoders package

Submodules

cvnets.text_encoders.base_text_encoder module

class cvnets.text_encoders.base_text_encoder.BaseTextEncoder(opts, projection_dim: int, *args, **kwargs)

Bases: Module

Base class for text encoder

__init__(opts, projection_dim: int, *args, **kwargs) → None

Initializes internal Module state, shared by both nn.Module and ScriptModule.

classmethod add_arguments(parser: ArgumentParser) → ArgumentParser

Add model specific arguments

reset_parameters()

Initialize model weights

get_trainable_parameters(weight_decay: float | None = 0.0, no_decay_bn_filter_bias: bool | None = False, *args, **kwargs)

freeze_norm_layers() → None

forward(text_tokens: Tensor, key_padding_mask: Tensor | None = None, attn_mask: Tensor | None = None, *args, **kwargs) → Any

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

dummy_input_and_label(batch_size: int) → Dict

Create dummy input and labels for CI/CD purposes. Child classes must override it if functionality is different.

cvnets.text_encoders.transformer module

class cvnets.text_encoders.transformer.TextTransformer(opts, projection_dim: int, *args, **kwargs)

Bases: BaseTextEncoder

__init__(opts, projection_dim: int, *args, **kwargs) → None

Initializes internal Module state, shared by both nn.Module and ScriptModule.

reset_parameters_clip_style()

This function resets the weights of Transformer model as done in the CLIP paper

classmethod add_arguments(parser: ArgumentParser) → ArgumentParser

Add model specific arguments

forward_embedding(text_tokens: Tensor)

build_attention_mask(context_length: int, batch_size: int)

encode_text(text_tokens: Tensor, key_padding_mask: Tensor | None = None, return_all_tokens: bool = False, *args, **kwargs) → Tensor

Returns token embeddings.

Parameters:

• text_tokens – a tensor of token indices. ([Batch, Seq_len])

• key_padding_mask – a tensor of boolean values as the padding mask.

• return_all_tokens – a boolean flag to return all tokens, defaults to False to return only EOT token embedding.

Returns:

a tensor of [Batch, Seq_len, hidden_dim] if return_all_tokens is True, otherwise a tensor of [Batch, hidden_dim].

forward_zero_shot(text_tokens: Tensor, key_padding_mask: Tensor | None = None, *args, **kwargs) → Tensor

forward(text_tokens: Tensor, key_padding_mask: Tensor | None = None, *args, **kwargs) → Tensor

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

Module contents

cvnets.text_encoders.arguments_text_encoder(parser: ArgumentParser) → ArgumentParser

Register arguments of all text encoders.

cvnets.text_encoders.build_text_encoder(opts, projection_dim: int, *args, **kwargs) → BaseTextEncoder

Helper function to build the text encoder from command-line arguments.

Parameters:

• opts – Command-line arguments

• projection_dim – The dimensionality of the projection head after text encoder.

Returns:

Text encoder module.