data.datasets.multi_modal_img_text package

Subpackages

• data.datasets.multi_modal_img_text.zero_shot package
  • Submodules
  • data.datasets.multi_modal_img_text.zero_shot.base_zero_shot module
    • BaseZeroShotDataset
      • BaseZeroShotDataset.__init__()
      • BaseZeroShotDataset.add_arguments()
      • BaseZeroShotDataset.class_names()
      • BaseZeroShotDataset.generate_text_prompts()
  • data.datasets.multi_modal_img_text.zero_shot.imagenet module
    • ImageNetDatasetZeroShot
      • ImageNetDatasetZeroShot.__init__()
      • ImageNetDatasetZeroShot.class_names()
      • ImageNetDatasetZeroShot.generate_text_prompts()
  • data.datasets.multi_modal_img_text.zero_shot.imagenet_class_names module
  • data.datasets.multi_modal_img_text.zero_shot.templates module
    • generate_text_prompts_clip()
  • Module contents
    • arguments_zero_shot_dataset()
    • build_zero_shot_dataset()

Submodules

data.datasets.multi_modal_img_text.base_multi_modal_img_text module

class data.datasets.multi_modal_img_text.base_multi_modal_img_text.BaseMultiModalImgText(opts, *args, **kwargs)

Bases: BaseImageDataset

Base class for Image-Text multi-modal learning

Parameters:

opts – command-line arguments

__init__(opts, *args, **kwargs) → None

get_zero_shot_dataset(*args, **kwargs) → BaseZeroShotDataset | None

If zero-shot evaluation is enabled, zero-shot dataset is returned. Otherwise, None is returned

get_dataset(*args, **kwargs) → Any

Helper function to get the dataset. Child classes must override this function

share_dataset_arguments() → Dict[str, Any]

Returns the number of classes in detection dataset along with super-class arguments.

classmethod add_arguments(parser: ArgumentParser) → ArgumentParser

Add dataset-specific arguments to the parser.

get_zero_shot_pair(img_index: int) → Tuple[Image, str | List[str] | List[List[str]], int]

Get image-text pair for zero-shot dataset along with classification label.

Parameters:

img_index – Image index

Returns:

A tuple of PIL image, captions, and class label

get_dataset_pair(img_index: int) → Any

Get image-text pair from the dataset. Sub-classes must implement this method.

extra_repr() → str

Extra information to be represented in __repr__. Each line in the output string should be prefixed with \t.

data.datasets.multi_modal_img_text.base_multi_modal_img_text.multi_modal_img_text_collate_fn(batch: List[Mapping[str, Tensor | Mapping[str, Tensor]]], opts: Namespace) → Mapping[str, Tensor | Mapping[str, Tensor]]

Combines a list of dictionaries into a single dictionary by concatenating matching fields.

data.datasets.multi_modal_img_text.flickr module

class data.datasets.multi_modal_img_text.flickr.FlickrDataset(opts, *args, **kwargs)

Bases: BaseMultiModalImgText

Dataset loader for Flickr-30k and Flickr-8k datasets.

For more info see:

http://hockenmaier.cs.illinois.edu/8k-pictures.html https://shannon.cs.illinois.edu/DenotationGraph/

Splits: train, val, and test

Also known in literature as Karpathy splits https://cs.stanford.edu/people/karpathy/deepimagesent/

Tracking license info:

Captions have CC BY 3.0 license (see links above). Splits are under BSD License (see Github of NeuralTalk by Karpathy et. al.). Images are from Flickr. We do not own them and are only used for research purposes.

Parameters:

• opts – command-line arguments

• is_training (Optional[bool]) – A flag used to indicate training or validation mode. Default: True

• is_evaluation (Optional[bool]) – A flag used to indicate evaluation (or inference) mode. Default: False

get_dataset(*args, **kwargs) → None

The data under self.root is expected to consist of:

dataset.json # Karpathy splits + captions images/ # Raw images

The metdatadata cap be downloaded from:

https://cs.stanford.edu/people/karpathy/deepimagesent/flickr30k.zip

Images can be obtained from:

Flickr-8k: http://hockenmaier.cs.illinois.edu/8k-pictures.html Flickr-30k: https://shannon.cs.illinois.edu/DenotationGraph/

data.datasets.multi_modal_img_text.img_text_tar_dataset module

data.datasets.multi_modal_img_text.img_text_tar_dataset.extract_content(tar_file: TarFile, file_name: str) → AnyStr

Extract the context of a particular file inside a tar file and returns it.

data.datasets.multi_modal_img_text.img_text_tar_dataset.decode_image(byte_data) → Image

Reads the byte image data and returns the PIL image.

data.datasets.multi_modal_img_text.img_text_tar_dataset.decode_text(byte_data) → str

Reads the byte text data and returns the decoded string.

data.datasets.multi_modal_img_text.img_text_tar_dataset.async_download_file_from_s3(opts: Namespace, tar_file_name: str, cache_loc: str, *args, **kwargs) → None

Helper function to download the files asynchronously from S3.

Parameters:

• opts – command-line arguments

• tar_file_name – Name of the tar file

• cache_loc – Caching location on the local machine

class data.datasets.multi_modal_img_text.img_text_tar_dataset.ImgTextTarDataset(opts, *args, **kwargs)

Bases: BaseMultiModalImgText

ImgTextTarDataset class for datasets that store Image-Text pairs as tar files, each tar file with multiple pairs.

The dataset should be stored in following format where img_text_tar_dataset is the location of directory that has all tar files.

img_text_tar_dataset |— 00000000_0_1000.tar.gz |——– 00000000_0_image |——– 00000000_0_text |——– 00000000_1_image |——– 00000000_1_text |——– …

|— 00000000_1000_2000.tar.gz |——– 00000000_1000_image |——– 00000000_1000_text |——– 00000000_1001_image |——– 00000000_1001_text |——– …

Parameters:

opts – An argparse.Namespace instance.

__init__(opts, *args, **kwargs) → None

get_dataset(*args, **kwargs) → Dict[str, str]

Reads the metadata file and returns a mapping of indices of files stored in a tar file and its name

classmethod add_arguments(parser: ArgumentParser) → ArgumentParser

Add dataset-specific arguments to the parser.

get_dataset_pair(img_index: int) → Tuple[Image, str, int]

For a given image index, read the image file, corresponding caption, and class label. If class label is not present, -1 is returned.

Module contents

data.datasets.multi_modal_img_text.arguments_multi_modal_img_text(parser: ArgumentParser) → ArgumentParser