data.datasets package
Subpackages
- data.datasets.audio_classification package
- data.datasets.classification package
- Submodules
- data.datasets.classification.base_image_classification_dataset module
- data.datasets.classification.base_imagenet_shift_dataset module
- data.datasets.classification.imagenet module
- data.datasets.classification.imagenet_a module
- data.datasets.classification.imagenet_r module
- data.datasets.classification.imagenet_sketch module
- data.datasets.classification.imagenet_synsets module
- data.datasets.classification.imagenet_v2 module
- data.datasets.classification.places365 module
- Module contents
- data.datasets.detection package
- data.datasets.multi_modal_img_text package
- Subpackages
- Submodules
- data.datasets.multi_modal_img_text.base_multi_modal_img_text module
BaseMultiModalImgText
BaseMultiModalImgText.__init__()
BaseMultiModalImgText.get_zero_shot_dataset()
BaseMultiModalImgText.get_dataset()
BaseMultiModalImgText.share_dataset_arguments()
BaseMultiModalImgText.add_arguments()
BaseMultiModalImgText.get_zero_shot_pair()
BaseMultiModalImgText.get_dataset_pair()
BaseMultiModalImgText.extra_repr()
multi_modal_img_text_collate_fn()
- data.datasets.multi_modal_img_text.flickr module
- data.datasets.multi_modal_img_text.img_text_tar_dataset module
- Module contents
- data.datasets.segmentation package
- Submodules
- data.datasets.segmentation.ade20k module
- data.datasets.segmentation.base_segmentation module
BaseImageSegmentationDataset
BaseImageSegmentationDataset.__init__()
BaseImageSegmentationDataset.add_arguments()
BaseImageSegmentationDataset.check_dataset()
BaseImageSegmentationDataset.adjust_mask_value()
BaseImageSegmentationDataset.color_palette()
BaseImageSegmentationDataset.class_names()
BaseImageSegmentationDataset.read_mask_pil()
BaseImageSegmentationDataset.convert_mask_to_tensor()
- data.datasets.segmentation.coco_segmentation module
- data.datasets.segmentation.pascal_voc module
- Module contents
- data.datasets.utils package
Submodules
data.datasets.dataset_base module
- class data.datasets.dataset_base.BaseDataset(opts: Namespace, is_training: bool = True, is_evaluation: bool = False, *args, **kwargs)[source]
Bases:
Dataset
,ABC
Base class for creating datasets. Sub-classes must implement __getitem__, _training_transforms, and _validation_transforms functions.
- Parameters:
opts – Command-line arguments
is_training – Training mode or not. Defaults to True.
is_evaluation – Evaluation mode or not. Defaults to False.
- …note::
is_training is used to indicate whether the dataset is used for training or validation. On the other hand, is_evaluation mode is used to indicate the dataset is used for testing.
Theoretically, is_training=False and is_evaluation=True should be the same. However, for some datasets (especially segmentation), validation dataset transforms are different from test transforms because each image has different resolution, making it difficult to construct batches. Therefore, we treat these two modes different.
For datasets, where validation and testing transforms are the same, we set evaluation transforms the same as the validation transforms.
- __init__(opts: Namespace, is_training: bool = True, is_evaluation: bool = False, *args, **kwargs) None [source]
- classmethod add_arguments(parser: ArgumentParser) ArgumentParser [source]
Add dataset-specific arguments
- static load_from_server(opts: Namespace, is_training: bool, is_evaluation: bool) Namespace | None [source]
Helper function to load dataset from server.
- get_augmentation_transforms(*args, **kwargs) BaseTransformation [source]
Helper function to get data transforms depending on the mode (training, evaluation, or validation)
Function that can be used by sub-classes to share dataset-specific options. It returns a mapping. An example is {“model.classification.n_classes”, 1000}
By default, we return an empty dictionary
- extra_repr() str [source]
Extra information to be represented in __repr__. Each line in the output string should be prefixed with
\t
.
- get_item_metadata(item_idx: int) Dict [source]
Returns the metadata for given @item_idx. This method could be used by samplers for sampling dynamic batches based on the metadata of the items.
- Parameters:
item_idx (int) – The index of sample to provide metadata for. The indexing should be aligned with how
self.__getitem__(item_idx)
sequences the dataset items.
- Returns: A dict containing the metadata. Each sampler may require a specific
schema to be returned by this function.
- class data.datasets.dataset_base.BaseImageDataset(opts: Namespace, is_training: bool = True, is_evaluation: bool = False, *args, **kwargs)[source]
Bases:
BaseDataset
,ABC
Base Dataset class for Image datasets.
- class data.datasets.dataset_base.BaseVideoDataset(opts: Namespace, *args, **kwargs)[source]
Bases:
BaseDataset
,ABC
Base Dataset class for video datasets.
- Parameters:
opts – Command-line arguments
- property clips_per_video: int
- property n_frames_per_clip: int
- classmethod add_arguments(parser: ArgumentParser) ArgumentParser [source]
Add dataset-specific arguments
- get_item_metadata(item_idx: int) VideoMetadataDict [source]
Returns the metadata for given @item_idx. This method is used by VideoClipSampler for sampling dynamic clips based on the duration of the items.
Subclasses should override this method if they use VideoClipSampler.
- Parameters:
item_idx (int) – The index of video file to provide metadata. The indexing should be aligned with how
self.__getitem__(item_idx)
sequences the dataset items.
Returns: A dict containing the metadata. Please see @VideoMetadataDict documentation.
- class data.datasets.dataset_base.VideoMetadataDict[source]
Bases:
TypedDict
This class is an alias of Dict, in addition to optional standard key names and value types. The fields will be required or optional depending on the Sampler.
- video_fps: float
- total_video_frames: int
- video_duration: float
- audio_fps: float
Module contents
- data.datasets.build_dataset_from_registry(opts: Namespace, is_training: bool = True, is_evaluation: bool = False, *args, **kwargs) BaseDataset [source]
Helper function to build a dataset from dataset registry
- Parameters:
opts – Command-line arguments
is_training – Training mode or not. Defaults to True.
is_evaluation – Evaluation mode or not. Defaults to False.
- Returns:
An instance of BaseDataset
- …note:
is_training is used to indicate whether the dataset is used for training or validation On the other hand, is_evaluation mode is used to indicate the dataset is used for testing.
Theoretically, is_training=False and is_evaluation=True should be the same. However, for some datasets (especially segmentation), validation dataset transforms are different from test transforms because each image has different resolution, making it difficult to construct batches. Therefore, we treat these two modes different. For datasets, where validation and testing transforms are the same, we set evaluation transforms the same as the validation transforms (e.g., in ImageNet object classification).
- data.datasets.get_test_dataset(opts: Namespace, *args, **kwargs) BaseDataset [source]
Helper function to build a dataset for testing.
- Parameters:
opts – Command-line arguments
- Returns:
An instance of BaseDataset
- data.datasets.get_train_val_datasets(opts: Namespace, *args, **kwargs) Tuple[BaseDataset, BaseDataset | None] [source]
Helper function to build a dataset for training and validation.
- Parameters:
opts – Command-line arguments
- Returns:
Training and (optionally) validation datasets.