Hyperparameters

typing.ParameterType

class pfl.hyperparam.base.HyperParam

Base class for defining adaptive hyperparameters. An adaptive hyperparameter can be used as a substitute for a static parameter where permitted (mostly in the configs and algorithms, see respective type signatures).

Make the subclass also a postprocessor or callback to access hooks where adapting the hyperparameters can take place.

Example:

This is an example of an adaptive hyperparameter (cohort size), which increases by a factor of 2 each central iteration.

class MyCohortSize(HyperParam, TrainingProcessCallback):
    def __init__(self, initial):
        self._value = initial

    def after_central_iteration(self, aggregate_metrics, model,
                                central_iteration):
        self._value *= 2

    def value(self):
        return self._value

abstractmethod value()

The current state (inner value) of the hyperparameter.

Return type:

TypeVar(ParameterType)

pfl.hyperparam.base.get_param_value(parameter)

If input is a HyperParam, extract its current value, otherwise return identity.

Example:

>>> get_param_value(1.0)
1.0
>>> get_param_value(MyCustomParam(initial_value=2.0))
2.0

class pfl.hyperparam.base.HyperParams

Base class for dataclasses that store parameters for model/training.

static_clone(**kwargs)

Returns a static clone of hyperparameters where each parameter has its current value (including adaptive parameters). This is used to access parameters in the algorithms (e.g. FederatedNNAlgorithm).

Return type:

TypeVar(HyperParamsType, bound= HyperParams)

get(key)

Get the current static value of a hyperparameter (which is a property of the dataclass). I.e. in the case of the hyperparam being a HyperParam, return the inner value state.

class pfl.hyperparam.base.AlgorithmHyperParams

Base class for additional parameters to pass to algorithms. By default, this base class has no parameters, but subclasses purposed for certain federated algorithms will have additional parameters.

class pfl.hyperparam.base.ModelHyperParams

A base configuration for training models. By default, this base class has no parameters, but subclasses purposed for certain models will have additional parameters.

class pfl.hyperparam.base.NNEvalHyperParams(local_batch_size)

Config to use for evaluating any neural network with an algorithm that involves SGD.

Parameters:

local_batch_size (Union[HyperParam[int], int, None]) – The batch size for evaluating locally on device. If None, defaults to no batching (full-batch evaluation).

class pfl.hyperparam.base.NNTrainHyperParams(local_batch_size, local_num_epochs, local_learning_rate, local_max_grad_norm=None, local_num_steps=None, grad_accumulation_steps=1)

Config to use for training any neural network with an algorithm that involves SGD.

Parameters:

• local_num_epochs (Union[HyperParam[int], int, None]) – The number of epochs of training applied on the device. If this is set, local_num_steps must be None.

• local_learning_rate (Union[HyperParam[float], float]) – The learning rate applied on the device.

• local_batch_size (Union[HyperParam[int], int, None]) – The batch size for training locally on device. If None, defaults to the entire dataset, which means one local iteration is one epoch.

• local_max_grad_norm (Union[HyperParam[float], float, None]) – Maximum L2 norm for gradient update in each local optimization step. Local gradients on device will be clipped if their L2 norm is larger than local_max_grad_norm. If None, no clipping is performed.

• local_num_steps (Union[HyperParam[int], int, None]) – Number of gradient steps during local training. If this is set, local_num_epochs must be None. Stops before local_num_steps if iterated through dataset. This can be useful if the user dataset is very large and training less than an epoch is appropriate.

• grad_accumulation_steps (int) – Number of steps to accumulate gradients before apply a local optimizer update. The effective batch size is local_batch_size multiplied by this number. This is useful to simulate a larger local batch size when memory is limited. Currently only supported for PyTorch.