Metrics#

class pfl.metrics.StringMetricName(description)#

A structured name for metrics.

Parameters:: description (str) – The metric name represented as a string.

class pfl.metrics.MetricName(description, population)#

A structured name for metrics which includes the population the metric was generated from.

Parameters:

description (str) – The metric name represented as a string.
population (Population) – The population the metric was generated from.

class pfl.metrics.TrainMetricName(description, population, after_training, local_partition=None)#

A structured name for metrics which includes the population the metric was generated from and training descriptions.

Parameters:

description (str) – The metric name represented as a string.
population (Population) – The population the metric was generated from.
after_training (bool) – True if the metric was generated after local training, False otherwise.
local_partition (Optional[str]) – (Optional) The name of the local dataset partition this metric is related to. This is mainly only relevant when you have a train and val set locally on devices.

class pfl.metrics.ComposableMetricName(metric_name)#

Base class of any decorate-able MetricName.

An instance of ComposableMetricName can be composed with another instances of ComposableMetricName without losing the original attribute.

The main purpose of this class and subclasses is to provide a way one can identify whether an attribute exist in a composed ComposableMetricName.

Parameters:: metric_name (StringMetricName) – an instance of MetricName.

class pfl.metrics.MetricNamePostfix(metric_name, postfix)#

Add a postfix to an existing metric name.

Parameters:

metric_name (StringMetricName) – An instance of MetricName.
postfix (str) – The postfix to append to the metric name string.

class pfl.metrics.SkipSerialization(metric_name)#: Metrics that shouldn’t be serialized when consuming metrics.

class pfl.metrics.MetricValue#

The value of a metric, values of which that come from different places can be combined (using +). Additionally, to allow for distributed computation, the value needs to be convertible to a fixed-dimensional vector of values. Addition in this space must perform the same operation as calling + on this object.

Technically, any subclass MetricValue with + must form a commutative monoid: + must be associative and commutative.

abstract property overall_value#: Return the overall value, e.g. an average or a total.

abstract to_vector()#

Get a vector representation of this metric value, with dtype=float32. Summing two vectors in this space must be equivalent to summing the two original objects.

Return type:: ndarray

abstract from_vector(vector)#

Create a new metric value of this class from a vector representation.

Note that this is a method on an object of this class, since it is possible that runtime attributes that do not change with addition are not serialized.

Return type:: MetricValue

pfl.metrics.user_average(metric_value)#

Take the overall value of the provided metric and re-weight by 1. Can be used as a postprocess function when specifying metrics to be able to create average of per-user metrics instead of the metric over all data of the cohort.

Parameters:: metric_value (MetricValue) – Metric value to evaluate and re-weight by 1.
Return type:: MetricValue
Returns:: A new metric value with weight 1.

pfl.metrics.get_overall_value(metric_value)#

Get the value of Union[MetricValue, float, int] in a type-safe way.

Return type:: float

class pfl.metrics.Weighted(weighted_value, weight)#

Represent a value with a weight. This allows a normalised value to be computed. These can be added; this adds the weighted values and the weights.

E.g. the value could be the sum of single values, and the weight could be the number of values; then value is the weighted value.

Parameters:

weighted_value – Value multiplied by the weight. E.g. if weighted value is 8.0 and weight is 2.0, the overall value is 4.0.
weight – Weight for the value. If this is 0, then the weighted value must be 0 too.

property overall_value#: Return the overall value, e.g. an average or a total.

from_vector(vector)#

Create a new metric value of this class from a vector representation.

Note that this is a method on an object of this class, since it is possible that runtime attributes that do not change with addition are not serialized.

Return type:: Weighted

to_vector()#

Get a vector representation of this metric value, with dtype=float32. Summing two vectors in this space must be equivalent to summing the two original objects.

Return type:: ndarray

class pfl.metrics.Summed(value)#

A metric value which simply wraps 1 number. Addition is equal to summing. Unlike Weighted which accumulates a weighted average across devices, Summed is useful when the metric to be accumulated across devices should simply be the sum.

Example:

>>> assert Summed(1) + Summed(2) == Summed(3)

property overall_value#: Return the overall value, e.g. an average or a total.

from_vector(vector)#

Create a new metric value of this class from a vector representation.

Note that this is a method on an object of this class, since it is possible that runtime attributes that do not change with addition are not serialized.

Return type:: Summed

to_vector()#

Get a vector representation of this metric value, with dtype=float32. Summing two vectors in this space must be equivalent to summing the two original objects.

Return type:: ndarray

class pfl.metrics.Histogram(bin_counts, bins)#

Collect values as a histogram. Use the from_values_range or from_values_bins to calculate the histogram from a list of values.

Parameters:

bin_counts (List[Number]) – A list of counts for each bin in the histogram.
bins (List[Number]) – A list of N+1 boundaries representing the N bins.

classmethod from_values_range(values, num_bins, min_bound, max_bound)#

Create histogram from a list of values and equally-spaced bins.

Parameters:

values (List[float]) – A list of values to create the histogram with.
num_bins (int) – Number of bins, equally-spaced between min_bound and max_bound.
min_bound (float) – The leftmost bin edge. Values lower than this are ignored.
max_bound (float) – The rightmost bin edge. Values higher than this are ignored.

classmethod from_values_bins(values, bins)#

Create histogram from a list of values and custom bin edges, which don’t need to be equally-spaced.

Parameters:

values (List[float]) – A list of values to create the histogram with.
bins (List[float]) – A list of N+1 boundaries representing the N bins. If a value is outside the bin range, the value is ignored.

property overall_value#: Return the overall value, e.g. an average or a total.

from_vector(vector)#

Create a new metric value of this class from a vector representation.

Note that this is a method on an object of this class, since it is possible that runtime attributes that do not change with addition are not serialized.

Return type:: Histogram

to_vector()#

Get a vector representation of this metric value, with dtype=float32. Summing two vectors in this space must be equivalent to summing the two original objects.

Return type:: ndarray

class pfl.metrics.MetricsZero#: Represent a set of metrics which has zero for all metric names. However, what the metric names are is not specified. This class only supports +. You probably want to use the singleton object Zero.

class pfl.metrics.Metrics(values=None)#

Represent a set of metrics.

It is possible to get a metric with []. It is also possible to set it, but only if it does not exist yet.

It is possible to iterate over the object, which yields tuples (metric_name, value).

It is possible to add two Metrics objects if they have the exact same set of metric names. It is possible to take the union (with |) of two Metrics objects if they have a completely disjunct set of metric names.

It is possible to convert the metrics into a list of variable-length Numpy arrays with dtype float32 and back. Adding these arrays is equivalent to adding the metrics.

Metrics is built such that you can always get the value by specifying the key in string format, even if the actual key is a StringMetricName.

Parameters:: metrics – (optional) An iterable of tuples (metric_name, value) to initialize the object with. This is equivalent to calling self[metric_name] = value for each entry. Use from_dict to initialize the class from a dict {metric_name: value}.

to_simple_dict(force_serialize_all_metrics=False, to_lowercase=False)#

Returns a python dictionary of name-value pairs of metrics and their values, e.g. {‘Loss’: 0.12, ‘Accuracy’: 0.45}. All metric names are capitalized.

Parameters:

force_serialize_all_metrics (bool) – Default to False. Indicate whether or not to include metrics that are marked to be ignored on serialization.
to_lowercase (bool) – Default to False. Indicate lowercasing entire metrics key name. This uniformity is necessary for downstream analytics pipelines such as Splunk and Delphi. On the contrary, Proper-casing might be better for readability in stdout in iPython notebooks and logs.

Return type:

Dict[str, Union[float, int]]

to_vectors()#

Get a list of vector representations of the metric values in this object.

Return type:: List[ndarray]
Returns:: A list of np.ndarray, which are vector representations of each metric value. The order of the vectors is the same for all Metrics objects with the same set of metric names. Performing element-wise addition of two of these vectors yields the same equivalent result of adding two Metrics objects (without the correctness checks).