Generic SFrame Eager Algorithms

Functions
template<typename S , typename T , typename TransformFn , typename = typename std::enable_if<sframe_impl::is_sarray_like<S>::value>::type, typename = typename std::enable_if<sframe_impl::is_sarray_like<T>::value>::type>
void	turi::transform (S &&input, T &&output, TransformFn transformfn, std::set< size_t > constraint_segments=std::set< size_t >())
template<typename S , typename T , typename FilterFn , typename = typename std::enable_if<sframe_impl::is_sarray_like<S>::value>::type, typename = typename std::enable_if<sframe_impl::is_sarray_like<T>::value>::type>
void	turi::copy_if (S &&input, T &&output, FilterFn filterfn, std::set< size_t > constraint_segments=std::set< size_t >(), size_t random_seed=size_t(-1))
template<typename S , typename T , typename FilterFn , typename TransformFn , typename = typename std::enable_if<sframe_impl::is_sarray_like<S>::value>::type, typename = typename std::enable_if<sframe_impl::is_sarray_like<T>::value>::type>
void	turi::copy_transform_if (S &&input, T &&output, FilterFn filterfn, TransformFn transformfn, std::set< size_t > constraint_segments=std::set< size_t >(), size_t random_seed=size_t(-1))
template<typename S , typename T , typename FilterFn , typename = typename std::enable_if<sframe_impl::is_sarray_like<S>::value>::type, typename = typename std::enable_if<sframe_impl::is_sarray_like<T>::value>::type>
void	turi::split (S &&input, T &&output1, T &&output2, FilterFn filterfn, size_t random_seed=std::time(NULL))
template<typename Iterator , typename SWriter , typename = typename std::enable_if<sframe_impl::is_sarray_like<SWriter>::value>::type>
void	turi::copy (Iterator begin, Iterator end, SWriter &&writer)
template<typename S , typename Iterator , typename = typename std::enable_if<sframe_impl::is_sarray_like<S>::value>::type>
void	turi::copy (S &&array, Iterator output, size_t limit=(size_t)(-1))
template<typename ResultType , typename S , typename FunctionType , typename = typename std::enable_if<sframe_impl::is_sarray_like<S>::value>::type>
std::vector< ResultType >	turi::reduce (S &&input, FunctionType f, ResultType init=ResultType())
template<typename S1 , typename S2 , typename T , typename TransformFn , typename = typename std::enable_if<sframe_impl::is_sarray_like<S1>::value>::type, typename = typename std::enable_if<sframe_impl::is_sarray_like<S2>::value>::type, typename = typename std::enable_if<sframe_impl::is_sarray_like<T>::value>::type>
void	turi::binary_transform (S1 &&input1, S2 &&input2, T &&output, TransformFn transformfn)
template<typename S , typename T , typename = typename std::enable_if<sframe_impl::is_sarray_like<S>::value>::type, typename = typename std::enable_if<sframe_impl::is_sarray_like<T>::value>::type>
void	turi::copy_range (S &&input, T &&output, size_t start, size_t step, size_t end)

Detailed Description

Function Documentation

binary_transform()

template<typename S1 , typename S2 , typename T , typename TransformFn , typename = typename std::enable_if<sframe_impl::is_sarray_like<S1>::value>::type, typename = typename std::enable_if<sframe_impl::is_sarray_like<S2>::value>::type, typename = typename std::enable_if<sframe_impl::is_sarray_like<T>::value>::type>

void turi::binary_transform	(	S1 &&	input1,
		S2 &&	input2,
		T &&	output,
		TransformFn	transformfn
	)

Writes input to output calling the transformfn on each input pair emitting the result to output.

This class accomplishes the abstract equivalent of

for each x in input1:
   read next y in input2
   write transformfn(x, y) to output

The input objects must be descendents of siterable, and the output object must be a descendent of swriter_base. sarray and swriter are two common instances of either.

Parameters

input1	The first input to read from. Must be a descendent of siterable
input2	The second input to read from. Must be a descendent of siterable
output	The output writer to write to. Must be a descendent of swriter_base
transformfn	The transform operation to perform on the inputs to generate the output

Definition at line 543 of file algorithm.hpp.

copy() [1/2]

template<typename Iterator , typename SWriter , typename = typename std::enable_if<sframe_impl::is_sarray_like<SWriter>::value>::type>

void turi::copy	(	Iterator	begin,
		Iterator	end,
		SWriter &&	writer
	)

Writes to an SWriter from a standard input iterator sequence.

This class accomplishes the abstract equivalent of

for each x in range(begin, end) (regular iterator):
    write x to output (swriter)

The input must be a pair of iterators, and the output object must be a descendent of swriter_base.

Note

The precise arrangement of the data in the segment will depend on the iterator type. If the iterator range is at minimal a forward_iterator, The resultant data is blocked across the segments of the writer. i.e. if the writer has 4 segments, the first 1/4 of the data will go to segment 1, the next 1/4 will go to segment 2, etc. Otherwise, if the iterator range is an input iterator, it will be striped across the segments.

Parameters

begin	The start of the forward iterator sequence to write to the writer.
end	The end of the forward iterator sequence to write to the writer.
output	The output writer to write to. Must be a descendent of swriter_base

Examples:

/build/src/core/storage/sframe_interface/unity_sframe.hpp.

Definition at line 416 of file algorithm.hpp.

copy() [2/2]

template<typename S , typename Iterator , typename = typename std::enable_if<sframe_impl::is_sarray_like<S>::value>::type>

void turi::copy	(	S &&	array,
		Iterator	output,
		size_t	limit = (size_t)(-1)
	)

Copies the contents of an SArray to regular output iterator.

This class accomplishes the abstract equivalent of:

for each x in sarray
    write x to output

The input object must be a descendent of swriter_base.

Parameters

array	The SArray to read from
output	The output iterator to write to.

Definition at line 441 of file algorithm.hpp.

copy_if()

template<typename S , typename T , typename FilterFn , typename = typename std::enable_if<sframe_impl::is_sarray_like<S>::value>::type, typename = typename std::enable_if<sframe_impl::is_sarray_like<T>::value>::type>

void turi::copy_if	(	S &&	input,
		T &&	output,
		FilterFn	filterfn,
		std::set< size_t >	constraint_segments = std::set<size_t>(),
		size_t	random_seed = size_t(-1)
	)

Filters input to output calling the filterfn on each input and emitting the input to output only if the filter function evaluates to true.

This class accomplishes the abstract equivalent of

for each x in input:
   if (filterfn(x)) write x to output

The input object must be a descendent of siterable, and the output object must be a descendent of swriter_base. sarray and swriter are two common instances of either.

The output object should be of the same number of segments as the input object. If they are of different number of segments, this function will attempt to change the number of segments of the output object. Changing the number of segments is generally a successful operation unless writes have already occured on the output. If the number of segments do not match, and if the number of output segments cannot be set, this function will throw a string exception and fail.

Parameters

input	The input to read from. Must be a descendent of siterable
output	The output writer to write to. Must be a descendent of swriter_base
filterfn	The filter operation to perform on the input. If the filterfn evaluates to true, the input is copied to the output.
constraint_segments	The set of segments to operate on. If empty (default) will operate on all segments. Only valid segment numbers will be operated on.

Definition at line 137 of file algorithm.hpp.

copy_range()

template<typename S , typename T , typename = typename std::enable_if<sframe_impl::is_sarray_like<S>::value>::type, typename = typename std::enable_if<sframe_impl::is_sarray_like<T>::value>::type>

void turi::copy_range	(	S &&	input,
		T &&	output,
		size_t	start,
		size_t	step,
		size_t	end
	)

Copies a range of elements to the output.

This class accomplishes the abstract equivalent of

for i = begin; i < end; i += step
   write row i in input to to output

The input object must be a descendent of siterable, and the output object must be a descendent of swriter_base. sarray and swriter are two common instances of either.

Parameters

input	The input to read from. Must be a descendent of siterable
output	The output writer to write to. Must be a descendent of swriter_base
start	The start row to begin reading
step	The row step
end	One past the last row to read

Examples:

/build/src/core/storage/sframe_interface/unity_sframe.hpp.

Definition at line 599 of file algorithm.hpp.

copy_transform_if()

template<typename S , typename T , typename FilterFn , typename TransformFn , typename = typename std::enable_if<sframe_impl::is_sarray_like<S>::value>::type, typename = typename std::enable_if<sframe_impl::is_sarray_like<T>::value>::type>

void turi::copy_transform_if	(	S &&	input,
		T &&	output,
		FilterFn	filterfn,
		TransformFn	transformfn,
		std::set< size_t >	constraint_segments = std::set<size_t>(),
		size_t	random_seed = size_t(-1)
	)

Filters input to output calling the filterfn on each input and emitting the transformed input to output only if the filter function evaluates to true.

This class accomplishes the abstract equivalent of

for each x in input:
   if (filterfn(x)) write transform(x) to output

The input object must be a descendent of siterable, and the output object must be a descendent of swriter_base. sarray and swriter are two common instances of either.

The output object should be of the same number of segments as the input object. If they are of different number of segments, this function will attempt to change the number of segments of the output object. Changing the number of segments is generally a successful operation unless writes have already occured on the output. If the number of segments do not match, and if the number of output segments cannot be set, this function will throw a string exception and fail.

The output object should be ready for write and the schema must match the output schema of the transform function.

Parameters

input	The input to read from. Must be a descendent of siterable
output	The output writer to write to. Must be a descendent of swriter_base
filterfn	The filter operation to perform on the input. If the filterfn evaluates to true, the input is copied to the output.
transformfn	The transform operation to perform on the input.
constraint_segments	The set of segments to operate on. If empty (default) will operate on all segments. Only valid segment numbers will be operated on.

Definition at line 211 of file algorithm.hpp.

reduce()

template<typename ResultType , typename S , typename FunctionType , typename = typename std::enable_if<sframe_impl::is_sarray_like<S>::value>::type>

std::vector<ResultType> turi::reduce	(	S &&	input,
		FunctionType	f,
		ResultType	init = ResultType()
	)

Performs a reduction on each segment of an Sarray returning the result of the reduction on each segment.

This class accomplishes the abstract equivalent of:

for each segment in sarray
    res = init
    for each x in segment:
       if (f(x, res) == false) break;
    returnval[segment] = res
return returnval

The input object must be a descendent of siterable

Parameters

array	The SArray to read from
function	The reduction function to use. This must be of the form bool f(const array_value_type&, reduction_type&).
init	The initial value to use in the reduction

Template Parameters

ResultType	The result type of each reduction.
S	The type of the input array. Must be a descendent of siterable
FunctionType	The type of the function

Note

This function assumes that writes to std::vector<T> can be made in parallel. i.e. ResultType of bool should not be used.

Definition at line 491 of file algorithm.hpp.

split()

template<typename S , typename T , typename FilterFn , typename = typename std::enable_if<sframe_impl::is_sarray_like<S>::value>::type, typename = typename std::enable_if<sframe_impl::is_sarray_like<T>::value>::type>

void turi::split	(	S &&	input,
		T &&	output1,
		T &&	output2,
		FilterFn	filterfn,
		size_t	random_seed = std::time(NULL)
	)

Split input to output1 and output2 calling the filterfn on each input and emitting the input to output1 if the filter function evaluates to true, output2 otherwise.

This class accomplishes the abstract equivalent of

for each x in input:
   if (filterfn(x)) write x to output1
   else write x to output2

The input object must be a descendent of siterable, and the output object must be a descendent of swriter_base. sarray and swriter are two common instances of either.

The output object should be of the same number of segments as the input object. If they are of different number of segments, this function will attempt to change the number of segments of the output object. Changing the number of segments is generally a successful operation unless writes have already occured on the output. If the number of segments do not match, and if the number of output segments cannot be set, this function will throw a string exception and fail.

Parameters

input	The input to read from. Must be a descendent of siterable
output1	The output writer to write to. Must be a descendent of swriter_base
output2	The output writer to write to. Must be a descendent of swriter_base
filterfn	The filter operation to perform on the input. If the filterfn evaluates to true, the input is copied to the output1, else output2.

Examples:

/build/src/core/storage/sframe_interface/unity_sframe.hpp.

Definition at line 293 of file algorithm.hpp.

transform()

template<typename S , typename T , typename TransformFn , typename = typename std::enable_if<sframe_impl::is_sarray_like<S>::value>::type, typename = typename std::enable_if<sframe_impl::is_sarray_like<T>::value>::type>

void turi::transform	(	S &&	input,
		T &&	output,
		TransformFn	transformfn,
		std::set< size_t >	constraint_segments = std::set<size_t>()
	)

Writes input to output calling the transformfn on each input emitting the result to output.

This class accomplishes the abstract equivalent of

for each x in input:
   write transformfn(x) to output

The input object must be a descendent of siterable, and the output object must be a descendent of swriter_base. sarray and swriter are two common instances of either.

The output object should be of the same number of segments as the input object. If they are of different number of segments, this function will attempt to change the number of segments of the output object. Changing the number of segments is generally a successful operation unless writes have already occured on the output. If the number of segments do not match, and if the number of output segments cannot be set, this function will throw a string exception and fail.

Parameters

input	The input to read from. Must be a descendent of siterable
output	The output writer to write to. Must be a descendent of swriter_base
transformfn	The transform operation to perform on the input to generate the output
constraint_segments	The set of segments to operate on. If empty (default) will operate on all segments. Only valid segment numbers will be operated on.

Examples:

/build/src/core/storage/sframe_interface/unity_sframe.hpp.

Definition at line 64 of file algorithm.hpp.