turi::lambda::lambda_master Class Reference

#include <core/system/lambda/lambda_master.hpp>

Public Member Functions
	lambda_master (size_t nworkers)
size_t	make_lambda (const std::string &lambda_str)
void	release_lambda (size_t lambda_hash) noexcept
void	bulk_eval (size_t lambda_hash, const std::vector< flexible_type > &args, std::vector< flexible_type > &out, bool skip_undefined, uint64_t seed)
void	bulk_eval (size_t lambda_hash, const sframe_rows &args, std::vector< flexible_type > &out, bool skip_undefined, uint64_t seed)
void	bulk_eval (size_t lambda_hash, const std::vector< std::string > &keys, const std::vector< std::vector< flexible_type >> &args, std::vector< flexible_type > &out, bool skip_undefined, uint64_t seed)
void	bulk_eval (size_t lambda_hash, const std::vector< std::string > &keys, const sframe_rows &args, std::vector< flexible_type > &out, bool skip_undefined, uint64_t seed)

Detailed Description

The lambda_master provides functions to evaluate a lambda function on different input types (single value, list, dictionary) in parallel.

set_lambda_worker_binary or must be called first to inform the location of the lambda worker binaries.

Internally, it manages a worker pool of lambda_workers.

Each evaluation call is allocated to a worker, and block until the evaluation returns or throws an exception.

The evaluation functions can be called in parallel. When this happens, the master evenly allocates the jobs to workers who has the shortest job queue.

std::vector<flexible_type> args{0,1,2,3,4};

// creates a master with 10 workers;
lambda_master master(10);

// Evaluate a single argument.
// plus_one_lambda is equivalent to lambda x: x + 1
auto lambda_hash = master.make_lambda(plus_one_lambda);

std::vector<flexible_type> out;
master.bulk_eval(lambda_hash, {0}, out);
ASSERT_EQ(out[0], 1);
master.bulk_eval(lambda_hash, {1}, out);
ASSERT_EQ(out[0], 2);


// Evaluate in parallel, still using plus_one_lambda.
std::vector< std::vector<flexible_type> > out_vec;
parallel_for(0, args.size(), [&](size_t i) {
   master.bulk_eval(lambda_hash, {args[i]}, out_vec[i]);
});

for (auto val : args) {
  ASSERT_EQ(out_vec[i][0], (val + 1));
}
master.release_lambda(plus_one_lambda);

Definition at line 75 of file lambda_master.hpp.

Constructor & Destructor Documentation

lambda_master()

turi::lambda::lambda_master::lambda_master

(

size_t

nworkers

)

Constructor. Do not use directly. Instead, use get_instance()

Member Function Documentation

bulk_eval() [1/4]

void turi::lambda::lambda_master::bulk_eval	(	size_t	lambda_hash,
		const std::vector< flexible_type > &	args,
		std::vector< flexible_type > &	out,
		bool	skip_undefined,
		uint64_t	seed
	)

Evaluate lambda on batch of inputs.

bulk_eval() [2/4]

void turi::lambda::lambda_master::bulk_eval	(	size_t	lambda_hash,
		const sframe_rows &	args,
		std::vector< flexible_type > &	out,
		bool	skip_undefined,
		uint64_t	seed
	)

This is an overloaded member function, provided for convenience. It differs from the above function only in what argument(s) it accepts.

bulk_eval() [3/4]

void turi::lambda::lambda_master::bulk_eval	(	size_t	lambda_hash,
		const std::vector< std::string > &	keys,
		const std::vector< std::vector< flexible_type >> &	args,
		std::vector< flexible_type > &	out,
		bool	skip_undefined,
		uint64_t	seed
	)

This is an overloaded member function, provided for convenience. It differs from the above function only in what argument(s) it accepts. Lambda takes dictionary argument.

bulk_eval() [4/4]

void turi::lambda::lambda_master::bulk_eval	(	size_t	lambda_hash,
		const std::vector< std::string > &	keys,
		const sframe_rows &	args,
		std::vector< flexible_type > &	out,
		bool	skip_undefined,
		uint64_t	seed
	)

This is an overloaded member function, provided for convenience. It differs from the above function only in what argument(s) it accepts.

make_lambda()

size_t turi::lambda::lambda_master::make_lambda

(

const std::string &

lambda_str

)

Register the lambda_str for all workers, and returns the id for the lambda. Throws the exception

release_lambda()

void turi::lambda::lambda_master::release_lambda ( size_t  lambda_hash )

noexcept

Unregister the lambda_str.

---

The documentation for this class was generated from the following file:

• core/system/lambda/lambda_master.hpp