turi::buffered_writer< ValueType, OutIterator > Class Template Reference

#include <core/storage/fileio/buffered_writer.hpp>

Public Member Functions
void	write (const ValueType &val)
void	flush (bool is_locked=false)

Detailed Description

template<typename ValueType, typename OutIterator>
class turi::buffered_writer< ValueType, OutIterator >

Provide buffered write abstraction. The class manages buffered concurrent write to an output iterator.

Example:

Suppose there are M data sources randomly flow to N sinks. We can use buffered_writer to achieve efficient concurrent write.

std::vector<input_iterator> sources; // size M
std::vector<output_iterator>  sinks; // size N
std::vector<turi::mutex>  sink_mutex; // size N

parallel_for_each(s : sources) {
  std::vector<buffered_writer> writers;
  for (i = 1...N) {
   writers.push_back(buffered_writer(sinks[i], sink_mutex[i]));
  }
  while (s.has_next()) {
    size_t destination = random.randint(N);
    writers[destination].write(s.next());
  }
  for (i = 1...N) {
    writers[i].flush();
  }
}

Two parameters "soft_limit" and "hard_limit" are used to control the buffer size. When soft_limit is met, the writer will try to flush the buffer content to the sink. When hard_limit is met, the writer will force the flush.

Definition at line 47 of file buffered_writer.hpp.

Member Function Documentation

flush()

template<typename ValueType , typename OutIterator >

void turi::buffered_writer< ValueType, OutIterator >::flush ( bool  is_locked = false )

inline

Flush the buffer to the output sink. Clear the buffer when finished.

Definition at line 87 of file buffered_writer.hpp.

write()

template<typename ValueType , typename OutIterator >

void turi::buffered_writer< ValueType, OutIterator >::write ( const ValueType &  val )

inline

Write the value to the buffer. Try flush when buffer exceeds soft limit and force flush when buffer exceeds hard limit.

Definition at line 63 of file buffered_writer.hpp.

---

The documentation for this class was generated from the following file:

• core/storage/fileio/buffered_writer.hpp