turi::general_ofstream Class Reference

#include <core/storage/fileio/general_fstream.hpp>

Public Member Functions
	general_ofstream (std::string filename)
	general_ofstream (std::string filename, bool gzip_compress)
bool	good () const
bool	bad () const
bool	fail () const
size_t	get_bytes_written () const
std::string	filename () const

Detailed Description

A generic output file stream interface that provides unified access to local filesystem, HDFS, S3, in memory files, and can automatically perform gzip decoding.

Usage:

general_ofstream fin("file");
// after which fout almost behaves like a regular std::ofstream object.

file can be:

• local filesystem
• S3 (in which case the filename must be of the form s3://... (see below)
• HDFS (filename must be of the form hdfs://...)
• In memory / disk paged (filename must be of the form cache://...)

Unlike standard std::ofstream, random seek is not allowed. In other words, only sequential write is permitted.

If the filename ends with ".gz", gzip compression is automatically performed.

S3 access keys are mediated by having the filename be of the form s3://[access_key_id]:[secret_key]:[endpoint/][bucket]/[object_name]

Endpoint URLs however, are set globally via the global variable S3_ENDPOINT.

Definition at line 130 of file general_fstream.hpp.

Constructor & Destructor Documentation

general_ofstream() [1/2]

turi::general_ofstream::general_ofstream

(

std::string

filename

)

Constructs a general ofstream object when opens the filename specified. The file may be on HDFS. If the filename has the ".gz" suffix, it will be gzip compressed.

Throw an std::io_base::failure exception if failing to contruct the stream.

general_ofstream() [2/2]

turi::general_ofstream::general_ofstream	(	std::string	filename,
		bool	gzip_compress
	)

Constructs a general ofstream object when opens the filename specified. The file may be on HDFS. This overloaded constructor allows you to explicitly specify whether the file is to to be gzip compressed, regardless of the filename.

Throw an std::io_base::failure exception if failing to contruct the stream.

Member Function Documentation

bad()

bool turi::general_ofstream::bad

(

)

const

Returns true if the stream is bad. See std::ios_base

fail()

bool turi::general_ofstream::fail

(

)

const

Returns true if the last stream operation has failed. See std::ios_base

filename()

std::string turi::general_ofstream::filename

(

)

const

Returns the local file name used by the stream.

get_bytes_written()

size_t turi::general_ofstream::get_bytes_written

(

)

const

Returns the number of bytes written to disk so far. Due to file compression and buffering this can be very different from how many bytes were wrtten to the stream.

good()

bool turi::general_ofstream::good

(

)

const

Returns true if the stream is good. See std::ios_base

---

The documentation for this class was generated from the following file:

• core/storage/fileio/general_fstream.hpp