Filesystem Abstraction
Arrow provides an abstract filesystem interface that works with local files, HDFS, S3, and other storage systems.
FileSystem
Abstract filesystem API.
Properties
Returns the filesystem type name (e.g., “local”, “s3”, “hdfs”)
Returns the IOContext associated with this filesystem
other
const FileSystem&
required
Filesystem to compare with
Path Operations
Normalizes path for the given filesystem. Default implementation is a no-op, but subclasses may normalize irregular path forms uri_string
const std::string&
required
URI or absolute path
Makes a URI from which FileSystemFromUri produces an equivalent filesystem path
const std::string&
required
Path to query
paths
const std::vector<std::string>&
required
Paths to query
select
const FileSelector&
required
File selector
paths
const std::vector<std::string>&
required
Paths to query
select
const FileSelector&
required
File selector
Directory Operations
path
const std::string&
required
Directory path to create
Whether to create parent directories
path
const std::string&
required
Directory path to delete
path
const std::string&
required
Directory path
Whether to allow missing directory
path
const std::string&
required
Directory path
Whether to allow missing directory
Deletes the root directory’s contents recursively. Implementations may raise an error if too dangerous
File Operations
path
const std::string&
required
File path to delete
paths
const std::vector<std::string>&
required
File paths to delete
src
const std::string&
required
Source path
dest
const std::string&
required
Destination path
src
const std::string&
required
Source path
dest
const std::string&
required
Destination path
Stream Operations
OpenInputStream
Result<std::shared_ptr<io::InputStream>>
path
const std::string&
required
File path to open
OpenInputFile
Result<std::shared_ptr<io::RandomAccessFile>>
path
const std::string&
required
File path to open
OpenInputStreamAsync
Future<std::shared_ptr<io::InputStream>>
path
const std::string&
required
File path to open
OpenInputFileAsync
Future<std::shared_ptr<io::RandomAccessFile>>
path
const std::string&
required
File path to open
OpenOutputStream
Result<std::shared_ptr<io::OutputStream>>
path
const std::string&
required
File path to open
metadata
const std::shared_ptr<const KeyValueMetadata>&
required
Optional metadata to attach
OpenAppendStream
Result<std::shared_ptr<io::OutputStream>>
path
const std::string&
required
File path to open
metadata
const std::shared_ptr<const KeyValueMetadata>&
required
Optional metadata to attach
FileInfo
FileSystem entry information.
File type (default: Unknown)
Returns the file type (File, Directory, or Unknown)
Returns the full file path in the filesystem
Returns the file base name (component after the last directory separator)
Returns the directory base name (component before the file base name)
Returns the size in bytes, if available. Only regular files are guaranteed to have a size
Returns the time of last modification, if available
Returns the file extension (excluding the dot)
Returns true if type is File
Returns true if type is Directory
FileSelector
File selector for filesystem APIs.
struct FileSelector {
std::string base_dir;
bool allow_not_found;
bool recursive;
int32_t max_recursion;
}
Directory in which to select files. Error if path exists but isn’t a directory
Behavior if base_dir isn’t found. If false, returns error. If true, returns empty selection
Whether to recurse into subdirectories
Maximum number of subdirectories to recurse into (default: INT32_MAX)
Factory Functions
FileSystemFromUri
Creates a new FileSystem by URI.
Result<std::shared_ptr<FileSystem>> FileSystemFromUri(
const std::string& uri,
std::string* out_path = nullptr)
Result<std::shared_ptr<FileSystem>> FileSystemFromUri(
const std::string& uri,
const io::IOContext& io_context,
std::string* out_path = nullptr)
uri
const std::string&
required
URI-based path (e.g., “file:///some/local/path”, “s3://bucket/key”)
IOContext for the filesystem
Optional output parameter for path inside the filesystem
FileSystemFromUriOrPath
Creates a new FileSystem by URI or local path.
Result<std::shared_ptr<FileSystem>> FileSystemFromUriOrPath(
const std::string& uri,
std::string* out_path = nullptr)
Result<std::shared_ptr<FileSystem>> FileSystemFromUriOrPath(
const std::string& uri,
const io::IOContext& io_context,
std::string* out_path = nullptr)
Specialized Filesystems
SubTreeFileSystem
A FileSystem that delegates to another implementation after prepending a fixed base path.
class SubTreeFileSystem : public FileSystem
base_path
const std::string&
required
Base path to prepend
base_fs
std::shared_ptr<FileSystem>
required
Underlying filesystem
base_fs
std::shared_ptr<FileSystem>
Returns the underlying filesystem
SlowFileSystem
A FileSystem that delegates to another implementation but inserts latencies.
class SlowFileSystem : public FileSystem
base_fs
std::shared_ptr<FileSystem>
required
Underlying filesystem
Average latency in seconds
Random seed for latency generation
Utility Functions
CopyFiles
Copies files, including between different filesystems.
Status CopyFiles(
const std::vector<FileLocator>& sources,
const std::vector<FileLocator>& destinations,
const io::IOContext& io_context = io::default_io_context(),
int64_t chunk_size = 1024 * 1024,
bool use_threads = true)
Status CopyFiles(
const std::shared_ptr<FileSystem>& source_fs,
const FileSelector& source_sel,
const std::shared_ptr<FileSystem>& destination_fs,
const std::string& destination_base_dir,
const io::IOContext& io_context = io::default_io_context(),
int64_t chunk_size = 1024 * 1024,
bool use_threads = true)