OmniStorage¶

Unified storage abstraction layer for Go

OmniStorage provides a single interface for reading and writing to various storage backends with composable layers for compression and record framing. Inspired by rclone.

Features¶

Single interface for multiple storage backends (local files, cloud drives, etc.)
Composable layers for compression (gzip, zstd) and formatting (NDJSON)
Sync engine for file synchronization between backends (like rclone sync)
Extended interface for metadata, server-side copy/move, and capability discovery
Multi-writer for fan-out to multiple backends simultaneously
Backend registration allowing external packages to implement backends

Quick Example¶

package main

import (
    "context"
    "io"
    "log"

    "github.com/plexusone/omnistorage-core/object/backend/file"
)

func main() {
    ctx := context.Background()

    // Create a file backend
    backend := file.New(file.Config{Root: "/data"})
    defer backend.Close()

    // Write a file
    w, _ := backend.NewWriter(ctx, "hello.txt")
    w.Write([]byte("Hello, World!"))
    w.Close()

    // Read it back
    r, _ := backend.NewReader(ctx, "hello.txt")
    data, _ := io.ReadAll(r)
    r.Close()

    log.Println(string(data)) // "Hello, World!"
}

Why OmniStorage?¶

Challenge	OmniStorage Solution
Different APIs for each storage provider	Single `Backend` interface
Provider-specific code	Backend abstraction with registration
Duplicated compression/formatting logic	Composable layers
Hard to switch providers	Configuration-driven backend selection
No sync between backends	rclone-inspired sync engine

Supported Backends¶

Core Backends (omnistorage-core)¶

Backend	Package	Status
Local Filesystem	`object/backend/file`	Stable
In-Memory	`object/backend/memory`	Stable
Go Channel	`object/backend/channel`	Stable
SFTP	`object/backend/sftp`	Stable

External Backends (separate packages)¶

Backend	Package	Status
S3-Compatible	omni-aws	Stable
Google Drive	omni-google	Stable
Google Cloud Storage	omni-google	Planned
Azure Blob Storage	Planned	-

Getting Started¶

- :material-download: **[Installation](getting-started/installation.md)** Install omnistorage and get started in minutes - :material-rocket-launch: **[Quick Start](getting-started/quick-start.md)** Learn the basics with hands-on examples - :material-book-open: **[Concepts](getting-started/concepts.md)** Understand the architecture and design

Documentation Sections¶

Backends - Storage backend documentation
Sync Engine - File synchronization (rclone-like)
Guides - How-to guides and tutorials
Reference - API reference and interfaces

omni-aws - AWS backends (S3, DynamoDB)
omni-google - Google backends (Drive, GCS)
rclone - Inspiration for backend coverage and sync capabilities
go-cloud - Google's portable cloud APIs
afero - Filesystem abstraction

License¶

MIT License - see LICENSE for details.