The SingularVault Protocol — Technical Architecture & Research Brief

01 — The Technical Problem

Cloud storage today operates on isolated silos. Each provider — AWS S3, Azure Blob Storage, Google Cloud Storage, and thousands of smaller providers — maintains completely independent storage infrastructure. When a user uploads a file to Provider A, and the same file (byte-for-byte identical) exists on Provider B, C, and D, no cross-provider awareness exists.

Within a single provider, deduplication is common. AWS, for example, uses internal dedup within S3 at the block level. Enterprise storage systems (NetApp ONTAP, Dell EMC, Veeam) have offered single-tenant dedup for years. But cross-provider deduplication does not exist at any meaningful scale.

The result: trillions of file-blocks stored redundantly across competing infrastructure, each copy consuming disk space, electricity for storage and retrieval, cooling water, and physical land. Conservative estimates suggest that cross-provider dedup could reduce global storage volume by 30–60%, depending on the file type distribution.

02 — Proposed Architecture

SingularVault would function as a middleware protocol layer that sits between users/applications and cloud storage providers. It is not a replacement for S3 or Blob Storage — it's a coordination layer that enables cross-provider content awareness.

Content-Addressable Storage (CAS)

At the core, every file (or file-block) is identified by its cryptographic hash. This is the same principle behind Git, IPFS, and BitTorrent. The hash becomes the address — if two files produce the same hash, they are the same content.

// Simplified content addressing
file_content = read("photo.jpg")
hash = SHA-256(file_content)  // → "a3f2b8c9..."

// This hash IS the storage address
// Same content always → same hash → same address
// No duplicates by mathematical guarantee

if global_index.exists(hash):
    return reference(hash)  // Already stored — just link to it
else:
    store(hash, encrypted_content)
    global_index.register(hash)

System Components

The architecture consists of four primary components:

03 — The Encryption-Deduplication Paradox

This is the hardest technical problem. Standard encryption (AES-256 with random IVs) produces unique ciphertext for every encryption operation — meaning two copies of the same file, encrypted by different users, produce completely different encrypted outputs. This defeats deduplication.

Convergent Encryption (CE)

The leading approach: derive the encryption key from the content itself.

// Convergent Encryption
content_key = Hash(file_content)    // Key derived FROM the content
ciphertext = Encrypt(content_key, file_content)

// Same content → same key → same ciphertext
// Deduplication works on ciphertext!

// To access: user stores their content_key securely
// The storage system never sees the key or plaintext

Known vulnerability: CE is susceptible to confirmation-of-a-file attacks. If an attacker knows (or can guess) the plaintext, they can compute the hash and confirm the file exists in the system. This is a real concern for low-entropy files (e.g., standard system files, common documents).

Message-Locked Encryption (MLE)

A more formal framework proposed by Bellare, Keelveedhi, and Ristenpart (2013). MLE formalizes CE and introduces additional constructions that can mitigate some attacks, including randomized convergent encryption and server-aided MLE schemes.

HCE² — Hybrid Convergent Encryption

A potential approach for SingularVault would combine CE with per-user key wrapping:

// HCE²: Hybrid approach
content_hash = SHA-256(file_content)
content_key = HKDF(content_hash, salt="singularvault-v1")
ciphertext = AES-256-GCM(content_key, file_content)

// User wraps the content_key with their own key
wrapped_key = RSA-OAEP(user_public_key, content_key)

// Storage sees: ciphertext (dedupable) + wrapped_key (per-user)
// User needs their private key to unwrap → access content
// System proves uniqueness via content_hash without seeing plaintext

This allows deduplication at the ciphertext level while maintaining per-user access control. The tradeoff is increased key management complexity.

04 — Protocol Flow

Here's how a file upload would work through the SingularVault protocol:

Download Protocol

Retrieval is straightforward: the client presents their reference (hash + wrapped key), the system locates the ciphertext via the hash registry, downloads the encrypted blob, unwraps the content key using the user's private key, and decrypts locally. The storage system never handles plaintext.

Deletion Handling

This is a critical design decision. When User A deletes “their” file, but User B still has a reference to the same content — the underlying data must persist. The system needs reference counting: the physical data is only deleted when the last reference is removed. This introduces garbage collection complexity and requires careful handling to prevent data loss or orphaned blobs.

05 — Prior Art & Landscape

Several projects have explored aspects of this problem. Understanding where they succeeded and failed is critical for SingularVault's design.

Key differentiator: IPFS and Filecoin are decentralized storage networks that aim to replace traditional cloud providers. SingularVault's approach is different — it aims to sit on top of existing providers as a coordination layer, requiring minimal changes to existing infrastructure. Think of it as a dedup protocol rather than a storage network.

06 — Open Challenges

These are the problems that need to be solved before SingularVault can move from theory to practice. This is exactly what we need specialists for.

07 — Research Questions

If you're a specialist considering joining this project, these are the open questions we most need help with: