Storacha Ignites: Engineering Highlights That Set 2024 Ablaze

2024 was an inferno of innovation for Storacha’s engineering team! From setting decentralized storage ablaze with game-changing protocols to pulling off spicy live demos, we’ve been firing on all cylinders to build the hottest storage network in web3. This year wasn’t just about pushing boundaries, it was about smashing them, one bold step at a time. Buckle up as we take you through the engineering highlights of a year that was anything but ordinary.

Transitioned to the Blob Protocol

The blob protocol enables data storage at a decentralized level. It is an evolution of our “store protocol” adding the ability to store _any_ data, not just DAGs, with the ability to verify delivery of a blob to a decentralized storage node from the client, as well as the network service. That’s a lot to take in, but put more simply, the blob protocol allows you to verify (cryptographically) that your data has been added to the storage network all via the magic of UCAN.

At a high level, the blob protocol looks like this:

1. A client requests to store a blob by sending the hash of the blob and it’s size to the upload service.
2. The upload service negotiates with a storage node, allocating space for a blob to be written.
3. The upload service responds to the client an instruction to PUT the blob to the URL contained in the signed allocation receipt it received from the storage node.
4. The client uploads the data, a HTTP PUT to the provided URL.
5. The client sends a signed receipt to the upload service to inform it that the blob was written to the allocated space.
6. The upload service asks the storage node to issue a signed receipt, accepting that it did in fact, receive the blob.
7. The client waits for the storage node to issue the acceptance receipt.
8. DONE! Almost ten steps, wow.

The blob protocol also represents a shift away from storing metadata on behalf of users in centralized databases/buckets. Any metadata, for example, lists of things stored, filenames, upload dates, and most importantly, indexes, is all simply not scalable in a centralized context without considerable cost and maintenance. We are instead designing protocols that allow users to manage metadata themselves on a decentralized network. The biggest source of metadata that we generated and stored for users prior to the blob protocol was indexes. Indexes tell us where to look in file for blocks that make up a DAG — they are a mapping of hash => byte offset and length. We need to know this information so that data can be retrieved peer-to-peer over bitswap or via an IPFS gateway. So, when the blob protocol was implemented, we also changed our client libraries to generate, store and register indexes for data they are storing. Later in the year we implemented decentralized indexing, which transitioned that data into IPNI.

Storacha is BORN!

Arguably the biggest and most exciting thing that happened this year was the birth of Storacha. We exploded into existence from a merge of teams in Protocol Labs, specifically the web3.storage team and the Saturn team. We decided the main focus of Storacha should be to build on the work web3.storage had already begun — decentralizing completely an already successful and well established, but partially centralized service. First though, we needed a name! After what felt like around 1,000 name suggestions or so, we realized that the job of decentralizing web3.storage was basically converting what web3.storage does best into a decentralized network. What web3.storage does best is HOT storage, I mean, to be fair, it also does a great job of aggregating and onboarding all uploaded data to Filecoin L1 to service that longer term, durable storage, but it’s not the main focus.

Anyhow, at some point we managed to associate hot storage with hot sauce, and the Storacha name popped out. They say there are 2 hard problems in computing: naming things, cache invalidation and off by 1 errors. After we named the beast, the rest came easily, we designed a logo, launched a website and rebranded console.

Implemented Ucanto in Go

We spent a considerable amount of time building Ucanto — an RPC framework for UCAN. A lot of the good ideas from Ucanto went into the specifications for UCAN 1.0. Ucanto was originally written in JS, to allow it to be used in both a server and client (browser) context. Our bread and butter is enabling application developers, so having a tool that is usable from the browser is first and foremost. That said, we have a multilingual team, and our protocols need to be available and accessible in other languages, not to mention we wanted to write our indexing service in Go.

Launched a Migration Tool

The NFT.Storage team transitioned their offering but users of the original offering can migrate any NFTs they wish to retain to Storacha super easily by using the console: https://console.storacha.network. Users of the pre-UCAN enabled web3.storage can transition to Storacha in exactly the same way. Ok, sounds super boring, but it’s not, promise! Let me tell you why — we content address EVERYTHING, migrating does not involve copying data, since we already have it! All migrating does is register a list of CIDs in your space. You’re not downloading and re-uploading everything, even big accounts can be migrated quickly.

One last thing, we built this because we give a ****. We don’t want to see any user left stranded.

Designed and implemented the Decentralized Indexing Service

Ok, first up this needs an entire blog post of it’s own. I talked about indexes earlier. Well yeah, we used to have a GIANT database that stored the offset and length of EVERY block in EVERY CAR file uploaded. We’re in the throws of deploying a new service to production that leans on IPNI for the majority of our indexing needs. Not only is it going to move our indexing to a credible decentralized context but it’s also going to speed up TTFB for retrievals! OMFG!

Launched an Alpha Network of Storage Nodes

At Devcon in Bangkok, Storacha held a launch PAR-TAY — the “Hot Ones and Zeros” — widely regarded as the best party @ Devcon and possibly in the history of the planet. We hosted a panel of industry experts who attempted to answer questions while consuming chicken wings laden with progressively spicier hot sauce. Hilarity ensues. We also did a big LIVE ACTION demo of the new decentralized indexing service, including launching a storage node onto our alpha test network LIVE in real time!

It represented a giant leap towards our eventual goal of a decentralized HOT storage network launch, demonstrating, for real, that the Storacha network can support multiple upload targets and retrieve stored data immediately afterwards from the decentralized network without issue! We are now at base camp of a giant mountain but dammit we can’t wait to start ascending.

The other major milestone here was that, prior to the party we launched a storage node that integrated with Curio to provide access to the newest proof (not yet) available on the Filecoin network — Proof of Data Possession (PDP for short). This is a new proof type that allows hot storage nodes to prove the fact that they have access to an unsealed copy of some data, and are ready to serve it to clients. We’ll be leaning heavily on this proof in the live network so it’s incredible (and super valuable for testing) to see an integration so early.

Took ownership of Spade

Broker in the house! We very recently took over ownership and management of the broker service we use — Spade. Our Filecoin aggregation pipeline builds aggregation “recipes” for Filecoin L1 Storage Providers (SPs). Spade is the software that finds SPs that are looking to onboard data according to our requirements (N copies, geographical diversity etc.). We say that Spade is a broker and finds SPs, but in reality it’s more like Ebay — a marketplace where aggregates are offered and SPs “buy” them. When an SP agrees to onboard an aggregate it is provided with the recipe to build it. The recipe is a list of pieces and URLs and the SP fetches the individual pieces from Storage Nodes on the Storacha network to build the aggregate that is eventually sealed, stored and registered on the L1 chain in a deal.

Implemented Egress Tracking via UCAN authorized gateway

The Storacha Network doesn’t really work unless we can reward Storage Nodes for serving content they have hot and fresh out the kitchen. We’re, right now, finishing up work on a protocol that allows users to authorize actors on the network to serve data that they have uploaded to a space. Actors you say? Yes, for example — IPFS gateways. Authorized actors can now record egress events against a space, which will eventually be tallied against signed egress events from storage nodes. The two together will eventually allow $RACHA rewards to be issued!

Turning Up the Heat: Storacha’s Sizzling Plans for 2025

As we close the book on 2024, one thing’s for sure: the fire’s just getting started. Every breakthrough, every protocol tweak, every spicy demo — it’s all been fuel for a decentralized storage revolution. None of this would’ve been possible without our unstoppable engineering team, the support of our fiery community, and the trust of our amazing partners.

2025 is calling, and we’re ready to crank up the heat even more. From blazing new trails to conquering new challenges, Storacha is primed for another year of explosive growth and innovation. Thanks for being on this wild ride with us — now let’s turn up the heat and make 2025 unforgettable! 🔥🌶️