Tarball → Ethereum blobs

Note written by Samuel: ERC7588 not followed because it consumes too much calldata gas. invented custom metadata format that is human-readable enough to decode without a spec document, and also efficient in terms of calldata gas

The code and documentation were written by gpt-6-astra high (high reasoning effort), except for Samuel's note above.

A small Node.js publisher using ethers and c-kzg. The production entry point is publish.mjs. It republishes the whole latest text_english_tarball_md_YYYYMMDD.tar, even when the file is unchanged. Sepolia runs every 10 minutes; mainnet starts a publication daily at 00:00 UTC.

Run

Requires Linux, Node.js 22+, flock (util-linux), and a C/C++ build toolchain and Python for c-kzg (sudo apt-get install build-essential python3 util-linux).

npm ci
npm test
npm run check                       # download + read-only Sepolia RPC check; no key needed
cp .env.example .env
chmod 600 .env
# Edit .env: enter a dedicated funded wallet's PRIVATE_KEY locally.
npm start                          # Sepolia, repeatedly
npm start -- sepolia --once         # one complete publication, or resume an interrupted one
# Mainnet uses the systemd credential setup below.

PRIVATE_KEY is used only for local signing. No keys are bundled. Keep .env, custom RPC URLs and runtime state private; .gitignore excludes local secrets, dependencies and downloads. package-lock.json pins dependencies. The package's private setting prevents accidental npm publication; the Git repo is publishable. Always launch through run.sh / npm so flock prevents overlapping workers.

The fee defaults are 0.001 ETH per transaction and 0.01 ETH per publication, including execution gas and blob gas at their maximum signed prices. Both are configurable in .env. Confirmations default to two (CONFIRMATIONS). Use a separate wallet for each network/service and do not send unrelated transactions from it. The publisher does not fund wallets.

Mainnet hot wallet

Create a new wallet locally using the OS cryptographic random generator:

sudo npm run wallet -- create

This prints only the address. It stores the private key encrypted by systemd-creds in /etc/credstore.encrypted/tarball-blobs-mainnet.cred and writes the public expected address and fee limits to /etc/tarball-blobs/mainnet.env. Both files are owner-only (600) in private directories (700). It refuses to overwrite files, verifies encryption/decryption, and fsyncs before reporting success. The private key is never written to a plaintext file, environment variable, command line, repository or log by the generator. No mnemonic is generated or stored. The npm command and service disable core dumps; direct invocation should do so too.

The publisher reads the decrypted key from systemd's service credential directory at runtime. Mainnet requires this credential and an EXPECTED_ADDRESS match; it will not use PRIVATE_KEY from an environment file. Sepolia's existing environment-file configuration remains supported.

This server has no usable TPM. Host-key encryption supports unattended restarts, but root access or a disk image containing both the credential and the host key can recover the wallet. A compromised publisher can also use its loaded key. Keep only the operating funds needed in a hot wallet; code fee caps do not constrain an attacker who obtains the key. JavaScript cannot guarantee wiping every memory copy of a key. See systemd credentials.

Make a portable encrypted recovery backup in your own SSH terminal before funding:

sudo npm run wallet -- backup /root/.config/tarball-blobs/mainnet.keystore.json

Enter and confirm a long unique passphrase (at least 16 characters) at the hidden prompts. Do not put it in chat, shell arguments or environment variables. The command uses the standard ethers encrypted JSON keystore, decrypts it to verify the address, and refuses to overwrite an existing backup. Copy the encrypted JSON off the server and store the passphrase separately, for example in your password manager. Loss of the server and host key without a portable backup loses access to the wallet. The encrypted systemd credential alone is not a portable backup.

Keep running after reboot

The included systemd service restarts after failures and boots automatically when enabled. Install the code in /opt/tarball-blobs and keep credentials outside it:

sudo install -d /opt/tarball-blobs /etc/tarball-blobs
sudo cp publish.mjs wallet.mjs run.sh package.json package-lock.json /opt/tarball-blobs/
sudo npm ci --prefix /opt/tarball-blobs
sudo install -m 600 .env.example /etc/tarball-blobs/sepolia.env
sudoedit /etc/tarball-blobs/sepolia.env
sudo install -m 644 tarball-blobs@.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now tarball-blobs@sepolia
journalctl -u tarball-blobs@sepolia -f

For production, first create and back up the mainnet wallet as above, then install the mainnet-only credential configuration:

sudo install -d /etc/systemd/system/tarball-blobs@mainnet.service.d
sudo install -m 644 mainnet-credentials.conf /etc/systemd/system/tarball-blobs@mainnet.service.d/credentials.conf
sudo systemctl daemon-reload
# When funded and ready to publish:
sudo systemctl enable --now tarball-blobs@mainnet

Review fee limits in /etc/tarball-blobs/mainnet.env before enabling the service. The template uses the system node on PATH; a Node installation inside a home directory is inaccessible to this service.

The archive and signed transactions are stored in systemd's persistent StateDirectory (/var/lib/tarball-blobs-sepolia or ...-mainnet). Manual runs use ${XDG_STATE_HOME:-$HOME/.local/state}/tarball-blobs/<network>. Preserve that directory when moving servers. Do not run a manual worker alongside systemd with the same wallet and a different state directory: their locks would be separate.

Mainnet waits until the next 00:00 UTC on its first scheduled start, then runs at UTC midnight each day, independent of the server's local timezone. Sepolia's ten-minute interval is measured from each publication's start. Deadlines are saved on disk; upgrading an existing mainnet checkpoint aligns its old 24-hour deadline to midnight after the previous publication's start.

Interrupted uploads resume immediately and finish before the next begins. If downtime, failures or slow inclusion miss a saved deadline, one fresh publication starts after recovery; there is no backlog of overlapping runs. Mainnet then returns to the next UTC midnight. These are start times; confirmations still depend on the network. --once publishes immediately regardless of the schedule, resuming any incomplete publication first.

Freshness and data format

Both directory listings and downloads use a unique query parameter and no-cache headers. The directory is checked again after download. Known cached/stale Cloudflare responses (HIT, STALE, UPDATING), ambiguous Cloudflare statuses, and responses with a positive Age are rejected. The filename's date determines the newest file, not the order of the HTML listing. The downloaded tar is published unchanged; its SHA-256 is checked when resuming.

A client cannot force Cloudflare to bypass a rule that ignores query strings. For guaranteed origin freshness, configure a Bypass cache rule for /raw/misc/*, or set SOURCE_URL to a trusted uncached origin directory (with a trailing slash). If freshness cannot be established, the worker retries rather than publishing a known cached copy. This checks Cloudflare's reported cache status, not the freshness of any hidden cache at the origin. See Cloudflare's cache response definitions.

Each 32-byte field stores 0x00 plus 31 payload bytes. The tar is one continuous stream across all transactions; only its end receives 0x80 and zero padding. Groups contain at most six blobs, using EIP-7594 cell proofs and wrapper version 1. Calldata is UTF-8 JSON: the first transaction contains i (zero), chain_total_tx_count (total transactions), content_type (application/x-tar) and description (text explaining how to reconstruct the chain); later ones contain f (the first transaction's hash) and i (the zero-based transaction index). Transactions send zero ETH to the zero address. Gas accounts for the EIP-7623 calldata floor.

This matches the public Sepolia example. The keyless failover endpoints come from ChainList mainnet and Sepolia, with the selection date in the code. RPC_URLS can replace the list. Every RPC attempt checks its chain ID; timeouts, unsupported methods and send failures fall through to the next endpoint. Public endpoints may rate-limit or refuse blob submissions even if read calls work.

Recovery and verification

Before broadcast, the signed transaction (including blobs) is atomically written and fsynced. After a crash or lost RPC response, the same signed bytes are checked and rebroadcast, preserving the nonce and first transaction hash. Receipt status, canonical block hash and confirmation depth are checked. All saved transactions are checked again on resume and before declaring the upload complete.

Transactions are never automatically replaced with higher-fee versions. This keeps the first transaction's hash stable for its followers. An underpriced saved transaction waits for fees to fall; nonce conflicts or failed transactions need operator attention. Do not delete state while a transaction may still be pending. Completed runs are not monitored indefinitely for reorgs; increase confirmations if that risk matters. Blobs provide temporary data availability, not permanent storage; periodic republication does not make old blob data permanent.

npm test exercises binary boundary round trips, calldata, cache rejection, latest-file selection, RPC failover, receipt checks, real KZG transaction signing, fee caps, a lost broadcast acknowledgement, restart recovery and reorg replay. --check tests the live source and a read-only RPC call; it does not send a transaction or prove a public RPC accepts blob writes. A funded --once run is the end-to-end network test.