Spent UTXO Pruning
This guide explains how to optimize disk usage in cardano-rosetta-java through spent UTXO pruning, including its impact on Rosetta API endpoints and configuration options.
Understanding Spent UTXO Pruning
Spent UTXO pruning is a disk optimization mechanism in cardano-rosetta-java, powered by its underlying indexer, Yaci-Store. This feature selectively removes data related to spent UTXOs from the local database.
Core Principles:
- Targeted Deletion: Only spent UTXOs are removed. All current, unspent UTXOs are preserved, ensuring the accuracy of the present blockchain state and balances.
- Distinction from Other Pruning: This mechanism differs from what is commonly understood as 'pruning' in some other blockchain contexts, including certain descriptions in the Coinbase Mesh API (formerly Rosetta). Unlike methods such as Bitcoin's pruning (which removes entire historical blocks), our approach retains full block history but selectively trims the UTXO set by removing only spent outputs.
How it Works: When enabled, the pruning process operates as follows:
- New UTXOs are indexed as transactions occur.
- UTXOs are marked as spent when consumed in subsequent transactions.
- A background job periodically permanently deletes spent UTXOs that are older than a configurable safety margin (default: 2,160 blocks, ~12 hours on mainnet). This buffer safeguards data integrity against chain rollbacks within Cardano's finality window.
Impact Summary:
| Aspect | Effect |
|---|---|
| Disk Storage | ✅ Significantly reduced (e.g., mainnet from ~1TB to ~500GB) |
| Current UTXO Set | ✅ Fully preserved; current balances remain accurate |
| Historical Spent UTXOs | ⚠️ Permanently deleted beyond the safety margin |
| Query Performance | ✅ Improved for queries against the current UTXO set |
Impact on Rosetta API Endpoints
Spent UTXO pruning affects Rosetta API endpoints differently based on their reliance on historical transaction data. The table below summarizes the impact. Note that "Recent" refers to data within the safety margin (default ~12 hours).
When pruning is enabled, the /network/status endpoint includes an additional oldest_block_identifier object in its response. This identifier corresponds to the latest fully queryable block with complete data. Below this block index, blocks might have missing data due to pruning, making historical queries unreliable.
| Endpoint | Current State | Historical Queries | Impact & Notes |
|---|---|---|---|
/account/balance | ✅ Works | ⚠️ Limited | Low - Current balances unaffected |
/account/coins | ✅ Works | ⚠️ Limited | Low - Current UTXO lists complete |
/block | ✅ Recent only | ❌ Incomplete | High - Missing old transaction inputs |
/block/transaction | ✅ Recent only | ❌ Incomplete | High - Missing spent UTXOs operation details |
/search/transactions | ⚠️ Recent only | ❌ Limited | Medium - Hash search works, address limited |
/network/status | ✅ Works | ✅ Works | None - Returns additional oldest_block_identifier when pruning enabled |
/network/* | ✅ Works | ✅ Works | None - Independent of UTXO data |
/construction/* | ✅ Works | ✅ Works | None - Uses current UTXOs only |
After enabling pruning, searching for transactions by their hash will always work, because transaction records themselves are never pruned. However, searching by address is limited: address-based searches rely on the UTXO set, and once spent UTXOs older than the pruning window are deleted, only transactions involving current or recently spent UTXOs can be found by address. Older history is not returned once pruned.
**Enable Spent UTxO removal **: Set REMOVE_SPENT_UTXOS=true in your environment (e.g., in .env.dockerfile or .env.docker-compose).
**Disable Spent UTxO removal ** (default): Set REMOVE_SPENT_UTXOS=false.
When Spent UTxO Removal should be enabled?
Pruning is beneficial in scenarios where optimizing disk space and focusing on current data is prioritized over maintaining a complete historical record. Consider enabling pruning if your use case aligns with the following:
- Exchange Integrations & Wallet Services: Primarily for tracking current balances, processing recent deposits/withdrawals, and validating recent transactions.
- Resource-Constrained Environments: Ideal when disk space is a significant limitation (e.g., under 1TB available for mainnet data).
- Tip-of-Chain Operations: For applications focused on the latest blockchain state rather than deep historical analysis.
- Development and Testing: Useful when a full historical dataset is not essential for development or testing purposes.
When to avoid setting UtxO Removal feature?
Avoid pruning if your operational or regulatory requirements necessitate access to complete and auditable historical blockchain data. Pruning is generally not suitable if you need:
- Complete Historical Data & Deep Queries: For comprehensive auditing, compliance, data analytics, or block explorer-like functionality that requires querying full transaction history from any point in time.
- Strict Compliance and Audit Trails: If regulatory mandates demand immutable, complete historical records. Pruned data cannot be recovered without a full resync, and historical queries for
/blockand/block/transactionbecome unreliable beyond the safety window.
Once data is pruned, it cannot be recovered without a full blockchain resynchronization. Assess your historical data needs carefully before enabling pruning.
Configuration
Spent UTXO pruning is configured via environment variables, typically set in your .env.dockerfile or .env.docker-compose file. Below is an example demonstrating the available settings and their default values:
# --- Spent UTXO Pruning Configuration ---
# Enable or disable spent UTXO pruning.
# Default: false (Pruning is disabled by default)
# To enable, set to: true
REMOVE_SPENT_UTXOS=true
# Safety margin: Number of recent blocks for which spent UTXOs are retained.
# Default: 2160 (approximately 12 hours of blocks on mainnet)
# This value balances safety for rollbacks against storage savings.
# Example: To keep ~24 hours of spent UTXOs, set to 4320.
# Note: Larger REMOVE_SPENT_UTXOS_LAST_BLOCKS_GRACE_COUNT values provide longer historical query support
# but use more disk space and delay the realization of storage benefits.
REMOVE_SPENT_UTXOS_LAST_BLOCKS_GRACE_COUNT=2160
- Start with the default settings (
REMOVE_SPENT_UTXOS=falsemeans pruning is initially off). - If enabling, the provided defaults (
REMOVE_SPENT_UTXOS_LAST_BLOCKS_GRACE_COUNT=2160) are sensible starting points. - Adjust
REMOVE_SPENT_UTXOS_LAST_BLOCKS_GRACE_COUNTbased on your specific needs for recent historical data access. A larger value offers a longer window for historical queries but increases storage.
Migration and Operational Notes
This section outlines key considerations when changing pruning settings or managing a system with pruning enabled.
Enabling Pruning on an Existing Deployment
- Configuration: Update the necessary environment variables (e.g.,
REMOVE_SPENT_UTXOS=true) and restart your cardano-rosetta-java services. - Initial Monitoring: After enabling, closely monitor your application for any errors. Disk usage benefits will typically appear after the first pruning cycle completes.
Disabling Pruning
- Halts Deletion, No Restoration: Setting
REMOVE_SPENT_UTXOS=falseand restarting services will stop further deletion of spent UTXOs. However, this action does not restore any data already pruned. - Full Resync for Complete History: To regain a complete historical dataset after pruning has been active, a full resynchronization of the blockchain data is required.
- Storage Growth Resumes: Once pruning is disabled, expect storage usage to gradually increase again as new UTXOs (both spent and unspent) accumulate.