flowchart LR
    subgraph Filecoin Snapshot Service
        subgraph Snapshot Generation
            A[Trigger Cron Job] --> B[Download Latest Filecoin Snapshot with aria2c]
            B --> C[Import Snapshot into Forest Node]
            C --> D[Start Forest Node & Install Utilities curl, aria2]
            D --> E[Sync Forest Node with Network Head]
            E --> F[Export Snapshot Including Last 2000 Blocks]
            F --> G[Run Initial Validation with forest-tool]
            G --> H[Stop Forest Node & Store Snapshot]
            H --> I[Destroy Forest DB to Save Space]
        end
        subgraph Snapshot Validation
            I --> J[Import Snapshot into Lotus Node]
            J --> K[Start Lotus Node & Install Utilities curl, jq]
            K --> L[Sync Lotus Node with Network Head]
            L --> M[Verify Lotus Node Sync Status]
            M --> N[Retrieve Head Epoch from Lotus Node using ChainHead RPC Method]
            N --> O[Calculate Adjusted Epoch: Head Epoch - 1950]
            O --> P[GET Block Hash from a remote Full Node using Adjusted Epoch with ChainGetTipSetByHeight]
            P --> Q[Verify Block Hash in Local Node using ChainGetBlock RPC Meethod]
        end
        Q --> T[Upload Validated Snapshot to Regular S3 Bucket]
        %% Failure Handling
        Q -- Validation Failure --> V[Move Snapshot to Forbidden Bucket]
        G -- Generation/Validation Failure --> V
        %% Notifications
        V --> W1[Send Alert to Team for Investigation]
        W1 --> W2[On-Call Team Investigates and Escalates]
    end

This document outlines the step-by-step process for generating and validating a Filecoin snapshot. The diagram above provides a quick overview of the snapshot service architecture.

Snapshot Service Process Overview

The snapshot process is initiated by a cron job that triggers the service container to execute hourly. Below is a detailed step-by-step outline of how snapshots are generated and validated:

1. Download the Snapshot:
   • A Filecoin snapshot is downloaded from the following URL: https://forest-archive.chainsafe.dev/<chain>/latest.
   • This snapshot represents the current state of the Filecoin blockchain at the time of download.
2. Import Snapshot into Forest Node:
   • Once the download is complete, the snapshot is imported into a Forest node.
3. Start the Forest Node:
   • After successful import, start the Forest node.
4. Export Snapshot:
   • Once the Forest node is fully synchronized, initiate the export process.
   • The export includes the last 2000 blocks of the blockchain.
5. Forest Tool Validate the Exported Snapshot:
   • After exporting, a simple validation is performed using the command: forest-tool snapshot validate forest_snapshot_*.forest.car.zst.
   • This checks the snapshot for integrity and consistency,
6. Stop the Forest Node and Store the Snapshot:
   • Upon successful validation, the Forest node is stopped, and the snapshot is stored.
7. Import Snapshot into Lotus Node:
   • Import the newly generated snapshot into a Lotus node to further verify its validity.
8. Start the Lotus Node:
   • After importing, starts the Lotus node and allow it to sync with the network head.
9. Check Node Sync Status and Health:
   • Once the Lotus node is synchronized, verify its sync status and overall health to ensure proper functionality.

Begin Main Validation

• The main validation process, which includes the following sub-steps:
  • 10.1. Retrieve Current Head Epoch - Query the local Lotus node to retrieve the current head epoch using the Filecoin.ChainHead RPC method. - Example: If the current head epoch is 4,000, the resulting epoch after subtraction will be 2,050 (4,000 - 1,950).
  • 10.2. Retrieve Block Hash for Adjusted Epoch - Use the adjusted epoch value (e.g., 2,050) to query a running Filecoin node (managed by the infrastructure team) via the Filecoin.ChainGetTipSetByHeight RPC method. - This retrieves the block hash from 1,950 epochs prior.
  • 10.3. Verify Block Hash Availability - Use the obtained block hash to query the local node with the Filecoin.ChainGetBlock method. - Ensure the block is available and matches the expected epoch.
  • 10.4. Finalize Validation - If all sub-steps are completed successfully, proceed to mark the snapshot generation and validation as successful. - Upload the validated snapshot to an S3 bucket to be served to the community.

Important Note:

If any step in this process fails, an alert is sent to the team. The snapshot is then moved to a forbidden bucket for further investigation by the on-call team. If necessary, the issue is escalated for resolution.