Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.phylax.systems/llms.txt

Use this file to discover all available pages before exploring further.

Backtesting runs your assertions against actual historical transactions from a specified block range. This helps ensure your assertions work correctly with real transaction patterns and don’t trigger false positives on legitimate protocol operations.
If you’re looking for the reference documentation on all configuration parameters, see the Backtesting Reference.

​
How It Works

Backtesting operates in two phases:
  1. Transaction Fetching: Identifies all transactions to the target contract in the block range using trace APIs (with automatic fallback)
  2. Transaction Validation: Replays each transaction with your assertion enabled
The tool automatically detects both direct calls and internal calls (e.g., when Contract A calls your target Contract B through a router or aggregator) using trace APIs. For validation, the tool forks the EVM at the exact transaction state and replays it against the specified assertion.

​
Example

// SPDX-License-Identifier: MIT
pragma solidity ^0.8.20;

import {CredibleTestWithBacktesting} from "credible-std/CredibleTestWithBacktesting.sol";
import {BacktestingTypes} from "credible-std/utils/BacktestingTypes.sol";
import {MyAssertion} from "../assertions/src/MyAssertion.a.sol";

contract MyBacktestingTest is CredibleTestWithBacktesting {
    function testHistoricalTransactions() public {
        BacktestingTypes.BacktestingResults memory results = executeBacktest(
            BacktestingTypes.BacktestingConfig({
                targetContract: 0x5fd84259d66Cd46123540766Be93DFE6D43130D7, // USDC on Optimism Sepolia
                endBlock: 31336940, // Latest block to test
                blockRange: 20, // Number of blocks to test
                assertionCreationCode: type(MyAssertion).creationCode,
                assertionSelector: MyAssertion.assertionInvariant.selector,
                rpcUrl: "https://sepolia.optimism.io",
                detailedBlocks: false,
                forkByTxHash: true
            })
        );

        // Check results
        assertEq(results.assertionFailures, 0, "Found protocol violations!");
    }
}
Backtesting tests inherit from CredibleTestWithBacktesting instead of CredibleTest. The executeBacktest function returns results in a BacktestingTypes.BacktestingResults struct.
Consider excluding backtesting from your CI/CD pipeline as it can take time to run. Run backtesting manually on demand to increase confidence in your assertions before deployment.

​
Common Configurations

​
Standard Configuration

BacktestingTypes.BacktestingConfig({
    targetContract: TARGET_CONTRACT,
    endBlock: LATEST_BLOCK,
    blockRange: 100,
    assertionCreationCode: type(MyAssertion).creationCode,
    assertionSelector: MyAssertion.assertionInvariant.selector,
    rpcUrl: vm.envString("MAINNET_RPC_URL"),
    detailedBlocks: false,
    forkByTxHash: true
})

​
Testing Against a Specific Block

When you want to test your assertion against a specific known transaction (such as a historical exploit), set blockRange: 1 with the block number containing that transaction: 
BacktestingTypes.BacktestingConfig({
    targetContract: BALANCER_V2_VAULT,
    endBlock: 23717632,     // Block containing the exploit transaction
    blockRange: 1,          // Only test this single block
    assertionCreationCode: type(BatchSwapDeltaAssertion).creationCode,
    assertionSelector: BatchSwapDeltaAssertion.assertionBatchSwapRateManipulation.selector,
    rpcUrl: vm.envString("MAINNET_RPC_URL"),
    detailedBlocks: false,
    forkByTxHash: true
})
This is particularly useful for:
  • Validating that your assertion catches known exploits
  • Testing edge cases found in production
  • Verifying assertion behavior on specific problematic transactions
See the Balancer V2 Rate Manipulation Exploit for a real-world example of using backtesting to verify an assertion catches a historical exploit transaction.

​
Testing a Single Transaction by Hash

For debugging false positives or validating against a specific transaction, use executeBacktestForTransaction: 
bytes32 txHash = 0x1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef;

BacktestingTypes.BacktestingResults memory results = executeBacktestForTransaction(
    txHash,
    TARGET_CONTRACT,
    type(MyAssertion).creationCode,
    MyAssertion.assertionInvariant.selector,
    vm.envString("MAINNET_RPC_URL")
);
This is particularly useful for:
  • Debugging potential false positives reported during staging (see Testing Strategy)
  • Quick validation of assertion behavior on a specific transaction
  • Investigating incidents without needing to know the block number

​
Understanding Results

When backtesting completes, transactions are categorized into different result types to help you quickly identify issues:

​
Result Categories

PASS - Assertion passed successfully
  • Transaction replayed and assertion validated without errors
  • Recommended Action: None - everything is working as expected
SKIP - Assertion not triggered
  • Transaction called a function not monitored by your assertion (function selector didn’t match)
  • This is normal behavior - your assertion only triggers on specific function calls
  • Recommended Action: None required, unless you expected this transaction to trigger your assertion
  • Note: These are NOT assertion violations - they indicate transactions that aren’t relevant to your assertion
REPLAY_FAIL - Transaction replay failed
  • Transaction failed to replay before the assertion could execute
  • Common causes:
    • State dependencies: Transaction depends on specific state that isn’t present
    • Context requirements: Transaction requires specific block context
    • Insufficient funds: Sender balance changed between original execution and replay
  • Recommended Action: Check the error message for details; this usually indicates an environmental issue rather than an assertion problem
  • Note: These are NOT assertion violations
FAIL - Assertion reverted
  • Your assertion reverted when validating this transaction
  • Most Common Causes:
    • False positive: Your assertion logic incorrectly flags legitimate protocol behavior (most common)
    • Gas limit exceeded: Assertion ran out of gas (300k limit)
    • Assertion bug: Logic error in your assertion code
    • Known exploit: If testing against a historical exploit block, this is expected behavior
  • Recommended Action:
    • Check if the transaction is a known exploit - if so, this confirms your assertion works correctly
    • Review your assertion logic to ensure it properly handles this transaction pattern
    • Check if the assertion is running out of gas and needs optimization
    • Verify the transaction on a block explorer to understand what it does
ERROR - Unexpected failure
  • An error occurred that doesn’t fit other categories
  • May indicate RPC issues, assertion bugs, or unexpected contract behavior
  • Recommended Action: Check the error message and retry; if persistent, file a bug report.

​
Reading the Summary

At the end of each backtest run, you’ll see a summary like: 
==========================================
           BACKTESTING SUMMARY
==========================================
Block Range: 23697580 - 23697590
Total Transactions: 15
Processed Transactions: 15
Successful Validations: 10
Skipped Transactions: 3
Failed Validations: 2

=== ERROR BREAKDOWN ===
Protocol Violations (Assertion Failures): 1
Replay Failures (Tx reverted before assertion): 1

Success Rate: 83%
================================

​
Running Tests

# Set RPC URL environment variable
export RPC_URL="YOUR_RPC_URL"

# Run backtesting tests
pcl test --ffi --match-test testHistoricalTransactions
The --ffi flag is required to enable foreign function interface for RPC calls.

​
Foundry Configuration

Enable FFI in your Foundry profile: 
[profile.backtest-assertions]
src = "assertions/src"
test = "assertions/test/backtest"
out = "assertions/out"
libs = ["lib"]
solc = "0.8.29"
optimizer = true
optimizer_runs = 200
ffi = true # To avoid setting --ffi in the test command
See CI/CD Integration for complete Foundry profile configuration.

​
Best Practices

  1. Start small - Test with 10-20 blocks first to verify your setup
  2. Use paid RPC providers - For block ranges over 1,000
  3. Check assertion failures - Ensure results.assertionFailures == 0
  4. Run before deployment - Catch false positives on real transactions
  5. Run manually - Exclude from CI/CD to avoid long test runs
  6. Debug with single transactions - Use executeBacktestForTransaction to investigate specific failures
Learn More: