Crypto Market Data Lake: Faster Backtests, No More Re-Downloads

You configure a backtest, click run, and then watch a progress bar crawl while the engine re-downloads the same BTC/USDT 1h candles you've fetched a dozen times before. Multiply that wait by 200 hyperopt epochs and the I/O alone burns half your afternoon. VolatiCloud's centralized market data lake removes that step entirely — historical OHLCV is fetched once, stored centrally, and made available instantly to every backtest, hyperopt run, and bot warm-up in your organization.

The Problem with Per-Runner Data Downloads

Self-hosted Freqtrade setups follow a straightforward but inefficient workflow: each backtest checks for local data, downloads whatever's missing, and only then begins the simulation. Switch runners, change pairs, or extend the date range — and the download cycle starts over.

At small scale this is annoying. At scale, it's a hard bottleneck:

• Hyperparameter optimization can require 200–500 individual backtest iterations. If each one waits on data, a meaningful fraction of total optimization time is pure I/O.
• Walk-forward optimization rolls the same date range across N windows. Without shared data, each window risks re-downloading what it already had.
• Multi-pair, multi-timeframe testing explodes the data requirement: 15 pairs × 4 timeframes = 60 datasets you'd otherwise download per runner.

Three additional problems compound the pain:

1. No visibility into what data you have. Without a dedicated inventory, you don't know whether your runner has 90 days or 2 years of ETH/USDT data until you try to run a backtest.
2. Data gaps are silent. A missing month produces results that look plausible but are statistically compromised. Without tracking completeness, gaps slip through.
3. Exchange rate limits punish repeated downloads. Major exchanges throttle their historical data APIs. Re-fetching the same candles burns through those limits and slows your next genuine download.

How VolatiCloud's Data Lake Works

The data lake is a shared OHLCV store that sits outside any individual runner. Instead of each runner managing its own history, a central pool of workers downloads market data from supported exchanges and stores it in a shared location every runner in your organization can access.

The data flow:

1. Configuration — You specify which exchange, pairs, and timeframes to maintain in your runner's data download settings.
2. Backfill — A worker claims the job and fetches historical candles, working backward from the present to fill in the requested history.
3. Upfill — Once initial history is loaded, the system monitors for new candles and keeps the dataset current.
4. Availability — When you run a backtest, the data is already present. The runner reads from the shared store rather than triggering a fresh exchange API call.

Workers run concurrently across multiple exchanges with exchange-aware rate limiting — Binance, OKX, Bybit, BingX, Bitget, and the rest each have different API constraints, and the downloader respects them without any configuration on your part. A lease and renewal mechanism ensures that if a worker fails mid-download, another picks up the job rather than leaving data permanently incomplete.

Backfill and Upfill: Two Distinct Processes

Backfill is the initial historical download. When you add a new pair and timeframe, a backfill job is created to fetch all available history. For major pairs like BTC/USDT, this can mean years of 1h candles — substantial, but only paid once.

Upfill is the ongoing maintenance process. After backfill completes, upfill takes over, checking for new candles and appending them. The system tracks the most recent candle and requests only the delta since the last update. Subsequent updates are near-instant.

The separation matters for planning. When you add a new pair, expect a one-time backfill that might take minutes (or longer for multi-year history on frequent timeframes). After that, you're paying only the trickle cost of upfill.

tip

Add pairs you're actively backtesting to your runner's data download config as soon as you know you'll be testing them. A backfill running in the background while you build your strategy means data is ready by the time you run your first backtest.

Monitoring Data Availability in the Dashboard

A data lake is only useful if you can see what's in it. VolatiCloud's Data Availability panel on the Runner detail page shows a live inventory of your downloaded market data.

Each configured pair and timeframe combination appears as a row with a status indicator:

Status	Meaning
Available	Data is fully downloaded and up to date
Downloading	A backfill or upfill job is actively running
Pending	A job is queued but not yet started
Error	The last download attempt failed (retryable)

Filter by status using the clickable chips — useful when you want to see at a glance which pairs are still in progress versus which are ready. Click "Downloading" to monitor in-progress jobs, or "Available" to confirm what's ready for your next backtest.

For in-progress jobs, the panel updates over WebSocket so you see percentage complete without refreshing. This visibility solves the silent-gap problem: if you're planning a backtest on SOL/USDT from January 2024 to January 2025 and that pair shows "Available," you can run with confidence. If it shows "Pending," you know to wait. The companion post on historical data availability walks through how to use this panel as a pre-backtest check.

Configuring Which Data to Download

When you create or edit a runner, the Data Download Config section lets you specify:

• Exchange — Which exchange's market data to download
• Trading pairs — Individual pairs (BTC/USDT) or all pairs on the exchange
• Timeframes — Which candlestick intervals to maintain (1m, 5m, 15m, 1h, 4h, 1d, etc.)

The pair selector supports infinite scrolling through the full exchange pair list and marks pairs the exchange doesn't support for your selected timeframe. Featured pairs — those commonly used in backtests — sort to the top.

Timeframe options are filtered by exchange. If a particular exchange doesn't offer 1-minute candles, that option won't appear. This prevents configuring downloads that would silently fail.

info

For most backtesting workflows, you need data for the same timeframe you intend to trade plus any higher timeframes your strategy uses for confirmation signals. A strategy running on 1h candles that checks 4h trend context needs both timeframes in the data lake. The multi-timeframe strategies post explains how informative pairs propagate to data requirements.

A practical starting configuration for a new runner:

Exchange: Binance
Pairs: BTC/USDT, ETH/USDT, SOL/USDT, BNB/USDT
Timeframes: 1h, 4h

This covers the most commonly backtested assets at intraday plus swing-trade granularity. Once backfill completes (typically 10–30 minutes for this set), you'll have years of history ready for instant use.

How This Changes Your Backtesting Workflow

The practical impact is most visible during iteration.

Before the data lake:

Build strategy → Configure backtest → Wait for data download
→ Run backtest → Review results → Adjust strategy → Repeat

If each iteration required a 3-minute data download and you made 20 strategy adjustments in a session, that's an hour of waiting on top of a few hours of actual work.

With the data lake:

Build strategy → Configure backtest → Run backtest
→ Review results → Adjust strategy → Repeat

The download step disappears from the loop. Backtests start as soon as you click run.

For hyperparameter optimization, the difference is even more pronounced. Hyperopt runs the same strategy hundreds of times with different parameters. With shared data, every iteration reads from the pre-loaded dataset. The same applies to walk-forward optimization, where rolling windows all read from the same store — no redundant downloads regardless of how many windows you evaluate.

Dry-Run Bots and Live Bots: Same Infrastructure

The data lake isn't only for backtesting. When you start a dry-run or live bot on a runner, it uses the same market history infrastructure. A bot warming up on a strategy that uses a 200-period EMA needs 200 candles of history to compute its initial indicator values — that history comes from the same shared store.

This consistency matters: the data your bot uses for warm-up is the same data your backtest used for validation. There's no divergence between backtesting history and live trading warm-up data.

VolatiCloud also supports running a dry-run bot without connecting a live exchange account — useful when you want to simulate a strategy's real-time behavior before committing API credentials. In that mode, the bot executes against paper fills using live market prices, with the data lake providing historical context.

Data Quality and Exchange Coverage

The data lake supports every exchange VolatiCloud integrates with: Binance, Bybit, OKX, Kraken, BingX, Bitget, KuCoin, Hyperliquid, and others. Each has specific quirks in its historical data API — different rate limits, different maximum page sizes, different behaviors when requesting data outside the available range.

The downloader handles exchange-specific behaviors transparently:

• BingX responds with a 100204 status for pairs with no trading history. The system classifies this as a user-facing error (the pair isn't tradeable) rather than retrying infinitely.
• Bitget has per-request time-window constraints. The downloader clamps request ranges to prevent HTTP 400 errors.
• OKX requires rate-limit jitter on concurrent requests. The worker pool adds randomized delays to stay within API limits.

You don't need to know any of this. The downloader surfaces clean, contiguous OHLCV data regardless of which exchange you're using.

Get Started with the Shared Data Lake

If you already have a runner configured, your data lake is ready to use. Navigate to your runner, click the Data tab, and you'll see the availability panel. If you haven't configured pairs yet, click Edit Runner and add them in the Data Download Config section.

If you don't have a runner yet, the runners overview walks through creating your first one — Docker, Kubernetes, or managed cloud. Once your backfill completes, run your first backtest against the pre-loaded data.

For teams sharing a runner, public runners share the same data lake — so if one team member has already backfilled BTC/USDT 1h data, every other member benefits immediately without any configuration step.

Open the VolatiCloud console to configure your data download settings. Less time waiting for downloads means more time iterating on strategies — which is the only part of the workflow that actually produces edge.