Why Full Re-Sync Matters
Missing or inconsistent data can break your dashboards. It throws off reporting, hides real issues, and can lead to poor decisions. Sometimes data just slips through the cracks—an API hiccup, expired credentials, or a delay on the source system’s side. Other times, historical data simply wasn’t pulled during the initial sync setup.
To handle these scenarios, we have built Full Data Re-Sync.
This feature reloads all historical data from your source systems—like Meta Ads, Google Analytics, or Shopify—and pushes it to your destination, such as BigQuery, Snowflake, or Databricks.
It ensures that your warehouse or lake reflects everything, not just recent changes. You gain full control over historical data and ensure consistency across reports and dashboards.
This article walks through when to use Full Data Re-Sync and how it works.
Introducing Full Data Re-Sync
Full Data Re-Sync is a simple but powerful feature that our customers love. It reloads all historical data from your connected source and sends it to your data destination—no manual patches or one-off fixes required.
Instead of backfilling data piece by piece, you get a clean, complete dataset in one go. It’s especially helpful when data was missed during setup, syncs were interrupted, or reporting needs have evolved.
And because the re-sync process is automated, you don’t need to manually handle pagination, rate limits, or batching logic. The system takes care of retries, slicing, and data delivery behind the scenes—making large historical loads as easy as selecting a time range.
By refreshing your historical data, you ensure your analytics environment reflects the full story—not just the last 30 days.
Let’s look at some real-world scenarios where Full Data Re-Sync is the right move.
Addressing Data Challenges with Full Data Re-Sync
Data pipelines operate in dynamic environments. APIs change, source systems get updated, and access credentials expire. Over time, gaps in data can appear.
Full Data Re-Sync makes it easy to correct those gaps and ensure your reporting layer remains accurate and complete—without rebuilding your pipelines from scratch.
🛠️ Recovering Lost Data
A source system was temporarily unavailable, or credentials expired, and a portion of your data wasn’t synced.
Instead of manually finding and reloading the missing records, just run a Full Data Re-Sync for the affected date range.
The system will fetch the data again using upsert mode, meaning:
-
Missing records are added
-
Existing ones are updated safely
-
No duplicates are created
📅 Backfilling Historical Data
Your reporting requirements have changed. You originally needed just 30 days of data, but now you need the full past year.
With Full Data Re-Sync, you can easily pull in historical data that wasn’t part of the original sync window. Simply choose the full date range you need, and the system takes care of the rest.
This is especially helpful during onboarding or when broadening your analytics scope.
🔄 Fixing Past Sync Issues
Maybe a transformation rule misclassified data, or a mapping field was incorrect. You’ve fixed the logic, but old records in your destination are still wrong.
Trigger a Full Data Re-Sync for the relevant time period, and outdated records will be corrected automatically—no manual updates needed.
Since the feature uses upsert mode, it only updates what’s necessary—avoiding duplication or further errors.
✅ Audit and Compliance
Preparing for an audit? Need to ensure complete data integrity?
Full Data Re-Sync ensures your destination accurately mirrors your source system across any time range—so you can meet compliance or internal data quality standards with confidence.
🚚 Migration or System Changes
If you’re migrating to a new warehouse or restructuring your data architecture, Full Data Re-Sync makes the process seamless.
Once your new destination is ready, use the feature to reload historical data—no need for manual exports, SQL scripts, or third-party loaders.
How Full Data Re-Sync Works in Dataddo
Running a Full Data Re-Sync in Dataddo is simple, even if you’re not a technical expert. Here’s how it works, step by step.
1. Navigate to the Flows Page
Go to the Flows page in your Dataddo dashboard. This is where all your data connections live.
Fig 1: Datadoo Flow Page
2. Click the Full Data Re-Sync Icon
Next to the flow you want to update, click the Full Data Re-Sync icon. This opens the setup panel for reloading data.
Fig 2: Full Data Re-Sync Option in the Flow
3. Select Your Date Range
Choose the timeframe you want to re-sync. You can reload all available history, or just a specific period. Once selected, confirm to start the process.
Fig 3: Selecting Time Range for Full Data Re-Sync
4. Monitor Job Progress
After triggering the re-sync, you’ll see a list of jobs under Full Data Re-Sync Jobs Overview.
This page shows:
- Job status (running, completed, failed)
- A link to the original flow
- Time of creation, last update, and finish date
- You can check the logs under Quick Actions
Behind the Scenes: How does it Work
Once started, Dataddo handles the sync process for you. If your dataset is large, it automatically splits the request into smaller chunks. This helps stay within source system limits (like API rate limits or data caps).
The sync runs in upsert mode, which means new records are added and existing ones are updated. This prevents duplicates and keeps your dataset clean.
A Few Things to Keep in Mind
- Full Data Re-Sync only works with destinations that support upsert mode, like BigQuery, Snowflake, and Databricks.
- It runs separately from your regular data syncs, so your live pipelines work normally.
- For large datasets, the re-sync might take time, as this is expected. You can track every step on the dashboard.
This process gives you full visibility and control while keeping your data pipeline stable.
Summary
Full Data Re-Sync transforms what used to be a time-consuming task into a simple, reliable process. Whether you’re recovering from a missed sync, updating old records, or expanding your analytics window, it helps you keep your data clean, complete, and ready for action.
It’s not just a safety net—it’s a foundation for trustworthy reporting.
Comments