Google Data Processing Delay Workaround

When working with data, getting updates in real time can be very beneficial for timely and fact-based decision-making. Unfortunately, when it comes to services provided by Google, there is often a processing delay ranging from 24 to 72 hours.

Most Google services will show a “no data available” message for the past two to three days. Google Analytics, Google My Business, or YouTube all have up to 48 hours of data delay. In the case of Google Search Console, data from up to 72 hours ago might not be available.

As for Google Analytics 4, although the data from the past 48 hours is being processed, you can still access it. It is, however, necessary to keep in mind that this data is not finalized and will most likely change as it continues to update.

In this article, we will show you how you can extract fresh Google Analytics 4 data to your warehouse or dashboard with the help of a data integration tool (in this case, Dataddo).

• Dashboard with Most Recent Data
  • Historical Data Source
  • Fresh Data Source
  • Add Both Sources to the Same Flow
• Store Fresh Data in Your Data Warehouse
• Conclusion: Why the Dataddo Connector?

Dashboard with Most Recent Data

To get fresh data to your dashboard, we will need to create two sources, one for historical data and one for fresh data. The second source will be a running window of fresh data which will keep updating with the most recent values which might not be the final version.

Historical Data Source

First, set up your sources. After you’ve created your first source, you can simply clone it to make sure both have identical metrics and attributes selected.

In your list of Sources, select the source for historical data. Make sure the snapshot keeping policy is Append.

Next, go to the Advanced Settings tab and change the Date_range_expression to 4d4 (or older).

In short, this source will be adding the most recently finalized data to your table every day.

Fresh Data Source

In your list of Sources, select the source for fresh data. For this source, we want the snapshotting policy to be Replace.

In the Advanced Settings tab, change the date range to 3d1 (data up until yesterday) or 3d0 (including today’s data).

This source will be the running window with your fresh data. Every day, the last three days’ worth of data will be replaced.

Add Both Sources to the Same Flow

Go to the Flows page and click on Create Flow to add a source and destination. As both sources have the same metrics and attributes, both can be added to the same flow and sent to your dashboarding tool.

Store Fresh Data in Your Data Warehouse

When setting up your source, choose your Date Range to be Three days ago.

You can also edit the date range by going to Sources > three dots at your source > Edit > Advanced Tab > Date_range_expression: 3d0.

Authorize and create your Destination by following the prompts provided by Dataddo. Next, head to Flows. Add your newly created source and destination.

If your warehouse allows deduplication, choose Upsert as your write mode. Upsert, however, is limited to 4 unique columns. 

In case you need more than 4 unique columns, you can instead deduplicate in your own storage using dataddo_extraction_timestamp.

Conclusion

Although we used Dataddo for this specific problem, the most important part is the logic behind the process. Of course, manually setting this up should also be possible but quite honestly not advisable. Due to multiple days’ worth of data changing very frequently, you would have to keep extracting, adding, and replacing data daily. This could easily lead to mistakes that are not noticed until much later, thus compromising your data integrity.

Moreover, Dataddo’s free tier lets you connect up to 3 data flows to dashboards. So, there is pretty much no need for you to set this up by yourself.

No matter what approach you choose, all that’s left now is for you to start working with your data insights. Just don’t forget that you’re working with data that might still be subject to change!