We have completed the final updates to the data (updating revenue) and the restore is now complete. We all apologize for the inconvenience this has caused and the length of time it took for the restore to be completed. We have more than doubled our volume this year and due to that growth things took longer than expected.
Posted Dec 18, 2014 - 14:55 EST
Update
Data for 12/16 has been completely re-loaded and verified. All Impressions, Click, and Conversion data is complete. Revenue numbers are being modified based on clicks and conversions and will be complete within the next 60 minutes.
Posted Dec 18, 2014 - 11:45 EST
Update
The upload of 12/16 is almost complete - once this is done all Impression and Click data will be completely accurate. We still need to process Clicks and Conversions for revenue data only which will be done by 1:00 EST.
Posted Dec 18, 2014 - 11:25 EST
Update
The reporting pipeline is almost caught up - it will be back to the normal 30 minute delay in the next couple of hours. We have re-processed all of 12/16 logs and will be re-loading that day once the pipeline is caught up. We anticipate this to be complete by the latest of noon tomorrow.
Posted Dec 17, 2014 - 23:23 EST
Update
Restore of data continues but we believe there has been some duplication of data and to be on the safe side we will be re-processing all of yesterdays data. As today's data starts to enter the system you can be sure that it is accurate - but please don't use data from yesterday until we give you the go ahead.
Posted Dec 17, 2014 - 07:06 EST
Update
Restore of the DB has completed and everything looks good - keyword forecasting/reporting is back online. The pipeline will be restarted shortly - we anticipate it to take most of the night for it to catchup and then we will run a verification step to ensure there were no issues. We will need to load keyword data from the raw longs which will take 2-3 days to complete.
Posted Dec 16, 2014 - 16:53 EST
Update
Things continue to restore - it is taking longer than anticipated. Once the restore is complete keyword reporting and forecasting will be restored and then reporting will begin to catch up.
Posted Dec 16, 2014 - 14:39 EST
Identified
We have found the root cause of the issue and are taking steps to resolve it - it will be 2-3 hours before reporting restarts and then 4-5 hours for it to catch-up for the day. Currently reporting and forecasting will work except for cases where a keyword is included, we are rebuilding the keyword database so that data will be unavailable for the next 2-3 hours.
Posted Dec 16, 2014 - 10:46 EST
Investigating
There is an issue in the reporting pipeline that we are investigating that is causing numbers to be delayed and possible accuracy issues for today's data only.