The publication backlog has been cleared, and publication times for all customers have returned to normal.
Ad serving, reporting, and the management UI & APIs were NOT impacted by this incident and are working normally.
Posted Mar 18, 2018 - 22:24 EDT
Update
Publication times for more customers have returned to normal. Some of our largest customers will continue to experience higher-than-normal publication delays, although delays should decrease as the system continues to process its backlog. We are continuing to monitor the publication system and will provide updates on its recovery. The system should recover fully over the next 2-4 hours.
Ad serving, reporting, and the management UI & APIs are NOT impacted. They are working normally
Posted Mar 18, 2018 - 16:42 EDT
Update
Publication times for most customers have returned to normal. For some of our largest customers, publication delays will take longer to recover, but should return to normal over the next 4-6 hours. We are continuing to monitor the publication system and will provide updates on its recovery.
Ad serving, reporting, and the management UI & APIs are NOT impacted. They are working normally
Posted Mar 18, 2018 - 11:31 EDT
Monitoring
Corrective actions have caused the publication system to begin recovering to its normal publication delay of several seconds. Publication times remain high for the moment, so changes via the UI and API may take several minutes to be propagated to the ad servers, but are recovering.
Ad serving, reporting, and the management UI & APIs are NOT impacted. They are working normally
Posted Mar 18, 2018 - 10:38 EDT
Identified
Our publishing queue service, which takes configuration changes from the management UI & API and applies them to the running ad servers, is currently running at a degraded performance. A change made in the UI usually takes effect in a couple of seconds, but right now a change takes upwards of fifteen minutes to take effect. We are taking steps and will advise as performance returns to normal.
Ad serving, reporting, and the management UI & APIs are NOT impacted. They are working normally