The queue is now caught up and changes are going through as normal now. We apologize for the issue. We have a ticket in progress to block this from happening again and will be doing db maintenance in the near future to give us more headroom to handle issues like this better.
Posted Jul 20, 2017 - 12:17 EDT
Monitoring
We have determined the cause of the issue. There was unintentional misuse of our system by a customer that caused our database to be overloaded and slow down processing of messages. We have disabled that customer until they can resolve the issue and we have a high priority ticket to block the usage path that caused the issue. Messaging are being processed again and the queue should be caught up in a little over an hour.
Posted Jul 20, 2017 - 11:41 EDT
Identified
We have identified the issue and the UI and API should be back to normal - we are working on bringing back the job processing queue.
Posted Jul 20, 2017 - 10:36 EDT
Update
We are still investigating issue and are also seeing some API and UI instability.
Posted Jul 20, 2017 - 10:21 EDT
Investigating
The publishing queue is backed up due to higher than usual database load. We've turned off the publishing queue temporarily to debug the database issue. This means that changes made in the UI or API in the past few hours (since 5am ET) are slow to get into the ad servers.
Ad serving is working fine. The management UIs and API may be a little slow due to the database load but are working normally otherwise.