This incident has been resolved, you will see a slightly longer delay (2 hours) in reporting tonight but things will catch up over the course of the night. Reporting was only interrupted for about an hour while we identified the issue and redirected it to our fail-over server. The issue was a large number of un-optimized queries being sent to the master server which slowed it down to the point where it couldn't process the reports coming from the UI. After moving the reporting system to the fail-over we added additional indexes to handle the queries to master, then switched back over.
Oct 1, 23:24 EDT
You should see reports reporting as normal now - it looks like the issues were due to a one of our machines being overloaded. We have switched over to the slave and things are working as expected.
Oct 1, 15:47 EDT
We have identified the issue and are working to resolve it. Reports are beginning to return now and will return to normal shortly.
Oct 1, 15:34 EDT
We are investigating reports that reports are not being returned in the UI.
Oct 1, 14:52 EDT