The queue is cleared and everything is operational. We will continue to monitor this service for the rest of the day. We are also patching the service to be more robust against this failure case.
Posted Nov 28, 2018 - 13:28 EST
Our post-processing reporting service was having intermittent issues writing its output to s3. We believe this was caused by a networking issue with these specific ec2 instances. New instances have been cycled in and are processing messages successfully. Monitoring until queue is cleared.
Posted Nov 28, 2018 - 12:59 EST
We are continuing to investigate this issue.
Posted Nov 28, 2018 - 12:42 EST
We are seeing intermittent issues with part of the reporting pipeline which is causing some reports to fail, and others to executing slower than normal. We are investigating the cause.