The queue is cleared and everything is operational. We will continue to monitor this service for the rest of the day. We are also patching the service to be more robust against this failure case.
Posted 7 months ago. Nov 28, 2018 - 13:28 EST
Our post-processing reporting service was having intermittent issues writing its output to s3. We believe this was caused by a networking issue with these specific ec2 instances. New instances have been cycled in and are processing messages successfully. Monitoring until queue is cleared.
Posted 7 months ago. Nov 28, 2018 - 12:59 EST
We are continuing to investigate this issue.
Posted 7 months ago. Nov 28, 2018 - 12:42 EST
We are seeing intermittent issues with part of the reporting pipeline which is causing some reports to fail, and others to executing slower than normal. We are investigating the cause.