API requests dropped during upgrade Monday 1st October 2018 17:33:00
We experiences a 3 min outage whilst upgrading some of our infrastructure. New servers were not started quickly enough to cope with the demand during the upgrade and as a result we dropped some requests. Traffic is normal now, and we are monitoring.
We've identified an error in our analytics platform that was causing sporadic errors and not allowing TVs to submit state information. We've disabled analytics whilst we understand the problem. All TVs that have failed to boot should now boot. Please contact support@nevaya.co.uk if you still have an issue
Latency inside of AQWS appears normal now however we have a small percentage of TVs that will not boot. We have replicated this in our lab and are investigating.
AWS now reporting EC2 issues in US-WEST-2
From AWS:
01:12 PM PDT We are investigating increased error rates and latencies for the EC2 APIs in the US-WEST-2 Region.
We are continuing to see high latency and disruption across all services. We have deployed a whole new infrastructure with our TV platform to isolate the cause but the situation remains.
We have seen network latency drop back to normal. Services are currently running normally but we are monitoring.
We are seeing network issues inside eu-west-1 on multiple applications, as a result our app is failing to load. AWS reports there are no issues, however we are seeing multiple reports of AWS issues on twitter and other sources.
