IDBS Blog | 9th March 2017
Amazon Web Services (AWS) Outage: An Update from Our SRE Team
As you may have seen in the news last week, AWS experienced some issues with the infrastructure it uses for some of its IaaS (Infrastructure as a Service) platform. Whilst IDBS was only minimally affected by this outage, we thought it would be good to provide you with an update on what we have learnt about this outage and the steps we are taking to improve our communication to our SaaS customers.
At around 18:50 GMT on Tuesday 28th Feb 2017, IDBS’ Site Reliability Engineering (SRE) team received email and text message alerts from New Relic (our application performance monitoring system) via VictorOps (our on-call management, incident notification and live infrastructure timeline system) about increased error rates with E-WorkBook Connect. The alert indicated more than 5% of requests were failing. The alert was acknowledged and SRE started to investigate. It was discovered AWS was experiencing an S3 outage resulting in some requests failing.
E-WorkBook Connect was the only product in our SaaS offering that was affected. The rest of The E-WorkBook Cloud suite reported no errors, with monitoring systems reporting all systems were performing as expected.
The SRE team escalated internally to our helpdesk, management and Global Professional Services (GPS) team. The GPS team provided an additional check of all systems in US-EAST-1 (the affected AWS region) and confirmed performance was as expected.
For the rest of the evening, SRE monitored our systems as the issue escalated within AWS and additional services started to fail. There were no further application alerts during this period and E-WorkBook performance remained static. E-WorkBook Connect also reported recovery and the alert was marked as resolved.
Our SRE team closed the incident at 06:30 GMT on Wednesday 1st March as AWS reported all systems as healthy and recovered. Since this time, IDBS have run an internal session to discuss what we have learnt from the AWS outage in US-EAST-1 and have refined and improved a number of our internal processes and communication protocols that we will be rolling out over the coming months. This will enable us to improve our service and the communication our customers receive should anything like this occur in the future.
The IDBS E-WorkBook Cloud was able to ensure our SLA was maintained despite a few challenges with one of our infrastructure partners.
If you’d like to read more about the AWS outage, then please visit the link below.
More news
Ensuring product, process and patient data integrity and traceability throughout the BioPharmaceutical lifecycle in the era of personalized medicine
The BioPharma industry is facing a multitude of challenging external trends, from the growth in personalized medicine to supply chain disruptions and more.
European Biomanufacturing Summit 2023
Come and join us at booth 130 in Amsterdam, to learn how an innovative tech transfer approach can revolutionize drug manufacturing in the pharmaceutical industry.
IDBS events roundup: Q1 highlights and Q2 events
IDBS associates had the opportunity to attend some events in Q1 and attendees were eager to see, hear and learn more about what’s happening in the industry.
How to improve time-to-value using GxP computer systems
Explore the benefits of GxP computer systems to improve time-to-value so that companies can speed up the development and manufacture of life-saving drugs.