At 15:38 on February 14, we suffered a little more than 40 minutes downtime due to a non scheduled database job on our primary database machine. This affected all parts of the system.
All system was back to normal at 16:20.
Due to human error a database upgrade was scheduled outside a maintenance window.
Human error working in the database interfaces. We have implemented a two step approach for this kind of work to prevent this to happen in the future.
The unscheduled upgrade job takes more than 5 hours to complete, and during this time stays offline. To bring the system up faster we restored the system from the backup made before the upgraded started.
_All users and all functions was affected during the incident. During the downtime there was a small window (approx 20 mins) where user could login and work in the system. Any saved work during this time is lost. This should affected very few customers.
_