Service Anomaly Detected (app.emarketeer.com)

Incident Report for eMarketeer

Postmortem

Overview
At around lunch of October 11st, we suffered a downtime due to a runaway process on our primary webhosting machine. This slowness caused roughly 100% of requests generated during this time to be completed out of SLA.
What Happened
one of the services running on the machine crashed
Resolution
restarted the service and replaced malicious code that was causing an issue
Root Causes
internal process consumed too much memory
Impact
all customers were affected by this

Posted Oct 11, 2019 - 14:51 CEST

Resolved

This incident has been resolved.

Posted Oct 11, 2019 - 12:48 CEST

Investigating

Monitoring systems have detected an anomaly with eMarketeer application. The Site Reliability Engineering team has been notified and is currently working to find to root cause. We will update this message when the anomaly has been identified. Automated monitoring systems will update affected components and will resolve operational status as systems recover.

Posted Oct 11, 2019 - 12:42 CEST