Alarms

Learn alarm features.

Predefined Alarms

Code Ocean natively reports metrics to CloudWatch. These include system metrics as disk space, CPU and memory usage as well as application metrics. As part of the deployment, alarms are also provisioned

Important Alarms

Alarm	Description	Threshold
services-unhealthy-host	The services machine is not responding to health checks	Whenever this happens
services-data-volume-usage-70/-90	The services machine disk space is getting low	Disk used is greater than 70%/90%
critical-error	Critical errors returned by CodeOcean services	Whenever this happens
services-cpu-usage-high	Services machine CPU usage is high	CPU Utilization is grater then 70% for 6 times during 30 minutes

How to Address Alarms

• services-unhealthy-host - We need to check the status of the machine:

  • First, check if the machine is running on the AWS ec2 console.

  • Connect to the machine using SSH and run sudo systemctl restart codeocean

• services-data-volume-usage - This alarm indicates low disk space. We need to increase the EBS backed volume to address this issue:

  • First we need to increase the volume size

  • Now we need to extend the file system on the services machine using the commend sudo xfs_growfs -d /data see here

• critical-errors - In case of a critical error, you will need to collect relevant error log lines and contact the Code Ocean support team.

• services-cpu-usage-high - Contact Code Ocean if the issue persists

Subscribe to Alarms

You can get notified when alarms happen using AWS SNS. All of the Code Ocean alarms are reported to the alarms-<id> SNS topic, which you can subscribe to.

Email subscription to an SNS topic

1. Open the Amazon SNS console at https://console.aws.amazon.com/sns/v3/home.

2. In the navigation pane, choose Topics. A list of topics should be visible. Find the pre-defined topic with the name alarms-<id> and copy its ARN value.

3. In the navigation pane, switch to Subscriptions, and select Create subscription.

4. In the Create subscription dialog box, for Topic ARN, paste the topic ARN that you copied in the previous step.

5. For Protocol, choose Email.

6. For Endpoint, enter an email address that you can use to receive the notification, and then choose Create subscription.

7. From your email application, open the message from AWS Notifications and confirm your subscription.

   Your web browser displays a confirmation response from Amazon SNS.

See more on how to connect AWS SNS with Slack or Microsoft Teams.