Backup and restore
Learn about Code Ocean VPC backup and restore capabilities.
Overview
Code Ocean VPC stores persistent data that needs to be backed up in the following storage components:
EBS data volume
S3 buckets
RDS analytics DB
The system comes with a default backup plan that includes: EBS & RDS daily snapshots using AWS Backup, S3 bucket versioning, and a 14 day retention period. The retention period is reflected in the number of days EBS & RDS snapshots are kept in the AWS Backup vault and in the number of days non-current (including deleted) S3 object versions are kept in S3 buckets that store persistent data.
You can configure the backup frequency (cron expression) to meet RPO (restore point objective) requirements, and the retention period (days) through the CloudFormation template parameters.
Disaster recovery
As part of a disaster recovery plan, you can protect production data on your Code Ocean VPC further by configuring automated backups to a different AWS account and/or region.
We provide a separate Code Ocean VPC CloudFormation backup stack to receive EBS & RDS snapshots and S3 replication from a production Code Ocean VPC deployment. The backup stack supports backup of a single Code Ocean VPC from VPC version 2.19.
The latest CloudFormation backup stack template version is available here: https://codeocean-vpc.s3.amazonaws.com/templates/codeocean-backup/v1.2.0/codeocean-backup.template.yaml
Note that newer versions of Code Ocean VPC can require upgrades to their backup stack to support new backup features. The Code Ocean VPC release notes will include the minimal required version of their backup stack.
Backup configuration
Prerequisites
To use cross-account backup, you must first enable the Cross-account backup feature in the master/management account in your AWS organization.
Log in using your AWS Organizations management account credentials. Cross-account backup can only be enabled or disabled using these credentials.
Open the AWS Backup console at https://console.aws.amazon.com/backup.
In My account, choose Settings.
For Cross-account backup, click Turn-on.
More information can be found in the AWS documentation.
Configuration steps
Deploy the Code Ocean VPC CloudFormation backup stack in your backup destination AWS account and region.
Specify the Code Ocean VPC source AWS account ID.
You can configure the Backup Retention Period parameter as needed. This will reflect in the number of days non-current (including deleted) S3 object versions are kept in S3 backup buckets.
If you are setting up cross-account and cross-region backup, then deploy a second intermediate Code Ocean VPC CloudFormation backup stack in your source AWS account but in the destination region. See diagram above.
Specify the Code Ocean VPC source AWS account ID.
Specify the Destination Backup Vault ARN. This is the backup vault in the destination account and region you deployed in step 1 above.
Update the source Code Ocean VPC CloudFormation stack with the following Backup Configuration parameters:
Destination Backup Vault ARN (e.g.
arn:aws:backup:us-east-1:0123456789:backup-vault:destination-stack-v1
). This would normally be the backup vault ARN in your destination account/region, but if you are setting up cross-account and cross-region backup, then this would be the backup vault of the intermediate Code Ocean VPC CloudFormation backup stack in your source account in the destination region.Destination Backup S3 KMS Key. This would be the AWS managed S3 KMS key alias ARN in the destination AWS account, eg
arn:aws:kms:us-east-1:0123456789:alias/aws/s3
.Destination Backup S3 Storage Class:
GLACIER_IR
. S3 Glacier Instant Retrieval delivers the fastest access to cheaper archive storage, which will allow you to restore a Code Ocean VPC without waiting to restore all S3 objects in the S3 backup buckets.Destination Backup S3 Bucket ARNs (Capsules, Datasets, etc.)
Wait for the stack update to complete.
Replicate existing S3 objects from source to destination:
Start an SSM session on the source deployment services instance
Generate and copy the S3 batch replication commands:
Start an AWS Console Cloud Shell, paste the commands into the shell and execute.
You can track the progress of the S3 batch replication jobs in AWS S3 console under Batch Operations. Note that S3 reports status Failed for batch jobs that try to replicate empty S3 buckets with no objects in them. These errors can be safely ignored.
Automatic backup
After completing the above backup setup, Code Ocean VPC will automatically copy EBS & RDS snapshots, as they are created, to the destination backup stack, and continuously replicate S3 data into the destination S3 backup buckets in the backup stack.
Manual backup
The following steps describe the process to generate on-demand snapshots of the system’s EBS data volume and the RDS analytics DB. This will be required during a system migration process to take a final snapshot of a current system. It is also recommended to generate a backup before upgrading the system to allow roll back to a previous state.
EBS data volume
Using the AWS console, navigate to AWS Backup > My account > Protected resources and click on Create on-demand backup.
Fill in the following fields:
Resource type:
EBS
Volume ID:
[Code Ocean deployment EBS volume ID]
Total retention period:
1 day
(or more as required)Backup vault:
[Code Ocean deployment backup vault]
IAM role:
[Code Ocean deployment BackupRole]
Click on Create on-demand backup.
RDS analytics DB
Using the AWS console, navigate toAWS Backup > My account > Protected resources and click on Create on-demand backup.
Fill in the following fields:
Resource type:
RDS
Database name:
[Code Ocean deployment RDS analytics DB name]
Total retention period:
1 day
(or more as required)Backup vault:
[Code Ocean deployment backup vault]
IAM role:
[Code Ocean deployment BackupRole]
Click on Create on-demand backup.
If the Code Ocean VPC is configured with backup to a destination backup vault, the above generated snapshots will be automatically copied into the destination backup vault.
Restore from backup
During disaster recovery, you can restore your Code Ocean VPC production deployment by provisioning another instance of Code Ocean VPC and performing a restore from the backup stack.
Deploy a new instance of Code Ocean VPC into the new account/region. Make sure to use the same version as the source Code Ocean VPC deployment you are restoring from.
Specify the backup AWS account ID under the Restore Configuration > Restore source account ID CloudFormation stack parameter.
Copy any other stack configuration parameter from the source deployment as needed.
Wait for the stack creation to complete.
SSM into the Code Ocean VPC services instance in the new deployment and stop the
codeocean
systemd service:This will prevent the system from performing any cleanup maintenance jobs while S3 data is being replicated into it, which will result in deleting replicated S3 data.
Update the Code Ocean VPC CloudFormation backup stack in your backup account with the following Restore Configuration parameters:
Destination Restore S3 KMS Key. This would be the AWS managed S3 KMS key alias ARN in the new deployment AWS account and region (e.g.
arn:aws:kms:us-east-1:0123456789:alias/aws/s3
).Destination Restore S3 Bucket ARNs (Capsules, Datasets, etc.). These would be the S3 bucket ARNs in the new deployment.
Wait for the stack update to complete.
Replicate S3 data from backup to the new deployment:
Start an SSM session on the new deployment's services instance
Generate and copy the S3 replication commands:
Fill in all required command line arguments, including the Backup Account ID, the IAM S3 Restore Role ARN, and the various bucket ARNs, all of which you can find the backup CloudFormation stack Resources list. The generated AWS CLI commands you copied will now be executed in the backup account.
Start an AWS Console Cloud Shell in the backup account, paste the commands into the shell and execute
You can track the progress of the S3 batch replication jobs in the backup account AWS S3 console under Batch Operations. Note that S3 reports status Failed for batch jobs that try to replicate empty S3 buckets with no objects in them. These errors can be safely ignored.
Select the EBS & RDS snapshots to restore:
In the backup account, open the AWS Backup console > Backup vaults > click the backup stack vault name.
Under Recovery points click on the matching EBS & RDS snapshots you'd like to restore and copy their ARNs.
Copy EBS & RDS snapshots from backup to the new deployment:
If you are performing a cross-account and cross-region restore:
First, deploy a second intermediate Code Ocean VPC CloudFormation backup stack in your new account in the same region as your main backup stack. See the diagram above.
Specify the backup AWS account ID under Code Ocean VPC source AWS account ID.
Start an SSM session on the new deployment's services instance.
Generate and copy the snapshot copy commands:
Fill in all required command line arguments:
The EBS & RDS snapshot ARNs.
The source backup vault name from the backup account.
The IAM role ARN. Specify the AWS Backup default service role in the backup account, e.g.
arn:aws:iam::0123456789:role/service-role/AWSBackupDefaultServiceRole
.
Change the
--destination-backup-vault-arn
argument to the backup vault ARN of the intermediate backup stack.Start an AWS Console Cloud Shell in the backup account, paste the commands into the shell and execute.
Wait for the snapshots copy to complete. The snapshots should appear in the intermediate backup stack backup vault under Recovery points. You can track the progress of the snapshot copy jobs in the backup account AWS Backup console under My account > Jobs.
Generate and copy a second pair of snapshot copy commands:
Fill in all required command line arguments:
The EBS & RDS snapshot ARNs.
The source backup vault name from the intermediate backup stack.
The IAM role ARN. Specify the AWS Backup default service role in the new deployment account, e.g.
arn:aws:iam::0123456789:role/service-role/AWSBackupDefaultServiceRole
.
Start an AWS Console Cloud Shell in the new deployment account in the intermediate region, paste the commands into the shell and execute.
Wait for the snapshots copy to complete. The snapshots should appear in the new deployment backup vault under Recovery points. You can track the progress of the snapshot copy jobs in the AWS Backup console under My account > Jobs.
If you are performing only a cross-account restore:
Start an SSM session on the new deployment's services instance.
Generate and copy the snapshot copy commands:
Fill in all required command line arguments:
The EBS & RDS snapshot ARNs.
The source backup vault name from the backup account.
The IAM role ARN. Specify the AWS Backup default service role in the backup account, e.g.
arn:aws:iam::0123456789:role/service-role/AWSBackupDefaultServiceRole
.
Start an AWS Console Cloud Shell in the backup account, paste the commands into the shell and execute.
Wait for the snapshots copy to complete. The snapshots should appear in the new deployment backup vault under Recovery points. You can track the progress of the snapshot copy jobs in the backup account AWS Backup console under My account > Jobs.
Restore a temporary EBS volume from the EBS snapshot:
In the AWS console of the new deployment navigate to AWS Backup > Backup vaults
Click on the backup vault name
Under the Recovery points section, click on the EBS snapshot to restore
Copy aside the Creation time of the EBS snapshot
Click on Restore
Fill in the following fields:
Volume type:
gp3
Size:
[match the size of the existing data volume]
Availability zone:
[choose AZ]
Throughput:
[match the throughput of the existing data volume]
Encryption: change KMS key to use the default
aws/ebs
key (copy the key ID from the AWS KMS console > AWS managed keys >aws/ebs
)
Click on Restore backup
Wait until the restore completes
Navigate to AWS Backup > My account > Jobs > Restore jobs
Copy aside the EBS Volume ID from the EBS restore job
Restore a temporary RDS DB instance from the RDS snapshot:
Navigate back to the AWS backup vault Recovery points and click on the RDS snapshot to restore
Click on Restore
Fill in the following fields:
instance type:
db.t3.micro
storage class:
gp3
Availability and durability:
Do not create a standby instance
DB Instance Identifier:
tmp-codeocean-restore
VPC:
[choose the Code Ocean deployment VPC]
DB parameter group:
[choose the Code Ocean DB parameter group]
IAM DB Authentication Enabled:
Disable
Click on Restore backup
Wait until the restore completes
Find and copy aside the current RDS analytics DB master password in AWS Console > Secrets Manager > Secrets >
/[codeocean-stack-name]/analytics/master-password-0
> Retrieve secret valueNavigate to the AWS RDS console > Databases and click on the
tmp-codeocean-restore
DBClick Modify
Update the master password using the master password you copied above
Update the security group to the current code ocean deployment analytics DB security group
Click Continue
Select Apply immediately
Click Modify DB instance
Wait until the update completes
Copy aside the endpoints for both the current RDS analytics DB and the restored RDS analytics DB
Start an SSM session on the new deployment's services instance and run the following commands:
Copy in data from the restored EBS volume:
Attach the restored volume to the services instance:
Override the data volume content with the snapshot content:
Copy in data from the restored RDS DB:
Restore the S3 buckets to the exact snapshot time:
Restore the system secrets:
Start codeocean service:
Run migration to set state of all datasets to “not-cached”:
Delete the restored EBS volume:
Unmount the volume
In the AWS console, navigate to EC2 > Elastic Block Store > Volumes > Filter by volume id and check the desirable volume
Click on Actions > Detach volume
Click on Delete
Delete the restored RDS DB:
Navigate to the AWS RDS console > Databases and click on the
tmp-codeocean-restore
DBClick on Actions > Delete
Uncheck Create final snapshot and Retain automated backups, acknowledge and click Delete
Migration
Migration of a Code Ocean VPC deployment to a new AWS account or region happens in two phases:
Setup, where you create a new Code Ocean VPC deployment in the destination AWS account or region, and copy in the bulk of data from the source deployment. This phase happens in the background while the source deployment is still up and running.
Switch, where you stop work on source deployment, take last snapshots of EBS & RDS, copy in remaining data, and finally perform the DNS switch.
Phase 1 - Copy Bulk of Data
Deploy a new instance of Code Ocean VPC into the destination account/region. Make sure to use the same version as the source Code Ocean VPC deployment.
Specify the source AWS account ID under the Restore Configuration > Restore source account ID CloudFormation stack parameter.
Copy any other stack configuration parameter from the source deployment as needed.
If you opt into ACM certificate validation via DNS, it's recommended to first set the new CloudFormation stack on a different domain than the source Code Ocean VPC deployment. Later, when switching DNS in phase 2 of the migration you can update the DNS configuration as needed.
Wait for the stack creation to complete.
SSM into the Code Ocean VPC services instance in the new deployment and stop the
codeocean
systemd service:This will prevent the system from performing any cleanup maintenance jobs while S3 data is being replicated into it, which will result in deleting replicated S3 data.
If you are performing a cross-account and cross-region migration, then deploy an intermediate Code Ocean VPC CloudFormation backup stack in your source AWS account but in the destination region. See diagram above.
Specify the Code Ocean VPC source AWS account ID.
Specify the Destination Backup Vault ARN. This is the backup vault in the destination account and region you deployed in step 1 above.
Update the source Code Ocean VPC CloudFormation stack with the following Backup Configuration parameters:
Destination Backup Vault ARN (e.g.
arn:aws:backup:us-east-1:0123456789:backup-vault:destination-stack-v1
). This would normally be the backup vault ARN in your destination deployment, but if you are preforming cross-account and cross-region migration, then this would be the backup vault of the intermediate Code Ocean VPC CloudFormation backup stack in your source account in the destination region.Destination Backup S3 KMS Key. This would be the AWS managed S3 KMS key alias ARN in the destination AWS account (e.g.
arn:aws:kms:us-east-1:0123456789:alias/aws/s3
).Destination Backup S3 Storage Class:
STANDARD
(notGLACIER_IR
).Destination Backup S3 Bucket ARNs (Capsules, Datasets, etc.). These would be the S3 bucket ARNs in the destination deployment.
Wait for the stack update to complete.
Replicate S3 existing objects from source to destination:
Start an SSM session on the source deployment's services instance.
Generate and copy the S3 replication commands:
Start an AWS Console Cloud Shell in the source account, paste the commands into the shell and execute.
You can track the progress of the S3 batch replication jobs in the source account AWS S3 console under Batch Operations. Note that S3 reports status Failed for batch jobs that try to replicate empty S3 buckets with no objects in them. These errors can be safely ignored.
Create an on-demand backup of the source deployment EBS data volume as described here.
Wait for the snapshot to be automatically copied into the destination deployment’s backup vault. The snapshot should appear in the destination deployment backup vault under Recovery points.
Copy in data from the EBS data volume snapshot to the destination deployment EBS data volume by executing the following steps from the restore section: 9, 11, 12, 18.
Phase 2 - Remaining Data, Switch
Stop work and bring down the source Code Ocean VPC deployment as described here.
Update the source deployment CloudFormation stack to remove S3 replication rules:
Remove the following Backup Configuration stack parameters, and update the stack
Destination Backup S3 KMS Key
Destination Backup S3 Bucket ARNs (Capsules, Datasets, etc.)
Wait for the stack update to complete
Stop the
codeocean
systemd service on the source deployment services instance:Create on-demand backups of the source deployment EBS data volume and RDS DB as described here.
The snapshots will be automatically copied into the destination deployment’s backup vault.
Copy in data from the EBS data volume and RDS DB snapshots to the destination deployment by executing steps 9-19 from the restore section. Skip step 14 as there is no need to restore S3 buckets to previous state.
Update the destination deployment CloudFormation stack to remove restore configuration:
Remove the Restore Configuration → Restore source account ID CloudFormation stack parameter and update the stack
Wait for the stack update to complete
Perform any required DNS switch
Last updated