Installation on AWS

From Ocean Framework Documentation Wiki
Jump to: navigation, search

Setting up Ocean on AWS is easy using the provided CloudFormation templates.

At the moment you can set up Ocean in these AWS regions:

  • us-east-1, North Virginia
  • us-east-2, Ohio
  • us-west-2, Oregon
  • eu-west-1, Ireland

We are no longer using Chef and TeamCity. Instead we're using the various SaaS DevOps offerings of the AWS platform (CodePipeline, CodeCommit, CodeBuild, CodeDeploy). The above four regions are the only AWS regions which support all four above DevOps tools.

If you modify the pipeline sources after creation (using the AWS console) to fetch from GitHub rather than from CodeCommit the following three regions are also available:

  • eu-central-1, Frankfurt
  • ap-southeast-1, Singapore
  • ap-southeast-2, Sydney

A quick spin of Ocean is well within the limits of the AWS Free Tier, if you keep to a Minimal default system. The only extra costs are the load balancers and DynamoDB costs, which in a Minimal system are very conservative. Running an Ocean Minimal system costs about $0.17 per hour using the costliest instance reservation class (On Demand).


Installation

Preparations

  • Optional: Set up a local S3 bucket to contain the contents of this repo. To evaluate Ocean, you may want to skip this step, but for real use you will want local copies of all scripts and templates.
  • Clone the Ocean source repos from GitHub to CodeCommit. Keep the names exactly as they are.

Helper Stacks

The auto subnet management facility lets you allocate subnets within a VPC without ever having to bother about collisions. We use a Lambda function and a small DynamoDB table, which will cost you about $5 a month.

  • Create the autosubnet/autosubnet.yaml stack. Name it AutoSubnet.

Ocean creates all DB and user passwords for you. To install the Lambda function that allows this:

  • Create the random.yaml stack. Name it Random.

Domain

  • Set up a domain on AWS Route 53.
  • Request a SSL certificate for your domain using AWS Certificate Manager. Take note of the ARN.

Ocean base

  • Create the ocean.yaml stack. Name it Ocean. This will create the Ocean VPC and export vital common data used by other CloudFormation templates. (If you created a local S3 bucket to hold the Ocean CloudFormation templates, enter its URL, otherwise accept the default.)

The Ocean AMI

Create the ami.yaml stack. Name it AMI. This configures an EC2 Ubuntu 16.04 LTS instance for Ocean's use, then shuts it down and creates an AMI from it. This AMI will be used as a base for all Ocean instances. It has the following software pre-installed:

  • cfn-init & friends
  • AWS CLI
  • AWS CodeDeploy Agent
  • CloudWatch Monitoring Scripts (mon-put-instance-data.pl and mon-get-instance-stats.p) configured to send 8 memory and EBS metrics every minute
  • Ruby 2.3.1p112 (ruby-full)
  • the MySQL client
  • ZeroMQ
  • Bundler

This allows near instant Rails installations, which is important for testing and deployment.

The Ocean Core Services

Explore the pipelines set up by Ocean, check out the log aggregation, the alarms, run the Ocean Tool... Check out the programming paradigm and make a few CloudFormation stack updates to change your Ocean system dynamically. Enjoy!

Bastion Server

To SSH into or make HTTP requests to instances in private subnets you need a Bastion server. You can create one when needed using the bastion.yaml template. It can be deleted when not in use.

Stack naming

You can of course name the stacks anything you like. The above names will allow you to use the defaults provided in the template forms, which is what you want if you install a single Ocean system.

A single Ocean system may have any number of environments within its VPC. Environments can be created and deleted dynamically, e.g. for automated testing without any human input. Each Ocean system has room for hundreds of environments, each with full network separation. There is no need to enter subnet ranges anywhere, as the AutoSubnet custom resource handles this automatically.

Types of deployment

The core.yaml template lets you control how many environments to deploy and the size of each environment.

To evaluate Ocean, simply accept the defaults. This will create a one-environment dev system running on a minimal set of 9 t2.micros. There will be no DNS failover in a Minimal environment, but auto-healing, full Varnish caching and pipelines are always included.

The Dev environment might be able to continue using a Minimal (t2.micro) or a Small (t2.small) size, even for professional development.

You will want a Medium or Large Prod environment as this will give you high-availability failover.

You get an AWS CodePipeline for each service.

Each pipeline uses CodeBuild for testing and artefact builds, something which is very important as a new test/build agent is created for each separate run - and instances which CodeBuild creates are paid for by the build minute, not per hour. The rate for such an instance starts at $0.005 per minute, and the first 100 build minutes per month are free. This means unbeatably low CI costs.

Deployment uses AWS CodeDeploy, which means you can switch between rolling in-place deployments and Blue/Green deployment for an environment simply by running a CloudFormation update.

Ocean pipeline.png

Continuous Deployment vs Continuous Delivery

Do you want Continuous Deployment? That's the ideal. With a Continuous Deployment pipeline, there are no manual steps to be performed at any time: test coverage is so thorough that there is implicit trust in the pipeline and the process. With Continuous Deployment, you don't need a staging environment (especially not in a microservice architecture where each service is deployed separately at different points in time - there are no releases of the whole system). Feature toggling can be used to switch on features being tested in production, or to perform A/B testing.

Ocean has a whole service dedicated to supporting feature toggling.

  • If you create only a Dev and a Prod environment but no Staging environment, pipelines will be set up for Continuous Deployment.

If you for some reason can't trust your tests, or must have manual QA for some stages or services, then what you're using is not Continuous Deployment, but Continuous Delivery. In this scenario, create all three environments: Dev, Staging, and Prod.

  • If you create all three environments, pipelines will be set up for Continuous Delivery: there will be manual approval required to deploy to Staging and to Prod.

In any case, this is a good read (basically, it says that Staging environments are evil):

Multiple Ocean systems

You can also create multiple Ocean systems in the same account. If you do, you will need a separate domain for each full Ocean system. Redo all above steps but skip the Ocean AMI, Auto Subnet, and Random sections.