Running Cypress Tests on AWS CodeBuild

Cypress is a relatively new web testing tool that is easier to use than Selenium, and it’s gaining in popularity. But using it in a continuous integration environment like AWS CodeBuild requires some additional steps compared to running it directly on your own computer.

This blog post contains helpful information to configure CodeBuild on AWS to run Cypress.

Requirements for running Cypress on CodeBuild

  1. An AWS Account with permissions to create S3 Buckets, run CloudFormation, and manage AWS CodeBuild jobs.
  2. A working local Cypress testing project

Things not covered here:

  • How to reset the environment before testing
  • Security via Cognito or other security systems
  • Additional security such as VPC configurations

Proper permissions for running these samples

  • AWSCodeBuildAdminAccess
  • AmazonS3FullAccess
  • AWSCloudFormationFullAccess
  • CloudWatchLogsFullAccess

Be careful to work with an AWS account with credits. Whatever you create and whatever bills you generate are your responsibility.

A sample application

Here’s the beginning of the CloudFormation script that sets up the S3 bucket, named based on the stack name you pass when you create it.

AWSTemplateFormatVersion:                           "2010-09-09"
Description: "Self hosted AWS application"

# See example at https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/quickref-s3.html
# Create an S3 bucket and using it to host a website written in Angular
Resources:

S3Bucket:
Type: AWS::S3::Bucket
Properties:
# Key setting: without this, the bucket cannot be accessed via an URL
AccessControl: PublicRead
# Give the bucket a predictable name based on the stack name passed
# when creating the CloudFormation stack
BucketName: !Sub "${AWS::StackName}-hosting-bucket"
WebsiteConfiguration:
IndexDocument: index.html
ErrorDocument: error.html
DeletionPolicy: Retain

In order to expose the bucket for web access, you need to assign a bucket policy:

BucketPolicy:
Type: AWS::S3::BucketPolicy
Properties:
PolicyDocument:
Id: MyPolicy
Version: 2012-10-17
Statement:
- Sid: PublicReadForGetBucketObjects
Effect: Allow
Principal: '*'
Action: 's3:GetObject'
Resource: !Sub "arn:aws:s3:::${S3Bucket}/*"
Bucket: !Ref S3Bucket

This policy allows all principals (everyone) to get objects from the bucket, allowing read access to the objects inside of it. Remember to remove this stack when you’re done with this example or you’ll pay for additional usage.

Next, you need to expose the URL and bucket name:

Outputs:
WebsiteURL:
Value: !GetAtt "S3Bucket.WebsiteURL"
Description: URL for website hosted on S3
S3BucketName:
Value: !Ref S3Bucket

A simple shell script installs this stack (you provide a stack name as the argument):

#!/bin/bash

# This script deploys the Angular app to an S3 bucket for hosting

aws cloudformation deploy
--stack-name ${1} \
--capabilities CAPABILITY_IAM --template-file \
./angular-app/cloudformation.yml \
|| { echo 'cloudformation deploy failed'; exit -1; }
WEBSITE_URL=`aws cloudformation describe-stacks \
--stack-name ${1} --output text \
--query "Stacks[*].Outputs[?OutputKey=='WebsiteURL'].OutputValue"`
BUCKET_NAME=`aws cloudformation describe-stacks \
--stack-name ${1} --output text \
--query "Stacks[*].Outputs[?OutputKey=='S3BucketName'].OutputValue"`
pushd ./angular-app
npm install || { echo 'npm install failed'; exit -2; }
npm run build || { echo 'build failed.'; exit -3; }
aws s3 cp dist/angular-app/ \
s3://${BUCKET_NAME} \
--recursive || { echo 'cp failed'; exit -4; }
echo "The website is now available at ${WEBSITE_URL}"
echo "The S3 bucket name is ${BUCKET_NAME}"
popd

Now your webapp can be deployed to a stack with ./setup-angular-stack.sh my-stack.

What are we deploying, anyway?

The Component

import { Component, OnInit } from '@angular/core';
import { ListItem } from '../support/list-item';
import { FormBuilder, FormGroup } from '@angular/forms';
import { ItemType } from '../support/item-type.enum';

@Component({
selector: 'app-shopping-list',
templateUrl: './shopping-list.component.html',
styleUrls: ['./shopping-list.component.scss']
})
export class ShoppingListComponent implements OnInit {
shoppingList: Array<ListItem> = [];
shoppingItemForm: FormGroup;
itemTypes = [ItemType.BAKERY, ItemType.DRINK, ItemType.FOOD, ItemType.PHARMACY,
ItemType.SALAD_BAR, ItemType.SNACKS, ItemType.TOILETRIES];
constructor(private builder: FormBuilder) { }

ngOnInit(): void {
this.shoppingItemForm = this.builder.group({
'purchased': [false, [], []],
'name': ['', [], []],
'category': ['', [], []]
}, []);
}

addItem() {
const formData = this.shoppingItemForm.value as ListItem;
this.shoppingList.push(formData);
this.shoppingItemForm.reset({purchased: false});
}
}

The component’s view

<h3>Add an Item</h3>
<form [formGroup]=shoppingItemForm
(ngSubmit)="addItem()">
<div class="form-group">
<label for="name">Item Name</label>
<input id="name" class="form-control"
type="text"
formControlName="name">
</div>
<div class="form-group">
<label for="category">Category</label>
<select id="category" class="form-control" formControlName="category">
<option [value]="itemTypes[0]">{{ itemTypes[0] }}</option>
<option [value]="itemTypes[1]">{{ itemTypes[1] }}</option>
<option [value]="itemTypes[2]">{{ itemTypes[2] }}</option>
<option [value]="itemTypes[3]">{{ itemTypes[3] }}</option>
<option [value]="itemTypes[4]">{{ itemTypes[4] }}</option>
<option [value]="itemTypes[5]">{{ itemTypes[5] }}</option>
</select>
</div>
<div class="button-group">
<button class="btn-large btn-primary">Add...</button>
</div>
</form>
<div *ngIf="shoppingList && shoppingList.length > 0">
<h3>Shopping List</h3>
<table class="table table-bordered table-striped">
<thead>
<th>X</th>
<th>Name</th>
<th>Category</th>
</thead>
<tbody>
<tr *ngFor="let item of shoppingList">
<td><input type="checkbox"></td>
<td>{{ item.name }} </td>
<td>{{ item.category }}</td>
</tr>
</tbody>
</table>
</div>

Testing with Cypress

A simple test against the app above would look like this:

describe('Adding a list item', () => {
beforeEach(() => {
// the visit is based on a "baseUrl" config
// setting that establishes the root
// website address
cy.visit('/');
});
it('adds an item', () => {
cy.get('input#name')
.type('Soap Flakes')
.should('have.value', 'Soap Flakes')

cy.contains('Add...').click();

cy.get('table').should('contain', 'Soap Flakes');

cy.get('input#name')
.should('have.value', '');
});
});

Cypress has a relatively easy to manage configuration system. Normally, with local development, the cypress.json configuration file can point to your local web server (so you can iterate on the application and re-run tests as you change the code).

For that purpose, the local repository points to the local Angular stack:

{
"baseUrl": "http://localhost:4200",
"viewportWidth": 1024,
"viewportHieght": 800
...
}

To test the app locally, you fire up the app in one terminal, and Cypress in another.

Creating the Cypress AWS CodeBuild project via CloudFormation

It is that method that we are going to use to run Cypress in an AWS CodeBuild project.

The CloudFormation configuration for the CodeBuild project has several major parts. It begins with two required parameters, GitHubProjectUrl and CypressBaseUrl, to fetch the project source code and point to the right website for testing:

AWSTemplateFormatVersion:               "2010-09-09"
Description: "Cypress Test"

Parameters:
GitHubProjectUrl:
Description: "The GitHub Project URL"
Type: "String"

CypressBaseUrl:
Description: "The base URL for the application under test"
Type: "String"

You need to provision an S3 Bucket to hold artifacts/reports (by default, it will
be a generated S3 bucket, so this gives you more control over where the assets will live):

Resources:

CypressReportBucket:
Type: AWS::S3::Bucket
Properties:
BucketName: !Sub "${AWS::StackName}-cypress-reports"

Next, you need to configure an IAM Role to run CodeBuild projects. This role needs to have access to the newly created reporting bucket, have access to the EC2 AMI registry to launch the CodeBuild VM, write to CloudWatch logs, and have the ability to read CloudFormation stack settings to get its configuration.

CypressRole:
Type: "AWS::IAM::Role"
Properties:
RoleName: !Sub "${AWS::StackName}-CypressRole"
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: "Allow"
Principal:
Service:
- "codebuild.amazonaws.com"
Action:
- "sts:AssumeRole"
ManagedPolicyArns:
- "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryPowerUser"
- "arn:aws:iam::aws:policy/AWSCloudFormationReadOnlyAccess"
Policies:
- # for writing to CloudWatch
PolicyName: CodeBuildLoggingPolicy
PolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: "Allow"
Action:
- "logs:CreateLogGroup"
- "logs:CreateLogStream"
- "logs:PutLogEvents"
Resource: "*"
- # For uploading artifacts
PolicyName: CodeBuildS3ArtifactBucketAccess
PolicyDocument:
Version: "2012-10-17"
Statement:
- # Static Hosting Bucket upload...
Effect: "Allow"
Action:
- "s3:*"
Resource:
- !Sub "arn:aws:s3:::${CypressReportBucket}"
- !Sub "arn:aws:s3:::${CypressReportBucket}/*"

Next, you define the Cypress runner as an AWS CodeBuild Build Project. We’ll break this down piece by piece, as it is a long configuration.

First, the overall runner settings:

CypressRunner:
Type: AWS::CodeBuild::Project
Properties:
# Define the codebuild project with the stack name
Name: !Sub "${AWS::StackName}-cypress-ui"
Description: Test the application via Cypress
# Tie in the codebuild role
ServiceRole: !GetAtt CypressRole.Arn
# set a reasonable time here
TimeoutInMinutes: 5
# for now, we're storing the report as an artifact
# the actual CodeBuild reports are trickier to configure
# and perhaps are a story for another day
Artifacts:
Type: S3
NamespaceType: BUILD_ID
Name: "reports.zip"
# Refer to the S3 bucket we just defined
Location: !Ref CypressReportBucket
Packaging: ZIP
# Remove this if you want to avoid storing build logs in CloudWatch
LogsConfig:
CloudWatchLogs:
GroupName: !Sub "/codebuild/${AWS::StackName}-cypress-logs"
Status: ENABLED
StreamName: "ci-log"

Continue with defining the environment of the CodeBuild job. This passes the
input S3 bucket hosting URL so you can override it in the Cypress build execution.

Environment:
Type: LINUX_CONTAINER
PrivilegedMode: true
ComputeType: BUILD_GENERAL1_SMALL
Image: aws/codebuild/amazonlinux2-x86_64-standard:3.0
EnvironmentVariables:
- Name: CYPRESS_BASE_URL
Value: !Ref CypressBaseUrl

Now you have to configure a source code repository to download the test artifacts.

In this example, we’ll use GitHub as our source, passed as the CloudFormation Stack input parameterGitHubProjectUrl above. Once you configure AWS to see your GitHub projects, you can execute this CodeBuild project job which will fetch the source from GitHub:

Source:
Auth:
Type: OAUTH
Location: !Ref GitHubProjectUrl
Type: GITHUB
GitCloneDepth: 1

Cypress Execution Options in CodeBuild

Finally, let’s review the build itself. This is a CodeBuild BuildSpec file, embedded in the CloudFormation definition. It will be uploaded to an S3 bucket backing the CodeBuild code.

The key challenges of this build were resolved with some good ole’ Stack Overflowment:

  • running Chromium instead of Chrome (as it is provided as part of the CodeBuild amazon-linux instance),
  • running Chromium headless (without a graphics card) to avoid installing a desktop environment,
  • properly generating the reports (we used Mochawesome as the report generator) and uploading them as build artifacts (not 100% the CodeBuild way, but the CodeBuild reporting engine didn’t work with our reports and this is good enough for our purposes),
  • and passing the baseUrl override into the Cypress launch command.
BuildSpec: |
# important: 0.2 enables some very useful features
version: 0.2
phases:
install:
runtime-versions:
nodejs: 12
# Pre-build - set up the Cypress runtime
pre_build:
commands:
- cd cypress-tests
- npm install
build:
commands:
- export CONFIG="baseUrl=$CYPRESS_BASE_URL"
- echo $CONFIG
- # NO_COLOR=1 disables ANSI special chars so you can read the output
- # Also: --headless is essential here. We MUST run headless
- # in order to avoid installing an X11 desktop and virtual X graphics
- # context...
- NO_COLOR=1 ./node_modules/.bin/cypress run \
--browser chromium \
--headless \
--config "$CONFIG"
finally:
- # You can grab this from the Cypress docs on running Cypress reports
- npx mochawesome-merge "cypress/reports/separate-reports/*.json" > mochawesome.json
- # yes, "marge"
- npx marge mochawesome.json
- # this one is stupid but somehow I can't get codebuild to copy multiple top level paths
- # so I just moved the report dir into place
- mv mochawesome-report cypress/
# these are available in the Build Details tab as the reports.zip archive
artifacts:
files:
- 'reports/**/*'
- 'screenshots/**/*'
- 'videos/**/*'
- 'mochawesome-report/**/*'
base-directory: 'cypress-tests/cypress'

My advice for those who dare: expect to tweak the options until you get it right. Try running the build script locally with the CYPRESS_BASE_URL set to http://localhost:4200 until it works, then deploy it to CodeBuild.

CodeBuild can be tricky, so we’ve put this project up on a GitHub repo for you to try. You can get all of the code above at https://github.com/chariotsolutions/aws-codebuild-cypress-example.

This post was originally written for the Chariot Solutions Blog by Ken Rimple, Chariot’s Director of Training & Mentoring. You can read the original post here.

Starting your next software project? Our software experts are here to help. Read more about our service offerings at Chariot Solutions, or contact us for a quote.

Chariot Solutions is a top IT consulting firm specializing in software and mobile development, and development in the cloud. Visit us at chariotsolutions.com.