Guest post: Proving that an application is as broken as intended

Posted Jun 25th, 2015

Typically you want to use end-to-end (e2e) tests to prove that everything works as intended in a realistic environment. In the Juice Shop application that idea is changed to the contrary. Here the main purpose of the e2e test suite is to prove that the application is as broken as intended!

Juice Shop: Broken beyond hope - but on purpose!

"WTF?" you might ask, and rightfully so. Juice Shop is a special kind of application. It is an intentionally insecure Javascript web application designed to be used during security trainings, classes, workshops or awareness demos. It contains over 25 vulnerabilities that an aspiring hacker can exploit in order to fulfill challenges that are tracked on a scoreboard. The job of the e2e test suite is twofold:
  1. It ensures that the overall functionality (e.g., logging in, placing products in the basket, submitting an order, etc.) of the application is working. This is the above mentioned typical use case for e2e tests.
  2. It performs attacks on the application that should solve all the existing challenges. This includes SQL Injections, Cross-Site Scripting attacks, business logic error exploits and many more.
When does Juice Shop pass its e2e test suite? When it is working fine for the average nice user and all challenges are solvable, so an attacker can get a 100% on the scoreboard! Juice Shop logo

Application Architecture

Juice Shop is created entirely in Javascript, with a Single-Page-Application frontend (using AngularJS with Bootstrap) and a RESTful backend using Express on top of NodeJS. The underlying database is a simple file-based SQLite with Sequelize as a OR-mapper and sequelize-restful to generate the simple (but not necessarily secure) parts of the API dynamically.

Test Stages

There three different types of of tests to make sure Juice Shop is not released in an unintendedly broken state:
  1. Unit tests make sure that the frontend services and controllers work how they should. The AngularJS services/controller are tested with Karma and Jasmine.
  2. API tests verify the RESTful backend is behaving properly when running as a real server. These tests are done with Karma and frisby.js for orchestrating the API calls.
  3. The e2e test suite performs typical use cases and all kinds of attacks via browser-automation using Protractor and Jasmine.
If all stages pass and the application survives a quick monkey-test by yours truly it will be released on GitHub and SourceForge.

Why Sauce Labs?

There are two reasons to run Juice Shop tests on Sauce Labs:
  1. Seeing the front-end unit tests pass on a laptop already gives a good feeling for an upcoming release. But there they run only on PhantomJS, so not in a real browser. Seeing them pass on various browsers increases confidence in the release.
  2. The e2e tests must be executed before shipping a release. Wanting to make sure they are not skipped due to laziness or overconfidence ("Oh' it's such a small fix, what could it possibly break?" - sound familiar?) the e2e suite must be integrated into the CI pipeline.
Having laid out the context the rest of the article will explain how both these goals could be achieved by integrating with Sauce Labs.

Execution via Travis-CI

Juice Shop builds on Travis-CI which Sauce Labs integrates nicely with out of the box. The following snippet from the .travis.yml shows the necessary configuration and the two commands being called to excecute unit and e2e tests.
  sauce_connect: true
- karma start karma.conf-ci.js
- node test/e2eTests.js
  - secure: <your encrypted SAUCE_USERNAME>
  - secure: <your encrypted SAUCE_ACCESS_KEY>

Frontend Unit Tests

The karma.conf-ci.js contains the configuration for the frontend unit tests. Juice Shop uses six different OS/Browser configurations:
var customLaunchers = {
    sl_chrome: {
        base: 'SauceLabs',
        browserName: 'chrome',
        platform : 'Linux',
        version: '37'
    sl_firefox: {
        base: 'SauceLabs',
        browserName: 'firefox',
        platform: 'Linux',
        version: '33'
    sl_ie_11: {
        base: 'SauceLabs',
        browserName: 'internet explorer',
        platform: 'Windows 8.1',
        version: '11'
    sl_ie_10: {
        base: 'SauceLabs',
        browserName: 'internet explorer',
        platform: 'Windows 8',
        version: '10'
    sl_ie_9: {
        base: 'SauceLabs',
        browserName: 'internet explorer',
        platform: 'Windows 7',
        version: '9'
    sl_safari: {
        base: 'SauceLabs',
        browserName: 'safari',
        platform: 'OS X 10.9',
        version: '7'
  In order associate the test executions with the Travis-CI build that triggered them, some extra configuration is necessary:
    sauceLabs: {
        testName: 'Juice-Shop Unit Tests (Karma)',
        username: process.env.SAUCE_USERNAME,
        accessKey: process.env.SAUCE_ACCESS_KEY,
        connectOptions: {
            tunnelIdentifier: process.env.TRAVIS_JOB_NUMBER,
            port: 4446
        build: process.env.TRAVIS_BUILD_NUMBER,
        tags: [process.env.TRAVIS_BRANCH, process.env.TRAVIS_BUILD_NUMBER, 'unit'],
        recordScreenshots: false
    reporters: ['dots', 'saucelabs']
  Thanks to the existing karma-sauce-launcher module the tests are executed and their result is reported back to Sauce Labs out of the box. Nice. The e2e suite was a tougher nut to crack.

End-to-end Tests

For the Protractor e2e tests there are no separate configuration files for local and CI, just one protractor.conf.js with some extra settings then running on Travis-CI to pass necessary data to Sauce Labs:
if (process.env.TRAVIS_BUILD_NUMBER) {
    exports.config.seleniumAddress = 'http://localhost:4445/wd/hub';
    exports.config.capabilities = {
        'name': 'Juice-Shop e2e Tests (Protractor)',
        'browserName': 'chrome',
        'platform': 'Windows 7',
        'screen-resolution': '1920x1200',
        'username': process.env.SAUCE_USERNAME,
        'accessKey': process.env.SAUCE_ACCESS_KEY,
        'tunnel-identifier': process.env.TRAVIS_JOB_NUMBER,
        'build': process.env.TRAVIS_BUILD_NUMBER,
        'tags': [process.env.TRAVIS_BRANCH, process.env.TRAVIS_BUILD_NUMBER, 'e2e']
  The e2e tests are launched via e2eTests.js which spawns a separate process for Protractor after launching the Juice Shop server:
var spawn = require('win-spawn'),
    SauceLabs = require('saucelabs'),
    colors = require('colors/safe'),
    server = require('./../server.js');

server.start({ port: 3000 }, function () {
    var protractor = spawn('protractor', [ 'protractor.conf.js' ]);

    function logToConsole(data) {

    protractor.stdout.on('data', logToConsole);
    protractor.stderr.on('data', logToConsole);

    protractor.on('exit', function (exitCode) {
        console.log('Protractor exited with code ' + exitCode + ' (' + (exitCode === 0 ?'SUCCESS') :'FAILED')) + ')');
        if (process.env.TRAVIS_BUILD_NUMBER && process.env.SAUCE_USERNAME && process.env.SAUCE_ACCESS_KEY) {
        } else {
  The interesting part regarding Sauce Labs is the call to setSaucelabJobResult(exitCode) in case the test is run on Travis-CI with Sauce Labs credentials which are passed in by the extra config part in protractor.conf.js. This function passes the test result from Protractor on to Sauce Lab's REST API:
function setSaucelabJobResult(exitCode) {
    var sauceLabs = new SauceLabs({ username: process.env.SAUCE_USERNAME, password: process.env.SAUCE_ACCESS_KEY });
    sauceLabs.getJobs(function (err, jobs) {
        for (var j in jobs) {
            if (jobs.hasOwnProperty(j)) {
                sauceLabs.showJob(jobs[j].id, function (err, job) {
                    var tags = job.tags;
                    if (tags.indexOf(process.env.TRAVIS_BUILD_NUMBER) > -1 && tags.indexOf('e2e') > -1) {
                        sauceLabs.updateJob(, { passed : exitCode === 0 }, function(err, res) {
                            console.log('Marked job ' + + ' for build #' + process.env.TRAVIS_BUILD_NUMBER + ' as ' + (exitCode === 0 ?'PASSED') :'FAILED')) + '.');
  This was necessary because there was no launcher available at the time that would do this out-of-the-box.

Determining solved Challenges

How does Protractor get its test result in the first place? It must be able to determine if all challenges were solved on the score board and cannot access the database directly to do that. But: It can access the score board in the application: Screenshot score board As solved challenges are highlighted green instead of red some simple generic function was used to assert this:
protractor.expect = {
    challengeSolved: function (context) {
        describe("(shared)", function () {

            beforeEach(function () {

            it("challenge '" + context.challenge + "' should be solved on score board", function () {
                expect(element( + '.solved')).getAttribute('class')).not.toMatch('ng-hide');
                expect(element( + '.notSolved')).getAttribute('class')).toMatch('ng-hide');

  When watching the e2e suite run Protractor will constantly visit the score board to check each challenge. This is quite interesting to watch as the progress bar on top moves closer to 100% with every test. But be warned: If you plan on trying to hack away on Juice Shop to solve all the challenges yourself, you will find the following screencast to be quite a spoiler! ;-)

Bjoern Kimminich is responsible for IT architecture and application security at Kuehne + Nagel and as a side job is giving lectures on Software Engineering at the private university Nordakademie. When not working on his ownJuice Shop, Bjoern thinks up Code Katas and regularly speaks at conferences and meetups on topics like application security and software craftsmanship. Twitter: @bkimminich

Written by

Björn Kimminich