Fast Debugging with Javascript Console Output

Sauce Labs developers Jonathan Lipps and Jeremy Avnet take a stroll through the recently released Sauce Labs feature — inline Javascript console log output.

One of the painful parts of dealing with Selenium tests is debugging Javascript issues. Sometimes we’ll go to click on an element or verify some text exists only to discover it is missing. Did a Javascript syntax error prevent part of the page from loading? Did an AJAX request fail? Finding the cause of these issues often requires manually performing each step in a browser running developer tools. We watch the console log hoping to catch untoward Javascript behavior.

Now, when you run tests on Chrome [1. Platforms will be added as we receive feedback.], Sauce will automatically show you Javascript console messages, network events, errors, and warnings. Let’s run through a quick example to see how these messages make it easier to diagnose issues just by looking at the test result on Sauce.

We created a test app, THE NAMINATOR!. It’s a silly app that adds -inator to the end of names. Here’s a screenshot of what it looks like:

The functionality is simple: type in a list of space-separated words and get back a list of “naminatorized” words. It’s powered by an AJAX call to a backend Flask app. Here’s the logic for responding to the AJAX request:

    @app.route("/naminatorize", methods=['GET', 'POST'])
def naminatorize():
    naminatorized = []

    text = request.form.get('text') or request.args.get('text')
    for name in (text or "").split(" "):
        if not name:
            continue

        suffix = "nator"
        if name.endswith('e'):
            name = name[:-1] + 'i'
        if not name.endswith1):
            suffix = 'i' + suffix

        naminatorized.append(name + suffix)
    return render_json(json.dumps(naminatorized))

And the Javascript which triggers the AJAX request when someone hits the “Naminatorize!” button:

$(function() {
    console.log("Page loaded!");
    $('#doItNau').click(window.doTheNaminating);
});

window.doTheNaminating = function() {
    console.log("Doing the naminating");
    $.ajax({
        url: '/naminatorize'
        , type: "POST"
        , data: {text: $('#names').val()}
        , success: function(res) {
            $('#naminated').html('');
            $('#naminated').css('font-size', '30px');
            $('#naminated').css('color', '#111');
            $.each(res, function(i, name) {
                var div = $('
‘ + name + ‘
');
                $('#naminated').append(div);
            })
        }
    });
}

Basically — query a remote server, expect a JSON array back, and iterate through the array values, displaying them on the page. It looks like this:

With awesome app power comes awesome selenium testing. Here’s the interesting part of our functional test:

    def test_naminator(self):
    self.driver.get('http://localhost:5050/')
    self.assertIn("naminator", self.driver.title.lower())

    # enter the names
    self.driver.find_element_by_id("names").click()
    self.driver.find_element_by_id("names").clear()
    self.driver.find_element_by_id("names").send_keys("sauce jquery selenium")

    # namiratorize!
    self.driver.find_element_by_id("doItNau").click()

    # get/wait for namiratorizations
    naminatorized = self.driver.find_element_by_tag_name("html").text
    for _ in xrange(20):
        if not "Naminatorized stuff" in naminatorized:
            break

        time.sleep(0.25)
        naminatorized = self.driver.find_element_by_tag_name("html").text
    else:
        assert False, "-ators never showed up"

    self.assertIn("saucinator", naminatorized)
    self.assertIn("jquerynator", naminatorized)
    self.assertIn("seleniuminator", naminatorized)

    self._passed = True

We can run this test file using Python on Sauce. If you’re interested, check out the test result page so you can see the Selenium commands executed successfully and the job marked “Passed”. In a nutshell, the test sends some text to the input box, clicks the button, and polls to see if the expected result got added to the page.

So far, so good, but what would happen if the “naminatorizing” web service we rely on changed? What if the results were returned in a different format? To illustrate, let’s change the Python web service code and act ignorant of the breaking changes. When we run our Naminator test on Sauce again:

$ python app_test.py
Test is running at https://saucelabs.com/jobs/6a5cd939131e432d8d40ed3eece4a18e
F
======================================================================
FAIL: test_naminator (__main__.Selenium2OnSauce)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "app_test.py", line 79, in test_naminator
    self.assertIn("saucinator", naminatorized)
AssertionError: 'saucinator' not found in u'The NAMINATOR\nA simple app that adds -inator to anything. Inator. See?\nHo

Looking at the test output, we see the text saucinator was not found in the body of the page. Interesting! If we navigate to the test page on Sauce, we see clearly why the test failed:

Something weird happened in the Javascript. But … what? We’re still getting a response from the web service. We see it’s getting turned into string representations of objects instead of “naminated” text. Luckily, Sauce captured the Javascript console output from the browser during the test run, so we can dig a little deeper. On the test page, we see this:

console message

And if we click “view data” on the network request, we can see the data that was actually returned when we clicked the “Naminatorize” button in the test:

Now we can see that instead of the array we expected, we got back this JSON:

{
  "version": "2.0", 
  "result": [
    {"saucinator": true}, 
    {"jquerynator": true}, 
    {"seleniuminator": false}
  ]
}

The little Naminating web service is growing up! Whoever maintains it *wink* decided to give the API a version and return naminated text as an array of hashes (they now convey information in addition to the names themselves). The Javascript we wrote to consume the API needs to be rewritten in order to handle the new format.

With example data already provided by the Sauce test page, we can code a fix now! The best thing is we can immediately apply a fix and we didn’t even have to load the webpage in our browser to figure how things were breaking or what we needed to do to fix the problem. All we had to do was look at the test run on Sauce and find the console output to examine what we were getting back from the server.[3. It should be obvious that in addition to applying the quick fix, you’d want to get the full v2 spec. Asking the author to support version numbers in the request wouldn’t hurt either. :-)]

So, run your Sauce Labs tests on Chrome (any platform) and you’ll start seeing these types of console output in your test results immediately. As always, please give us feedback on how you find this update and the ways it’s made your testing less painful and more fun.

Happy Testing!

PS If you’re interested in the changes to the web service API, check out v2 of the naminatorize function.

Apart from adding the version, we are also including whether we did anything linguistically clever in order to add -inator to the end of a word, like handling vowels and such.

Written by

The Sauce Labs Team

Topics

Programming languagesSelenium