Posts Tagged ‘optimization’

Announcing Support for Internet Explorer 9 and Firefox 4 in Sauce OnDemand

March 15th, 2011 by Santiago Suarez Ordoñez

Here at Sauce Labs, we’re committed to keeping Sauce OnDemand, our browsers-in-the-cloud service, up to date with every new browser release. That’s why we’re especially thrilled to bring you fully supported IE9 and Firefox 4.0 browsers in the cloud, ready now for everyone – even free accounts – to use.  Also, at the end, for the do-it-yourself Selenium users, we share some of our “secret sauce” for installing these browsers.

Watch them running

Firefox 4 in the coud (see the full job in OnDemand):

Internet Explorer 9 (full job in OnDemand):

Use them yourself!

All you have to do is to create an account if you don’t have one yet, then start your Selenium RC tests using the right browser version and OS in the special Sauce browser string.
Of course you can find the browser string to use in our supported browsers page, but to save you some time, these are the ones you’re looking for:

"os": "Windows 2008", "browser": "firefox", "browser-version": "4"
"os": "Windows 2008", "browser": "iexplore", "browser-version": "9"
Notice: IE9 only runs in Windows Vista or later, so we’ve created new Windows 2008* VMs just for this! This means, though, that it’s important to provide “Windows 2008″ as the OS version requested.

This is how we did it

We founded Sauce Labs in part because we believe most people working on software testing have higher value uses for their time than maintaining test infrastructure.  And also because we believe in the power of open source to make a positive contribution.  So for those Selenium users for whom using Sauce OnDemand is not yet an option, we want to give back to the community and share the following How To section so those users can more easily install both FF4 and IE9 in their own test lab.

The following outlines how to configure and install both browsers.

Setting up Firefox 4

firefox 4 logo
To make Firefox 4 ready for Selenium testing we must ensure it is visible to Selenium.  Here’s how. Choose the Custom option during installation and make sure firefox is installed in C:\Program Files\Mozilla Firefox.

Selenium will find Firefox once it is installed in that location. No further configuration needs to be done, since Selenium takes care of the rest using its own custom profile.

Setting up Internet Explorer 9

ie9 logo

Internet Explorer 9 requires more work to set up. We’ve seen users having issues bringing up the browser as well as performing basic driving actions.

With the accumulated lessons learned from 4 million Selenium tests and counting, our VMs contain a series of registry edits that make Internet Explorer work like a charm with Selenium, as well as run tests faster.

We have made Sauce registry edits available in the their own public gist.  Please only follow these instructions if you are complete comfortable with the process and terms.  To run them just download the files and run the following from the command line:

From the administrator account in your test machine, enter:

regedit /s admin_regedits.reg

As the user running Selenium (may be the administrator user too):

regedit /s user_regedits.reg

Warning: The configurations resulting from these registry edits make should only be used to run in dedicated testing servers. Using them on a machine intended for everyday usage, such as checking email and working, will be a security risk.

If you run into problems with this setup, feel free to ask questions in our Sauce OnDemand forums or our Selenium forums (depending on whether you’re using our cloud service or setting this up locally). We do our best to answer any questions that come through. If you’re looking for faster and personalized responses, we do offer paid Selenium support packages, which give you access to our Selenium experts via phone, email, and chat.

* Windows Server 2008 RC1 is Microsoft’s server OS most closely related to Windows Vista (they have the same NT kernel version number).

Share

It Takes Time To Go Fast

October 12th, 2010 by Jason Huggins

Since I started my career in software development, I’ve witnessed many changes in how we release software.


My first job was as a technical consultant at a large ERP software vendor customizing their Student Administration package for a large state university. I worked on several things* on that project, and at one point, I was in charge of change management from the Dev to Test system environments. This was the process:

* On a weekly basis, email all the developers on the team, asking them for their finished code projects.
* Create a spreadsheet listing the complete set of projects to be migrated.
* Get various approvals from the team leads for the final migration list.
* Manually migrate each project from the Dev database to the Test database.
* Finally, let the testers know they’ve got new stuff to test.

Waterfall can best be described as a “walled-garden” methodology. Each function — development, test, and operations — mostly worked in silos. Interactions between teams only happened occasionally — weeks at best, months at worst. And even then, only a person with people skills, like me, handled the interaction between the teams.

This was the peak of waterfall software development. A complete code cycle from initial development to production only happened a few times a year. (“Once per quarter” was a pretty aggressive release goal for ERP projects in those days.) No one thought this was a “bad” process. Tedious, yes, but not terrible.


A few years later, I found myself working for a hip, boutique software consultancy in Chicago. I became the technical lead on an in-house project to replace the company’s time and expense system. This company was really into XP and Agile, so we “ate our own dog food,” and ran the project The Agile Way, the same way we ran our client projects. We had weekly iterations, and pushed to production about once a month. Initial project kick-off to first production users was about 3 months.

The big innovation I noticed with Agile was the distinct feeling that development no longer lived in a “walled garden” with regards to testing. The wall that traditionally existed between development and testing no longer existed. In fact, every developer wrote tests. We even wrote cross browser testing tools to make the task easier for the development team. And we used continuous integration to make sure our tests were run all the time.

We were good-to-above-average for a typical agile project. We only pushed to production once a month, but we were proud of ourselves. During this whole time, no one thought this was a bad process. In fact, general consensus was that it was good, bordering on great — especially compared to the Waterfall horror stories everyone knew of.


More recently, I co-founded a startup focused on solving software testing problems. As a small startup with a new Amazon EC2 account, cloud computing provided us direct and fast access for deploying code to production whenever we wanted.

With no more long cycles waiting for IT to procure machines, and no lengthy approval processes for dealing with production DBAs, I could hear another wall collapsing. We had obliterated the line between dev/test and operations. Some people call this “DevOps.” In my opinion, it should be called “DevTestOps,” because no sane person would push to production without running tests first!

At first, even though we now had little process friction to go from dev to test to production, we still did things the Agile Way. We coded features at a weekly or biweekly scale, and pushed to production once a week or every two weeks. This was still an improvement over my first Agile project, though.

With the walls down between development, test, and operations, we’ve started to optimize for speed. As we strive to improve our process, it helps to see pioneers show the way.

We now release at a much faster pace than a typical Agile project. Our internal goal is to *always* push to production at least once a day. We often do better than that. The faster you go, the more motivated you are to go even faster.

One benefit of fast release cycles is that it frees up time to tear down more walls in the development process. We have production metrics to watch user behavior in production — what they do and what they click on — that can inform what we work on next week. And we even have time to talk to the sales team for input on what to work on. Wherever we can, we destroy friction in our development process.

We’ve noticed that as our release cycle speeds up, so does our reliance on fast feedback systems. Our tests and our continuous integration system is the heartbeat of development. If the build is green, we deploy. If not, we have to wait. When you get used to pushing new code to production several times a day, it’s really painful to go back to the old way of doing it. After every process improvement, I look back at the old way we used to do things, and wonder what took us so long to get here.

Footnote:
* My first assignment was writing the program for the Admissions department that had a simple workflow. Input: GPA and ACT/SAT score. Output: a “personalized” (mail-merged) acceptance or rejection letter. I still remember thinking how a bug in my code would alter the course of someone’s life. If you didn’t get into Boise State University around 1999-2000, sorry about that!

Share
Posted in Featured Posts, General

Continuous Try Server Integration

December 24th, 2009 by John Dunham

Collin Jackson of Betable talks about their successful use of “Continuous Try Server Integration” using Selenium together with Sauce OnDemand cloud-hosted Selenium service.

Share
Posted in Selenium Knowledge

Running your Selenium tests in parallel: Python

September 2nd, 2009 by Santiago Suarez Ordoñez

This is the first post in our series “Running your Selenium tests in parallel”, in which we’re going to explain how to set up a concurrent execution environment and considerably reduce your testing times.

The first client language we’re going to address, as the title says, is Python. To start, let’s get a set of Selenium Python tests to use:

The tests are stored in a public github project. You can see the code there or even download them in a zip file: Python_tests.zip (28KB)

This set of tests validates our site. It checks basic structure, our login form, our signup form, and our feedback tab (UserVoice powered). The tests are grouped into Python files based on the functionality they address.

Note: These tests are written to run against SauceRC, our own service, which takes care of all the concurrency work on the RC server side and launches multiple concurrent browsers.

If you run these tests in your local environment, you should not send concurrent jobs to a single Selenium RC server. More than 2 tests at the same time will consistently affect performance such that any reduction in test time will not be significant.

Another alternative would be to use Selenium Grid and manage a group of test servers yourself. SauceRC offers a suite of useful extras and eliminates the maintenance headaches of the DIY approach.

First approach: One by one execution

So,  the regular way to run these would be to run each file from the command line, like so:

$ python testIntegrity.py
$ python testLogin.py
$ python testSignup.py
$ python testFeedback.py

As a first approach it’s not that bad; your tests will run, and you’ll get decent output with the results.
Execution time: 16.2 minutes

16.2 minutes isn’t that bad. The problem is that once your test base starts growing, the time and effort it will take you run these tests will increase considerably. We’re talking about hours in some cases.

The next step to take will be to write a small script that runs them automatically, so the only thing the user will have to do is to run a single Python script that takes care of the rest:

$ python ordered_run.py

The code in this script is:

import os
import glob

tests = glob.glob('test*.py')
for test in tests:
    os.system('python %s' % test)

Easier to run, but not much faster…
Execution time: 16 minutes

Second approach: A process per each test set

Let’s improve this using subprocess, one of Python’s multiple process libraries.

from subprocess import Popen
import glob

tests = glob.glob('test*.py')
processes = []
for test in tests:
    processes.append(Popen('python %s' % test, shell=True))

for process in processes:
    process.wait()

Now our tests run concurrently, using a separate process per python file (4 processes total).
Execution time: 8.3 minutes

This is much faster! It reduces the whole execution time from the time it takes to run all the tests one after another to the time it takes the longest of the four sets of tests to end.

Note: One drawback to this method is that the output we receive isn’t in any particular order. You can change that by setting the stdout parameter in the Popen instantiation and then concatenating the output in order.

The definitive solution: A process per test

We can further reduce the execution time by running all 14 individual test methods in parallel. The easiest way we’ve found to do that in Python is to use nose.

First, install it:

$ easy_install nose==0.11 multiprocessing

Now, place yourself where your tests are, run nose and enjoy:

$ nosetests --processes=14

Much easier to run (no helper script for this), cleaner output…
Execution time?
3 minutes!

Do you have a better way to run your tests concurrently? Tell us about it in the comments.

Share
Posted in Selenium Knowledge