Appeasing the Butler or The Commit Heard Round the Company

June 28th, 2011 by joe

Sauce Labs does not employ any artists

 

This is what our build looked like. He was not happy. Notice how his eyes are red. That’s because of failing tests, which as we know turn builds red. These failing tests weren’t normal failing tests, which are actually pretty happy, because they are telling you how to fix your broken stuff. No, these tests were intermittent. That’s largely the fault of the fact that they were selenium tests – go figure, the selenium shop that sells selenium as a service is having selenium issues, lol (but sadface).

Well, to get out of sadfaceland, we dug into one of the sadtests and saw something surprising. It was a case of selenium looking for an element before it appeared. Wait, what? We already know how to fix that. In fact, we already told the whole world how to fix it: with spinning! Basically, the way a spinassert works is by trying the assertion over and over until it passes. This is a big deal with selenium because trying to interact with an element on a page before the page has loaded is a frequent source of flakey tests. We even went so far as to write Test::Right (source on github), a ruby test framework which goes out of its way to make it impossible for you to do write selenium tests without spin asserts. So what happened?

Well, we don’t use Test::Right for all our own tests. Test::Right is ruby and our tests are written in the same language as our back-end; python. And while we were using spin-style asserts for many of our tests, some of them were written before we discovered the magic of spinning, and some were written when we just didn’t remember to use spinning. Also, there were some things that were not assertions but still required spinning; like, if you want to click on an element in a page, it makes sense to spin-wait for that element to be present before clicking on it. Test::Right does the right thing in all these cases; we just had to be more like Test::Right in-house, to prevent sadness from creeping back into the codebase again.

So the fix was clear: modify our own test framework to always spin-wait all the time for every DOM element to show up, every time anyone tried to interact with a DOM element, automatically. We did that, and it made the butler happy. All of a sudden we could trust the build again, and it didn’t take hours to wait for a nonflakey build so that we could deploy new changes. This in turn meant that we never had to hotpatch production when there was an issue, which meant we didn’t have to spend time cleaning up hotpatches when a deploy happened. It also unblocked people who wanted to push new changes to trunk. Basically the whole dev process got better.

So here’s the thing – our customers are having the same issue. Not everyone uses Test::Right. We know this because we have a support team and customers contact us all the time asking for help with flakey tests. In our over 3,000 support requests, race-condition flakey tests are the #1 cause of pain. The shocker is that except for Test::Right (and now our own internal python nose framework), nobody has this universal fix.

So we’re hoping to make test frameworks for all the common languages – PHP, Java, Python, at least – that do what Test::Right did. We already have our own PHPUnit extension (thanks Jan Sorgalla!) which will probably be the first to get the Test::Right makeover.

Sadly, we are a focused team, and we have many tasks to accomplish, so this fix to selenium client libraries will likely not be imminently released by us. Thus we decided to at least tell you all how to Fix Everything on your own side. Now that I’ve done that, back to programming! Good night and good luck.

Share

The Half Dome Syndrome

April 1st, 2011 by John Dunham

Thanks and a hat-tip to Dale Emery for his authentic, anguished tweet triggered by the “zero programming” required benefit cited in our announcement of Sauce Builder yesterday.

In his angst, Dale highlights the need for Sauce Labs to communicate its view on automated testing in general, the role of record and playback capabilities and most specifically about skills correlated with the establishment of successful automated software testing systems.  With this post, we set ourselves about that task.

Let’s start with a metaphor.   Consider visitors to Yosemite’s Half Dome:

Standing in the valley at the base of Half Dome’s legendary vertical wall, they may look up and see exultant climbers at the top,

and think about how wonderful it must be to be up there, with such a clear and far reaching view.  But as the valley visitors stand looking up that sheer face, they feel frustration borne of the impossibility of scaling it.  So they snap a few pictures, buy some souvenirs, then board the bus back home.

Some visitors notice a trail leading off to the side, a non-obvious way to the top.  While certainly not the most direct route, they decide to pursue it in the hopes that somehow, by hook or by crook, they may reach that top so they too can enjoy that clear and far reaching view.  Some of those aspirants may reach the top.  Others may wander about in the woods for extended periods.  They may even lose sight of the fact that reaching the top was the original goal.  Of those who reach the top, they may notice that the face-climbers carry special tools (pitons, ropes, karabiners hexes and so forth) and set about learning how to adopt and use those as a more direct way to ascend.   Of course the vast majority of Half Dome climbers begin exactly this way.

And so it is with automated software testing.

The sheer-face climbers skillfully hand code tests in the language of their application and know the virtues and method of the page object model.

But the significant majority of the world’s 5+ million practicing software test professionals do not.  The well established trend toward the need of programming skills in the world of agile software QA accompanies a fast-rising need for professionals skilled in open source automation tools like Selenium.  Sauce Labs lives in that world.  We don’t want the valley visitors to get back on the bus.  We aim to provide them with both the documentation and the tools to help them begin today to become better climbers, and provide a realistic path.

To date, we have focused our products and our communication pretty heavily toward the experienced sheer-face climbers.  With Sauce Builder, we begin to expand our coverage to address the needs of the valley visitors who aspire to climb.  While we continue to craft tools and systems for the sheer-face crowd, watch this space for a growing body of work on how to climb the just-getting-started path.

So, in hindsight, our Sauce Builder announcement ought have said “zero programming required to get started.”  Thank you again to Dale Emery for drawing our attention, and for creating the opening for us to begin this communication process.  Stay tuned for further expression of Sauce Labs’ view on the practice, process and technology of manual and automated software testing in the new age of agile development and open source tool chains and deployment platforms.

Share

Hey, Selenium. Hurry The Hell Up.

February 11th, 2011 by The Sauce Labs Team

I love automated browser tests, but I *really* love fast builds. The faster the better. So when a build that I’m waiting for starts taking over 5-10 minutes, all I can think to myself is… “Hurry the hell up!” And we all go through it, even here at Sauce. So here are a few things we’ve found to really help speed up Selenium testing:

1.) Paralellism

The more tests you can run in parallel, the faster your build can finish. So when writing your tests, always ask yourself, “Can I run this test by itself and have it pass?” If you think to yourself, “No… another test has to run *before* for this one to pass,” then stop. Put the shovel down, and start climbing your way out of that hole. Tests that rely on previous tests cause random, weird bugs to pop up, are hard to run when you want to reproduce those bugs, and are just slow. So keep your tests independent. It’s how they were meant to be anyway.

When none of your tests are inter-dependent, that means they can all be run in parallel, and in theory, your entire build should only take as long as your single longest test (in practice, it’s a bit longer, but not too much). Which brings us to the next point…

2.) Short Tests

Write your tests to only test a single tiny bit of functionality. Keep the number of steps in your tests under 20. All the parallelism in the world won’t save you from long-running tests, because a single test can’t be parallelized. The best written tests we’ve seen generally run in under 10 seconds. Yeah, I know. That’s short. If you’re wondering how that’s possible, check out step #4 (but be patient, we still have to get through step #3. Don’t rush me.)

A pleasant side-effect of this is that when a build fails, you’ll be able to narrow it down to a very specific area point and fix it all the faster.

3.) Sauce Connect 2, BOOST MODE!

Take a breath. A deep one. Then repeat “Sauce Connect 2, BOOST MODE!” to yourself, slowly, with some real emphasis on the “BOOST MODE”. Nice huh? Even if that’s all it were, it would be worth it, but there’s more. A whole lot more.

Sauce Connect 1 was a python-based tool that could easily and securely connect a staging server behind a firewall to the Sauce OnDemand browser cloud. Raved about by developers, sysadmins, and network admins alike, it was a cornerstone in enabling (very) security-minded orgs to bask in the glory of the Sauce Cloud. Even so, we knew it could go further, so why not push it?

Sauce Connect 2 is java-based (we lovingly call it “Enterprise Edition” internally), and in addition to some handy things like not having to specify each domain by hand anymore, it comes with BOOST MODE. Not ‘boost mode’, BOOST MODE. It looks faster that way.

When launching Sauce Connect 2 with the -b flag, it activates some magic under the hood and causes reduced traffic to your server, and speeds up resource-intensive tests considerably. And though it works for everyone, for our friends in distant lands in particular, you’re really going to love it.

Best part is, it’s just a -b flag. Couldn’t be simpler.

(Note: For the time being, this only works on Firefox and IE, though we have support coming for Safari and Chrome. Also, this isn’t “released” yet, so it’s my gift to you, our loyal reader. Let’s just keep it a secret between us, shall we? Check it out here)

4.) Pre-populate the cookies

This is a really nice trick.

Every browser you get from Sauce Labs is in a clean state. This is great, and you really want this. Even if Fred says you don’t want it, you want it. Tell Fred, “Fred, last night I spent an hour debugging tests, and found out nothing was wrong. You know what the problem was? Someone had installed the fuzzy_warm_critters_asciis browser extension, and it replaced every instance of my javascript with an ASCII cat. I want clean browsers every time.” Do it. Fred will understand.

ASCII cats are so cute!

Who needs javascript when you can have warm fuzzy asciicatz?

But this poses a bit of a problem with shorter builds, because now each browser session has to go through a login process, and possibly a few more steps depending on your application logic, just to get to where you want to test. Previously, I simply recommended abstracting the login process into a separate method so that the tests looked cleaner, but when you’re waiting for the build to finish, that’s not good enough. You want them clean *and* fast. One way to get this is to set the cookie via Selenium after the browser starts up.

Log in to your user via server-side logic (e.g. generate a row in your sessions table, or however you have it setup locally), generate a cookie for that login session, and push it into the browser via Selenium. Bam. You’re done logging in, and your test can continue on as though it had already spent the 5 seconds logging in and the 10 seconds adding items to the shopping cart, all so you can just test the “confirm purchase” button works on your site as expected in IE7.

This isn’t to say that you shouldn’t test out your login form or your “add fuzzy critter to cart” logic via browser tests. Because you should. But you shouldn’t do it for every test. Test it once, test it good, and leave it at that.

This works on anything that relies on state to be built up. Logins, Shopping Carts, Playlists, etc. You get a clean browser for every tests, and your tests are speedy. And you know what? Even Fred’ll love it. And that – in and of itself – is worth it.

Got any tips on how you speed up your Selenium builds? Take us to school and share ‘em. We’re all ears!

Oh, and for Fred’s sake…

|\      _,,,---,,_
 /,`.-'`'    -.  ;-;;,_
 |,4-  ) )-,_..;\ (  `'-'
 '---''(_/--'  `-'\_)
Share

Announcing the Mobile Testing & Development Survey

December 9th, 2010 by Jason Huggins

We are witnessing a shift to mobile software development that feels very similar to the late ’90s shift from desktop to web development. Since we’re currently in the middle of this industry-wide shift, there are still some questions about how developing for mobile will be done by most developers. Specifically, will everyone write Objective-C for iOS, or write Java for Android? Or will the mobile web win, with everyone writing JavaScript, HTML, and CSS and using one of the many mobile web frameworks available today?

To answer these questions about where mobile web development (and consequently, testing) is going, today, Sauce Labs is launching our first Mobile Testing & Development Survey.

At Sauce Labs, we want to provide you the best tools and support for testing your applications, wherever they may live. Currently, most of those applications live on the web and are viewed with desktop web browsers. But the future is not so certain.

Please take a few minutes of your time to give us your opinion on where you think the mobile application development world is going. Sauce Labs will, of course, publish the results. And if you’re one of the first 100 respondents, we’ll send you a bottle of hot sauce. (Yes, really!) We’ll keep the survey open about a month, closing it on January 15, 2011.

Share

The Future of Testing

November 18th, 2010 by The Sauce Labs Team

It’s gratifying to have one’s story mirrored by people like Elisabeth Hendrickson and Chris McMahon, but you’ve got to wonder if we’re all not just drinking the same koolaid. We think there’s more to this than sugar, water and artificial coloring. We think there’s a fundamental change in software development going on.

We welcome your comments after you read this post or add your voice to the habanero chorus by taking our new Selenium Survey. We’ll give you a bottle of Sauce Labs Hot Sauce for your time.

The question we pose to you: What’s the future of testing?

If you are a QA director or programmer who writes tests and follows Sauce Labs, you probably know agile development is exploding. You also most likely agree that Selenium is the lead technology in test automation programming and that automated testing is transforming software development. These ideas should, in our view, be obvious to development teams who are running hundreds or thousands of Selenium scripts in their deployment process, and perhaps with our browsers in the cloud.

This shift doesn’t seem to be an opinion or a hunch. Here’s a bit of evidence that the movement is gaining momentum.

On October 20th, Elisabeth Hendrickson from Quality Tree Software wrote a blog post questioning whether the rise in agile is affecting the QA job market in terms of skills and qualifications. She found it did – and that a trend is emerging.

On October 29th, Chris McMahon wrote another blog post, referencing Elisabeth Hendrickson and our very own Jason Huggins, in which Chris posited that we were seeing a “general across-the-board increase in demand for technical skills in traditional UI-based software testing”.

We couldn’t agree more. Agile development and continuous integration is changing the job landscape for testing professionals, and the technical skills being increasingly demanded is Selenium because the traditional definition of a QA skill-set just doesn’t work if teams want to be agile.

What of “Traditional” QA?

“I have seen a number of reports of a radical increase in the rate of adoption of agile practices among US companies of every size and description,” wrote McMahon. “And the agile whole-team approach to software development makes dedicated, siloed traditional Q&A test departments irrelevant.”

Wow. Do you think that QA departments are irrelevant?

Well, instead of traditional QA departments, we are finding that Quality Directors now know Selenium or Watir, among other technologies. They make sure that their development team uses modern testing practices, and developers write user interface acceptance tests with Selenium for features they have developed. As teams roll out features and fixes, they re-run tests against their applications at blazing speeds, get feedback, make fixes, re-run tests – and soon the new feature is deployed, failure free across all browsers. In other words, the trend we have seen amongst our customers concurs with McMahon.

Automated cross browser testing with Selenium: highly sought?

Growing evidence indicates the answer is yes, Selenium experience is a highly sought after skill.

Elizabeth Hendrickson set out to quantify the demand for programming skills for a small sample of tester job postings. They found that out of the 187 jobs sampled, 112 indicated that programming was required and 39 jobs indicated that programming was desired. In other words, a whopping 80% of test jobs in her sample required or desired programming skills.

Hendrickson’s research then considered test automation technologies. Out of their sample, 27 job ads explicitly said that they require knowledge of test automation tools and an additional 50 ads stated that test automation tool knowledge is a nice to have. We imagine many more actually required automation.

Hendrickson’s study makes a case for Selenium being the leader with the counts of jobs specifying an automation technology:

  • Selenium, including Selenium RC (31)
  • QTP (19)
  • XUnit frameworks such as JUnit, NUnit, TestNG, etc. (14)
  • LoadRunner (11)
  • JMeter (7)
  • Winrunner (7)
  • SilkTest (6)
  • SilkPerformer (4)
  • Visual Studio/TFS (4)
  • Watir or Watin (4)
  • Eggplant (2)
  • Fitnesse (2)

The present (and future) of testing is automated.

We’ve blogged about Selenium job trends already, so we were not surprised by Hendrickson’s report of Selenium skills seemed the most commonly required test automation technology.

Don’t believe us? Just check the QTP vs Selenium Job posting trends from Indeed.com below. Based on these trends, we’re going out on a pretty solid limb and predicting that openings for Selenium jobs will overtake QTP jobs within the next two years.

So, if you’re QA tester serious about a career in Quality and you don’t know a programming language like Java, Ruby, Python, and now JavaScript, learn one. However, you should be adept in automation technologies. We suggest, for obvious reasons, you learn Selenium.

And if you are a QA tester who already knows Selenium who hasn’t tried sending your tests to our browsers in the cloud? We’re always happy to hear yourfeedback on our service after you try it.

We welcome your comments!

Share

Cross browser testing with Soda and Selenium

September 16th, 2010 by Jason Huggins

Today, the cool kids at LearnBoost Labs, the creators of a pretty sweet online gradebook service, released Soda, a web testing library for NodeJS and Selenium, with built-in integration for cross-browser testing with Sauce Labs. It’s truly awesome. (It’s earned an exception to my general rule against the over-use of the word ‘awesome‘.)

Node.js is the new kid in web technologies, but it’s experiencing rapid growth in developer adoption, at a rate not seen since the early rise of Rails.

Here’s a sample acceptance test written with soda:

var soda = require('soda');

var browser = soda.createSauceClient({
  'url': 'http://yourwebsite/'
, 'username': ''
, 'access-key': ''
, 'os': 'Windows 2003'
, 'browser': 'googlechrome'
, 'max-duration': 300
});

browser
  .chain
  .session()
  .setTimeout(5000)
  .open('/')
  .waitForElementPresent('username')
  .type('username', 'invalid')
  .type('password', 'invalid')
  .click('//input[@value="Submit"]')
  .waitForElementPresent('css=ul.global-errors')
  .assertText('css=ul.global-errors li',
    'Please check your username / password')
  .testComplete()
  .end(function(err){
    if (err) throw err;
    console.log('Passed!');
  });



Using chained methods, Soda very elegantly solves Node’s “ugly callback syntax” problem. I suspect more NodeJS libraries will copy this pattern.

Seriously, check out Soda now! :-)

Share

A Quick Request to Her Majesty’s Government

August 4th, 2010 by Jason Huggins

Earlier this week, the UK government posted their reply to a petition, stating that they will not upgrade their systems away from Internet Explorer 6 (IE6) anytime soon. The online petition, created in February 2010, encouraged all government departments to upgrade their browsers. More than 6,000 UK citizens signed the petition.

The petition explains its reasoning: “Most creative and software development companies are forced by government department clients to build websites for IE6 when most of the industry has moved on.”

We at Sauce Labs feel for the citizens of the UK, and we’d like to help. Unfortunately, in its reply, the UK government cites the high cost and time involved in testing an upgrade as the reason it will not upgrade IE6. The official government response states: “Upgrading these systems to IE8 can be a very large operation, taking weeks to test and roll out to all users. To test all the web applications currently used by HMG departments can take months at significant potential cost to the taxpayer. It is therefore more cost effective in many cases to continue to use IE6″

Sauce Labs is kindly offering Her Majesty’s Government some help in upgrading, and a free evaluation of Sauce OnDemand, which will save development time and cost the citizens of the UK far less.

With the Selenium cross browser testing tool, any UK government department can test their site in IE6 and IE8 in one shot. Since Selenium is free and open source software, licensing fees won’t be an issue for UK taxpayers. And with cloud computing services like Sauce OnDemand, there’s no need to invest in expensive and labor intensive test lab infrastructure. Instead, Sauce Labs provides a low-cost cross-browser testing service in the cloud.

We challenge the UK government to create an account at saucelabs.com now and try Selenium today. To back this up, we’re offering any UK government department a free trial account to try out the service to test their department websites in IE6 and IE8.

All in favor, petition your parliamentarian directly and Demand Your Sauce!

Share

The Over-Exaggerated Death of Flex and Flash

July 20th, 2010 by Jason Huggins

The war between Adobe’s Flash and Apple’s support for HTML5 continues to heat up. The first set of battles between Apple and Adobe have reached a standstill for the moment. Adobe is pushing Flash for Android (which is awesome, on my new Google Nexus One, by the way). Meanwhile, the Apple-backed SproutCore project makes HTML5 a viable platform for developers to adopt today.

In spite of the Apple marketing machine, we’re betting that Flex and Flash still have their parts to play in the future of the web. After all, how can we just forget about the massive installed base? Flash is now on 98 percent of computers. More importantly, Flex and Flash have comprehensive tool sets that developers still need. Flash is the single browser plug-in that provides consistency for a chaotic and scattered web world. There are things that HTML and Javascript can’t do well, like webcam capture, media-streaming, and full-screen video. And let’s not forget — a huge segment of the web is gaming, which wouldn’t be possible (so far) without Flash. As YouTube recently noted: “While HTML5’s video support enables us to bring most of the content and features of YouTube to computers and other devices that don’t support Flash Player, it does not yet meet all of our needs.” Should we just abandon everything that we’ve known for HTML5 because it is new and exciting?

I’ll admit that even I have drunk the Apple Kool-Aid and see HTML5 for its role in the future of the web. Sauce specializes in cross-browser web app testing and HTML5 is easier to integrate with Selenium. However, the web is bigger than HTML and JavaScript. HTML5 is the future, but we live in the now. Though growing, HTML5 does not yet have the adoption that Flex and Flash have.

It’s this installed base that Sauce Labs is committing to with our announcement of the Sauce Flash-Flex Testing System.

The newest Selenium-based solution, the Sauce Flash-Flex Testing System provides testing of Flex applications on the Flash platform. Other tools in the marketplace can only test Flex applications, but the Sauce Flash-Flex Testing System covers the entire Flash platform. While Flex is a great tool for many developers, the two biggest markets on the web — gaming and advertising — are based on Flash, not Flex.

So, while we’re all excited for HTML5, rumors of Flash’s death are greatly exaggerated. Demand for Flash wont die. To a lot of people, Adobe may seem like yesterday’s news. But since not everyone has an iPad or an iPhone, the answer is still Flash and Flex, even if it’s not the new hotness.

To this strong base of Flex and Flash users: try out our Sauce Flash-Flex Testing System and give us your feedback.

Share

Selenium Rules Web Testing World…And We Couldn’t be Happier

July 16th, 2010 by The Sauce Labs Team

A new survey by ActiveState shows Selenium to be the most highly adopted testing framework among the automated testing community.

According to the author’s analysis, the “results were a mixed bag as the test automation community has a lot of frameworks and testing approaches from which to choose,” but we read things a bit differently.

With 40% of users preferring Selenium, and the next most popular approach being “other,” it’s clear who the winner is. But of course, there are “lies, damn lies, and statistics” to keep in mind, and at Sauce Labs, we admit to having a special bias towards Selenium :-).  It’s hard not to, though, when Jason Huggins, our co-founder, is the original creator, and was recently named one of the 25 “Best CTOs of 2010″ by InfoWorld.

With that being said, it’s great there are many choices for automated testing, and we’re glad that our favorite testing platform is at least winning “by a nose.”

If you’d like to see useful tips and tricks for using Selenium, check out some of our other blog posts under the category “Selenium Knowledge.” Also, make sure to keep your eyes peeled, we have some good stuff planned for next week.

Share

Trends in Testing: Continuous Integration, Mobility, Open Source, Cloud

May 10th, 2010 by The Sauce Labs Team

Functional testing is the automation of web app testing across several platforms or browsers. In the past, functional testing was less common due to long development cycles and the lack of multiple browser options. It was done primarily in-house via proprietary software like Rational or QuickTestPro.

Today, there are four trends converging to radically change how organizations test web applications:

  1. Continuous integration – An increased emphasis on performance of the software through the end customer’s perspective is leading agile development. Gartner predicts that by 2012 agile development methodologies will be used by 80 percent of all software development projects. Teams are shifting away from rigid quality control to quality that is demonstrable to the end user. With this shift, comes a need for quick, simple and automated testing tools.
  2. Open source – Open source tools, like Selenium, are viable for functional testing. Today’s development environment has dramatically changed and a lot more people are contributing than ever before. The major advantages of open-source are speed, time to market, time to value, and the ability to reach and grow a developer and user community. In fact, Gartner predicts that by 2012, 80 percent of all commercial software will include elements of open source technology.
  3. Cloud computing – Testing in the cloud is an affordable and scalable alternative to testing behind a firewall. It is expensive and time consuming to maintain on site test infrastructures that cover a vast number of browsers (and versions) of operating systems in several languages. The increase in the adoption of cloud computing creates an opportunity to leverage the space for functional testing. We believe the life cycle for cloud-based applications will look different in the future. New solutions that are cloud based will support cloud-base applications.
  4. Mobility – Websites need to support multiple browsers such as Internet Explorer, Firefox, Chrome, Rockmelt, Opera on various platforms including Windows and Mac. Smartphones only add to this “Browser War 2.0” battle with the plethora of mobile operating systems. These various environments make the infrastructure required more complex and the need for functional testing more relevant that ever before.

Are there other trends beyond continuous integration, mobility, open source and the cloud that will change how organizations test their web applications?

Share
« Older Entries