Selenium Testing Framework Pt. 1: Testing Concepts

November 2nd, 2011 by Jason Smiley

This is part 1 of a 3-part guest blog series by Jason Smiley, QA Engineer at Gerson Lehrman Group.

When writing test scripts, you might find yourself trying to solve similar problems over and over again. So why not re-use the same solutions over and over again? The following framework will help you get the most out of your code without needing to do tons of maintenance every time something changes.

Before getting into specifics, lets first talk about what should be implemented when writing test scripts. At GLG, our tests typically go through a set of steps on the website. They also have to interact with elements on the site in order to accomplish these steps. Then there is the actual tests themselves, which review the displayed content on the site. Essentially, we have tests that assert the content on the site, a set of steps we need to take in order to perform these tests (which, for the sake of this blog post, I will describe as actions), and then the pages with elements that we are checking or interacting with.

When drawn as an ERD, we have something like this:

How can we best make use of this structure? We want tests to focus on use cases, actions that focus on what steps to take, and pages that allow us to interact with them. From a development standpoint, this means we have tests containing a list of actions that can be used to obtain the values required to check or change the state of the system under testing. Actions can interact with pages in order to complete these steps and make sure they accomplished their task. Pages can be used by actions or tests to add, edit, clear, or read data. Adding these relationships to the ERD, we get something like this:

 

What does this look like in code? How would we best accomplish this task? Well, that depends on what you are doing. For example, maybe you need to test 5 different types of projects, or maybe you need to test same page differently each time. Either way, as a coding standard, you should always design classes with a specific purpose. Anything that is shared should go into a base class for common functionality.

In the next blog post, I will describe how to set up different base classes for tests, pages, and actions. Then in part three of this blog post series, I will show you how to bring all of this together for testing your local projects.

Share

Free Sauce Hack Day This Thursday

November 1st, 2011 by Ashley Wilson

The Saucers are opening our office doors for our first ever Sauce Hack Day this Thursday, Nov. 3, from 2pm-6pm PST. If you’re a developer or tester working with Selenium, functional testing, continuous integration, javascript testing (really, anything to do with testing), then come on by to Sauce Labs at 500 3rd St, Ste 240 in San Franicsco for a couple hours of chatting, hacking and hanging out.

Perhaps you’ve been pondering whether or not to set up your own Selenium Grid. Or maybe your boss has tasked you with overhauling your company’s current testing process in favor of automation. Or maybe you’ve heard a lot about Sauce but just haven’t quite figured out what you would get out of using it.

No matter your motives, this hack day is your chance to get some insights and advice from folks who were once in the same boat as you, and to walk away knowing more than you did. We’ll be on hand to answer questions and chat about topics that might include how to get started using Sauce with Selenium, the pros and cons of switching from Selenium 1 to Selenium 2, the basic “things-to-know” if you want to offload your testing to Sauce Labs, testing best practices, and more.

We’ll provide space for you to work in, food, and plenty of beer, soft drinks and water. We ask that you provide a RSVP to the event, your own laptap and power supply. See you there!

Share

So you want to start a Selenium Meetup Group. Now what?

October 24th, 2011 by Ashley Wilson

The is part one in a two part series on Starting a Selenium Meetup Group.

Organizing Selenium meetups has become my thing. On my first day at Sauce back in April 2010, I was tasked with helping to organize a meetup that was happening two days later. I hadn’t done much event planning nor did I even know what Selenium was (a little embarrassed to admit that now), but I jumped in, proved I (sort of) knew what I was doing and was handed the reigns to Sauce’s monthly Selenium Meetups. 18 San Francisco Selenium meetups later, plus three in New York City, one in Boston, and the inaugural Selenium Conference, I’ve learned a thing or two about organizing events for the Selenium community and would like to share some of that with you.

I’m writing this post because my Selenium meetup history tells me there are Selenium users all over the world who want to meet other testers and developers, geek out about testing, and find out how others are utilizing this awesome tool – they just need an avenue to do so. We saw this in action at the Selenium Conference back in April. Everyone who showed up was committed to open source, wanted to make this project the best it can be and loved getting together to talk about Selenium. But while a big Selenium conference is great, realistically it can only happen once or twice a year. Meetups, on the other hand, can happen much more frequently, in just about any city in the world, with the help of one or two committed organizers.

Which is where you, future Selenium Meetup organizer, come in :-) If you don’t live in San Francisco, New York, Boston, Seattle, London, Toronto, MelbournePhoenix, and soon DC (cities that have an existing meetup group), I hope to arm you with the knowledge and confidence to start one yourself and see it flourish. So, without further ado, let’s dive into the nitty gritty of starting your first Selenium meetup!

1. Ask yourself: How active do you want this meetup group to be?
It’s important to set this expectation early on so you know what you’re signing up for. Will you meet every two weeks? Once a month? Once a quarter? I’ve found once a month to be solid, but I’m also fortunate that organizing these meetups is part of my duties at Sauce.

2. Decide what your first meetup topic will be
For a first event, I’d suggest opening the meetup with “getting to know you” time and then follow that up with either a speaker or a workshop that you (or someone else qualified) will lead. Organizing it in this way gives you time as the organizer to meet attendees, get a feel for their technical level and perhaps ask about future meetup topics. An agenda for the evening might look like this:

7:00pm: Registration, Welcome, pizza (or local favorite) and drinks
7:15pm: Welcome, Announcements
7:30pm: Mingling, networking, etc
8:00pm: Workshop / Presentation
8:30pm: Q&A / Wrap up workshop
9:15pm: Lights Out

If you’re experience is anything like mine, you’ll find you have to shoo people out so the cleaning people can come in to do their job.

3. Find a Speaker
This one can be difficult when you’re first starting out. Assuming you work at a company that uses Selenium, I’d ask your coworkers if anyone would like to present. If you strike out there, I’d go to Linkedin and filter by location and whether they have Selenium in their profile. You can also send a note to the LinkedIn Selenium Group or the Selenium user list on Google groups. If all those options fail, get in touch with me and I’ll do what I can to help you secure a speaker.

4. Pick a date for your first meetup
It’s a good idea to not only pick a date for your first event, but also establish the day of the week to have regular meetups. This will make it easier for attendees to remember and also (hopefully) keep you motivated to continue organizing. In San Francisco, we generally have our meetups on the third or fourth Tuesday of the month. In New York, we do meetups on the third Thursday of the month. I’d avoid doing events on Mondays and Fridays.

5. Look for a (free) venue
Check first with your own company. Meetups are a great recruiting tool and you can use that angle when convincing your boss to host a bunch of geeks. If your own company won’t do, put a note out to the Selenium user list saying that you’re starting this meetup and need a venue. Do a google search for other tech companies in your area and see if you can track down the developer advocate or QA director. Once again, if all these options fall short, send me an email. We’ve got a database of companies using Selenium and I know plenty of them would be thrilled to offer up their space.

Some things to remember about a venue: Open space is better than a conference room. You’ll need to confirm that there is a projector and screen set up for the presenters (with adapters for Mac and PC users!). Also be sure there are plenty of chairs set up. Theatre style tends to work best but if you’re doing a workshop, obviously round table seating is more preferable.

6. Find a sponsor for food and beer
Free food and drinks are essential to meetups so I wouldn’t skimp on this. I usually order pizza from a place close to the venue and have beer delivered from a liquor store. Every once in a while I find a company that provides a venue, food and drinks. When that happens, I get really happy and feel like I’ve hit the jackpot. You will too. (Remember to ask the company that is providing the venue if they’ll also pony up food and drinks. The worst they can say is no). For a group of 50 attendees, plan on it costing about $400 for food, drinks, and tip.

7. Set the group up on Meetup.com and announce your first meetup
It’s time to announce your meetup! When setting up your meetup page, be sure to use Selenium and your city in the headline (e.g. San Francisco Selenium Meetup Group) so it’s easily searchable. Go ahead and announce your first meetup, knowing that at first, it will be a meetup for one. Then tweet and/or blog about it. If you don’t tweet or blog, tell Sauce about it so we can do the tweet to our followers.

8. Have your meetup!
After weeks of organizing, it’s time to have your first meetup! Here are some parting tips:

      • Plan to arrive to your meetup 1 hour in advance to make sure everything is set up to your liking.
      • Come armed with nametags and sharpies.
      • Create a sign in sheet so you know how many people came vs. how many people RSVP’d.
      • Keep everything on time according to the agenda.
      • Smile, mingle, and, most importantly, have fun :-)

The next post will cover best practices for organizing subsequent meetups and growing your group.

Share

Announcing Selenium 2.9.0 support

October 21st, 2011 by Santiago Suarez Ordoñez

Wait! What?! Didn’t you guys just announce Selenium 2.8.0?! Well, the Selenium dev team moves crazy fast and we do our best to keep up.

Selenium 2.9.0 (released yesterday!) comes with several improvements and some bugfixes. Here’s the changelog for more information.

You can start using this new version right now by adding the following Desired Capabilities/JSON key-value:

"selenium-version": "2.9.0"

If you see any issues after moving your tests to this new release, we definitely want to hear about it.

For more information about the current version being used by Sauce, using other Selenium versions and the selenium-version flag you can check our docs on Sauce OnDemand additional configuration.

Share

Announcing Selenium 2.8.0 support

October 20th, 2011 by Santiago Suarez Ordoñez

Selenium 2.8.0 is Selenium’s latest release. It includes some important bug fixes as well as great features people have been waiting for:

  • setFileDetector and support for file uploads in RemoteDriver for java
  • Native event support in Firefox 7

Here’s the changelog and the official announcement in Selenium’s blog for more information.

The current default version for our service is 2.6.0. But once thoroughly tested, we’ll be announcing the move to 2.8.0 as the default version for all of our users’ tests to run. In the meantime, you can start using this new version right now by adding the following Desired Capabilities/JSON key-value:

"selenium-version": "2.8.0"

If you see any issues after moving your tests to this new release, we definitely want to hear about it. And remember, once we move everyone over, you’ll still be able to test with previous versions using the “selenium-version” capability outlined above, in case you notice any issues with the default version.

For more information about the “selenium-version” flag, you can check our docs on Sauce OnDemand additional configuration.

Share

Top Selenium Tips From The Sauce Codebase

October 13th, 2011 by Ashley Wilson

Having run more than 8 million tests in the Sauce cloud, we’ve learned a thing or two about the common pitfalls folks encounter when writing and running Selenium tests.

To help others not make those same mistakes, we recently started hosting bi-weekly webinars led by Sauce Ninja Santiago Suarez Ordoñez. This week’s webinar covered various Selenium tips, including implicit waits, timeouts, and the reasons to avoid complex locators.

If you’d like to attend an upcoming webinar, register here. Happy testing!

Share

Massive Reduction of Selenium Costs with Sauce Labs

October 4th, 2011 by Ashley Wilson

This post was originally written for Okta’s blog by QA Lead Denali Lumma. Okta is an identity & access management company for cloud/SaaS applications. It has been reprinted with her permission.

At Okta we provide enterprise grade identity management from the cloud.  Specifically, Okta delivers single sign-on across all your web applications, centralize user management and control, and integration with Active Directory.  We invest heavily in test automation and validation.  A lot of great testing tools are utilized, including Selenium.  We currently have about 1000 Selenium tests in our production test system.   These tests are deployed and run in the cloud with our trusted partner; Sauce Labs.  Currently, Selenium tests run continuously on check-in across master and release branches against our supported browser/ version/ OS matrix.  This matrix currently includes Firefox 3.6 on Linux, Firefox 7 (latest stable version) on Windows, Internet Explorer 8 on Windows, Internet Explorer 9 on Windows, Google Chrome on Windows, Safari 4 on Windows and Safari 5 on OSX.  (Safari 5 on OSX is run locally until the up and coming Mac VM feature is released by Sauce Labs.) The decision to use Sauce Labs as the Selenium client rather than running tests internally on a roll your own Selenium Grid configuration was methodically considered by our engineering team.

Even before we began to consider working with Sauce Labs, internally a general question would come up every now and again: why bother with Selenium?  The answer was simple: we have no choice. Because of our business focus, we must follow industry best practices with test automation across all levels of the stack including the GUI in order to ensure excellent product software quality.  Yet, there is no escaping that web application test automation with Selenium is expensive.  Engineers at our company report spending anywhere from 10 – 60% of their time writing, reviewing and maintaining Selenium tests.  But the cost of not implementing Selenium testing is beyond high—it’s a game-ender.  Without constant deep and broad test feedback developers are unable to introduce changes safely and responsibly at the constantly accelerating pace achieved by the world’s best software companies—our competitors and your competitors.  But we’ve managed to reduce these costs significantly with Sauce Labs.  What we’ve found through our trial investigation and now full deployment is that Sauce Labs saves us tremendous time, and thus tremendous money.

Read the rest of this post on Okta’s blog

Share

Flying Saucers All ‘Round The World

September 29th, 2011 by Ashley Wilson

Here’s a breakdown of upcoming events we’ll be speaking at, drinking at, hanging out at, etc. If you’re in the area, give us a shout on Twitter and find us to say hello!

  • 10/01: Sauce Developer Adam Christian (Windmill, Jelly.io) has been traversing around Europe for the last few weeks. Soon he’ll make his final stop at JSConf EU in Berlin, Germany, held October 1-2. He won’t be speaking, but he will be around to chat about Sauce over a beer or two (bonus: he said he’s buying). If you’re attending the conference and are involved in testing, get in touch with him!
  • 10/02: Jason Huggins, co-founder of Sauce, will speak at the *free* Jenkins Users Conference this Sunday at the Marines’ Memorial Hotel in San Francisco. Last we heard, nearly 400 people had registered, but if you manage to get in, drop by his talk, Extreme Testing with Jenkins and Selenium. He’ll start with an overview of how Selenium and Jenkins usually connect together and then cover browser testing at the extremes, including how a major U.S. airline uses Selenium and Jenkins to test its website every 20 minutes across dozens of concurrent threads. Sauce is also a sponsor so if you want some goodies, stop by our booth!
  • 10/20: New Yorkers should mark their calendars for the next NYC Selenium Meetup, scheduled for 7pm on October 20 at Yodle. The topic for this free event is mobile testing with Selenium 2.0. I’ll be around to host and meet with folks who are interested in hearing more about Sauce and Selenium. We’ve also got a speaking slot open, so if you’d like to show off how your company is handling mobile testing, please send me an email with a brief abstract.
  • 10/25: Jason will speak at the Software Test Professionals (STP) Conference in Dallas. His talk, Web Testing with Selenium 2.0 – Better, Faster, More Awesome, will explain the benefits of using Selenium 2.0 and cover some of the big differences between Se v.1 and Se v.2. He’ll also cover trends in testing and give guidance on what should and should not be tested with Selenium.
  • 10/27: And Adam Christian is once again hitting the conference circuit with a talk at GTAC titled Browser Automation with NodeJS and Jellyfish. This event is currently closed to new registrants, but if you were able to snag a slot, make sure you tweet about how awesome it is :-)

And that’s what we’ve got so far. Happy testing!

Share

Security Through Purity — Bypassing the Red Queen

September 27th, 2011 by jeremy avnet .:. brainsik

As a cloud service, Sauce Labs uses quite a few virtual machines to run our browsers. With Windows images taking about 15 minutes to boot in EC2 and other cloud providers, the most efficient way to use these resources is to cleanup between customers and reuse. Unfortunately, reusing VMs paves a path for malicious users to install software that will continue to run while other customers use them. This means sensitive data could be captured and sent to a 3rd party or, even worse, a 3rd party could gain access to a customer’s internal network if they are testing with a tunneling service.1 How can a VM be protected from malicious software installation?

There are a variety of tools one can use to lock down Windows. The tools provided by Windows itself are the Software Restriction Policy and AppLocker technologies. And a cursory Google search will reveal current dll load hijacking methods for getting around them. The reality is it’s a Red Queen race between securing a system and finding a new exploit — “It takes all the running you can do, to keep in the same place.” For all intents and purposes, it’s like submitting your laptop to a hacking contest. Is there anything you could do to feel comfortable using it again?

the only winning move is not to play

Like Joshua learned, “the only winning move is not to play,” and that’s how we roll at Sauce Labs. Every single session, whether you’re using Scout or OnDemand, gets a fresh VM that’s never been used by anyone else.2 At the end of your session — whether it’s 10 minutes or 10 seconds — the VM (with all its data) is completely destroyed. By avoiding the unsolvable problem of guaranteeing a system secure across anonymous use, we instead guarantee a system that has never and will never be used by any customer besides you.3

A desirable side-effect of the Security Through Purity model is increased reliability. By always using pristine VMs for every session, every session is run with the same settings. This avoids intermittent failures that arise from the OS being in a state dependent on what the previous users did.

At Sauce Labs core is the ability to offer secure, remote browsers to our customers. Our attention to security provides confidence you can test any part of your external or internal services without having to build and manage your own browser testing infrastructure.

  1. Did you know you can use Sauce Labs to test your internal network? Check out Sauce Connect for secure way to use our cloud service to test your internal infrastructure. It’s been vetted by a well-known financial institution, a well-known domestic airline, and a variety of other high-profile customers.
  2. In this way, Sauce Labs has lunched nearly 1.3 million VMs in the last 30 days.
  3. Since we give you a fresh VM in under 20 seconds, a 15 minute boot time would require us to keep a lot of expensive, idle resources around to handle demand spikes. That’s why we developed a cloud technology which allows us to cold boot VMs in under 40 seconds. Aside from meaning demand spikes are handled with minimal delay, this also means there is no additional expense for providing the best security.
Share

Announcing Selenium 2.7.0 support

September 26th, 2011 by Santiago Suarez Ordoñez

Selenium 2.7.0 is Selenium’s latest release. Released just Friday, it includes some important bug fixes to make your tests in the cloud even more reliable than before. Here’s the official changelog for more information.

Once thoroughly tested, we’ll be announcing the move to 2.7.0 as the default version for all of our users’ tests to run. In the meantime, you can start using this new version right now by adding the following Desired Capabilities/JSON key-value:

"selenium-version": "2.7.0"

If you see any issues after moving your tests to this new release, we definitely want to hear about it. And remember, once we move everyone over, you’ll still be able to test with previous versions using the “selenium-version” capability outlined above, in case you notice any issues with the default version.

For more information about the “selenium-version” flag, you can check our docs:
http://saucelabs.com/docs/sauce-ondemand#selenium-version.

Happy testing!

Share