A Functional Tester Looks at Performance

Posted by Ashley Hunsberger in Best PracticesGuest Blog PostsQAQuality Assurance

Even if you aren’t directly responsible for performance, it is important to consider it under the umbrella of quality. As a tester, how do you move forward and help drive performance quality (especially when you are new to this area, like me)? What are the ramifications of not considering performance within QA? Let’s take a look at what performance is, the questions QA can ask during design and implementation, some of the types of testing that can be done, and making performance part of your acceptance criteria (and, therefore, part of your Definition of Done).

What is software performance, and why is it important?

As an end user, I think of performance as just how fast or stable something is. If I click on something, does it take forever to load in a website? Does my app crash every time I try to open it or submit something? Do I give up and find a better solution to meet my needs? Of course we want a feature to work, but do we think about the system holistically?

I can tell you now that if a website or app that I am using crashes, I instantly think that the quality is just not there. If I have a choice in what I use, I quickly delete it and find another that does work. You may be tied into an app and not have a choice, but your opinion of that app (and the company) can quickly plummet based on stability alone.

Although performance is multi-faceted, some basic topics to think about include:

  • Response time – How quickly does the system react to user input?
  • Throughput – How much can the system accomplish in a specified amount of time?
  • Scalability – Can the system increase throughput under an increased load when resources are added?
  • Availability – Is the system available for service when requested?

To retain customers, you must consider performance as part of overall quality.

Understanding performance during development

The problem I’ve seen is that performance is always deemed important, but is not necessarily addressed up front. All too often I recall discussing performance long after a feature was coded and tested. It was pushed until the end, and it can be difficult to make your features meet performance expectations after they’ve been built. This was difficult in a Waterfall world, but how do you adapt what was an afterthought as more and more companies are moving towards a Continuous model? Performance needs to be considered first and understood by the team.

Here are some sample questions to be considered DURING design and development to help ensure you are considering performance needs early on as you go through your non-functional requirements (remember, you will need to discuss as a team and get guidance in determining what is expected):

Edit
Category Non-Functional Requirement Questions
Response Time What is the acceptable waiting time for users?

Do we need to consider users on various devices and speeds? Do we need to simulate slower speeds? Some may be on modern desktop/laptops on high speed Internet, or modern mobile devices on 3g and above — but others may not.

Example - Changing a password. How long before I can expect a change to take effect? Do I need to show progress feedback?
Data Volume How do we ensure that data volume does not impact user experience?

What's the maximum and typical volume of data that will be involved?

Example - Entering a page that lists users. Do I show all 20,000 users in the system? Do I show the first 25? How long does it take? Can I perform other actions while the list generates? Do I see a blank screen while I wait?
Caching Cache is king. Queried and calculated data can be reused to eliminate duplicate work.

When do we need to invalidate the cache? Which data cannot tolerate staleness?

How long can the cache live for?

Could cache staleness impact the system or the user session only?

Example - Notification badges. How long do those notifications last once a user has viewed, or just upon first login? If the cache is stale, does it impact just the authenticated user?

Testing performance

There are several types of testing that can help ensure your apps are performing as expected. Please note that this is not a comprehensive list, just a high-level overview to get you started1)Summarized from http://goo.gl/RC4AaS and http://goo.gl/5ukqAi :

Edit
Type of Test Overview
Load Application is tested for response times during normal and peak usage. How does the app respond with a few users completing a few interactions vs. thousands of users completing thousands of interactions at a time?
Stress Finds ways to break the system by increasing the load. Start with a good benchmark (identified during your load testing), and increase the load until you see which components start lagging and fail first.
Volume Test if application performance degrades with more data volume. Do you access the database directly? How does it handle the query if there are millions of records?
Reliability/Recovery If your app does fail, testing will show if and how it recovers, and how long it takes to get back to an acceptable state.
Scalability Tests if your app’s performance improves if you add resources (hardware, memory, etc.)

Improving performance quality faster

It’s time to stop pushing performance to the end and hoping for the best. As stories are designed, add performance to your acceptance criteria. Make sure that everything in your acceptance criteria is marked as complete (part of your Definition of Done).

As with anything, the longer you put something off, the more difficult (and/or expensive) it is to implement later. Be proactive, and build performance in.

Ashley Hunsberger is a Quality Architect at Blackboard, Inc. and co-founder of Quality Element. She’s passionate about making an impact in education and loves coaching team members in product and client-focused quality practices. Most recently, she has focused on test strategy implementation and training, development process efficiencies, and preaching Test Driven Development to anyone that will listen. In her downtime, she loves to travel, read, quilt, hike, and spend time with her family.

Discuss: A Functional Tester Looks at Performance
0 Comments

Free Trial

Get access to a free 14-day trial version, or contact Sales for more information.

\n:

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
Edit
Type of TestOverview
LoadApplication is tested for response times during normal and peak usage. How does the app respond with a few users completing a few interactions vs. thousands of users completing thousands of interactions at a time?
StressFinds ways to break the system by increasing the load. Start with a good benchmark (identified during your load testing), and increase the load until you see which components start lagging and fail first.
VolumeTest if application performance degrades with more data volume. Do you access the database directly? How does it handle the query if there are millions of records?
Reliability/RecoveryIf your app does fail, testing will show if and how it recovers, and how long it takes to get back to an acceptable state.
ScalabilityTests if your app’s performance improves if you add resources (hardware, memory, etc.)
\n

Improving performance quality faster

\n

It’s time to stop pushing performance to the end and hoping for the best. As stories are designed, add performance to your acceptance criteria. Make sure that everything in your acceptance criteria is marked as complete (part of your Definition of Done).

\n

As with anything, the longer you put something off, the more difficult (and/or expensive) it is to implement later. Be proactive, and build performance in.

\n

Ashley Hunsberger is a Quality Architect at Blackboard, Inc. and co-founder of Quality Element. She’s passionate about making an impact in education and loves coaching team members in product and client-focused quality practices. Most recently, she has focused on test strategy implementation and training, development process efficiencies, and preaching Test Driven Development to anyone that will listen. In her downtime, she loves to travel, read, quilt, hike, and spend time with her family.

","useMarkdown":true}), document.getElementById("react_7xqBxevvy0aa6Pm5VedD3Q")); ReactDOM.render(React.createElement(Components.RichTextArea, {}), document.getElementById("react_qBk9HG3UiBpLnGy5XMtA")); ReactDOM.render(React.createElement(Components.Share, {"title":"Share","href":"https://saucelabs.com/blog/a-functional-tester-looks-at-performance"}), document.getElementById("react_ajpq3BLHc0q6ZzeGwAlihw")); ReactDOM.render(React.createElement(Components.BlogPostCommentContainer, {"referenceName":"CommentsForContentView459","id":2145,"title":"A Functional Tester Looks at Performance","lang":{"cancelReply":"Cancel","commentCount":"%s Comments","commentCountSingular":"%s Comment","commentLabel":"Your comment*","emailLabel":"Email","like":"Like","liked":"Liked","nameLabel":"Name*","newestLabel":"Newest","oldestLabel":"Oldest","reply":"Reply","replyingTo":"Replying to %s","submit":"Post Comment","submitted":"Your comment has been submitted for approval.","title":"Discuss: %s"}}), document.getElementById("react_TOytnLNyG0GTFcZ4bdxARw")); ReactDOM.render(React.createElement(Components.ButtonGroup, {"buttons":[{"key":3630,"text":"Subscribe to our blog","href":"http://feeds.feedburner.com/saucelabs/DzbW","target":"_blank","styles":["tertiaryFilled"]}],"styles":["extraBottomMargin"]}), document.getElementById("react_TlpXw3yCPU6dZwt13OUhuQ")); ReactDOM.render(React.createElement(Components.Heading, {"text":"Categories","tag":"h3","styles":["h2LineHeight"]}), document.getElementById("react_96tyKwqA7ESi6Kq3zs0VUw")); ReactDOM.render(React.createElement(Components.BlogCategoryList, {"categories":[{"key":5274,"title":"Accessibility Testing","href":"/blog/category/accessibility-testing"},{"key":1546,"title":"Agile","href":"/blog/category/agile"},{"key":4316,"title":"Android","href":"/blog/category/android"},{"key":1547,"title":"AngularJS","href":"/blog/category/angularjs"},{"key":1548,"title":"Appium","href":"/blog/category/appium"},{"key":1549,"title":"Appium Resources","href":"/blog/category/appium-resources"},{"key":1550,"title":"Automation","href":"/blog/category/automation"},{"key":1551,"title":"Best Practices","href":"/blog/category/best-practices"},{"key":4194,"title":"Chef","href":"/blog/category/chef"},{"key":1552,"title":"Contests","href":"/blog/category/contests"},{"key":1553,"title":"Continuous Delivery","href":"/blog/category/continuous-delivery"},{"key":1554,"title":"Continuous Integration","href":"/blog/category/continuous-integration"},{"key":1555,"title":"Continuous Testing","href":"/blog/category/continuous-testing"},{"key":3587,"title":"Cross Browser Testing","href":"/blog/category/cross-browser-testing"},{"key":1556,"title":"DevOps","href":"/blog/category/devops"},{"key":1557,"title":"Events","href":"/blog/category/events"},{"key":1558,"title":"Guest Blog Posts","href":"/blog/category/guest-blog-posts"},{"key":4257,"title":"IoT","href":"/blog/category/iot"},{"key":3584,"title":"Java","href":"/blog/category/java"},{"key":1559,"title":"JavaScript","href":"/blog/category/javascript"},{"key":3607,"title":"Jenkins","href":"/blog/category/jenkins"},{"key":3476,"title":"JUnit","href":"/blog/category/junit"},{"key":4174,"title":"Load Testing","href":"/blog/category/load-testing"},{"key":1560,"title":"Mobile","href":"/blog/category/mobile"},{"key":1561,"title":"Mobile Development & Testing","href":"/blog/category/mobile-development-testing"},{"key":1562,"title":"News & Product Updates","href":"/blog/category/news-product-updates"},{"key":1563,"title":"Open Sauce","href":"/blog/category/open-sauce"},{"key":3655,"title":"Open Source","href":"/blog/category/open-source"},{"key":4175,"title":"Performance Testing","href":"/blog/category/performance-testing"},{"key":1564,"title":"Press","href":"/blog/category/press"},{"key":1565,"title":"Product Updates","href":"/blog/category/product-updates"},{"key":1566,"title":"Protractor","href":"/blog/category/protractor"},{"key":1567,"title":"QA","href":"/blog/category/qa"},{"key":1568,"title":"Quality Assurance","href":"/blog/category/quality-assurance"},{"key":1569,"title":"Quality Engineering","href":"/blog/category/quality-engineering"},{"key":1570,"title":"Releases","href":"/blog/category/releases"},{"key":1571,"title":"Ruby","href":"/blog/category/ruby"},{"key":1572,"title":"Sauce Product Info","href":"/blog/category/sauce-product-info"},{"key":3573,"title":"Scrum","href":"/blog/category/scrum"},{"key":1573,"title":"Security Testing","href":"/blog/category/security-testing"},{"key":1574,"title":"Selenium","href":"/blog/category/selenium"},{"key":1575,"title":"Selenium Resources","href":"/blog/category/selenium-resources"},{"key":1576,"title":"Software Development & Testing","href":"/blog/category/software-development-testing"},{"key":1577,"title":"The Story of Sauce","href":"/blog/category/the-story-of-sauce"},{"key":1578,"title":"Uncategorized","href":"/blog/category/uncategorized"},{"key":1579,"title":"Videos","href":"/blog/category/videos"},{"key":1580,"title":"WebdriverIO","href":"/blog/category/webdriverio"},{"key":1581,"title":"Webinars","href":"/blog/category/webinars"}]}), document.getElementById("react_mbcmG68EHEqnSVR9pbEFbQ")); ReactDOM.render(React.createElement(Components.ButtonGroup, {"buttons":[{"key":2497,"text":"Free Trial","href":"/signup/trial","styles":["inverted"]},{"key":2498,"text":"Contact Us","href":"/contact","styles":["inverted"]}],"styles":["center","noMargin"]}), document.getElementById("react_RQWkw5sJfU6elW1H9BPdg")); ReactDOM.render(React.createElement(Components.SiteHeader, {"navigation":{"links":[{"key":149,"title":"Solutions","children":[{"key":160,"title":"Enterprise","href":"/enterprise","children":[{"key":162,"title":"Automated Testing Platform","href":"/enterprise#automated-testing-platform","children":[],"isActive":false},{"key":161,"title":"Cross Browser Testing","href":"/enterprise#cross-browser-testing","children":[],"isActive":false},{"key":2373,"title":"Mobile Testing","href":"/enterprise#mobile-testing","children":[],"isActive":false},{"key":2374,"title":"Manual Testing","href":"/enterprise#manual-testing","children":[],"isActive":false},{"key":163,"title":"Request a Quote","href":"http://info.saucelabs.com/enterprise-request-lp.html","target":"_blank","children":[],"isActive":false}],"isActive":false},{"key":775,"title":"Small Team","href":"/small-team","children":[{"key":772,"title":"Automated Testing Platform","href":"/small-team#automated-testing-platform","children":[],"isActive":false},{"key":771,"title":"Cross Browser Testing","href":"/small-team#cross-browser-testing","children":[],"isActive":false},{"key":2375,"title":"Mobile Testing","href":"/small-team#mobile-testing","children":[],"isActive":false},{"key":2376,"title":"Manual Testing","href":"/small-team#manual-testing","children":[],"isActive":false},{"key":774,"title":"Pricing","href":"/pricing","children":[],"isActive":false}],"isActive":false},{"key":944,"title":"Open Source","href":"/open-source","children":[{"key":942,"title":"Automated Testing Platform","href":"/open-source#automated-testing-platform","children":[],"isActive":false},{"key":941,"title":"Cross Browser Testing","href":"/open-source#cross-browser-testing","children":[],"isActive":false},{"key":2377,"title":"Mobile Testing","href":"/open-source#mobile-testing","children":[],"isActive":false},{"key":2378,"title":"Manual Testing","href":"/open-source#manual-testing","children":[],"isActive":false},{"key":943,"title":"Signup","href":"https://saucelabs.com/beta/signup/OSS/None","target":"_blank","children":[],"isActive":false}],"isActive":false},{"key":1356,"title":"Additional Products","children":[{"key":2280,"title":"JavaScript Unit Testing","href":"/products/javascript-unit-testing","children":[],"isActive":false},{"key":1357,"title":"Open Source Frameworks","href":"/products/open-source-frameworks","children":[],"isActive":false},{"key":1358,"title":"Integrations","href":"/products/integrations","children":[],"isActive":false},{"key":3466,"title":"Platforms","href":"/platforms","children":[],"isActive":false},{"key":4916,"title":"Professional Services","href":"/support/services","children":[],"isActive":false}],"isActive":false}],"isActive":false},{"key":150,"title":"Stories","children":[{"key":249,"title":"Enterprise","href":"/enterprise/stories","children":[{"key":491,"title":"Case Studies","href":"/enterprise/case-studies","children":[],"isActive":false},{"key":492,"title":"Industry Examples","href":"/enterprise/stories#industry-examples","children":[],"isActive":false},{"key":493,"title":"Testimonials","href":"/enterprise/stories#testimonials","children":[],"isActive":false}],"isActive":false},{"key":790,"title":"Small Team","href":"/small-team/stories","children":[{"key":786,"title":"Case Studies","href":"/small-team/case-studies","children":[],"isActive":false},{"key":787,"title":"Industry Examples","href":"/small-team/stories#industry-examples","children":[],"isActive":false},{"key":788,"title":"Testimonials","href":"/small-team/stories#testimonials","children":[],"isActive":false}],"isActive":false},{"key":950,"title":"Open Source","href":"/open-source/stories","children":[{"key":946,"title":"Case Studies","href":"/open-source/case-studies","children":[],"isActive":false},{"key":947,"title":"Industry Examples","href":"/open-source/stories#industry-examples","children":[],"isActive":false},{"key":948,"title":"Testimonials","href":"/open-source/stories#testimonials","children":[],"isActive":false},{"key":949,"title":"Community","href":"/community","children":[],"isActive":false}],"isActive":false}],"isActive":false},{"key":1425,"title":"Pricing","href":"/pricing","children":[],"isActive":false},{"key":152,"title":"Resources","href":"/resources","children":[{"key":1271,"title":"Featured Resources","href":"/resources","children":[],"isActive":false},{"key":607,"title":"Blog","href":"/blog","children":[],"isActive":true},{"key":841,"title":"Compare Us","href":"/compare-us","children":[],"isActive":false},{"key":1272,"title":"Videos","href":"/resources/videos","children":[],"isActive":false},{"key":606,"title":"Articles","href":"/resources/articles","children":[],"isActive":false},{"key":700,"title":"White Papers","href":"/resources/white-papers","children":[],"isActive":false},{"key":698,"title":"Data Sheets","href":"/resources/data-sheets","children":[],"isActive":false},{"key":654,"title":"Case Studies","href":"/case-studies","children":[],"isActive":false},{"key":634,"title":"Webinars","href":"/resources/webinars","children":[],"isActive":false},{"key":2855,"title":"Best Practices","href":"/resources/best-practices","children":[],"isActive":false}],"isActive":false},{"key":153,"title":"Support","href":"/support","children":[{"key":3325,"title":"Overview","href":"/support","children":[],"isActive":false},{"key":810,"title":"Knowledge Base","href":"/support#knowledge-base","children":[],"isActive":false},{"key":811,"title":"Documentation","href":"/support#documentation","children":[],"isActive":false},{"key":813,"title":"Training","href":"https://training.saucelabs.com/","target":"_blank","children":[],"isActive":false},{"key":3326,"title":"Services","href":"/support#services","children":[],"isActive":false},{"key":809,"title":"Contact Support","href":"/support#contact","children":[],"isActive":false}],"isActive":false},{"key":154,"title":"Company","href":"/company","children":[{"key":1156,"title":"Company","href":"/company","children":[{"key":1198,"title":"Team","href":"/company/team","children":[],"isActive":false},{"key":1157,"title":"Values","href":"/company/values","children":[],"isActive":false},{"key":1448,"title":"Careers","href":"/company/careers","children":[],"isActive":false},{"key":1167,"title":"Partners","href":"/company/partners","children":[],"isActive":false},{"key":1430,"title":"Contact Us","href":"/contact","children":[],"isActive":false}],"isActive":false},{"key":1238,"title":"News","href":"/news","children":[{"key":1239,"title":"Press Releases","href":"/news/category/press-releases","children":[],"isActive":false},{"key":1240,"title":"Press Coverage","href":"/news/category/press-coverage","children":[],"isActive":false},{"key":1241,"title":"Awards","href":"/news/category/awards","children":[],"isActive":false}],"isActive":false},{"key":1042,"title":"Community","href":"/community","children":[{"key":1043,"title":"Join Secret Sauce","href":"/community#secretsauce","children":[],"isActive":false},{"key":1044,"title":"Events","href":"/community#events","children":[],"isActive":false},{"key":1132,"title":"Appium","href":"/community#appium","children":[],"isActive":false},{"key":1133,"title":"Selenium","href":"/community#selenium","children":[],"isActive":false}],"isActive":false}],"isActive":false}],"cta":{"text":"Free Trial","href":"/signup/trial"},"signIn":{"text":"Sign In","href":"/beta/login"}}}), document.getElementById("react_nYVgCLYvkkO1uimTDfjI6w")); ReactDOM.render(React.createElement(Components.SiteFooter, {"lang":{"captoraHeading":"Trending Topics","copyright":"© 2017 Sauce Labs. All rights reserved.","subscribeButtonText":"Join","subscribeEmailLabel":"Email*","subscribeFormID":"385","subscribeHeading":"Join our mailing list"},"links":[{"key":108,"title":"Solutions","children":[{"key":323,"title":"Enterprise","href":"/enterprise","children":[],"isActive":false},{"key":767,"title":"Small Team","href":"/small-team","children":[],"isActive":false},{"key":951,"title":"Open Source","href":"/open-source","children":[],"isActive":false}],"isActive":false},{"key":112,"title":"Stories","children":[{"key":376,"title":"Enterprise","href":"/enterprise/stories","children":[],"isActive":false},{"key":800,"title":"Small Team","href":"/small-team/stories","children":[],"isActive":false},{"key":952,"title":"Open Source","href":"/open-source/stories","children":[],"isActive":false}],"isActive":false},{"key":122,"title":"Resources","href":"/resources","children":[{"key":1273,"title":"Featured Resources","href":"/resources","children":[],"isActive":false},{"key":609,"title":"Blog","href":"/blog","children":[],"isActive":true},{"key":993,"title":"Compare Us","href":"/compare-us","children":[],"isActive":false}],"isActive":false},{"key":802,"title":"Support","href":"/support","target":"_blank","children":[{"key":803,"title":"Contact Support","href":"/support#contact","children":[],"isActive":false},{"key":805,"title":"Knowledge Base","href":"/support#knowledge-base","children":[],"isActive":false},{"key":806,"title":"Documentation","href":"/support#documentation","children":[],"isActive":false},{"key":808,"title":"Training","href":"https://training.saucelabs.com/","target":"_blank","children":[],"isActive":false},{"key":3566,"title":"Status","href":"https://status.saucelabs.com/","target":"_blank","children":[],"isActive":false}],"isActive":false},{"key":144,"title":"Company","href":"/company","children":[{"key":1160,"title":"Team","href":"/company/team","children":[],"isActive":false},{"key":1158,"title":"Values","href":"/company/values","children":[],"isActive":false},{"key":1237,"title":"Careers","href":"/company/careers","children":[],"isActive":false},{"key":2879,"title":"Partners","href":"/company/partners","children":[],"isActive":false},{"key":3467,"title":"Contact Us","href":"/contact","children":[],"isActive":false}],"isActive":false},{"key":2875,"title":"News","href":"/news","children":[{"key":2876,"title":"Press Releases","href":"/news/category/press-releases","children":[],"isActive":false},{"key":2877,"title":"Press Coverage","href":"/news/category/press-coverage","children":[],"isActive":false},{"key":2878,"title":"Awards","href":"/news/category/awards","children":[],"isActive":false}],"isActive":false},{"key":116,"title":"Community","href":"/community","children":[{"key":1045,"title":"Join Secret Sauce","href":"/community#secretsauce","children":[],"isActive":false},{"key":1046,"title":"Events","href":"/community#events","children":[],"isActive":false},{"key":1134,"title":"Appium","href":"/community#appium","children":[],"isActive":false},{"key":1135,"title":"Selenium","href":"/community#selenium","children":[],"isActive":false}],"isActive":false}],"disclaimerLinks":[{"key":354,"title":"Terms of Service","href":"/terms-of-service","children":[],"isActive":false},{"key":355,"title":"Privacy Policy","href":"/privacy-policy","children":[],"isActive":false}],"copyright":"© 2017 Sauce Labs. All rights reserved.","socialLinks":[{"key":1462,"service":"Facebook","href":"https://www.facebook.com/saucelabs","icon":"facebook"},{"key":1463,"service":"Twitter","href":"https://twitter.com/saucelabs","icon":"twitter"},{"key":1464,"service":"Google Plus","href":"https://plus.google.com/111293610298852124914","icon":"google-plus"},{"key":1465,"service":"LinkedIn","href":"https://www.linkedin.com/company/891955","icon":"linkedin"}],"marketoId":"468-XBT-687","captora":"\n\n"}), document.getElementById("react_fmBOkn4Dq0ODacmAv2k0w"));