Our friends in the Plone community recently took to Open Sauce for their testing needs to save time. The results have been stellar; with the time saved they're able to focus on improving Robot Framework, according to their release manager Eric Steele. Check out the rest of what they have to say below.
When I took over as release manager for the Plone CMS project, we ran our test suite nightly, but that only covered our Python code and some simple form submissions. The entire JavaScript layer remained largely untested, save a few click-arounds by hand before each release. The suspicion that some critical feature might have broken in a browser combination we hadn't tried kept me up at night. As I began preaching the need for continuous integration and in-browser testing, it was surprising to find a whole team's-worth of people excited to obsess over running tests, improving coverage, and collecting a fleet of VMs to run the few Selenium tests we'd put together at that point. The latter proved to be our undoing; we spent more time managing our testing infrastructure than we did doing actual testing.
Thankfully, Sauce Labs' Open Sauce came along to save us.
Open Sauce has freed up my testing team to do far more interesting things. We've put quite a bit of effort into helping Robot Framework grow. Robot's Behavior-Driven Development abstraction seems to fit everyones' heads a bit better and allows us to easily alter tests based on which features are active. Asko Soukka, previously featured on this blog, became Plone's Person of the Year for 2013 based on the work he put into extending Robot Framework for our community.
Asko has created a set of Robot keywords to enable automated screenshots for our Sphinx documentation. This allows our documentation to show the Plone user interface in the same language as the document. Groups deploying Plone sites can regenerate our end-user documentation with screenshots featuring their own design customizations. It's a huge win; users see examples that look exactly like their own site. Finally, in a bit of pure mad science, Asko has piped those image generation scripts through a text-to-speech program to create fully-automated screencasts.
The Plone community is currently at work on the upcoming release of Plone 5. With its new widgets layer and responsive design, there are so many new ways that bugs could creep into our system. Happily, that's not the case. I get a nightly report full of screenshots of Plone in action across browser, device, and screen size. Basic accessibility gotchas are quickly caught. Content editing and management features are automatically tested on both desktop and mobile. Open Sauce allows us to focus on getting things done and done correctly. Finally, I can sleep soundly — or at least find something else to worry over. -Eric Steele, Release Manager, Plone.org. Read Eric's blog here or follow him on Twitter.
