Diego Molina, Staff Software Engineer at Sauce Labs and core contributor to the Selenium project, gave a Selenium Grid Build vs. Buy webinar in August, 2021 (watch it here). In this blog post he answers some of the questions posed during the webinar.
Q: How does a Selenium Grid work with Appium?
A: Grid 4 does not yet work with Appium, but there is work in progress to make them compatible. It will be presented at AppiumConf in September 2021, in a presentation on Selenium Grid 4 and Appium Together in Harmony.
Q: Is Sauce To Go compatible with Appium?
A. Not yet. We could probably look into that based on feedback and requests we get at https://github.com/saucelabs/sauce-togo.
Q: Is it possible to add executions of mobile tests in a Selenium Grid dashboard?
A: Mobile tests will be visible when we have implemented an integration with Appium. Once again, a work in progress and will be presented at AppiumConf in September 2021, in a presentation on Selenium Grid 4 and Appium Together in Harmony.
Q: Will I be able to replace my Zalenium set-up on Kubernetes with Selenium Grid 4? How complex will it be to migrate to Selenium Grid 4?
A: The short answer is yes, with some adjustments and without the "dynamic" feature available. The long answer is: we are taking a conservative approach with Grid 4, where we want to see what the community needs and what type of general use cases are out there. With Zalenium, I personally took an approach of offering lots of features and integrations that were really hard to maintain in the long run, which is why we are holding into any type of "out-of-the-box" integration with Kubernetes. The first thing we will tackle after moving to release candidate will be the pending work on a Helm chart and after that we’ll pause and wait for feedback.
Q: Which tech is used for the event bus? RabbitMQ/Kafka/Redis?
A: When running on Standalone, the server uses the GuavaEventBus, and in the other roles it uses https://zeromq.org/. Future plans include adding other backends, such as Redis, given that the interfaces are well defined. Other components such as the SessionMap are already backed by Redis, if desired. Q: Can the node be shut down remotely as it was possible to do in Selenium Grid 3 - http://{nodeUrl}:{nodePort}/servlet/extra/LifecycleServlet?Action=shutdown?
A.Yes, this is possible, however, with a slight difference. In Grid 4, we have a new endpoint to "drain" the Node, which means that the Node will stop accepting new sessions and when the last active session ends, the Node will shut down. Please see the documentation for more information.
Q: Is it good to sacrifice execution speed when using a third party? Since in a local grid everything will be much faster.
A: It always depends on the context and the scope/coverage you want to have in your tests. It also depends on your test strategy - if you have hundreds or thousands of tests, it might not be the best idea to run them on every commit using a third party as there won't be quick feedback. But if you have a more balanced test suite, testing at different layers of the app, and only running tests on each commit that guarantee that the major workflows are stable, then it might be a good idea to use a third party as you can focus on the quality of tests rather than on the test infrastructure.
Ideally, we should test in an environment that is similar to what our customer uses, and if you have a local Grid that covers the environments your customers use, then a local Grid might be a good decision. In general, our decisions and test strategy should consider what tests and environments we need to offer our teams and the company enough confidence after each test run. Other folks mix both, running quick tests during the development iterations on a local Grid and when they open a PR, they use Sauce Labs to cover more environments and with that have a good confidence that things are working well. As you can see, this is not a "yes/no" question, the answer will always come after defining a test strategy and after considering the context. Q: What’s the level of security with Sauce To Go? My customers might ask how secure it would be for the transformation of their sensitive data (test results, screenshots) etc.
A: Sauce To Go will run inside your infrastructure, no need to make any network changes or use a VPN for now, so it is a good starting point for a proof of concept with Sauce, given that you might have strong security concerns. Regarding sensitive test data, Sauce To Go creates the logs, screenshots and videos during execution time and will upload them via HTTPS after being able to authenticate to the Sauce Labs endpoint. When the data is in Sauce, it will already be protected by our certified security procedures. We have data centers in the US and the EU, so you can also decide which area to use based on your own policies. Therefore, in the end, the test data produced will be secured during the whole flow. Q: Regarding the machines, is it possible to create a machine (turning on) and kill the machine after the test with Selenium Grid?
A: Technically, this is possible. This is what conceptually happens with the Docker integration - a machine is created (Docker container created), and the machine is stopped when test ends (Docker container is stopped and removed). However, one would need to understand what APIs or mechanisms are available to do what you want in the environment you are considering. For example, if you are thinking about using VirtualBox, then you would need to understand the VirtualBox APIs to start and stop virtual machines. It goes without saying that this is work that needs to be done, and it is part of the decision process when someone wants to build their own Grid.
Q: Any thoughts on how, or whether, Sauce To Go might “play” with Sauce Bindings in the future?
A: Absolutely. Based on feedback and requests we get at https://github.com/saucelabs/sauce-togo, we might have a look into that. Q: Will Grid 4 have older browser version support?
A: Older browser versions is rather ambiguous, but yes, Grid 4 supports all browsers and versions that use W3C as their browser driver. Q: What's the biggest challenge for hosting a Selenium Grid locally?
A: Depending on how many operating systems, browsers, devices, and versions you are supporting, I would say the biggest challenges are: performance (keep the Grid fast based on the amount of tests running in parallel), maintenance (keep up with the new releases, a new browser version every 6 weeks, and new operating systems every year), and testing (yes, when a new browser or operating system comes out, the infrastructure needs to be tested before rolling it out to all users and team, and with that also having good rollback strategies). In the end, if you want to be serious about having your own testing infrastructure, it has to be rock solid as it is part of the critical pipeline for releases. Q: I understand that Sauce Labs has some features which come at an additional cost, e.g., Performance Testing). It would be good to see if it’s possible to do these things also on the grid without having to opt-in for those products (e.g., running Lighthouse yourself). A: One of the advantages, and disadvantages at the same time, is the chance to customize and add as many features as needed to your own Grid. There are ways to add things like Lighthouse or Accessibility testing to a Grid, but it comes with the cost of maintenance. For example, if you offer Lighthouse as a feature in your own Grid, the integration needs to be maintained and tested on every new Chrome release. I like to say that all this is technically possible, but you need to evaluate how much sense it makes for you to implement it based on your skills, available time, and work responsibilities.
Please also note that Sauce Performance has been free of charge since October 2020. All Sauce Labs users can use it.
If you missed Diego's webinar you can watch it here.
