The time to continue our journey of reviewing e2e testing frameworks has come. If you missed our previous articles on the topic don’t hesitate to take a look, we already covered Cypress and Nightwatch.js.
Read part 1 of "On a way to find the perfect e2e testing tool"
Read part 2 of "On a way to find the perfect e2e testing tool"
To summarize, our goal is to build an independent test environment for one of our projects and automate the process of features validation. The whole test suite will be a standalone project without any access to the raw codebase, so we won’t mock anything. The result should be a real world user journey, executed directly on the dev website, providing basic insurance that all main features are working as expected and our latest changes didn’t cause any regression bugs. This way we try to eliminate the user factor in the process by automating some of the manual tests we usually do after every push. And in order to achieve it we thought it’s a good idea to make a small research of the market and find the best e2e library for our case. Next up we’ll try to cover our impressions of Webdriver.io after subjecting it to a series of inspections.
It’s a next-gen browser and mobile automation test framework for Node.js, as they say. In reality it’s an extendible, feature rich product for writing tests using your favorite framework (Jasmine, Mocha, etc.) and targeting your modern webapps built with React, Angular, Vue.js (or any other web technology). And last but not least you can use different protocols and test runners in order to reach wide coverage of the modern web browsers. Looks promising so let’s go to see it in action.
Note: The purpose of this article is to review the different testing libraries and provide you some feedback in order to help you choose the right tool for your case. Thus you cannot rely on this article as a tutorial or step-by-step guide, instead you can go and read the official docs.
Webdriver.io is built with Node.js so it’s not a surprise that the easiest way to install it is by using the npm one-liner, as expected. Although we don’t want to fall into specific details, they can change in time so we prefer to point you to the official guide for the latest version of the instructions. Once you’re ready with the initial setup and the installation of all dependencies we can write our first basic test, and check whether everything is running fine.
describe('Home page', () => {
it('should load successfully', () => {
browser.url('/');
const heading = $('.description h1');
expect(heading).toHaveText('Hello world');
})
})Once again, we start with a simple test - navigating to the homepage and checking whether a specific text exists. Webdriver allows us to choose which testing framework to use and since the official guidelines rely on Mocha, we decided to stick our examples to that. Because of this we inherit its BDD syntax and the descriptive style guide, so you can easily understand every line of your tests. As expected element selection is performed by a regular query selectors, so nothing surprising here. Then you can use the rich set of matchers provided by the expect assertion in order to perform some validations.That’s the basics for now, let’s try to run our first test.
"spec" Reporter: ------------------------------------------------------------------ [chrome 83.0.4103.61 linux #0-0] Spec: /home/valentin/Documents/MTR Design/2020/reachadvance-e2e-tests-source/webdriver.io/test/specs/guest/home.js [chrome 83.0.4103.61 linux #0-0] Running: chrome (v83.0.4103.61) on linux [chrome 83.0.4103.61 linux #0-0] Session ID: d883eeef20c13c43a4820925a4d956f7 [chrome 83.0.4103.61 linux #0-0] [chrome 83.0.4103.61 linux #0-0] Home page [chrome 83.0.4103.61 linux #0-0] ✓ should load successfully [chrome 83.0.4103.61 linux #0-0] [chrome 83.0.4103.61 linux #0-0] 1 passing (1.5s) Spec Files: 1 passed, 1 total (100% completed) in 00:00:05
Once again when we start the test runner we see a browser window and all of the interactions with the webpage, then it closes and there is an output with the results of the execution in the console. A new browser window opens for every test, which makes the process a bit slower, but it ensures the isolated environment for every suite. If some of your tests fail you can debug the problem by yourself relying on the console output. There are no fancy features like “time-travel” (as it is in Cypress) but you can use the implemented debugging tools to ease the process. So far so good, time to write a few more tests.
We’ll follow the same path as we did in our previous articles, now when we have Webdriver up and running we’ll move on to test some website forms. Tradition dictates to start with the Sign Up form because of its rich set of input components. But before we jump straight to the test code itself we have to do some prerequisite steps. Webdriver.io is designed to use Page Object Pattern which in my opinion is a great way to add an abstraction layer to your test suites. The goal of using page objects is to move any page information away from the actual tests. Ideally, you should store all selectors or specific instructions that are unique for a certain page in a page object, so that way you still can run your test after you've completely redesigned your page. So we did it, here is our Sign Up page wrapper.
export class Page {
constructor() {}
open(path) {
browser.url(path)
}
// Other shared methods will reside here soon
}
class SignUpPage extends Page {
get inputFirstName() { return $('#registration--first-name') }
get inputLastName() { return $('#registration--last-name') }
get inputCompanyName() { return $('#registration--company-name') }
get inputEmail() { return $('#registration--email') }
get inputPhone() { return $('#registration--phone') }
get inputCity() { return $('#registration--city') }
get inputState() { return $('#registration--state + .select2') }
get inputMarketingStrategies() { return $('#registration--marketing-strategies + .select2') }
get submitBtn() { return $('form button[type="submit"]') }
get alertSuccess() { return $('.alert.alert-success') }
get alertErrors() { return $('.alert.alert-danger') }
open() {
super.open('signup');
}
submit() {
this.submitBtn.click();
}
}
export default new SignUpPage();We decided to use some inheritance instead of duplicating small snippets across all future pages, so that’s the reason we define a simple Page class first. Soon enough we’ll extend it with more base methods, but for now it’s enough. Then we create our Sign Up page and define selectors for all of our input elements, buttons and presentation divs which will be used in the tests later. This will help us make our tests look more focused and well described, stripping all of the overhead that query selectors carry. Also this makes our tests more flexible, we can adapt easily to any changes of the UI markup simply by updating the page object. Last but not least we add some basic methods for interactions with the page object itself, but let’s move on to the tests so you can see all of this in action.
import SignUpPage from '~/pages/signup.page';
describe('Sign Up page', () => {
// ...
it('should submit the form successfully', () => {
SignUpPage.open();
SignUpPage.inputFirstName.setValue(faker.name.firstName());
SignUpPage.inputLastName.setValue(faker.name.lastName());
SignUpPage.inputCompanyName.setValue(faker.company.companyName());
SignUpPage.inputEmail.setValue(faker.internet.email());
SignUpPage.inputPhone.setValue(faker.phone.phoneNumber());
SignUpPage.inputCity.setValue(faker.address.city());
SignUpPage.inputState.click();
$('.select2-results__option:nth-child(3)').click();
SignUpPage.inputMarketingStrategies.click();
$('.select2-results__option:nth-child(1)').click();
SignUpPage.submit();
expect(SignUpPage.alertSuccess).toHaveText('Your submission has been successfully submitted.');
});
});Looks pretty good, right? All of our predefined elements are chainable with the built-in element methods so you can use them directly without any additional overhead. Yet, the code looks clean and all of the actions are well described. You may see how valuable this approach is when you start to add more complex tests which work on multiple pages, it keeps the context of every action and is a way easier to read and understand the test cases. Similarly we use faker.js in order to generate dummy test values increasing the variability of our tests on every run. And the last thing in this test example is the way we interact with the custom input elements on the page. We already have some negative experience with the select2 elements (you can read more about our problems in the previous article) so we knew that this part will be tricky.
Note: If you’re now aware of its capabilities, Select2 is a 3rd party library which provides an extended and customizable version of the standard html element. It comes with support for searching, tagging, remote data sets, infinite scrolling, and many other highly used options. We use it a lot in our project so it’s crucial to be able to interact with the tests. Generally it’s a great tool so If you’re still not convinced of its powers we strongly recommend you to give it a try.
Back to the board, our first try was unsuccessful, select2 component is way different than its parent and yes, you can’t interact with it using the built in methods. However we found a way to select a value simply by interacting with the UI directly. You can see that we select the element and then click on one of its options. We hope that this will do the job for your case too, it was definitely enough for us.
"spec" Reporter: ------------------------------------------------------------------ [chrome 83.0.4103.61 linux #0-0] Spec: /home/valentin/Documents/MTR Design/2020/reachadvance-e2e-tests-source/webdriver.io/test/specs/guest/signup.js [chrome 83.0.4103.61 linux #0-0] Running: chrome (v83.0.4103.61) on linux [chrome 83.0.4103.61 linux #0-0] Session ID: 32c939cf714f992bc3565a15e9047761 [chrome 83.0.4103.61 linux #0-0] [chrome 83.0.4103.61 linux #0-0] Sign Up page [chrome 83.0.4103.61 linux #0-0] ✓ should report form errors on submit if provided empty form [chrome 83.0.4103.61 linux #0-0] ✓ should report form errors on submit if provided invalid data [chrome 83.0.4103.61 linux #0-0] ✓ should submit the form successfully [chrome 83.0.4103.61 linux #0-0] [chrome 83.0.4103.61 linux #0-0] 3 passing (10.2s) Spec Files: 1 passed, 1 total (100% completed) in 00:00:14
The test finished successfully and all of the actions were executed as expected so we can mark this part of the job as done. We have to mention the speed time as well, it works fast enough but we have to wait how it will go when the number of test suites increases. Since we have passed this threshold successfully we can move on and start to cover the more complex scenarios.
We already have tested the Sign Up process which leads us to the next step - find a way to authenticate the users and test protected routes. Unfortunately there is no way to make a POST requests without using any plugins (you can try to use the InterceptService if you want) so our goal includes the following key points:
Starting from the top to bottom to solve each of those requirements. First up, remember when we defined our page object and told you we’ll use it later? Well, the moment has come, it’s the perfect time to go back and extend it. Since we want to define a method which can be reused across all test suites and it requires the browser’s object to interact with the webpage, we think that the main page object is the best place to do this.
export default class Page {
constructor() {}
open(path) {
browser.url(path)
}
login() {
browser.url('signin');
$('#email').setValue(browser.config.user.email);
$('#password').setValue(browser.config.user.password);
$('form input[type="submit"]').click();
}
}Yes, it’s that simple. Just fill the correct values and submit the form. As you can see the credentials are obtained from some browser config which is nothing more than the default “wdio.conf.js” which was created when you initialized the project. So there is no magic, just add your details and wire up the correct variable names in your tests. You can use this approach for anything you want to be somehow dependent on the environment and not hardcoded across the whole codebase. And finally we have to find a place to add our login methods instead of prefixing every single test. Well, it's not that hard of a decision since we are using Mocha and can take advantage of the built-in hooks, more particularly the “before” hook which is run before all tests in the suite. Here is how it goes, a basic example to test the accessibility of the Dashboard page.
import DashboardPage from '~/pages/profile/dashboard.page';
describe('Dashboard page', () => {
before(() => {
DashboardPage.login();
});
it('should show the Dashboard page', () => {
DashboardPage.open();
expect(DashboardPage.breadcrumb).toHaveText('Dashboard');
});
});Once again, we use our new login method in the hook before the execution of all tests, and as you can see we access it as a member of the Dashboard page object, because it inherits all of the main page object properties and methods. Then we simply check whether we can see the Dashboard breadcrumb in order to assume that the user was authenticated and the page was loaded successfully. We already covered most of our requirements and Webdriver seems like an easy to go framework with great capabilities, but there are a few more steps before we can figure out our conclusion.
"spec" Reporter: ------------------------------------------------------------------ [chrome 83.0.4103.61 linux #0-0] Spec: /home/valentin/Documents/MTR Design/2020/reachadvance-e2e-tests-source/webdriver.io/test/specs/profile/dashboard.js [chrome 83.0.4103.61 linux #0-0] Running: chrome (v83.0.4103.61) on linux [chrome 83.0.4103.61 linux #0-0] Session ID: 5fd1513ac22289d827aa940964a52c8d [chrome 83.0.4103.61 linux #0-0] [chrome 83.0.4103.61 linux #0-0] Dashboard page [chrome 83.0.4103.61 linux #0-0] ✓ should show the Dashboard page [chrome 83.0.4103.61 linux #0-0] [chrome 83.0.4103.61 linux #0-0] 1 passing (14.7s) Spec Files: 1 passed, 1 total (100% completed) in 00:00:19
We’re ready to jump in our most complex scenario so far. Let’s take a moment to describe our goals. We’re going to access a protected route and interact with a bunch of new components and elements including a table, checkboxes and a modal. Here is a basic step-by-step breakthrough:
it('should assign multiple tags to a contact', () => {
ListContactsPage.open();
for (let index=1; index <= 3; ++index) {
ListContactsPage.contactsTable
.$(`tr:nth-child(${index}) td input[type="checkbox"]`)
.click();
}
ListContactsPage.manageTagsBtn.waitForDisplayed();
expect(ListContactsPage.manageTagsBtn.isDisplayed()).toBe(true);
ListContactsPage.manageTagsBtn.click();
ListContactsPage.tagsModal.waitForDisplayed();
expect(ListContactsPage.tagsModal.isDisplayed()).toBe(true);
const tag1 = faker.random.word();
const tag2 = faker.random.word();
ListContactsPage.tagsSelect.click();
ListContactsPage.tagsSelect.$('input.select2-search__field').addValueToSelect2(tag1);
expect(ListContactsPage.tagsInput.getValuesOfSelect2()).toEqual([tag1]);
ListContactsPage.tagsSelect.$('input.select2-search__field').addValueToSelect2(tag2);
expect(ListContactsPage.tagsInput.getValuesOfSelect2()).toEqual([tag1, tag2]);
ListContactsPage.submitTagsModal();
expect(ListContactsPage.tagsModalAlertSuccess).toHaveText('The tags have been successfully changed.');
ListContactsPage.tagsModalCloseBtn.click();
ListContactsPage.tagsModal.waitForDisplayed({ reverse: true });
expect(ListContactsPage.tagsModal.isDisplayed()).toBe(false);
for (let index=1; index <= 3; ++index) {
expect(ListContactsPage.contactsTable.$$(`tr:nth-child(${index}) td .badge`).length).toEqual(2);
expect(ListContactsPage.contactsTable.$$(`tr:nth-child(${index}) td .badge:nth-child(1)`)).toHaveText(tag1);
expect(ListContactsPage.contactsTable.$$(`tr:nth-child(${index}) td .badge:nth-child(2)`)).toHaveText(tag2);
}
});
Once again we’re going to use a page object to interact with the different elements. We’ll omit to post here the definition because it doesn’t contain anything different than the first example above, instead we’ll focus on the workflow itself. Once the page has been opened and loaded (after the user has been authenticated) we use a regular loop to iterate through some of the table rows and click the corresponding check boxes in order to select them. Then we open a modal and wait for it to be displayed because its content is fetched asynchronously. And here we go again, time to interact with the custom select2 element, but this time we have to add new values manually (previously we used one of the existing options). And once again simply using the general select element which is controlled by its enhanced successor was not enough, we had to interact with the UI (which actually is not that bad, because it’s the way the users will potentially use it).
/**
* Gets executed before test execution begins. At this point you can access to all global
* variables like `browser`. It is the perfect place to define custom commands.
* @param {Array.} capabilities list of capabilities details
* @param {Array.} specs List of spec file paths that are to be run
*/
before: function (capabilities, specs) {
// Custom command to focus a specific element
browser.addCommand('focus', function () {
browser.execute(function (domElement) {
domElement.focus();
}, this);
}, true);
// Custom command to type a specific value in a select2 input
browser.addCommand('addValueToSelect2', function (value) {
this.focus();
browser.keys(value);
browser.keys("\uE007");
}, true);
// Custom command to get the selected values in select2 input
browser.addCommand('getValuesOfSelect2', function () {
return this.$$('option[data-select2-tag="true"]').map(option => option.getValue());
}, true);
}We’re going to use another built-in approach provided by Webdriver.io to define some additional commands and extend the default set of abilities for interactions with the elements. In the example above you may notice three new commands. The first one is used to focus a specific element in the browser, we found it as a mandatory pre-condition when we want to interact with the select2 so we decided to define it as a standalone command in terms of simplicity of the code. The second one is used to add a new value to a select2 component, and if you see it does the same 3 steps as you would do as a user if you want to do the same thing. You’ll select the element, type some text via your keyboard and then press
"spec" Reporter: ------------------------------------------------------------------ [chrome 83.0.4103.61 linux #0-0] Spec: /home/valentin/Documents/MTR Design/2020/reachadvance-e2e-tests-source/webdriver.io/test/specs/profile/contacts/tags/manage-multiple.js [chrome 83.0.4103.61 linux #0-0] Running: chrome (v83.0.4103.61) on linux [chrome 83.0.4103.61 linux #0-0] Session ID: dc4da0753fc4c8eca11f2356a1d613be [chrome 83.0.4103.61 linux #0-0] [chrome 83.0.4103.61 linux #0-0] Dashboard page [chrome 83.0.4103.61 linux #0-0] ✓ should show the Contacts page [chrome 83.0.4103.61 linux #0-0] ✓ should assign multiple tags to a contact [chrome 83.0.4103.61 linux #0-0] [chrome 83.0.4103.61 linux #0-0] 2 passing (14.5s) Spec Files: 1 passed, 1 total (100% completed) in 00:00:21
Well, we can officially say that Webdriver.io passed all of our base tests and as we did with Cypress before, we’ll stop here. We’re happy to announce that we liked it, the way we write our code and how the test runner behaves. The project comes with great docs so you can always find the answer to your questions when needed. Also, there is a well developed society around the community forums so you can learn from the experience of the other before you. Definitely we can say it’s a well designed and developed testing tool which deserves your attention if you’re on a search for a new tool, as we did. You don’t have to hesitate anymore, give it a try and let us know what you think.
Our journey comes to an end and we think it will be good if we try to summarize our conclusions. We’ve reviewed three different tools which provide different approaches for solving the same use-cases. We’ve collected some experience and shared it with you but... What about if we try to compare them all in a single view and let you choose for yourself. We’ll cover the most important aspects (for us) and try to rate them from 1 to 3, here is what it looks like.
| Cypress | Nightwatch.js | Webdriver.io | |
| Documentation | 3 | 1 | 2 |
| Community | 3 | 1 | 1 |
| Code | 3 | 3 | 3 |
| Speed | 2 | 3 | 3 |
| Debug | 3 | 1 | 2 |
| Test Runner | 3 | 1 | 1 |
| Completeness | 3 | 0 | 3 |
| Total | 20 | 10 | 15 |
And the winner is… Cypress (cymbal splash) with its perfect score of 20pts. Of course this score is well deserved since the tool did so well during our tests in all aspects. Initially they provide a great documentation and a wide community of supporters but actually you don’t need to rely so much on them, because everything happens easily enough and the whole process was really smooth. We can’t miss to mention again the great test runner and debugging options they provide with the handy time-travel feature. After all, we managed to implement all of our base tests with Cypress which makes it suitable for our needs and we can rely on its services in the future.
As you expected, the score of Nightwatch.js is significantly lower but one of the main reasons for that is the lack of completeness, we couldn’t manage to implement all of our base tests. Although it’s a full featured project so you don’t have to rely only on our rate, give it a try and who knows, it may serve you better than us.
And last, but not least - second in our scoreboard comes the last tool we reviewed above in this article - Webdriver.io. With results in the middle it fully describes our impressions - a really good tool which covered all of our requirements and behaved really well during our tests. What takes it away from the first place is the better test runner and the handy UI which Cypress offers, so nothing bad at all with Webdriver itself. Actually we really liked the Page Object pattern and the way it makes our tests contextually dependent. So we definitely will suggest you to use this tool if you want a bit more speed and the cool UI interface doesn’t fool you, the decision is up to you.
After all we’re glad of the work we’ve done... and that we started it at all. Not only that we found a tool for our job but we extended a bit our horizon and gained some experience which is never a superfluous resource. We hope that our results will help you decide and save you some time. At least you can base your own opinion on our experience and the thoughts we shared. That’s all from us for now, stay tuned for our future projects and updates.
Struma No 2
Dobrich 9300
+359 899 181044
3 East Point, High Street
Seal, Sevenoaks
Kent TN15 0EG
+44 7512 320760