News

Over the years we have worked on tons of different projects and lots of them challenged us to acquire new skills. Our team always embraces the chance to improve and develop the services we provide.

We mainly focus on satisfying the functional requirements of our clients (and getting things done) but we always keep in mind the importance of testing our products especially when it comes to big projects with longer development time and dynamically changing requirements. Taking this into account we decided to try a new testing approach in one of our latest projects.

The project in question is a marketing platform for real estate professionals that combines social media posting & scheduling, image/video content creation, turnkey websites, and email & SMS marketing. As for the tech stack - the main web application is built with Laravel + Vue.js, with a bunch of microservices written in Python and Go.

We have good experience with the standard approaches for testing such web projects, both the smaller and granular unit and feature http tests with PHPUnit, and the more complex browser tests with tools like Behat or Dusk. All of those methods are great but we have experienced one particular problem in the past, the bigger the project - the slower the build. Yet we couldn’t skip the testing process so we decided to try a different approach. In order to ensure the quality of the application and to reduce the regression bugs we would stick only to e2e testing. This time we wanted to build a standalone testing environment which would not depend on the code itself. Our ultimate goal is to be able to test the system as it is available to the end users, so we can ensure that the onboarding process of every new user is working as expected. In particular we wanted to cover all main cases of usage and the whole array of features in the specific user journey workflow.

Last but not least we wanted to experiment with new tools and to extend our tech stack. The JavaScript world is still booming and there are a lot of great services out there, so we decided to see what the market is offering and choose the best e2e testing library to build a standalone automated testing environment instead of the repeatable manual user testing cycle. And so began our search for the best tool we could use in our case.

Chapter 1: Cypress

Our first choice was Cypress. It was allegedly a fast, easy and reliable testing tool for anything that runs in a browser. Besides the expected testing features (such as toolkit with wide range of assertions and possibilities to write tests, test runner with support for multiple browsers and different types of reports generation) this library provides a few game changers which makes it stand out. Cypress takes snapshots as your tests run so you can time travel to see exactly what happened at each step of your tests. It offers great debugging capabilities as well. Stop guessing why your tests are failing - debug directly from familiar tools like Chrome DevTools, which provides readable errors and stack traces making debugging lightning fast. Furthermore there is built-in support for taking screenshots of the failed tests and even videos of entire test suites (handy, right?). All of those marketing hooks seemed attractive enough so we decided to give Cypress a try.

Getting started with Cypress

Note: The purpose of this article is to review the different testing libraries and provide you with some feedback so you can choose the right tool for your case. Thus you cannot consider that this article is a tutorial or step-by-step guide. If you need some official guidelines you can visit the Cypress’ page and read the official docs.

Our first task is to download and set up the required dependencies and because we’re in the JavaScript world now we simply have to rely on npm (at least it was our choice, you can see how to do it or what are all available methods here). It may take a while (depending on your internet connection) but soon you’ll be ready to write your first test.

describe('Home Page', () => {
  it('should load successfully', () => {
    cy.visit('/');
 
    cy.get('.description h1')
      .should(($item) => {
        expect($item).to.contain('Hello world!');
      });
  });
});

As you can see it comes with clean and descriptive BDD syntax which makes it easy to read and understand every step of your tests. You can interact with the web page with standard query selectors and use a predefined set of assertions in order to verify whether a specific action has been performed, i.e. to verify whether the expected results are met. One more thing to acknowledge - there is built-in smart automatic waiting. You don’t have to add waits or sleeps to your tests (it’s kind of a nightmare with all of those asynchronous actions). Cypress automatically waits for commands and assertions before moving on which in my opinion makes the job much easier (yeah, no more async hell). Well, we’re ready to move on and run our first test, I know you’re all curious to see the test runner.

As you can see on the screenshot above, it does not simply run a browser and execute the set of actions in the tests, it also provides a UI wrapper with details for all tests and steps. You can click on each action and the preview will automatically time travel to the captured snapshot so you can inspect carefully and see the state of your app at every moment (usually the other test services simply report the error in the console and you have to replicate the issue by yourself). This tool is helpful first when you write scenarios or later when you have to debug some of the failed tests but you can omit it and start the test runner headlessly when you are ready to integrate it into your build process.

Let’s add some real world tests

We’ve got Cypress up and running so we can go further and test some real world scenarios containing more interactions with the webpages. Usually you’ll have at least one form in your website (in our project we’ve got a lot) so it’s a good start to test the capabilities of the framework. For example we will use our Sign Up form because it’s made up of fields of different types, and gives us plenty of different components to work with.

import faker = require('faker');
 
describe('Sign Up Page', () => {
 
  const formValues = {
    firstName: faker.name.firstName(),
    lastName: faker.name.lastName(),
    companyName: faker.lorem.company(),
    email: faker.internet.email(),
    phone: faker.phone.phoneNumber(),
    city: faker.address.city(),
    state: 'California',
    marketingStrategies: ['Website']
  };
 
  it('should submit the form successfully', () => {
    cy.visit('/signup');
 
    cy.get('#registration--first-name')
      .type(formValues.firstName)
      .should('have.value', formValues.firstName);
 
    // skipped some input fields...
 
    cy.get('#registration--state')
      .select(formValues.state, { force: true })
      .should('have.value', formValues.state);
 
    cy.get('form').submit();
 
    cy.get('.alert.alert-success')
      .should('be.visible')
      .should(($alert) => {
        expect($alert).to.contain('Message for the results');
      });
 });
 
});

The workflow is simple and straightforward - visit a page, select the different input fields, add some test values, finally submit the form and check for a successful message. You may notice that we are using faker.js in order to generate dummy test values. This way we don’t stick to hard-coded values which will remain fixed forever. Also, by using different values on every execution, we may increase our chance to catch some corner cases. The other thing in the above example which may draw your attention is that we force the selection of the select element value. The reason for this is that we’re not using a standard select element for this particular field, instead we rely on the select2 library which provides a more complex component with a rich set of additional features (if you’re not familiar with it you should definitely take a look). Still it’s a select element. Well with a lot of additional helper markup, but yet a select tag so we can target it the same way as the standard html elements. The only difference is that we have to force the change of the value because it has to propagate the event to the whole chain.

it('should report form errors on submit if provided empty form', () => {
   cy.visit('/signup');
 
   cy.get('form').submit();
 
   cy.get('.alert.alert-danger').should('be.visible');
 
   cy.get('.alert.alert-danger ul > li')
     .should(($list) => {
       expect($list).to.have.length(8)
       expect($list.eq(0)).to.contain('The first name field is required.');
       expect($list.eq(1)).to.contain('The last name field is required.');
       expect($list.eq(2)).to.contain('The company name field is required.');
       expect($list.eq(3)).to.contain('The email field is required.');
       // ...
     });
 });

Of course sometimes things mess up so we won’t forget to add a bunch of negative tests. We will use the occasion to show the handy syntax when you want to assert multiple expectations of fields which belong to the same query selector. Before we dive deeper into even more complex test scenarios have a break and take a look again at the awesome test runner.

Test protected routes and complex workflows

Our tests are executed in an isolated environment which slightly differs from the real website usage. In this case there are a lot of prerequisite steps which are required in order to test some scenarios. One of the most common examples is that of features hidden behind protected routes. As a user you have to login to the site once and then you can use all of its features but we can’t apply the same logic when we write our tests. Also we can’t afford to open the login UI, to fill and submit the form before every test, this will add an enormous overhead which will significantly slow down the whole process. Fortunately there is a solution how to avoid such complications - Browser Requests and Custom Commands.

Cypress.Commands.add('login', (user) => {
  if (!user) {
    user = Cypress.env('user');
  }
 
  cy.request('/signin')
    .its('body')
    .then((body) => {
      const $html = Cypress.$(body)
      const csrf = $html.find('input[name=_token]').val()
 
      cy.request({
        method: 'POST',
        url: '/login',
        failOnStatusCode: false,
        form: true,
        body: {
          email: user.email,
          password: user.password,
          _token: csrf,
        },
      }).then((resp) => {
        expect(resp.status).to.eq(200);
      });
 });
 
});

Cypress comes with its own API for creating custom commands which can be used to define a specific set of actions reusable across your test suites. This allows us to define a login command and use it later when we have to test some of the features for the authenticated users. Also, you can see that this command is not interacting with the UI (almost*). Then we send a direct POST request to the login route with the valid credentials in order to authenticate the user without any interaction with the UI (at least we don’t type in values in the input fields). Now we can use the command in our tests before the actual test logic.

*Note: We have to fetch the markup of the login form in order to obtain the security csrf token, otherwise we can’t submit the form successfully. But this is only because we’re using Laravel, you can omit this part if you can authenticate your users only with their credentials.

beforeEach(() => {
  cy.login(user);
});

Cypress support hooks, blocks which are executed before/after a specific test/suite (you may be familiar with them, such hooks are widely supported across many other testing frameworks like Jasmine, Mocha, etc.) and we think it’s the best place to use our login command. This way we ensure that all of the tests inside the current suite will be executed for successfully authenticated users without unnecessary code duplications. You may notice that we are using another unknown method in the command snippet above. Cypress comes with a built-in support for environment variables so you can use them to provide some non-static data to your tests. In our case, we use env variables to obtain the default user credentials so we don’t have to hardcode any production data in the codebase.

{
  "baseUrl": "http://localhost:8000",
  "user": {
    "email": "user@mtr-design.com",
    "password": "our-secure-password"
  }
}

Once we are ready with all of those pre-conditions we can finally try to execute some slightly more complicated tests. Our goal is to verify whether Cypress is able to interact with as many different page components as possible. We’ve got a perfect match for this job, here is the workflow that we’ll try to cover:

1. Login to the application (use the command instead of real login via the UI)
2. Go to a specific listing page with multiple records
3. Interact with table element, select a few rows via checkbox elements
4. Open a modal and wait it to fetch its body from the server (it’s dynamic, so again this is an async operation)
5. Use a Select2 component in order to apply some changes and submit an inline form. Wait for the response of the async submission, verify the results and close the modal.
6. Verify whether the performed changes are applied by inspecting the UI of the page after reload.

it('should assign multiple tags to a contact', () => {
   cy.visit('/profile/contacts');
 
   for (let i=1; i <= 3; ++i) {
     cy.get('#contacts-table tr').eq(i)
       .find('td input[type="checkbox"]')
       .check()
       .should('be.checked');
   }
 
   cy.get('.btn-manage-tags').click();
   cy.get('#modal-global').should('be.visible');
 
   const tag1 = faker.random.word();
   const tag2 = faker.random.word();
 
   cy.get('#contact--tags + .select2')
     .click()
     .find('input.select2-search__field')
     .type(`${tag1}{enter}`);
 
   cy.get('#contact--tags')
     .invoke('val')
     .should('deep.equal', [tag1]);
 
   cy.get('#contact--tags + .select2')
     .click()
     .find('input.select2-search__field')
     .type(`${tag2}{enter}`);
 
   cy.get('#contact--tags')
     .invoke('val')
     .should('deep.equal', [tag1, tag2]);
 
   cy.get('#form--contacts-tags')
     .submit();
 
   cy.get('#form--contacts-tags .alert.alert-success')
     .should('be.visible')
     .should(($alert) => {
       expect($alert).to.contain('The tags have been successfully changed.');
     });
 
   cy.get('#form--contacts-tags .btn[data-dismiss="modal"]').click();
   cy.get('#modal-global').should('be.not.visible');
 
   for (let i=1; i <= 3; ++i) {
     cy.get('#contacts-table tr').eq(i)
       .find('td .badge')
       .should(($badges) => {
         expect($badges).to.have.length(2);
         expect($badges.eq(0)).to.contain(tag1);
         expect($badges.eq(1)).to.contain(tag2);
       });
   }
});

The purpose of this code snippet is to perform the actions described above. You may notice a few new ways to interact with the UI - iteration through multiple objects of the same type via loops and visibility check. In the current case we have to select multiple table rows and we don’t have to copy the code N times in order to be sure of the synchronous execution of the statements. We simply can rely on a regular for loop and it does the trick. Also in this test case we introduce another assertion - we check whether a specific element is visible or not. There’s nothing special about it but reminds me to give you the complete list of supported Cypress Assertions so you can get familiar with it. And last but not least, take a look again at the test runner and the execution of all those complex steps. Enjoy.

We can spend more time exploring the abilities of the Cypress library but our goal here is to check multiple services and choose the best one that works for us. So it’s time to stop here. We performed various tests and learned how to use basic features and how to construct more complex scenarios. All of those observations are sufficient to conclude that Cypress is a great tool, built with a lot of effort and we would be glad to use it in our project later and so on. It comes with a seamless learning curve, great documentation and a wide community so you can always find the answer of your questions easily.

That’s it. Before we proceed to the next tool in our list we’ll use the moment to suggest you again to take a look at the Cypress project and give it a try. Let us know if you do this and what your impressions are. We’re curious to hear your experience and whether you liked it or not.

Read part 2 of "On a way to find the perfect e2e testing tool"