User:SimTran/Cypress vs WebdriverIO: What are the differences?

1. Some facts about Cypress

Cypress is a next generation front end testing tool built for the modern web. [1]

Cypress ecosystem consists of a Test Runner and a Dashboard Service for recording the tests. [1]


 * Cypress does not use Selenium.
 * Cypress focuses on doing end-to-end testing
 * Cypress works on any front-end framework or website.
 * Cypress tests are only written in JavaScript.
 * Cypress is all in one
 * Cypress doesn’t support cross browser testing, only support Chrome
 * Cypress doesn’t support native mobile apps, but they do intend to support mobile web browsers in the future

2. Cypress vs WebdriverIO

Below is a comparison between Cypress and WebdriverIO, it shows the pros and cons of each in 4 tasks:


 * Setting up tests
 * Writing tests
 * Running tests
 * Debugging

The download trend of Cypress and WebdriverIO in past 1 year. We can see that number of Cypress downloads are growing more than WebdriverIO downloads. At the beginning, they are the same but now, Cypress are 2 times higher. [2]

Practical experiment

To verify the above comparisons, I conducted a practical experiment: automation tests for Homepage page, using both Cypress and WebdriverIO [3]. There are a total of 16 tests used to test the modules and sub-modules on the Homepage page and their operation. The tests in Cypress and WebdriverIO have exactly the same scenarios and follow the same steps. There are some different behaviors between Cypress and WebdriverIO that occurred in this experiment:

- Steps of setting up and running tests:

Setting up:

·        Cypress: installing with “npm install cypress”, all necessary dependencies are installed

·        WebdriverIO: installing webdriverio and then other necessary dependencies, browser driver.

Running test:

·        Cypress: open Cypress: “$(npm bin)/cypress open” and select the test to run for the first time. After that, the test is run automatically each time code is changed

·        WebdriverIO:

o  Run chromedriver

o  Run tests for every test run

- The number of lines of code of Cypress and WebdriverIO are not much different. With every command in the WebdriverIO, we can almost find a corresponding command in Cypress to replace. But in some cases, Cypress is better at having some extra commands such as checking/ unchecking checkbox. That makes Cypress code is easier and shorter to write.

Cypress code: {{Code|style=} unselectDefaultHomepageCheckbox{ this.scrollToHomepageCheckBox; cy.wait(500); this.homepageBox.check; cy.wait(500); this.defaultHomepageBox.uncheck; this.save.click; } }

WebdriverIO code: unselectDefaultHomepageCheckbox{ this.scrollToHomepageCheckBox; this.homepage.waitForVisible; if (!this.homepageBox.isSelected) {       this.homepage.click; this.save.click; }else{ this.defaultHomepage.waitForVisible; if (this.defaultHomepageBox.isSelected) {          this.defaultHomepage.click; this.save.click; }    } }

- The test runner: this is a highlight of Cypress compared to WebdriverIO.

As we can see in the screenshot [4], when the test runs, the commands that are executed are shown in the side panel, while they are being executed. Moreover, once the test ends, we can click on each command in the side panel, and we can see a screenshot of how the page looked like when the command was executing. This is called Time traveling.

This is phenomenal for debugging a test! Cypress screenshots and seeing how the tests progress is a marvelous way to debug the tests, and a marvelous way to start to understand why the test failed. The whole experience helps in writing, understanding, and debugging tests, and in analyzing their results.

- Test run time: I ran the 16 tests of Cypress and WebdriverIO 10 times and took the average value. The results are: Cypress took 690 seconds to complete the entire tests while the WebdriverIO took 590 seconds to complete. WebdriverIO runs a little faster than Cypress (however, one WebdriverIO test run did not finish yet)

- Consistent Results: Cypress has the same result in 10 test runs while WebdriverIO has 6 different results in 10 test runs. Even in one test run, WebdriverIO was unable to complete all 16 tests when it failed at "before each" hook of sixth test.

- Special failed cases: there are some special failed cases in Cypress which were passed in WebdriverIO

·        Verify text of username in heading (4th test)

·        Xpath was changed in Cypress browser leads to test failure (10th test) Conclusion:

As we can see, on the surface, although the two tools Cypress and WebdriverIO – seem similar, there are actually many differences between the two. Although Cypress still has some limitations compared with WebdriverIO, but for end-to-end testing, these limitations are mostly not a big problem, and enable Cypress to give frontend developers the advantages that they do need: speed and consistency. In my opinion, Cypress is better than WebdriverIO in end-to-end testing

Full report in Google Docs: https://docs.google.com/document/d/1Qc88RmeReBBEUVuzJHKpevmmsNdn212PKvLihey_9xw/edit?usp=sharing

References:

[1] https://www.cypress.io/how-it-works

[2] https://www.npmtrends.com/cypress-vs-webdriverio

[3] https://github.com/simkawaii/GrowthExperiement_Cypress_WebdriverIO

[4] https://docs.cypress.io/guides/core-concepts/test-runner.html#Overview