I’ve had to use Cypress at work, and I really don’t like it. I find the command chaining and assertion queuing to be very unintuitive. (“Do I chain this one or not? Maybe stick it in a callback? No idea!”)
It’s been a few months since I’ve touched it, so I’m struggling to give more specific criticisms.
I’ve had a really great time using Kent Dodd’s DOM Testing Library [1] for unit tests. The readability and writability of these tests has been fantastic. I wish I could use the exact same API for e2e tests. It looks like they actually do have a Cypress version of the Testing Library [2], but it seems to suffer from the same Cypress-isms that I was just complaining about.
Has anyone else had a similar experience? It seems like I’m in the minority opinion. Maybe I’m just bothered because I don’t understand Cypress well.
I spent a few weeks with it, and it was very frustrating.
The programming model is very odd. I found it next to impossible to share any code between tests, and most of the programming skills I've learned over the decades didn't seem to apply.
It is vastly superior to anything else we've seen in stability and polish. If I only could write reasonable code for it, things would be great...
I found their idea about defining commands very odd, but when we ignored these entirely and just went with the good old page pattern, the tests became very tame again. All the cypress-internals are done in small methods within page-objects, the tests contain only the abstract page-object-methods.
But yes, when we tried to run with the patterns they advertise, the tests became indeed hard to write and read; maybe we did it all wrong but it's at least consistent with the experience expressed by you and others in the thread here.
The one slight gotcha is if you go for the page object pattern the intuition is that each page object holds a reference to the cypress element that formed it so you can do new LoginForm(cy.get('#loginForm')) or similar which is a mistake because of the way cypress chaining works - it will result in only being able to use a single method once on the page object. Instead you need to query fresh again each time.
Which is fine when the page object is a login form, but a bit more comfortable complicated when it's notification 5 or something.
Good point, I'd gotten used to that and forgotten how it is kind of counterintuitive at first. Our base test helper object does this:
get root() {
// It is important here to run cy.get()
// each time this property is accessed.
// Lots of methods of this class chain
// off of `this.root` so it needs to be
// re-evaluated every time.
return cy.get(rootComponentCss);
}
find(selector: string, options?: any) {
return this.root.find(selector, options);
}
...so that component-specific test helper objects (subclasses) can just do something like:
get sendMissileAlertButton() {
this.root.find('.x-hawaii-missile-incoming');
}
That way, it looks up the target element every time it is used in a test.
I've compared a few thousand tests in each testing library from real companies and I've found Cypress to be with "as many defects as Selenium in real projects".
Most of the other open source tools (not to mention enterprise tools like Testim.io where I work) support the things that make Cypress "fast" like mocking network requests or automatically waiting for elements. I wrote an article about this somewhere I can dig it if you want.
If you have data showing differently I'd love to see it.
Did you ever need to mix cypress tests with anything else? Say for example I wanted to have cypress click through a frontend, and then make a separate backend call to verify something happened in the same test run. Is there a clean way to do that in a cypress context, or is playwright / puppeteer going to be a better fit?
What's your backend written in? I'm in a similar position where work is pushing cypress, but there is no library for cypress in our backend language, there is for playwright. I want to do something in the frontend, and check the backend, so it would be nice to use our existing backend language for both so we have access to our internal library. I was told we could write a bash script and have cypress exec. Of course we can do a script in any language, but suddenly I'm seeing a bash QA library emerging at my work. Does that sound absurd to anyone else?
What do you mean with "check the backend"? You can e.g. do assertions on HTTP-responses coming from the backend - is this what you head in mind?
If you want to test the backend on its own, I personally would recommend using a different framework than cypress (or a different framework than the one being used for frontend e2e testing in general). Otherwise you risk that people conflate these two things and start "testing the backend" in e2e-tests for the frontend, which are super slow.
Yeah, specifically that interfacing with the ui causes some side effects we would expect to happen (web hooks sent, reports generated, etc). I definitely wouldn't want to write all of our tests that way, because that sounds like a nightmare to maintain and run, but was considering a handful for covering a few major features. We do already have a very extensive backend integration test suite (http / rest interactions) that doesn't do anything with UIs, though.
Cypress is a recent addition for us, so this is more a curiousity than anything else. I've done similar fully e2e / functional tests with puppeteer in the past.
Our backend is in node / typescript, so it's theoretically possible to mix the two. I would be able to test what I need using REST APIs, though, so in theory the backend language doesn't really matter for me.
I haven't had much time to try anything yet, but cypress's override of promises and async / await makes me think it'll be a pain to try adding non-cypress-specific checks to a cypress test.
Ideally you should not combine Cypress Frontend end to end tests with backend testing.
You use Cypress to do something on the frontend, which triggers something on the backend and ideally that change is reflected on the frontend, you test that frontend change using Cypress.
For backend testing just directly call the backend API's and then check the database directly.
There is a difference between integration testing (Cypress UI click through UI) and backend unit testing.
We run integration tests with Cypress and then rspec or some other Rails testing platform for the backend.
Cypress uses fixtures so ideally you would not let it hit a backend API unless you don't care about reproducibility of the response. For example a PUT will augment the data on the backend so when you run the test again the backend data will probably give you a different response to assert and fail your expected.
Due to the limitations of Cypress you end up using them both.
Cypress for the simple tests, and for the more complex case, e.g. opening tabs, testing your SSO login paths, I frame widgets, you end up using Playwright. I spend quite a lot of time to even fork Cypress to add support for testing WebKit browsers. Probably was quicker to write a Cypress kind of solution for Puppeteer/Playwright
We have Cypress tests that call our database directly and get information out, call 3rd party APIs to get information back, and call our APIs we've specifically created ourselves to help Cypress tests. It's pretty flexible; I'm just not sure how much this flexibility might bite us in the future.
One thing I didn't see mentioned is to just have some extra routes in /tests/* that are only enabled for tests to call or return data from arbitrary back-end code.
cy.exec() lets you run general system commands iirc. Istr seeding a database with it before running tests, I think if that command fails with Nonzero, the test would fail.
What made it hard to reuse code? You can just import/export stuff like normal, unless you mean it was hard to find any logic that could be usefully extracted into a helper?
We have a lot of data in UI tables, so I wanted to write code that extracted the cell values to a ROWSxCOLUMNS matrix, that I could reuse in each test.
Did you try custom commands? I have some for more basic stuff like 'get data-value attribute for the element with data-name attribute of x' which I can use in any test.
That said, I find checking for specific values in the page against other page values to be really difficult so if you're doing that with the table... good luck!
if that functionality isn't already present in the UI, e.g. by selecting a filter or exporting, then perhaps it belongs in the domain of a scraper rather than a testing tool
I love Cypress overall, but wouldn't advocate using Cypress commands unless you are doing something weird.
Their examples show using Cypress commands for pedestrian things like "log in to app" but there is no need to get all Cypress-specific for things like that.
Instead, we typically write "test helper" objects for that, which are conceptually similar to the Page Objects pattern from the Selenium era. So like:
class GroupUiTester {
get groupChooser() {
return cy.get('#group-chooser-button');
}
}
Then, in your test, you can just be like:
tester.groupChooser.click();
That is kind of simplified, but even that would be useful if you had dozens of tests interacting with that group chooser element. (Especially since, unlike string-based "cy.get('.some-class.whatever')" it will all auto-complete in any decent editor.)
An example of "doing something weird" where we do use Cypress.Commands is like:
We do that because the local storage clearing breaks a third party lib we use for a few tests. But that is something you can't just do with the regular Cypress/DOM API, so it is a good candidate for using Cypress "commands".
we have extensive Page Objects which let us re use code everywhere. i’m not a huge fan of cypress commands because they pollute a public prototype and i don’t like modifiying those
Basically, I think all the code to locate UI elements and perform common interactions with them should be in re-usable utility code, so that the actual tests can focus on just testing what they test.
I don't think the Cypress docs probably do a great job emphasizing how easy this is (especially since they go out of their way to de-emphasize the 'page objects' pattern, but without offering much in the way of a replacement).
Yes, I'll echo my disagreement with the Cypress advice against the page object pattern - the advantage was always that the logic to get the element could be updated in a single place, and that applies as much whether that code is 3-5 lines of webdriver code or one line of cypress code.
Yeah, "no DOM in tests" is actually a pretty good mantra.
Code that is just inspecting the DOM to find elements, or to do stuff to them that is just working up to the state where you can actually test something, should be in some shared object/library, but that is as easy as:
import { Whatever } from '/foo/bar';
Having that kind of code outside of the test probably makes the introductory "test hello world" example more complicated, but I think it's definitely a good practice in the real world, and isn't documented well.
Last time I checked you couldn't even organize tests in different folders, I wonder if they finally fixed that. I always find it odd using test frameworks developed by a for profit company. They pick and choose what code to merge and bugs to fix based on their own priority, not what's best for the users.
This even model is sooo confusing, clumsy and error-prone that writing any non-trivial test is a challenge. Whole product uses async/await or Promises and suddenly you have to switch to this baroque model just to write a test. Reusing logic between production code and tests is pretty much impossible.
Dynamic pages etc. often confuse cypress. I'm actually not a frontend developer, so I don't have a lot of insight while exactly that is, but I just know that it has been a huge drag.
Other than that, it would be OKish. Being able to test the UI is great, but I would advice everyone not to base the whole testing strategy on it. It's just too heavy, slow and unreliable. It doesn't not scale even moderately with the product growing.
e2e testing is better done on API level, IMO. You'll want redesigns, changes of UX, and suddenly your changes will require updating tests, which otherwise (if done on the API level) would require no modifications.
It’s possible (and indeed uh ... recommended) to write frontend tests that are resilient to DOM changes by not depending on things that are not essential to the functionality being tested. Eg select a form field by its label, not by its position in the DOM, and use specific data attributes if there’s no better choice. Then when you gut the HTML for UI overhaul, if the functionality hasn’t changed, the cypress test doesn’t change, you just make sure you attach the attributes to the right new elements.
Of course if the functionality changed (like you split something up into 2 pages) yeah you need to update your UI tests cause you are testing a different experience now. But they are UI tests, so yeah. That’s appropriate.
I dunno. It sounds like you found other parts of it unwieldy as well, but the idea that the tests are inherently brittle is wrong. Time spent maintaining UI based testing is saved many times over when compared with the manual clicking around that it saves imo.
> Then when you gut the HTML for UI overhaul, if the functionality hasn’t changed,
How often website redesigns are only about html? Usually everything changes. New flows, new dialogs, new toolbars, new frameworks.
Often the web UI gets and Android UI and iOS UI siblings. Gets replaced altogether by a completely different UI, developed by a different team.
APIs change much less often, and can be shared between many UIs. You can also version them. How are you going to version a web UI? Like reddit with "turn back on classic look"? How long can you keep the old UI around just because you have 1000s tests around that use it.
> but the idea that the tests are inherently brittle is wrong
I don't think so. Tests are great calcifyiers (check my blogpost about it if you want). If you test though your UI, you're calcifying your UI. The more you test though it, the more you calcifying it. No way around it.
I’m not against testing APIs. But testing APIs does not tell you if your latest CSS change broke pointer events on your “add to cart” button, so you can no longer receive orders, etc.
If you are wholesale redesigning/rebuilding a whole site and all the workflows yes of course you need new tests. But also your users would get pretty impatient if entire ways of working are changing all the time in a way that makes testing a giant moving target.
The purpose of a UI test in my mind is to make sure that the core business things a user is supposed to be able to do, are doable. In the context of your blog posts, I think those things should be calcified.
Like, I want a test to fail if I remove a workflow that used to be there. I want a test to fail if a form field suddenly has no label. There should be those alerts when functionality that was previously understood to be correct has been changed. Lightweight UI tests with some easily re-approved screenshot diffs goes a long way.
The idea behind the code being async is that you don't know how long things will take, especially talking to a backend, or even sometimes DOM manipulations in dynamic pages like you mention.
Having said that, I find parts of it v confusing (although not the asyn stuff) especially around assertions - there seem to be multiple ways to do things and I'm never clear which is which.
As a sibling reply said, try to write cypress (or other similar product) tests not to rely too much on layout - I use html data attrivutes rather than text or Dom based selectors, so the tests can survive most redesigns, and if they don't, it's because the user journey has changed enough that we _should_ be changing the tests.
There are much simpler programming models to deal with long-running actions, like processes and threads. The essential point of async is to be thread-like, but with lower overhead; or in environments where you cannot use processes and threads - but there's no very convincing reason for testing to be it. And choosing to prioritize dealing with runtime overhead over dealing with coding complexity isn't an obvious win. Note that promises make sense for processes and threads too; the key question is whether you really block, or virtually block.
Nothing wrong with async code; but it's often sold as the "obvious" solution to, well, several things happening and some of them taking long - and it's just not that great a solution.
Would threads or processes be a great way of simulating how a user interacts with a web app though?
I tend to think of them clicking things and waiting for a response then moving on to the next thing when that happens (or failing the test when it doesn't) which seems to fit well with an async model.
Cypress _does_ do a lot of magic behind the scenes to stop you explicitly using async/await or promises in the code... Which occasionally bites it.
Especially when you have to use a .then to use values you pull from the Dom.
You can wait in a threaded model too, right? The only difference is that if you do so, you're blocking that thread. The pain comes when you mix the two models, because async often deals really poorly with blocking. In general, the differences between the two models when used cleanly are quite small. When most waits are no-ops, threads have significantly lower overhead; but when waits are common, especially if long-lived, async overhead is much lower.
In fact, if you want to minimize the chance of deadlocks, using threads or processes for every action is safest; although of course that means you may get issues with concurrent modification. But though that risk is serious, it's worth noting that async code doesn't actually solve it; after all, every await may mean a context switch, and thus if you have shared mutable state living across an await, all bets are still off. async-only code with exactly 1 thread only provides safety to the extent that modification is encapsulated and that the module within which the internal datastructures are updated does not use async. And that's not nothing; it's kind of as if a java object was synchronized on all methods. But it's also not reliable enough that you can just forget about concurrent modification, either.
But if all you want to do is automate a browser, those overheads are completely irrelevant, and this story about concurrent modification likely is too. You could use either model equally well. To be explicit: there's nothing about promises that means you need to use async. It's only once you start chaining a bunch of them that you enter async territory, but why would you ever do that in a threaded model when automating a browser? The browser is so heavy weight, just fork a bunch of processes already, and KISS: no need to worry about any of this, at all.
I am quite surprised. Our experience has been quite positive. We were able to do full coverage of our web app in less than a month without prior experience with cypress. This is what we learned from it: https://blog.servmask.com/8-tricks-i-learned-from-cypress/
To hop on the unhappy train, it's actually quite hard to get an internal mirror mirror working correctly because the platform for which the extra file it downloads is based on a query string, not the path. Argh.
I have been developing tests using Cypress for 6+ months.
I have used multiple other variations of test frameworks that were built on the Selenium WebDriver protocol. The official documentation [1] states that Cypress differs fundamentally and architecturally as compared to Selenium.
Admittedly I found Cypress web testing framework frustrating initially, and yes, I have performed many re-writes because Cypress documentation behind command chaining and assertion queuing to be not intuitive.
Thanks for my management's support, perseverance afforded me with the stubbornness with a purpose of checking out Cypress if it has a better DOM manipulation, how it operates directly in the browser, interactive test runner, and its API and configuration.
It required 3+ months of perseverance in getting the Cypress tests organized [2] for handling web testing core functionality of a commercial-grade corporate web site. My current layout required adding a lot of extensions to the core folders under /cypress: /support, /plugins & /config.
I am motivated in writing a series of posts on Medium, because I like Cypress. Is it perfect, no, some aspects are frustrating. In other words, I have not found a solution yet, but I will continue to use Cypress.
From the minimal amount of Cypress development experience, suggestions:
a) Switch everything to TypeScript
b) Declare Commands within Cypress namespace
c) Understand selectors
d) Add querying attributes, like 'data-cy', to expedite finding elements.
e) Wait for element to appear using 'cy.get(...).contains([ element attributes]).then(...)'
f) Avoid if at all possible using XPath for selectors.
g) Use 'cy.wait()' to wait for an alias to become available, and not for millisecond delays.
h) Use available eslint extensions for Cypress.
I will try to get a couple of posts on Medium as soon as possible. They may be helpful.
If you're willing to try something a bit different, we're working on a no-code approach to test automation that tries to avoid the pitfalls of code-based testing [1]. We're focused on building a E2E testing platform that (1) makes it as easy as possible to create tests and (2) makes tests easy to maintain. Easier said than done but that's our goal.
What are the benefits over selenium ide? It’s been years since I worked heavily on testing but I don’t recall there ever being a good record/playback testing tool. I’m intrigued.
Selenium IDE uses a Chrome or Firefox extension to record your actions, but they're rewriting it to be an Electron app. I'm not sure about the status of that rewrite though.
But the benefits we have vs. what you experienced with Selenium IDE is:
- Instead of using an extension to record your actions, there's nothing to install to use Reflect. We spin up an instrumented browser inside a virtual machine and screenshare that with you within our web app. It records your actions and translates those actions into a repeatable test automatically.
- Since we have complete control of the test environment, we can do cool things like make recording file uploads really easy or present a nice workflow for getting coverage of visual regressions via screenshot testing.
- Our recorder is more accurate :)
- For every test you run within Reflect, you get a video synced up to the steps in your test, along with console and network logs from the test run.
- Instead of emitting Selenium code that you then maintain like a normal code-based testing tool, we are completely no-code meaning that even updating a test is completely codeless. So for example if you need to make bigger changes to an existing test, we expose a way to re-record only the portions of the test you want to change, and keep the rest as-is.
It is visually appealing but sluggish and sucks at non-standard authentication to the point of not being usable.
Also no out of the box way to run more than one instance with incognito tabs to have communication to test multi-user bugs.
Perhaps radical, but I think it should support hooks with Playwright/Puppeteer that you can hand control back and forth to as a safety valve just like being able to shell out to the command line when your favorite programming language doesn't have a library for what you need.
The difference here is that the browser context needs to be passed - that means some janky handoff (which with crypto will be a PITA) or using the Playwright/Puppeter connector under the hood.
Same experience here, moved from cypress to testcafe after getting frustrated by cypress API.
It was some time ago so I may not remember correctly but I felt like cypress was not for people already fluent in javascript, a bit like jquery but pushed even further (you need to do everything the cypress way now, you can't even use variables or promises...).
No, page objects (AKA page models) are a recommended but optional pattern in TestCafe.
We use them on my team because they make the tests more readable and keep us from having copies of the same selectors in multiple places, but you can easily use TestCafe without them.
I really enjoyed the testing library when I used it. I recently came back to it and felt like they removed a lot of the documentation that I liked about the library. I'm not quite sure why they did, but it was very helpful to see their examples for integrating with common third party libraries.
it took a lot of time but i do like the cypress approach. there are weird things like how you can’t get the current time because all lines execute immediately and are evaluated later. but once you get your handle on .then and .wrap it is good. they should have picked a different name than .then because it’s not a promise but looks like one
It’s been a few months since I’ve touched it, so I’m struggling to give more specific criticisms.
I’ve had a really great time using Kent Dodd’s DOM Testing Library [1] for unit tests. The readability and writability of these tests has been fantastic. I wish I could use the exact same API for e2e tests. It looks like they actually do have a Cypress version of the Testing Library [2], but it seems to suffer from the same Cypress-isms that I was just complaining about.
Has anyone else had a similar experience? It seems like I’m in the minority opinion. Maybe I’m just bothered because I don’t understand Cypress well.
[1]: https://testing-library.com/docs/dom-testing-library/intro
[2]: https://testing-library.com/docs/cypress-testing-library/int...