Every type of automated testing has a maintenance tax. Everything from simple unit tests up through end-to-end tests that are performed on the GUI of your application. Of course this tax rate varies depending on the complexity of the test. For unit tests (which typically deal with a small, isolated piece of functionality), the tax is cheap. You’re dealing with an atomic unit of your application that isn’t likely to be affected by changes around it. Only changes that are made specifically to the code you’re testing will have an impact, therefore the required maintenance is minimal.
End-to-end testing is on the opposite side of the spectrum. You’re interacting with, in most cases, every layer of your application:
- A button is clicked in your application’s GUI in a web browser
- That buttons triggers some JavaScript code
- That JavaScript code makes an API call to your backend
- Your backend server fields the request and runs some code of it’s own
- That backend code queries your database for some data
- The data is processed and sent back to the frontend of your application
- The application processes the data in JavaScript and updates your GUI
Phew! Look at all those layers of your application that are being touched (*). That’s why it’s called end-to-end testing. It’s also why it’s on the opposite end of the “maintenance tax” spectrum. Any change to any of those layers could have an impact on your test. It’s far more prone to breakage simply due to it’s exposure to each of those layers and thus the cost of upkeep is much higher.
However, touching each of those layers is exactly where the benefit of end-to-end testing comes from. You’re testing your application as a whole, across the board. You’re confirming that every aspect from UI to server to database is playing nice together. After all, that’s how your end users are going to experience your application.
I’ll give a sports analogy here. Pretend you’re a basketball player. A unit test might be dribbling the ball, or passing the ball, or taking a free throw. An end-to-end test would be dribbling down the court, passing the ball off, getting it back, faking out another player, then sinking a shot. Now, what should you be doing at practice? Well, primarily working on each of those isolated skills. But ultimately, playing basketball requires putting them all together, so you should be devoting at least some effort to that as well — even it’s a more complicated endeavor.
So, given that end-to-end tests have a higher maintenance tax, are they still worth it? Absolutely — but, like the analogy above, one must understand their place within the greater scope of application testing. Fortunately, the test pyramid does a good job of illustrating this:
As you can see, unit tests make up the foundation and largest portion of the test pyramid. As the complexity of the tests increase, so too does their maintenance cost, and so we scale down the volume of tests. In case you’re wondering, that “Intergration Tests” layer in the middle is exactly what it sounds like. One notch up in complexity from unit tests where you’re testing multiple portions of code to ensure that they integrate together properly. However, they typically don’t touch the “full stack” like an end-to-end test.
Relative to the full testing picture, end-to-end tests should be more like “the sugar on top” than the foundation, though the actual volume can vary as your testing strategy scales. Perhaps you only have 5-10 end-to-end tests which run through the core features of your application via the GUI (like the on-boarding journey, the purchasing journey, and so on). Or, maybe that number scales up to 50 or 100 as your testing coverage grows and you develop a sound strategy for creating and maintaining your end-to-end tests along side of your application.
End-to-end tests bring a unique and valuable angle to your testing approach by touching many different layers and replicating real world scenarios. But, with that power comes complexity and a higher maintenance tax that must be factored in when leveraging them. In the end, it’s the decision of the testing team as a whole to decide when and where end-to-end tests make sense within your application testing strategy.
Acknowledgements & Related Links
- (*) The number of layers that an end-to-end test touches can be minimized by “mocking” specific portions, like the database or API responses. However, from my experience, it’s more common that they’re run on a fully operational application (often in a staging environment).
- The test pyramid is a concept developed by Mike Cohn, described in his book “Succeeding with Agile”.
- Martin Fowler’s TestPyramid post covers this same topic. While he’s critical of recording tools in his post, which Ghost Inspector offers, we add robust functionality for editing and centralizing logic after the recording process to help mitigate the issue of “brittle” tests.
