Sometimes I wonder how quality assurance (QA) got this bad. Humanity’s engineering achievements are nothing short of mind blowing. Buildings that reach the stars, tunnels bored under cities, bridges that span seas. But when it comes to software that doesn’t break, quite frankly, we suck.
If you agree with me, the chances are you feel the situation is not getting any better. If you’re an engineer, you probably feel like you don’t test your software well enough. If you are a product manager, you likely feel like in the time it takes for you to explain how to test your software, you could have just done it yourself. If you run QA, you probably feel that people don’t understand what you do or really value it, but gosh, is it hard! And as a CEO, you probably feel like your team doesn’t do enough QA because the last release had an embarrassing bug that showed up during your VC pitch.
If any of that rings true, then welcome to the club. The bottom line: QA is hard, and it is mostly because, up until now, much of the tooling out there has sucked. However, things changed drastically for us when we began relying on both API testing and UI testing solutions that are so easy to use, anyone from a developer to a business person can take them for a spin.
QA-ing Our QA Processes
Not long ago, the best QA solution was to throw people at the problem; pay an offshore outfit bags of cash to write a custom Selenium framework for your app, which unfortunately ceases to be useful the moment you discontinue the contract because, guess what? Maintaining 1,100 Selenium tests is impossible for anyone but a gang of offshore QA guys.
Two years ago, we replaced our monolith system with a microservices stack. We got into trouble pretty fast with integration issues—services at the core would change and everything else would break because of response differences and so forth. The unit tests on the core would pass, but those tests didn’t cover all the ways that other services were interacting with them. As a result, sometimes the platform remained broken until someone manually tested it or another service was rebuilt. Sometimes defects went undetected for longer than you would typically like them to, making pinpointing those issues time-consuming.
We realized pretty quickly that the best people to contribute integration tests for a service are the people who depend on them—which isn’t necessarily the people who historically write the tests. We also understood that integration tests need to run constantly, not just at build time in the dev environment, but also at runtime in production. As the platform grew and was used in ways we didn’t anticipate, defects would appear. Our users became the one mutagenic factor that the build server couldn’t truly account for.
Decentralizing QA with Runscope and Ghost Inspector
This notion of democratizing the test capability started to germinate with us. We wanted to empower anyone in our company to write tests, especially the people on our team who maybe didn’t know how to code, but worked very closely with our clients. We started using Selenium and Postman, but the learning curve was too great and they lacked an easy framework for continually running and contributing to the corpus of services we have in our stack.
Once we signed up for Runscope for API testing and Ghost Inspector for browser and UI testing, they quickly bridged this gap for us. At the danger of this sounding liking a puff piece, I have to say that they are incredibly usable pieces of software that let you write functional tests and integration tests for your web app in an absolutely delightful way. I honestly love their software.
However, the victory for us though is how both Runscope and Ghost Inspector have changed the face of QA in our company. We are beginning to grow what I believe is a QA culture, in which people’s feelings of dread and malaise around QA are now replaced with feelings of empowerment—even our chairman writes Ghost Inspector tests now!
Ghost Inspector is instrumental here because it allows us to observe the exact experience that a user would have visiting a site or app because of its video capability. Knowing something failed is one thing, but having a recorded video of the failure case, plus browser logs, is quite another. We also use Runscope heavily to monitor our APIs because it gives us probes that we couldn’t easily achieve in any other way.
Cutting through Complexity
When I think about what this all means to me, I’m reminded of something Kent Beck once said, something along the lines of, “Writing software without tests makes me anxious… and I don’t think software developers deserve to feel anxious about their jobs.” When you boil down what we do as technologists, we try to make cool stuff that makes people’s lives better and absolutely, positively, never ever breaks. Trouble is, sometimes that last part is a whole lot more consuming than the first, and that’s a bummer.
So I guess I’m here to report that QA might be easier than you think nowadays. With tools like Runscope and Ghost Inspector, you can all do it. While you are on the couch, on the way to work, in the shower (well, maybe not in the shower).
