How EnergyCAP Automated 3,600 Daily Test Runs with Ghost Inspector [Case Study]

Introduce yourself

Hi, my name is Zac ODonnell. I am a Manager of Software Engineering at EnergyCAP Inc, and I am passionate about making products that help people without making the developers who build them miserable.

What is your business?

EnergyCAP offers a suite of products that we call an energy and sustainability ERP. It helps our clients get their arms around their energy usage, their utility bills, and their carbon footprint and save a ton along the way.

What challenges were you facing that made you seek out Ghost Inspector?

Our product offering has been around for around 40 years. That means it’s pretty big keeping something that big working when you’re making changes regularly, sometimes only days or hours apart can be really challenging. That’s especially the case when you’re depending on a lot of libraries that have security patches coming out every few weeks and you don’t really know what they might have changed or what that could affect. Ghost Inspector has given us a way to build an integration testing suite where we have the confidence to make those releases all the time. Even when the changes that are happening aren’t really in our control.

How did Ghost Inspector help you overcome those challenges?

Ghost Inspector has enabled us to dramatically change our development cycle at EnergyCAP. While we used to have a separate development and QA team, primarily with manual testing being done by the QA and unit testing being done by the developers. We were running into this problem where the developers would think that something was working and they would kind of throw it over the wall and then move on and then later on find out that it’s not really working or it doesn’t work in a certain case. And then we end up with this mix of what are we working on? And when is it really done? And it was a real problem for us.

So what we did instead was we changed it so that the developers had to actually build those integration tests along with the code and turn it all into review at the same time. This made it so that things like adding in automation to help interacting with the code worked right off the right off the bat, no friction because the developers are the ones that needed to use it. So they just added them in – this has made a really big deal for us because the ownership of the code has gone way way up. The understanding of the real user flows has gone way way up. And because there’s no cheating using the browser with real everything, we’ve really had a lot of coverage that we just didn’t have before. Ghost Inspector makes this process seamless enough that those tests are just running all the time.

So every morning we’re able to come in and make sure that the whole suite of everything is covered and we spot problems early on. This has done a lot of great things for us. One of them is reducing our release cadence. We used to release every several weeks, maybe a couple of months and now we can turn a release around in basically a day, a few hours if we want to do targeted testing and still have that confidence to make sure that everything that we think we didn’t touch is really working the way it did before. This has been really critical for our team getting changes out there quickly, bug fixes security patches which can affect lots of different things. It’s been totally transformational for us.

Can you walk us through the process of implementing Ghost Inspector in your organization?

Implementing Ghost Inspector at our organization was kind of an iterative process. We started by just having the QA team author the tests. And then we found that we got into kind of a maintenance problem as we made the test suites bigger and bigger. We had brittle selectors. We had gaps in coverage, duplicate coverage, and we didn’t really have the procedures we were looking for to make reliable tests. So we built up some of those procedures, we built up some infrastructure, and we got the developers involved in making them work. Once we had that, we, we had people using those procedures to build tests. We established a Chief Ghost Inspector to keep the eye on the maintenance as a rotating assignment every sprint and we were able to get 3600 tests to run. It takes about 12 hours of execution. Although most of it is in parallel every night covering the application covering all of our new features and telling us any time we create a regression.

What specific features on Ghost Inspector do you find the most helpful?

Some of the top features that we use in Ghost Inspector every day are cascading variable assignments so that you can have different overrides at different suite levels. Modular tests, definitely for login, but even for including repeated setups and clean up kind of things in tests, being able to call javascript functions that we embedded in the app to make certain interactions much smoother for Ghost Inspector, reusable pieces that way. 

We also use Ghost Inspector’s email handling to generate fake email accounts that we can send messages to and inspect the contents. It’s been fantastic to keep finding new features along the way. One of the things that we make full advantage of is the API handling and Ghost Inspector. So by having an API key, we were able to build our own tooling to do bulk image acceptance across screenshot failures. When we do something like change a button style that’s on hundreds of screens. It’s been really, really useful for us.

Why do you recommend Ghost Inspector to other businesses?

Ghost Inspector can give you the confidence you need to build out your product, to release it all the time, to keep up to date with security patches, to eliminate manual QA to increase ownership on your team and to do it without all the pain that often comes with this sort of automation. Their support is great. They’ve been able to help us out when we hit problems, they respond quickly. The feature set is useful, the price is reasonable and the number of bugs that it catches up front and helps us track down and find out when they happened by looking at historical logs. It’s paying for itself. It’ll probably pay for itself for you too.