How Ghost Inspector ‘Transformed’ StellarWP’s QA Processes [Case Study]

 Introduce yourself 👋

Hey there, my name’s Ben Meredith and I’m the Head of Support at GiveWP, as well as the Head of Quality Assurance for GiveWP and a number of other WordPress brands within StellarWP (a Liquid Web brand). I head up the technical support teams, and I also launched the quality assurance teams for a number of those brands. So that’s the context that I use Ghost Inspector in.

What is your business?

StellarWP is a group of brands that are all WordPress related products. So they’re WordPress plugins that allow you to add functionalities to your website. GiveWP specifically is a donation platform for WordPress. That’s the main software product that we sell. It’s a distributed product. And so we have to test it as widely as possible on as many different scenarios as possible before we send it out. So quality assurance just has extra levels of wrinkles because it’s not on servers that we even have control over. 

What challenges were you facing that made you seek out Ghost Inspector?

GiveWP is where I started with Ghost Inspector. The biggest problem we were having was the way we were doing quality assurance. It was really kind of guess and check and not very systematized at all in the sense that we just had to be lucky to test as many specific scenarios as possible. 

We needed automated tests for all of these scenarios, so for anything that we were manually testing we said – hey, let’s make an automated test for that so that the next time around we can manually test something else. And, oh, let’s make an automated test for that so that each release became more and more stable. In the case of GiveWP, customers are doing donations on their WordPress website, and an update could potentially take down donation functionality.

That’s obviously a humongous “no-no” – we don’t want to do that. So we said, that’s our top priority. And since then, we’ve branched out. Now we test all sorts of things. Before each release, the major thing that was a huge source of heartburn for us as a company was not having the assurance to be able to tell our customers: we know that when you click “update,” it’s not going to take down donations on your site. And now we can say that.

How did Ghost Inspector help you overcome those challenges?

It has just night and day transformed our QA processes, and very early on as we were creating these automated tests, we ran into some edge case bugs that would have caused hundreds of support tickets of angry people saying “my website is not working.” We were able to head those off at the pass, and it really comes down to Ghost Inspector catching those because we were able to do that.

Can you walk me through the process of implementing Ghost Inspector in your organization?

I’ll be honest, implementing Ghost Inspector was intimidating for me. I’m enough of a developer to get myself into trouble. I know a little bit of javascript. I know a little bit of PHP and those kinds of things. And so the first thing I did was kind of get overwhelmed. I got in there and I was like, man, I can do all of these different things. I don’t really know where to start.

So, where did you start?

One of the engineers on our team walked me through and said, “let’s just start small, start with one little, simple repeatable test and build from there and then do it again.”

The very first few tests I ran into some troubles where our deployment system was not working right to get me the correct zip. I kind of ran into a dead end, but then I took a step back and said, OK, let’s come at it from a different way.

We have a site that is our persistent quality assurance site that we run all tests on for this one specific test suite. I started by ensuring everything on the test site looked right. Then I went into Ghost Inspector and ran the tests.

That was kind of version one – this QA site as a persistent site. 

What was version two like?

The next thing we did was start using a service that spins up these disposable sites with templates. So I can spin up a site, use it for tests, use it for a suite of tests and then delete it when I’m done because we’re honestly still in the very early stages of implementing that whole procedure. And getting with our development team to make sure that the QA team has the right resources and the support team has the right resources to make that all happen there.

Now we’ve moved on to getting this system of disposable sites that we can spin up with very specific edge case scenarios. Then we can test against those and not have to maintain a dozen of QA sites that are just sitting there for QA. We can use those templates to do that. 

Now we’ve got a team of folks – it’s actually 11 guys who help us make the tests. So as we’re deploying new functionality or finding bugs, we send them to the team and ask them to make a new test for this specific functionality or this specific bug and make sure that we’re catching it. So that’s been a humongous process. I think now we’re up to 75 or 80 tests on that one main QA site that we can run on every single release and know that we are not releasing bugs into the system.

What specific features on Ghost Inspector do you find the most helpful?

One feature is the video recorder – to be able to go and watch the video and see exactly where the test went wrong if it failed or what the test fixed because like I said, I’m not making all the tests now. I’ve delegated that, and someone else is making them, so now I can hop in, run the video and see what’s going on, approve it, and move it into an official status.

The 2nd thing that I think revolutionized my experience with Ghost Inspector has been the concept of reusable tests. I mentioned that we work with a lot of payment gateways. Well, some of them have specific test credit cards that you have to use. And so as opposed to having 15 different tests that all test Stripe test credit cards, now, I can just create one reusable test that’s says enter Stripe credit card and it goes in, it enters that Stripe credit card. And if something changes, if something’s broken about that test, I can go and fix it in that one reusable test spot and not have to go fix it in 15 different places.

Why do you recommend Ghost Inspector to other businesses?

I’d recommend Ghost Inspector to other organizations simply because it’s peace of mind. It’s knowing that you’re releasing software that’s not broken.  So if I’m shipping new code, I want to make sure that this old bug that I fixed three versions ago isn’t back. I’ve got a test for that. Now, every time there’s a bug that comes through that’s testable, we make a new test. Every time I’m shipping a new code, I know that it’s not breaking in the same way. It might be broken, but it’s not breaking in the same way as it was before. And that peace of mind is just wonderful, and it gives us the ability to talk to our customers in a much more confident way to say, hey, this new version has been tested. And we know that it’s not broken. And we know that your donations are going to continue so that has been humongous for us.