Storing & Executing Tests in Your Version Control

Ghost Inspector now provides the ability to store tests offline in your own system and execute them on-demand. This can be useful when you want to keep your test in your own code base or version control system, instead of within the Ghost Inspector service.

One scenario where this can be beneficial is with code branches; on-demand tests will allow you to store your tests in your VCS and then modify them within your branch. Tests in your branch can be modified to suit the changes that you are making and then executed on-demand using Ghost Inspector in your CI process. This allows for immediate pass/fail feedback on the status of the branch.

Table of Contents

On-demand Test Example

To create an on-demand test you can either download one of your tests from your account in Ghost Inspector (JSON) format, or you can create one manually.

On-demand tests only require three attributes, namestartUrl, and steps:

  "name": "On-demand Test",
  "startUrl": "",
  "steps": [
      "command": "assertTextPresent",
      "target": ".site-logo",
      "value": "Ghost Inspector"
      "command": "execute",
      "value": "5e34b17ea9096c7650f1df99",
      "notes": "Import Log In test"

Now I can save this JSON to a file called myTest.json on my file system, or in my version control system like Git.

Executing an On-demand Test with the CLI

The quickest way to execute to execute our newly-created test is using the Ghost Inspector CLI. Assuming you have saved the above code sample to a file called myTest.json, we can execute it using the following example:

$ ghost-inspector test execute-on-demand <organiationId> ./myTest.json

By default this command will wait until the execution is finished before returning, however you can have the command return right away with --immediate. You can also have the command return a failing (non-0) exit code when your test is failing by passing --errorOnFail for the passing status and --errorOnScreenshotFail for the screenshotComparePassing status. This can be handy in continuous integration environments when you want to fail a build step based on the test status.

Executing an On-demand Test with Node.js

Our official Node.js client also supports executing tests on-demand. In this next snippet, we will load up myTest.json, execute it using the Ghost Inspector API, and then poll for the status of the result:

// load the Ghost Inspector API client with our API key
const GhostInspector = require('ghost-inspector')('{{my-api-key}}')
// load our test
const myTest = require('./path/to/myTest.json')

// this function will let us execute the test using async/await
async function main() {
  const myOrganizationId = '{{my-organization-id}}'
  // we pass the `wait: true` option to wait for test completion
  const result = await GhostInspector.executeTestOnDemand(myOrganizationId, myTest, { wait: true })
  console.log(`Test status: ${result.passing}`)

// execute the test!

Running the Test

Assuming we save the above code sample to a file called on-demand-test.js, we can execute it using Node.js:

$ node ./on-demand-test.js
Test status: passing

Test Attribute Reference

Here is a list of the variables that are available to a test (and steps) for your on-demand tests:

Link the execution to an existing test in your organization.
Valid options: 24 character hex string
Automatically retry the test run if it fails when this is set to true.
Valid options: false (default), true
Specifies the browser to use during execution.
Valid options: chrome (default), chrome-<version>firefoxfirefox-<version>. Pass multiple values as an array to trigger multiple executions, eg: ['chrome', 'firefox']
The ID of the data source to use for this execution.
Valid options: 24 character hex string
Disable the screenshot and video features for this test run when true.
Valid options: false (default), true
Fail text execution when a JavaScript error is detected when true.
Valid options: false (default), true
The length (in ms) the test will wait before taking the final screenshot.
Valid options: Any numeric value, defaults to 5000 (5 seconds)
The length (in ms) the test will wait before taking the final screenshot.
Valid options: Any numeric value, defaults to 250
The max time (in ms) to wait for an XHR request to finish.
Valid options: Any numberic value, defaults to 10000 (10 seconds)
The max time (in ms) to wait for an element to be available.
Valid options: Any numberic value, defaults to 15000 (15 seconds)
The name of the test.
Valid options: Any string value
Specifies the global region go execute the test in.
Valid options: us-east-1 (default), us-west-1ca-central-1eu-central-1eu-west-1eu-west-2eu-west-3eu-north-1me-south-1ap-east-1ap-northeast-1ap-northeast-2ap-southeast-1ap-southeast-2ap-south-1sa-east-1. Pass multiple values as an array to trigger multiple executions, eg: ['sa-east-1', 'ap-southeast-2']
The ID of an existing test result to use for the screenshot comparison baseline.
Valid options: 24 character hex string
Perform a screenshot comparison using the result specified in screenshotCompareBaselineResult when true.
Valid options: false (default), true
The URL to begin the test from.
Valid options: Any string value
The list of test steps to execute.
Valid options: Array of step objects, available properties are command,commandoptionaltargetvalue
Specifies the type of action this step will take.
Valid options: assertElementNotPresentassertElementNotVisibleassertElementPresentassertElementVisibleassertEvalassertNotTextassertTextassertTextNotPresentassertTextPresentassignclickdragAndDropevalexecuteexitextractextractEvalkeypressmouseOveropenpauserefreshscreenshotstore
JavaScript script that will execute in the browser session to determine if the step should be run (if true is returned) or skipped (if false is returned).
Valid options: Any valid JavaScript code, ex: return false;
Test will continue if this step fails when true.
Valid options: false (default), true
CSS or XPath element target to use for interactive test steps.
Valid options: Any valid CSS or XPath selector, ex: .submit-button
Value to use with relative step action.
Valid options: String value will change according to step[N].command:

  • assertEval – JavaScript code
  • assertNotText – text value to check
  • assertText – text value to check
  • assertTextNotPresent – text value to check
  • assertTextPresent – text value to check
  • assign – variable value
  • eval – JavaScript code
  • execute – test ID to import
  • extractEval – JavaScript code
  • keypress – key code to use
  • open – URL to open
  • pause – time (in ms) to pause
  • store – string value to store
The size to set the browser viewport for the test run
Valid options: Object with width and height fields, ex: { width: 1200, height: 800 } (default). Pass multiple values as an array to trigger multiple executions, eg: [{width: 800, height: 600}, {width: 1024, height: 768}]