Storing & Executing Tests in Your Version Control

Ghost Inspector now provides the ability to store tests offline in your own system and execute them on-demand. This can be useful when you want to keep your test in your own code base or version control system, instead of within the Ghost Inspector service.

One scenario where this can be beneficial is with code branches; on-demand tests will allow you to store your tests in your VCS and then modify them within your branch. Tests in your branch can be modified to suit the changes that you are making and then executed on-demand using Ghost Inspector in your CI process. This allows for immediate pass/fail feedback on the status of the branch.

On-demand Test Example

To create an on-demand test you can either download one of your tests from your account in Ghost Inspector (JSON) format, or you can create one manually.

On-demand tests only require three attributes, name, startUrl, and steps:

{
  "name": "On-demand Test",
  "startUrl": "https://www.ghostinspector.com/",
  "steps": [
    {
      "command": "assertTextPresent",
      "target": ".site-logo",
      "value": "Ghost Inspector"
    },
    {
      "command": "execute",
      "value": "5e34b17ea9096c7650f1df99",
      "notes": "Import Log In test"
    }
  ]
}

Now I can save this JSON to a file called myTest.json on my file system, or in my version control system like Git.

Executing an On-demand Test with the CLI

The quickest way to execute to execute our newly-created test is using the Ghost Inspector CLI. Assuming you have saved the above code sample to a file called myTest.json, we can execute it using the following example:

$ ghost-inspector test execute-on-demand <organiationId> ./myTest.json

By default this command will wait until the execution is finished before returning, however you can have the command return right away with --immediate. You can also have the command return a failing (non-0) exit code when your test is failing by passing --errorOnFail for the passing status and --errorOnScreenshotFail for the screenshotComparePassing status. This can be handy in continuous integration environments when you want to fail a build step based on the test status.

Executing an On-demand Test with Node.js

Our official Node.js client also supports executing tests on-demand. In this next snippet, we will load up myTest.json, execute it using the Ghost Inspector API, and then poll for the status of the result:

// load the Ghost Inspector API client with our API key
const GhostInspector = require('ghost-inspector')('{{my-api-key}}')
// load our test
const myTest = require('./path/to/myTest.json')

// this function will let us execute the test using async/await
async function main() {
  const myOrganizationId = '{{my-organization-id}}'
  // we pass the `wait: true` option to wait for test completion
  const result = await GhostInspector.executeTestOnDemand(myOrganizationId, myTest, { wait: true })
  console.log(`Test status: ${result.passing}`)
}

// execute the test!
main()

Running the Test

Assuming we save the above code sample to a file called on-demand-test.js, we can execute it using Node.js:

$ node ./on-demand-test.js
Test status: passing

Test Attribute Reference

Here is a list of the variables that are available to a test (and steps) for your on-demand tests:

_id

Link the execution to an existing test in your organization.
Valid options: 24 character hex string

autoRetry

Automatically retry the test run if it fails when this is set to true.
Valid options: false (default), true

browser

Specifies the browser to use during execution.
Valid options: chrome (default), chrome-<version>, firefox, firefox-<version>. Pass multiple values as an array to trigger multiple executions, eg: ['chrome', 'firefox']

dataSource

The ID of the data source to use for this execution.
Valid options: 24 character hex string

disableVisuals

Disable the screenshot and video features for this test run when true.
Valid options: false (default), true

failOnJavaScriptError

Fail text execution when a JavaScript error is detected when true.
Valid options: false (default), true

finalDelay

The length (in ms) the test will wait before taking the final screenshot.
Valid options: Any numeric value, defaults to 5000 (5 seconds)

globalStepDelay

The length (in ms) the test will wait before taking the final screenshot.
Valid options: Any numeric value, defaults to 250

maxAjaxDelay

The max time (in ms) to wait for an XHR request to finish.
Valid options: Any numberic value, defaults to 10000 (10 seconds)

maxWaitDelay

The max time (in ms) to wait for an element to be available.
Valid options: Any numberic value, defaults to 15000 (15 seconds)

name
(Required)

The name of the test.
Valid options: Any string value

region

Specifies the global region go execute the test in.
Valid options: us-east-1 (default), us-west-1, ca-central-1, eu-central-1, eu-west-1, eu-west-2, eu-west-3, eu-north-1, me-south-1, ap-east-1, ap-northeast-1, ap-northeast-2, ap-southeast-1, ap-southeast-2, ap-south-1, sa-east-1. Pass multiple values as an array to trigger multiple executions, eg: ['sa-east-1', 'ap-southeast-2']

screenshotCompareBaselineResult

The ID of an existing test result to use for the screenshot comparison baseline.
Valid options: 24 character hex string

screenshotCompareEnabled

Perform a screenshot comparison using the result specified in screenshotCompareBaselineResult when true.
Valid options: false (default), true

startUrl
(Required)

The URL to begin the test from.
Valid options: Any string value

steps
(Required)

The list of test steps to execute.
Valid options: Array of step objects, available properties are command,command, optional, target, value

steps[#].command
(Required)

Specifies the type of action this step will take.
Valid options: assertElementNotPresent, assertElementNotVisible, assertElementPresent, assertElementVisible, assertEval, assertNotText, assertText, assertTextNotPresent, assertTextPresent, assign, click, dragAndDrop, eval, execute, exit, extract, extractEval, keypress, mouseOver, open, pause, refresh, screenshot, store

steps[#].condition

JavaScript script that will execute in the browser session to determine if the step should be run (if true is returned) or skipped (if false is returned).
Valid options: Any valid JavaScript code, ex: return false;

steps[#].optional

Test will continue if this step fails when true.
Valid options: false (default), true

steps[#].target

CSS or XPath element target to use for interactive test steps.
Valid options: Any valid CSS or XPath selector, ex: .submit-button

steps[#].value

Value to use with relative step action.
Valid options: String value will change according to step[N].command:

assertEval - JavaScript code
assertNotText - text value to check
assertText - text value to check
assertTextNotPresent - text value to check
assertTextPresent - text value to check
assign - variable value
eval - JavaScript code
execute - test ID to import
extractEval - JavaScript code
keypress - key code to use
open - URL to open
pause - time (in ms) to pause
store - string value to store

viewportSize