Test a hint

When testing a hint you might need to do different things like forcing a fail request, return a binary, etc. This page documents what you need to do to start testing a hint and how to configure the test server to do what you need.

Getting started

If you have used the built-in tools to create a new hint (core or custom), everything should already set up to use hint-runner.ts under (utils-tests-helpers) and the testHint method.

If not, you need to:

1. Create a tests.ts file in the hint folder like hint-<hint-id>/src/tests/tests.ts.
2. Have the following template:

import { generateHTMLPage } from '@hint/utils-create-server';
import { getHintPath, HintTest, testHint } from '@hint/utils-tests-helpers';

const hintPath = getHintPath(__filename);

const tests: HintTest[] = [
    {
        name: 'This test should pass',
        serverConfig: generateHTMLPage()
    },
    {
        name: `This test should fail`,
        reports: [{ message: `This should be your error message` }],
        serverConfig: generateHTMLPage()
    }
];

testHint(hintPath, tests);

The high level overview of what’s is happening in the code above is as follows:

1. tests (of type HintTest[]) contains the list of things to test, the server configuration to use (serverConfig) and the expected result(s) (reports). If no results are defined that means webhint should not fail that configuration. Otherwise the results should match the ones defined.
2. hintRunner.testHint will take an HintTest[] and create a web server for each one of the items in it. It will also create a webhint object with just the hint to test configured and run it against the web server for that particular test. The results from executing it are compared to those defined in HintTest.reports. If they match, then everything is good.
3. If the reports property is not specified, the actual results of the test will be logged. This can be used to debug what is being returned by the tests.

There’s more information and detail in the following sections.

HintTest

HintTest defines a test that needs to be validated. Its properties are:

• name: The name of the test. It’s a good practice to say what the test is testing and what is the expected output. E.g.: “meta charset in body should fail”
• serverConfig: This is the server configuration used for that particular test. When running the tests, a local web server will be created for each HintTest on a random port that webhint will analyze. There’s more information about serverConfig below.
• reports: An array of Reports to match with the output of running webhint to the specified configuration. A Report is what webhint returns when it finds an issue. In this scenario, only the properties defined on each Report will be matched. This means you can decide to ignore some that are not relevant to you, i.e.: in the code above you could decide to remove position and the test engine will not try to validate that property.

position

When specified, position can be a ProblemLocation or a string of text in the source to derive a ProblemLocation from. The latter is recommended when possible as it is less error-prone.

// `position` as a `ProblemLocation`
const tests: HintTest[] = [
    {
        name: 'Name of the tests',
        reports: [{
            message: 'Error message targeting the word "HTML"',
            position: { column: 0, line: 0 }
        }],
        serverConfig: 'HTML to use'
    }
    // ...
];

// `position` as a `match`
const tests: HintTest[] = [
    {
        name: 'Name of the tests',
        reports: [{
            message: 'Error message targeting the word "HTML"',
            position: { match: 'HTML' }
        }],
        serverConfig: 'HTML to use'
    }
    // ...
];

Execute code before or after collecting the results

In some scenarios you need to execute some code before or after the actual tests (e.g.: if you need to change the current working directory). For those cases you can use the before and after properties of HintTest:

const tests: HintTest[] = [
    {
        after() {
            // Code to execute right before calling `connector.close` goes here.
        },
        before() {
            // Code to execute before the creation of the engine object goes here.
        },
        name: 'Name of the tests',
        reports: [{ message: 'Message the error will have' }],
        serverUrl: 'https://example.com'
    }
    // ...
];

Override hint dependencies when running tests

In some scenarios you need to override or mock dependencies to the hint being tested. For those cases you can use the overrides property of HintTest:

const tests: HintTest[] = [
    {
        name: 'Name of the tests',
        overrides: { 'dependency-package-name': { /* Overridden helpers from dependency */ } },
        reports: [{ message: 'Message the error will have' }],
        serverUrl: 'https://example.com'
    }
    // ...
];

An example will be if the hint integrates with another service. You don’t want to actually connect to that service during the tests (slow down, need to force an specific output, etc.) so you will mock the connection to that service in the overrides property. An example of hint that does this is ssllabs where the call to the server is completely mocked to return different grades.

serverConfig

serverConfig defined the web server configuration to use for a given test. It can be of different types depending on your particular needs:

• string containing the response for / (HTML, plain text, etc.).
• object with paths as properties names and their content as values:

const serverConfig = {
    '/': 'some HTML here',
    'site.webmanifest': 'other content'
};

This code will create a local web server that will return some HTML here to all requests done to / and with other content to the requests done to site.webmanifest.

You can even specify the headers and status code for the response for a specific path, by using the headers and status properties:

const serverConfig = {
    '/': 'page content goes here...',
    '/example.js': {
        content: 'script content goes here...',
        headers: {
            'Content-Type': 'text/javascript; charset=utf-8',
            Header: 'value'
            // ...
        },
        status: statusCode
    }
};

Notes:

• If content is not specified, it will default to an empty string ''.
• To remove any of the default HTTP response headers, set their value to null (e.g.: headers: { '<response_header>': null }).
• status defaults to 200, so it only needs to be specified if its value is different.

Conditional responses

Sometimes you need the server to respond differently to a route depending on the contents of a request, e.g.: when requesting an asset that can be compressed with different formats. The following is an example of how you can return a different value depending on the content of the Accept-Encoding header:

const serverConfig = {
    '{ "request": { "headers":{ "Accept-Encoding":"br" }}}': {
        '/': {
            content: { /* content here */ },
            headers: { /* headers here */ }
        }
        // ...
    },
    '{ "request": { "headers":{ "Accept-Encoding":"gzip" }}}': {
        '/': {
            content: { /* content here */ },
            headers: { /* headers here */ }
        }
        // ...
    }
};

Throwing an error

If you need to force an error in the connector when visiting a URL you have to make the content null. This will force a redirect to test://fail, thus, causing an exception.

Testing an external URL

If you need to test an external resource (because you are integrating with a third party service) you need to use the property serverUrl:

const tests: HintTest[] = [
    {
        name: 'Name of the tests',
        reports: [{ message: 'Message the error will have' }],
        serverUrl: 'https://example.com'
    }
    // ...
];

hintRunner.testHint()

hintRunner is in charge of executing and validating the tests. The signature of hintRunner.testHint is:

• hintPath, the name of the hint being tested.
• tests, an HintTest[].
• configuration (optional), allows you to modify the defaults of how the tests are run.

configuration can have the following properties:

{
    "browserslist": [],
    "https": boolean, // default is false
    "hintOptions": {
        // hint properties
    }, // default is an empty object
    "serial": boolean, //default is true
}

• browserslist: You can change the targeted browsers to check that the hint adapts correctly with this property. It uses the same format as the one in .hintrc.
• https: By default all tests are run over HTTP. If you need to test something over HTTPS you want to set this property to true. NOTE: Do not mix HTTP and HTTPS tests in the same file as it will not run correctly.
• serial: By default all tests are run in parallel. If you need to run them serially set it to true
• hintOptions: Some hints allow further configuration. You can test those configurations with this property.

Each web server is started on a random port. If the message of a report contains localhost, it will be replaces automatically with localhost:USEDPORT so you don’t have to worry about it.

Note: hint-runner will automatically test the hint in as many connectors as possible, that’s the reason why you might see tests being run more than once.