Overview

Yath (Yet Another Test Harness / Test2::Harness)

Yath is a new alternative to prove that allows you to take full advantage of Test2.

Brief list of features:

Output
Normal

Normal Output


Here is a run with multiple tests, one of which fails:

While tests are actively running yath will maintain a status at the bottom of all output that looks like this:

Verbose

Verbose Output

In verbose mode you see all test-generated events, each on their own line (or multiple lines)

Noisy

Very Verbose Output

You can always increase verbosity by adding another 'v' to '-vv'. Doing so will cause extra information from the harness to be displayed:

Logging
Log

Event Logging

You can keep a log of all events using the '-L' (Log), '-G' (Gzipped log) or '-B' (Bzip2 log) flags. '-B' is recommended as these logs can be quite large. Yath will tell you where to find the log at the end:

The log is a jsonl file with every single event generated during the test run, in order they were seen.

Replay

Replay from a log

When you replay from a log, the log is used as the source of events. No tests are actually run. The original source does not need to be present, the log file is all you need.

One major benefit of being able to replay a log is that you can use a different verbosity, or even a different output format. You can also tell it to only show you specific test jobs.

The above command will replay events from the specified log, but will only show jobs 11, 34, and 42. Notice that '-v' was used, so we will have verbose output. The original run may not have been verbose, it does not matter.

Times

The 'times' command

The 'times' command will analize a log file and tell you how much time was spent in each test file:
  • Startup is the time between launch and the first 'primary' event.
  • Events is time spent between the first and last 'primary' event.
  • Cleanup is time spent between the last 'primary' event and exit.
Note: Some of this data is best-estimate, for instance 'exit' is when the harness noticed the test was done, not necessarily the exact moment the test finished. If a test is bypassing the Stream formatter and outputting TAP directly then all the numbers could be wrong.

Note: Primary events are assertions, plans, and similar state-changing events.

Preload
Simple

Simple Preloading

You can preload any module with the '-PModule' option. You can specify any number of preloads.

Example, preload Moose before running all tests:

Moose test suite time without preloading Moose (prove has similar numbers)

Moose test suite time preloading Moose first:

Advanced

Advanced Preloading

You can create preload modules with advanced behavior. If the module you preload subclasses Test2::Harness::Preload it will get special treatment.

  • Optional OO - If you provide a new() method then your preload will be instanciated.
  • preload hook - called when loaded
  • pre_fork hook - called before forking for tests
  • post_fork hook - called right after forking, in the test
  • pre_launch hook - called just before the harness drops down to the test
You can also define stages. An example use of stages is when you want to preload multiple sets of conflicting modules. You can load each set independantly (or one after the other if you want to build on top of the previous one). Tests can be marked to run in specific stages.
Project
Init

Init command

The 'init' command is a simple command designed for cpan module authors.

This command generates a 'test.pl' file that will execute all tests under the 't' and 't2' directories using yath.

'test.pl' is executed by most build tools when tests are run. The difference between test.pl and other test files is that it is not run via Test::Harness, and as such only the exit value matters. This is the easiest way to make sure people using your cpan modules will have the tests run by the correct harness.

Config

Configure with .yath.rc file

A .yath.rc file in your project root will provide the default arguments for yath commands. Each command gets a section, and each item in that section is passed to yath as an argument BEFORE any command line arguments. Many options can be negated on the command line if needed, so .yath.rc simply defines the defaults.

Directives
About

What are directives?

Directives are comment you can put in the header of your test file. These comments can control how the harness runs or processes your test.

These comments should appear before any run-time statement. They may follow the shbang (#!) line, and any use statements, or 1-line BEGIN blocks.

Timeout

Timeout Directives

This tells the harness that it should not timeout when running this test. Not recommended.

This tells the harness that it should not timeout on events in less than 90 seconds. By default a test will timeout if no events are recieved for 60 seconds, if you know your test may take longer than that you can set this. If a test exits true, but does not appear to be really finished (maybe it forked a child), the harness will wait up to 15 seconds for more events to come in. If you know it will take longer than this you can use this directive.

Preload

Preload

This tells the harness that this test cannot be run with preloads. The harness will instead execute the test using IPC::Open3 to run it with a completely new interpreter instance.

At the moment there is no difference between this and the HARNESS-NO-FORK directive. That may change.

Fork

Fork

This tells the harness that this test cannot be run by forking (the default). The harness will instead execute the test using IPC::Open3 to run it with a completely new interpreter instance.

At the moment there is no difference between this and the HARNESS-NO-FORK directive. That may change.

Stream

Stream

This tells the harness that the test will break if Test2::Formatter::Stream is used instead of the TAP formatter.

Use this if your test mixes regular tools and hard-coded TAP. Note: DO NOT DO THIS!

Category

Category

The category is used when sorting tests (but only with a -j setting > 1). The primary goal is to make sure to start running long tests first in n-1 slots, while running other tests in the remaining slot.

This is the default, no need to actually specify it:

This is the default for any test with NO-FORK or NO-PRELOAD specified.

Use this to mark tests that are long.

Use this to mark tests that should not be run concurrently with other IMMISCIBLE tests.

Use this to mark a test that cannot be run concurrently with any other tests.

Stage

Stages

The stage directives work along with preload stages. They let you tell the harness which preload stage should run the test. If you specify an invalid stage, the test will run under the 'DEFAULT' stage.

Persist
Start

The 'start' command

This command is used to start a new persistent yath instance.

The above example will start a persistent yath instance with 2 jobs slots. The instance will preload Moose.

Run

The 'run' command

'run' is the same as 'test' except it hands the jobs off to a persistent runner instead of starting a new runner. Note: The job-id will be the process of the 'run' command followed by a sequential number.
Which

The 'which' command

This command tells you where the persistant instance lives (or if there is none).

Watch

The 'watch' command

This command lets you monitor the output of a persistent runner. This will only show you output produced by the runner, it will not show any test results.

Note: STDOUT and STDERR and mixed here, so things may appear out of order.
Reload

The 'reload' command

This command is used to send a sighup to the persistent runner, forcing it to reload anything that was preloaded.

This is normally not necessary as yath will watch files for changes and reload as necessary (excluding any module that changed when it does).

The benefit of this is that it will clear the blacklist allowing all preloads to load once more.

Stop

The 'stop' command

This is used to stop a persistent runner. Output logs will also be rendered.
Plugins

Yath Plugins

Plugins allow you to add custom options to commands or modify how files are found and processed. They must subclass App::Yath::Plugin.

You can use a plugin with the '-p' option: The above will load the App::Yath::Plugin::MyPlugin module. You can specify a fully qualified module name by prefixing it with a +.

Available hooks:

Speed

Performance Example

This table compares the performance of 'prove' and 'yath' when running the test suite for Moose-2.2006. All times were taken on the same system around the same time.

command WallclockCPU TimeNotes
prove -r t 96s 097.74Just prove
prove -r t -j3 49s 145.22prove with 3 procs
 
yath 97.97s097.63Just yath
yath -j3 50.17s146.66Yath with 3 procs
yath -PMoose 40.92s040.22Yath with Moose preloaded
yath -j3 -PMoose20.05s055.68Yath with 3 procs and Moose preloaded

Misc
Stream

The Stream Formatter

By default the harness will tell tests using Test2 to use Test2::Formatter::Stream. This formatter outputs all events to a jsonl file. In addition this formatter prints a timestamp to both STDOUT and STDERR on every event, which allows the 3 feeds to be synchronized later.

Example output:

events.jsonl:

stdout:
stderr:
Model

Directory Structure

  • Each 'run' has a temporary data dir
  • Each test has a directory under the 'run' dir
  • You can ask yath to keep the entire run dir with the '-k' option
The run directory is the 'model' layer. As well anything that processes a log will treat the log as its model layer.
Files in the run dir:
Files in a test dir:
View

The App::Yath namespace

This namespace is where the code for the 'yath' script lives.

Code in this namespace should be UX, this is essentially the view layer.

Controller

The Test2::Harness::Namespace

This namespace is where logic for running and processing tests goes.

This is essentially the controller layer.

TAP

TAP Parsing

Test2::Harness has a TAP parser that is uses when the Stream formatter is not used, or when it detects TAP events.

Unlike prove/Test::Harness this parser WILL parse subtests.

The TAP parser is line based, the subtests are re-assembled and verified after parsing.