shard-spectator/ARCHITECTURE.md

Architecture and Design of Spectator

This document explains the structure and design decisions behind Spectator. It is broken up into the logical components of Spectator:

• Terms
• DSL - Domain Specific Language. Macros and methods that build a spec.
• Matchers
• Examples and groups
• Runner and harness
  • Hooks
• Mocks and doubles
  • Stubs
  • Doubles
• Formatting

Terms

The following are terms and concepts frequently used in the project. They are listed in alphabetical order, but you may find it useful to jump around when learning what they mean.

Assertion

An assertion is a fundamental piece of a test. It checks that a condition is satisfied. If that condition isn't met, then an exception is raised.

Builder

A builder is a type that incrementally constructs a complex object. Builders are primarily used to create specs and example groups. See: https://sourcemaking.com/design_patterns/builder

Config

Short for configuration, a config stores information about how to run the spec. The configuration includes parsed command-line options, settings from .spectator and Spectator.configure.

Context

A context is the scope or "environment" a test runs in. It is a part of an example group that provides methods, memoized values, and more to an example block. From a technical standpoint, it is typically an instance of the class defined by an example_group block. It can thought of as a closure.

Double

Stand-in for another type. Doubles can be passed to methods under test instead of a real object. They can be configured to respond to methods with stubs. Doubles also track calls made to them. An important note: a double is not the same type (nor does it inherit) the replaced type.

DSL

DSL stands for Domain Specific Language. It is the human-like language that comprises a spec. Keywords in the DSL, such as describe, it, and expect, are macros or methods. Those macros and methods make calls to Spectator to describe the structure of a spec. They are combined in such a way that makes it easy to read.

Example

An example is essentially a test and metadata. Spectator makes a distinction between test and example. An example can have a description, group, context, result, and tags. That is to say: an example is the test and information for execution. An example is a type of node.

In the DSL, an example is created with example and it blocks and their variants.

Example Group

An example group (or group for short), is a collection of examples. Groups can be nested in other groups, but can only have one parent. Groups can have hooks. Groups have extra properties like a name and metadata. A group is a type of node.

In the DSL, an example group is created with example_group, describe, and context blocks and their variants.

Expectation

An expectation captures a value or behavior and whether it satisfies a condition. Expectations contain match data. They are bubbled up from the harness to the runner. Expectations can be thought of as wrappers for assertions.

In the DSL, an expectation is the code: expect(THIS).to eq(THAT) An expectation target is just the expect(THIS) portion.

Formatter

A formatter takes results and reports them to the user in a specific format. Examples of formatters are XML, HTML, JSON, dots, and documentation. The runner will call methods on the formatter. The methods called depend on the type of result and the state of the runner. For instance, #example_started is called before a an example runs, and #dump_summary is called at the end when all results are available.

Harness

A harness is used to safely wrap test code. It captures expectations and creates a result based on the outcome.

Filter

A filter selects nodes to be included in a running spec. There are multiple types of filters.

Hook

A hook is a piece of code to execute at a key time. For instance, before a test starts, or after everything in a group completes. Hooks can be run in the same context as a test. These are known as "example hooks." Hooks that don't run in a context, and instead run independent of examples, are called "example group hooks." Hooks are attached to groups.

Label

A label is a string from the spec that identifies a expression. Labels are captured to improve the readability of results and match data.

In the following code, the labels are: does something useful, the_answer, and 42.

it "does something useful" do
  expect(the_answer).to eq(42)
end

Matcher

A matcher defines an expected value or behavior. Matchers are given an "actual" value from a test and produce match data. The match data contains information regarding whether the value or behavior was expected (satisfies a condition). They behave similarly to an instance of a Regex.

In the following code, the eq(42) portion returns an instance of a matcher expecting the value 42.

expect(the_answer).to eq(42)

Match Data

Match data is produced by matchers. It contains information regarding whether an expectation is satisfied and values from the match. The values are key-value pairs identifying things such as "expected value" and "actual value." Match data is similar in concept to Regex::MatchData.

Mock

A mock is a type that can have its original functionality "swapped out" for a stub. This allows complex types to be "mocked" so that other types can be unit tested. Mocks can have any number of stubs defined. They are similar to doubles, but use a real type.

Node

A node refers to any component in a spec. Nodes are typically examples and example groups. A node can have metadata associated with it, such as a label, location, and tags.

Procsy

A procsy is a glorified Proc. It is used to wrap an underlying proc in some way. Typically used to wrap an example when passed to a hook.

Profile

A profile includes timing information for examples. It tracks how long each example took and sorts them.

Report

A report is a collection of results generated by running examples. It provides easy access to various metrics and types of results.

Result

A result summarizes the outcome of running an example. A result can be passing, failing, or pending. Results contain timing information, expectations processed in the example, and an error for failing results.

Spec

A spec is a collection of examples. Conceptually, a spec defines the behavior of a system. A spec consists of a single, root example group that provides a tree structure of examples. A spec also contains some config describing how to run it.

Stub

A stub is a method in a double or mock that replaces the original functionality. Stubs can be attached to a single instance or all instances of a type.

Tag

A tag is an identifier with optional value. Tags can be used to group and filter examples and example groups. Some tags have special meaning, like skip indicating an example or group should be skipped.

Test

The word "test" is overused, especially when using a testing framework. We make an effort to avoid using the word "test" for everything. However, test has a technical meaning in Spectator. It refers to the code (block) executed in an example.

it "does a thing" do
  # Test code starts here. Everything inside this `it` block is considered a test.
  expect(the_answer).to eq(42)
  # Test code ends here.
end

DSL

The DSL is made up of methods and macros. What look like keywords (describe, it, expect, eq, etc.) are just macros and methods provided by the DSL. Those macros and methods are defined in multiple modules in the Spectator::DSL namespace. They are logically grouped by their functionality.

Each module is included (as a mix-in) to the base Spectator context that all tests use. The SpectatorTestContext class includes all of the DSL modules.

The DSL methods and macros should be kept simple. Their functionality should be off-loaded to internal Spectator "APIs." For instance, when the DSL creates an example, it defines a method for the test code and calls another method to register it. While Crystal macros are powerful, excessive use of them makes maintenance harder and compilation slower. Additionally, by keeping logic out of the DSL, testing of internals becomes less dependent on the DSL.

TODO: Builders...

TODO: Tricks...

Matchers

TODO: Base types

Examples and groups

TODO

Runner and harness

TODO

Hooks

TODO

Mocks and doubles

TODO

Stubs

TODO

Doubles

TODO

Formatting

TODO