Writing a runner

This package ships the conformance set of the Curly Message Format and a runner for it in JavaScript. The fixtures are the artifact; the runner is one reading of them. An implementation in another language needs a runner in that language, and this document is what such a runner is held to.

README.md states what a fixture file holds and what an adapter supplies. This document states what a runner does with them, and how a runner is measured before it is trusted to measure anything.

What a runner does

A runner reads the fixture files, drives an implementation through the adapter of section 14.3, and answers, case by case, whether what the implementation produced is what the specification requires. It decides nothing else. Where an implementation and a case disagree, the specification says which is wrong, and a runner that resolves the disagreement itself has stopped measuring conformance and started defining it.

That makes the runner's correctness load-bearing in a way its convenience is not. A runner that compares nothing passes every implementation, including the ones that are wrong, and reports the same green summary either way. The catalogue described under Auditing the runner is how a runner is kept honest.

Reading the set

The set ships as JSON: index.json lists the fixture files with their level — or, for a file that pins the tree rather than a resolution, their kind — their section and their case count, and each file under fixtures/ holds the cases. schema/fixture.schema.json describes a file. A runner reads the manifest, or the directory; both are sorted by file name, and a runner reports in that order so two runs of the same set line up.

format in every file is the versioned identifier of the format the file targets, and is curly-message-3 for version 3. A file whose format a runner does not know is a file it must refuse rather than skip: a set a runner could not read is not a set an implementation passed.

The files a release ships are held to the schema, and to objects that repeat no key, by the set's own tests, so a runner reading a release may rely on both. A runner that reads files from anywhere else checks them first: against the schema, or, on a host with no JSON Schema validator, by hand for what the schema states — the exact keys each object may hold, the members it requires, the type of each value, the values an enumeration or a constant allows, and the patterns, lengths and alternatives the schema adds — rather than taking on a dependency to run the set. It also refuses a file in which an object repeats a key: hosts' parsers differ over which of the two they keep, and the schema, which sees the object once it is parsed, cannot state it. The JavaScript runner makes neither check: load and --fixtures read a directory as it stands, so files from anywhere but a release are checked before they are handed to them. A runner that reads the set from the release's archive, conformance-<version>.zip, and pins it by its digest checks the archive before it opens it, against the sha256 the release notes state or, from 4.2.0 on, the conformance-<version>.zip.sha256 the release carries beside it, which sha256sum -c reads.

Decoding a case's inputs

A case's inputs are JSON, and the format's data model (section 4) is not. Four constructions cannot be written as JSON, and a case writes each as an object carrying a $curly key, which a runner replaces with the host value the tag names:

The replacement is by the value's own shape, everywhere in the inputs and at every depth, and nowhere in the expectations: an expectation names text, and a tagged value read in one would name something a case cannot state.

Each tag is host-specific by design, which is why it is a tag and not a literal. A host with no notion of absence, or whose serialization never follows a reference twice, has its own answer for what stands in, and the set does not prescribe one.

The adapter states its stand-in for undefined: the value its implementation reads as the caller having passed nothing. Where it states none, the runner leaves out whatever holds the tag — the entry of an object, or a message the caller then does not supply — which is what the specification makes of the host's undefined almost everywhere: an entry that holds it is absent, and a layer that holds it names nothing. A wrapper is the exception, because a wrapper owns at least one key (section 4.1): with its one member left out, { "value": { "$curly": "undefined" } } is an empty object, which is a value. An implementation that reads a wrapper owning a member that holds nothing states a stand-in for it. So does one that runs the constructions at and one past the conversion limit, which hand over an object whose members, and the members of the object it shares, hold the host's undefined (see Building a generated case).

A runner on a typed host may decode a case into a neutral value of its own — the kinds JSON spells, and a kind of its own for each tag — and have the adapter map that onto the implementation's types. The cases are measured on what the implementation answers, so the mapping is the adapter's business as the rest of its shape is; the comparison is the runner's, and does not change.

A case's messageId is any JSON value, because no step of resolution reads it (section 4). An implementation whose ids are text hands any other shape on as no id. Nothing observes the difference: an expectation names an id only as text, so a case whose id is of another shape leaves it out of what it expects.

The adapter

The adapter is the implementation's side of the contract, and its shape in another language is that language's business; what it must carry is not. README.md states it. Five things are worth repeating here, because they are what a runner reads before it runs anything:

• levels selects the fixtures. An adapter must claim core.
• limits builds the generated cases.
• unexpressible, where the adapter supplies it, leaves out the cases whose formatting request reads what the host's facility cannot express: a property, or a value of one, in every request or in the requests whose other properties hold given values. An entry is a property name, or names the property and narrows it to the values it lists, to the requests meeting the condition when states, or to both. An entry that names no property, lists no value, states an empty condition or holds another member is an error, as a claim outside the vocabulary below is.
• cst, where the adapter supplies it, is the tree of CST.md: the unit its spans are counted in, and the call that produces one. An adapter that supplies none has the tree cases left out.
• reentry, where the adapter states it, is the bound section 13 lets an implementation keep on the counted reads producing their text at once on one thread of execution, the resolution asked for not counted, and builds the constructions of kept reads. keeps, stated as false, says that no modifier can keep an input past its call.

The optional claims are unexpressible, cst, reentry and keeps; levels and limits are not. A runner on a typed host, whose adapter cannot leave a member out, reads unexpressible and cst loosely: an unexpressible that names no property, or a cst stated empty — no unit and no call that produces a tree — is no such claim. It reads reentry and keeps loosely by no value they can hold, since 0 is a bound and false a claim: it gives the adapter a member it can leave unset, an optional or a nullable one, or a keeps named for the claim it makes, whose zero value is the claim unmade. An unset reentry states no bound, and an unset keeps reads as true. It reads an entry of unexpressible loosely too: values or when stated empty is no narrowing, so the entry stands for every value of its property, or in every request. An adapter that can leave a member out states no claim by leaving it out, and the JavaScript runner refuses a cst that holds no call and an entry whose values or when is empty. An answer is not a claim either: an empty list of reports is reports observed and none made, and an empty list of requests is requests exposed and none made, both measured as they stand.

A claim outside the vocabulary of sections 2, 11.2, 11.5 and 13, or of section 4 of CST.md — a level that is not one of the three, a limit that is not a positive count, a reentry that is not a count of 0 or more, a keeps that is not true or false, a facility the format does not name, a span unit that document does not name — is an error, and the runner refuses the whole run. It is not a skip and not a failure: an adapter that cannot say what it satisfies has not been measured, and a run that reports a number for it reports a number that means nothing.

Planning: what runs and what is left out

A case runs where its file's level is one the adapter claims. A case left out is listed with the reason it was left out, never dropped silently: a set whose skipped cases are invisible is a set whose coverage cannot be read off the summary.

Seven reasons leave a case out, and no others:

• Its level is one the adapter does not claim, or one the caller excluded.
• Its formatting request reads what the adapter declared it cannot express (section 11.2): a property it named, a value of one among the values it listed, in a request whose properties hold every value its condition states. A value is compared as a request's are, by its entries and with the sign of zero.
• It pins the tree and the adapter offers none. A tree file declares no level, because CST.md is not one of them; whether the adapter offers a tree is what selects its cases.
• It is conversion-at-limit or conversion-over-limit, and the adapter states no stand-in for undefined. Each counts its nodes in members that hold nothing, and with those members left out it is an object of empty objects far short of the limit, which pins none.
• Its inputs hold a refusing tag, and the host has no value the implementation reads whose entries raise where they are read, as a host whose payload and props reach the implementation as a value model of plain data has none (section 4). The runner states it, since it is what builds the value; an adapter declares nothing for it. Leaving the tag out instead would hand over props that refuse nothing, and the case would fail for the reports it no longer meets rather than pin what it states.
• It is one of the five constructions of kept reads, and the adapter states no reentry: each is built from that bound, and an implementation that keeps none has no bound to meet. reentry-own-reads-free is left out at a reentry of 0 as well, where no counted read stands for the walk's own reads to be made inside. A written case that registers nest under keeping is left out where the adapter states a reentry of 0, and runs where it states none: an implementation whose bound counts resolutions and is documented as 1 states 0 and refuses the resolution nest asks for.
• It reads an input a modifier kept after that modifier returned, and the adapter states keeps as false: the five constructions of kept reads, and a written case that registers keep or first under keeping. The set writes no case that reads a kept input so through keep-default, take, taking and nest alone, and its tests hold every case to that, so a case that reads a kept input while the call of the modifier that kept it is in progress still runs, the read a conversion or a nested resolution makes included.

Section 13 also lets an implementation bound, across the process, the resolutions asked for inside another, and sets no minimum for that bound. No adapter statement covers it: the set assumes, whatever reentry an adapter states and where it states none, that such a bound, where one is kept, lets one resolution asked for inside another stand, as the written case that registers nest asks for one.

Building a generated case

Section 13 states minima and requires an implementation to document what it permits, so a case at a limit cannot be written out: the runner builds it from the limits the adapter declared, or from the bound on counted reads it states. README.md gives the twenty-one constructions, the message and payload each builds, and what each expects. A runner builds them from the adapter's own numbers, not from section 13's minima; an implementation that permits more is exercised at what it documents.

Each of the four limits section 13 states a minimum for is built at its bound and one past it, and each construction is shaped so that it spends one budget alone. The output constructions write their text from an option value, which is the message's own text and so costs no reading; the read construction hands over a long value the placeholder selects nothing from, so almost nothing of it reaches the output. The constructions at and one past the conversion limit hand over an object that holds one object several times and members that hold the host's undefined, and the shared object's own members hold it too: section 13 counts a value reached twice twice, and a member the walk visits and then omits, so the serialization visits as many nodes as the construction states and writes little more than the names of the shared references, and the node count meets the limit long before any bound on the text. The construction past the limit that continues hands over an array of strings instead, whose text would run past the read limit, because what it pins is that a value no conversion describes spends none of that budget. A construction that spent two budgets would fail an implementation for the limit the case was not about. Five nesting constructions are the exception, and meet a second limit, a report of the payload or a second message error at a placeholder nested too deep because that is what they pin: section 9 tests the read limit first, then looks the value up and converts it, and only then decides the nesting limit, before any other message error.

The nesting limit is built once more, far past it. A refused placeholder takes its fallback chain, and the chain's last link is its inline default, whose placeholders are refused in turn. Section 13 has no limit raise, so a chain of them twenty thousand levels past the limit resolves like any other message. The runner builds that message without recursion of its own, and so should a runner in another language.

The bound on counted reads is built from inputs a modifier keeps. The behaviours keep, keep-default, take, first, taking and nest share one list of what was kept, and its claims, and a runner builds them afresh for each resolution it asks for where none of its own is in progress, so that nothing one resolution kept is read in another but the one a nest asks for while its call is in progress, which shares the list; the JavaScript runner builds them with keeping(adapter), outside the behaviours record, which holds no state. reentry-at-limit keeps B defaults that each take, and its last take reads the one kept last, which reads the one kept before it, so B counted reads stand one inside another; reentry-over-limit keeps one more, so the innermost read is refused and answers the empty string, and the take that made it, written between brackets in the second input's default (the take after them where B is 0), answers empty; reentry-refused-not-recorded then reads the input kept first, which the refused read did not record as produced; reentry-over-limit-by-a-conversion has the read past the bound made by the conversion of taking's answer, where reentry-over-limit has a take; and reentry-own-reads-free has the innermost of B counted reads render modifiers that read their own defaults N − 1 levels deep, reads the walk makes that count toward no bound. keep answers the empty string and take and first the text they read, or empty for the empty string, so no result is longer than [empty] whatever B is, and the value they read is the empty string, which spends none of the read budget; each renders a default at the depth the message writes it, so none of them meets the output, read or nesting limit.

taking answers a host value whose ordinary string conversion (section 4) runs code: an instance of a class with a toString of its own in JavaScript, an fmt.Stringer in Go, Value::host of a type that implements Display in Rust, whose fmt makes the read, a Stringable in PHP, an object with __str__ in Python. A runner never builds it as text computed in the call, which moves the read out of the conversion, and never as a plain object or map, which serializes and fails the case. A correct implementation refuses the read a runner that builds text makes in the call the same way it refuses the conversion's, so that runner passes the set and no defect the audit applies tells it apart; a port checks in its own tests that the read is made by the conversion. Section 13 has a resolution convert a value no more than once observably, so the conversion reads once.

nest is defined by what it resolves, not by the entry point it calls: a resolution, by the implementation under test, of {{x:take; default:F}}, asked for on the thread of execution its call runs on while its call is in progress, whose modifiers are the six behaviours sharing the list of what was kept, and its claims, with the resolution around it. A runner that hands an adapter the behaviours themselves, as the JavaScript runner does, builds nest over the adapter the case runs against, so that adapter's resolve must be re-entrant — called again on the same thread while a call to it is in progress, each call answering its own resolution with its own reports — and in an audit the inner resolution goes through the defect as well. An adapter that builds the behaviours from their names builds nest itself, resolving with the implementation over the list it built for the resolution in progress, never through a call that builds a fresh list, whose take finds nothing kept and answers none; its inner resolution goes through no defect in an audit, which changes no outcome, since every defect that wraps resolve already alters the resolution around it, and the inner one reports nothing. Either way, nest raises where the resolution it asks for raises or answers no text, which the implementation contains as a failed modifier, and the case fails.

Executing a case

A runner hands the adapter one resolution's inputs and compares what came back, in this order. The first mismatch is the failure; what follows it is not reached.

1. The answer's shape. An adapter that raised, or answered with something that is not a resolution — on a host that has them, a promise of one is not — has failed that case, and only that case. A raise from the implementation is a result, not an accident: the run continues and the next case is put to the same adapter.
2. The output, or the formatting request. Below.
3. The case's own check, where the construction carries one.
4. The reports, where the adapter observes any.

The output

The output is compared exactly: character for character in the host's own string unit, with nothing trimmed, collapsed or normalized. Section 8 makes whitespace significant text, and a runner that trims an output before comparing it passes an implementation that loses it. This is the single easiest way to write a runner that measures nothing, because the trimming reads as tidiness.

A formatting request

A case at the Intl level states a request — the facility, its properties and its input — rather than the text a host makes of it. Section 11.2 lets an implementation expose the requests a resolution made, and where the adapter does, a formatting case is compared on the request and not on the output; a selection case, below, is compared on both. That is what lets an implementation whose locale data is not the runner's conform: the request is what the specification pins, and the text is what two hosts' CLDR data would have to agree on.

Where the adapter exposes no request, the runner performs the case's request on its own host and compares the text. Both readings are correct; they measure different things, and a runner supports both.

The JavaScript runner performs a request through ECMAScript's Intl, which takes it as the case writes it. On a host whose facility does not take ECMA-402's requests, the runner performs one through a performer of its own, which maps the request's facility, properties and input onto the host's facility (section 11.2). The performer imports nothing of the implementation's formatting path: a translation the two shared would agree with the implementation whatever either of them got wrong. It is as wide as the requests of the cases the run holds, which leaves out those that read what the adapter declared it cannot express, and a request of the run outside it — a facility, a property or a value it does not map — ends the run rather than failing or skipping the case, since the gap is the runner's and not the implementation's. The port documents the library the performer formats through, and its version, so that a text it expects can be read against the locale data that made it. A recording of another host's answers to the set's requests, kept in the port and read in place of a facility, is such a performer as well. The set ships none: section 11.2 lets two hosts' locale data differ, and a recording the set shipped would pin one host's on every other.

A request is compared by its entries and never by the text a serialization makes of it, because the order of a request's properties is the host's. A case that states a request writes the placeholder alone, so the resolution it describes makes exactly one request: more than one, or none, is a failure.

A request's numbers are compared with the sign of zero, which JSON writes alike for both. Section 11.2 applies the sign of a delta to the count it rounds to, so a negative delta under half a second asks for a count of -0, and a relative time spelled out reads that as the past and 0 as the future; a runner that compared the two alike would pass an implementation that drops the sign. A case writes such a zero as -0.0, because a decoder that reads -0 as an integer reads it as 0.

A selection request (section 11.5) answers with a category, not with text, and the case names the output each category selects. On the text reading the runner performs the request, takes the category its host answers and expects the output named for it; a category the case names nothing for is the set's defect and ends the run. Where the adapter exposes the request, it exposes the category its host answered with it, and the runner compares the request and then holds the output to the one named for that category. An exposed selection request that carries no category, or one the plural rules do not name, is a failure: the output would be measured against nothing.

The reports

Reports are compared in order. Section 14.3 has an implementation report in walk order: where neither of two reporting placeholders holds the other, the one the message writes first reports first, and where one holds the other, the inner reports while the outer's value is being read. The order is part of what a case pins, and comparing them as a set loses it. The count is compared as written: a report more is a failure, and so is a report fewer.

A report is compared by its code, always. Its origin, its id and its limit are compared where the adapter's report carries them and the case expects something of them, because the specification prescribes a report no shape — only what a code names and which origin it declares. A field the adapter does not carry is not a failure and not a pass; it is unobserved.

An adapter that supplies no reports at all says the implementation does not report, which section 14.3 permits: reporting is a SHOULD. Every expectation about reports is then skipped — and said to be. A case that passed without its reports checked passed less than one that passed with them, and a runner that reports the two alike overstates what it measured.

Executing a tree case

A file whose kind is tree pins the concrete syntax tree of CST.md rather than a resolution. Its cases run wherever the adapter offers a tree. A runner puts the message to that tree and compares what came back, in this order; again, the first mismatch is the failure.

1. The answer's shape. An adapter that raised, or answered with something that is not a message node, has failed that case.
2. The root's span, which is the whole message.
3. Every node, in document order: its kind is one section 6 of that document names, its span is a pair of offsets whose end is at or after its start, it lies within the message and within the node that holds it, neither boundary falls inside a code point, a name kind carries a name, and an escape carries cancels.
4. The properties of section 5: the leaves tile the message in order and spell it back.
5. The case's own expectation, node by node.
6. Determinism: the same message put to the tree again answers the same tree.
7. Agreement, where the case states resolves: the same message put through resolve produces that output. The tree and the resolution then read the same placeholders, which is what property 4 requires and what a second scan written separately is how an implementation loses.

The order is what makes a failure legible. A tree whose leaves do not tile the message is wrong about the message whatever the case says of its nodes, so that is what the failure names.

Two of those need care in a port.

A runner reads spans in the unit the adapter declared, and in no other. A case states the text a node spans rather than a number, so that one implementation's unit is not pinned on every other. A runner turns the declared unit into offsets into the string its own host holds, once per message, and reads each span through that; where a boundary has no offset, because it falls inside a code point, the case fails, which is what section 4 requires. An implementation that counts one unit and declares another fails on the first message that tells the two apart.

A tree is a structure the implementation supplies, and nothing stops it holding itself. A runner walks it under a bound derived from the message — a tree describes the message, so it holds no more nodes than the message can spell — and a cyclic answer fails that case rather than hanging the run.

Outcomes

A runner answers one of four things for a case, and a summary that cannot say which is a summary that cannot be acted on:

A failure names the section, because a failure is a disagreement with a sentence and the section is where that sentence is, and names the document too wherever that is CST.md rather than the specification — the two number their sections separately. A failure that only says which case failed makes the reader find it.

Auditing the runner

A runner is trusted with a verdict, so it needs a verdict of its own. This package ships defects.json: a catalogue of ways an adapter can be wrong, each with what a correct runner answers where it is present. schema/defect.schema.json describes the file.

The procedure is the same in any language. Take an adapter that passes the set. Apply one defect to it — wrap it, so the adapter is correct in every way but one. Run the set again. A defect is realized as its sentence reads for the adapter in hand: the one that misdeclares the unit of the tree declares a unit other than the adapter's own, whichever that is, since declaring the adapter's own would alter nothing. The two that declare what the adapter cannot express add their entry to what the adapter declared itself, since replacing it would run the cases the adapter documented a gap in, and fail them. Each is unreachable where the adapter's own declaration already leaves out every case of the run whose request the defect's entry covers: the defect then leaves out no case the run without it held. A correct runner answers what the catalogue states; a runner that answers nothing has found a hole in itself.

expects says what that answer is:

A defect that the adapter never gives the runner anything to catch is neither caught nor missed. An implementation that observes no reports cannot carry a defect of its reports; one that claims Core alone cannot carry a defect of a level it does not claim; one that states no stand-in for undefined runs no construction at the conversion limit, so a conversion limit stated one higher alters nothing it answers, though one stated one lower is still caught past the limit, by the construction that continues. One that states no reentry, or states keeps as false, runs no construction of kept reads, so a bound stated one higher or one lower alters nothing it answers, and one that states a bound of 0 has no bound one lower to state. A bound stated one higher is caught by every construction of kept reads, and one stated one lower only by reentry-over-limit, reentry-refused-not-recorded and reentry-over-limit-by-a-conversion, since at the bound either bound lets every read through. A defect is unreachable where applying it changed nothing the adapter answers, and an audit that calls those missed sends a port hunting for holes that are not there. The catalogue is most informative against an adapter that claims every level its implementation satisfies and observes what it reports.

The defects are in defects.json, a sentence each, in five families: whether an adapter answers at all (it raises, or answers with nothing), what it answers for the output (truncated, trimmed), what it answers for the reports (dropped, added, reordered, each field altered in turn, or withheld entirely), what it states about itself — its limits and its bound on counted reads among them — and about a formatting request, and what it answers for the tree (a node dropped, a node retyped, a span moved, a name left as the message spells it, an escape read the other way, and the unit misdeclared or unnamed). Each defect names the section it pins, and a defect of the tree names the document that section is a heading of. Each one pins a decision a runner makes, and a runner that misses one is a runner making that decision by not making it.

This package's own runner carries the catalogue as mutations and runs it through audit:

import { audit } from '@curly-message/conformance';

const missed = audit(adapter).filter(({ outcome }) => outcome === 'missed');

audit answers one entry per defect: what the catalogue expected, what this runner observed, and whether that counts as caught, missed or unreachable. It requires an adapter that passes the set, because a defect is measured by what it changes and a baseline that already fails changes nothing legible.

What a runner does not decide

A runner does not decide what the format means. Where a case and an implementation disagree, SPEC.md decides which is wrong, and the answer is a change to the set, to the implementation, or to the document — never to the runner. A runner that special-cases a case to make it pass has made the set measure that runner instead of the format.

A runner also does not extend the set. A behaviour the specification leaves to the host is not a case, and a runner that adds one has made an implementation non-conformant for doing what the document allows.