Limitations

This is a test generator, not an oracle. It knows only what the schema says, and the schema is written by a person who may be wrong — or by a heuristic that may be. The failures worth worrying about are the ones it does not report.

What generated tests cannot know

• Intent. A schema says "a string of 1 to 64 characters". It cannot say that the string must be a valid shipping reference, or that two customers must not share one.
• Anything about state. A generated harness constructs an object when new: is present, but it does not set up a database, a fixture or a sequence of calls. Behaviour that only appears after three operations stays untested.
• Side effects. The harness checks the return value and whether the call died. It does not assert what was written to a file, sent over a socket, or stored in a global.
• Concurrency and timing. Tests run one call at a time; races are out of scope.

These are properties of black-box testing, not defects. The response is to keep writing ordinary unit tests next to the generated ones, and to use mutation testing to find out which of them the suite does not really check.

Extraction is a heuristic

• Types are inferred from how a parameter is used. A parameter that is only ever compared for equality looks like a string, whatever it really is.
• Constraints are inferred from comparisons near the parameter. A bound enforced in another function, or by a database, is invisible.
• Inheritance is not followed: a constructor defined in a parent class leaves the schema with an unenrichable new: key.
• Heuristics can also be too eager. A validation regex in the body can be read as an enum; a documented type can conflict with the code. The confidence level says how much evidence there was, and --strict-pod fatal turns a documented-type disagreement into a hard error.

The confidence levels are the part of extraction to read first.

Mutation testing costs time — and can hang

• No per-mutant timeout. The per-mutant prove run is unbounded, so a mutant that causes infinite recursion or a wait on a process stalls the whole run. Skip markers are the workaround; a watchdog around the runner is the other.
• Four operators. Absence of survivors does not mean absence of untested code — only that these four transformations are caught.
• Equivalent mutants. Some mutants cannot be distinguished from the original by any test. They count as survivors and are not a defect.
• In-place writes. Mutants are applied to the real lib/ file and restored afterwards. If the process is killed in between, the file stays mutated; git restore lib/ fixes it.
• A baseline is required. If the suite does not pass before mutation begins, every mutant is reported as killed for the wrong reason.

The dashboard is expensive

• The suite runs twice — once under Devel::Cover, once under the LCSAJ debugger. On a large module this dominates CI time.
• Coverage-guided fuzzing needs Devel::Cover. Without it, fuzzing still runs but gets no branch feedback and no corpus minimisation benefit.
• Property tests need Test::LectroTest and Test::LectroTest::Compat; a generated file skips its property section when they are absent.
• The CPAN Testers section makes HTTP requests unless NO_NETWORK_TESTING is set. With network access and a slow API, a run can wait minutes per report.
• The trend line's y-axis is the arithmetic mean of per-file percentages, while the table's Total row is Devel::Cover's weighted aggregate. They differ by a fraction of a percent, always, by construction.

Platform and portability

• Perl 5.036 or later. Not negotiable; the code does not compile on older perls.
• Windows. The close_stdin option is ignored and generate-test-dashboard is a bash script. The module itself and all the Perl commands work.
• BSD::Resource is used only to cap the signature-evaluation subprocess, and is not available everywhere. Its absence is handled, not fatal.
• Generated harnesses assume a POSIX-ish shell when they shell out; the test suite that ships with the distribution is verified on Linux, macOS and Windows CI, but a harness for code that invokes system commands inherits that code's portability problems.

Things the schema format does not express

• Nested container element types — a hashref of arrayref of objects is a hashref, and only its size is constrained. The generator reports this rather than guessing.
• Relationships between parameters, such as "max must be greater than min". Each parameter is checked against its own spec.
• Ordering constraints between calls.

The distribution tests itself

Two integration tests run every module in lib/ through the extractor and the generator:

EXTENDED_TESTING=1 TEST_JOBS=4 prove -Ilib t

They are the reason a change to extraction is expensive to get wrong, and the reason a failure is usually easy to locate — the end-of-run table names every routine and why it was skipped.