I'm using fast-check to create zod, typebox, valibod, arktype and JSON schemas

[ahrjarrett]

Hello fast-check maintainers,

I've been using fast-check as a core piece of infrastructure while building various libraries that are part of @traversable/schema.

These libraries generate "seeds", which are deeply nested arrays of integers (for performance reasons), and ship a handful of "seed interpreters" that can be used to generate arbitrary schemas, valid data, and even slightly invalid data, for testing purposes.

• @traversable/arktype-test
• @traversable/json-schema-test
• @traversable/typebox-test
• @traversable/valibot-test
• @traversable/zod-test

There are still a few features I'd like to add, and one regrettable configuration option that I need to deprecate (turns out minDepth is a bad idea 😅), but they're almost ready.

I also built a huge OpenAPI v3.0 arbitrary a while back, which needs a lot of work before it's ready to be published. Notably, one of the features that took a long time to build was a "self-referencing" OpenAPI document (which involved mutating the document to move some schemas to components/schemas, to make the generated documents more realistic.

I'd like to revisit building an OpenAPI arbitrary in the near future, but first, I was hoping to get some feedback on the direction of these libraries, to make sure I'm heading in the right direction with the implementation.

Anyway, thanks for taking a look, happy to answer any questions and would love to hear what you think.

[ahrjarrett]

Here's the entrypoint for one of them, in case it helps:

https://github.com/traversable/schema/blob/main/packages/zod-test/src/generator.ts#L259-L297

[ahrjarrett]

To make things more concrete: here's how I'm using it to test that a function that accepts a Valibot schema and turns it into a string results in generating the same schema:

StackBlitz

[gruhn]

Hey, sorry I'm familiar with schemas and rewrite systems but I still don't quite understand what this library is doing and how it relates to fast check.

Is this like a more composable way to construct schemas? Does this allow you to derive fast-check arbitraries from schemas or something?

[ahrjarrett]

@gruhn hey, thanks for checking it out.

Did you look at the StackBlitz? Here's my post about it.

tl;dr, basically these are seed arbitraries that can be interpreted to generate schemas. For example, if you're writing a library that accepts a JSON Schema input, you can use @traversable/json-schema-test to test certain invariants given any JSON Schema object.

The same seed can also be interpreted as a valid data generator, and also an invalid data generator.

[gruhn]

Ah, so you're not testing a function whose input data follows a schema. You're testing a function that operates on schema objects themselves? But this is not the main use case, right? I feel I still don't see the big picture.

[ahrjarrett]

@gruhn yes, that's a valid use case I think.

Here's how I've been thinking about it / using it:

A feature developer would probably be most interested in a library that accepts a concrete schema, and generates some code that would be very painful to have to write/maintain by hand (like a fast-check arbitrary, or TypeScript types, etc.).

If you're working at a level of abstraction above that (for example, if you're building infrastructure for feature developers, or building libraries that are schema driven), there are fewer tools to make building and testing your program easier.

I've also found that the process of building these kinds of libraries/applications seems daunting enough that many simply don't attempt it. They assume (perhaps rightfully so) that building it would take too long, would be too hard, a never-ending source of bugs, etc.

So the main idea behind the library is to make this process easier. The goal is to make building schema-driven libraries and applications trivial – and while we've still got a long way to go, building these arbitraries have been super helpful for my own productivity, and there are certain features I don't think I would have ever dared to attempt had fast-check not had my back :)

[ahrjarrett]

Actually, since you're familiar with term rewriting, there's a simpler analogy I think:

If you think of a schema like an AST, then these arbitraries generate ASTs.

In a way, term rewriters are the opposite of AST arbitraries: arbitraries build up an AST (they're co-algebraic), and term rewriters "tear down" the AST (are algebraic).

I wrote about it here, if you're interested in hearing more about it. In that post I implement both (without using any fancy words, except for in one place where I mention Functors 😅).

[gruhn]

Wow that Function constructor trick is definitely something I would have never thought of 😮

[ahrjarrett]

how it relates to fast check

I didn't answer this part – fast-check generates the seeds.

My goal in sharing this was to see if the fast-check maintainers have any feedback about how I implemented things. But now that some time has passed since I asked the question, I'm realizing that it's probably not very realistic to expect anyone to sit down and read through my code.

I'll close this here, but if anyone does have any thoughts or questions I'm more than happy to discuss! Thanks for taking a look.

[dubzzz]

I was planning to read it but I had some important personal matter to look for. I just reaching the end of the tunnel and will be soon a bit more available.

[dubzzz]

Something I looked for for so long was the fuzz helper you provide from @traversable/zod-test.

It's something I initially planned to put directly within fast-check then dropped as probably too complex and too opinionated to fit all needs. For that reason I never added it directly in the core and let it as a possible side-project that will land as another library plugged on top of fast-check. That's great to see something like that appearing.

Regarding the helpers seedToValidData and seedToInvalidData I have not looked to their implementation yet but I'm very interested into seeing how you implemented the invalid one. I'll probably look at your implementation one day (a bit curious).