How we test · The Plain Web Stack · diadem.fun
The Plain Web Stack6 / 15

How we test

The three libraries have about 9,800 unit tests between them. They all passed, and the libraries still had real bugs. Each of those bugs only appeared when the code ran the way an app runs it:

  • a real server talking to a real browser;
  • a real database file written by an older version;
  • default options rather than the convenient ones tests use. So every guarantee in this book is checked by one of three kinds of test: | Kind | What it is | Example | |---|---|---| | Generated matrices | thousands of seeded cases compared with an independent answer | 3,072 random queries, each compared with hand-written SQL on the same data; failures print the seed and the smallest failing case | | Real-run tools | scripts that run the real product end to end | the starter app in headless Chrome (30 steps); an old published version builds a database file and the new version opens it | | Measurements | numbers against a baseline, compared on the same machine | ORM speed as a fraction of hand-written bun:sqlite | Three rules make these tests worth trusting:
  1. A fix counts only if reverting it makes a test fail. Every fix in the changelogs was checked this way, for example "66 of 71 cases fail without it".
  2. Look at the screenshots. The browser tool saves one per step. One fix came only from looking at them: every note had an open edit box, so a page of 20 notes was a wall of text boxes.
  3. Measure the same way twice. Machine load swung speed results by about 35% between runs, so speed is reported as a ratio to plain SQL measured in the same run. A change that did not measure faster was not released, even when it looked clever. Fig. 1: a wrong password brings the login form back with a message (starter app, real-browser run) Next: chapter 7, what this found.