A model release is a starting question

Notebook principle 02: turn each model announcement into a question I can test.

I want release notes here to name a useful task and a way to check it. Provider claims and my own measurements should remain visibly separate.

This is the notebook’s approach to new models and emerging ideas. Announcements provide hypotheses. Small, repeatable tasks provide a way to investigate them. Until a result has been measured, I treat it as a question rather than a conclusion.

Four things every model note should record

1. The task. A precise input and a useful, inspectable outcome.
2. The conditions. The model version, tools, instructions, and available context.
3. The cost of the whole attempt. Include retries and the time spent correcting the answer.
4. The failure. Record the case that did not work, not just the response worth sharing.

Compare the same job

For comparisons in this notebook, I want the task to stay stable. Change the model, keep the question and success criteria, and inspect the outputs. If a model needs different instructions to do well, record that too. The workflow is part of the result.

Leave room for “not yet”

Some ideas are interesting before they are dependable. I want to keep that distinction visible here: research to read, experiments to run, and results that have actually been checked. New release notes will earn their place by connecting to a concrete question.