• Describe the input and the output you need.
• List required features and constraints.
• Name the candidates and the versions or access tiers being compared.
• Prepare the same shareable tasks for each candidate.
• Define how you will score correctness, usability, and review effort.
• Record relevant costs from current official sources and note the date checked.

Keep provider claims separate from results you observed. If you have only read an announcement, link it and explain what you still want to test.

Share the comparison plan or a completed result set, including limitations. Please explain the use case behind any preference so another reader can judge whether the conclusion applies to their workflow.

A useful contribution includes:

• The official announcement or release-note link and publication date.
• The exact product and version or model identifier.
• The change described by the maintainer.
• The workflow you think could be affected.
• A small test you ran or propose to run.
• Observed results, limitations, and questions still open.

If you compare before and after behavior, record both environments and use the same inputs. Label reports from other people as second-hand and link the original source.

What would you need to verify before adopting the update in your own project?

Propose a test case with:

• A task name and the capability you want to examine.
• A public, self-authored, or clearly synthetic input.
• The complete instructions.
• An expected result or scoring rubric.
• Known ambiguities and what the test does not measure.

Possible task shapes include following a fixed output format, extracting fields from supplied text, or identifying missing information. These are suggestions for test design, with no model performance implied.

Keep cases short enough for another member to inspect manually. When sharing a result, include the model identifier, settings, date, and output. Which first test would help you evaluate models for the work you actually do?