Skip to main content

​
PRD-007: Holdout test plan (TAT-378)

Status: Draft v0.1, Nov 5, 2026 (PT). Writing this plan is unblocked. Running it is blocked by TAT-379 (studio consent) and TAT-328 (quote data). This commit is a draft, not a completed pre-registration.

​
Research question, variables and hypotheses

The following question, variables and hypotheses are from TAT-378. Research question: Does the UQP PricingModel’s predicted range contain the price the shop actually charged, often enough to be useful? Variables
  • Predicted: the low–high range the PricingModel outputs for a quote request.
  • Actual: the final price charged.
Hypotheses
  • H0: the model’s hit rate is no better than the pass threshold (or no better than a naive baseline).
  • H1: final prices fall inside the predicted range at or above the threshold, and better than the baseline.
The source wording overlaps at equality. The operational decision rule, including how threshold equality and baseline superiority are assessed, is TBD (Gavin).

​
Pass threshold and pre-registration

Pass threshold: TBD (Gavin). TAT-378’s example only: at least 70% of final prices fall inside the predicted range. This example is not an adopted threshold. Gavin must approve the threshold and power target before any sample-size number is chosen. Power target: TBD (Gavin). Sample size: TBD (Gavin). Confidence level and test decision procedure: TBD (Gavin). Pre-registration date: TBD (Gavin). Commit the completed decisions in a dated commit before anyone sees any quote data, including tuning data. Record its commit identifier: TBD (Gavin). Do not tick the checklist until that dated commit exists; the identifier can be recorded in a subsequent audit commit, still before data access.

​
Holdout split

Tune factors and weights on earlier quotes only. Freeze the PricingModel and analysis decisions before testing once on a later, time-based holdout that has never been used for tuning. Do not tune against holdout results or rerun it to select a better model. Split date: TBD (Gavin), pending data availability. Choose and commit the boundary before opening data; if availability cannot be established without viewing data, keep this gate blocked rather than inspect records to select a favorable split. Time field, eligibility rules, handling of missing or ambiguous final prices, duplicate quotes, repeated customers and boundary ties: TBD (Gavin). Frozen model identifier and tuning cutoff procedure: TBD (Gavin).

​
Blinding

Artists do not see predictions during pilots, so predictions cannot anchor their prices. Keep predictions out of artist-facing pilot materials and interfaces. Blinding custodian and verification procedure: TBD (Gavin). This document does not implement pilot tooling.

​
Metrics and naive baselines

  • Hit rate: the share of eligible held-out quotes whose final price is inside the inclusive interval [low, high]. Report the aggregate hit count and denominator alongside the rate; do not replace final prices with deposits.
  • Band width: for each quote, divide high minus low by the midpoint of low and high, expressed as a percentage. Report both the average and median of these per-quote normalized widths. Invalid or nonpositive midpoints follow the pre-registered eligibility rule: TBD (Gavin).
  • Naive baselines: compare against the shop minimum and the median final price for the same size bucket. Freeze the bucket definitions and compute historical medians using earlier tuning quotes only, never holdout outcomes. Shop minimum source and effective-date policy, missing-bucket fallback, point-to-band construction and fair comparison rule: TBD (Gavin). Report baseline hit rates and normalized widths under that frozen comparison rule; do not claim superiority from an uncalibrated point-versus-band comparison.
  • Uncertainty: use a Wilson score confidence interval for the holdout hit rate. Confidence level: TBD (Gavin). Report its limits with the hit rate. Independence assumptions, repeated-studio dependence and limitations of the interval must be reviewed before access; treatment: TBD (Gavin). No sample-size number is set before Gavin chooses the threshold and power target.

​
Ways the test could mislead

  • Wide or flat bands can pass without skill. Interpret hit rate together with band widths and naive baselines; the band-width usefulness criterion is TBD (Gavin).
  • Consenting studios may differ from nonconsenting studios. State that selection bias and do not generalize beyond the consented cohort without evidence.
  • Deposits are not final prices. Reconcile the meaning of the final-charge field before analysis and apply the frozen eligibility rules to ambiguous records.
  • Time shifts, leakage and dependence can overstate usefulness. Disclose them in aggregate reporting rather than silently changing the split or analysis.
Execution depends on TAT-379 and TAT-328; these issue references are execution gates, not authorization to access data. Consent must be recorded per studio before inclusion. Consent record location and authorized custodian: TBD (Gavin). Any later authorized study must use read-only, anonymized inputs and release aggregates only. Do not include customer or studio identifiers, raw quotes, payment records or row-level examples in this repository or reports. Access approvals, anonymization review, retention/deletion policy and aggregate suppression rules: TBD (Gavin). This task accesses no study data, including Stripe, Whop, GoHighLevel or tattoo.co data. It performs no scraping, outreach, spending or writes to external systems. This is a plan only: no evaluator changes, site copy changes or PRD-008 work.

​
Pre-registration checklist

Every box must be ticked before any data is opened. Unresolved decisions block execution; consent and data gates cannot be waived by completing this document.
  • Gavin approves the question, variables and operational H0/H1 decision rule.
  • Gavin fixes the pass threshold, power target, sample-size justification, confidence level and decision procedure, without adopting an example by default.
  • Gavin fixes the time-based split, eligibility and missing-data rules.
  • Gavin fixes the tuning/freeze procedure and holdout isolation controls.
  • Gavin fixes baseline construction, bucket definitions, fallbacks and comparison.
  • Gavin fixes width usefulness criteria and dependence treatment.
  • Gavin assigns the blinding custodian and verifies artists cannot see predictions.
  • TAT-379 studio consent and TAT-328 quote-data readiness are confirmed, with consent recorded for every participating studio.
  • Gavin approves read-only access, anonymization, aggregate-only reporting, retention and disclosure controls; no data has yet been opened.
  • Every unresolved decision has been resolved and the completed plan committed in a dated pre-registration commit, whose identifier is recorded before access.