Six ways to score generated order flow

The same continuations. Different comparisons. This guide follows the Wasserstein scoring path from input trajectories to feature scores and aggregates. Shared operations are black; a different colour marks what each protocol changes.

Two meanings of “conditional”

These are separate choices. Feature-conditioned scoring does not ask the model to regenerate data at a specified spread or volatility. Here, a continuation means events after the historical context, not a later phase of training.

Both descriptions assume historical context was supplied during generation. Neither scoring mode preserves the identity of each history–future pair; that is the purpose of context_paired.

Naming and implementation status. The two-word names below are explanatory names. Existing CLI identifiers remain listed alongside them; the unified rename is not yet deployed. context_paired is a new local implementation, defaulting to one generated continuation per history. It is not a new set of measured leaderboard results and does not retroactively change historical scores.

Six methods at a glance

Choose a method by the question it answers. All six can score continuations generated from historical context; what differs is how observations are compared after generation. The table describes the implementations documented here, including the new local paired scorer.

For feature_conditional, context_binned and time_lagged, the current code skips undefined group scores without renormalizing their weights. Report the covered weight alongside the score. Scores across these six protocols are not directly interchangeable because their comparison units, weighting and normalization differ.

0. Shared inputs and notation

One history, one observed future, possibly many generated futures

Let \(i=1,\ldots,M\) index historical contexts. \(C_i\) contains \(L\) observed events. \(R_i\) is its one realized continuation of \(T\) events. Each \(G_i^{(k)}\) is a separately sampled continuation from that same context, not a subdivision of a single continuation.

For example, \(L=T=1000\) means 1,000 historical events followed by 1,000 future events. It is an illustration, not a requirement of every benchmark artifact. The new paired protocol requires a common continuation event count within a scoring run.

Reading \(K\) throughout this guide. \(K_i\) always counts complete generated continuations for history \(i\), not events, bins, feature pairs or generation windows. Write \(K_i=K\) only when every history has the same count. For the existing modes, the number must be read from the run's generation manifest or matched generated file pairs; it cannot be inferred from the mode name. This guide specifies supported counts, not an audited numerical count for every historical leaderboard artifact.

Independent repeated generation should restore the same context state before each sample and vary the sampling randomness. A start token alone does not establish whether historical information was used.

Features and empirical distributions

\(f\) is a target feature; \(g\) is a conditioning feature. Write \(V_f(X)\) for the list of feature values extracted from trajectory \(X\). Its length need not equal the event count: spread is usually event-level, rolling OFI yields fewer values, and a trajectory-level volatility or return yields one scalar.

Symbols used in the next equation: \(X\) is any trajectory; \(A\) is any nonempty list of extracted values; \(a\) is one entry in that list. \(V_f\) is the extraction rule for feature \(f\). \(\widehat P_i^f\) is the observed continuation's feature distribution, while \(\widehat Q_{ik}^f\) is the corresponding distribution for generated member \(k\). \(P\) always denotes the real side and \(Q\) the generated side in these comparisons.

Here \(\delta_a\) is a unit point mass. These expressions construct distributions; they do not reduce a trajectory to its mean. The symbol \(\uplus\) below concatenates lists while preserving duplicates.

The common distance

\(P,Q\) are probability distributions over a scalar feature, such as spread. \(F_P(x)\) is the probability under \(P\) of a value at most \(x\); \(F_Q\) is defined the same way. \(u\) is a percentile level between 0 and 1. \(F_P^{-1}(u)\) means the quantile function, not \(1/F_P(u)\): the smallest threshold at which the cumulative probability reaches \(u\) (more precisely, its infimum). For an empirical list \(A\),

Here \(x\) is a numerical threshold, and \(\leq\) means “less than or equal to.” For example, for \(A=[1,1,3]\), \(F_{\mathcal E(A)}(1)=2/3\).

\(W_1\) denotes Wasserstein distance of order 1: it uses absolute distance as its transport cost. The subscript 1 is not a feature number. For one-dimensional distributions with finite first moments,

If both empirical distributions have \(n\) equally weighted observations, sorting gives

In the sorted formula, \(n\) is the common list length, \(r\) is the rank from 1 to \(n\), and \(a_{(r)},b_{(r)}\) are the \(r\)-th smallest entries of the two lists. Parentheses around \(r\) mark a sorted rank. They have a different role from the generated-member label \((k)\) on \(G_i^{(k)}\).

Unequal sample counts are valid; each distribution still has total mass one. A single value on each side gives \(W_1(\delta_a,\delta_b)=|a-b|\). The implementation uses SciPy's one-dimensional Wasserstein distance.

Normalization in the existing scorer

For each distance call, the existing scorer combines the real and generated target values, \(Z=A\uplus B\), and computes their joint sample standard deviation:

Here \(A\) and \(B\) are the real and generated value lists for this particular comparison; \(Z\) is their concatenation; \(z\) is one entry of \(Z\); \(\bar z\) is their pooled arithmetic mean. The denominator \(|Z|-1\) defines the sample variance. \(s(A,B)\) is its square root. \(\mathcal W\) denotes the normalized distance, whereas \(W_1\) denotes the distance in the feature's original units.

Both sides are centered and scaled by these same values. Common translation cancels, so for \(s(A,B)>0\) the resulting score is

This scale depends on both real and generated observations. In grouped modes, it is recomputed within each group. It is not a fixed real-only reference scale. Empty sides or zero variance are not evidence of a perfect match.

1. feature_marginal

Generated continuations per history: \(K_i\in\{1,2,\ldots\}\), determined by the input files; this scoring mode does not fix a numerical \(K\). It pools feature observations from all \(K_i\) members. If each history has one generated file pair, \(K_i=1\); if each has \(K\) pairs, \(K_i=K\). Different histories may have different counts.

Question: Does each generated feature have the right overall distribution?

Step 1 — Pool before comparing

Reading the pooling symbol. \(\biguplus\) means “join all these lists and keep duplicates.” It does not add their numerical values and does not remove repeated values as a set union would.

For two histories, if \(V_f(R_1)=[1,1,2]\) and \(V_f(R_2)=[2,3]\), then

The result has five observations. Its empirical distribution assigns probability \(2/5\) to 1, \(2/5\) to 2 and \(1/5\) to 3. It is not the sum 9 or the mean 1.8.

In the next equation, \(A_f\) is the complete real feature-value list; \(B_f\) is the complete generated feature-value list. The first blue operator loops over histories. On the generated side, the outer operator loops over histories and the inner operator loops over the generated members for each history.

Difference: the context identity disappears before the distance is computed. A trajectory producing more feature observations contributes more mass to the pooled distribution. More generated members for a context also contribute more generated observations.

Step 2 — Compute one normalized distance per feature

\(D_f^{\mathrm{marginal}}\) is one scalar distance for feature \(f\) under this protocol; its superscript is a name, not an exponent.

Step 3 — Average the complete feature set

\(j\) indexes features, \(f_j\) is the \(j\)-th feature, and \(S_{\mathrm{WS21}}\) is the final equal-feature average. “WS21” labels the complete 21-feature score. For the complete 21-feature contract,

Features are equally weighted, regardless of their observation counts. A mean over fewer available features must not be silently presented as complete WS21.

What it misses. Suppose the real outcomes for two histories are 0 and 1, but the generated outcomes are 1 and 0. Both pooled distributions are \(\{0,1\}\), giving zero distance even though both historical pairings are wrong. This is the distinction between context-conditioned generation and context-agnostic scoring.

2. context_paired

Question: Does a generated future match the realized future associated with its own history?

Step 1 — Keep the context identity

The new default uses \(K_i=1\). Write \(G_i=G_i^{(1)}\) and \(\widehat Q_i^f=\widehat Q_{i1}^f\).

Difference: compute the distance inside each historical pairing before averaging across histories.

Here \(d_{i,f}\) is the unnormalized distance for one history \(i\) and one feature \(f\). The subscript \(i\) is retained so that the two distributions cannot be exchanged across histories before scoring.

Step 2 — Give each history equal weight

\(D_f^{\mathrm{paired}}\) is the arithmetic mean of those per-history distances for feature \(f\).

For scalar outcomes, \(d_{i,f}=|f(R_i)-f(G_i)|\). For event-level features, it compares within-path empirical distributions. These are different representations of a future, not interchangeable measurements.

Step 3 — Use fixed reference scales for an optional feature aggregate

\(J\) is the number of selected features; \(j\) indexes them; \(s_{f_j}^{\star}\) is a supplied positive reference scale in the units of feature \(f_j\). Dividing by it makes each term dimensionless. \(S^{\mathrm{paired}}\) is the scalar average across all selected scaled feature scores. The star labels a fixed reference, not an optimized value.

Difference: the positive scales are fixed from separate reference data and reused across models. They are not joint real/generated standard deviations recomputed for each model. Without supplied scales, the implementation reports raw feature means and leaves the cross-feature aggregate undefined. Its numerical values are not directly comparable with WS21.

The new CSV adapter supports:

These four representations are not the 21-feature contract. Insufficient observations for a requested feature cause an explicit error rather than silently becoming a zero score.

Step 4 — Optional wrong-pairing control

\(I\) is the set of histories eligible for this control; \(|I|\) counts them. \(\pi\) is a reassignment of indices within each allowed block, and \(\pi(i)\ne i\) means a history never receives its own generated continuation in the control. \(\Delta_f\) is the shuffled score minus the matched score for feature \(f\).

Reassign generated ensembles within an allowed block, such as a ticker-day, with \(\pi(i)\ne i\). On the same eligible set \(I\) on both sides,

A positive gap means correct pairing performs better on this score. Singleton blocks cannot be shuffled and their exclusion is reported. This is an offline pairing diagnostic, not an inference rerun, a permutation p-value, or a causal hidden-state intervention. Initial-book constraints and other covariates can also explain the gap.

One sample: what it does and does not establish

With one observed and one generated path, the sides have symmetric trajectory counts. But single-path discrepancy does not identify the full conditional distribution or establish calibrated uncertainty. Return and volatility reduce to absolute error. The primary output is paired_w1; pooled_w1 is identical at \(K=1\).

The implementation can bootstrap whole group_id blocks, not individual events, before recomputing equal-context means. The default ticker-day blocks need to be broadened if dependence crosses those boundaries. Fewer than two blocks yields no interval.

Optional ensembles

Multiple members require explicit opt-in. In the following equations, \(A_{i,f}\) is a scalar average distance, distinct from the pooled observation list \(A_f\) above. \(F_{i,f}\) is its ensemble-corrected score. \(k,\ell\) index two generated members, and \(k<\ell\) visits each unordered pair once; \(\ell\) here is not the lag parameter used in the next section. The implementation also reports

In the expectation expression below, \(X,X'\) are independent generated feature representations under the same history, \(y\) is the observed representation, and \(d\) is their Wasserstein distance (absolute difference for scalars). The prime marks a second independent draw, not a derivative. For independent, identically distributed ensemble members, the score estimates \(E[d(X,y)]-E[d(X,X')]/2\) without self-pair bias. It is fair CRPS for scalar outcomes and a Wasserstein energy score for empirical feature distributions. It is not a full path-distribution metric. Finite-sample scores can be negative. For \(K_i=1\) it is unavailable, not zero; the implementation reports coverage rather than silently discarding ineligible contexts.

3. feature_conditional

Generated continuations per history: \(K_i\in\{1,2,\ldots\}\), determined by the input files, not by the conditional scoring mode. All members contribute target-condition observations to the pooled groups. \(K_i=1\) is valid. The three configured feature pairs are three scores, not three generated continuations.

Question: Are relationships between features realistic, such as spread at a given volatility?

Example: spread given volatility

Here, spread is the target being compared, and volatility is the feature used to form groups: \(\mathrm{spread}\mid\mathrm{volatility}\).

The two groups illustrate the idea; the implemented default uses the quantile boundaries described below, rather than a fixed low/high split. Both sides use the same boundaries, but each continuation is assigned using its own volatility. The within-group distances are then weighted by the real-plus-generated observation counts, as in Step 4.

To see what grouping adds, suppose low- and high-volatility groups contain equal numbers of spread observations:

Both pooled spread distributions contain equal proportions of 1 and 3 ticks, so their marginal Wasserstein distance is zero. Within each volatility group, however, the generated spread is wrong. Feature-conditioned scoring detects this reversed relationship even though the marginal spread distribution matches.

Step 1 — Attach a conditioning value to every target observation

\(\mathcal A_{f,g}\) and \(\mathcal B_{f,g}\) are lists of pairs, not lists of distances. Each pair contains a target value and its conditioning value. \(n_R,n_G\) count real and generated feature observations, respectively; \(r\) indexes an observation on the relevant side. Write these lists as

\(a_r,b_r\) are target values; \(u_r,v_r\) are aligned condition values. A continuation's scalar volatility can be repeated as the condition for its event-level spread observations.

Step 2 — Derive shared boundaries

\(\tau_b\) is the \(b\)-th numerical boundary, \(\boldsymbol\tau\) is the list of boundaries, and \(q\) is the number of boundaries. \(\operatorname{Quantile}_{p}(A)\) extracts the \(p\)-quantile of list \(A\); \(p=b/10\) below is a percentile level, not a bin weight. For the default continuous-condition rule,

Difference: boundaries normally come from the real and generated continuation conditions together. The percentile levels are fixed; their numerical thresholds are data-dependent. Ties mean groups need not contain exactly 10% of observations.

\(b_{\boldsymbol\tau}(x)\) is a function returning a bin ID, not a feature observation. It counts how many boundaries the numerical value \(x\) has reached or exceeded. With ordered boundaries \(\boldsymbol\tau\), the implementation's right-sided lookup is

Discrete conditions use unique observed values. Explicit thresholds override the default quantile rule. The quantile implementation also handles repeated boundaries by removing excess duplicates and applying a small offset to remaining repeats.

Step 3 — Each side groups by its own condition

The colon in \(\{a_r:\text{condition}\}\) means “keep those observations satisfying this condition.” \(A_{f,b},B_{f,b}\) contain target-feature values assigned to bin \(b\). The unadorned \(b\) is a bin ID; \(b_r\) is a generated target value. These are different quantities despite sharing a letter.

A generated observation is not assigned using its real counterpart's volatility. It uses its own condition value.

Step 4 — Compare within groups, then weight by counts

\(d_{f,g,b}\) is the distance for target \(f\), condition \(g\), and bin \(b\). \(w_b\) is that bin's fraction of all real-plus-generated observations. \(c\) in the denominator is a second bin index, used to sum over all bins without confusing it with the selected bin \(b\). \(D_{f\mid g}^{\mathrm{conditional}}\) is the resulting scalar score for one target–condition pair; the vertical bar reads “target \(f\) given condition \(g\).”

Weights count real plus generated feature observations. They are not equal-bin weights or equal-context weights. For group counts 500, 300, 200 and normalized distances 0.1, 0.2, 0.4, the result is \(0.5(0.1)+0.3(0.2)+0.2(0.4)=0.19\).

The three configured pairs

The time configuration has 24 boundary points, spaced about 62.61 minutes apart, not exact hourly bins. Volatility uses midpoint log returns resampled at 0.1 seconds. The scorer does not automatically evaluate every ordered pair among 21 features. An additional mean over these three pairs is a separately declared aggregate, not WS21.

Missing groups matter

The existing aggregation uses np.nansum(weights * losses). Undefined groups can be omitted without renormalizing their original weights:

“Finite” means a defined numerical value, neither NaN nor infinity. The summation subscript restricts the sum to those valid bins. \(D_{\mathrm{code}}\) is the resulting implementation output, and coverage is the fraction of original bin weight with a valid distance, between 0 and 1.

Check coverage alongside the score. Missing observations must not be interpreted as zero discrepancy. This caveat also applies to other modes using the same grouped aggregation pattern.

4. context_binned

Generated continuations per history: \(K_i\in\{1,2,\ldots\}\), determined by the input files; the mode does not prescribe \(K\). All generated members are assigned to bins using their own continuation features. Historical context supplies the boundaries. The number of bins is unrelated to \(K_i\).

Question: Do continuation feature distributions match within regimes whose boundaries are derived from historical data?

Step 1 — Change where the boundaries come from

\(H_g\) is a list of historical values of the conditioning feature \(g\). \(\operatorname{Boundaries}\) means the configured rule for constructing a threshold list: in this default it takes unique historical spread values. It is not a distance or an average. \(\boldsymbol\tau\) has the same meaning as in the preceding method.

Difference: historical context determines the boundaries. The default pair is best-ask volume given spread. Because discrete=True, unique historical spread values take precedence over n_bins=10; ten groups are not guaranteed.

Step 2 — Group continuation values using those boundaries

The conditions \(u_r,v_r\) are still the continuations' own spread values. The implementation does not assign an entire future to a regime determined by its historical context.

Step 3 — Reuse within-group distances and count weights

There is one default feature pair. Crucially, this mode does not compute one distance for each historical context. The more specific name avoids confusing historical bin calibration with historical pairing.

5. time_lagged

Generated continuations per history: \(K_i\in\{1,2,\ldots\}\), determined by the input files; \(K_i=1\) is supported. Each member is separately appended to the same historical context, then its lagged feature pairs enter the pooled comparison. The lag (for example, 500 aligned positions) is not the number of generated members. Too-short members may be skipped, so report the scored coverage as well as the input count.

Question: Are relationships between earlier features and later features realistic?

Step 1 — Prepend historical context

\(X_i^R\) is history \(i\) followed by its real continuation; \(X_{ik}^G\) is that history followed by generated member \(k\). Superscripts \(R,G\) label the real/generated branch. Extract target and condition arrays on the merged sequence. The implementation tail-aligns them to a common length \(n\); call the aligned arrays \(\widetilde f\) and \(\widetilde g\).

Step 2 — Offset target and condition by a lag

Here \(\ell\) is the positive lag measured in aligned array positions; \(t\) indexes an earlier position and \(t+\ell\) a later one. The tilde on \(\widetilde f,\widetilde g\) denotes the tail-aligned feature arrays. \(n_i\) and \(n_{ik}\) are the available aligned lengths for the real trajectory and generated member, respectively. \(\mathcal A_{f,g,\ell},\mathcal B_{f,g,\ell}\) collect the resulting (later target, earlier condition) pairs. This \(\ell\) is a lag, not the ensemble-member index in the preceding section.

Difference: the target occurs later than the condition. The actual slices are condition [:-lag] and target [lag:]. Slicing both arrays identically would only truncate them. For event-aligned features the lag is an event offset, not seconds; rolling or length-changing features require checking the alignment explicitly.

Step 3 — Reuse conditional binning and aggregation

\(A_{f,g,\ell,b}\) and \(B_{f,g,\ell,b}\) are the target-value lists selected into bin \(b\) by the lagged condition. \(w_b\) is calculated from these lists' real-plus-generated counts, using the same definition as before. \(D_{f\mid g,\ell}^{\mathrm{lagged}}\) is one scalar score for the target, condition and lag.

The current default pairs, each configured with lag 500, are best-ask volume given earlier spread, best-ask volume given earlier OFI, and best-bid volume given earlier OFI. They remain pooled conditional comparisons, not one score per history.

This mode is closer to a past-to-future dependency check than same-stage feature conditioning. It still does not assess every aspect of \(p(Y\mid C)\). Because scoring uses the merged sequence, targets can include historical positions when the history exceeds the lag. Too-short aligned sequences are skipped; coverage must be checked.

6. temporal_divergence

Generated continuations per history: \(K_i\in\{1,2,\ldots\}\) complete generated trajectories, determined by the input files. Each trajectory is then split into successive windows. These windows do not increase \(K_i\). A stage pools the available windows from those trajectories; later-stage coverage can differ when lengths differ. \(K_i=1\) still produces a multi-stage curve.

Question: Does feature-distribution discrepancy change as generation proceeds?

Step 1 — Slice the generated continuation

\(H\) is the number of generated events per window; \(h=1,2,\ldots\) is the window index. \(G_{i,h}^{(k)}\) is window \(h\) of generated member \(k\) for history \(i\). The slice uses zero-based positions, includes its left endpoint and excludes its right endpoint. For window length \(H\),

These are successive windows, not cumulative prefixes. The adapter's default is 100 messages per window. Available windows depend on actual sequence length and the slicing implementation.

Step 2 — Keep the full real reference

\(A_f\) is again the pooled real feature-value list. \(B_{f,h}\) pools generated feature values only from stage \(h\). \(D_{f,h}^{\mathrm{temporal}}\) is one normalized feature distance at that stage. Feature \(f\) and stage \(h\) are different indices.

Difference: only the generated side is divided by generation stage; each stage is compared with the full real feature distribution. It is not a matched comparison of real and generated windows at the same historical position.

Features are extracted separately on each generated slice. A short slice may not support a rolling feature. The joint normalization scale can also change with the stage.

Step 3 — Report a curve

The braces above collect successive values into a curve; they do not sum the stages. \(S_h\) below is a feature average at one stage. If all 21 features are valid at stage \(h\), an explicitly defined stage-wise mean is

Do not silently change the feature set when comparing stages, or collapse the whole curve to one number without specifying the temporal weights. This formula describes a stage-wise aggregate, not a claim that every historical artifact already contains it.

7. The important order of operations

Pool weights empirical distributions by their actual feature-observation counts. It becomes an equal mixture only when those counts are equal.

No label alone proves that a model uses its hidden state effectively. A stronger causal claim needs an inference intervention that controls other inputs. These protocols measure complementary properties; their numerical scores are not interchangeable.

8. Implementation and provenance

The existing mode descriptions were checked against the sigma-0 source snapshot at 63b86a429d291acbb524043dd87193f8ecb55902. The new paired implementation and its single-sample default were checked at local commit 4c1bdef8 on feat/context-paired-20260927.

Historical artifacts can use different scorer revisions. Identify an artifact's own provenance before interpreting it through this guide. This page adds no measured scores and makes no change to the leaderboard's recorded mode labels or numbers.

See also the existing evaluation-mode reference, including its implementation caveats and earlier source snapshot.