event_metrics

Callback that scores a fixed slice of the validation set on events — beats and bar positions on full-length tracks — during training.

Every number the training log has ever shown is a frame metric: how well the model labels 23 ms frames of a 16-second clip taken from the middle of a track. What the model is judged on is events: beats recovered from a full-length track, and bars numbered across it. The two do not track each other closely, and the gap runs in both directions — plans/06_metric_calibration_and_eval_consolidation.md measured val/position_acc at 0.661 against val_event/position_acc of 0.581 — the same quantity, read per frame on the clip and per event on the full track (the clip is cut from the middle, deliberately avoiding intros, so the frame number is measured on the easiest 16 seconds of every track) — while frame acc_beat at a 70 ms window reads 0.959 against an f_beat of 0.845 (balanced accuracy averages in a true-negative rate that is pinned near 1.0 on any sane model).

So the event numbers had to be reconstructed after the fact, from a checkpoint, with tools/eval_beat.py. This logs them as the run goes, from score() — the same scoring path the CLI uses, so a number seen in W&B during training and the same number recomputed afterwards cannot disagree.

Two things make it affordable. It runs every every_n_epochs epochs rather than every epoch, and it runs on a fixed subsample of the validation split rather than all of it. The subsample is stratified across corpora (stratified_sample()): a split file is written corpus by corpus, so taking the first N tracks of a merged split yields N tracks of whichever corpus was written first.

Functions

stratified_sample(refs, n_tracks[, seed])

Pick n_tracks refs spread as evenly as possible across the corpora present.

Classes

EventMetricsLogger(refs, postprocess, *[, ...])

Logs val_event/* metrics for a fixed validation subsample.

class EventMetricsLogger(refs, postprocess, *, n_tracks=50, every_n_epochs=5, sample_rate=22050, hop_length=512, group_size=4, binary_only=False, tolerance=0.07, seed=0, name='val')

Bases: Callback

Logs val_event/* metrics for a fixed validation subsample.

Parameters:

• refs (list) – Validation TrackRef entries to sample from — the same refs the validation dataloader is built over (see resolve_beat_split_refs()).

• postprocess (dict) – Tuned decode knobs per task, handed straight to BeatEvaluator. It has to be the same block tools/eval_beat.py scores with, or a number logged during training and the same number recomputed afterwards mean different things — two sources disagreeing about the decoder is exactly how they drifted apart before.

• n_tracks (int | None) – Size of the fixed subsample; None scores every ref. Cost is roughly one full-track model pass per track, so this trades directly against how often it can run.

• every_n_epochs (int) – Score every N epochs. Values below 1 are treated as 1. The final epoch is always scored regardless, so the run ends with a fresh event number for the model it finished with.

• sample_rate (int) – Audio sample rate; must match training.

• hop_length (int) – Frame hop, in samples; must match training.

• group_size (int) – Beats per group the position head predicts over.

• binary_only (bool) – Passed to BeatDataset; must match how the split was built, or tracks are dropped here that the dataloader keeps.

• tolerance (float) – Event matching window, in seconds.

• seed (int) – Seed for stratified_sample().

• name (str) – Label for the dataset in report lines.

property dataset: BeatDataset

The subsample as a dataset, built on first use.

Built lazily so that constructing the callback — which happens before the trainer exists — does no annotation I/O, and so a run that never reaches a scoring epoch never pays for it.

on_validation_epoch_end(trainer, pl_module)

score(pl_module)

Decode and score the subsample with the model as it stands now.

A fresh BeatEvaluator per call: its probability cache is per-instance and exists to make several decoder settings share one model pass, which is the opposite of what is needed here — the weights are different every epoch, so a reused evaluator would report epoch 0’s numbers forever.

Return type:

list[dict]

should_run(trainer)

Whether this epoch is a scoring epoch.

Skips Lightning’s pre-training sanity check (an untrained model, and the resulting point would sit on the curve as if it were epoch 0’s real score) and every rank but zero, since the pass is duplicated work on every rank and log() is rank-zero-only anyway.

Return type:

bool

stratified_sample(refs, n_tracks, seed=0)

Pick n_tracks refs spread as evenly as possible across the corpora present.

Round-robin over corpora — one track from each in turn — so a corpus contributing 7 tracks out of 249 is represented on the same footing as one contributing 104, and a small corpus running out simply drops out of later rounds instead of truncating the sample.

Within a corpus the order is a fixed-seed shuffle rather than the split file’s own order. Split files are grouped by corpus and often ordered within it (ballroom is written genre by genre), so taking the first few of each corpus can quietly mean taking one dance style.

Deterministic in seed: the same refs and seed give the same tracks on every epoch, every restart and every machine, which is what makes the logged curve a curve rather than a walk over different test sets.

Parameters:

• refs (list) – TrackRef entries, typically a split’s validation half.

• n_tracks (int | None) – How many to keep. None (or a count at or above len(refs)) keeps all of them.

• seed (int) – Seed for the per-corpus shuffle.

Returns:

The selected refs, interleaved by corpus.

Return type:

list