Corpus average_cpWER still uses the concatenation algorithm that PR #15573 replaced #16093

Describe the bug

OfflineDiarWithASR.evaluate() reports a corpus-level average_cpWER computed with the
concatenation algorithm that PR #15573 replaced, so it disagrees with — and systematically
under-reports relative to — every per-session cpWER printed beside it.

PR #15573 ("[Fix] Make cpWER calculation identical to meeteval") replaced cpWER with MeetEval
semantics: each (ref_speaker, hyp_speaker) pair is scored independently, so an edit cannot cross a
speaker boundary. That PR changed nemo/collections/asr/metrics/der.py (where cpWER lived at the
time; it is now nemo/collections/asr/metrics/cpwer.py) and its tests, but it did not touch
nemo/collections/asr/parts/utils/diarization_utils.py.

nemo/collections/asr/parts/utils/diarization_utils.py:1367-1371 still does:

cpWER_values, hyps_spk, refs_spk = concat_perm_word_error_rate(spk_hypotheses, spk_references)

# Take an average of cpWER and regular WER value on all sessions
wer_results['total'] = {}
wer_results['total']['average_cpWER'] = word_error_rate(hypotheses=hyps_spk, references=refs_spk)

hyps_spk / refs_spk are the space-joined per-speaker transcripts
(cpwer.py:153-154: min_perm_hyp_trans = " ".join(hyp_texts) / ref_trans = " ".join(spk_reference)),
so running plain word_error_rate on them is exactly the old concatenation approach — edits are
free to cross speaker boundaries again.

The value is user-facing: print_errors (diarization_utils.py:1519-1521) prints it under the
label cpWER, and write_session_level_result_in_csv writes it into the results CSV. It is
reachable from two shipped scripts and a tutorial:

• examples/speaker_tasks/diarization/clustering_diarizer/offline_diar_with_asr_infer.py:75
• scripts/speaker_tasks/eval_diar_with_asr.py:169 / :176
• tutorials/speaker_tasks/ASR_with_SpeakerDiarization.ipynb

Steps/Code to reproduce bug

This drives the public OfflineDiarWithASR.evaluate() entry point with three sessions, each being
the scenario already pinned by
tests/collections/speaker_tasks/utils/test_cpwer.py::TestCpWERExpectedValues::test_cross_boundary
(and quoted in PR #15573's own description as the bug it fixed):

import os, tempfile
from nemo.collections.asr.parts.utils.diarization_utils import OfflineDiarWithASR

# CTM columns: <uniq_id> <speaker> <start> <duration> <word>
def write_ctm(path, uniq_id, spk_texts):
    with open(path, "w") as f:
        t = 0.0
        for spk_idx, text in enumerate(spk_texts):
            for w in text.split():
                f.write(f"{uniq_id} speaker_{spk_idx} {t:.2f} 0.10 {w}\n")
                t += 0.10

tmp = tempfile.mkdtemp()
hypdir = os.path.join(tmp, "hyp"); os.makedirs(hypdir)
audio_file_list, ref_ctms, hyp_ctms = [], [], []
for i in range(3):
    uid = f"sess{i}"
    audio = os.path.join(tmp, f"{uid}.wav"); open(audio, "w").close()
    ref = os.path.join(tmp, f"{uid}.ctm"); hyp = os.path.join(hypdir, f"{uid}.ctm")
    write_ctm(ref, uid, ["the cat", "sat on"])       # reference: 2 speakers
    write_ctm(hyp, uid, ["the cat sat", "on"])       # hypothesis: boundary shifted by one word
    audio_file_list.append(audio); ref_ctms.append(ref); hyp_ctms.append(hyp)

wer_results = OfflineDiarWithASR.evaluate(
    audio_file_list=audio_file_list,
    hyp_trans_info_dict=None,
    hyp_ctm_file_list=hyp_ctms,
    ref_ctm_file_list=ref_ctms,
)
for k in sorted(wer_results):
    print(k, "->", wer_results[k])

Actual output:

sess0 -> {'cpWER': 0.5, 'WER': 0.0}
sess1 -> {'cpWER': 0.5, 'WER': 0.0}
sess2 -> {'cpWER': 0.5, 'WER': 0.0}
total -> {'average_cpWER': 0.0, 'average_WER': 0.0}

Every session reports cpWER = 0.5, while the corpus value reports 0.0. print_errors() then
shows the user:

    DER                : 0.0000
    FA                 : 0.0000
    MISS               : 0.0000
    CER                : 0.0000
    Spk. counting acc. : 1.0000
    cpWER              : 0.0000
    WER                : 0.0000

0.0 is the exact value PR #15573's description cites as the bug it fixed:

For example, hyp=["the cat sat", "on"] vs ref=["the cat", "sat on"] would incorrectly return
cpWER=0.0 instead of the correct cpWER=0.5.

The same thing can be seen without any files, at the level of the two functions involved:

from nemo.collections.asr.metrics.cpwer import concat_perm_word_error_rate
from nemo.collections.asr.metrics.wer import word_error_rate

spk_hypotheses = [["the cat sat", "on"]]
spk_references = [["the cat", "sat on"]]

cpWER_values, hyps_spk, refs_spk = concat_perm_word_error_rate(spk_hypotheses, spk_references)
print("per-session cpWER:", cpWER_values)
# verbatim diarization_utils.py:1371
print("average_cpWER    :", word_error_rate(hypotheses=hyps_spk, references=refs_spk))
print("strings actually scored -> hyp:", hyps_spk, "ref:", refs_spk)

per-session cpWER: [0.5]
average_cpWER    : 0.0
strings actually scored -> hyp: ['the cat sat on'] ref: ['the cat sat on']

The concatenation collapses the speaker boundary, so the two strings become identical.

Expected behavior

The corpus-level cpWER should be the MeetEval-consistent aggregate of the session values — total
errors over total reference words — so that a corpus of N sessions each at cpWER 0.5 reports 0.5,
not 0.0.

This also matches the metric printed next to it: average_WER is computed by word_error_rate over
the list of sessions, which accumulates scores and words across the whole list and divides once
(nemo/collections/asr/metrics/wer.py:35-71) — i.e. it is already length-weighted across sessions.
average_cpWER should be aggregated the same way.

Environment overview (please complete the following information)

• Environment location: Bare-metal
• Method of NeMo install: from source (git clone + pip install -e .), Python 3.10, PyTorch 2.12.0

Environment details

• OS version: macOS 26.5.1 (arm64)
• PyTorch version: 2.12.0
• Python version: 3.10.18

Additional context

tests/collections/speaker_tasks/utils/test_cpwer.py covers calculate_session_cpWER and
concat_perm_word_error_rate thoroughly, but nothing covers the corpus aggregation in
diarization_utils.py, which is why the gap survived.

The streaming evaluator in the same file (diarization_utils.py:652) is not affected — it returns
per-chunk cpWER lists and never aggregates them, so evaluate() is the only corpus-level
aggregation site.