LisanResearch companion
Evaluation protocols.
How the systems were evaluated, how to read their scores and how to reproduce the comparisons.
M3
Multilingual meeting transcription
Multilingual meeting intelligence. Compare Arabic and English transcription results, explore the meeting workflow and examine the scoring protocol.
| Evaluation element | Recorded scope |
|---|---|
| Evaluation material | Arabic and English meeting speech is compared with human-verified reference transcripts. The language-specific results are presented separately; product language coverage extends beyond these evaluated languages. |
| Matched comparison | M3 and the external ASR configurations receive the same audio and clip boundaries within each language evaluation, with every system explicitly configured for Arabic or English as appropriate. Outputs are scored against the same human-verified references; references are not provided during transcription. The Arabic and English evaluation recordings and references were not used to train M3. |
| Meeting architecture | M3 connects eleven functional components, including three preprocessing models, across capture, speech preparation, contextual recognition and traceable reporting. Preprocessing labels describe proposed functional roles rather than official model names. |
| Meeting context | In the evaluated Arabic and English meetings, M3 kept each speaker linked to the corresponding transcript using available participant and platform identity context. Speaker linkage was evaluated for M3; external speaker identification was not evaluated. This check is separate from CER and does not establish independent speaker-recognition or diarization accuracy. |
| Automatic language selection | Before transcription, M3 selects the meeting's primary language from its audio; users do not need to select it. The same M3 system handles Arabic and English, including switches within a meeting. A separate check selected the correct primary language for every evaluated Arabic and English meeting. This check was outside the explicitly language-configured CER comparison and does not establish a universal language-detection rate. |
| Product coverage | MeetriX supports 40 languages and 32 Arabic dialects. The CER results in this paper apply to separate Arabic and English evaluations under their respective scoring rules. |
| Arabic scoring | Arabic CER is 1.62% after the defined lexical-equivalence rules. These cover dialect and standard Arabic, equivalent technical terms, Latin and Arabic spellings, numeric values and Arabic letter forms. CER is an edit-distance metric. |
| English scoring | English CER is 1.83%. Scoring ignores punctuation, letter case and ordinary spacing, while a merged word boundary counts as an error. The evaluated M3 English path uses soxr resampling, one second of silence padding and the English language setting. |
| Batch processing | On one L4 GPU, batch transcription processes an hour of recorded audio in 45 seconds. This is recorded-audio processing throughput, not end-to-end live latency. |
| Concurrent capacity | A separate operating measurement supports 82 concurrent meetings on one L4 GPU. Batch speed and capacity describe different workloads; they are not a combined benchmark, provider-by-provider latency comparison or guarantee for every deployment. |
Interpretation and reproduction. Read Arabic and English CER within their respective normalization rules. Neither result is a universal accuracy guarantee across all supported languages. Reproduction requires the same audio, references, pinned model configurations and scoring rules. Selected audio examples illustrate output behaviour; they do not form an additional benchmark.
L3
Language Tasks
Arabic language intelligence for precise, contextual edits. Explore error detection, correction fidelity and efficient deployment.
| Evaluation element | Recorded scope |
|---|---|
| Language tasks | The evaluation measures Arabic error detection, localized correction and whole-output fidelity. Expert-reviewed source/reference pairs provide the same task and reference material to every configuration. |
| Controlled intervention | Models are instructed to correct spelling and grammar when needed, preserve valid wording and avoid paraphrasing. Unchanged references form the no-op evaluation subset. Only the first L3 suggestion is scored. |
| Primary metric | Diacritic-insensitive F0.5 weights precision more heavily than recall. |
| Correction fidelity | The normalized replacement score evaluates correction quality strictly on the error spans successfully detected and targeted by the model. In addition, whole-output exact match is measured under two conditions—both with and without the symmetric removal of diacritics and tatweel to account for surface-level variations. |
| Operational measures | End-to-end service throughput uses a shared corrected-output tokenizer. Accelerator energy is a separate controlled operating measurement; a full-stack estimate applies an explicit overhead factor and is identified as modeled. |
Interpretation and reproduction. The comparison uses one expert reference for each input; a different valid correction can disagree with that reference. Scores are point estimates without confidence intervals. Reproduction requires the same tokenizer, alignment rules, normalization, provider configuration and timing procedure.
T3
Document translation
Multilingual document intelligence. Explore languages, layout reconstruction, professional review and measured translation quality.
| Evaluation element | Recorded scope |
|---|---|
| Product and language scope | TranslateX supports multilingual text and document workflows, with named languages detailed in the paper. Glossaries, translation memory, CAT review and integration options describe the product and enterprise workflow. The measured document results remain Arabic-to-English and English-to-Arabic. |
| Formats and deployment scope | Product inputs include PDF, DOCX, PPTX, XLSX, TXT, CSV and JSON, depending on configuration. OCR/layout reconstruction and the enterprise reference architecture are capability and design descriptions, not additional scored tests or certification of deployment controls. |
| Document workflow | Saved evaluations cover DOCX, PPTX and PDF. The evaluation is primarily DOCX; XLSX is demonstrated in the separate visual comparison collection. |
| Language direction | Arabic-to-English and English-to-Arabic results are grouped by each document’s source and target language. The groups contain different documents. |
| Quality and structure | Equal file weighting yields 93.34/100 for translation quality and 97.93/100 for structural fidelity. Null structural criteria are excluded within each file. These are automated evaluation scores. |
| Operational timing | Cached outputs store zero translation time. Only uncached records contain positive timings, so the stored timings do not establish an end-to-end speed advantage. |
| Visual inspection | The paper presents one representative case for each of DOCX, PPTX, XLSX and PDF input. The comparison viewer retains the complete supplied files and available report pages. These examples illustrate layout and translation behaviour separately from the aggregate scores. |
Interpretation and reproduction. Automated ratings measure the evaluated outputs and should be read alongside the document examples. The format mix is primarily DOCX, and the language-direction groups are not a matched experiment. Reproduction requires the source/output mapping, scoring implementation and judge configuration.
The papers, figures and this companion describe the same saved evaluations and scoring definitions.