Research · Evaluation report · September 2026

M3: multilingual meeting transcription,
measured on real meetings

M3, the multilingual meeting transcription system inside MeetriX, recorded the lowest character error rate in two separate evaluations on real business meetings: 1.62% on multi-dialect Arabic meetings and 1.83% on English meetings, lower than each of the six external ASR model configurations it was benchmarked against.

Read the announcementSee it in MeetriXCite this report

Research

  • Published
  • VersionArabic and English, September 2026
  • AuthorshipLisan Research
Read the announcementSee it in MeetriXCite

Executive summary

M3 recorded the lowest character error rate in both language-specific evaluations: 1.62% on the Arabic set and 1.83% on the English set. Each evaluation compares M3 with six external ASR model configurations on real business meetings: the same six models, each set to the language under test.

  • Accuracy. M3 recorded a lower CER than every external configuration on both evaluation sets.
  • Real-meeting evidence. The results are paired with human-verified references and segment-level examples covering short turns, names, numbers and business terminology.
  • Deployment. The evaluated workflow processed one hour of recorded audio in 45 seconds on an L4 GPU and can run inside customer-controlled infrastructure.

Arabic and English were evaluated on separate source material, so each headline figure is read within its own language table, not as a ranking of the two languages.

1.62%Arabic character error rateReal Arabic meetings, human-verified references
1.83%English character error rateReal English meetings, human-verified references
28% to 83%Lower Arabic CERM3 versus each external configuration
42% to 77%Lower English CERM3 versus each external configuration
45 sPer hour of audio80x real time on an L4 GPU
6External ASR configurations compared with M3The same six models, set to each language

Arabic benchmark

Arabic results: character error rate (CER)

Why Arabic meetings are different

A business meeting in the region is rarely held in one register. The chair may open in Modern Standard Arabic, a colleague answers in a Gulf or Levantine dialect, and the technical terms, product names and numbers arrive in English inside the Arabic sentence. The evaluated Arabic meetings contain all of this: multiple Arabic dialects, English technical terminology inside Arabic speech, several participants in one recording, overlapping speakers, sub-second utterances, silence, and sentences split across recorder boundaries. Qualitative.

Each of those conditions defeats a transcription system in a different way. A sub-second acknowledgement can be classified as silence and dropped. An English abbreviation can be replaced by a similar-sounding Arabic word. A pause can become an invented filler, and a hard segment boundary can cost the first words of a turn. A meeting record is only useful if the numbers, the names and the decisions survive these conditions, which is why the evaluation was run on real meetings rather than read speech.

The M3 system records the lowest character error rate against six external ASR model configurations on the same human-verified meetings

Lower is better. Character error rate (CER) after defined lexical normalisation.

Same source audio, same human-verified reference, same scoring rules for every entry. This comparison evaluates transcription only. External configurations received the same participant-separated segments, so external speaker diarization was not tested. Evaluation snapshot September 2026. See "How we measured the Arabic benchmark" for the scoring rules.

Measured. The chart shows the character error rate of the M3 system and the six external ASR model configurations on the same human-verified Arabic evaluation set, after the lexical normalisation described under "How we measured the Arabic benchmark". M3 recorded 1.62%, followed by Google Chirp 3 at 2.26%. Cohere Transcribe Arabic (4.93%), ElevenLabs Scribe v1 (5.01%) and Deepgram Nova-3 (6.09%) form a middle group, and Google Latest Long at 9.01% and Sonix at 9.8% close the field. CER is the share of characters that had to be added, removed or changed to match what a human verified was said. In practical terms, 1.62% CER corresponds to roughly 16 character edits per 1,000 reference characters after the stated normalisation rules.

The gap to Google Chirp 3 is 0.64 percentage points, 28% lower CER. The gap to Sonix is 8.18 points, 83% lower.

The leaderboard

The benchmark includes seven evaluated entries: the M3 system and six external ASR model configurations. The table below carries their character accuracy (100 minus CER) and M3's relative reduction against each configuration. Read the reduction column as the gap on this evaluation set, not as a ranking of the external configurations by quality.

Character error rate and character accuracy on Arabic meetings for the M3 system and the six external ASR model configurations, with M3's relative reduction versus each configuration. Lower CER is better. Measured on the same human-verified meetings; scoring rules under How we measured the Arabic benchmark.
RankEvaluated entryCER (%), lower is betterCharacter accuracy (%)M3 vs this entry
1M31.6298.38baseline
2Google Chirp 32.2697.7428% lower (0.64 points)
3Cohere Transcribe Arabic4.9395.0767% lower (3.31 points)
4ElevenLabs Scribe v15.0194.9968% lower (3.39 points)
5Deepgram Nova-36.0993.9173% lower (4.47 points)
6Google Latest Long9.0190.9982% lower (7.39 points)
7Sonix9.890.283% lower (8.18 points)

The practical result. M3's advantage was not limited to a lower aggregate error rate. Across the reviewed examples, it more consistently retained numbers, short utterances, product names, and English technical terms in a form that remains readable and searchable in a business record.

Arabic examples: what the transcripts look like

Aggregate rates hide what actually goes wrong in a transcript. The cards below show the human-verified reference, the M3 system's output and every external configuration's output for individual Arabic segments, copied verbatim from the evaluation. Speaker labels are fictional placeholders. Each card carries the source clip: an excerpt of an internal Lisan meeting, the same audio every system transcribed. We chose segments that a reader without Arabic can still inspect: an empty cell, a Latin term, a digit, or a switch into another script is visible in any language. Where M3 itself makes a mistake, the caption says so. Under the scoring rules a phonetic spelling of an English term is not an error, so several of the differences below affect readability and search rather than the CER.

Arabic example 1English terminology

SDK, rendering, editor, and Chrome extension

Source clip
  • Preserved term
  • Unsupported text
  • Filler-like insertion
  • No transcript
M3

صباح الخير بالنسبة لي امبارح كان معظم الشغل مع تيم العربي الجديد بخصوص الـ SDK (preserved term) اتسكر تقريبا مشكلتين و إلهن علاقة بالـ rendering (preserved term) الـ highlights (preserved term) بالإديتور اللي معمل له الـ Configuration (preserved term) من عندن بصفى مشكلة وحدة طلبنا منهم اجتماع مشان نكمل معهم اليوم ان شاء الله، كمان امبارح صرت اشتغل على موضوع الـ Chrome extension (preserved term) بين كوركتو و Lisan (preserved term) ما كثير يعني خلصت هذا الموضوع، اليوم ان شاء الله بدي كمل مثل ما قلت بخصوص العربي الجديد وكمان الـ Chrome extension مع كوركتو، يعطيكم العافية.

Reference

صباح الخير، بالنسبة لي امبارح كان معظم الشغل مع تيم العربي الجديد بخصوص الـ SDK اتسكر تقريبا مشكلتين ولهم علاقة بالـ rendering للـ highlights بالـ editor اللي معمل له configuration من عندن بصفى مشكلة وحدة طلبنا منن اجتماع مشان نكمل معن اليوم ان شاء الله كمان امبارح صرت اشتغل على موضوع الـ Chrome extension بين كوركتو و Lisan ما كتير خلصت هذا الموضوع اليوم إن شاء الله بدي كمل متل ما قلت بخصوص العربي الجديد وكمان الـ Chrome extension مع كريكتو يعطيكم العافية

Why this example matters. A daily stand-up full of developer vocabulary. M3 keeps SDK, rendering, highlights, configuration, Chrome extension and Lisan in Latin script, which is what a search across your archive will find. ElevenLabs Scribe v1 keeps them too, with heavy filler. Google Chirp 3 is complete but phonetic, which the scoring rules accept.

Arabic example 2Filler and punctuation

Sprint Planning and VAT

Source clip
  • Preserved term
  • Unsupported text
  • Filler-like insertion
  • No transcript
M3

صباح الخير يعطيكم العافية. امبارح كان في عندنا متابعة باجتماعات الـ Sprint Planning (preserved term) كان في كمان متابعة مع الشي المطلوب من هيئة الزكاة والضرائب بالسعودية موضوع الـ VAT (preserved term) ضريبة القيمة المضافة. كنت عم بتابع بهدول المواضيع يعطيكم العافية.

Reference

صباح الخير يعطيكم العافية امبارح كان في عنا متابعة باجتماعات السبرنت بلاننج كان في كمان متابعة مع الشي المطلوب من هيئة الزكاة والضرائب بالسعودية موضوع الفات ضريبة القيمة المضافة كان كنت عم بتابع بهدول المواضيع يعطيكم العافية

Why this example matters. A project update containing Sprint Planning and VAT. M3 preserves both terms in Latin script and keeps the Arabic expansion of VAT, making the record easier to search and read.

Arabic example 3Short speech

A sub-second spoken greeting

Source clip
  • Preserved term
  • Unsupported text
  • Filler-like insertion
  • No transcript
M3

كيفكم؟

Reference

كيفكم؟

Why this example matters. A greeting shorter than a second. M3, Cohere Transcribe Arabic, Sonix and Google Chirp 3 match the reference exactly. Deepgram Nova-3 and Google Latest Long returned nothing, which scores as deletions, and ElevenLabs Scribe v1 produced an English word that was not spoken.

Arabic example 4Numbers and terms

Files, quality figures, and model work

Source clip
  • Preserved term
  • Unsupported text
  • Filler-like insertion
  • No transcript
M3

صباح الخير امبارح خلصت الملفات كلهم طلعوا جوا تبع TranslateX (preserved term) حوالي 200 (preserved term) file (preserved term) الدقه طلعت يعني بالـ style (preserved term) حوالي 98 (preserved term) يعني بالـ translation (preserved term) كجوده بال 94 (preserved term) هلا هذا بعد طبعا عده تعديلات هلا اليوم بدي ارفع الـ version (preserved term) وكان في كمان شغل على MOE (preserved term) حاولنا نعمل fine tuning (preserved term) هلأ اليوم كمان حأتابع

Reference

صباح الخير امبارح خلصت الملفات كلهم تبع نجوى تبع TranslateX هلأ هنن حوالي 200 file الدقه طلعت يعني بالـ style حوالي 98 يعني بالـ translation كجوده بال 94 هلأ هذا بعد طبعا عدة تعديلات هلأ اليوم بدي ارفع الـ version وكان في كمان شغل على MOE حاولنا نعمل fine-tuning، هلأ اليوم كمان حتابع بنفس الموضوع

Why this example matters. Three spoken figures: 200 files, a style score of 98, and a translation quality score of 94. M3 preserves all three, along with TranslateX, version, MOE, and fine tuning. Cohere Transcribe Arabic and Google Chirp 3 also preserve the three figures, while several other systems alter or omit one or more values or technical terms. M3 omits the name Najwa in this segment.

Arabic example 5English terminology

SDK, dark theme, web app, and monitoring

Source clip
  • Preserved term
  • Unsupported text
  • Filler-like insertion
  • No transcript
M3

صباح الخير، بالنسبة ليوم الخميس الشغل كان مع فضاءات، كان في عندهم مشاكل العلاقة بالـ SDK (preserved term) والـ suggestions (preserved term) اللي عم تطلع، انحل اغلب الاخطاء، صفيان وحدة بدنا نعمل كول عليها اليوم إن شاء الله، كمان بدي اشتغل بالـ بنسخة الـ dark theme (preserved term) بالـ web app (preserved term)، وكمان بدي أعمل monitoring (preserved term) للنسخة الجديدة تبع الـ AI agent (preserved term)، يعطيكم العافية.

Reference

صباح الخير، بالنسبة إلي يوم الخميس الشغل كان مع فضاءات، كان في عندهم مشاكل العلاقة بالـ SDK والـ suggestions اللي عم تطلع، انحل اغلب الاخطاء، صفيان وحدة بدنا نعمل كول عليها اليوم إن شاء الله، كمان بدي اشتغل بالـ بنسخة الـ dark theme بالـ web app، وكمان بدي أعمل monitoring للنسخة الجديدة تبع الـ AI agents، يعطيكم العافية.

Why this example matters. Consecutive product terms. M3 keeps SDK, suggestions, dark theme, web app, monitoring, and AI agent in Latin script. The reference uses the plural “AI agents,” while M3 returns the singular “AI agent.”

Arabic example 6Dialects and terminology

Saudi dialect, Kubernetes, scaling, and paper

Source clip
  • Preserved term
  • Unsupported text
  • Filler-like insertion
  • No transcript
M3

صباح الخير كنت أمس أشتغل على موضوع في الـ Kubernetes (preserved term) من ناحية scaling (preserved term) كان في مشكلة بسيطة قاعد أدور لها حل وأيضا قاعد أشوف الموضوع البيبر ويعطيكم العافية

Reference

صباح الخير كنت أمس أشتغل على موضوع في الـ Kubernetes من ناحية scaling كان في مشكلة بسيطة قاعد أدور لها حل وأيضا قاعد أشوف الموضوع البيبر ويعطيكم العافية

Why this example matters. A Saudi-dialect status update containing Kubernetes, scaling, and paper. M3 preserves Kubernetes and scaling in Latin script and captures the full update. ElevenLabs Scribe v1 preserves all three English terms in Latin script, while several other systems render them phonetically or omit them.

Output behaviour across systems on the Arabic evaluation

Qualitative. Alongside the CER, the reviewers recorded how the M3 system and each external ASR model configuration behaved on the evaluation set: whether English terms survived inside Arabic speech, whether content was lost at the start of segments, whether fillers, event tags or repeated punctuation crept in, and whether outputs looped. No separate aggregate rates were calculated for these observations. The reviewers' phrase behind every cell is published under the matrix, so the mapping to strong, mixed, weak and not rated can be checked against the wording.

Google Chirp 3 deserves the credit the review gives it: strong handling of English technical terminology within Arabic speech, low content loss, and substantially fewer empty outputs than Google Latest Long. On several of the example segments above it is as complete as M3, with the difference limited to phonetic rather than Latin spellings. ElevenLabs Scribe v1 preserved English terminology particularly well, but produced frequent spurious filler-like vocalisations and one switch into an unexpected language or script. Cohere Transcribe Arabic was highly accurate on Classical Arabic, while the review identified content omissions, inconsistent terminology, repeated punctuation and repetition loops. Deepgram Nova-3 showed participant-name errors and segment-start omissions. Google Latest Long correctly returned no transcript for many non-speech segments but missed a small number of speech-bearing ones and frequently omitted English terms. Sonix showed errors in names and English terms, segment-start omissions and excessive sentence-final punctuation. The M3 system and all six external ASR model configurations produced Arabic transcripts across the dialects represented in this evaluation set.

M3 recorded the lowest CER and, in the reviewers' notes, preserved business-critical content, numbers and terminology more reliably and did not show the same concentration of filler, tag or repeated-punctuation artifacts in the review. Speaker identification was not evaluated for the external configurations: each received the same participant-separated audio prepared by the M3 capture workflow and was scored only on transcription.

Nine criteria, seven evaluated entries (the M3 system and six external ASR model configurations), one level per cell. The M3 column is marked with the brand rule only; status colours are never used for emphasis. The final row repeats the measured CER for reference.

Observed output behaviour by criterion and system, from the reviewers' notes on the Arabic evaluation set. Levels follow one rule from the report's wording: strong where the reviewers state a positive result, weak where they state a defect, mixed where the note is qualified or says not separately scored or no recurring issue was isolated, and not rated where the criterion was not evaluated. The reviewers' phrase behind each cell is in the notes below. Separate aggregate rates were not calculated.
M3Cohere Transcribe ArabicElevenLabs Scribe v1Deepgram Nova-3Google Latest LongGoogle Chirp 3Sonix
Classical ArabicStrongStrongStrongStrongWeakStrongStrong
English inside ArabicStrongWeakStrongMixedWeakStrongWeak
Text omissionStrongWeakMixedWeakMixedStrongWeak
PunctuationMixedWeakMixedMixedMixedMixedWeak
Spelling and terminologyStrongWeakMixedWeakWeakStrongWeak
Word repetitionStrongWeakMixedMixedMixedMixedMixed
DialectsStrongStrongStrongStrongStrongStrongStrong
Speaker identificationStrongNot ratedNot ratedNot ratedNot ratedNot ratedNot rated
Filler words and event tagsStrongWeakWeakMixedMixedMixedMixed
Character error rate (CER), lower is better1.62%4.93%5.01%6.09%9.01%2.26%9.8%
Reviewer notes behind each cell
Classical Arabic
M3 Highly accurate. · Cohere Transcribe Arabic Highly accurate. · ElevenLabs Scribe v1 Accurate. · Deepgram Nova-3 Accurate. · Google Latest Long Many errors. · Google Chirp 3 Accurate. · Sonix Accurate.
English inside Arabic
M3 Preserved English technical and business terminology more consistently within Arabic speech. · Cohere Transcribe Arabic English technical and business terms were sometimes omitted, mistranscribed, or rendered inconsistently within Arabic speech. · ElevenLabs Scribe v1 English technical terminology was preserved particularly well in the reviewed output. · Deepgram Nova-3 English-term performance was not scored separately; reviewed examples showed inconsistent rendering alongside segment-start omissions. · Google Latest Long English technical terms were frequently omitted. · Google Chirp 3 English technical terminology was preserved strongly within Arabic speech. · Sonix Names and English technical terminology were frequently transcribed incorrectly or omitted.
Text omission
M3 Preserves business-critical content more reliably and reduces the loss of important numbers, terminology, and organisation or product names. · Cohere Transcribe Arabic May omit or distort business-critical content, including numbers, technical terms, and organisation or product names. · ElevenLabs Scribe v1 No separate omission rate was calculated; the principal qualitative issue was inserted filler-like content rather than a recurring segment-start omission pattern. · Deepgram Nova-3 Substantial content loss was observed, particularly at the beginning of source segments. · Google Latest Long Correctly rejected many non-speech segments, but missed a small number of speech-bearing segments and sometimes omitted English terms or numbers. · Google Chirp 3 Low content loss was observed, with substantially fewer empty outputs than Google Latest Long. · Sonix Content loss was observed particularly at the beginning of source segments.
Punctuation
M3 Not separately scored; M3 produced clearer sentence boundaries, but punctuation was not evaluated as a separate aggregate measure. · Cohere Transcribe Arabic Repeated punctuation patterns that reduce readability were observed in the reviewed output. · ElevenLabs Scribe v1 Not separately scored; readability was affected more by filler-like insertions than by a recurring punctuation defect. · Deepgram Nova-3 Not separately scored; no recurring punctuation defect was isolated in the qualitative review. · Google Latest Long Not separately scored; no recurring punctuation defect was isolated in the qualitative review. · Google Chirp 3 Not separately scored; no recurring punctuation defect was highlighted in the qualitative review. · Sonix Punctuation quality was weak, with excessive sentence-final marks observed in the reviewed output.
Spelling and terminology
M3 Produces more consistent business terminology for reading and search. · Cohere Transcribe Arabic Technical and business terms may appear inconsistently or in phonetic form. · ElevenLabs Scribe v1 English technical terms were handled well, although filler-like insertions reduced overall transcript cleanliness. · Deepgram Nova-3 Participant names were frequently transcribed incorrectly. · Google Latest Long English technical terms were frequently missed, and some numbers were omitted. · Google Chirp 3 English technical terms were rendered reliably within Arabic speech. · Sonix Performance was weak on participant names and English technical terminology.
Word repetition
M3 Showed substantially fewer repetition problems in the reviewed output. · Cohere Transcribe Arabic Token and phrase repetition loops were observed in the reviewed output. · ElevenLabs Scribe v1 No separate repetition rate was calculated; frequent filler-like vocalisations were the more prominent insertion issue. · Deepgram Nova-3 No recurring repetition issue was isolated in the qualitative review. · Google Latest Long No recurring repetition issue was isolated in the qualitative review. · Google Chirp 3 No recurring repetition issue was isolated in the qualitative review. · Sonix No recurring word-repetition issue was isolated; punctuation was the more prominent readability problem.
Dialects
M3 Produced Arabic transcripts across the dialects represented in this evaluation set. · Cohere Transcribe Arabic Produced Arabic transcripts across the dialects represented in this evaluation set. · ElevenLabs Scribe v1 Produced Arabic transcripts across the dialects represented in this evaluation set. · Deepgram Nova-3 Produced Arabic transcripts across the dialects represented in this evaluation set. · Google Latest Long Produced Arabic transcripts across the dialects represented in this evaluation set. · Google Chirp 3 Produced Arabic transcripts across the dialects represented in this evaluation set. · Sonix Produced Arabic transcripts across the dialects represented in this evaluation set.
Speaker identification
M3 Uses participant-linked audio channels and platform participant information. Platform-provided speaker attribution remained linked throughout the evaluated meetings. · Cohere Transcribe Arabic Not evaluated for the external configurations. Each configuration received the same participant-separated audio segments prepared by the M3 capture workflow and was scored only on transcription. · ElevenLabs Scribe v1 Not evaluated for the external configurations. Each configuration received the same participant-separated audio segments prepared by the M3 capture workflow and was scored only on transcription. · Deepgram Nova-3 Not evaluated for the external configurations. Each configuration received the same participant-separated audio segments prepared by the M3 capture workflow and was scored only on transcription. · Google Latest Long Not evaluated for the external configurations. Each configuration received the same participant-separated audio segments prepared by the M3 capture workflow and was scored only on transcription. · Google Chirp 3 Not evaluated for the external configurations. Each configuration received the same participant-separated audio segments prepared by the M3 capture workflow and was scored only on transcription. · Sonix Not evaluated for the external configurations. Each configuration received the same participant-separated audio segments prepared by the M3 capture workflow and was scored only on transcription.
Filler words and event tags
M3 Did not show the same concentration of filler-like, event-tag, or repeated-punctuation artifacts in the review. · Cohere Transcribe Arabic Filler-like insertions, event-style tags, and repeated-punctuation artifacts were observed in the reviewed output. · ElevenLabs Scribe v1 Frequent spurious filler-like vocalisations such as “آآآ” were observed, and one segment switched to an unexpected language or script. · Deepgram Nova-3 No recurring filler-word or event-tag pattern was isolated in the qualitative review. · Google Latest Long Handled many non-speech segments correctly, although a small number of speech-bearing segments were also rejected. · Google Chirp 3 No recurring filler-word or event-tag pattern was highlighted in the qualitative review. · Sonix No recurring event-tag pattern was isolated; excessive punctuation was the more prominent artifact.

English benchmark

English results: character error rate (CER)

This section reports the English comparison. It uses the same metric as the Arabic benchmark, character error rate, on a separate English evaluation set, so its values are read within their own source material rather than as a ranking of the two languages.

M3 recorded 1.83%. In practical terms, 1.83% CER corresponds to roughly 18 character edits per 1,000 reference characters after the stated normalisation rules. The closest external configuration, Google Chirp 3, recorded 3.18%, a gap of 1.35 percentage points: M3's CER is 42% lower on this English set. Against the other five external configurations the relative reduction ranges from 48% to 77%. These are comparisons on this evaluation set, not universal vendor rankings.

1.83%M3 English CERLowest value in the English table
98.17%English character accuracy100 minus the M3 CER
42% to 77%Lower English CERM3 versus each external configuration
7Evaluated entriesM3 and six external ASR model configurations

On English meetings, the M3 system also records the lowest character error rate against the six external ASR model configurations

Lower is better. Character error rate (CER) after the stated normalisation rules.

Same source audio, same human-verified reference, same scoring rules for every entry, with every system set to English. This comparison evaluates transcription only. Character accuracy and M3's relative reduction are derived from the aggregate CER values. See "How we measured the English benchmark" for the model and language settings.
Character error rate and character accuracy on English meetings for the M3 system and the six external ASR model configurations, with M3's relative reduction versus each configuration. Lower CER is better. Accuracy and relative reduction are derived from the aggregate CER values; settings under How we measured the English benchmark.
RankEvaluated entryCER (%), lower is betterCharacter accuracy (%)M3 vs this entry
1M31.8398.17baseline
2Google Chirp 33.1896.8242% lower (1.35 points)
3Cohere Transcribe Arabic3.5496.4648% lower (1.71 points)
4Sonix4.2195.7957% lower (2.38 points)
5Deepgram Nova-35.5894.4267% lower (3.75 points)
6ElevenLabs Scribe v16.4193.5971% lower (4.58 points)
7Google Latest Long7.9892.0277% lower (6.15 points)

English qualitative review by criterion

This second English table applies the same qualitative structure used for Arabic. It separates short-utterance handling, omissions, punctuation, names and terminology, repetition, speaker identification and unsupported insertions, instead of turning aggregate CER into a second one-row ranking. Its source is the English reference-error review; the final row repeats the aggregate CER.

Observed English output behaviour by criterion and system. Qualitative levels summarise the English reference-error review; the final row repeats the separately reported aggregate CER values. The assignment method is in the notes below.
M3Cohere Transcribe ArabicElevenLabs Scribe v1Deepgram Nova-3Google Latest LongGoogle Chirp 3Sonix
Overall English transcriptionStrongStrongWeakMixedWeakStrongMixed
Short utterancesStrongStrongMixedMixedWeakMixedMixed
Text omissionStrongMixedStrongMixedMixedStrongWeak
PunctuationMixedMixedWeakMixedMixedMixedMixed
Names, numbers and terminologyStrongStrongMixedWeakMixedMixedWeak
Word repetitionStrongStrongWeakWeakMixedMixedMixed
Speaker identificationNot ratedNot ratedNot ratedNot ratedNot ratedNot ratedNot rated
Unsupported fillers and insertionsStrongStrongWeakMixedMixedMixedMixed
Character error rate (CER), lower is better1.83%3.54%6.41%5.58%7.98%3.18%4.21%
Reviewer notes and assignment method
Overall English transcription
The level is the displayed CER band: Strong at 4% or lower, Mixed above 4% and below 6%, and Weak at 6% or higher.
Short utterances
Two clips in the English review are shorter than five seconds. M3 and Cohere Transcribe Arabic match both references; Google Latest Long shows the clearest loss on the short clips, while the other four configurations show minor transcript differences.
Text omission
The reviewed outputs show the lowest deletion concentration for ElevenLabs Scribe v1, Google Chirp 3 and M3; Cohere Transcribe Arabic, Deepgram Nova-3 and Google Latest Long form a qualified middle group; Sonix shows the most concentrated content loss. ElevenLabs Scribe v1 is rated strong on omission only; its insertions are rated in a separate row.
Punctuation
Punctuation is excluded from CER, so every entry stays Mixed unless the review shows a recurring defect. ElevenLabs Scribe v1 is Weak here because it produced substantially more punctuation than the reference and repeated punctuation sequences.
Names, numbers and terminology
The review checked that names, acronyms, numerals and business terms such as RFP, OSS, IP, Matrix and MVP stayed visible. M3 has the most consistent result; Deepgram Nova-3 and Sonix show the most recurring name or terminology loss.
Word repetition
M3 shows no unsupported adjacent repeat and Cohere Transcribe Arabic one isolated repeat. Repetition is concentrated in ElevenLabs Scribe v1 and Deepgram Nova-3; Google Latest Long, Google Chirp 3 and Sonix show a qualified middle pattern.
Speaker identification
Speaker identification was not scored as a separate accuracy metric in the English benchmark. M3 kept the platform participant link as a product-workflow capability; the external configurations received participant-separated clips and were evaluated for transcription only. Every entry is therefore Not rated in this row.
Unsupported fillers and insertions
M3 has the cleanest reviewed output. ElevenLabs Scribe v1 shows the clearest recurring filler and insertion pattern; the other entries are rated from the relative concentration visible in the reviewed outputs.
Scope
The qualitative rows summarise the English reference-error review and follow the same assignment rule as the Arabic matrix: Strong for a clear positive result, Weak for a recurring or concentrated defect, Mixed for a qualified result, and Not rated where the criterion was not evaluated. They are not additional CER measurements or statistical-significance claims. Only the final row repeats the aggregate CER values.

English examples: three selected clips from the evaluation

For each example, every system's output is shown alongside the human-reviewed reference transcript.

The examples are qualitative illustrations and do not recalculate the aggregate English CER. Human-verified reference transcripts are available for all three clips, so each output can be compared directly with what was spoken.

English example 1Terminology and number

Core42 and Compass in a two-second turn

Source clip
  • Preserved term
  • Unsupported text
  • Filler-like insertion
  • No transcript
M3

access to core 42 (preserved term) compass models (preserved term), right?

Reference

access to core 42 compass models, right?

Why this example matters. M3 matches the reference exactly, keeping “core 42,” “compass models” and the closing “right.” Cohere Transcribe Arabic also keeps the wording, while the other five configurations alter one or more of these terms.

English example 2Single-word meaning

“Clear” versus “good” or “great” changes the meeting record

Source clip
  • Preserved term
  • Unsupported text
  • Filler-like insertion
  • No transcript
M3

No, no, everything is clear (preserved term) for now. Thanks.

Reference

No, no, everything is clear for now. Thanks.

Why this example matters. The reference says “clear,” and M3 keeps it, as do Cohere Transcribe Arabic, ElevenLabs Scribe v1 and Deepgram Nova-3. Google Chirp 3 and Sonix substitute “good” and “great,” and Google Latest Long turns the answer into “Not everything is clear,” which reverses it.

English example 3Role and participant name

Note taker and “Adam” in a short question

Source clip
  • Preserved term
  • Unsupported text
  • Filler-like insertion
  • No transcript
M3

This is not your note taker (preserved term), Adam (preserved term)?

Reference

This is not your note taker, Adam?

Why this example matters. M3 matches the reference exactly, keeping the negation, the role “note taker” and the participant name “Adam.” Cohere Transcribe Arabic also matches; the other five configurations add a filler, omit a word, or alter the role, the name, or both.

How we measured the English benchmark

The English evaluation uses real English-language business meetings. This page reports one explicitly English-configured entry for the M3 system and for each of the six external ASR model configurations.

  1. Source material

    Real English-language business meetings. Every scored segment was paired with an English reference transcript verified by a human reviewer listening to the source audio.

  2. Model and language settings

    The reported M3 entry used M3 + soxr + 1 s silence with --language en. Every external configuration was also set to English before transcription: Google Chirp 3 (Speech-to-Text V2, model chirp_3) and Google Latest Long (Speech-to-Text V1, model latest_long) with en-US; ElevenLabs Scribe v1 (scribe_v1) with eng; Deepgram Nova-3 (nova-3) with en; Cohere Transcribe Arabic (cohere-transcribe-arabic-07-2026, Cohere's latest transcription model, which covers Arabic and English) with language=en; and Sonix with en. For the separate Arabic evaluation, Cohere Transcribe Arabic used language=ar rather than automatic language detection.

  3. Same evaluation inputs

    M3 and all six external ASR model configurations were evaluated on exactly the same audio clips and segment boundaries, against the same human-verified reference transcripts.

  4. Metric

    All English figures on this page are character error rates: character substitutions, deletions and insertions divided by the number of reference characters after normalisation.

  5. Normalisation

    Punctuation, letter case and whitespace are ignored. The exception is a missing space that fuses two words together, which counts as one error.

  6. Derived reporting

    Character accuracy is 100 minus CER. M3's relative reduction versus another entry is (that entry's CER minus M3's CER) divided by that entry's CER, rounded to the nearest whole percent.

Evidence map for the English benchmark

ClaimStatus
M3 1.83% CER versus 3.18% for Google Chirp 3, 3.54% for Cohere Transcribe Arabic, 4.21% for Sonix, 5.58% for Deepgram Nova-3, 6.41% for ElevenLabs Scribe v1 and 7.98% for Google Latest Long.The aggregate English results table; lower CER is better.Reported in the results.
The English evaluation uses real English-language business meetings with human-verified reference transcripts.A human reviewer verified every scored reference while listening to the source audio.Verified in the evaluation process.
Every evaluated system was explicitly set to English before transcription.M3 + soxr + 1 s silence with en; Google Chirp 3 and Google Latest Long with en-US; ElevenLabs Scribe v1 with eng; Deepgram Nova-3, Cohere Transcribe Arabic and Sonix with en.Verified in the run metadata.
All seven entries were scored on exactly the same audio clips with the same normalisation rules.Punctuation, letter case and whitespace are ignored; a missing space that fuses two words counts as one error.Confirmed for the English evaluation.
M3 character accuracy is 98.17%, and its CER is 42%, 48%, 57%, 67%, 71% and 77% lower than the six external configurations respectively.Arithmetic on the reported aggregate CER values; relative reductions rounded to whole percentages.Derived from reported values.
The English matrix summarises short-utterance handling, text omission, punctuation, names and terminology, repetition, speaker identification and unsupported insertions for the seven entries.Criterion-level review of the reference-error analysis. The overall row uses the displayed CER bands; the other rows are qualitative summaries of the review, not additional aggregate rates.Observed in the review; assignment notes under the matrix.
Three short English clips reproduce the source audio, the M3 output and the outputs of the six external configurations.Human-verified reference transcripts are available for all three clips.Documented qualitatively against the human-verified references.

Speed

Validated. The M3 meeting transcription workflow processes a one-hour meeting in 45 seconds on an L4 GPU, 80x real time. The figure is batch throughput for recorded audio, measured on a meeting-duration basis. Live delivery, while a meeting is in progress, is measured separately and is not part of this report.

One hour of meeting audio in 45 seconds

80x real time on an L4 GPU.

Batch throughput of the M3 meeting transcription workflow on an L4 GPU, for recorded audio.

From transcript to meeting record

Product capability. The benchmarks above score transcription only. In MeetriX, transcription is one stage of a longer workflow. It is described here as a product capability so that the two are not confused.

  • Meeting capture. Audio is captured with the available participant identity attached from the moment of capture, producing a clearer record of who said what.
  • Participant and time context. Each contribution is kept with its speaker identity and meeting time throughout processing. Platforms that provide separate participant channels and platforms that provide mixed audio with speaker-activity information are both supported.
  • Speech preparation. Silence, short turns, pauses and speech boundaries are handled before transcription, reducing cut-off words, repetition and non-speech artifacts.
  • Arabic transcription and terminology. Multi-dialect Arabic is transcribed, and English names and technical terms spoken inside Arabic sentences are written in their recognised business form so transcripts are easier to read and search.
  • Structured meeting record. Every transcript segment stays connected to its original speaker, meeting time and source audio segment, giving a speaker-linked, time-aligned record that can enter knowledge bases, analytics, CRM or archives.
  • Summary. A post-meeting stage turns the record into a concise summary that can include the meeting title, date and language, attendees, a general summary, key discussion points, decisions and recommendations, and next steps. The template is configurable.
  • Email delivery. The finished summary is sent to the registered email addresses of the participants, following the organisation's access, consent and retention policies.

In the evaluated deployment, the transcription path called no external transcription service. M3 can operate within customer-controlled infrastructure, keeping meeting audio, transcripts and summaries inside the organisation's data-governance boundary, and it adapts its workload to the available compute while preserving speaker identity, timestamps and transcript order. Platform-provided speaker attribution remained linked throughout every evaluated meeting in the M3 participant-channel workflow; this is a coverage statement, and no speaker-attributed error rate is reported.

Note

Product capability, not benchmark. The summary and delivery stages were not scored in this evaluation. The CER figures on this page cover transcription only.

How we measured the Arabic benchmark

For the Arabic benchmark, the M3 system and six external ASR model configurations were given the same material and scored by the same rules. The steps below condense the comparison protocol and the scoring rules from the report. They are published so that a reader can reproduce the scoring on their own meetings.

  1. Source material

    Real multi-dialect Arabic business meetings, with human-verified reference transcripts prepared for every scored segment.

  2. Evaluation-data separation

    The meetings and human-verified reference transcripts used in this evaluation were not used to train M3.

  3. Same audio for every system

    The M3 system and the six external ASR model configurations received the same source audio, pre-segmented by the M3 capture workflow into participant-separated segments. External configurations were scored on transcription only; their speaker diarization, the step that splits audio by who is speaking, was not used or evaluated. Every figure for the external configurations on this page was produced by running the vendor's API on the evaluation audio; none is taken from a vendor publication.

  4. Same evaluation inputs

    The M3 system and all six external ASR model configurations were evaluated using the same audio segments and segment boundaries. All outputs were scored against the same human-verified references using the same normalisation and scoring rules. The reference transcripts were used only for scoring and were not provided to any system during transcription.

  5. Configurations

    Google Chirp 3 is Google Cloud Speech-to-Text V2 with model chirp_3 and ar-XA; Google Latest Long is Google Cloud Speech-to-Text V1 with model latest_long and ar-SA; ElevenLabs Scribe v1 is model scribe_v1 with language_code=ara; Cohere Transcribe Arabic is model cohere-transcribe-arabic-07-2026, Cohere's latest transcription model, set explicitly to language=ar for this Arabic evaluation (and to language=en for the separate English evaluation); Deepgram Nova-3 and Sonix (Sonix.ai) ran as named, with no further model string given in the report. M3 ran as the complete meeting system with participant-linked audio channels from its capture workflow.

  6. Why the comparison uses ElevenLabs Scribe v1

    To verify that the use of Scribe v1 did not disadvantage ElevenLabs, we reran the same evaluation audio with Scribe v2 under the same input and scoring conditions. Scribe v2 did not improve transcription accuracy on this material, and Scribe v1 performed slightly better. We therefore retain the Scribe v1 result in the main comparison. This choice reflects the observed results on this evaluation set and should not be interpreted as a general ranking of ElevenLabs model versions.

  7. Metric

    CER = (substitutions + deletions + insertions) divided by the characters in the human-verified reference, computed after normalisation. Character accuracy is 100 minus CER.

  8. Dialect and standard forms

    An Arabic word is correct whether written in its dialect form or its Modern Standard Arabic form, as long as the meaning is the same.

  9. English words

    An English word is correct whether written in Latin letters, spelled phonetically in Arabic script, or translated into an equivalent Arabic word.

  10. Numbers

    Compared by value: 15, ١٥ and خمسة عشر count as the same thing. Numbers and English business terms stay in the scored content; they are not stripped out.

  11. Punctuation, case, letter variants, whitespace

    Punctuation is ignored entirely. Latin text is case-insensitive. Arabic letter variants such as the forms of alef are normalised. Whitespace is ignored, except that a missing space that fuses two words counts as one error.

  12. Remaining differences

    After these equivalences, every remaining missing, added or substituted character is an error. Only segments with a verified reference enter the CER. An empty output against a speech-bearing reference counts as deletions; extra unsupported output counts as insertions.

  13. Qualitative review

    Reviewers recorded content omission, names and English terminology, filler-like insertions, unexpected language changes, repeated punctuation and repetition loops for the M3 system and each external ASR model configuration. No separate aggregate rates were calculated.

Evidence map for the Arabic benchmark

ClaimStatus
M3 1.62% CER versus 2.26% for Google Chirp 3, 4.93% for Cohere Transcribe Arabic, 5.01% for ElevenLabs Scribe v1, 6.09% for Deepgram Nova-3, 9.01% for Google Latest Long and 9.8% for Sonix; relative reductions of 28%, 67%, 68%, 73%, 82% and 83%.The same human-verified evaluation set of real meetings, same reference, same scoring rules.Measured.
Deepgram Nova-3 showed participant-name errors and segment-start omissions; ElevenLabs Scribe v1 preserved English technical terms well but produced frequent spurious filler-like vocalisations and one unexpected language or script switch; Google Latest Long handled many non-speech segments correctly but missed a small number of speech-bearing segments and frequently omitted English terms, with occasional number omissions; Google Chirp 3 handled English technical terms strongly and showed low content loss; Sonix showed errors in names and English technical terms, segment-start omissions and excessive sentence-final punctuation.Qualitative review of the same evaluation set.Observed qualitatively; separate aggregate rates not calculated.
Platform-provided speaker attribution remained linked throughout every evaluated meeting.M3 participant-channel workflow; external configurations were evaluated for transcription only.Measured for the M3 participant-channel workflow.
80x batch throughput; 45 seconds per hour of meeting audio.M3 meeting transcription workflow for a one-hour meeting on an L4 GPU.Validated; batch throughput for recorded audio.
No external transcription call.Evaluated transcription path.Verified in the evaluated deployment.

Limitations

  • One evaluation set per language. Every CER figure comes from a single human-verified evaluation set of real meetings in its language, one Arabic and one English. Each shows how the M3 system and the six external ASR model configurations compared on that material and should not be read as a universal rate for any of them.
  • Separate language-specific evaluation sets. Both the Arabic and English figures are CER, but the two sets use different source material. Compare systems within each language table; do not read 1.62% and 1.83% as a ranking of the two languages.
  • A snapshot in time. The benchmark was run in September 2026 using the language-specific configurations described in the English and Arabic methodology sections. Vendors update their models, so the figures describe the versions available at the time of each run.
  • A small margin at the top. In the Arabic benchmark, the gap between M3 and Google Chirp 3 is 0.64 percentage points, and on several example segments Google Chirp 3 is as complete as M3.
  • Throughput scope. The 45 seconds per hour and 80x figures describe batch processing of recorded audio on one hardware configuration, an L4 GPU. Live latency is measured separately.
  • Qualitative observations carry no aggregate rates. Statements about fillers, omissions, repetition and punctuation come from reviewer notes. The matrix levels follow one rule applied to the wording of those notes, and the phrases are shown so the mapping can be challenged.
  • The English qualitative levels are review summaries. Only the overall-transcription row uses the displayed CER thresholds. The other Strong, Mixed, Weak and Not rated cells summarise the reference-error review and the observed transcript differences; they are not additional aggregate rates or statistical-significance tests.
  • External speaker diarization was not evaluated. Every external configuration received participant-separated audio prepared by M3, so the comparison isolates transcription and says nothing about how those configurations attribute speakers on mixed audio. The M3 speaker-attribution result is a coverage statement; no speaker-attributed error rate is reported.
  • M3 is not perfect. The Arabic example cards show M3 omitting a spoken first name, returning a singular where the reference has a plural, and adding words that the reference does not contain.

To cite this report, use the reference below or the BibTeX record.

Cite this page

Lisan Research (2026). M3: multilingual meeting transcription, measured on real meetings. Arabic and English, September 2026. Lisan, published 5 October 2026. https://lisan.com/company/research/m3/
BibTeX
@techreport{lisan2026m3,
  author = {Lisan Research},
  title = {M3: multilingual meeting transcription, measured on real meetings},
  institution = {Lisan},
  type = {Evaluation report},
  year = {2026},
  month = {October},
  note = {Arabic and English, September 2026. Published 5 October 2026},
  url = {https://lisan.com/company/research/m3/},
}

Run M3 on your own meetings

The benchmark that matters is your audio. A pilot runs M3 on a sample of your meetings with a human-verified reference, and we publish the scoring rules to you before the run, so the result is measured the same way this report was.

Book a pilotWrite to support@lisan.com

Frequently asked questions

What is CER, and does this report use WER?

CER, the character error rate, is the share of characters in a transcript that had to be added, removed or changed to match what a human verified was said. This report scores CER after the normalisation rules listed under the two method sections, so for Arabic, dialect and standard spellings, Latin and phonetic English, and numeric forms count as equivalent. Word error rate is not reported here, and earlier MeetriX figures published as WER are superseded by this evaluation.

What does the 1.83% English result measure?

It is the character error rate of M3 on the English evaluation set, the same metric as the Arabic result. In practical terms, 1.83% CER corresponds to roughly 18 character edits per 1,000 reference characters after the stated normalisation rules. Every evaluated system was run on exactly the same audio clips with an explicit English language setting, against reference transcripts verified by a human listening to the audio. Punctuation, letter case and whitespace are ignored, except that a missing space that fuses two words counts as one error.

Can the Arabic and English CER figures be compared directly?

No. Both are CER, but they come from separate language-specific evaluation sets. Compare systems within each language table; do not read 1.62% and 1.83% as a ranking of the two languages.

Was the evaluation independently verified?

No. Lisan Research ran the evaluation. The protocol, scoring rules, examples and evidence maps are published on this page so the scoring can be reproduced, and a pilot on your own meetings with a human-verified reference is the way to check the result on your material.

Does M3 send meeting audio to a cloud speech API?

In the evaluated deployment the transcription path called no external transcription service. M3 can run within customer-controlled infrastructure, keeping meeting audio, transcripts and summaries inside the organisation's data-governance boundary.

Can we evaluate M3 on our meetings?

Yes. A pilot runs M3 on a sample of your meetings against a human-verified reference, with the scoring rules shared before the run. Write to support@lisan.com or book a pilot through MeetriX.

References

  • Lisan Research (2026). M3: multilingual meeting transcription, measured on real meetings. Evaluation Edition, September 2026 (the white paper behind the Arabic benchmark on this page).
  • Google Cloud Speech-to-Text V2 documentation, Chirp 3 (chirp_3), language codes ar-XA and en-US (step 5 of "How we measured the Arabic benchmark"; step 2 of "How we measured the English benchmark").
  • Google Cloud Speech-to-Text V1 documentation, Latest Long (latest_long), language codes ar-SA and en-US (the same steps).
  • ElevenLabs documentation, Scribe v1 (scribe_v1), language codes ara and eng (the same steps).
  • Cohere documentation, Cohere Transcribe Arabic (cohere-transcribe-arabic-07-2026), Cohere's latest transcription model, covering Arabic and English, run with language=ar and language=en (the same steps).
  • Deepgram Nova-3 (nova-3) and Sonix.ai: vendor documentation for the products as named; the report gives no further model string for Deepgram Nova-3 in Arabic or for Sonix.