Executive Summary

Discussions about developing Arabic models often focus on model size, parameter count, and architecture. Yet no model, regardless of scale, can exceed the limits of the data from which it learns.

 

This is why we did not begin with a single data repository: each source presented a different limitation. The Arabic web is vast but noisy. High-quality knowledge is concentrated in other languages, while a substantial share of Arabic knowledge remains locked inside books, periodicals, and PDF files that language models cannot consume directly. We therefore designed a three-stage pipeline whose components are technically independent but integrated through a unified data schema: Wasm for retrieving and cleaning the Arabic web while preserving its structure, Mutarjim for adding high-quality translated knowledge, and Baseer for converting document pages into structured Markdown.

 

Terminology note: In the research paper, “Wasm” refers to a comprehensive web-data processing framework comprising multiple stages: language detection; node- and document-level filtering; a dedicated Arabic language model for perplexity scoring; and an advanced method for removing near-duplicate content.

Overall Architecture: Three Pipelines, One Data Schema

 

The separation between these pipelines is deliberate. Web pages require parsing and decomposing the DOM tree to remove noise such as advertisements and sidebars. Translation requires bilingual semantic alignment. Scanned books require recovering text and reading order from pixels. Applying a single tool to all three sources would have obscured errors and made their origins harder to trace.

How We Built a Multi-Source Arabic Data Pipeline

Stage One: Collecting the Arabic Web with Wasm

1. Reducing the Search Space Before Downloading Data

One of the first challenges in working with Common Crawl is access cost. Each dump contains an enormous number of WARC records, while Arabic represents only about 0.6% of Common Crawl content, according to the Common Crawl Foundation estimate directly cited by the Wasm paper. We therefore began with the Common Crawl index, searching for pages whose metadata indicated the presence of Arabic and whose fetches had succeeded. For each page, we extracted its URL, WARC filename, byte offset, record length, detected languages, and domain.

 

This filtering takes place before the WARC content is downloaded, allowing most irrelevant pages to be discarded before consuming network bandwidth, memory, or storage. Each crawl snapshot was also processed independently, making it easier to rerun batches, retain temporal provenance, and perform deduplication without turning the process into one monolithic job that would be difficult to resume.

2. From Raw HTML to a Learnable Document

After retrieving the record identified by its offset and length, Wasm converts the page into HTML and cleans it in two stages. First, spaces and line breaks are normalized, while HTML comments, CSS content, and recurring elements unrelated to the main materialsuch as headers, footers, navigation bars, and menusare removed.

 

Second, the DOM tree is parsed and converted into Markdown that preserves headings, paragraphs, ordered and unordered lists, and tables, while retaining image positions in their natural sequence. Adjacent text elements of the same type are merged without breaking context. The page then becomes suitable for two use cases: training a language model on text alone, or training a multimodal model on an interleaved image–text sequence.

 

This is the practical distinction between Wasm and several earlier Arabic web datasets. Its output is not merely a bag of scattered sentences, but a document with an internal hierarchy and explicit relationships among headings, paragraphs, images, and captions.

3. Two-Level Filtering Instead of a Single Page-Level Decision

The quality system first operates at the level of an HTML tag or textual nodesuch as a paragraph, list, or sectionand then reapplies a set of similar checks, with different thresholds, to the full document. This prevents an advertising paragraph or repeated navigation menu from causing an otherwise valuable article to be discarded in its entirety.

 

Adapting the rules to the characteristics of Arabic was essential. Rules borrowed from English-language pipelines may treat lexical repetition, a low proportion of stopwords, or sparse punctuation as indicators of poor quality. In Arabic, however, repetition may serve a legitimate rhetorical function; a sentence can be grammatically sound despite containing few function words; and many Arabic pages use inconsistent punctuation without necessarily being low quality. We therefore reduced the weight assigned to repeated-word ratios, removed filters based on stopword, punctuation, and common-word ratios, and adopted more tolerant handling of special characters, abbreviations, and emojis.

 

At the same time, language detection was tightened to ensure that each unit was fundamentally Arabic without rejecting the natural presence of an English technical term or a product name written in Latin script. This balance allows the system to retain an Arabic programming page containing API names or technical excerpts while rejecting a foreign-language page that contains only a short Arabic list.

4. Perplexity as a Quality Gate

At the core of Wasm is a KenLM-based language model trained on carefully selected Arabic data covering a range of dialects and topics. The model calculates perplexity for each unit. Coherent human-written text generally falls within an acceptable range, whereas perplexity rises for concatenated words, corrupted encoding, low-value advertisements, arbitrarily mixed-language pages, and poor machine-generated text. Rather than applying an off-the-shelf English threshold, we iteratively calibrated the thresholds on large Arabic samples to prevent the filtering process from erasing dialects or non-journalistic writing styles.

 

The paper’s examples illustrate this clearly: Arabic strings with incorrect separators, text mixed with remnants of Latin encoding, and pages saturated with repetitive commercial keywords are excluded because of their high perplexity. Wasm therefore also addresses encoding corruption behaviorally: even when malformed text passes character-level detection, its probabilistic structure fails the language-model test.

5. Removing Near-Duplicates at the Node Level, Not the Entire Document Level

Web duplication does not always appear as two identical pages. Thousands of articles may share a category menu, advertisement, or legal template while the body of each article remains unique. Wasm therefore uses the Needleman–Wunsch algorithm to compare textual units and remove elements exceeding an 80% similarity threshold.

 

The decision is precise: the repeated node is removed while the rest of the document is retained. Compared with dropping an entire page or performing document-level deduplication alone, this approach increases data diversity and reduces the loss of unique content. Image links are preserved rather than downloaded during extraction, unsuitable domains are excluded through a blocklist, and computationally expensive image checks are deferred until required by the training task.

6. What Did This Stage Produce?

According to the MSDD (Misraj Structured Data Dump) dataset card published on Hugging Face, the public Wasm output contains 23 million documents drawn from multiple Common Crawl snapshots, with a reported file size of 63.4 GB. Each sample contains Markdown text interspersed with the <image> marker, an ordered list of image URLs, and a list of their associated captions. These figures describe the published output of the web pipeline; they do not represent the total size of the unified repository after translation and document extraction.

Stage Two: Enriching Knowledge with Mutarjim

1. Translation as a Coverage Decision

We do not translate the entire English web. This stage begins with a gap map: which domains are insufficiently represented in native Arabic web data? High-value sources are then selected across technology, science, healthcare, law, culture, and other fields. Each segment retains the source identifier, original language, domain, license, and retrieval date.

 

The goal is to introduce knowledge that is unavailable in Arabic while minimizing the stylistic imprint of machine translation. A text is not accepted merely because the model produced a grammatical Arabic sentence. It must demonstrate that the target language is correct, output length is proportionate to the source, the content is complete and non-duplicative, and the terminology remains faithful to the domain.

2. Why Mutarjim Is Suited to This Role

Mutarjim is a 1.5-billion-parameter decoder-only model built on Kuwain-1.5B, an Arabic-centric bilingual Arabic–English model. It was adapted in two stages. First, continued pretraining was performed on approximately 10 billion tokens of Arabic–English pairs using a next-token prediction objective. Two special language tokens<|Arabic|> and <|English|>were added, and the order of the two sentences in each pair was randomized so the model would not learn a dominant translation direction.

 

Second, the model was fine-tuned on approximately six million high-quality parallel pairs. During this stage, the source-side loss was masked through causal masking, enabling the model to learn to generate the target from the source rather than spend capacity predicting the input text again. Training used approximately three billion tokens across two epochs.

 

This design is particularly important for translation into Arabic. Many multilingual models learn primarily from English-origin text, leaving Arabic as a secondary output language. Mutarjim’s fine-tuning data assigned a 2:1 weight to Arabic-origin samples over English-origin samples, with coverage spanning cultural, legal, scientific, medical, religious, and technical content. The objective was not only to improve Arabic-to-English translation, but also to expose the Arabic generator to enough original Arabic to learn its rhythm, syntax, and terminology rather than imitate English structure.

3. Quality Checks Before Accepting a Translation

The training data underwent a clear sequence of checks. Pairs were dropped if either side contained fewer than three tokens. A pair was rejected if the translated sentence was not in the required language. Large length discrepanciesoften indicating an incomplete translation or unjustified additionswere excluded, after which duplicate pairs were removed.

 

Review was not fully automated. Datasets exhibiting recurring patterns of context drift, hallucination, incorrect information, or weak alignment were excluded, and linguists inspected representative samples of generated translations. For Tarjama-25, 30,000 original passages of 50–100 words were collected, half originally written in Arabic and half in English. A bilingual subset of 5,000 samples was then selected, reviewed by professional translators, and corrected by domain experts.

 

In practice, translated text is admitted into training only after passing all of these gates: language verification, expected-length matching, deduplication, and terminology validation.

4. Results That Measure Translation Quality

On Tarjama-25, Mutarjim achieved a COMET score of 83.41, chrF++ of 68.67, and BLEU of 43.71 for English-to-Arabic translation. On the same dataset, GPT-4o mini scored 83.36, 66.36, and 38.52, respectively. These results do not mean that every output is automatically suitable for training. They do, however, provide empirical evidence that a small specialized model trained on original Arabic and domain-balanced data can compete with far larger models on the more difficult direction: generating Arabic.

Stage Three: Extracting Knowledge from Books and Research with Baseer

1. Why Extracting Text from a PDF Is Not Enough

Many Arabic PDFs contain no reliable text layer. Even when one exists, word order may be reversed, columns interleaved, or footnotes inserted into the middle of the body text. The Baseer pipeline therefore treats each page as a structured image rather than as a sequence of characters hidden inside the file.

 

The pipeline documented in the paper focuses on scanned PDF pages or pages generated from documents. Reflowable EPUB files are normally extracted directly from XHTML through a DOM-based pipeline similar to Wasm. They are routed to Baseer only when they use a fixed layout or consist of page images. This separation avoids running OCR on digital text that is already available, reducing both errors and cost.

2. From Pixels to Structured Markdown

Baseer is built on Qwen2.5-VL-3B-Instruct. This foundation was selected after a qualitative comparison of open vision-language models in terms of Arabic text accuracy, right-to-left handling, and overall consistency. Initial testing showed that the base model could occasionally revert to left-to-right order or struggle with diacritized text, which led us to specialize it for Arabic.

 

Rather than retraining every component, we froze the vision encoder and updated the language decoder and the remaining non-visual parameters. Experiments showed that this approach outperformed both full fine-tuning and LoRA, achieving a ChrF score of 89.79 compared with 84.79 and 85.52. A 4,096-token context window also outperformed 2,048 and 8,192 tokens: it provides sufficient context for a dense page without the excessive padding that can dilute the visual signal.

 

The output is not flat text. Content is written in Markdown; tables are represented in HTML so merged cells and complex structures are not lost; and dedicated tags identify page numbers, watermarks, and image positions. This makes the output suitable for serialization while retaining the layout signals required by a document model.

3. Training on the Diversity of Real Pages

Baseer’s training data comprises 500,000 image–text pairs. Of these, 300,000 were synthetic. The process began with web-sourced Markdown documents, which were converted from Markdown to HTML, then to Word, PDF, and finally high-resolution images. Controlled variation included 39 Arabic fonts, multiple page sizes and orientations, one- to three-column layouts, and font sizes ranging from 8 to 22 points.

 

Among the synthetic data, 150,000 examples underwent transformations designed to simulate real-world printing and scanning through 29 types of augmentation, including watermarks, printing defects, handwritten marks over the page, folds and yellowing, digital noise, perspective distortion, poor lighting, and various forms of blur.

 

The remaining 200,000 pairs came from real documents: books, magazines, educational materials, and academic research. Vision algorithms selected diverse pages based on the alignment and overlap of text boxes and page colors, ensuring the inclusion of tables, figures, index pages, columns, skewed layouts, images, and colored backgrounds. Experts then reviewed a representative sample of the vision-model-generated ground truth at both textual and structural levels.

4. Handling Footnotes, Tables, and Equations

Footnotes are not merely small text. They must be linked to their locations without interrupting the flow of the main body. Baseer’s training on real pages included complex footnotes, multiple columns, and non-standard typography, while TEDS measures its ability to recover tree structure rather than characters alone. Tables are converted into structured HTML, and tag variants are normalized before evaluation so the model is not penalized for formal differences that do not change meaning.

 

The Baseer paper does not provide an independent benchmark for converting mathematical equations into LaTeX; this capability should therefore not be presented as proven by the paper. In a research-focused production pipeline, equation regions should be detected and routed to a specialized mathematical OCR system, then reinserted in their correct positions within the Markdown.

5. Accuracy and Knowledge Value

On Misraj-DocOCRa 400-page, expert-reviewed benchmark covering diverse layoutsBaseer achieved a WER of 0.25, a TEDS score of 66, and a MARS score of 76.885. Its WER was better than Gemini 2.5 Pro’s 0.37 and Azure AI Document Intelligence’s 0.44, and it also outperformed them on overall structure metrics.

 

The value of this stage is not simply the number of pages processed. Web data tends toward news, short-form text, repeated content, and material optimized for search engines. Books and research papers contribute long-form arguments, extended definitional sequences, specialized terminology, results tables, footnotes, and citations. This structure gives a model deeper knowledge and longer context than a typical web page, provided that source licensing and document provenance are preserved.

Integration: From Three Sources to a Unified Training Database

1. A Standard Schema That Preserves Provenance

Each pipeline ultimately produces a unified record containing, at minimum: a persistent document identifier; a source typeweb/original, translated, or document-ocr; structured text; visual elements; language and translation direction, where applicable; domain; quality indicators; extraction method and model version; original URL and crawl timestamp; and licensing information.

 

Retaining these fields makes it possible to reconstruct the training mixture, exclude a source later found to be weak, and measure the contribution of each layer to the model instead of treating the corpus as a black box.

2. Normalization and Cross-Source Deduplication

Before comparison, text passes through a common normalization layer for Unicode characters, spaces, line breaks, and Markdown forms, while preserving potentially meaningful information such as diacritics in datasets created for that purpose. Exact matches are then removed using fingerprints computed on normalized text, after which near-duplicates are examined at the segment level. This prevents the same article from entering once from the web, again from a PDF, and a third time as a republished translation.

 

Wasm’s use of the Needleman–Wunsch algorithm with an 80% threshold for HTML-node-level deduplication is documented. The three papers, however, do not disclose the cross-pipeline near-duplicate removal method or the fingerprinting mechanism used. Before final publication, the actual production algorithm and thresholds should therefore be inserted rather than assuming the use of MinHash or SimHash without evidence.

3. Balance Does Not Mean Equal Quotas

If the sources were mixed strictly according to raw volume, web data would overwhelm the others. Quotas are therefore based on the desired effect: original Arabic remains the foundation; translation fills domain gaps without imposing a translated style on the model; and books and research increase knowledge density and context length.

 

Balance is monitored across source, domain, time period, Modern Standard Arabic and dialects, document length, proportion of translated text, and OCR quality. Rare, high-quality samples can receive carefully calibrated weights instead of being duplicated verbatim, while an independent validation set is retained for each segment to test for bias or degradation.

The Numbers: What Is Publicly Known and What Requires Internal Validation

 

In line with Misraj’s vision of building an Arabic data infrastructure capable of supporting advanced model development, we created an internal repository containing approximately two trillion Arabic tokens. The data was collected through integrated pipelines spanning Arabic web content, translated knowledge, and content extracted from books and research. MSDD represents only a published sample of Wasm’s output; it does not reflect the full scale of Misraj’s repository or all of its sources.

Available Evidence and the Required Evaluation Design

The papers demonstrate the value of specialization at the component level. The 1.5-billion-parameter Mutarjim model competed with substantially larger models and outperformed them on several metrics for translation into Arabic. Baseer reduced WER to 0.25 while demonstrating a clear advantage in structural recovery. A sample of Wasm’s output was also used in Baseer’s training data.

 

The difference should be evaluated across Modern Standard Arabic comprehension, knowledge questions, reasoning, long-form generation, translation, and dialect tasks, with separate results for each domain and dialect rather than a single average that conceals variation. Once results are available, the proposed final formulation is:

 

“Holding the architecture and training budget constant, the unified mixture increased the average [metric names] by [percentage] compared with a web-only baseline, with a [percentage] improvement in Modern Standard Arabic and [percentage] in dialects, while reducing [metric] on long-form generation tasks from [before] to [after].”

Conclusion

The hardest part of building Arabic training data was not finding text, but deciding what to retain, what to repair, and what should never enter training in the first place. Wasm transformed Common Crawl from a noisy HTML archive into structured Arabic documents that preserve page hierarchy. Mutarjim added knowledge unavailable in Arabic through a small specialized model and quality gates that prevent translation noise. Baseer opened the layer of books and research by converting visible pages into sequence-ready Markdown, with independent measurements for text and structural accuracy.

 

The real value emerges when these stages are combined without erasing their differences: provenance remains intact, quality is auditable, duplication is removed at multiple levels, and the mixture can be reweighted and measured. In this form, the corpus becomes an interpretable, improvable engineering assetnot merely a large token count.