WASM

A processing pipeline for building Arabic textual and multimodal datasets that are ready for training, analysis, and reuse.

 

In one sentence
WASM by Misraj AI collects Arabic web pages, reconstructs their content in a structured format, cleans and filters it using criteria tailored to Arabic, and preserves the natural relationship between text and images within each document.

 

Based on an analysis of the research paper Wasm: A Pipeline for Constructing Structured Arabic Interleaved Multimodal Corpora, first version, November 2025.

Executive Summary

The web may appear to be a vast source of Arabic data, yet this data does not reach AI teams in a ready-to-use form. A single page may combine original text with navigation menus, advertisements, duplicated content, styling code, and images detached from their context. When that page is converted into flat text, its headings, lists, tables, and the sequence connecting each image to its explanation are lost.

 

Misraj AI developed WASM to address this gap. WASM is an Arabic-focused processing pipeline built on Common Crawl data. It identifies pages containing Arabic content at an early stage, cleans their HTML, converts the content into structured Markdown, separates textual and visual elements while preserving their original order, and then applies filtering and deduplication at two levels: the individual text block and the complete document.

 

What makes WASM valuable to organizations is its ability to collect data and preserve its provenance, understand its structure, and adapt it for training language or multimodal models, building knowledge repositories, or conducting large-scale content analysis. According to the paper’s introduction, a portion of the resulting data was used to train Baseer, Misraj AI’s document-understanding model.

The Problem: Arabic Content Exists, but Accessing It Does Not Make It Usable Data

Organizations that rely on the web to build linguistic or knowledge assets face four interconnected challenges:

 

• Noise: Advertisements, navigation menus, footers, code comments, CSS, and excessive whitespace are mixed with the actual content.

• Loss of structure: Traditional text extraction flattens the page, erasing the distinctions between headings, paragraphs, lists, tables, and image captions.

• Filters poorly suited to Arabic: Rules designed for English may reject valid Arabic text because of rhetorical repetition, limited punctuation, or differences in the distribution of common words.

• Coarse deletion decisions: In some processing pipelines, a single duplicated element can cause the entire document to be discarded, including unique content unrelated to that duplication.

 

Why is this difficult?
The paper notes that Arabic accounts for only about 0.6% of Common Crawl content. Overly aggressive filtering therefore does not necessarily improve quality; it may instead discard a substantial portion of an already scarce linguistic resource.

Why Are “Image + Caption” Pairs Not Enough?

Multimodal models learn from the presence of an image and adjacent text, but they also benefit from the sequence of the full document: a heading introduces the topic, a paragraph explains a figure, two images are compared, and a table summarizes the result. Interleaved data preserves this extended context, whereas paired datasets separate the image from the broader narrative in which it originally appeared.

The Solution: A Processing Pipeline That Preserves Meaning Before Reducing Noise

WASM was built using the OBELICS framework as a starting point, then adapted and extended for Arabic while preserving the page’s hierarchical structure in Markdown. This allows the output to be used either as text-only content or as multimodal documents in which text and images alternate according to their original positions on the page.

1. Early Detection of Arabic Content

The Common Crawl index is first examined to select pages that contain Arabic, even when they are not entirely in Arabic. Filtering before downloading complete WARC files reduces processing time, memory use, and storage requirements. For every page, the system retains its URL, offset and length within the WARC file, detected language or languages, and source domain.

2. Page Retrieval and HTML Normalization

The system retrieves the raw content using the page’s location, offset, and length metadata. It then normalizes whitespace and line breaks and removes HTML comments, headers, footers, navigation menus, non-semantic components, and CSS content.

3. Conversion to Structured Markdown

The page is converted into Markdown while preserving the distinctions among headings, paragraphs, ordered and unordered lists, and tables. Images and figures are captured as visual elements, while adjacent text elements carrying the same tag are merged to preserve the natural flow of the content.

4. Text-Block-Level Filtering

Each coherent blocksuch as a paragraph, list, or sectionis evaluated independently rather than treating the page as a single undifferentiated unit. Criteria that would unfairly penalize Arabic are relaxed, while checks for Arabic dominance are strengthened without excluding naturally occurring foreign terminology.

5. Coherence Measurement Using an Arabic Model

The team developed a custom perplexity model based on the KenLM framework and trained it on high-quality content spanning multiple dialects and subject areas. Its purpose is to identify corrupted text, low-quality advertising content, and poorly generated machine text while retaining coherent, human-authored Arabic.

6. Fine-Grained Deduplication

Instead of deleting an entire page because it contains duplicated elements, WASM uses the Needleman–Wunsch algorithm with an 80% similarity threshold to identify and remove near-duplicate blocks. This preserves the unique portions of the document.

7. Full-Document Validation

Quality criteria are then reapplied to the page as a whole, using thresholds different from those applied at block level. This helps control the quality of long-form content without unnecessarily sacrificing the volume of available Arabic data.

The Most Important Engineering Decision: Adapting Quality Criteria to Arabic

The team did not simply transfer English-language filters unchanged. The paper shows that some indicators commonly used in English processing pipelines can become sources of linguistic bias when applied outside the language for which they were designed.

 

 

Table 1. Adaptation of quality indicators to the characteristics of Arabic, based on Section 1.6.1 of the paper.

What Does WASM Produce?

WASM produces a structured representation of each document, retaining both the content needed by the model and the structural signals that explain how its parts relate to one another:

 

• The natural sequence of paragraphs and images as they appeared on the page.

• The hierarchy of headings and sections, as well as lists and tables.

• Relationships among an image, its caption, and its preceding and following context.

• Metadata about the source, the page’s location within a Common Crawl snapshot, its language, and its domain.

• The flexibility to derive a text corpus, a multimodal training dataset, or task-specific formats.

Results Supported by the Paper

The researchers evaluated the coherence model on random samples of 100,000 examples from each dataset. The “rejection rate” represents the proportion of examples that exceeded the acceptable perplexity threshold.

 

 

These percentages measure only what the WASM team’s filter rejected when applied to the samples. They do not constitute a comprehensive assessment of each dataset’s quality or an absolute ranking among them.

 

The qualitative examples in the paper provide practical explanations for rejection cases: reversed text or corrupted encoding, incoherent mixing of Arabic with other languages, automatically generated filler pages, and repetitive advertising content.

 

How should these figures be interpreted?
These percentages measure what the team’s filter rejected when applied to the samples. They do not represent a comprehensive evaluation of each dataset or an absolute quality ranking. In addition, WASM’s result reflects data that had already passed through the same processing pipeline and should therefore be understood primarily as an internal consistency test of the filtering process.

When Is WASM a Suitable Component for Organizations?

AI Laboratories

When building pre-training or continued-pre-training datasets for Arabic language or multimodal models, particularly when teams require controllable quality and richer structure than raw text can provide.

Media Organizations and Publishers

When transforming a large web archive into structured content, analyzing its topics, or building semantic search that retains headings, tables, and images in context.

Government and Cultural Institutions

When collecting public Arabic content from multiple sources to build knowledge repositories or research datasets while preserving provenance and structure.

Data and Product Teams

When a reusable pipeline is needed upstream of RAG, classification, extraction, or model training, rather than cleaning web data from scratch for every project.

Arabic-Language Researchers

When studying dialects, topics, and digital language patterns at scale without relying on English-language filters that may discard valid Arabic samples.

What Distinguishes WASM from Traditional Extraction?

Conclusion

WASM addresses the challenge at the stage preceding model training: how can the Arabic web be transformed from pages designed for human reading into structured data that a system can understand, without losing the structure that carries part of the meaning?

 

Early filtering, preservation of Markdown structure, adaptation of quality criteria to Arabic, and block-level deduplication make WASM a practical foundation for building richer and more flexible Arabic data assets. This is its value to organizations: not merely accessing the web, but transforming it into reliable material on which new systems can be built.

Source 

Hennara, K., Bastati, A., Hreden, M., Hamed, M. M., Aldallal, Z., Chrouf, S., & AlModhayan, S. (2025). Wasm: A Pipeline for Constructing Structured Arabic Interleaved Multimodal Corpora. arXiv:2511.07080v1.