MSDD - Misraj Structured Data Dump

Twenty-three million Arabic documents drawn from several Common Crawl snapshots, 63.4 GB in 361 Parquet shards. Each sample is Markdown text with the image marker left inline, an ordered list of the image URLs, and the list of their captions, so the position of every picture within the article survives, which is the whole point of the format. The images themselves are not included; the dump carries their links.

MSDD is the published output of the Wasm pipeline, and inherits its decisions end to end: index-level filtering before download, two-pass HTML cleaning, conversion to structured Markdown, quality filters applied at both the text-block and document level and recalibrated for Arabic, perplexity scoring by a KenLM model trained on Arabic, and node-level near-duplicate removal at an 80 per cent Needleman-Wunsch threshold. Each crawl snapshot was processed independently, which made batches restartable and kept the time provenance of every document traceable.

Pre-training Arabic language models on text only, or Arabic multimodal models on interleaved image-text documents, from the same dump. It is a sample of one pipeline's output and is not a slice of Misraj's full two-trillion-token repository, so it should be described as web-derived Arabic data and not as a representative cross-section of Arabic knowledge: web text skews towards news, short forms and material written for search engines.

Apache 2.0, listed publicly on Hugging Face with access granted on request. The listing, the card and the file manifest are readable by anyone, and the Parquet shards are handed out after an access request. Because image links are carried rather than image files, anyone training on the multimodal form fetches the pictures themselves under the originating sites' terms.

The Wasm paper documents the pipeline that produced this dump and the comparison against other Arabic corpora. MUDD is its sibling from the translation pipeline. Baseer's training data drew on a sample of this same web output.