Why Malayalam Needs Its Own Corpus

Malayalam is not underrepresented in LLMs by accident. It is underrepresented by design — or rather, by the economics of data collection. Web scraping produces enormous quantities of English text cheaply. Malayalam has a rich web presence, but it is not structured in ways that translate cleanly into training-quality data.

The actual failure is not quantity — it is domain coverage and annotation quality.

Scraping the Malayalam Wikipedia and Common Crawl gets you broad vocabulary coverage. It does not get you:

Administrative Malayalam (government forms, legal documents, land registry language)
Educational Malayalam (school curricula, examination language across Kerala Board and CBSE Malayalam medium)
Professional Malayalam (medical, financial, agricultural sector terminology)
Conversational registers (formal written vs. spoken colloquial vs. regional dialect variation)

Our benchmark found ~73% failure rate on administrative queries specifically. This is the domain gap.

What We're Building

The Araskova Malayalam corpus is being built across four domain categories:

1. Administrative & Legal

Government circulars, panchayat meeting minutes, land registry documents, court judgments translated into Malayalam, and official notices from Kerala state departments. These are the texts that a real Malayalam LLM needs to understand to be useful in government, legal, and civil service contexts.

Collection method: Public records, KSFE publications, RTI responses, Kerala Gazette archives.

2. Educational

Kerala State Syllabus textbooks (Classes 1–12, all subjects in Malayalam medium), SCERT publications, examination papers, and teacher training materials. This feeds directly into Aerial Board, our offline classroom AI product.

Collection method: Direct licensing from SCERT where possible; OCR + manual correction on printed materials.

3. News & Periodicals

Manorama, Mathrubhumi, Deepika, Kerala Kaumudi archives — spanning multiple decades to capture vocabulary and idiom evolution. Malayalam journalism is among the oldest and highest-quality print traditions in any Indian language.

Collection method: Publisher partnerships where available; public archive scraping with robots.txt compliance.

4. Conversational & Spoken

Transcriptions of public lectures, legislative assembly proceedings (Vidhan Sabha debates in Malayalam), and community radio. This is the hardest to collect and the most valuable — spoken and formal Malayalam diverge significantly.

Collection method: Manual transcription, ASR + human review pipeline.

Quality Over Quantity

We do not add data to the corpus until it passes three filters:

Language ID: confirmed Malayalam (not transliterated or code-switched)
Domain tag: labelled to one of our four domain categories
Quality score: grammatical completeness, sentence segmentation quality, and absence of OCR noise

At 47.5M tokens, we are nowhere near done. The goal is 200M tokens at comparable quality before we begin serious fine-tuning work. We estimate reaching that threshold in late 2026.

The Connection to Aerial Board

Aerial Board is our offline classroom AI for schools in Kerala, Tamil Nadu, and other regional language medium environments. It is the first product that will run on this corpus.

Aerial Board runs entirely on-device on commodity hardware (Raspberry Pi class), uses RUSTAMA for inference, and requires no internet connectivity. The Malayalam corpus is what makes the difference between a generic multilingual model that occasionally produces correct Malayalam and a model that actually sounds like a Kerala government school teacher.

Open Questions We're Working Through

License compatibility: Some SCERT materials are not clearly licensed for ML training. We are in conversations with the department.
Dialect coverage: Malabar Malayalam (northern dialect) is significantly underrepresented vs. Kottayam-Ernakulam dialect in print sources. We are actively sourcing northern Kerala materials.
OCR quality on historical text: Pre-1990 Malayalam print uses older typesetting conventions that modern OCR handles poorly. We are fine-tuning an OCR model specifically for this.

Updates will be posted here as the corpus grows. If you work in education, government, or media in Kerala and want to contribute or partner, get in touch.

Building the Malayalam Corpus: 47.5M Tokens and Why It Matters