IndicJevBench: Measuring Decision AI in Indian Languages
IndicJevBench is the first open benchmark for Indic-language structured decision models: 69,402 items across seven languages, four enterprise domains, and a composite score that penalizes overconfidence as much as wrong answers.
The first open benchmark for Indic-language structured decision models. 69,402 items across seven languages, four enterprise domains, and a composite score that penalizes overconfidence as much as wrong answers.
The Gap
India’s enterprise AI stack runs on models trained mostly on English text and evaluated on English benchmarks. These models are deployed into systems where the user writes in Hindi, Tamil, Bengali, or a mix of all three.
They are not asked to generate text. They are asked to decide: route this support ticket, score this complaint’s urgency, flag this UPI transaction for review. The output the system needs is a probability, not a sentence. The right failure mode is a wide distribution, not a confident wrong answer.
No public benchmark measured whether models can do this in Indian languages. IndicJevBench does.
Three Kinds of Questions
Every benchmark item is a (state, question) pair. The state is a customer message in one of seven languages. The question is typed:
| Type | Example | Output |
|---|---|---|
choice | What is the intent of this message? | Probability per option (2–77 options) |
score | Rate urgency from 1–5 | Distribution + expected level |
noul | Should this be escalated? | P(true) |
A single message can carry more than one question: intent and urgency and escalation in the same pass.
What’s in the Benchmark
Four tasks, 69,402 items, all CC BY 4.0.
intent_massive: 57,922 items. Six native-script languages (Hindi in Devanagari, Bengali, Tamil, Telugu, Kannada, Malayalam), 60 intent classes. Drawn directly from the MASSIVE test split; no relabeling.
fintech_banking77: 7,816 items. The Banking77 dataset translated from English into six Indic languages using NLLB-200. 77 banking intent classes. Labels are unchanged by translation; 130 rows with ChrF < 10 were dropped. The domain is UK retail banking, so UPI and NEFT intents are absent. We plan to close this gap in v2.
hinglish_lid: 3,220 items. Hinglish language identification from COMI-LINGUA (IIT Gandhinagar). Roman-script code-mixed Hindi, Twitter register. The only Hinglish task in this release.
synthetic_enterprise: 444 items. UPI payments, Indian e-commerce, and customer-support routing in Hindi and Hinglish. Generated by a two-model pipeline: Qwen3-235B writes the customer message and predicts the label; DeepSeek-Chat-V3 independently verifies. Only items where both models agreed were kept. A sample of 100 was hand-reviewed before inclusion.
The test split was checksummed, frozen, and never touched again. No training or calibration data enters this repository.
The Score
IndicJevScore (0–100) is a weighted geometric mean across four axes:
| Axis | Weight | What it captures |
|---|---|---|
| Intelligence | 35% | Accuracy: is the model right? |
| Calibration | 25% | ECE + Brier: are stated confidences accurate? |
| Speed | 20% | Latency on a log scale: 50ms = 100, 5s = 0 |
| Cost | 20% | Per-1k inference cost on a log scale; local models score 100 |
Because it’s a geometric mean, a zero on any axis drives the composite toward zero. A model that is accurate but overconfident loses on calibration. A correct model with 16-second latency loses on speed. This is intentional: in production, all four matter.
Per-task metrics additionally include macro-F1, NLL, Brier score, ECE (15 bins), MAE of expected level for score tasks, and automatable share (the fraction of decisions passable at under 5% error). Everything breaks down by language, source dataset, and question type.
What the Baselines Show
We ran five models on 1,000 items per dataset (2026-09-22):
| Model | Type | fintech | synthetic | intent | hinglish_lid |
|---|---|---|---|---|---|
| OpenJev (Qwen3.5-4B) | local | acc=0.620 / IJ=78.3 | acc=0.606 / IJ=79.6 | — (16-opt limit) | acc=0.998 / IJ=87.2 |
| GPT-4o-mini | API | acc=0.775 / IJ=56.1 | acc=0.670 / IJ=46.6 | acc=0.580 / IJ=51.3 | acc=0.175 / IJ=21.2‡ |
| Laya-multilingual | local | acc=0.455 / IJ=69.3 | acc=0.351 / IJ=58.7 | acc=0.350 / IJ=61.7 | acc=0.493 / IJ=73.7 |
| Qwen3-4B (logprob) | local | acc=0.373 / IJ=55.3 | acc=0.525 / IJ=68.5 | acc=0.316 / IJ=49.6 | acc=0.840 / IJ=2.3† |
| Laya (base) | local | acc=0.256 / IJ=56.2 | acc=0.369 / IJ=64.5 | acc=0.199 / IJ=50.2 | acc=0.569 / IJ=74.0 |
Three things stand out.
GPT-4o-mini has the highest raw accuracy on structured Indic tasks: 77.5% on fintech, 67% on synthetic enterprise. It still loses the composite to local models because API latency (~2s/call) and per-call cost crush the speed and cost axes.
OpenJev wins the composite on every task it can run. It runs locally (cost = 0), and is calibrated reasonably well. Its hard limit of 16 options means it cannot run intent_massive, which has 60 classes.
Hinglish is where everything falls apart for general models. GPT-4o-mini scores 17.5% accuracy on hinglish_lid. This is the one language no public training corpus covers well, and the results confirm it.
† Qwen3-4B logprob scoring runs one forward pass per option on hinglish_lid, approximately 16 seconds per item, collapsing the speed axis to near zero.
‡ GPT-4o-mini on hinglish_lid had 46 JSON parse failures in 200 items (pre-fix run). Accuracy and IJ score are underestimates.
On Calibration
Accuracy is the easy number to report. Calibration is the number that determines whether a model can be used in production.
A model that says “70% likely this is a payment complaint” should be right 70% of the time it says that. None of the models we tested come close to this. ECE scores are poor across the board.
For agent routing and payment flagging, overconfidence means missed escalations and wrongly auto-approved transactions. IndicJevBench makes this cost visible. The calibration axis, weighted at 25%, means a model cannot hide behind high accuracy. A weak calibration score drags down an otherwise strong result.
A model can win an axis and still lose the composite — an accurate but overconfident model, or a fast but wrong one, scores poorly. This is by design.
What’s Missing
Marathi and Gujarati are absent. MASSIVE, the main source for intent tasks, did not include these in the mirror we used. v2 will add them.
India-specific banking intents (UPI transfers, FASTag recharge, NEFT status queries) are not in fintech_banking77, which derives from UK Banking77. The synthetic task partially covers this. Dedicated tasks are planned.
Non-Hindi Hinglish. Tamil-English and Bengali-English code-mixing are absent. COMI-LINGUA covers only Hindi-English.
We release these gaps explicitly. v2 timelines depend on finding or generating CC BY 4.0 data for each.
One note on datasets/v1/hinglish_lid.jsonl: the options field is serialized as a stringified Python repr rather than a plain JSON array. The dataset is frozen for reproducibility and all published results were computed against these files. v2 will re-emit clean option arrays alongside re-verified baselines.
Where This Fits
Any Indic JEV model that returns calibrated probability distributions for typed questions in Indian languages needs a benchmark to measure against. IndicJevBench is that benchmark. It is model-agnostic: it works against any server implementing the /v1/systemone contract, whether that is a local checkpoint, a hosted endpoint, or any OpenAI-compatible API.
Get It
git clone https://github.com/CMUL8/IndicJevBench
cd IndicJevBench
uv sync --extra dev
uv run pytest -q
To run against your own model, implement one method, decide(self, task: Task) -> DecisionResult, and point the harness at it. The adapter contract is documented in docs/ADAPTERS.md. The leaderboard is live on HuggingFace at cmul8-hf/IndicJevBench.
View on GitHub · Leaderboard on HuggingFace
IndicJevBench v1.0.0 · MIT license · Datasets: CC BY 4.0 · September 2026