Pretraining and semantics matter
Reinitializing attention and FFN weights substantially hurts performance. Replacing merchant names with opaque IDs also creates a persistent performance gap.
Can a small language model outperform competitive tabular classifiers and a production-grade deep learning model? Our results say yes.
Different signals. Stronger together.Ensembling adds 7.6 percentage points over VRLLM alone.
Large language models understand text. Can that knowledge translate into better predictions on industrial tabular data?
We fine-tuned Llama 3.2 1B on billions of labeled payment transactions, retaining its native tokenizer, vocabulary, and language modeling head. The result is VRLLM: the Visa Research LLM.
On our temporal holdout test set, this compact model outperforms strong tree-based, neural, and tabular foundation model baselines—and edges ahead of our purpose-built production classifier.
View the technical report on arXivUse next-token prediction to produce a continuous classification score. A familiar language-model interface becomes a compact decision model.
A compact LLM, tested on internal Visa payment transaction data. Every result below uses the same large-scale temporal holdout test set.
Internal data, offline evaluation. OFPR is a proprietary operating false-positive rate. Tabular foundation models use an 800K-example support set; the other individual models are task-trained or fine-tuned. Absolute metrics on this curated test do not represent live production performance.
Source notes. Links identify the checkpoints or implementations listed in the report. DCNv2 was reproduced from its architecture paper. The production model is proprietary, with no public model source.
Trained on the same features, the two models still discover different predictive patterns. Averaging their scores improves both recall and PR-AUC.
Keep the language model’s native interface. Adapt the data, the training process, and the readout for tabular classification.
Preserve meaningful text, keep field order fixed, and turn each record into a 600–1,300-token sequence.
Synthetic fields shown for illustrationTrain only on the final 0 or 1 label token. LoRA adapts attention and FFN projections with 1.8% trainable parameters.
Frozen embeddings and language modeling headRenormalize the 0 and 1 token logits, then apply business-specific calibration. No free-form text generation is needed.
A bounded, single-score outputRich field descriptions improve data efficiency with limited training examples. At the largest training scale, the shorter value-only template performs best.
Explore the three serialization styles. These synthetic examples illustrate the approach; they contain no production data.
Northstar Cafe | 24.50 | restaurant Label:
Compact, fixed-order values minimize token overhead. This template performs best at the largest training scale in our experiments.
Reinitializing attention and FFN weights substantially hurts performance. Replacing merchant names with opaque IDs also creates a persistent performance gap.
Training begins with a batch size of 32. After approximately 100M unique records, gradient accumulation increases the effective batch size to 512, balancing stable minority-class signals with update frequency.
Fixed input formatting, padding, GPU kernels, and software and hardware configurations help maintain deterministic scores. Business-specific probability calibration is applied before evaluation.
A strong prediction is a starting point. We look inside VRLLM to understand the learned priors, the components it relies on, and its ability to generalize.
Ablating either layer 0 or layer 1 causes linear-probe accuracy to plateau early, well below the unablated model. Other individual FFN layers and attention heads have much smaller effects.
The intervention supports the importance of these components for linearly decodable task information. It does not establish exactly what computation they perform.
Swapping a low-rate code for a high-rate code raises predicted probability in both classes—consistent with a learned, per-code lookup.
Higher amount buckets raise predicted probability even with similar historical positive rates. Pretraining is one possible source of this prior.
Feature swaps measure model sensitivity. They can produce off-manifold inputs and do not establish real-world causal effects.
VRLLM loses less recall than the production model on an unseen-merchant population.
Recall is normalized to each model’s own transaction-day-matched control. Both models also follow similar performance trajectories over eight consecutive evaluation windows.
Seventeen LLMs and decision models. Ten thousand internal transactions. A closer look at what improves—and what still limits zero-shot decisions.
Results use the model versions evaluated in the report. Zero-shot evaluation is shown separately from the large-scale, task-trained benchmark above.
Every frontier model reaches zero recall at this stringent operating point. Better overall ranking has yet to meet the task’s production requirements.
PR-AUC (%) in this zero-shot experiment. GPT-6-Astra leads the reported benchmark.
The three top-ranked models produce mean scores near the sample’s 10% positive rate. Lower-ranked models can be badly shifted: Gemini-2.5-pro has a median score of 0.965.
Perplexity-Decider-v1.2 places 99.9% of scores in the narrow interval [0.59, 0.69].
GPT-6-Astra produces 266 distinct scores across 10K inputs; Claude-fable-5.1 produces 114. OpenAI Decisions API, Claude-sonnet-4, and Claude-opus-4.5 produce only 40–49.
Coarse score grids constrain ranking granularity, even when the underlying model can reason well.
GPT-6-Astra assigns average scores of 0.26 to positive transactions and 0.09 to negatives—a gap of 0.17. Claude-fable-5.1 and Gemini-4-argon each have a 0.15 gap.
Perplexity-Decider-v1.2 separates the classes by only 0.01 (0.61 vs. 0.60).
The three top-ranked models show stronger merchant-category priors and assign higher positive-class probabilities to larger-dollar transactions, echoing the amount-bucket pattern in VRLLM.
These are useful starting signals. Task-specific supervision remains necessary to capture the finer predictive patterns in this study.
GPT-6-Astra, Claude-fable-5.1, Claude-opus-4.5, Kimi-K3, and Grok-4.7 form a tightly coupled prediction cluster, with rank correlations of 0.63–0.86.
Gemini-4-argon stands apart: its mean correlation with the other listed models is only 0.26; its maximum is 0.44, with GPT-6-Astra.
Training recipes and calibration strategies are possible directions for investigation. The correlation analysis describes behavior; it does not establish the cause.
We use the semantically rich serialization template on 10,000 transactions sampled from the internal temporal holdout, with a 1:9 positive-to-negative ratio. Frontier LLMs are prompted to return scores between 1 and 1,000.
Perplexity-Decider-v1.2 ↗ is hosted on-premises; its class logits are converted into probabilities. Other models are accessed through secure enterprise APIs. The OpenAI Decisions API ↗ returns a positive-outcome probability directly. Scores and probabilities are converted and calibrated before metric computation.
Dive into the experiments, training strategies, frontier benchmarks, and interpretability analysis. We welcome perspectives from researchers and practitioners in LLMs, tabular learning, and decision intelligence.
All trademarks are the property of their respective owners, are used for identification purposes only, and do not necessarily imply product endorsement or affiliation with Visa. Performance results are limited to the specific research task, evaluation dataset, model versions, and experimental settings described in this paper. They should not be interpreted as statements about the general capabilities, commercial suitability, or relative market standing of the referenced models, products, or organizations. To comply with legal and privacy policies, we do not release the dataset or any model trained on it.
Preprint. Work in progress.