Best Loss Function for Tabular Regression: Study Findings

👁 113 reads

A research paper published in October 2026 by Danil Rudenko, Georgii Bulgakov, Nikolai Kartashev, Ivan Rubachev, Daniil Krasylnikov, and Artem Babenko takes a hard look at objective functions for deep regression on tabular data. Their experimental code is publicly accessible via the link provided in the paper.

The Problem: No Rigorous Comparison of Regression Losses

Neural networks applied to tabular data remain an active research field, yet the choice of objective function in regression tasks has long been - as the authors state - an under-explored degree of freedom. While multiple research directions have proposed alternatives to standard Mean Squared Error (MSE), conflicting datasets, varying architectures, and inconsistent evaluation setups made comparing practical gains nearly impossible. The authors set out to close that gap.

What the Authors Evaluated

The team conducted a systematic re-evaluation of regression objectives. They tested candidate losses across an extensive collection of established tabular datasets and across multiple distinct neural network architectures (not explicitly listed by name in the document). Their primary baseline was a properly tuned MSE - the researchers emphasize that this reference point was carefully optimized rather than treated as an afterthought.

Key Benchmark Results

The findings are sobering for many popular proposals in the literature: most alternatives offer only limited or inconsistent gains over a properly tuned MSE baseline. The sole standout was quantile loss, which proved to be the only objective delivering consistent aggregate improvements over MSE across the evaluated architectures.

A Hidden Limit: Global Early Stopping

Even with an improved objective function, the researchers uncovered an underlying structural issue: early stopping. In standard workflows, training halts as soon as aggregate validation metrics plateau. However, the authors observe that distinct regions across the data - characterized by varying noise levels in the target variable - reach optimal predictive accuracy at different training epochs. A single global early stopping checkpoint inevitably forces a compromise across these regions, even when using an objective capable of modeling heteroskedastic target distributions. In short: one unified stopping point cannot fit feature spaces where different subsets converge at different speeds.

The Proposed Workaround

To address this imbalance, the authors introduce a heteroskedasticity-aware extension of quantile loss. The method relies on input-dependent label smoothing. Described by the researchers as a potential remedy rather than a silver bullet, this approach delivered measurable performance gains over standard quantile loss across several evaluated architectures.

What the Paper Omits

The abstract leaves out precise figures detailing the exact magnitude of performance gains, such as percentage drops in prediction error or the exact count of benchmark datasets involved. It also does not explicitly name the specific neural network architectures or the underperforming loss functions tested during the evaluation.

Why This Matters

In our view, this research serves as a valuable reality check against reflexively switching to trendy loss functions - most simply do not outperform a properly tuned MSE baseline. The insight into early stopping is arguably more impactful: this limitation quietly affects anyone training tabular deep learning models, yet it is rarely addressed directly. While the paper does not reinvent tabular deep learning, it cuts through literature hype and highlights a genuine structural bottleneck in common training pipelines.

Source: arXiv cs.AI: Revisiting Tabular Deep Regression: Objectives and the Limits of Global Early Stopping (published October 12, 2026). This analysis is based directly on the original document text.

// read moreBest AI for Google Sheets: Gemini and the =AI() FunctionAI ToolsFunction Calling in AI: How Models Call Your FunctionsGuidesYann LeCun: Turing Award Winner and LLM SkepticMagazine

Kinetyra