publications

2023

1. NeurIPS

   ReHLine: Regularized Composite ReLU-ReHU Loss Minimization with Linear Computation and Linear Convergence

   Ben* Dai, and Yixuan* Qiu

   In Thirty-seventh Conference on Neural Information Processing Systems, 2023

   Abs PDF Slides

   Empirical risk minimization (ERM) is a crucial framework that offers a general approach to handling a broad range of machine learning tasks. In this paper, we propose a novel algorithm, called ReHLine, for minimizing a set of regularized ERMs with convex piecewise linear-quadratic loss functions and optional linear constraints. The proposed algorithm can effectively handle diverse combinations of loss functions, regularizations, and constraints, making it particularly well-suited for complex domain-specific problems. Examples of such problems include FairSVM, elastic net regularized quantile regression, Huber minimization, etc. In addition, ReHLine enjoys a provable linear convergence rate and exhibits a per-iteration computational complexity that scales linearly with the sample size. The algorithm is implemented with both Python and R interfaces, and its performance is benchmarked on various tasks and datasets. Our experimental results demonstrate that ReHLine significantly surpasses generic optimization solvers in terms of computational efficiency on large-scale datasets. Moreover, it also outperforms specialized solvers such as liblinear in SVM, hqreg in Huber minimization and lightning(SAGA, SAG, SDCA, SVRG) in smooth SVM, exhibiting exceptional flexibility and efficiency.

2. JMLR

   RankSEG: A Consistent Ranking-based Framework for Segmentation

   Ben Dai, and Chunlin Li

   Journal of Machine Learning Research, 2023

   Abs PDF Slides

   Segmentation has emerged as a fundamental field of computer vision and natural language processing, which assigns a label to every pixel/feature to extract regions of interest from an image/text. To evaluate the performance of segmentation, the Dice and IoU metrics are used to measure the degree of overlap between the ground truth and the predicted segmentation. In this paper, we establish a theoretical foundation of segmentation with respect to the Dice/IoU metrics, including the Bayes rule and Dice-/IoU-calibration, analogous to classification-calibration or Fisher consistency in classification. We prove that the existing thresholding-based framework with most operating losses are not consistent with respect to the Dice/IoU metrics, and thus may lead to a suboptimal solution. To address this pitfall, we propose a novel consistent ranking-based framework, namely RankDice/RankIoU, inspired by plug-in rules of the Bayes segmentation rule. Three numerical algorithms with GPU parallel execution are developed to implement the proposed framework in large-scale and high-dimensional segmentation. We study statistical properties of the proposed framework. We show it is Dice-/IoU-calibrated, and its excess risk bounds and the rate of convergence are also provided. The numerical effectiveness of RankDice/mRankDice is demonstrated in various simulated examples and Fine-annotated CityScapes, Pascal VOC and Kvasir-SEG datasets with state-of-the-art deep learning architectures. Python module and source code are available on Github at (https://github.com/statmlben/rankseg).

3. TMLR

   Supervised Knowledge May Hurt Novel Class Discovery Performance

   ZIYUN LI, Jona Otholt, Ben Dai, and 3 more authors

   Transactions on Machine Learning Research, 2023
4. AoAS

   Data-adaptive discriminative feature localization with statistically guaranteed interpretation

   Ben Dai, Xiaotong Shen, Lin Yee Chen, and 2 more authors

   The Annals of Applied Statistics, 2023

   Abs PDF Slides

   In explainable artificial intelligence, discriminative feature localization is critical to reveal a blackbox model’s decision-making process from raw data to prediction. In this article, we use two real datasets, the MNIST handwritten digits and MIT-BIH Electrocardiogram (ECG) signals, to motivate key characteristics of discriminative features, namely adaptiveness, predictive importance and effectiveness. Then, we develop a localization framework based on adversarial attacks to effectively localize discriminative features. In contrast to existing heuristic methods, we also provide a statistically guaranteed interpretability of the localized features by measuring a generalized partial R2. We apply the proposed method to the MNIST dataset and the MIT-BIH dataset with a convolutional auto-encoder. In the first, the compact image regions localized by the proposed method are visually appealing. Similarly, in the second, the identified ECG features are biologically plausible and consistent with cardiac electrophysiological principles while locating subtle anomalies in a QRS complex that may not be discernible by the naked eye. Overall, the proposed method compares favorably with state-of-the-art competitors. Accompanying this paper is a Python library dnn-locate (this https URL) that implements the proposed approach.

5. CVPR

   ImbaGCD: Imbalanced Generalized Category Discovery

   Ziyun Li, Ben Dai, Furkan Simsek, and 2 more authors

   In The 2nd Workshop on Computer Vision in the Wild, CVPR 2023, 2023

   PDF

2022

1. Optica

   Full Poincaré polarimetry enabled through physical inference

   Chao He, Jianyu Lin, Jintao Chang, and 8 more authors

   Optica, 2022

   arXiv PDF
2. IEEE TNNLS

   Significance tests of feature relevance for a black-box learner

   Ben Dai, Xiaotong Shen, and Wei Pan

   IEEE Transactions on Neural Networks and Learning Systems, 2022

   Abs PDF Slides

   An exciting recent development is the uptake of deep neural networks in many scientific fields, where the main objective is outcome prediction with a black-box nature. Significance testing is promising to address the black-box issue and explore novel scientific insights and interpretations of the decision-making process based on a deep learning model. However, testing for a neural network poses a challenge because of its black-box nature and unknown limiting distributions of parameter estimates while existing methods require strong assumptions or excessive computation. In this article, we derive one-split and two-split tests relaxing the assumptions and computational complexity of existing black-box tests and extending to examine the significance of a collection of features of interest in a dataset of possibly a complex type, such as an image. The one-split test estimates and evaluates a black-box model based on estimation and inference subsets through sample splitting and data perturbation. The two-split test further splits the inference subset into two but requires no perturbation. Also, we develop their combined versions by aggregating the p -values based on repeated sample splitting. By deflating the bias-sd-ratio, we establish asymptotic null distributions of the test statistics and the consistency in terms of Type 2 error. Numerically, we demonstrate the utility of the proposed tests on seven simulated examples and six real datasets. Accompanying this article is our python library dnn-inference (https://dnn-inference.readthedocs.io/en/latest/) that implements the proposed tests.

3. AP

   Revealing complex optical phenomena through vectorial metrics

   Chao He, Jintao Chang, Patrick S Salter, and 8 more authors

   Advanced Photonics, 2022
4. JASA

   Coupled Generation

   Ben Dai, Xiaotong Shen, and Wing Wong

   Journal of the American Statistical Association, 2022
5. JASA

   Embedding learning

   Ben Dai, Xiaotong Shen, and Junhui Wang

   Journal of the American Statistical Association, 2022

   Abs

   Numerical embedding has become one standard technique for processing and analyzing unstructured data that cannot be expressed in a predefined fashion. It stores the main characteristics of data by mapping it onto a numerical vector. An embedding is often unsupervised and constructed by transfer learning from large-scale unannotated data. Given an embedding, a downstream learning method, referred to as a two-stage method, is applicable to unstructured data. In this article, we introduce a novel framework of embedding learning to deliver a higher learning accuracy than the two-stage method while identifying an optimal learning-adaptive embedding. In particular, we propose a concept of U-minimal sufficient learning-adaptive embeddings, based on which we seek an optimal one to maximize the learning accuracy subject to an embedding constraint. Moreover, when specializing the general framework to classification, we derive a graph embedding classifier based on a hyperlink tensor representing multiple hypergraphs, directed or undirected, characterizing multi-way relations of unstructured data. Numerically, we design algorithms based on blockwise coordinate descent and projected gradient descent to implement linear and feed-forward neural network classifiers, respectively. Theoretically, we establish a learning theory to quantify the generalization error of the proposed method. Moreover, we show, in linear regression, that the one-hot encoder is more preferable among two-stage methods, yet its dimension restriction hinders its predictive performance. For a graph embedding classifier, the generalization error matches up to the standard fast rate or the parametric rate for linear or nonlinear classification. Finally, we demonstrate the utility of the classifiers on two benchmarks in grammatical classification and sentiment analysis. Supplementary materials for this article are available online.

2021

1. JASA

   Scalable collaborative ranking for personalized prediction

   Ben Dai, Xiaotong Shen, Junhui Wang, and 1 more author

   Journal of the American Statistical Association, 2021
2. EJS

   Two-level monotonic multistage recommender systems

   Ben Dai, Xiaotong Shen, and Wei Pan

   Electronic Journal of Statistics, 2021
3. EJS

   Sentiment analysis with covariate-assisted word embeddings

   Shirong Xu, Ben Dai, and Junhui Wang

   Electronic Journal of Statistics, 2021

2019

1. JMLR

   Smooth neighborhood recommender systems

   Ben Dai, Junhui Wang, Xiaotong Shen, and 1 more author

   Journal of Machine Learning Research, 2019
2. EJS

   Query-dependent ranking and its asymptotic properties

   Ben Dai, Junhui Wang, and  others

   Electronic Journal of Statistics, 2019
3. NC

   Complex vectorial optics through gradient index lens cascades

   Chao He, Jintao Chang, Qi Hu, and 8 more authors

   Nature Communications, 2019