Kirill Skobelev

Recent work by Doshi and Hauser (2024), Bisbee et al. (2024), and Xie et al. (2026) raises concerns that outputs from large language models (LLMs) tend to be under-diverse: they repeat or resemble one another more often than responses from the population they are meant to represent, a phenomenon known as mode collapse. In this work, we show that whether mode-collapse, or its opposite, occurs depends on the specific model and dataset used. Further, with sufficient supervised fine-tuning (SFT) data, LLM output diversity converges toward that of the target distribution from which fine-tuning data are sampled. To quantify this comparison, we measure the probability that two responses sampled independently from the same fixed prompt coincide (collide), or their expected similarity under a kernel. We derive a bias-variance decomposition of the expected gap between the model's and target's collision probabilities, showing that SFT is not inherently biased toward mode collapse or its opposite: finite-sample SFT can leave a model either under- or over-dispersed, depending on the model and dataset. Finally, we show that the absolute gap is bounded by the square root of the Kullback-Leibler (KL) divergence from the target distribution to the model. Consequently, a model sufficiently close to optimal under population cross-entropy cannot exhibit arbitrarily miscalibrated diversity. We test the decomposition and the bound in three experiments: small transformers on synthetic languages, four LLMs fine-tuned on human surveys, and these LLMs fine-tuned on CodeNet, a dataset of human code solutions. More target data moves model diversity toward the human (or synthetic target) level in all experiments, consistent with our theoretical predictions. These results show that diversity miscalibration can arise from finite-sample error and shrink as SFT better approximates the target distribution.

Recent Artificial Intelligence (AI) models have matched or exceeded human experts in several benchmarks of biomedical task performance, but have lagged behind on surgical image-analysis benchmarks. Since surgery requires integrating disparate tasks --- including multimodal data integration, human interaction, and physical effects --- generally-capable AI models could be particularly attractive as a collaborative tool if performance could be improved. On the one hand, the canonical approach of scaling architecture size and training data is attractive, especially since there are millions of hours of surgical video data generated per year. On the other hand, preparing surgical data for AI training requires significantly higher levels of professional expertise, and training on that data requires expensive computational resources. These trade-offs paint an uncertain picture of whether and to-what-extent modern AI could aid surgical practice. In this paper, we explore this question through a case study of surgical tool detection using state-of-the-art AI methods available in 2026. We demonstrate that even with multi-billion parameter models and extensive training, current Vision Language Models fall short in the seemingly simple task of tool detection in neurosurgery. Additionally, we show scaling experiments indicating that increasing model size and training time only leads to diminishing improvements in relevant performance metrics. Thus, our experiments suggest that current models could still face significant obstacles in surgical use cases. Moreover, some obstacles cannot be simply ``scaled away'' with additional compute and persist across diverse model architectures, raising the question of whether data and label availability are the only limiting factors. We discuss the main contributors to these constraints and advance potential solutions.

Large Language Models (LLMs) have become powerful tools for annotating unstructured data. However, most existing workflows rely on ad hoc scripts, making reproducibility, robustness, and systematic evaluation difficult. To address these challenges, we introduce DELM (Data Extraction with Language Models), an open-source Python toolkit designed for rapid experimental iteration of LLM-based data extraction pipelines and for quantifying the trade-offs between them. DELM minimizes boilerplate code and offers a modular framework with structured outputs, built-in validation, flexible data-loading and scoring strategies, and efficient batch processing. It also includes robust support for working with LLM APIs, featuring retry logic, result caching, detailed cost tracking, and comprehensive configuration management. We showcase DELM’s capabilities through two case studies: one featuring a novel prompt optimization algorithm, and another illustrating how DELM quantifies trade-offs between cost and coverage when selecting keywords to decide which paragraphs to pass to an LLM. DELM is available at \href{https://github.com/Center-for-Applied-AI/delm}{\texttt{github.com/Center-for-Applied-AI/delm}}.

U.S. bank regulators generally view their role as overseeing banks' exposure to climate-related risks rather than actively shaping climate policy. However, the boundary between risk oversight and policy advocacy is a gray area. We examine how U.S. bank regulators navigate this boundary by analyzing the rulemaking process for the 2023 interagency guidance on climate-related financial risk management. Our analysis consists of two parts. First, we examine the role of public pressure in shaping the final guidance. Using large language models (LLMs) to classify the public comments submitted on the draft, we find that the comments are highly polarized: the majority of individuals and climate advocacy groups support climate action by bank regulators, while banks and banking associations oppose it. We then construct textual alignment measures between each comment and the corresponding paragraphs in the draft and final guidance, and find that highly climate-engaged comments are less likely to be reflected in the final version, suggesting regulators discounted pro-climate advocacy. Second, we use LLMs to simulate regulators operating under different incentives. We implement a two-stage procedure in which the LLM first decides which public comments a regulator in a given role would accept, then revises the draft guidance using only those accepted comments. The evidence at the first stage is robust: both commercial and fine-tuned LLMs systematically accept or reject comments in line with their assigned role. In the second stage, we find suggestive evidence that regulators' revisions resemble those of a risk-averse bureaucrat most closely. However, the second stage outcomes are sensitive to the choice of model, echoing cautions in the literature on using LLMs as human surrogates.

This paper explores the use and interpretation of ML methods in asset-pricing. First, I propose a robust market anomaly test based on analyst errors. I derive a link between excess returns and market's errors in earnings forecasts relative to a conditionally optimally estimator. I then contrast a Gradient Boosted Decision Tree algorithm to IBES consensus forecasts to show that companies for which analyst predictions exceeded ML forecasts earned lower out-of-sample returns and vice versa, which is consistent with the theoretical result. Further, I identify behavioral causes of analyst errors: I illustrate that analyst errors are associated with earnings growth in previous quarters (negatively), with book values (positively), and with capital expenditures (positively), thereby explaining some pricing anomalies. Second, I explore the application of ML for studying analyst behavior. I find that they submitted more Net Income forecasts for companies with intrinsically more predictable earnings as measured by R-Squared of ML forecasts after controlling for size and reporting history length.

This study examines the Bitcoin mining industry. It proposes a novel approach rooted in behavioral patterns for estimating the structure of the equipment utilized in the Bitcoin network. This study models industry’s capital expenditures, aggregate efficiency of mining hardware, and aggregate miner profitability. It finds that profits cluster in time and are relatively low compared to pure Bitcoin returns. This study considers two Bitcoin valuation methods proposed earlier in the literature: the marginal cost of production approach and 51-percent attack arbitrage. The study finds that the marginal cost hypothesis is invalid, while 51-percent attack arbitrage theory is well-grounded. This study introduces the hashrate adjustment hypothesis, which states that miners in aggregate adjust network hashrate to their revenues, which are primarily a function of the price of Bitcoin. The study presents evidence in its support.