Viktoria Schram

Predicting model performance at larger scales enables the design of training strategies and architectures tailored to specific performance targets. Empirical scaling law research identifies functional forms to aid this prediction task. These describe the relationship between loss and compute using a loss-compute frontier defined by learning curves. Due to the empirical nature of this approach, the computational burden is substantial, making strategic resource allocation essential—yet it remains surprisingly underexplored. In this work, we address this shortcoming by exploring the suitability of Successive Halving (SH) and SH combined with parametric and non-parametric surrogate models. In addition to enabling a more systematic allocation of a given compute budget, our findings show that SH paired with surrogate models yields a set of learning curves that includes one with a lower loss-compute value than what naive uniform allocation or an SH-only approach can obtain. Our experiments demonstrate mean relative improvements of up to 2.84% and 5.47% on real-world and synthetic learning curve datasets. This strategic resource allocation enables us to obtain accurate scaling laws at significantly reduced computational costs, saving up to 98.7% over the traditional exhaustive approach.

The prediction of learning curves for Natural Language Processing (NLP) models enables informed decision-making to meet specific performance objectives, while reducing computational overhead and lowering the costs associated with dataset acquisition and curation. In this work, we formulate the prediction task as a multitask learning problem, where each task's data is modelled as being organized within a two-layer hierarchy. To model the shared information and dependencies across tasks and hierarchical levels, we employ latent variable multi-output Gaussian Processes, enabling to account for task correlations and supporting zero-shot prediction of learning curves (LCs). We demonstrate that this approach facilitates the development of probabilistic scaling laws at lower costs. Applying an active learning strategy, LCs can be queried to reduce predictive uncertainty and provide predictions close to ground truth scaling laws. We validate our framework on three small-scale NLP datasets with up to 30 LCs. These are obtained from nanoGPT models, from bilingual translation using mBART and Transformer models, and from multilingual translation using M2M100 models of varying sizes.

Performance prediction for Natural Language Processing (NLP) seeks to reduce the experimental burden resulting from the myriad of different evaluation scenarios, e.g., the combination of languages used in multilingual transfer. In this work, we explore the framework of Bayesian matrix factorisation for performance prediction, as many experimental settings in NLP can be naturally represented in matrix format. Our approach outperforms the state-of-the-art in several NLP benchmarks, including machine translation and cross-lingual entity linking. Furthermore, it also avoids hyperparameter tuning and is able to provide uncertainty estimates over predictions.

With the rapid expansion of wireless technologies, monitoring and regulating the Radio Frequency (RF) spectrum usage becomes more important than ever. In this paper, we present a Deep Learning (DL) based approach to analyze the RF spectrum by detecting, localizing, and classifying active signals in RF frequency bands. We represent the radio signals in wideband spectrograms and formulate the signal detection and classification problem as an object detection task related to the computer vision field. To this end, You Only Look Once (YOLO), a state-of-the-art object detector, is adapted and optimized to detect, localize, and classify signals in spectrograms. For the experimental evaluation of YOLO as a signal detector, a rich dataset was simulated, consisting of diverse signals modulated with digital and analog modulation schemes and transmitted over channels with realistic propagation conditions. Our proposed method achieves an Average Precision (AP) of almost 87% and an average Intersection over Union (IoU) of 90%, thus demonstrating significant potential for analyzing RF spectral activity with high accuracy.

Continuous research progress in THz communications including communications and networking paradigms as well as devices and implementation aspects indicate that this technology has the potential to become a key component in future beyond 5G systems. Still, some research challenges have to tackled before, e.g. related to the peculiarities of the highly frequency-selective THz channel resulting in a much longer channel impulse response (CIR) compared to channels in the lower frequency bands. In this paper, first, various equalization techniques such as minimum mean-squared error (MMSE) linear equalization and decision-feedback equalization (DFE) are studied for single-carrier transmission in order to combat the frequency selectivity of a deterministic THz channel. Furthermore, orthogonal frequency-division multiplexing (OFDM) communication with zero-forcing (ZF) equalization is considered, and the performance of single-carrier and multi-carrier transmissions are compared. Secondly, the advantages of a stochastic channel model are pointed out for the analysis of a multiple-input multiple-output (MIMO) transmission, and ZF and MMSE linear frequency-domain equalization are investigated for a single-carrier frequency-division multiple access (SC-FDMA) transmission over a MIMO THz channel. We conclude that SC-FDMA along with MMSE linear equalization is promising for practical THz systems.

Compressed sensing (CS) deals with the problem of reconstructing a sparse vector from an under-determined set of observations. Approximate message passing (AMP) is a technique used in CS based on iterative thresholding and inspired by belief propagation in graphical models. Due to the high transmission rate and a high molecular absorption, spreading loss and reflection loss, the discrete-time channel impulse response (CIR) of a typical indoor THz channel is very long and exhibits an approximately sparse characteristic. In this paper, we develop AMP based channel estimation algorithms for indoor THz communications. The performance of these algorithms is compared to the state of the art. We apply AMP with soft- and hard-thresholding. Unlike the common applications in which AMP with hard-thresholding diverges, the properties of the THz channel favor this approach. It is shown that THz channel estimation via hard-thresholding AMP outperforms all previously proposed methods and approaches the oracle based performance closely.

The theory of finite blocklength coding provides powerful tools to approximate the maximum achievable rate for short blocklength coding. In this work, the normal approximation is applied for the transmission over the THz channel assuming a point-to-point single-carrier system. Here, a transmission with bit interleaved coded modulation (BICM) is assumed and compared to coded modulation (CM). The performance is analysed for higher order modulations considering PSK and QAM. These results are verified by simulations, considering terminated convolutional codes with different coding rates, minimum mean-squared error linear equalization (MMSE-LE) and soft decision decoding. Our simulation results show that a rate of about 100 Gbps is achievable for 32QAM with a blocklength of 100 and a bit error rate of 1e-3.

Terahertz (THz) communication is a new emerging technology which has the potential to satisfy the steadily growing demands for high data rates. However, transmission over the THz channel is a difficult task since perfect channel state information (CSI) is not available in real systems. Due to the high transmission rate and a high molecular absorption, spreading loss and reflection loss, the discrete-time channel impulse response (CIR) of the THz channel is very long and exhibits an approximately sparse characteristic. Conventional least-squares (LS) channel estimation does not incorporate the sparsity assumption into the estimation process. Therefore, in this work, sparse channel estimation for an indoor THz transmission example scenario using compressive sensing (CS) techniques is analyzed. A CS method based on solving a convex program by using the Dantzig selector (DS) and a CS approach using a greedy pursuit method called compressive sampling matching pursuit (CoSaMP) are investigated. All methods are analyzed with respect to mean squared error (MSE) performance, computational efficiency and numbers of observations needed. The numerical results show that significant advantages over LS estimation are achievable in all categories.