Drefs, Jakob
(2022)
Evolutionary Variational Optimization for Probabilistic Unsupervised Learning.
PhD, Universitätsbibliothek Oldenburg.
Abstract
Probabilistic generative modeling is a powerful machine learning paradigm suitable for a variety of tasks, such as missing value estimation, denoising, clustering, outlier detection, or compression. A key factor determining the efficiency and effectiveness of algorithms based on generative models is the internally used approximate inference scheme. This work focuses on inference and learning in generative models with binary latents and studies a novel approach that combines variational optimization of truncated posteriors with evolutionary algorithms. As a distinction to related Gaussian and factored (a.k.a., mean field) variational approximations, the use of truncated variational distributions allows for flexible adaptation to complex posterior landscapes with multiple modes and correlations. Through the application of evolutionary algorithms as integral part of the variational optimization scheme, the approach becomes generically applicable without the need for model-specific analytical derivations (‘black box’ inference). Distinctions to related ‘black box’ inference schemes include that the approach studied here does neither involve sampling approximations nor gradient-based optimization of variational parameters, thereby circumventing auxiliary mechanisms such as continuous relaxations of discrete distributions and variance reduction techniques required by stochastic gradient estimators. Compared to previous truncated variational approximation schemes, the approach studied has the distinguishing feature of guaranteeing a monotonic increase of a log-likelihood lower bound. The studies presented in this thesis consider applications of the novel evolutionary variational optimization to a variety of data models with a wide range of characteristics, including: binary and binary-continuous priors, binary and continuous noise models, shallow and deep model architectures, linear and non-linear latent interaction models including learnable and non-learnable non-linearities, and global and latent-specific variance encodings. When studying deep generative models, the evolutionary variational optimization is combined with automatic differentiation tools for model parameter optimization, thereby demonstrating the suitability of the investigated approach for ‘black box’ learning and inference. The numerical experiments presented focus on image patch modeling and provide extensive performance evaluations using established denoising and inpainting benchmarks that allow for comparison to a broad range of related methods. These investigations reveal that evolutionary variational optimization of expressive data models such as spike-and-slab sparse coding or variational autoencoders results in competitive performances in benchmark settings with only a single noisy image available for training. Analyses of the learned data representations show that the models use comparably sparse encodings in theses cases. Experiments with SARS-CoV-2 microscopy imaging data illustrate potential contributions that the developed methods may deliver in applications, for instance, possibly improving visualization of image details to potentially ease image interpretation. In general, this work highlights the importance of effective procedures for learning and inference in generative models, and opens the door to a variety of possible future research directions.
Actions (login required)
 |
View Item |
