Variational inference for probabilistic programs and generative models

Black-box stochastic variational methods have become one of the dominant methods for approximate inference and form the basis for amortized inference methods for structured generative models. One class of particularly structured models are probabilistic programs, which can model complex generative processes that include hierarchical structures and discrete latent variables such as stochastic control flow choices. For these models, standard variational inference methods often produce sub-par results, caused not by a lack of expressivity of the variational family but by the inability to escape local minima during optimization. Models with additional differentiability constraints, introduced by discrete components, calls to external simulators, or aforementioned stochastic control flow, can significantly add to those challenges by precluding the computation of gradients w.r.t. (a subset of) the latent variables of the model. As a result, many gradient-based optimization strategies, especially those relying on reparameterization, are no longer viable, and practitioners have to resort to less efficient methods that can suffer from prohibitively high variance. These difficulties compound considerably in scenarios with limited sampling budgets, imposed by computationally expensive models, where drawing more samples to reduce variance is no longer feasible. In this thesis, we develop ideas and techniques to address these limitations of structured and discrete latent variable models, which are encountered in probabilistic programming and generative modeling, and still present significant challenges for modern variational inference.