If you're interested in probabilistic programming and want something a little more hands-on, I recommend The Design and Implementation of Probabilistic Programming Languages http://dippl.org/ . It's an online course/textbook that gets you programming right away and makes the power of probabilistic programming immediately clear.

thanks for this. there is also Probabilistic Models of Cognition [0] by one of the authors. I wish however that they stuck to Church language [1]

[0] https://probmods.org/

[1] http://web.stanford.edu/~ngoodman/papers/POPL2013-abstract.p...

I too preferred church, though I understand why the authors of dippl chose a more popular language (javascript) for their book. Church (a dialect of Scheme, which is a dialect of Lisp) also ties probabilistic programming back to its intellectual roots in McCarthy's amb operator [1].

That said, you can get pretty far with probabilistic programming in any language with decent monad support [2]. I did most of my probabilistic programming work in Scala. (You lose the ability to do really fancy inference if you go the monad route, as you can't analyze the program structure, but a lot of the time, this is fine.)

[1] http://community.schemewiki.org/?amb

[2] http://mlg.eng.cam.ac.uk/pub/pdf/SciGhaGor15.pdf

I wish I could have Church as a Jupyter notebook/lab Kernel.

Would make it much easier to play around with the language when trying to wrap my mind around the church version of probmods.

Not exactly the same thing but try clojure with anglican in nextjournal to see if it fits your needs.

Last time I tried putting a lisp into jupyter as a playground for church, I realized that I was missing all the plotting features to really get anywhere and before I realized it, I had to stop my self from reimplementing church’s standard library.

I’ll have a look on google on the names you mentioned :)

In nextjournal you can plot clojure output trivially with plotly or vega-lite. It’s just a matter of adding metadata to your output, like this:

    ^{:nextjournal/viewer :plotly}
    {:data [{:x '(0 10 20 30 40)} 
            {:y '(5 10 15 20 25)}]
     :layout {:autosize false :width 800 :height 400 
              :xaxis1 {:title "year"} 
              :yaxis1 {:title "revenue"}}}

See the result here:

https://nextjournal.com/kommen/plotting-from-clojure

Regarding anglican, here is a link. It’s basically a church port:

https://probprog.github.io/anglican/

does this require stats/probability knowledge?

on that topic, can anyone recommend an online stats/probability course? I tried the coursera one by Sebastian Thrun and couldn't get far into it because the "TA" examples were unintelligible.

Stat110[0] (Harvard) on EdX! The professor, Joe Blitzstein, is incredible. Easily one of the best classes I have ever taken.

[0] https://www.edx.org/course/introduction-to-probability

for machine learning i can recommend the course by Philipp Henigg from Uni of Tuebingen

https://www.youtube.com/playlist?list=PL05umP7R6ij1tHaOFY96m...

There is also ProbLog (Probabilistic Prolog)[1][2] and even extension of it with a deep learning - DeepProbLog[3]. Personally, though, I hope that new ISO Prolog implementation in Rust, aiming for the performance, Scryer Prolog[4] will add the probabilistic capability [5].

[1] https://dtai.cs.kuleuven.be/problog/

[2] https://github.com/ML-KULeuven/problog

[3] https://github.com/ML-KULeuven/deepproblog

[4] http://github.com/mthom/scryer-prolog/

[5] https://github.com/mthom/scryer-prolog/issues/69

I like this passage: In this and the next two chapters of this introduction we will present the key ideas of probabilistic programming using a carefully designed first-order probabilistic programming language (FOPPL). The FOPPL includes most common features of programming languages, such as conditional statements (e.g. if), primitive operations (e.g. +,-, etc.), and user-defined functions. The restrictions that we impose are that functions must be first order, which is to say that functions cannot accept other functions as arguments, and that they cannot be recursive. These two restrictions result in a language where models describe distributions over a finite number of random variables. In terms of expressivity, this places the FOPPL on even footing with many existing languages and libraries for automating inference in graphical models with finite graphs.

This gives a nice picture of what's happening. At the same time, does this mean that in the end, you're basically operating on a single distribution with only a few canned global transformations?

Can someone please give an example of how this could be used in practice?

On the surface probability seems antithetical to the explicit well-defined determinism of programming.

The best example I've seen is the birthday paradox. How likely is it for 2 people in a N person class to have the same birthday? You can solve this deterministically using math pretty easily.

But if you modify the question to ask how likely it is for 3 people, or you add things like February 29th and leap years, or you add the fact that births are more likely during summer months, then it becomes extremely difficult to solve this deterministically. Instead, you run a Monte Carlo simulation to get approximate probabilities. This is much simpler to code and can be easily modified to fit new conditions.

A Monte Carlo simulation is probabilistic because you use random numbers in the simulation and you won't get a 100% perfect answer but you'll get close enough (and you can do some math to get error bounds).

It will also be more time consuming to compute, won't it?

You can generate a posterior distribution for any overdetermined problem. Basically, any scientific problem, as they must infer models inputs (parameteric or non-parametric) from observed model outputs (data).

Determinism is a nice illusion that quickly breaks down on real data. Scientific problems are non-unique and noise or inadequate models cause data and model to not mesh with one another. Deterministic answers have the property of being precisely wrong as opposed to mostly correct.

Rather than focus on the non deterministic random sampling, frame it as a programming language or library for manipulating and fitting statistical distributions.

An HTTP library is good for making web calls. A probabilistic program is good for estimating probabilities.

One example would be polling. You could write down a program that estimates votes from political polls. Then feed it polling data and get estimates of how people will vote.

It’s super interesting for combining scientific models with modern statistical computation in a flexible way

Also, randomized algorithms for all kinds of things are very useful in general-purpose computing

I wouldn’t classify floating point computation as deterministic

Floating point computation is fully deterministic, every time you do computation on floating point it will give the same results for the same inputs. What you probably meant is that it's not compatible with humans decimal model of floating point math.

Has probabilistic programming been shown to solve problems better than machine learning or deep learning approaches?

I remember it being pretty hyped 5-6 years ago …

I wouldn't contrast it with machine learning or deep learning. Probabilistic programming is focused on building languages or libraries that incorporate fundamental probabilistic building blocks, model building statements and inference strategies as first class citizens within the language. Its been remarkably successful, check out Stan, Pyro, PyMC, Tensorflow Probability, JAGS, BUGS and Turing as very successful projects that have been used to tackle challenging and diverse problems with probabilistic modeling . The more modern probabilistic programming languages are actually designed to incorporate the advances of deep learning by making it easy to embed ANNs into models by using them to parameterize random variables, Pyro and Tensorflow Probability are probably the best examples of this.

note that these are not exclusive. you could divide ML into a traditional statistical approach and a probabilistic one that is concerned with deriving the underlying probability distribution. probabilistic programming is kind of like a domain specific language for achieving this. there is also differential programming that works on the same principle. there are certainly industrial usages of this paradigm. look up pyro (http://pyro.ai/examples/intro_part_i.html) for ppl and jax (https://github.com/google/jax) for differential programming

It's the standard tool for building Bayesian statistical models, which aren't glamorously crushing SotA records on AI-like prediction tasks, but are valuable in both academia and industry.

Not an either-or situation. Many ML algorithms can be formulated in a probabilistic way, and PPLs can represent them. But the choice of the underlying inferential algorithm (MCMC, variational, etc) greatly affects the scale and type of solvable problems, so it's not just the language itself.

PPLs using MCMC (e.g., Stan and pymc) are first-choice tools for Bayesian inference on small to medium data analysis, which is lots more common than Google-scale data analysis.

My professor in school years ago was working on probabilistic hardware and showing pretty good thermal benefits. It’s a drastically different way to build software so I don’t know how commercially viable it’ll become until someone shows it has a drastically better power per watt ratio (like 1000x or 10000x, maybe higher to account for the cost of transitioning existing software)

I'd be interested in learning more about that if you could share a name of the professor or project?

Dr Todd L. Veldhuizen. I looked through his research and don’t see anything there so either I’m misremembering or the research hadn’t gone anywhere (or still incomplete).

On the fun side, in the book :

"Every little kid knows that even the slightest variation in the placement of a firecracker or the most seemingly minor imperfection of a glue joint will lead to dramatically different model airplane explosions."

I've never made my model airplane explode (after the many hours needed to build them).

But I burnt ants :-)

Anyone have a Haskell version? This or quantum computing seems like you could really just package up with some fancy monad, instead of just a single state it encapsulates a distribution, and you just apply abstract transformations ontop as usual.

What is the "hello world" version of PP(L) ?

nice! also interesting that they have chosen a lisp syntax for the book. note however that the book is written to be language agnostic

> It is a Lisp-like language which, by virtue of its syntactic simplicity, also makes for efficient and easy meta-programming, an approach many implementors will take. That said, the real substance of this book is language agnostic and the main points should be understood in this light.

is Probabilistic Programming the same thing as doing MCMC (Markov chain Monte Carlo)? How do these two ideas relate? Or is one a subset of the other?

Probabilistic programming uses computer science techniques to do automated statistical modeling. For example, imagine I have a coin, and I want to discover if it is biased, i.e. if it lands on heads more often than tails. In a probabilistic programming framework, I can express my model as a simple Bernoulli model, `x ~ Bernoulli(p)`, and then automatically estimate the bias parameter `p` given some data (do "inference").

You can easily do this calculation by hand or in Python, but this does not generalize to more complex real-world scenarios. For complex probabilistic models, we must rely on numerical approximations. MCMC is just one algorithm for doing this approximate inference. Another popular technique is called variational inference [2]. Another commenter mentioned HMC [3], which is just a specific instance of MCMC.

[1] https://mc-stan.org/

[2] https://arxiv.org/abs/1601.00670

[3] https://arxiv.org/abs/1206.1901

I am not an expert in this but I find the math behind HMC to be out of this world as described in this mindblowing twitter thread:

https://twitter.com/betanalpha/status/1234576972132626445

Probabilistic programming can be done via MCMC approaches, but you can also infer the necessary quantities by using variational inference (which approximates the distribution described by your program with something that's simpler and easier to estimate).

Basically probabilistic programming is a way of describing a distribution, and then MCMC is one way of inferring the quantities in that distribution.

I'd argue that probabilistic programming is a language or framework of programming that lets you easily build and then fit probabilistic models to estimate or predict things of interest.

If you're taking a Bayesian approach to statistical modelling and inference, then they're probably a fairly good tool to consider. With the Bayesian approach you're trying to compute some posterior probability distribution that summarises your prior information (this might capture domain knowledge, information from related studies) and information from observations.

There are different ways to compute a posterior distribution. In very simple or contrived cases you might be able to manually grind out an answer analytically with pen and paper and lots of algebra and integrals. But that isn't very efficient or scalable. Also, nice algebraic structure is very easily broken by small perturbations to the problem statement -- need to add a weird bit onto the model to capture some real world behaviour? Good chance that ruins your algebraic structure and previous analytic "attack".

MCMC can be used to estimate the integrals you need when computing a posterior distribution. MCMC isn't the only way to estimate or approximate these calculations -- e.g. another approach is variational inference where a bunch of approximations are introduced to replace the original calculation with an approximation that is easier to compute -- this likely introduces bias into the results but can give you something that can then be solved analytically or semi analytically (e.g. approximate everything as Gaussian distributions and a lot of integration collapses to efficiently computable algebraic identities).

Some probabilistic programming platforms like Stan let you define your probabilistic model and parameters and decouple it from the computational backend used to estimate the posterior distribution. E.g. in Stan you can switch the computational backend between MCMC (https://mc-stan.org/docs/2_18/stan-users-guide/sampling-diff...) and ADVI (auto-differentiation variational inference).

MCMC has practical problems in that it is only guaranteed to give you the correct (unbiased) estimate asymptotically, in the limit if you run it for an infinite amount of time. If you're trying to approximate the integral of a function that is very multi-modal -- where it would be difficult for a global optimisation algorithm to locate the global optima -- then MCMC will likely also struggle to produce a good estimate. MCMC is difficult to parallelise effectively as the algorithm is inherently like an iterative local search procedure -- the next state in the chain is some mutation of the previous state. You can run n MCMC chains in parallel from n different initial configurations, but it's not obvious that you'll get a better estimate from n short chains vs a single long chain -- the longer a chain runs, the more chance it has of being able to discover and explore higher probability (more realistic, more plausible) configurations of the parameter space.

MCMC isn't only used for probabilistic programming, you can apply it for other things. E.g. it gets used in material science to study statistical properties of molecular dynamics simulations etc.

I think, and this is WAY outside my area, is that Hamiltonian Monte Carlo (a Markov chain monte carlo method) is used. Beyond that and I'm lost.

NB updated as of two days ago, despite the 1809 title.

What's the point of this? Will it make you rich?

it can be used to build better statistical modelling and computational tools that might, in the future, enable scientists and researchers to more efficiently or effectively figure out statistical or causal relationships -- and in some niche applications, e.g. in pharma or finance or insurance or advertising, that information might be valuable and help some people get rich. but the people able to capture the most value from those developments are unlikely to be the researchers or the people building the software tools.