 |
ML Newsnew | past | best | rss | submit | about |
| Constrained Bayesian Optimization with Noisy Experiments (arxiv.org) | |
| 3 points by MrQuincle on Jun 24, 2017 | hide | past | pdf | 1 comment on HN | |
|
In plain words: A search method that picks which settings to test in noisy real-world experiments, scoring batches with an approximation that handles measurement error and limits. It beat search tools that assume clean measurements on noisy problems with limits, and tuned Facebook's ranking and compiler settings. AbstractRandomized experiments are the gold standard for evaluating the effects of changes to real-world systems. Data in these tests may be difficult to collect and outcomes may have high variance, resulting in potentially large measurement error. Bayesian optimization is a promising technique for efficiently optimizing multiple continuous parameters, but existing approaches degrade in performance when the noise level is high, limiting its applicability to many randomized experiments. We derive an expression for expected improvement under greedy batch optimization with noisy observations and noisy constraints, and develop a quasi-Monte Carlo approximation that allows it to be efficiently optimized. Simulations with synthetic functions show that optimization performance on noisy, constrained problems outperforms existing methods. We further demonstrate the effectiveness of the method with two real-world experiments conducted at Facebook: optimizing a ranking system, and optimizing server compiler flags. | |
|
add comment on HN
Also discussed: Jun 2017 (1 point, 0 comments)
|
1. A Bayesian approach for A/B testing is advocated where noise is handled properly (not through heuristics).
2. Computationally efficiency is addressed through quasi-Monte Carlo.
3. It's practical use is shown through a production ranking system and optimization of a web server.
+ The Expected Improvement acquisition function (Mockus, 1978, Jones, 1998) is used to perform Bayesian optimization of the hyperparameters of a particular model.
+ Intuitively, it is an iterative procedure that picks a new point to fit a function in a sequential manner using the previously picked points.
+ Aside, to play with Gaussian processes, see e.g. https://github.com/fmfn/BayesianOptimization. To summarize, rather than searching for random variables it's searching for random functions.
+ Aside, Bayesian optimization is interesting because it reduces the number of evaluations required and replaces it by assumptions in the form of a (Gaussian Process) prior. See article http://papers.nips.cc/paper/4522-practical-bayesian-optimiza... by Snoek et al where by the way duration of execution time is also taken into consideration.
+ Aside, Bayesian optimization (in this form) does not use derivatives.
+ Aside, rather than ordinary MCMC, the "fit" of the posterior distribution is used to inform the sequential sampler through the acquisition function about where to sample next. Note that Bayesian optimization can even be used for adaptive MCMC: http://proceedings.mlr.press/v22/mahendran12/mahendran12.pdf.
+ Normally the Expected Improvement acquisition function assumes that observations are not noisy. (The Gaussian process prior defines stochastic relationships between parameters, not observations). The author introduce the Noisy Expected Improvement acquisition function.
+ The Noisy Expected Improvement integral is cast to an integral over the unit cube to be able to perform quasi Monte Carlo.
+ Aside, if you never read on quasi-randomness, check https://en.wikipedia.org/wiki/Low-discrepancy_sequence. Intuitively, the proportion of points in an (s-)interval is proportional to the (s-dimensional measure) length of that interval. This can be grid-like and does not need to be uniform randomly distributed. Note, quasi-randomness is not pseudo-randomness!
+ 6.2 describes the application for optimization of the JIT compiler part of the HipHop Virtual Machine (HHMV) to translate php/Hack into x86 machine code. Tunable parameters are about hot/cold code paths to reduce cache misses, choice for which type of functions to inline. The described algorithm is subsequently used to A/B test compiler flags w.r.t. CPU time, memory usage, database fetches, etc.
+ Results were good enough to be integrated in the HHVM: http://hhvm.com/.
Really nice work!