Wiki source code of Intensive course on Bayesian inference
Last modified by corander@helsinki_fi on 2024/03/27 10:17
Show last authors
| author | version | line-number | content |
|---|---|---|---|
| 1 | = Intensive course on Bayesian inference for the Nordic biostatistics network, spring 2013 = | ||
| 2 | |||
| 3 | === Lecturer === | ||
| 4 | |||
| 5 | [[Jukka Corander>>url:http://www.helsinki.fi/bsg/||shape="rect"]] | ||
| 6 | |||
| 7 | === Scope === | ||
| 8 | |||
| 9 | 7.5 cu. This course is primarily intended for participants from the Nordic biostatistics network, excluding University of Helsinki. Statistics students at other Finnish universities are also welcome. Course explains the fundamental issues in Bayesian inference, role of prior probabilities, predictive modeling, hierarchical models, model selection, asymptotics. These concepts are also put into several application contexts to demonstrate how and why Bayesian inference works, what are the benefits and potential pitfalls. No prior knowledge about Bayesian inference is expected. The aim is to equip participants with good intuition on how the Bayesian machinery works, rather than focus on exact mathematical formalism. | ||
| 10 | |||
| 11 | === Type === | ||
| 12 | |||
| 13 | advanced level course. | ||
| 14 | |||
| 15 | === Prerequisites === | ||
| 16 | |||
| 17 | First course on calculus, linear algebra and probability. | ||
| 18 | |||
| 19 | === Lectures === | ||
| 20 | |||
| 21 | First part 4.3.-8.3. Every day 10-12, 13-16. Room B120 in the Exactum building. | ||
| 22 | |||
| 23 | Second part 13.5.-17.5. Every day 10-12, 13-16. Room C124 in the Exactum building. | ||
| 24 | |||
| 25 | === Exams === | ||
| 26 | |||
| 27 | The participants will have to solve exercises and do home projects to gain the credits from the course. Sets of exercises: [[1>>url:http://www.helsinki.fi/bsg/filer/BT2012excercises1.pdf||shape="rect"]],[[2>>url:http://www.helsinki.fi/bsg/filer/BT2012excercises2.pdf||shape="rect"]],[[3>>url:http://www.helsinki.fi/bsg/filer/BT2012excercises3.pdf||shape="rect"]],[[4>>url:http://www.helsinki.fi/bsg/filer/BT2012excercises4.pdf||shape="rect"]],[[5>>url:http://www.helsinki.fi/bsg/filer/BT2012excercises5.pdf||shape="rect"]], return solutions by email to the lecturer by May 31st. During the break between part 1 and 2 of the course, participants should familiarize themselves with the [[WinBUGS software>>url:http://www.mrc-bsu.cam.ac.uk/bugs/||shape="rect"]] for Bayesian inference (many tutorials available from the website, there is even a YouTube video introducing it). Using WinBUGS fit [[these models>>url:http://wiki.helsinki.fi/download/attachments/70230785/Examples.pdf?version=1&modificationDate=1347966935874&api=v2||shape="rect"]] from the Bayes course of Jukka Ranta. Summarize as joint report and send to the lecturer before the second part starts. The final part of examination involves doing any two out of these [[small project alternatives>>url:http://web.abo.fi/fak/mnf//mate/jc/miscFiles/Projects%20for%20Bayesian%20theory%202010.pdf||shape="rect"]] and sending the report back to the lecturer by August 31st. | ||
| 28 | |||
| 29 | === Registration === | ||
| 30 | |||
| 31 | To register for the course, send an email to the lecturer (first dot last at helsinki dot fi). | ||
| 32 | |||
| 33 | === Preliminary lecture diary - Part 1 === | ||
| 34 | |||
| 35 | Many of these [[lecture slides>>url:http://web.abo.fi/fak/mnf//mate/jc/miscFiles/BayesianTheory2010.pdf||rel="nofollow" shape="rect" class="external-link"]] will be used during the course. Introduction to subjective and epistemic vs physical perspective on probability, Bayes' theorem (see this simple [[eye-opener>>url:http://www.helsinki.fi/bsg/filer/SistersParadoxByBayes.pdf||rel="nofollow" shape="rect" class="external-link"]] on our perception of probabilities and information), dynamic revision of uncertainty using Bayes' theorem; see [[the example on perception and sensory integration>>url:http://www.helsinki.fi/bsg/filer/AistiHavaintojenIntegrointi.pdf||rel="nofollow" shape="rect" class="external-link"]], [[Search & Rescue game>>url:http://archives.math.utk.edu/software/msdos/probability/bayes/bayes.zip||rel="nofollow" shape="rect" class="external-link"]] and usefulness of systematic use of prior information in the context of infant mortality and SIDS (see [[this article by Gilbert et al. 2005>>url:http://ije.oxfordjournals.org/cgi/reprint/dyi088v1.pdf||rel="nofollow" shape="rect" class="external-link"]]). Potential consequences of seemingly innocent and vague priors are illustrated by [[this example>>url:http://www.helsinki.fi/bsg/filer/WhatCanPriorsCause.pdf||shape="rect"]]. | ||
| 36 | |||
| 37 | [[Hierachical models - a vanilla introduction>>url:http://web.abo.fi/fak/mnf//mate/jc/miscFiles/cancer%20The%20story.PDF||rel="nofollow" shape="rect" class="external-link"]], exchangeability, de Finetti's representation theorem, prior and posterior predictive distributions, illustrations with probabilistic classification of documents, see also the following two papers about predictive classification in addition to the lecture slides: [[paper 1>>url:http://dx.doi.org/10.1007/s11222-011-9291-7||shape="rect"]], [[paper 2>>url:http://dx.doi.org/10.1016/j.jspi.2012.07.013||shape="rect"]]. [[Example of inference and utility>>url:http://web.abo.fi/fak/mnf//mate/jc/miscFiles/EurosInTheJar.pdf||rel="nofollow" shape="rect" class="external-link"]]. More about hierarchical models - [[a solid frozen vanilla cracker example of a hierarchical model>>url:http://mbe.oxfordjournals.org/content/28/1/673.full.pdf+html||shape="rect"]]. The cracker example uses advanced importance sampling, a nice introduction to importance sampling can be found [[here>>url:http://ib.berkeley.edu/labs/slatkin/eriq/classes/guest_lect/mc_lecture_notes.pdf||shape="rect"]]. About ABC (approximate Bayesian computation) inference, see [[this introduction>>url:http://www.ploscompbiol.org/article/info%3Adoi%2F10.1371%2Fjournal.pcbi.1002803||shape="rect"]]. More details on choosing priors through formal rules are found in [[this review>>url:http://www.jstor.org/stable/10.2307/2291752||shape="rect"]]. Gu, L. [[Notes on Dirichlet distribution with relatives>>url:http://www.cs.cmu.edu/%7Eepxing/Class/10701-08s/recitation/dirichlet.pdf||rel="nofollow" shape="rect" class="external-link"]] provides a concise recapitulation of some of the central formulas around the Dirichlet distribution. [[Example of Bayesian meta-analysis>>url:http://www.helsinki.fi/bsg/filer/WoodworthMetaanalysis.pdf||shape="rect"]] from the biostatistics book of George Woodworth. [[Finite mixture models and EM-algorithm,>>url:http://www.helsinki.fi/bsg/filer/Koski_mixturesEM.pdf||shape="rect"]] [[Bayesian learning for Markov chains>>url:http://www.helsinki.fi/bsg/filer/Koski3.pdf||shape="rect"]], [[introduction to hidden Markov models>>url:http://www.helsinki.fi/bsg/filer/KoskiBookHMM1.pdf||shape="rect"]] (HMMs) and [[recursions for various posterior probabilities in HMMs>>url:http://www.helsinki.fi/bsg/filer/KoskiCh14.pdf||shape="rect"]], all from the [[HMM book>>url:http://www.amazon.co.uk/Hidden-Markov-Models-Bioinformatics-Koski/dp/B008KXGQBQ/ref=sr_1_4?ie=UTF8&qid=1362554925&sr=8-4||shape="rect"]] by prof Timo Koski at KTH.A biological example of the use of HMM is [[here>>url:http://nar.oxfordjournals.org/content/early/2011/11/07/nar.gkr928.full||shape="rect"]]. HMMs are also relevant for a multitude of engineering applications, such as dynamic tracking, an excellent technical review of this field by Arnaud Doucet is [[here>>url:http://www.cs.ubc.ca/~~arnaud/doucet_johansen_tutorialPF.pdf||shape="rect"]], another [[excellent review>>url:http://ais.informatik.uni-freiburg.de/teaching/ss10/robotics/slides/stachniss06phd-basictechniques.pdf||shape="rect"]] by Cyrill Stachniss and an excellent short introduction by Bryan Minor is [[here>>url:http://eecs.wsu.edu/~~cook/ml/presentations/bryan.pdf||shape="rect"]]. | ||
| 38 | |||
| 39 | A nice tutorial on Bayesian non-parametric models is available [[here>>url:http://web.mit.edu/sjgershm/www/GershmanBlei12.pdf||shape="rect"]], see also [[these slides>>url:http://www.cs.ubbcluj.ro/~~csatol/gep_tan/Bishop-CUED-2006.pdf||shape="rect"]] on mixture models by Christopher Bishop. | ||
| 40 | |||
| 41 | === Preliminary lecture diary - Part 2 === | ||
| 42 | |||
| 43 | Model comparison and selection, asymptotic behavior of model selection procedures, see [[this proof of asymptotic consistency for the discrete case>>url:http://www.helsinki.fi/bsg/filer/DiscreteProof.pdf||rel="nofollow" shape="rect" class="external-link"]] and [[this article>>url:http://mbe.oxfordjournals.org/content/24/8/1639.long||shape="rect"]] by Ziheng Yang, formal rules for choosing priors in the context of model comparison, Occham's razor principle, see this [[primer on Occham's razor and Bayesian model comparison for Markov chains>>url:http://www.helsinki.fi/bsg/filer/PrimerOnOccham.pdf||rel="nofollow" shape="rect" class="external-link"]], Occam's razor in curve fitting - a [[demo>>url:http://alumni.media.mit.edu/%7Etpminka/statlearn/demo/||rel="nofollow" shape="rect" class="external-link"]], a nice [[review of information-theoretic criteria>>url:http://www.sal.ufl.edu/eel6935/2008/01311138_ModelOrderSelection_Stoica.pdf||shape="rect"]] for model selection, a paper on [[cross-validation and predictive inference>>url:http://pdf.aminer.org/001/023/618/bayesian_model_assessment_and_comparison_using_cross_validation_predictive_densities.pdf||shape="rect"]], predictive evaluation of forecasts was considered in [[this article>>url:http://www.sciencedirect.com/science/article/pii/S1352231011010491||shape="rect"]], [[information-theoretic book by D MacKay where Ch 28 contains a detailed explanation of the Occham's razor principle and Bayesian model comparison>>url:http://www.inference.phy.cam.ac.uk/mackay/itila/||rel="nofollow" shape="rect" class="external-link"]], Bayesian learning of the order of a discrete-time Markov chain (see [[this excerpt>>url:http://www.helsinki.fi/bsg/filer/Koski3.pdf||rel="nofollow" shape="rect" class="external-link"]] from the book: Timo Koski. Hidden Markov models for bioinformatics. Kluwer, 2001). Model selection under improper priors with fractional marginal likelihood (see course slides and these articles: [[paper1>>url:http://www.mattiasvillani.com/wp-content/uploads/2009/08/fracrankneerlandicafinal.pdf||shape="rect"]], [[paper2>>url:http://www.mattiasvillani.com/wp-content/uploads/2009/08/corandervillanijtsa1978.pdf||shape="rect"]], [[paper3>>url:http://onlinelibrary.wiley.com/doi/10.1111/j.1467-9469.2011.00785.x/full||shape="rect"]]), model choice using [[Bayesian entropy criterion>>url:http://www.helsinki.fi/bsg/filer/JLLI.pdf||shape="rect"]], with application to [[structural learning of time-series dynamics>>url:http://link.springer.com/article/10.1007/s10994-009-5101-2||shape="rect"]], effects of proper priors in model selection, a case with [[clustering of cancer genome data>>url:http://www.biomedcentral.com/content/pdf/1471-2105-10-90.pdf||shape="rect"]]. What happens in Bayesian inference when null hypothesis should not be favored by Occham's razor? This is a general problem in forensics applications and a recent solution to it is presented (see paper: Blomstedt P, Corander J. (2012) Posterior predictive comparisons for the two-sample problem. Communications in Statistics – Theory and Methods, in press). Other forensics related problems that were discussed are [[crime linking>>url:http://www.helsinki.fi/bsg/filer/LCP%2dBehavioural%20Crime%20Linking%20Using%20Bayes%92%20Theorem.pdf||shape="rect"]] and GSR evidence (see Romain Gauriot, Lawrence Gunaratnam, Rossana Moroni, Tapani Reinikainen, Jukka Corander. (2012). Statistical Challenges in the Quantification of Gunshot Residue Evidence. Journal of Forensic Sciences, in press). For a general introduction to Bayesian networks and expert systems, see [[this paper>>url:http://www.jstor.org/stable/10.2307/2245959||shape="rect"]]. Introduction to [[minimum description length principle>>url:http://arxiv.org/pdf/math/0406077v1.pdf||shape="rect"]] and [[an application of it to clustering genetic data>>url:http://www.mdpi.com/1099-4300/12/5/1102/pdf||shape="rect"]]. [[Non-reversible MCMC>>url:http://link.springer.com/article/10.1007/s11222-006-9391-y||shape="rect"]] and its [[application to graphical model learning>>url:http://link.springer.com/article/10.1007/s10618-008-0099-9||shape="rect"]]. | ||
| 44 | |||
| 45 | === Bibliography === | ||
| 46 | |||
| 47 | Examples of useful books on Bayesian theory and modeling are Bernardo & Smith (1994), O'Hagan (1994), Schervish (1995), Gelman et al. (2004), see also the lecture slides collection. | ||
| 48 | |||
| 49 |