Bayesian theory with applications, spring 2010

Advanced studies. Bayesian theory is currently applied throughout the whole spectrum of scientific modeling and it is also a very important tool in a multitude of technological and engineering fields. The aims of the course are to decipher the Bayesian machinery, how and why it works, as well as to gain detailed understanding of an array of its applications.

14

15

=== Prerequisites ===

16

17

Probability calculus, calculus, linear algebra are important pre-requisites. Stochastic processes and computational statistics are useful, but not obligatory.

=== Lectures ===

see main page.

=== Lecture diary (only tentative schedule). ===

26

27

Week 11: [[Recent article in NY Times about Bayesian statistics>>url:http://www.nytimes.com/2014/09/30/science/the-odds-continually-updated.html?action=click&pgtype=Homepage&region||shape="rect"]], [[Course introduction>>url:http://www.helsinki.fi/bsg/filer/BTintro.pdf||shape="rect"]], Introduction to subjective and epistemic perspective on probability, see [[Stanford Encyclopedia on probability>>url:http://plato.stanford.edu/entries/probability-interpret/||shape="rect"]], [[Bayes' theorem>>url:http://www.helsinki.fi/bsg/filer/Introduction2BayesTheorem.pdf||shape="rect"]], dynamic revision of uncertainty using Bayes' theorem; see [[the example on perception and sensory integration>>url:http://www.helsinki.fi/bsg/filer/AistiHavaintojenIntegrointi.pdf||shape="rect"]] which is demonstrated live in [[this BBC clip>>url:http://www.youtube.com/watch?v=G-lN8vWm3m0||shape="rect"]], [[Search & Rescue game sw>>url:http://archives.math.utk.edu/software/msdos/probability/bayes/bayes.zip||shape="rect"]], Search and Rescue [[demo case as a pdf>>url:http://www.helsinki.fi/bsg/filer/SearchAndRescue.pdf||shape="rect"]], (note also that there is [[a real Bayesian search & rescue sw>>url:http://www.uscg.mil/acquisition/international/sarops.asp||shape="rect"]] in use by coast guards, see [[here for a SAROPS demo>>url:http://www.ifremer.fr/web-com/stw2004/sar/pdf/spaulding_ppt.pdf||shape="rect"]]), [[sequential Monte Carlo related computation>>url:http://www.cs.ubc.ca/~~nando/smc/index.html||shape="rect"]]. Revision of uncertainty and predictions for a 'cigar-box sampling problem', usefulness of systematic use of prior information in the context of infant mortality and SIDS (see [[this article by Gilbert et al. 2005>>url:http://ije.oxfordjournals.org/cgi/reprint/dyi088v1.pdf||shape="rect"]]). Use of Bayesian statistics to locate a missing plane, [[an example of Air France flight 447 and its relevance to MH370 search>>url:http://www.bbc.com/news/magazine-26680633||shape="rect"]]. [[This paper >>url:http://arxiv.org/abs/1405.4720||shape="rect"]]in Statistical Science discusses the AF447 case using SAROPS, and there are many other success stories listed [[here>>url:http://projecteuclid.org/euclid.ss/1399645719||shape="rect"]] (a special Bayes issue of Statistical Science, the papers are also available on arxiv.org). Discussion of Bayesian inference and likelihood ratio calculations in forensics, for DNA evidence see these excellent slides [[(1)>>url:http://pub.math.leidenuniv.nl/~~gillrd/teaching/graphical/FSandGMslides1.pdf||shape="rect"]] , [[(2)>>url:http://pub.math.leidenuniv.nl/~~gillrd/teaching/graphical/FSandGMslides3a.pdf||shape="rect"]] , [[(3)>>url:http://pub.math.leidenuniv.nl/~~gillrd/teaching/graphical/FSandGMslides2.pdf||shape="rect"]] and [[(4)>>url:http://pub.math.leidenuniv.nl/~~gillrd/teaching/graphical/FSandGMslides4.pdf||shape="rect"]] from Richard Gill's homepage, for gunshot residue analysis, see [[this paper>>url:http://onlinelibrary.wiley.com/doi/10.1111/1556-4029.12179/full||shape="rect"]] and [[these slides>>url:http://www.helsinki.fi/bsg/filer/EAFS2012Corander.pdf||shape="rect"]]. [[This paper>>url:http://arxiv.org/pdf/1302.4404.pdf||shape="rect"]] discusses more complicated evidence calculation in cases of mixture of DNA and [[this review>>url:http://www.annualreviews.org/doi/pdf/10.1146/annurev-statistics-022513-115602||shape="rect"]] by David Balding discusses several challenges in DNA based evidence.

28

29

Week 12: Exchangeability, de Finetti's representation theorem, subjective probability modeling, prior and posterior predictive distributions, illustrations with probabilistic classification of documents, [[SpamAssasin>>url:http://spamassassin.apache.org/||shape="rect"]] is the most widely used spam protection system with a Bayesian filter. Cormack and Lynam at U Waterloo [[present a nice study on the efficiency SpamAssasin and other filters>>url:http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.86.2190&rep=rep1&type=pdf||shape="rect"]], instead of modeling presence/absence of words, it is also possible to [[use data compression on text sequences for spam filtering>>url:http://machinelearning.wustl.edu/mlpapers/paper_files/BratkoCFLZ06.pdf||shape="rect"]]. In a more general setting, this[[ recent paper discusses predictive classification and exchangeability>>url:http://www.springerlink.com/content/v11673087354713x/||shape="rect"]], see also its [[sequel paper>>url:http://dx.doi.org/10.1016/j.jspi.2012.07.013||shape="rect"]] and [[these illustrative slides>>url:http://www.helsinki.fi/bsg/filer/OBrother2015.pdf||shape="rect"]] that summarize behavior of predictive inference under various classification circumstances. A February 2015 [[paper about predictive classifiers based on graphical models>>url:http://link.springer.com/article/10.1007/s11634-015-0199-5||shape="rect"]] illustrates several improvements over the previous approaches. A useful approach to calculating integrals in Bayesian inference via 'visual pattern recognition' is explained in [[this document>>url:http://web.abo.fi/fak/mnf//mate/jc/inferens/kernelformel.pdf||shape="rect"]]. A catalogue of conjugate prior distributions is [[here>>url:http://www.google.fi/url?sa=t&source=web&cd=1&ved=0CBoQFjAA&url=http%3A%2F%2Fciteseerx.ist.psu.edu%2Fviewdoc%2Fdownload%3Fdoi%3D10.1.1.157.5540%26rep%3Drep1%26type%3Dpdf&rct=j&q=compendium%20of%20conjugate%20priors&ei=tjWLTeK5GIbBswbKkPWICg&usg=AFQjCNEyQcx9zQ_zWmPED0fOwg0kErsMsw&cad=rja||shape="rect"]]. Gu, L. has provided these useful [[Notes on Dirichlet distribution with relatives>>url:http://www.cs.cmu.edu/%7Eepxing/Class/10701-08s/recitation/dirichlet.pdf||rel="nofollow" shape="rect" class="external-link"]] provides a concise recapitulation of some of the central formulas around the Dirichlet distribution.

30

31

Week 13: marginal and conditional independence, DAGs, graphs for representations of hierarchical models (see also [[this>>url:http://www.cs.berkeley.edu/~~jordan/papers/statsci.ps||shape="rect"]] introductory article by M Jordan), choosing prior distributions. [[Vanilla introduction to hierarchical models as a case study on kidney cancers>>url:http://web.abo.fi/fak/mnf//mate/jc/miscFiles/cancer%20The%20story.PDF||shape="rect"]] from the book by Gelman et al.

32

33

Week 14: kidney cancer story continued (with simulation in classroom), for a more realistic example of Bayesian smoothing of disease rates, see [[the excellent slides of Aki Vehtari>>url:http://becs.aalto.fi/en/research/bayes/publications/Vehtari_Liverpool2013.pdf||shape="rect"]], Bayesian inference procedures in practice, illustrating case-study with IQ estimation (with simulation in classroom), choosing priors continued, [[these slides>>url:http://www.helsinki.fi/bsg/filer/BASTAmodelillustration.pdf||shape="rect"]] and [[this article>>url:http://www.biomedcentral.com/content/pdf/1471-2105-10-90.pdf||shape="rect"]] illustrate the impact of different prior/model choices for clustering data of genomic aberrations observed in cancers.

34

35

(% class="external-link" %)

36

Week 15: more about priors, hierarchical models, partial exchangeability. We do several experiments related to specifying subjective probability intervals. In one of them participants collectively examine a glass jar with Euro coins (actually plastic pearls) and settled on a prior for the 'Number of Euros in the jar' problem, see [[this document>>url:http://web.abo.fi/fak/mnf//mate/jc/miscFiles/EurosInTheJar.pdf||shape="rect"]]. For a discussion about scoring probabilistic forecasts using Brier score, see [[this article>>url:http://journals.ametsoc.org/doi/pdf/10.1175/WAF1034.1||shape="rect"]]. For a discussion about combining expert information using probabilities, see [[this article>>url:http://www.sciencedirect.com/science/article/pii/0377221795002332||shape="rect"]]. Problems of getting reliable statements from experts are discussed [[here>>url:http://www.sequentialunmasking.org/su/Dror_Contextual_FSI_2006.pdf||shape="rect"]] and [[here>>url:http://www.aridgetoofar.com/documents/Dror_Why%20Experts%20Make%20Errors_2006-1.pdf||shape="rect"]]. Advanced hierarchical modeling: [[a solid frozen vanilla cracker example of a hierarchical model>>url:http://mbe.oxfordjournals.org/content/28/1/673.full.pdf+html||rel="nofollow" shape="rect" class="external-link"]] and [[a summary of it>>url:http://www.helsinki.fi/bsg/filer/CoranderMCMSki2014.pdf||shape="rect"]] (to appreciate the concept of genetic drift you may wish to watch this simple [[animation>>url:http://nortonbooks.com/college/biology/animations/ch16a01.htm||shape="rect"]]). [[Example of Bayesian meta-analysis>>url:http://www.helsinki.fi/bsg/filer/WoodworthMetaanalysis.pdf||rel="nofollow" shape="rect" class="external-link"]] from the biostatistics book of George Woodworth. Examples of [[hierarchical models in fisheries management>>url:http://arxiv.org/abs/1405.4696||shape="rect"]].

37

38

(% class="external-link" %)

39

Week 17-18: model selection issues, [[fair-coin and star-tree paradox>>url:http://mbe.oxfordjournals.org/content/24/8/1639.short||shape="rect"]], a [[review of information-theoretic criteria>>url:http://www.sal.ufl.edu/eel6935/2008/01311138_ModelOrderSelection_Stoica.pdf||rel="nofollow" shape="rect" class="external-link"]] for model selection, Bayes factor, see [[this>>url:http://www.jstor.org/stable/2291091||shape="rect"]] paper by Kass & Raftery (1995), Occam's razor - see a [[demo>>url:http://alumni.media.mit.edu/~~tpminka/statlearn/demo/||shape="rect"]], sampling from two cigar boxes and dynamically updating model uncertainty (classroom simulation), discussion of choosing priors by formal rules, see [[this>>url:http://www.jstor.org/stable/2291752||shape="rect"]] article in particular, asymptotic behavior of model selection

40

procedures, see [[this proof of asymptotic consistency for the discrete case>>url:http://www.helsinki.fi/bsg/filer/DiscreteProof.pdf||shape="rect"]]. Model selection under improper priors with fractional marginal likelihood (see course slides and these articles: [[paper1>>url:http://www.mattiasvillani.com/wp-content/uploads/2009/08/fracrankneerlandicafinal.pdf||rel="nofollow" shape="rect" class="external-link"]], [[paper2>>url:http://www.mattiasvillani.com/wp-content/uploads/2009/08/corandervillanijtsa1978.pdf||rel="nofollow" shape="rect" class="external-link"]], [[paper3>>url:http://onlinelibrary.wiley.com/doi/10.1111/j.1467-9469.2011.00785.x/full||rel="nofollow" shape="rect" class="external-link"]]), What is wrong with Bayes factors or posterior probabilities when null hypothesis must NOT be favored? For an answer see [[paper1>>url:http://www.tandfonline.com/doi/abs/10.1080/03610926.2012.745563#.UxyO3YWSe8x||shape="rect"]] and [[paper2>>url:http://onlinelibrary.wiley.com/doi/10.1002/cem.2566/abstract||shape="rect"]], Bayesian model averaging, [[see this article>>url:http://www.jstor.org/stable/2676803||shape="rect"]], asymptotic behavior of Bayesian inference, see also [[the free book by David MacKay>>url:http://www.inference.phy.cam.ac.uk/mackay/itila/||shape="rect"]], which contains chapters on Bayesian modeling and in particular a very nice discussion about Occham's razor principle. About ABC (approximate Bayesian computation) inference, see [[this introduction>>url:http://www.ploscompbiol.org/article/info%3Adoi%2F10.1371%2Fjournal.pcbi.1002803||rel="nofollow" shape="rect" class="external-link"]]. [[Finite mixture models and EM-algorithm>>url:http://www.helsinki.fi/bsg/filer/Koski_mixturesEM.pdf||rel="nofollow" shape="rect" class="external-link"]] from the [[HMM book>>url:http://www.amazon.co.uk/Hidden-Markov-Models-Bioinformatics-Koski/dp/B008KXGQBQ/ref=sr_1_4?ie=UTF8&qid=1362554925&sr=8-4||rel="nofollow" shape="rect" class="external-link"]] by prof Timo Koski at KTH. A nice tutorial on Bayesian non-parametric models is available [[here>>url:http://web.mit.edu/sjgershm/www/GershmanBlei12.pdf||rel="nofollow" shape="rect" class="external-link"]], see also [[these slides>>url:http://www.cs.ubbcluj.ro/%7Ecsatol/gep_tan/Bishop-CUED-2006.pdf||rel="nofollow" shape="rect" class="external-link"]] on mixture models by Christopher Bishop.

=== Exams ===

Written exam and assignments (weekly assignments downloadable from the main page).

45

46

See the course page for current year for the exam date. Participants are allowed to bring all the lecture and assignment materials with them to the exam.

47

48

A list of possible topics for a larger assignment task is available [[here>>url:http://web.abo.fi/fak/mnf//mate/jc/miscFiles/Projects%20for%20Bayesian%20theory%202010.pdf||shape="rect"]], choose freely one project from the list. Reports on the larger assignment can be produced by working alone or in pairs. In case you decide to do the project jointly with another participant, return only a single report with names of both participants. The reports should be returned within three months from the written exam date. By completing both the written exam and a larger assignment participants will gain 8 credits for the course. In case you wish to suggest own topic for a larger assignment, contact the lecturer.

=== Bibliography ===

Lecture slides are available [[here>>url:http://web.abo.fi/fak/mnf//mate/jc/miscFiles/BayesianTheory2010.pdf||shape="rect"]]. In addition several classroom demonstrations and various case study materials are considered. Examples of useful books on Bayesian theory and modeling are Bernardo & Smith (1994), O'Hagan (1994), Schervish (1995), Gelman et al. (2004).

|=(((

)))|=(((

)))|=(((