Introduction to Apache Spark

Kevin Mader (4Quant and ETH Zurich)

The presentation given at the IBM Data Science Connect Meeting introducting Apache Spark with several examples from statistics, SQL, image processing, and graphs.

Kevin Mader is a lecturer in the X-ray Microscopy Group within the Department for Information Technology and Electrical Engineering at ETH Zurich. His research focuses on turning big hairy 3D images into simple, robust, reproducible numbers without resorting to black boxes or magic. In particular, as part of several collaborations, he is currently working on automatically segmenting full animal zebrafish images, characterizing rheology in 3D flows, and measuring viral infection dynamics in cell lines.

Learn more at

4Quant: From Images to Statistics - http://www.4quant.com
Spark Demo from this presentation - https://gist.github.com/kmader/755c2d99c23f4cbe2e74
Setting up Spark on top of Sun Grid Engine - https://github.com/4Quant/sge_spark

Imaging

X-Ray Imaging Group at ETH Zurich - http://bit.ly/1gD8wKb
Quantitative Big Imaging Course at ETH Zurich - http://bit.ly/1kj9mnq
Presentation at Spark Summit 2014 - https://rawgit.com/4Quant/spark-summit-2014-presentation/master/ssPresentation.html

Graphs

FEM Demo on Gist - https://gist.github.com/kmader/6456262935af381c8dbe