Publishing Workflows for IPython/Jupyter Notebooks

class: center, middle

# Publishing Workflows <br>for Jupyter

## Andrew Odewahn |  @odewahn
##    Kyle Kelley |  @rgbkrk
##    Rune Madsen |  @runemadsen

http://odewahn.github.io/publishing-workflows-for-jupyter

---
# Goal

<div style="font-size: 150%">
Demonstrate an authoring and publishing process for long-form content (books, courses, tutorials, etc) that can include the IPython Notebook computing environment, but are authored outside of the IPython authoring environment.  (And, hopefully, Jupyter very soon!)
</div>

---

# Write in Markdown, AsciiDoc, or HTML

* Easily diff-able formats (vs. JSON, which is not)
* Collaboration / GitHub friendly 
* Specify executable code cells using a simple, declarative format.  For example:

```html
<pre data-executable="ipython" data-code-language="python">
%pylab inline
x = np.linspace(0, 10)
plt.plot(x, np.sin(x), x, np.cos(x))
</pre>
```

---

# Generate a static site with embeddable code widget

* Write in tools like [Jekyll](http://jekyllrb.com/), [Pelican](http://blog.getpelican.com/), or [Atlas](https://atlas.oreilly.com/)
* Code cells run in a widget, everything else is HTML, CSS, and JavaScript

.center[<img src="images/jupyter-plugin.png" width=70%/>]

---

# Kernels run in Docker containers

* Each user gets his or her own kernel
* Kernels run in Docker container
  * [ipython-kernel](https://github.com/oreillymedia/ipython-kernel) is a reference implementation
* Based on the original [singlecell](https://github.com/minrk/singlecell) server by Min RK
  * Need a singlecell for IPython 2 and 3, not just 1

---
# Scaling strategy

* Short-term, [tmpnb](https://github.com/jupyter/tmpnb) can spin up new, transitory kernels
* Long-term, we need a new system with an API that can handle scaling and security

.center[<img src="images/tmpnb.png" width=60%/>]

---
# Reference implementation in O'Reilly Atlas

* Write in Atlas using HTML, Markdown, or Asciidoc
* Mark up code examples as follows:

```html
<pre data-executable="ipython" data-code-language="python">
... your code here ...
</pre>
```
* Specify the `oreillymedia/jupyter_theme` theme
* Install the [Atlas CLI](https://github.com/oreillymedia/atlas-cli) utility for building and publishing (Mac OSX only for now)
* Clone your project to your local machine
* Run the following commands within your cloned directory :

```bash
$ atlas build --html
$ atlas publish --public
```

* Your project will be published on [sites.oreilly.com](http://sites.oreilly.com/odewahn/jem-test/ch01.html)

---

# Edit using Visual Editor (HTML), Markdown, or AsciiDoc

<center>
<video autoplay="true" loop="true" muted="true" width="640">
   <source src="https://s3.amazonaws.com/orm-atlas-media/atlas-jupyter/edit-with-atlas.mp4" type="video/mp4"> Your browser does not support the video tag.
</video>
</center>

---

# Specify a Jupyter-enabled theme

.center[<img src="images/jupyter-theme.png" width=80%/>]

---

# Build and publish a static site

* [atlas-cli](https://github.com/oreillymedia/atlas-cli) provides a simple, command-line interface for building and publishing

<center>
<video autoplay="true" loop="true" muted="true" width="640">
   <source src="https://s3.amazonaws.com/orm-atlas-media/atlas-jupyter/publish-atlas-cli.mp4" type="video/mp4"> Your browser does not support the video tag.
</video>
</center>

---

# Run code via the widget

<center>
<video autoplay="true" loop="true" muted="true" width="640">
   <source src="https://s3.amazonaws.com/orm-atlas-media/atlas-jupyter/static-site.mp4" type="video/mp4"> Your browser does not support the video tag.
</video>
</center>

* Sample is available on [sites.oreilly.com](http://sites.oreilly.com/odewahn/jem-test/ch01.html)
* Data is persistent from widget to widget, just like the Notebook

---

# Advantages / Disadvantages

* Advantages
  * More design possibilities
  * Ability to embed widget on blogs, nbviewer, etc.
  * Static assets served by CDN
  * Simple writing and publishing process
  * Support for publishing-specific features like cross references, sidebars, index terms, etc
* Disadvantages
  * No easy way in tmpnb to specify a different container (i.e., one with your own dependencies)
  * No authentication framework

---
# Future Directions to explore

To generalize and expand the concept, we could:

* Create singlecell servers for IPython 2 and Jupyter (i.e., update the singlecell example for those versions)
* Develop a remote service that you call and it returns a kernel for a specific language
  * Use https://github.com/ipython/ipython/wiki/IPEP-25%3A-Registry-of-installed-kernels
* Front end javascript that can work with the above remote API
* Write a minimized set of IPython/Jupyter javascript that can handle the outputs without being the notebook