I Love Symposia!

Introducing napari: a fast n-dimensional image viewer in Python

Juan Nunez-Iglesias — Thu, 24 Oct 2019 13:59:54 GMT

I'm really excited to finally, officially, share a new(ish) project called napari with the world. We have been developing napari in the open from the very first commit, but we didn't want to make any premature fanfare about it… Until now. It's still alpha software, but for months now, both the core napari team and a few collaborators/early adopters have been using napari in our daily work. I've found it life-changing.

The background

I've been looking for a great nD volume viewer in Python for the better part of a decade. In 2009, I joined Mitya Chklovskii's lab and the FlyEM team at the Janelia [Farm] Research Campus to work on the segmentation of 3D electron microscopy (EM) volumes. I started out in Matlab, but moved to Python pretty quickly and it was a very smooth transition (highly recommended! ;). Looking at my data was always annoying though. I was either looking at single 2D slices using matplotlib.pyplot.imshow, or saving the volumes in VTK format and loading them into ITK-SNAP — which worked ok but required me to constantly be changing terminals, and anyway was pretty inefficient because my disk ended up cluttered with “temporary” VTK volumes.

I tried a bunch of things after that time, including Mayavi and even building my own orthogonal views viewer with Matplotlib (which was super slow), but nothing really clicked. What's worse, I didn't see any movement on this — I even found a very discouraging thread on MNE-Python in which Gaël Varoquaux concluded, given the lack of support, that 3D visualisation was not mission critical to anyone and not worth pursuing further. (!) Throughout that time, I kept living with either manually picking out 2D slices, or going back to tools outside of Python, with all the context switching that entailed. The end result is that I was looking at my data far less than I should have been, which slowed down my work. I can't count the number of silly mistakes I made and dead ends I chased, just because I wasn't looking at the data enough.

The beginning

In the meantime though, I was busy improving support for n-dimensional images in scikit-image. In May last year, Nelle Varoquaux organised a joint sprint with developers from scikit-learn, scikit-image, and dask at UC Berkeley, and I was one of the lucky invitees. I'd recently seen that Loïc Royer, a good friend from my Janelia days, had moved to San Francisco to start his lab at the Chan Zuckerberg Biohub, so I asked him if I could crash at his place for a few days before the sprint.

I knew Loïc as a Java guy. In Janelia he'd repeatedly extolled the virtues of Java to me, and since then he'd worked on some incredible Java software, including ClearVolume, a super-fast 3D viewer, and ClearControl, a super-fast microscope control library (you might detect a theme there ;). In fact, although I love Python and its ecosystem and what we've built with scikit-image, I've always been intimidated by Fiji and its ecosystem and its ~~10,000~~ 15,000 citations. Was I deluding myself that Python was important in biology?

So I was both shocked and delighted when Loïc casually mentioned that he needed a fast nD viewer in Python, and that we should build it together. Loïc had been pushing the state of the art of deep learning in microscopy (and continues to do so), and, despite the hard work of many to integrate Fiji with the major deep learning learning libraries (which are Python), as well as the wider scientific Python ecosystem, it remains challenging to use both together. It might well remain so, because the build, installation, and dependency tooling is so disparate between the two communities. And so that evening, in Loïc's apartment, napari (lowercase n) was born (though it did not yet have a name).

Loïc walking around San Francisco the weekend after I landed.

Following on from his work on ClearVolume, Loïc and I agreed on some founding design principles for our tool:

n-dimensional first;
native (neither of us likes opening a browser to work);
blazing fast (GPU-powered);
native 2D and 3D viewing; and
minimal new windows and pop-ups, choosing layering instead.

In terms of speed, we knew it was doable, thanks to Martin Weigert's excellent 3D volume viewer, spimagine.

The next week, at the Berkeley sprint, I finally met Kira Evans, a contributor to scikit-image who had made several incredibly technical pull requests, on my favourite topic of extending n-dimensional support, and whom I'd thus asked Nelle to invite to the sprint. She had turned out to be a freshman at Northeastern University, but despite her age she was already doing great things in open source. In general, life is a big accumulation of small decisions you don't even think about. On certain rare occasions, though, you make decisions on which you look back and think, hot damn that was a good decision! Inviting Kira is one of those for me. (Nelle agreed.)

John Kirkham explains dask-image to Kira, Emmanuelle Gouillart, and me at the Berkeley sprint.

That week at the sprint, Loïc was able to hack out a quick prototype. Meanwhile, Kira was busy helping scikit-image with n-dimensional rotations, LowLevelCallables in C and Cython, and various other crazy things.

I'll take a short moment to acknowledge VisPy, which is the basis for all of napari's visualisation: napari's canvas is a VisPy canvas, its events are VisPy events, and its visual building blocks are VisPy Visuals. I don't know the history of VisPy, but I want to thank all its contributors, especially Almar Klein, who drove it for a very long time, and David Hoese, who resurrected it when Almar could no longer maintain it, and who maintains it today. VisPy is yet another example of open source software being inadequately supported by academia.

The next weekend, on the Caltrain to the Computing History Museum, we worked on choosing a name. After playing around with some terrible acronyms, we decided to continue the theme of Pacific islands begun by Fiji (though Fiji itself is a spectacular recursive acronym, “Fiji Is Just ImageJ”). We needed a starting point, and we settled on the geographic midpoint between Loïc's base of San Francisco and mine of Melbourne. That's in the middle of the ocean, of course, but not too far from the tiny village of Napari, in the Republic of Kiribati. We thought it had a nice ring to it, and the name stuck.

There was a big risk of napari never getting off the ground. Starting a new lab is hard work, and Loïc knew he would not have time to develop it further. He did, however, have some funds available for a summer intern, and after seeing the work Kira did that week, he offered her the internship. (As a result, Kira is now officially a college dropout and full-time software engineer at CZI, the Chan Zuckerberg Initiative.) That summer, under the guidance of Loïc, Stéfan van der Walt, and myself, Kira put together the first implementation of napari as you see it today.

By late summer/early fall, although internally napari was quite a mess, as tends to happen in new, fast-growing projects, functionally it was already impressive enough that Jeremy Freeman and Nick Sofroniew, from CZI's Computational Biology division, started paying close attention. Nick, who'd had similar experiences to mine working with Python and nD images, fell in love with it, and literally could not wait for us to move it forward. He took matters into his own hands.

Initially, Nick brought some brilliant management to napari, instituting weekly meetings that he drove (and continues to drive) like a champion. Those meetings were critical to bringing others into the loop, including Ahmet Can Solak, Kevin Yamauchi, Bryant Chunn, and Shannon Axelrod, who started to make pull requests to improve napari. Pretty soon, though, Nick made his own pull request, and after that it was like seeing a racecar take off. No one has done more for napari than Nick, and Loïc and I owe him an enormous debt of gratitude.

Soon after that, the StarFISH team at CZI adopted napari as their main image viewer, which again helped us discover bugs, improve our UI, and add features. By February, everyone in the project was using it regularly, despite its warts. We started to think about broadening the group of people with eyes on napari.

Nick shows napari to attendees of the Neurodegeneration Challenge Network Investigators Meeting in Februrary. Photo by CZI Science.

CZI was organising a meeting for grantees of its Imaging Scientists and Imaging Software Fellows programs in March, which we thought would be a good opportunity to grow the team. Among others, we invited John Kirkham (NVIDIA) and Eric Perlman (then at Johns Hopkins, now freelancing). The CZI meeting was a dramatic demonstration of the power of bringing together people with complementary experience, as John added support for viewing bigger-than-RAM dask arrays in about 20 minutes flat, and Eric massively improved the performance of our segmentation visualisation layer.

Loïc explaining napari's design to John Kirkham and Wei Ouyang at the Chan Zuckerberg Biohub the day after the CZI meeting.

I came out of the March meeting thinking hey, this napari thing might have legs! Not only had newcomers been able to improve it quickly without knowing the codebase (which usually points to at least decent design), but we heard over and over from the imaging scientists that viewing and annotating large datasets remained a challenge for them.

So, we started gently expanding the user base, but in hushed tones, full of caveats: “This software we're working on might be useful for you… If you're brave and can work around all the missing functionality… Be sure to report any issues…” One of our most frequent requests, which was indeed part of the napari plan from the very beginning, was 3D volume rendering, and Pranathi Vemuri, a data scientist at the Biohub, implemented that just in time for SciPy 2019. At that conference, Nick presented napari to the public for the first time, because many SciPy attendees tend to not mind working with development versions of libraries. Indeed, we got contributions that week from Alex de Siqueira (from the scikit-image team) and Matthias Bussonnier (IPython).

The biggest revelation for me, though, came immediately after SciPy, when I went to Jackson Lab to teach a workshop on scikit-image. I've taught scikit-image many times before, but for the first time, I went off-script and analysed new data live, on IPython, rather than using Jupyter notebooks pre-populated with toy data. I used napari instead of matplotlib, and found napari's layers model insanely useful to visualise every step of the analysis, one on top of the other, toggling their visibility on and off to compare them. And it was super easy to add interactivity to the mix, clicking on some points in napari, then grabbing those points to use as watershed seeds, and popping the resulting segmentation as another layer to napari.

Combining interactive annotation and segmentation algorithms with napari.

In the months since then, we've spent a lot of time improving the code, and polishing the little edge cases, like making sure the coordinates reported by the cursor are pixel perfect, merging the previously separate Image (2D) and Volume (3D) layers, and extending 3D and multi-resolution support to all layers.

Thanks to CZI's massive in-house investment in open source, we also actually got an audit for napari's GUI from Lia Prins, a UX designer at CZI — an incredible opportunity that few academic projects get. And, to repeat a familiar pattern, Nick immediately got to work and implemented a bunch of fixes from the audit. To me, this update is what's most pushed me to write this post, as the UI now feels more like a coherent whole than the mishmash of features that results from organic growth of a project.

napari now

As I mentioned at the start, currently, everyone on the team uses napari on a daily basis for their own work. Our workflows actually look quite different, yet napari meets the needs of all of them, in some form or another. Last month we were surprised to find that Constantin Pape, at EMBL, was developing his own viewer, Heimdall, on top of napari, completely independently from us. I'm a big fan of Constantin's work, so little has brought me more joy than finding his repo and reading this comment:

most of the credit goes to napari; it's so great to finally have a decent python viewer.

So, what are those use cases?

1. Just looking at NumPy ndarrays quickly

This is my most common use case. I like to think about and use 2D, 3D, and 4D images interchangeably. So, I often find myself calling plt.imshow on a 3D array and being greeted by a TypeError: Invalid dimensions for image data, after the figure window has popped up, which now stares back at me blank and forlorn. No more, with napari.view_image. By default, napari will display the last two dimensions of an array, and put in as many sliders as necessary for the remaining dimensions.

To illustrate, I'll create a syntethic 4D array of blobs using scikit-image's binary_blobs function.

import numpy as np
from skimage import data, filters


blobs_raw = np.stack([
    data.binary_blobs(length=256, n_dim=3, volume_fraction=f)
    for f in np.linspace(0.05, 0.5, 10)
])

blobs = filters.gaussian(blobs_raw, sigma=(0, 2, 2, 2))
print(blobs.shape)

(10, 256, 256, 256)

Now we can look at the volume in napari:

viewer = napari.view_image(blobs)

Thanks to VisPy and OpenGL, the canvas is just blazing fast, and a joy to navigate.

Using napari, we can immediately see that the blobs grow dramatically along the leading axis. With my previous matplotlib workflow, I might have missed that, as I would only look at one or two slices before continuing my workflow in whatever preconceived direction I'd chosen.

We can click on the little cube icon to switch to a 3D view (or type viewer.dims.ndisplay = 3 in our IPython terminal). Napari will remove one of the sliders, and display a maximum intensity projection of the volume.

White blobs turn out to not look so great in maximum intensity, so let's look at the 3D view with the famous mitosis dataset from Fiji's sample data (which I think comes from this paper by Griffis, Stuurman and Vale, but I'm not sure...):

from skimage import io

mitosis = io.imread('mitosis.tif')  # make sure your working directory is right
print(mitosis.shape)

(51, 5, 2, 196, 171)

Those axes are time, z, channels, y, and x (TZCYX).

viewer = napari.view_image(mitosis, name='mitosis')

As above, you can see the last two dimensions displayed, then sliders for the remaining dimensions. The sliders are proportionally sized to the number of slices along each axis, making it easy to figure out which axis is what. (This was one of Lia's recommendations that Nick implemented.) (And yes, labeled axes are in our roadmap!)

Edit: 2019-10-28: removed broken use of view_multichannel function.

We want to see the two channels as individual images to overlay, rather than as an extra dimension. We can do this manually with two add_image calls, adding a scale keyword argument to account for the different scales in xy and in z:

viewer = napari.Viewer()
viewer.add_image(mitosis[:, :, 0, :, :], colormap='magenta',
                 name='aurora-B', scale=[1, 10, 1, 1],
                 contrast_limits=[1500, 6500])
viewer.add_image(mitosis[:, :, 1, :, :], colormap='green',
                 name='tubulin', scale=[1, 10, 1, 1],
                 contrast_limits=[1600, 16000])

2. Overlaying computation results

As I mentioned earlier, the first time it really hit me how useful napari could be was when I was teaching scikit-image at Jackson lab. Napari uses a layered display model, so you can keep adding layers on top of each other, and change their opacity or visibility on the fly. This makes it really convenient to examine the steps of a pipeline as you process data, and maybe try different methods if the results don't look so great mid-pipeline.

In this example, I open napari from IPython and I keep adding layers as I try different segmentation methods on the coins example image from scikit-image:

3. Annotating data

Sometimes, it can be difficult to get an algorithm to pick out exactly what you want in an image. With the right UI, however, annotation can be extremely fast, and just a little interaction can dramatically help automated algorithms.

from skimage import data
from skimage import filters
from skimage import segmentation
from skimage import morphology

import napari


coins = data.coins()

viewer = napari.view_image(coins, name='coins')

edges = filters.sobel(coins)

edges_layer = viewer.add_image(edges, colormap='magenta', blending='additive')

pts_layer = viewer.add_points(size=5)
pts_layer.mode = 'add'
# annotate the background and all the coins, in that order

coordinates = pts_layer.data
coordinates_int = np.round(coordinates).astype(int)

markers_raw = np.zeros_like(coins)
markers_raw[tuple(coordinates_int.T)] = 1 + np.arange(len(coordinates))
# raw markers might be in a little watershed "well".
markers = morphology.dilation(markers_raw, morphology.disk(5))

segments = segmentation.watershed(edges, markers=markers)

labels_layer = viewer.add_labels(segments - 1)  # make background 0

4. Viewing very large (dask) arrays

Thanks to John Kirkham's work, napari only loads the data that it needs to display. Therefore, virtual arrays such as Dask arrays, HDF5 datasets, and zarr files can be loaded quickly and easily into napari, provided that the individual slices are small enough.

Today's microscopes produce datasets ranging in the hundreds of GB to multiple TB in size. Berkeley's Gokul Upadhyayula has made his lattice light sheet data publicly available. Nick converted this to a zarr file for ease of access.

Now, we can use a dask array to view this 100GB array quickly and easily:

import os
from dask import array as da

image = da.from_zarr(
    os.path.expanduser('~/data/gokul-lls/aollsm-m4-560nm.zarr')
)

viewer = napari.view_image(image, name='560nm', colormap='magma',
                           contrast_limits=[0, 150_000])

5. Quickly looking at images

When you pip install napari, you also get a command-line client that lets you quickly view image stacks. Here is a downsampled, isotropic version of this image from my colleagues, Volker Hilsenstein and André Nogueira Alves:

napari ~/data/ovarioles/droso-ovarioles-isotropic.tif

Or the 4D C. elegans embryo from the scif.io example images (unzipping gives a folder of images):

napari ~/data/EmbryoCE

Or a data set of many images, such as the red blood cell spectrin network from my work with Adam Blanch in Leann Tilley's lab (data available from the Open Science Framework here):

napari ~/data/schizonts/*.tif

6. Parameter sweeps

A hat tip to Davis Bennett at Janelia, who came up with this one. Suppose you want to manually find the best threshold for an image. Thanks to dask's lazy evaluation, you can attach a function call to a dimension slider, and use that slider to call the function with many different parameter choices. And napari's dimensions model tries to mirror NumPy's broadcasting rules, so that you can view data of different dimensionality together.

from skimage import data
import dask


coins = data.coins()

def threshold(image, t):
    arr = da.from_array(image, chunks=image.shape)
    return arr > t

all_thresholds = da.stack([threshold(coins, t) for t in np.arange(255)])

viewer = napari.view_image(coins, name='coins')
viewer.add_image(all_thresholds,
    name='thresholded', colormap='magenta', blending='additive'
)

We can even have a bit of fun at the end with 3D rendering... 🙃

This is not just a cute demo. I actually found this trick really useful when investigating a bug in scikit-image. By varying the threshold for h-maxima, I could see detections blinking in and out of existence, even though they are supposed to only decrease as the parameter value is increased, thus confirming that there is a bug in the scikit-image implementation of h-maxima! 😬

And indeed, improving that example for this post, the broadcasting works with points, too, not just images!

image = np.load('heatmap.npy')

viewer = napari.view_image(image, name='original', colormap='green',
                           blending='additive')
result = np.array([hmaxima(image, f) for f in np.linspace(0.2, 100.0, num=100)])
points = np.transpose(np.nonzero(result))
viewer.add_points(points, name='maxima coordinates', size=3, opacity=0.5)
result_image = viewer.add_image(result, name='result', colormap='magenta',
                                blending='additive')

7. Overlay polygons

One of Nick's earliest contributions was one of his biggest: a vector-based "shapes" layer for polygons, rectangles, circles, and other annotations. This makes it very easy to replicate things like the active contours scikit-image example, without having to flip coordinates, because napari is an image-first library:

from skimage import data
from skimage import filters
from skimage.color import rgb2gray
from skimage.segmentation import active_contour


astro = data.astronaut()
astro_gray = rgb2gray(astro)

s = np.linspace(0, 2*np.pi, 400)
r = 100 + 100*np.sin(s)
c = 220 + 100*np.cos(s)
init = np.array([r, c]).T

snake = active_contour(filters.gaussian(astro_gray, 3),
                       init, alpha=0.015, beta=10,
                       gamma=0.001, coordinates='rc')

viewer = napari.view_image(astro, rgb=True, name='astro')

viewer.add_shapes([init], name='initial',
                  face_color=[(0, 0, 0, 0)],
                  edge_color='red', edge_width=2
                  shape_type='polygon')

viewer.add_shapes([snake], name='snake',
                  face_color=[(0, 0, 0, 0)],
                  edge_color='blue', edge_width=2
                  shape_type='polygon')

8. More!

Nick has been working hard to improve our tutorials, which will have more details about all of the above. Meanwhile, Ahmet Can and Bryant are busy making it easier to use napari to view images live from a microscope, or another source. We look forward to hearing more use cases from you! Your first point of contact to ask for help should be the image.sc forum — don't forget to use the "napari" tag! — as well as our GitHub issues for bug reports and feature requests.

This might be a good time to emphasise that napari is still in alpha, which means that we might change the API at any time. This is not to discourage you from using it, but rather to point out that you should be ready to evolve quickly with it, or pin to the current minor version.

Our vision for the future

At the risk of sounding like a broken record, napari has already proven itself insanely useful for all of us at the team, as well as a few extended members of our community. But we are just getting started. We want to napari to help not just Python practitioners, but also biologists and other scientists who want to access Python's enormous scientific ecosystem, but don't necessarily want to learn Python.

We are working on a plugin system to allow the data from napari layers to be the input to Python functions, and have their outputs appear as new layers. A sneak peek:

We are building this in collaboration with the developers of ImJoy to make our plugins cross-compatible.

In addition to applying functions and seeing the outputs (what ImJoy's Wei Ouyang refers to as functional plugins), we see napari as a basis for more complex interactivity, such as closed-loop learning in the style of Ilastik, skeleton tracing, 3D annotation, and more. We already provide a framework to add or modify keyboard shortcuts or mouse interactivity, so it's possible to write this functionality on top of napari now. But we want to provide a unified framework that will prevent plugins from clobbering each other's functionality.

Our aim for the plugin framework is for every function to produce valid and readable Python code, so that path from performing a set of operations to producing a Python module — and thus a plugin — is perfectly smooth.

At the March CZI meeting, someone asked me whether we would provide a headless mode for napari processing. Yes and no: Our motto developing the plugin system is that napari-headless is just Python. With the recording capability and built-in IPython console, we think that napari could be a platform to teach non-computational scientists the basics of Python, thus providing an accelerated loop converting scientists from users to plugin contributors, and then to library contributors, and improving the whole ecosystem in the process.

Join us!

In a little over a year, napari has grown beyond what I thought possible. But we still have a lot of work to do! In addition to our issues list, we have an open roadmap. Feel free to jump in and join the fray!

We also encourage you to contribute to the many open source projects on which we depend, starting with VisPy: if you have OpenGL experience, they could really use your help! Other libraries we build on include NumPy, SciPy, IPython, QtConsole, ImageIO, scikit-image, QtPy, and PyOpenGL.

Acknowledgements

I've tried to name people's contributions to napari as they happened, but so many have helped to build it, it was impossible to name everyone. Our ever-growing committers list can be found on GitHub, but many more have contributed in countless ways. CZI's Shannon Axelrod and Ambrose Carr helped us make early design decisions when they adopted napari as the main viewer for StarFISH. Charlotte Weaver helped us build a (still upcoming) binary installer, and serves on our Code of Conduct (modelled after SciPy's). Greg Johnson and his team at the Allen Institute have used napari to build a corpus of annotated images — and helped drive the design of napari for that use case.

At the Imaging Kickoff Meeting last March, since CZI had invited me, I was staying at a hotel downtown rather than at Loïc's. One night, Loïc sent me an exasperated message that still brings a smile to my face: “You know, if you'd stayed at your stupid hotel a year ago, there would be no napari.” It constantly amazes me how many things had to line up for napari to exist, let alone be where it is today.

I'm so grateful to everyone who has helped napari get to get to this point. If I haven't mentioned your contribution, please email me and I'll make it right. One nice thing about blog posts is that they are easy to update!

Why citations are not enough for open source software

Juan Nunez-Iglesias — Tue, 28 May 2019 08:41:54 GMT

A few weeks ago I wrote about why you should cite open source tools. Although I think citations important, though, there are major problems in relying on them alone to support open source work.

The biggest problem is that papers describing a software library can only give credit to the contributors at the time that the paper was written. The preferred citation for the SciPy library is “Eric Jones, Travis Oliphant, Pearu Peterson, et al”, 2001. The “et al” is not an abbreviation here, but a fixed shorthand for all other contributors. Needless to say many, many people have contributed to the SciPy library since 2001 (GitHub counts 716 contributors as of this writing), and they are unable to get credit within the academic system for those contributions. (As an aside, Google counts about 1,200 citations to SciPy, which is a breathtaking undercounting of its value and influence, and reinforces my earlier point: cite open source software! Definitely don't use this post as an excuse not to cite it!!!)

Not surprisingly, we have had massive contributions to scikit-image since our 2014 paper, and those contributors miss out on the citations to our paper.

From a maintainer point of view, one way to deal with this is to periodically write “update” publications that become the preferred way to cite the software. This is the approach taken by the CellProfiler team, for example, and the one we will take for scikit-image. This allows new contributors to benefit, while also preventing infinite returns for the original authors, which is as it should be.

I think this is as good as project maintainers can do now, but leaves open the questions of authorship, publication frequency, and the potential dilution of the authors' message.

A proper fix requires structural change that directly recognises the value of open source software to science. Software papers are in fact a hack of the academic system, a currency conversion, and as with any conversion, there are losses and inefficiencies involved. Indeed, the paper provides less value than good documentation for the software. (Last year, when someone tweeted about their latest software paper, many replies asked for the GitHub link in the abstract! People wanted to look at the software, not the paper.)

So what does open source software really need, if not citations? Of course, it's money. Citations are useful because they can potentially be translated into grants, promotions, and jobs, but wouldn't it be great if we could cut out the middleman and just have great open source translate directly into grants, promotions, and jobs?

The insane thing is that granting bodies actually give enormous amounts of money to closed source software developers, since grant money is routinely spent on software licenses. Instead of subsidising proprietary software, that money could be used to fund open source developers, and improve software for everyone.

How to fund open source

I might have left the how for a later post, but while I was drafting this, CZI announced direct funding specifically for open source software. Needless to say I think this is wonderful. I am funded by an earlier closed funding round to develop scikit-image, and I thought a lot at the time about all the other deserving, unfunded projects that I use. An open call is absolutely the right thing to do, and supporting maintenance rather than development of hot new things, as this CZI call is doing, is also the right thing to do. (Others have flagged that a single year of support, even if renewable, makes it difficult to support careers, and I agree, but this is a new space, and it makes sense that CZI would dip its toes before jmuping in.)

The response to CZI's announcement has been absolute fire. There is enormous pent-up need and frustration over funding maintenance and improvement for open source tools underpinning an enormous amount of science. (Even a month later, reading Andreas's tweet makes me want to scream in frustration in the crowded train in which I write this. Not impactful enough!?) Given the buzz around CZI's call, I wonder whether CZI has enough staff to even read through the avalanche of applications it will receive. My hope is that national funding bodies will take notice and start funding open source maintenance work, as Germany has recently done. (It's unclear to me whether Germany has repeated that call since 2016. If you know, please leave a comment below!)

Since I'm an academic, for a long time I was stuck in the mindset that granting bodies need to step up and recognise that established open source is a valuable thing to fund, and this is certainly true. More recently though, as I dipped my toes into some private sector consulting, it occurred to me that the situation is completely ridiculous — “Hey look at this incredible thing we built, it's serving thousands to millions of scientists, can we get some money to keep it going?” “It's done and it's free, why would we give you money?” It's a fundamental misunderstanding at the very top of funding agencies of how software maintenance works.

There are plenty of people that do understand the value of the software, though: its users, many of whom are in a position to use grant money to help maintain it. But open source developers have not thus far made it easy to ~~donate money to~~ invest in their projects. I'm not talking about a little donate button on the homepage, which will never substantially support a project. Rather, open source projects should sell a product, be it a maintenance contract, a support contract, or a new feature development, or even a feel-good contract. But “buying” an open source software package should feel exactly like buying closed-source software, and to University purchasing departments it should look exactly like a normal software purchase.

As I was thinking about this idea, I moved my blog from wordpress.com, for which I was paying $100 per year, to Nikola, an open source static site generator written in Python. I love Nikola, and was ready to put my $100 towards the project, but to my surprise, even a donate button was absent. As Andreas Mueller and Chris Holdgraf have pointed out, many projects haven't even considered what they would do with a large influx of money if they got one. That needs to change.

Thankfully, organizations including NumFOCUS, QuanSight, Tidelift, and others are working hard to rectify this situation. I'm trying to help where I can and I really look forward to seeing what they come up with.

If you are interested in this topic, we are organising a Birds of a Feather session to take place at the SciPy 2019 in Austin in July. Please join us!

Why you should cite open source tools

Juan Nunez-Iglesias — Thu, 02 May 2019 02:31:30 GMT

Every now and then, a moment or a sentence in a conversation sticks out at you, and lodges itself in the back of your brain for months or even years. In this case, the sentence is a tweet, and I fear that the only way to dislodge it is to talk about it publicly.

Last year, I complained on Twitter that a very prominent paper that was getting lots of attention used scikit-image, but failed to cite our paper. (Or the papers corresponding to many other open source packages.) I continued that scientists developing open source software depend on these citations to continue their work. (More on this in another post...) One response was that surely the developers of the open source scientific Python stack were not scientists per se, and that citations were not a priority for them.

I still sigh internally when I think of it.

That tweet manifests a pervasive perception that open source scientific software is written by God-like figures. These massively experienced software developers have easy access to funds for their work, and are at the service of all the other scientists, who are their users. I used to share this perception, but it is utterly false.

I certainly hadn't thought about the funding question, but I did think of packages like NumPy and SciPy as being written by "pros" (whatever that means), whose main job was to produce these amazing libraries.

My ideas started to change only after I attended the SciPy 2012 conference, and I was invited to the scikit-image sprint by its lead author, Stéfan van der Walt. I was totally starstruck, and even after meeting him I just assumed his job was "open source guru", or somesuch. It is only later that I learned that he was a postdoctoral researcher and lecturer in applied mathematics at Stellenbosch University, in South Africa. As with other academics, his main job was to produce research and to teach. Open source was something he produced on the side.

(Years later, as we were scrambling to finish the final chapter for Elegant SciPy, Stéfan revamped the build infrastructure for our book — the scripts and configuration files that converted the Markdown text we were writing to executed code and html. "You have poor prioritisation skills, you know?", I taunted. He responded: "Yeah. But a lot of the SciPy documentation toolchain exists because of that." And yes: the revamped scripts ended up being extremely useful during the editing phase with O'Reilly.)

After the conference I continued to contribute to scikit-image, eventually joining the core development team, but throughout the process I continued to feel like an impostor, someone who had somehow managed to gain entrance to this hallowed and otherworldly community despite his inferior skills and knowledge.

Only after years of interacting with this community did I internalise the fact that nearly all of this software stack has been produced by practising scientists who took the extra care and effort to ensure that their code was robust, well-tested, and easily accessible to all. Despite a recent influx of interest and contributions from industry, as far as I can tell, most contributions to the SciPy stack still come from practising scientists in academia.

I know this now, but until recently I was suffering from another fallacy: that if I knew it, clueless as I was, then surely everybody knew it. That tweet disabused me of that notion, and this is why you're reading this. I hope it is instructive, and that you'll find it worth sharing widely.

This idea, that the SciPy stack is made by active scientists, is important because it affects how this work can be supported. Sadly, neither university hierarchies nor national funding bodies recognise code as valuable output. (There are some exceptions, but this remains the norm.) By and large, the only things that count are papers, grants, and, to a lesser extent, teaching evaluation scores.

So, yes, many of the original contributors to the SciPy libraries are now in industry. But they were academics at the time that they contributed and were driven out because academia did not value their contributions. Academics should not have to sacrifice their careers to contribute to open source. Citations to software papers are an imperfect solution to this problem (again, more soon), but they sure as hell are better than nothing.

So, if you are a user of open source tools in the Scientific Python stack, I have two requests for you:

When you publish your work, cite every library that you import. Most scientific software has a notice on their homepage or README file pointing to a paper you can cite. By definition, if you've imported a library, you've found it useful, and if you've found it useful, then you probably care about supporting its authors. This is a small way you can contribute to their success.
You are good enough to contribute. If you have an issue with an open source package you are using, look at the source code. Submit an issue to the project's bug tracker (usually GitHub). And try your hand at fixing it. The software's authors will usually offer guidance on how to do this, and you will improve your own skills as a result. Good software development practices is one of the most transferrable skills you can gain.

Of course, citations alone will not solve the wider problem that open source software is chronically undervalued. I have many thoughts about how open source should be supported, especially in science, but I'll expand on that in an upcoming post.

Update: I'm adding a link to a related post: What do scientists know about open source?

Summer school announcement: 2nd Advanced Scientific Programming in Python (ASPP) Asia Pacific!

Juan Nunez-Iglesias — Thu, 30 Aug 2018 04:48:05 GMT

The Advanced Scientific Programming in Python (ASPP) summer school has had 10 successful iterations in Europe and one iteration here in Melbourne earlier this year. Another European iteration is starting next week in Camerino, Italy.

Now, thanks to the generous sponsorship of CSIRO, and the efforts of Benjamin Schwessinger and Genevieve Buckley, two alumni from the Melbourne school, and Kerensa McElroy, Agriculture Data School Coordinator at CSIRO, the Asia Pacific fork of ASPP gets its second iteration in Canberra, Jan 20-27, 2019.

Key details

The workshop runs January 20-27, 2019 at the Australian National University in Canberra, Australia.
topics include git, contributing to open source software with github, testing, debugging, profiling, advanced NumPy, Cython, and data visualisation.
hands-on learning using pair programming
free to attend (but students are responsible for travel, accommodation, and meals)
30 student places, to be selected competitively
application deadline is Oct 7, 2018, 23:59 Anywhere On Earth
website: https://scipy-school.org
FAQ: https://scipy-school.org/faq
apply: https://scipy-school.org/applications

Background

Three years ago, I had the privilege of teaching the 2015 ASPP school in Munich. It turned out to be a fantastic teaching experience (I have taught in 2 more since), and more importantly, it was a fantastic experience for the students. Students are selected for the school to fit a certain profile, neither too novice nor too advanced. As such, participants selected for the school are almost guaranteed to learn a great deal.

Indeed, almost every iteration of the school has been co-organised by former students. Sure enough, with the help of two students from the Melbourne instance, we will be able to have a new iteration in Canberra this January.

Course description

Scientists spend increasingly more time writing, maintaining, and debugging software. While techniques for doing this efficiently have evolved, only few scientists have been trained to use them. As a result, instead of doing their research, they spend far too much time writing deficient code and reinventing the wheel. In this course we will present a selection of advanced programming techniques and best practices that are standard in industry, but especially tailored to the needs of a programming scientist. Lectures are devised to be interactive and to give the students enough time to acquire direct hands-on experience with the materials. Students will work in pairs throughout the school and will team up to practice the newly learned skills in a real programming project — an entertaining computer game.

We use the Python programming language for the entire course. Python works as a simple programming language for beginners, but more importantly, it also works great in scientific simulations and data analysis. We show how clean language design, ease of extensibility, and the great wealth of open source libraries for scientific computing and data visualization are driving Python to becoming a standard tool for scientists.

Who is eligible?

This school is targeted at Master/PhD students, postdocs, and academic staff and technicians from all areas of science. Competence in Python or in another language such as Java, C/C++, MATLAB, or Mathematica is absolutely required. Basic knowledge of Python and of a version control system such as git, subversion, mercurial, or bazaar is assumed. Participants without any prior experience with Python and/or git should work through the proposed introductory material before the course.

We have strived to get a pool of students that is international and gender-balanced, and have succeeded, with gender parity in the last five schools.

The road to scikit-image 1.0

Juan Nunez-Iglesias — Thu, 12 Jul 2018 18:58:35 GMT

This is the first in a series of posts about the joint scikit-image, scikit-learn, and dask sprint that took place at the Berkeley Insitute of Data Science, May 28-Jun 1, 2018.

In addition to the dask and scikit-learn teams, the sprint brought together three core developers of scikit-image (Emmanuelle Gouillart, Stéfan van der Walt, and myself), and two newer contributors, Kira Evans and Mark Harfouche. Since we are rarely in the same timezone, let alone in the same room, we took the opportunity to discuss some high level goals using a framework suggested by Tracy Teal (via Chris Holdgraf): Vision, Mission, Values. I'll try do Chris's explanation of these ideas justice:

Vision: what are we trying to achieve? What is the future that we are trying to bring about?
Mission: what are we going to do about it? This is the plan needed to make the vision a reality.
Values: what are we willing to do, and not willing to do, to complete our mission?

So, on the basis of this framework, I'd like to review where scikit-image is now, where I think it needs to go, and the ideas that Emma, Stéfan, and I came up with during the sprint to get scikit-image there.

I will point out, from the beginning, that one of our values is that we are community-driven, and this is not a wishy-washy concept. (More below.) Therefore this blog post constitutes only a preliminary document, designed to kick-start an official roadmap for scikit-image 1.0 with more than a blank canvas. The roadmap will be debated on GitHub and the mailing list, open to discussion by anyone, and when completed will appear on our webpage. This post is not the roadmap.

Part one: where we are

scikit-image is a tremendously successful project that I feel very proud to have been a part of until now. I still cherish the email I got from Stéfan inviting me to join the core team. (Five years ago now!)

Like many open source projects, though, we are threatened by our own success, with feature requests and bug reports piling on faster than we can get through them. And, because we grew organically, with no governance model, it is often difficult to resolve thorny questions about API design, what gets included in the library, and how to deprecate old functionality. Discussion usually stalls before any decision is taken, resulting in a process heavily biased towards inaction. Many issues and PRs languish for years, resulting in a double loss for the project: a smaller loss from losing the PR, and a bigger one from losing a potential contributor that understandably has lost interest.

Possibly the most impactful decision that we took at the BIDS sprint is that at least three core developers will video once a month to discuss stalled issues and PRs. (The logistics are still being worked out.) We hope that this sustained commitment will move many PRs and issues forward much faster than they have until now.

Part two: where we're going

Onto the framework. What are the vision, mission, and values of scikit-image? How will these help guide the decisions that we make daily and in our dev meetings?

Our vision

We want scikit-image to be the reference image processing and analysis library for science in Python. In one sense I think that we are already there, but there are more than enough remaining warts that they might cause the motivated user to go looking elsewhere. The vision, then, is to increase our customer satisfaction fraction in this space to something approaching 1.0.

Our mission

How do we get there? Here is our mission:

Our library must be easily re-usable. This means that we will be careful in adding new dependencies, and possibly cull some existing ones, or make them optional. We also want to remove some of the bigger test datasets from our package, which at 24MB is getting rather unwieldy! (By comparison, Python 3.7 is 16MB.) (Props to Josh Warner for noticing this.)
It also means providing a consistent API. This means that conceptually identical function arguments, such as images, label images, and arguments defining whether an input image is grayscale, should have the same name across various the library. We've made great strides in this goal thanks to Egor Panfilov and Adrian Sieber, but we still have some way to go.
We want to ensure accuracy of our algorithms. This means comprehensive testing, even against external libraries, and engaging experts in relevant fields to audit our code. (Though this of course is a challenge!)
Show utmost care with users' data. Not that we haven't cared until now, but there are places in scikit-image where too much responsibility (in my view) rests with the user, with insufficient transparency from our functions for new users to predict what will happen to their data. For example, we are quite liberal with how we deal with input data: it gets rescaled whenever we need to change the type, for example from unsigned 8-bit integers (uint8) to floating point. Although we have good technical reasons for doing this, and rather extensive documentation about it, these conversions are the source of much user confusion. We are aiming to improve this in issue 3009. Likewise, we don't handle image metadata at all. What is the physical extent of the input image? What is the range and units of the data points in the image? What do the different channels represent? These are all important questions in scientific images, but until now we have completely abdicated responsibility in them and simply ignore any metadata. I don't think this is tenable for a scientific imaging library. We don't have a good answer for how we will do it, but I consider this a must-solve before we can call ourselves 1.0.

Our values

Finally, how do we solve the thorny questions of API design, whether to include algorithms, etc? Here are our values:

We used the word "reference" in our vision. This phrasing is significant. It means that we value elegant implementations, that are easy to understand for newcomers, over obtaining every last ounce of speed. This value is a useful guide in reviewing pull requests. We will prefer a 20% slowdown when it reduces the lines of code two-fold.
We also used the word science in our vision. This means our aim is to serve scientific applications, and not, for example, image editing in the vein of Photoshop or GIMP. Having said this, we value being part of diverse scientific fields. (One of the first citations of the scikit-image paper was a remote sensing paper, to our delight: none of the core developers work in that field!)
We are inclusive. From my first contributions to the project, I have received patient mentorship from Stéfan, Emmanuelle, Johannes Schönberger, Andy Mueller, and others. (Indeed, I am still learning from fellow contributors, as seen here, to show just one example.) We will continue to welcome and mentor newcomers to the Scientific Python ecosystem who are making their first contribution.
Both of the above points have a corrolary: we require excellent documentation, in the form of usage examples, docstrings documenting the API of each function, and comments explaining tricky parts of the code. This requirement has stalled a few PRs in the past, but this is something that our monthly meetings will specifically address.
We don't do magic. We use NumPy arrays instead of fancy façade objects that mask their complexity. We prefer to educate our users over making decisions on their behalf (through quality documentation, particularly in docstrings).
We are community-driven, which means that decisions about the API and features will be driven by our users' requirements, and not the whims of the core team. (For example, I would happily curry all of our functions, but that would be confusing to most users, so I suffer in silence. =P)

I hope that the above values are uncontroversial in the scikit-image core team. (I myself used to fall heavily on the pro-magic side, but hard experience with this library has shown me the error of my ways.) I also hope, but more hesitantly, that our much wider community of users will also see these values as, well, valuable.

As I mentioned above, I hope this blog post will spawn a discussion involving both the core team and the wider community, and that this discussion can be distilled into a public roadmap for scikit-image.

Part three: scikit-image 1.0

I have deliberately left out new features off the mission, except for metadata handling. The library will never be "feature complete". But we can develop a stable and consistent enough API that adding new features will almost never require breaking it.

For completeness, I'll compile my personal pet list of things I will attempt to work on or be particularly excited about other people working on. This is not part of the roadmap, it's part of my roadmap.

Near-complete support for n-dimensional data. I want 2D-only functions to become the exception in the library, maybe so much so that we are forced to add a _2d suffix to the function name.
Typing support. I never want to move from simple arrays as our base data type, but I want a way to systematically distinguish between plain images, label images, coordinate lists, and other types, in a way that is accessible to automatic tools.
Basic image registration functionality.
Evaluation algorithms for all parts of the library (such as segmentation, or keypoint matching).

The human side

Along with articulating the way we see the project, another key part of getting to 1.0 is supporting existing maintainers, and onboarding new ones. It is clear that the project is currently straining under the weight of its popularity. While we solve one issue, three more are opened, and two pull requests.

In the past, we have been too hesitant to invite new members to the core team, because it is difficult to tell whether a new contributor shares your vision. Our roadmap document is an important step towards rectifying this, because it clarifies where the library is going, and therefore the decision making process when it comes to accepting new contributions, for example.

In a followup to this post, I aim to propose a maintainer onboarding document, in a similar vein, to make sure that new maintainers all share the same process when evaluating new PRs and communicating with contributors. A governance model is also in the works, by which I mean that Stéfan has been wanting to establish one for years and now Emmanuelle and I are onboard with this plan, and I hope others will be too, and now we just need to decide on the damn thing.

I hope that all of these changes will allows us to reach the scikit-image 1.0 milestone sooner rather than later, and that everyone reading this is as excited about it as I was while we hashed this plan together.

As a reminder, this is not our final roadmap, nor our final vision/mission statement. Please comment on the corresponding GitHub issue for this post if you have thoughts and suggestions! (You can also use the mailing list, and we will soon provide a way to submit anonymous comments, too.) As a community, we will come together to create the library we all want to use and contribute to.

As a reminder, everything in this blog is CC0+BY, so feel free to reuse any or all of it in your own projects! And I want to thank BIDS, and specifically Nelle Varoquaux at BIDS, for making this discussion possible, among many other things that will be written up in upcoming posts.

Update: Anonymous comments are now open at https://pollev.com/juannunezigl611. To summarise, to comment on this proposal you can:

comment on the GitHub issue
submit a comment below
submit an anonymous comment at https://pollev.com/juannunezigl611

What do scientists know about open source?

Juan Nunez-Iglesias — Wed, 20 Jun 2018 11:19:46 GMT

A friend recently pointed out this great talk by Matt Bernius, What students know and don't know about open source. If you have even a minor interest in open source it's worth a watch, but the gist is: in the US alone, there are about 200,000 students enrolled in a computer science major. Open source communities are a great space to learn real-world programming, so why don't these numbers translate into massive contributions to open source?

At the core of the issue, Matt identifies two main problems: (1) colleges and universities simply don't teach open source, or even collaborative coding; and (2), many open source communities make newcomers feel unwelcome in a variety of ways.

I want to comment about this in the context of programming in science. That is, programming where the code is not the main product, but rather a useful tool to obtain a scientific result, for example in biology or physics. Here, we still see relatively little contribution to open source, for related but different cultural issues.

I've sent my scientists should code in the open post to a few people and the response from most remains sceptical. I hope this post will address some of their concerns.

Scientific culture is ridiculously secretive

The most common objection is to my assertion that people won't scoop you by looking at your code. I remember a tweet (that I sadly can't find now) that really got to the gist of the problem. It went something like this:

Someone in science having a new idea: "Ooh, I hope I don't get scooped!"
Someone in open source having a new idea: "Ooh, I hope someone has implemented this already!"

Update: I found the source! It's this tweet by Elizabeth Seiver.

This is a huge gap in culture that won't soon go away, but there are encouraging steps towards narrowing it. For example, PLOS Biology, a leading journal, recently announced that they would consider "scooped" studies for publication within six months of the "scooping". That goes some way towards re-aligning incentives towards collaborative and open science.

I've come across many collaborations that have started because of open source. I have not heard of someone getting scooped because of open source, but of course that sort of information would be hard to trace and come by. Several people did write to me that they were concerned about very specific groups rifling through their code expressly for the purpose of scooping them. For me it's hard to imagine someone even having that attitude, and my advice is that if you do face such a toxic community, it might be wise to change your chosen field of study.

Nevertheless, I want to emphasise here that open source programming can take many forms, with the zip file attached to the paper being the lowest, coding in the open being the highest, and several other models in between. Any steps you can take towards the higher models will ultimately help you. My preferred mode for code that really does have to be private is to use a private GitHub repository, and just make that repo public once the paper is accepted.

A lot of people prefer the "code dump with no revision history" model of post-publication sharing, but this tosses out a lot of valuable information for people coming after you: what have you tried that didn't work? What issues did you have with the code? Have you considered coding in one style or another? The code dump model also makes you less likely to use GitHub in the first place, depriving you of an opportunity to learn some valuable real-world skills.

For coding, scientists have even more severe impostor syndrome

As I mentioned in my original post, and this I find completely uncontestable, publishing shitty code is not a bad thing. Everybody writes bad code, and nearly everybody knows it. Here's Hadley Wickham, creator of dplyr, tidy data, ggplot2, among other things; in other words, someone who knows a thing or two about elegant code and about as close as one gets to coding royalty in science:

The only way to write great code is to write lots of shitty code first.

Publishing your raw code is a good thing and will absolutely not be a black mark on your career. Indeed, in open source circles, it is often a bare GitHub contribution history that is a black mark. (And this is another problem, but in my opinion a better one.)

Scientists don't know about open source

If knowledge of open source is lacking in computer science, what chance does it have in other fields? The truth is that outreach and education need to become a massive part of open source culture, especially in science.

I credit Stéfan van der Walt for my life in open source. After I gave a talk at SciPy 2012, he invited me to join the scikit-image sprint at the end of the conference. If it hadn't been for that, I probably would have just wandered around the hall, too shy to join any sprint (see "impostor syndrome", above), and my life would be very different right now.

Anyway, at that point I'd made my code "open source", which meant it was on GitHub. I had only added a license to submit to the conference. As a reminder, unlicensed code doesn't count as open source. But I had never really collaborated in open source. My idea of collaboration was my workflow with my colleague: a single branch (master), from which we both pulled and to which we both pushed. When I sat down with Stéfan and Tony Yu, and I figured what I wanted to work on, I asked: "So, should I just push to master, or what?" I still remember, with some embarrassment, the dubious look Stéfan and Tony exchanged, as they silently figured out which of them would introduce this newbie to pull requests.

But that's the thing: I shouldn't feel embarrassment. Scientists for the most part don't get introduced to coding in their education, much less to open source.

What can scientists in open source do?

A lesson from my continued contributions to the SciPy ecosystem, I hope, is that some light mentorship can yield enormous dividends later on. Stéfan and Tony took the time to walk me through the open source contribution process, when they could have dismissively sent me a link to some page explaining it. I'm a big fan of writing good documents for newcomers, but nothing beats a good hand-holding. It's very easy for me to imagine an alternate reality where I had not felt welcome or rewarded by the scikit-image project and my life had not taken this productive turn.

Continuing on imaginary themes, it is only slightly less plausible that the open source scientific world should be awash with new contributors at every level of science. How do we turn this dream into a reality?

If you are a scientist and this post is among your first encounters with the term "open source", and you think you might be interested in learning more, here are a few things I recommend, in order of easiest to hardest:

Read the preface and epilogue of my book with Stéfan and Harriet Dashnow. (Free online!) I feel a bit icky recommending my own book, but why repeat myself? In those chapters I tried to distill my thoughts on joining the SciPy community, which is a fantastic, rewarding space in which to do open source programming as a scientist. I expect many things we wrote generalise well to e.g. the tidyverse.
Look for upcoming software carpentry workshops near you. These are free two-day programming boot camps to introduce you to computational thinking, and, crucially, to version control with git.
Go to a SciPy conference. I know of SciPy, EuroSciPy, and SciPy India, but I have a vague memory of offshoots in Africa and South America.

If you are in a boat similar to mine (intermediate/advanced open source contributor in science), and you feel like you would like your work to feel a bit more crowded, I can tell you what I'm going to be doing in response to this talk:

Sign up to deliver (more) software carpentry training (or similar). Getting the word out is the number one thing.
In software carpentry, emphasise the role of git in collaboration. (I think the official program does not go far enough in this direction, and focuses instead on the initial linear history.)
If you are located in a university, talk to your CS department to see whether they have any courses in open source development. If not, see whether you can guest lecture in a suitable course to make students aware of the open source opportunities out there.
Similarly, follow up software carpentry with more advanced sessions on open source collaboration. I gained an enormous fraction of my programming skills from collaborating on open source. I really think there is no better tool for long-term learning in this space. An idea that I'd like to try out is to curate a bunch of open issues on prominent repos and get SWC students to sprint on them for a day¹. I know about the "good first issue" tag on GitHub. Unfortunately, my experience with it is mixed. I think many repos are overly optimistic with theirs (this includes scikit-image), and, furthermore, a large proportion of these tagged issues get "claimed" quickly — and often half-heartedly!
Write, write, write! Did you get a cool PR merged? Write a blog post about it! Or at least tweet! We need to get the message out that writing PRs is for everyone. =)

If you have any further ideas, I'd love to hear them.

Actually I drafted this post a while back, and tried this yesterday, with mixed success. I'll write about that experience soon. ;) ↩

1st ASPP Asia Pacific evaluation survey

Juan Nunez-Iglesias — Mon, 09 Apr 2018 03:29:53 GMT

In January of 2018, we had the first ASPP summer school outside of Europe. (This was a parallel workshop to the European one, which will be held in Italy in September 2018.) In general, it was a great success, with some caveats that we will elaborate on below.

First we want to note that this school was a bit different than the European ones, in that we only had attendees from Australian institutions, where the European school has broad international representation, including some from out of Europe. This was in some ways inevitable, as it is more expensive to travel to Australia from almost anywhere than to travel within Europe. On the other hand, we advertised relatively late, and we were unable to secure travel grants during the advertising period, so there is hope that a future edition would be able to attract a more international crowd from the Asia Pacific region.

Given all this, there was a question as to whether we would be able to capture the atmosphere of the school, which normally sees the students living together and socialising for basically the whole week. In this case, most students just went home after classes were finished. But although some of that atmosphere was missing, by the end of the week we did manage to get some close links between all the students and the faculty. The evaluations below show that most of the value of the school was preserved.

We note that 100% of the respondents (29/30 of the students) would recommend the course to their peers. So, although some lectures were better received than others, and although the programming project was not universally loved, we managed to provide value for everyone. All of this is in line with the evaluations at previous schools (available at https://python.g-node.org/wiki/archives.html).

The project, which consists of programming a videogame bot, is controversial every year, but, consistently, more people like it than don't, and people get to practice git, pair programming, and programming as a team, which is the single most difficult skill to practice when programming for science. Indeed when we walk around during the project programming sessions, we see people extremely engaged in what they are coding. It's difficult to imagine a scientific problem engaging such diverse people as the school's attendees (which come from very disparate scientific fields).

Of all the feedback, two particular statements, we hope from people in the same project group, broke our hearts. We decided not to include them in this report, because they might be easy to de-anonymise by group members, but they boil down to the following: a group member, by being combative and rude to others in their team, and deciding to essentially complete the project by themselves, ruined the programming project for all of their team members, with some even feeling that they were not good enough to contribute. This is tragic, because we want everyone in the school to feel empowered to do anything at all in Python.

Absolutely every student has something to offer in this project. Here, as in life, teams are comprised of members of varying skills. But we know from our selection that everyone has the skills to contribute (and this is confirmed by the fact that most attendees, for most lectures, felt that the difficulty level was "just right"). So if a student felt inadequate, it can only be because of the toxic team member.

Ned Batchelder recently wrote an excellent blog post about what he calls "Toxic experts" and what Tiziano Zito calls, somewhat more bluntly, "Arrogant assholes". (In discussions about this post, Tiziano and others noted that one does not have to be an expert to be toxic, or arrogant, or an asshole. No matter: the points below apply equally to anyone meeting any of the above characteristics regardless of expertise.)

The feedback we received should serve as a warning to selection committees and hiring managers everywhere about how damaging it is to allow such a person into your ranks. Due to the anonymous nature of the survey, we can't tell whether there was one or two toxic experts in our midst, but if it's one, they soured the school for five other people. If it's two, then that's ten people, a third of the school, that might have had a terrible experience. The problem with toxic experts is that they can so quickly cause damage to so many others. Thus, even if they are a mythical "10x engineer", they are not worth it.

Literally nothing that the above-described team member could have done, coding-wise, could make up for the damage they caused. Despite their strong opinions, they missed the entire point of the programming project, which is not to win a medal, but to learn about working in a team.

We try to avoid toxic experts in our selection process for the school, but they slip through every so often. In response to this feedback, we will aim to be even more vigilant in our selection, and also make the aims of the project as a learning exercise more explicit during its introduction. We will also make sure to be more aware of group interactions during the actual school; we apologise to the students involved that we did not catch this behaviour this time. We are truly sorry.

If you are in the position of being an expert during a school or workshop, don't go it alone. That is a waste of your time, because you can do a programming project on your own whenever you damn well please. Slow down, and think instead about practicing your teaching and mentoring skills. They are also important in life, and, in many contexts, they are your responsibility.

You can access the full survey results here.

-- Juan, and the Organisers.

Summer School Announcement: ASPP Asia-Pacific 2018

Juan Nunez-Iglesias — Thu, 05 Oct 2017 03:21:30 GMT

The Advanced Scientific Programming in Python (ASPP) summer school has had 10 extremely successful iterations in Europe. (You can find past materials, schedules, and student evaluations at https://python.g-node.org/archives.) Now, thanks to the INCF, we will be holding its first iteration in Australia, to cater to the Asia Pacific region. (Note: the original ASPP will still take place in Europe next Northern summer; this is a fork of that school.)

Key details

The workshop runs January 14-21 at the Melbourne Brain Centre, University of Melbourne, Australia
topics include: git, contributing to open source software with github, testing, debugging, profiling, advanced NumPy, Cython, data visualisation.
hands-on learning using pair programming
free to attend (but students are responsible for travel, accommodation, and meals)
30 student places, to be selected competitively
application deadline is Oct 31, 2017, 23:59 UTC.
website: https://melbournebioinformatics.org.au/aspp-asia-pacific
apply: https://melbournebioinformatics.org.au/aspp-asia-pacific/applications (make sure you read the FAQ on that page)

Background

Two-and-a-bit years ago, Tiziano Zito asked me if I could join the faculty at the 2015 ASPP school in Munich (then in its 8th iteration). It turned out to be a fantastic teaching experience, and, more importantly, it was a fantastic experience for the students. Students selected for the school fit a certain profile, neither novice nor advanced. As such, you can be sure that if you participate in the school, you will learn a great deal. We teach tools that will immediately improve your scientific practice. I decided that I wanted to replicate the school in Australia. Now it is finally here!

Course outline

Who is eligible?

This school is targeted at Master/PhD students and postdocs from all areas of science. Competence in Python or in another language such as Java, C/C++, MATLAB, or Mathematica is absolutely required. Basic knowledge of Python and of a version control system such as git, subversion, mercurial, or bazaar is assumed. Participants without any prior experience with Python and/or git should work through the proposed introductory material before the course.

We have strived to get a pool of students that is international and gender-balanced, and have succeeded, with gender parity in the last four schools.

Prettier LowLevelCallables with Numba JIT and decorators

Juan Nunez-Iglesias — Tue, 14 Mar 2017 13:14:49 GMT

In my recent post, I extolled the virtues of SciPy 0.19's LowLevelCallable. I did lament, however, that for generic_filter, the LowLevelCallable interface is a good deal uglier than the standard function interface. In the latter, you merely need to provide a function that takes the values within a pixel neighbourhood, and outputs a single value — an arbitrary function of the input values. That is a Wholesome and Good filter function, the way God intended.

In contrast, a LowLevelCallable takes the following signature:

int callback(double *buffer, intptr_t filter_size, 
             double *return_value, void *user_data)

That’s not very Pythonic at all. In fact, it’s positively Conic (TM). For those that don’t know, pointers are evil, so let’s aim to avoid their use.

“But Juan!”, you are no doubt exclaiming. “Juan! Didn’t you just tell us how to use pointers in Numba cfuncs, and tell us how great it was because it was so fast?”

Indeed I did. But it left a bad taste in my mouth. Although I felt that the tradeoff was worth it for such a phenomenal speed boost (300x!), I was unsatisfied. So I started immediately to look for a tidier solution. One that would let me write proper filter functions while still taking advantage of LowLevelCallables.

It turns out Numba cfuncs can call Numba jitted functions, so, with a little bit of decorator magic, it’s now ludicrously easy to write performant callables for SciPy using just pure Python. (If you don’t know what Numba JIT is, read my earlier post.) As in the last post, let’s look at grey_erosion as a baseline benchmark:

>>> import numpy as np
>>> footprint = np.array([[0, 1, 0],
...                       [1, 1, 1],
...                       [0, 1, 0]], dtype=bool)
>>> from scipy import ndimage as ndi
>>>
>>> %timeit ndi.grey_erosion(image, footprint=fp)
1 loop, best of 3: 160 ms per loop

Now, we write a decorator that uses Numba jit and Numba cfunc to make a LowLevelCallable suitable for passing directly into generic_filter:

import numba
from numba import cfunc, carray
from numba.types import intc, CPointer, float64, intp, voidptr
from scipy import LowLevelCallable

def jit_filter_function(filter_function):
    jitted_function = numba.jit(filter_function, nopython=True)
    @cfunc(intc(CPointer(float64), intp, CPointer(float64), voidptr))
    def wrapped(values_ptr, len_values, result, data):
        values = carray(values_ptr, (len_values,), dtype=float64)
        result[0] = jitted_function(values)
        return 1
    return LowLevelCallable(wrapped.ctypes)

If you haven’t seen decorators before, read this primer from Real Python. To summarise, we’ve written a function that takes as input a Python function, and outputs a LowLevelCallable. Here’s how to use it:

@jit_filter_function
def fmin(values):
    result = np.inf
    for v in values:
        if v < result:
            result = v
    return result

As you can see, the fmin function definition looks just like a normal Python function. All the magic happens when we attach our @jit_filter_function decorator to the top of the function. Let’s see it in action:

>>> %timeit ndi.generic_filter(image, fmin, footprint=fp)
10 loops, best of 3: 92.9 ms per loop

Wow! numba.jit is actually over 70% faster than grey_erosion or the plain cfunc approach!

In case you want to use this, I’ve made a package available on PyPI, so you can actually pip install it right now with pip install llc (for low-level callable), and then:

from llc import jit_filter_function

The source code is on GitHub. Currently it only covers ndi.generic_filter's signature, and only with Numba, but I hope to gradually expand it to cover all the functions that take LowLevelCallables in SciPy, as well as support Cython. Pull requests are welcome!

SciPy's new LowLevelCallable is a game-changer

Juan Nunez-Iglesias — Sun, 12 Mar 2017 03:41:41 GMT

... and combines rather well with that other game-changing library I like, Numba.

I've lamented before that function calls are expensive in Python, and that this severely hampers many functions that should be insanely useful, such as SciPy's ndimage.generic_filter.

To illustrate this, let's look at image erosion, which is the replacement of each pixel in an image by the minimum of its neighbourhood. ndimage has a fast C implementation, which serves as a perfect benchmark against the generic version, using a generic filter with min as the operator. Let's start with a 2048 x 2048 random image:

>>> import numpy as np
>>> image = np.random.random((2048, 2048))

and a neighbourhood “footprint” that picks out the pixels to the left and right, and above and below, the centre pixel:

>>> footprint = np.array([[0, 1, 0],
...                       [1, 1, 1],
...                       [0, 1, 0]], dtype=bool)

Now, we measure the speed of grey_erosion and generic_filter. Spoiler alert: it’s not pretty.

>>> from scipy import ndimage as ndi
>>> %timeit ndi.grey_erosion(image, footprint=footprint)
10 loops, best of 3: 118 ms per loop
>>> %timeit ndi.generic_filter(image, np.min, footprint=footprint)
1 loop, best of 3: 27 s per loop

As you can see, with Python functions, generic_filter is unusable for anything but the tiniest of images.

A few months ago, I was trying to get around this by using Numba-compiled functions, but the way to feed C functions to SciPy was different depending on which part of the library you were using. scipy.integrate used ctypes, while scipy.ndimage used PyCObjects or PyCapsules, depending on your Python version, and Numba only supported the former method at the time. (Plus, this topic starts to stretch my understanding of low-level Python, so I felt there wasn’t much I could do about it.)

Enter this pull request to SciPy from Pauli Virtanen, which is live in the most recent SciPy version, 0.19. It unifies all C-function interfaces within SciPy, and Numba already supports this format. It takes a bit of gymnastics, but it works! It really works!

(By the way, the release is full of little gold nuggets. If you use SciPy at all, the release notes are well worth a read.)

First, we need to define a C function of the appropriate signature. Now, you might think this is the same as the Python signature, taking in an array of values and returning a single value, but that would be too easy! Instead, we have to go back to some C-style programming with pointers and array sizes. From the generic_filter documentation:

This function also accepts low-level callback functions with one of the following signatures and wrapped in scipy.LowLevelCallable:

c int callback(double *buffer, npy_intp filter_size, double *return_value, void *user_data) int callback(double *buffer, intptr_t filter_size, double *return_value, void *user_data)

The calling function iterates over the elements of the input and output arrays, calling the callback function at each element. The elements within the footprint of the filter at the current element are passed through the buffer parameter, and the number of elements within the footprint through filter_size. The calculated value is returned in return_value. user_data is the data pointer provided to scipy.LowLevelCallable as-is.

The callback function must return an integer error status that is zero if something went wrong and one otherwise.

(Let’s leave aside that crazy reversal of Unix convention of the past 50 years in the last paragraph, except to note that our function must return 1 or it will be killed.)

So, we need a Numba cfunc that takes in:

a double pointer pointing to the values within the footprint,
a pointer-sized integer that specifies the number of values in the footprint,
a double pointer for the result, and
a void pointer, which could point to additional parameters, but which we can ignore for now.

The Numba type names are listed in this page. Unfortunately, at the time of writing, there’s no mention of how to make pointers there, but finding such a reference was not too hard. (Incidentally, it would make a good contribution to Numba’s documentation to add CPointer to the Numba types page.)

So, armed with all that documentation, and after much trial and error, I was finally ready to write that C callable:

>>> from numba import cfunc, carray
>>> from numba.types import intc, intp, float64, voidptr
>>> from numba.types import CPointer
>>> 
>>> 
>>> @cfunc(intc(CPointer(float64), intp,
...             CPointer(float64), voidptr))
... def nbmin(values_ptr, len_values, result, data):
...     values = carray(values_ptr, (len_values,), dtype=float64)
...     result[0] = np.inf
...     for v in values:
...         if v < result[0]:
...             result[0] = v
...     return 1

The only other tricky bits I had to watch out for while writing that function were as follows:

remembering that there’s two ways to de-reference a pointer in C: *ptr, which is not valid Python and thus not valid Numba, and ptr[0]. So, to place the result at the given double pointer, we use the latter syntax. (If you prefer to use Cython, the same rule applies.)
Creating an array out of the values_ptr and len_values variables, as shown here. That’s what enables the for v in values Python-style access to the array.

Ok, so now what you’ve been waiting for. How did we do? First, to recap, the original benchmarks:

>>> from scipy import ndimage as ndi
>>> %timeit ndi.grey_erosion(image, footprint=footprint)
10 loops, best of 3: 118 ms per loop
>>> %timeit ndi.generic_filter(image, np.min, footprint=footprint)
1 loop, best of 3: 27 s per loop

And now, with our new Numba cfunc:

>>> %timeit ndi.generic_filter(image, LowLevelCallable(nbmin.ctypes), footprint=footprint)
10 loops, best of 3: 113 ms per loop

That's right: it's even marginally faster than the pure C version! I almost cried when I ran that.

Higher-order functions, ie functions that take other functions as input, enable powerful, concise, elegant expressions of various algorithms. Unfortunately, these have been hampered in Python for large-scale data processing because of Python's function call overhead. SciPy's latest update goes a long way towards redressing this.

I Love Symposia!

Introducing napari: a fast n-dimensional image viewer in Python

The background

The beginning

napari now

1. Just looking at NumPy ndarrays quickly

2. Overlaying computation results

3. Annotating data

4. Viewing very large (dask) arrays

5. Quickly looking at images

6. Parameter sweeps

7. Overlay polygons

8. More!

Our vision for the future

Join us!

Acknowledgements

Why citations are not enough for open source software

How to fund open source

Why you should cite open source tools

Summer school announcement: 2nd Advanced Scientific Programming in Python (ASPP) Asia Pacific!

Key details

Background

Course description

Who is eligible?

More questions

The road to scikit-image 1.0

Part one: where we are

Part two: where we're going

Our vision

Our mission

Our values

Part three: scikit-image 1.0

The human side

What do scientists know about open source?

Scientific culture is ridiculously secretive

For coding, scientists have even more severe impostor syndrome

Scientists don't know about open source

What can scientists in open source do?

1st ASPP Asia Pacific evaluation survey

Summer School Announcement: ASPP Asia-Pacific 2018

Key details

Background

Course outline

Who is eligible?

More questions

Prettier LowLevelCallables with Numba JIT and decorators

SciPy's new LowLevelCallable is a game-changer