Monday, November 17, 2008

Worryingly senior...

Astronomy is one of the more computing intensive of the sciences, and historically we've pushed the boundaries of the available computing resources. But we're also dependent on a thinning cadre of dedicated hero programmers...

Our ability to provide comprehensive software suites to our users hinges on our ability to hire staff experienced in both scientific data analysis and software engineering... In the absence of such people, much larger teams containing both astronomers and industry programmers under formal project management need to be formed. - Economou et al. 2004
However with the data reduction systems and the telescopes themselves becoming more and more automated replacements for those hero programmers are becoming hard to find because of the lack of experienced developers with an appropriate astronomical background, and the fact that it's not really a respectable profession...

We have found it extremely hard to hire good people to work on astronomical software. There is no career path within the universities for software specialists, despite the fact that there's no logical distinction between building hard- and soft-ware instruments. Smart and sensible graduate students, desirous of a career in astronomy, simply don't choose to specialise in the software required to reduce modern observational datasets. - Lupton et al. 2001
Which of course is the reason we're having to replace those hero programmers in the first place, without some sort of established career path the astronomical software community is suffering from 'leakage' around the edges. My own situation is typical, I'm generally described by faculty as a "worrying senior" fellow.

Despite industry-led criticisms of the hero programmer paradigm, such software-scientists are a required. Building complicated bespoke systems to do science takes domain knowledge, not just of software engineering, but also of the underlying science behind what you want to accomplish. Simply put scientists, and the institutions the employ them, can't afford to support the large structured software teams that would be necessary if those hero programmers didn't consitently punch above their weight. Scientists also generally aren't that keen to get involved in the software design process made necessary by more formal processes that larger teams would entail.

Unfortunately historically those same scientists have been reluctant to provide the necessary support and career advancement that would be required to keep people like me around, sometimes through a misguided belief that software is easy and robust software can be produced by any wet-behind-the-ears graduate student.

While there is of course a huge oversupply of hopeful candidates for any long term posting in astronomy, but if you talk to software-scientists at those watering holes where we usually congregate, like the recent ADASS conference in Quebec, you'll find more than the expected amount of doom-and-gloom going around. My situation isn't unique, I'm not the only worrying senior programmer living contract-to-contract...

Of course up until recently, despite our complaints, it's been other people worrying how senior we've become, not us. Most of the programmers that have managed to stick around inside academia for any length of time, and there are many that just come and go, are usually fairly good at what they do. That means they knew they could go out and get a 'real job', probably paying more than they were earning in academia, when or if it came to it...

Unfortunately, amougst other things, the current economic turmoil has taken away our comfort blanket and left us very much out in the cold. Although, perhaps, with a better winter coat and a set of decent boots than many these days. None the less, its not a situation that's going to encourage people to specialise in software.

I don't see any of this changing in the near future. In fact I see the situation getting worse, the current generation of students are further away from the software, and underlying hardware, than I've ever seen. A culture of black boxes is very much in evidence. But you have to ask, what happens when the black boxes break?