Wednesday, August 31, 2005

Hubble begins "Two-Gyro" operations

NASA is reporting that engineers have shut down one of the three remaining working gyroscopes aboard the Hubble Space Telescope (HST) placing the spacecraft into two-gyro mode. This move is an attempt to extend the operating life of the ageing telescope by keeping one of the gyros in reserve, and should hopefully mean that the HST continues operations into 2008.

Sign up for Gmail

While I was away the Google Blog reported that you can finally sign up for Gmail without an invitation. Perhaps people will now finally stop asking me for invitations?

The real news however, is that you have to give them your mobile phone to sign up, and Russell Beattie has a similar take on things to me. It's a good move on their part to harvest all those numbers, but probably not a smart move on our part to hand them over without a serious think about the future consequences...

Apple rumours...

Along with just about everyone else, the BBC is reporting on Apple's announcement of a press event on the the 7th of September. The invite itself seems to hint that the announcement will be as crucial to the company future as the original iPod turned out to be, and this has raised the typical Apple rumour mongering to a new level of crazyness.

Simultaneously Motorola has announced a press event on the same day, and as Gizmodo comments this is rather odd behaviour...

As I've said before the Motorola E790 is not the long awaited Apple iPhone, but I'm still hopeful, surely this much press can't be devoted to a phone running iTunes mobile, surely the long delay means that there will be shiny Apple hardware as well?

Update: The latest rumours suggest that Apple and Motorola will be partnered with O2 here in the UK.

Update: Looks like O2 have accidentally confirmed the rumours...

Tuesday, August 30, 2005

I'm back

Just got back into the country, but I'm still on holiday, so you can't see me, I'm not here. Back to work tomorrow though, I'm definitely not looking forward to the email backlog that's going to have built up in my absence...

The wedding...
The honeymoon...

Thursday, August 18, 2005

On holiday...

For those of you who don't already know, I'm getting married at the weekend, and then I'm off on honeymoon. I'll be back in early September...

Thursday, August 11, 2005

Autonomous Underwater Vehicles

Last weekend saw the 8th annual Autonomous Underwater Vehicle Competition (via Slashdot and Engadget) which this year was held in San Diego.

The competition consisted of teams from 19 schools, colleges and universities competing for $20,000 in scholarship money. To win the vehicles need to complete three mission goals, completely autonomously, without any direct human supervision. These were to,
  • Rendezvous with a "docking station."
  • Find and mark a break in an underwater pipeline.
  • Home in on an acoustic beacon within a marked surface zone.
The winning team was from the University of Florida. However each of the top four teams managed to complete only one of the three competition missions. For their efforts, Florida was awarded $5,500, ETS $4,500, MIT $4,000 and Duke $2,000.

CREDIT: University of Florida
The winning entry, Subjugator...

However if you're interested in underwater vehicles, you might want to start with something a bit simpler that an autonomous robot and try to build a homebrew underwater ROV (via Slashdot and Engadget).

Tuesday, August 09, 2005

Landing at Edwards

Due to bad weather Discovery didn't land at Kennedy on schedule, and was diverted into Edwards Air Force Base a day late. So I didn't miss the landing like I thought I would, and I'm watching it live, via the web feed...


Mission Control Centre at Houston

Update: Discovery is now approaching the Californian coast and is 135 miles from Edwards...


Tracking Discovery inbound to Edwards

Update: At 35 miles and 4 minutes to go to touch down I've just lost the web feed. Presumably the server has crashed under the load, although the feed had been holding up well until this point.

Update: Successful touchdown for Discovery...


Down, safe, and in Infrared, at Edwards

Sunday, August 07, 2005

Discovery is coming home...

I'll be in the air somewhere over the Atlantic when Discovery is scheduled for touchdown, and despite my objections to the boondoggle that the Shuttle programme has become, or perhaps always was, I wish them well.

This was the first flight of the Shuttle since the Columbia disaster in 2003, and after the difficulties with this mission, it's possible it may be the last. However even if NASA does continue Shuttle operations, this flight marks at least the start of the end of the shuttle era. After this, nobody, not even NASA, is going to try and justify a long term future for the Shuttle...

Update: The landing was postponed...

In transit

Back in transit, and heading back into the UK. I'm taking the same route back as I did on the way out. So in a couple of hours I'll be heading up to Vancouver on yet another Dash 8, and from there, back to Heathrow getting in sometime on Monday afternoon.

Friday, August 05, 2005

Final keynote

The convention closed with a final keynote session hosted, as always, by the brightly clad Nat Torkington...


Nat Torkington in yet another shirt...

First up were the results of the HP Photo Contest, and I was amazed to find out that my photo of Brad had made it onto the finalist list. Although unsurprisingly, considering the quality of some of the other photos, I didn't win.

The conference closed with a talk from Miguel de Icaza talking about the migration to Linux and Open Office at Novell. He discussed the usability labs that Novell are doing, and how they're making extensive use of the Mono Project. According to Miguel,
If you're not running Mono, you're a moron.
and along with a number of digs at Mac OS X, I ran out of fingers trying to count them, he finished off with some impressive demos of the next version of Suse Linux.

...and that's a wrap. You don't have to go home, but you can't sleep here!

Hacking your home phone

I thought I'd go to something a bit off the wall for the final session of the conference, so I ended up in Brian Aker's talk on Hacking Your Home Phone System. He talked about his experiences with Asterisk, an open sourced Linux PBX system, and Sipura telephone kit. The talk was a bit US-centric for obvious reasons, but interesting stuff none the less...


Brian Aker and hacking the phone system

Easily Extensible Perl

I read his blog so I thought I'd sit in on Ask Bjørn Hansen's talk on building easily extensible Perl programs. I'm sort of hoping that I'm going to know all of this...


Ask Bjørn Hansen and extensible Perl

He recommend that you embed hooks through out your program and use a plugin architecture, and he's used his own SMTP daemon qpsmtpd, to illustrate his approach to plugin architectures. I sort of agree, plugins are a good solution for extensibility, and this is pretty much the approach I used to swap the data analysis algorithms in and out of scope for the eSTAR intelligent agents. It's not "the" solution however, and plugin architectures have a lot of overhead, although there are ways around this if you're willing to put the work in to properly encapsulate things, so it's interesting that he concentrated so heavily on this option.

Update: Brad was also blogging this sesison...

Friday morning keynote

First thing that struck me when I walked in, much smaller room. The movable walls of the room where the keynote is being held hadn't been, well, moved as much. Attendence on the Friday is down on the rest of the week, I guess with the exhibit hall closed today all those guys went home...

After the kick-off from Nat Torkington, the first speaker was Asa Dotzler the "community guy" for the Mozilla Foundation, who was talking about Linux on the desktop after his controversial post in his blog last month.

As I briefly mentioned on Wednesday, in response to one of the other keynotes, I think this is pretty much irrelevant now. The entire point of Web 2.0 is that the desktop isn't important any more. Why are we still worried about this? Asa made some good points, although I disagreed with a lot of what he said, but I'm confused as to why we're even talking about this...

Perhaps to follow on from last years discussion with Freeman Dyson about hacking the physical world, the next speaker was Drew Endy, from the MIT's Biological Engineering group, talking about Open Source biology.

He started off talking about genetic sequencing, apparently you can just go to to a company website, type in the DNA sequence you want, and they'll ship you the genetic material. That's pretty odd...


Drew Endy talking about open source biology

He went on to talk about genetic engineering and the trend towards patenting DNA, and the counter revolution of open source DNA, and the worrying legal issues surrounding this new technology which currently holds out so much promise. This was really interesing stuff, I think I'm going to have to start doing some reading.

The next speaker was Tony Gaughan from Computer Associates, who talked about their open sourcing of Ingres. I guess we've been lucky, this was pretty much the first corporate shill of the conference. If it wasn't obvious from their stand in the exhibit hall that Computer Associates just didn't get it, I think this talk cleared up any doubt. I haven't a clue what the guy said, my brain turned off.


On Evil...

The penultimate talk was given by Danny O'Brien, from the Need to Know site, who talked "On Evil", which was perhaps the funniest talk of the conference so far although he had some serious points. Mind you, these
Clowns have been at stage two, they laugh at you, for 400 years...
and
Ruby, who went from 'they ignore you' to 'you win' in three weeks...
weren't them.


Nat Torkington introducing Saul Griffith...

The final speaker of the session was Saul Griffith, from Squid Labs, who talked about hardware hacking and teaching kids how to build their own toys.

The Stonehenge party

Last night we were back at Ground Kontrol for the traditional Stonehenge party, and as usual, everyone had a great time. However Randal apparently gave out over a thousand t-shirts in the lead up to the event, so for those people who didn't have the sense to turn up really early, there was a bit of a queue to get in...

Video games and beer at Ground Kontrol

Thanks Randal, as always, a great party.

Extreme Perl Makeover

I browsed through a copy of his new book downstairs in Powell's, so I thought I'd attend Peter Scott's talk on legacy Perl applications. Normally when you use the word legacy in relation to Perl, you're talking about Perl as the solution to some horrible legacy problem. However as a mature language, there is also more and more legacy Perl code around. For some people Perl is no longer the solution, it's also part of the problem...

Amongst other things he recommends Perl::Tidy to clean up poorly styled code, or even to be run automatically at CVS check-in and check-out so you never have to see any other coding style other than your own. Cute idea...

Another good idea he suggested was a cure for "monolithic madness",
{
use strict;
[ ... ]
{
no strict;
[ ... ]
}
}
gradually migrating code from inside to outside.

Make: at OSCON

During the afternoon break I dropped by the O'Reilly stand in the exhibit hall where I came across Philip Torrone giving a demonstration...


Philip Torrone and the Make: demo...

Amongst other cool stuff, including some good Sony PSP hacks, he was having fun with lasers.

Thursday, August 04, 2005

I want a ponie...

I'm sitting listen to Nick Clark talk about Ponie, the project to allow us to run Perl 5 programs on top of Parrot, the virtual machine designed for Perl 6.

Perl 6 is a language and Parrot the virtual machine, whereas Perl 5 is both a language and a virtual machine. So we want to be able to write and run Perl 6 code, and run our existing Perl 5 and XS code linking to external libraries. In other words we want to be able to call from one virtual machine into another. We already have Inline, so why not just use that? It's not impossible, although it might mean some changes were needed to the Perl 5 internals. But it's not the best approach...

We need one single virtual machine, so can we compile Perl 6 down to Perl 5? Difficult, both threads and Unicode were bolted on to Perl 5 almost as an after thought and we want them to work well in Perl 6. How about compiling Perl 5 to Perl 6? This will almost certainly give us problems with existing XS code. So is there another way...?

There is only one Perl implementation, and the documentation might not specify behaviour. In reality the Perl 5 language is defined by the C source, with lots of corner cases. We need to present to all the Perl 5 code the illusion that it's running on the Perl 5 virtual machine, when it's actually running on Parrot.

This is what Ponie is all about...

Don't drop the SOAP...

After lunch I decided to go and see what Randy Ray had to say about the SOAP protocol...


Don't drop the SOAP...

Randy was giving his talk in response to an article a few months back which advocated REST services over the complexity of SOAP. Randy argues that you should not confuse complexity with detail, and gave a vigourous defence of SOAP, but I definitely sensed some reservations in the audience. I'm must admit to being confused. Like Randy, I view SOAP and REST as very different beasts, and I don't understand how people can advocate REST over SOAP when they're clearly useful for different things.

Update: Randy has a post about his talk on his blog...

More damage to Discovery

The BBC is reporting that following earlier repairs to the orbiter NASA has ruled out a fourth spacewalk to fix a torn thermal blanket below the cockpit window.

CREDIT: NASA
Windows Media | Real Video

JSAN

With all the buzz about AJAX I decided to go and see what the Javascript guys are doing with JSAN...

The killer application for Perl isn't actually an application at all, its always been CPAN, so it's interesting to see a Perl hacker like Casey West trying to take the concept to another language, especially one like JavaScript. JSAN looks good, but trying to get the CPAN concept working for Python seems to have stumbled and failed so I guess time will tell...

Thursday morning keynote

Hosted again by Nat Torkington the Thursday morning keynote kicked off with Nick Gall talking about TCP/IP and Shipping Containers and how to build freedom into software architectures.

...and here's Nat Torkington
Nick Gall talking about building free architectures

He talked about how software gets harder to change as it gets older due to bit rot. He argued that "internetwork" architectures can serve as an example for sustained freedom to change. An internetwork architecture should be: interoperable, composable, extensible, generic, federated and simple.

Interestingly containerised shipping can be fitted into this model, like HTTP which sits ontop of TCP/IP, containerised shipping sits ontop of existing infrastructure, the existing railways and ships, but revolutionised the way people worked.

He argues that you should design your software not only to do the job you want it to do, but also to be infrastructre for everyone else....

The next speaker was David Heinemeier Hansson who talked about the secrets behind Ruby on Rails. While there is no one thing you can point to and say that this is why its swept the community so quickly...


David Heinemeier Hansson and Ruby on Rails

Ruby on Rails is an integrated stack of web-application frameworks, it's a framework ontop of frameworks, in other words its,
A bunch of stuff that makes web developers happy!
Why is Rails interesting for several reasons. First of all, it follows convention over configuration, called the golden path, it does what most of the people want, most of the time. Secondly chanages are instant, it follows a
Use it, break it, fix it, trash it, change it...
paradigm. Finally, Rails ships as a complete, integrated, and familiar stack.
Using one language, all the time...
David claims that,
Flexibility is overrated and constraits are liberating
which is herecy for a Perl hacker like me, but the success of Ruby on Rails is hard to argue with...

Next up was Kartik Subbarao from HP, talking about Enterprise IT, who seemed to be trying to emulate Larry Wall's presentation style. Did his speaking style remind anyone else of Larry's, or was it just me?

Following on from Kartik we had Robert Lang who talked about Origami, this was pretty off the wall, but then so was last year's talk by David Rumsey on maps which got the only standing ovation of the conference. Can you see where I'm heading here?

Robert Lang talking about Origami

It seems that over the last fifty years the number of published Origami grew from about 100 to over 20,000 mostly because people have started applying mathematics to the subject, and more recently computer-aided Origami design. Interestingly there are practical applications to fold mirrors for space telescopes, and of course for solar sails.


The Japanese Solar Sail deployment

The the penultimate keynote was an Nat interviewing Mitchell Baker the chief lizard wrangler from the Mozilla Foundation...

Finally we closed with a talk from Dick Hardt called Identity 2.0 who discussed what defines identity,
What identity is, is what I say about me, and what others say about me... it's about trust
He talked about how today our identity online is constrained inside walled gardens, and how we'll need to move away from this model, and away from centralised identity providers and although the next logical step for the existing walled gardens is federation, and he argues that this is a poor model.

Update: Brad was also blogging the keynotes...

Copyright infringement

One thing I really dislike is that some people think they can just reuse content without permission and without giving credit. These guys for instance lifted some of my images without giving any indication that they didn't belong to them. I wouldn't mind that too much, except that they're directly linking to my site for the image and burning my bandwidth, that isn't nice.

Would a "CREDIT: Alasdair Allan" have killed them? That's all it would have taken to keep me happy...

Update: Looks like the photo is gone from their page, how odd!

Autonomous Robots

I work with agent technologies so the word "autonomous" immediately sprang out to me when I was glancing over the programme to see what I should be going to next, so in the end I decided to go to the Open Source Paths for Autonomous Robots session given by Joe Bosworth and Joe Monti from Smart Robots, Inc.


Autonomous Robots (Windows Media)

It looks like they're looking to gather a group of open source developers around their platform to work on autonomous systems and to make a serious push towards a standardised open source solution for self-learning systems. Interesting...

Update: The guys got a grilling over their choice of Java for their systems, okay, I might have had something to do with that. On the other hand it does look like they have a document literal XML interface at least at a higher level for interchange of information between the robots themselves. Again, interesting. Still, it's a pity about the Java...

Wednesday, August 03, 2005

The Conway channel, part 2

Tuning back in to the Conway channel, Damian is talking about twelve new Perl modules hes written over the last year, all but one of which are on CPAN, and the lst of which will be on by midnight, err, probably...

He kicked off talking about smarter IO and the IO::Prompt and IO::Interactive modules which look quite handy. Although perhaps not as handy as logging and the Log::StdLog module, which is what he talked about next.

Next up was Debug::Phases module which is an incredibly simple piece of code. Following that we had the the Leading::Zeros module, which I don't think I could ever find a use for...

However at the half way point we have Sub::Installer which allows you to reinstall a subroutine, now that I can find a use for...

Then we had a module to manipulate the pre-, post- and match magic variable called the Regexp::MatchContext, which looking at his CPAN directory looks to be the one that hasn't been released yet.

Following that Damian talked about Robin Houston's Want module, which didn't quite do everything he wanted, so he wrote Contextual::Return which will allow you to overload your routines to return different things in different contexts. This one looks amazingly useful stuff.

Next up was a discussion of the Exporter module, which Damian doesn't like very much, so he wrote Perl6::Export::Attrs.

Damian then said that having tried all of the configuration file parsers on CPAN, of which there are many, and he didn't like any of them, so he wrote Config::Std. Having basically to having had to do this myself at one point I could feel his pain, although his needs look very different from mine so it doesn't look like I can get rid of me awful hack of a config file parser just yet.

After writing a configuration line parser, he's also gone on to write a command line argument parser module, sort of like POD::Usage but on steroids. It's called Getopt::Euclid which uses your POD documentation to specify what your command line arguments should be...

The next module was Toolkit which allows you to specify a default list of modules and pragmas to using an AutoLoad, amongst other clever techniques, funnily enough that doesn't seem to be in his CPAN directory either, I thought there should only be one missing?

So what's the message? That Perl isn't the perfect tool to do the job, but it's the perfect tool for building the perfect tools for getting the job done...

What's new in Perl 5

After listening to the status of Perl 6, I thought I'd sit in on Nick Clark and Rafael Garcia-Suarez talk on the status of Perl 5. It looks like things are going fairly steadily. However one thing that did spring out at me is that as of Perl 5.9.*, the pack( ) function will work in character mode by default. Since I make fairly extensive use of pack( ) I'll have to figure out how this is going to affect me...

Oh, and there still isn't any progress in sharing objects across Perl ithreads unfortunately!

Perl 6: End-game

My first session of the conference proper was on the status of Perl 6 given by Damian Conway and Larry Wall,


Larry and Damian talking about Perl 6
Are we there yet...
the answer to which seems to be, almost...

The language design is undergoing simplification and consolidation, but there are a whole bunch of new features including three new yada yada yada operators,
while get_next( ) -> $next { ... } # fail
while get_next( ) -> $next { ??? } # warn
while get_next( ) -> $next { !!! } # die
go figure. Apparently we can also now do,
say $data.perl()
instead of
use Data::Dumper;
print Dumper( $data );
which got a big round of applause, and has to be a good thing...

Listening to Damian and Larry I'm reassured, Perl 6 is actually going to happen. We're close, we have prototypes and people are already being ported to Perl 6, so Damian is hoping that this time next year he'll be teaching people how to use it. Here's hoping...

Wednesday morning keynote

The Wednesday morning keynote is as usual hosted by Nat Torkington in his traditional blindingly bright shirt.


Nat Torkington kicking things off...

The first speaker was Marc Hedlund who launched O'Reilly's & SpikeSource's CodeZoo for Python and Ruby. What, no Perl. Oh, no, wait, we already have CPAN...


Nat Torkington and Tim O'Reilly

He then handed over to Tim O'Reilly himself for the O'Reilly Radar who immediately promised us he wasn't going to give the same Open Source Paradigm Shift talk as the year before. He then ran through a a few things that he thinks will be interesting, of which his top hit was Ruby on Rails
Ruby on Rails may by the Perl of Web 2.0...
He went on to talk about Greasemonkey and Findory, which attempts to do something along the lines of Amazon Recommendations. If you like this blog entry, you'll probably like this other one...

Tim claims that the thing that is cropping up around the computer industry at the moment is that users add value, and he seems to feel that this is a driving theme of Web 2.0. Next on the radar was open source telephony and,
If you're not thinking about internet telephony, you should be...
along with the transformation of radio and television. The interesting thing here is that, according to Yahoo! Research, the number two PVR behind Tivo is homebrew. Not what I'd expected...

After briefly mentioning Make: Magazine which O'Reilly launched at last years OSCON, Tim went on to talk about the computing book market.

He then handed over to Kim Polese, the CEO of SpikeSource, who talked about how companies are building things onto of the open source phenomenon and the long tail. There didn't seem to be much here that people inviolved in the open source of community wouldn't have heard before, she talked a lot about people become their own vendors and how there is a new generation of software companies emerging which are sitting between the software and the more traditional companies.


Kim Polese saying things we're heard before...

Apparently SpikeSource is all about solving the problem of velocity mismatch and testing, the ugly stepchild of software, and how to scale testing frameworks and automation of dependency testing. Definately an odd choice for an OSCON keynote...

Following Kim was Andrew Morton of OSDL, who talked about the commodisation of open source software and the comapnies are embbedding their own developers into the development teams of the software they have come to rely on,
Haven't seen much from Sun lately, but that might change when they finally give up on Solaris...
He also talked about Linux on the desktop, altough I'm not really sure why because its becoming increasingly irrelevant these days. Surely that's the entire point of Web 2.0, that the operating system running on someones desktop isn't particularly important any more. Disappointgly this presumably means that OSDL is still fighting the last battle, along with Microsoft and Sun. That's okay I guess, the rest of us can get on with building the future...

The next speaker was Jeremy Zawodny from Yahoo! who talked about why Open Source is a good thing, I'm confused, again surely we all agree about that? He did mention Yahoo's recent acquisitions of Flickr and Konfabulator which I thought was, well, interesting...

As an aside it looks like just as predicted yesterday the wireless network has collapsed under the weight of the assembled masses, there are apparently over two thousand people at this conference. Unfortunately it also took my PowerBook with it as all the networked applications I was running hung solidly, go figure. So I've had to return to basics, this post coming to you curtsey of vi...


Nat Torkington talking to Jonathon Schwartz

The morning keynote closed with Nat Torkington interviewing one of the most prolific commentors on open source Jonathon Schwartz about, amongst other things, the Apache open source Java project. Which, despite Jonathon's comments, I'm sure Sun is very upset about...

Update: Brad was also blogging the keynote speeches.

EVA to fix Discovery

The BBC is reporting that the additional EVA to repair Discovery has been declared successful by NASA.

CREDIT: NASA
Windows Media | Real Video

This, the third spacewalk of the mission, was the first ever in-orbit repair of the Shuttle heat shield. Mission Specialist Steve Robinson removed gap fillers that were protruding from two areas between heat-shield tiles on Discovery.

Update: Slashdot has the story...

The Tuesday Extravaganza

Brad and I are sitting about four rows back from the front after the various alarms to experience one of the highlights of the conference, the Tuesday Night Extravaganza. We kicked off with the Google-O'Reilly Open Source Awards...

Nat Torkington hosting the festivities
The Open Source Awards

Following the awards we have Larry Wall with his annual State of the Onion address and the "five year" plan for the People's Republic of Perl, which was amazingly off the wall, exceptionally eccentric and generally funny, as usual.

The State of the Onion

We then David Pennock had talking about the The Tech Buzz Game, which I'd never heard of, but apparently is a prediction market for high-tech products, concepts, and trends based on search volume at Yahoo!.


The Tech Buzz Game

The Tech Buzz Game v0.2 is coming soon, and will have an open source focus, who knew?

Following this we had one of my favourite dispensers of wisdom Paul Graham, author of Hackers & Painters, who talked off the cuff about open source software.


Paul Graham talking about Open Source

Paul discussed the difference between amateur and professional, and how with the arrival of blog the old model of channels is breaking down, and despite the fact that the average quality of writing online is pretty bad, the mainstream press isn't competing against the average writing but against the best,
The New York Times front page is a list of articles in the New York Times, del.icio.us is a list of articles that is interesting...
along with,
Live by the channel, die by the channel...
and on the traditional office environment, and fixed working hours, he had this to say,
If they can't make people work, they can at least stop them having fun!
which has to be one of the best quotes to come out of the day. Paul's pitch seems to be for a new model of investment in start-ups rather than the traditional employment model. Interesting, as always...

Following Paul we then had David Adler presenting the 2005 White Camel Awards before we moved on to last talk of the evening which was given by the irrepressible Damian Conway who gave a talk entitled "Fun with Dead Languages"


As Brad says, "All Damian, all the time"...

How anyone can can go from this,
There is one good thing about being a professional Lara Croft impersonator in Russia, you have much better access to real armaments.
to this,
Lisp might not be dead, but it's definitely pining for the Fijords...
in only a handful of slides, and make it seem natural, continues to bemuse and amaze me...

Update: Brad was blogging the session as well...

Fire and Alarms (various)

Arriving from downtown about twenty minutes before the Tuesday Night Extravaganza, Brad and I got to the conference centre only to find the fire alarm sounding, the building burning, and crowds of geeks standing around outside the building. On the bright side, sticking next to the doors meant we were guaranteed good seats when it was all over...

Various alarms...

Planet OSCON

Last night at the Blogging BOF, which weirdly enough I didn't actually blog, people were bemoaning that there was no easy way to aggregate all the OSCON blog entries. So overnight Casey West went and built one using the Planet aggregator code...

Tuesday, August 02, 2005

Making Programs Faster

This afternoon I'm in Mark-Jason Dominus' tutorial Making Programs Faster. He started off by saying that the main point of this class is performance tuning and profiling tools, and that he's going to talk about how to build custom profiling tools from Perl's darkest recesses, but apparently,
Performance tuning is really, really, hard...
and the main theme in performance tuning is that nobody guesses right, so guessing doesn't work, which is a bit of surprise because there is a lot of programming that you can get through thorough flying by the seat of your pants. You have to be scientific and methodical.

MJD then went on to talk about the Schwartzian Transform as a classic example of optimising things,
@sorted_names = 
map { $_ ->[0] }
sort { $b->[1] <=> $a->[1] }
map { [ $_, -M $_ }
readdir D;
as opposed to,
sort { -M $b <=> -M $a } readdir D;
where the former runs about 50% faster than the latter...

The goal of the Schwartzian Transform is to reduce the amount of system time, although at the expense of spending some extra user time.
Optimisation, like any engineering, is a trade off...
However, you have to be careful about what you optimise, because you might even up making things slower. Always remember to ask what you spend and what you get in return.

Wall clock time is the most understandable way to measure performance, but measuring wallclock time directly is really hard due to pre-emptive multitasking. So although the amount of CPU time used won't vary too much between runs, the amount of wall clock time can vary enormously. Because of this we normally concentrate on measuring CPU time because we can, rather than because we want to...

For CPU bound programs if you can reduce the amount of computation carried out then you can significantly increase the user experience. But what about programs which are IO bound? You have to address the network latency time, there isn't any point in reducing the amount of computation is doomed to failure. What to do instead? The only real solution is to parallelise the IO. The final case of memory bound programs, where the run time is dominated by swapping time, are probably the hardest, here the only way to optimise things is to reduce the amount of memory your code uses, which isn't always that easy.

So how do you measure time? Well of course most Unix systems come with a time command, and Perl has its own built in time( ) function. But since Perl 5.7.2 Perl has come with the Time::HiRes module,
use LWP::Simple 'get';
use Time::HiRes 'time';

my $url = shift;
my $start = time( );
my $doc = get( $url );
my $elapsed = time( ) - $start;
print "$elapsed second(s) elapsed.\n";

1.49982798099518 second(s) elapsed.
Time::HiRes also provides high-resolution versions of sleep and alarm.

There is also the built-in times( ) function which measures CPU time,
($u, $s, $cu, $cs) = times( );
which can form the basis of a simple benchmarking program although the benchmarking apparatus itself can bias your results and you need to introduce null tests. So you might be better advised to use Perl's own benchmarking module called Benchmark which runs its own null tests. However MJD doesn't use Benchmark.pm anymore because,
The numbers are all over the place...
Heisenberg said that it's impossible to measure something without altering the measurement, and that's certainly true of benchmarking. Every benchmark introduces some bias into the thing it measures, you can either try and minimise this or try and quantify it and correct for it, which is the Benchmark.pm approach. This doesn't always work well...

Profilers are really interesting, they allow you approach optimisation scientifically rather than by guesswork. They will tell you which parts of the program contribute the most to the run time. If the program is CPU-bound, you should use a profiler and find out which bits are slowest and then concentrate on those. Although of course you should always try and figure out whether it would just be cheaper to throw hardware at the problem.

Perl comes with its own profiler called Devel::DProf and a post-processor called dprofpp so that you can use to poke around inside your code, including XS,
although apparently we're going to learn how to write our own profilers later on in the tutorial. But right now, it's coffee time...

Update: Turns out Brad was sitting in the back and blogging the tutorial as well, and I hadn't realised till we broke for coffee...

Update: ...and we're back after coffee, chocolate and popcorn.

We start off with the 90/10 rule, which says that 10% of your code accounts of 90% of the run time, and if anything MJD says that this may be too conservative. So you should optimise the program concentrating on the the crucial 10%, but it's vitally important to test your optimised program to make sure you still get the same answer as before. Oddly enough this is a step which many people miss out, although of course optimisation may fix an outstanding bug, which can become confusing.

The other thing to bear in mind is that you have to know when to stop, you'll reach a point where it's just not worth the effort to optimise any further. People waste a tremendous amount of time on performance improvements that aren't actually worthwhile, a lot of time, buying new hardware is a lot more cost effective and more likely to be successful.

At this point MJD went off and talked us through the optimisation of perldoc, which was a useful real world example of how to go about this stuff...

Moving onwards, another handy profiling tool is Devel::SmallProf which measures the contribution perl line rather than per subroutine like Devel::DProf, although the output format is somewhat opaque, and MJD claims it's actually easier to write your own profiler using the hooks intended for the Perl debugger. If your module is called Devel::* and it has a DB::DB( ) function then you get access to various magic variables...

At this point MJD talked us through how you can do this, and it doesn't actually look too bad, and he's right the output from his simple profiler is a lot easier to understand. Who'd have thought it?

He closed the tutorial by talking about optimisation blunders, using pseudo-hashes as his main example. Ouch!

Update: Brad had this say about the tutorial post-coffee break...

Perl Best Practices

Unlike me, this morning Brad decided to go to Damian Conway's tutorial Perl Best Practices. He wrote it up fairly thoroughly in two parts [1, 2]...

Mighty Mouse

As predicted earlier in the year, Apple has just released a multi-button mouse. It looks pretty nice, but I guess I'll wait for the Bluetooth version...

CREDIT: Apple
The Apple Mighty Mouse

Update: Russell Beattie has managed to get his hands on one, and has the first hands-on review of the mouse I've seen so far...

Update: Russell gets Macdotted after reviewing the new Mighty Mouse.

Learning AJAX

My first tutorial on the second day of OSCON is Learning AJAX given by Alex Russell. I've been looking forward to this one, every year at OSCON I pick one tutorial on sheer interest alone rather than on whether it looks like it'll actually be directly useful. Last year I went to Cross Platform Rendezvous given by Stuart Cheshire from Apple, and while it wasn't immediately useful it gave me a lot to think about and certainly influenced the direction I took eSTAR over the last year, I'm hoping for the same sort of thing out of Alex's AJAX tutorial.

Update: Laura Thompson is also blogging Learning AJAX...

Update: Interesting stuff, a lot to think about, although I think I'm going to stick with my current plan which is to use perlVNC in nefarious ways rather than go with AJAX. What am I going to do with perlVNC? I'll tell you when it's done...

Wet string...

The wireless network at OSCON is working about as well as wet string this morning, I'm getting a steady dribble of a few hundred bytes per second. I could probably do better falling back to GPRS roaming, so I'm not looking forward to tomorrow when several hundred more people turn up...

Geotagging

Matt Croydon just published an article about the KML project he's been working on for Python for Series 60, including a link to his simple KML Python library. Looks interesting, wish I could justify buying myself a Bluetooth GPS.

However since I'm at OSCON I'm doing more than the usual amount of blogging, so Matt's post immediately tempted me to brush off a bunch of geotagging links which have been sitting in my edit queue for a month or more, in some cases anyway, here goes...

The first stop for anyone thinking about geotagging has to be the excellent MAKE magazine HOW TO which explains how to integrate Flickr, Mappr and Google Earth with GPS data.

Integration with Google Earth is certainly the way to go, for instance the thing that started Matt thinking about his new KML project was the integration of Google Sightseeing with Google Earth.

Rui Carmo has some suggestions, and some snippets of PHP, for tagging Google Earth, but there are lots of other Google Earth hacks floating around especially in the wake of the O'Reilly Where 2.0 conference in June.

Off course unless you can integrate real GIS data there is only so far you can go, but a good example of the powerful stuff that these hacks allow you to do is the fact that can so this, for instance drawing the postcode boundaries on top of the map data.

So there you go, a few interesting geotagging links that, so far, I haven't had the time to do anything with...

Tips for attendees

I agree with Brad, turn off your cell phones. But if you forget, and your phone does ring, for the love of God don't answer it! I'm totally amazed that anyone could so impolite as to do that when someone else is speaking...

Running into people...

Funnily enough I keep running into people I know at OSCON. Notably, since I haven't see him for a couple of years, I ran into Randy Ray at lunch time, but as he mentioned in his own blog, I hardly recognised him when I saw him. It's amazing how the lack of a beard can change someone's looks so much...

Shuttle emergency repairs

The BBC is reporting that NASA has now scheduled emergency repairs to Discovery to fix damage done to the orbiter during launch...

Update: Slashdot have picked up the story...

Update: More from BBC News...

Update: The repairs have been declared successful by NASA...

Slashdot sinks to new low?

The latest post to Slashdot is entitled "Hackers Forced Announcement of 10th Planet Find" and with that, Slashdot, once somewhere to go to get a geek orientated view of the world, has sunk to a new low. I won't even start on the supposedly sensible members of the main stream press. Does nobody understand the scientific process anymore?

Monday, August 01, 2005

Open Source Data Warehouses

My second tutorial for today is Building Open Source Data Warehouses by John Paul Ashenfelter. I'm not entirely sure how relevant this is going to be as he seems to be pitching heavily towards the industry guys, and putting things in their terms, however since just about all of the backend databases in the Astronomy community are data warehouses rather than the more common transactional style databases, I should be able to get something out of this...

Update: I'm about an hour into the tutorial, and I'm a bit worried as John seems to be giving a talk about data warehousing rather than giving a tutorial. The two things are very different and I'm not entirely sure he understands this...

Update: I'm not getting as much out of this as I hoped, I usually get one busted tutorial out of the four at OSCON, I guess this is this years. Pity, but tomorrow I've got Mark-Jason Dominus talking about Making Programs Faster and Alex Russell talking about Learning Ajax. They should both be good, looking forward to them...

Steganography with Flickr

I've just come across Keith McDuffee's article on Steganography with Flickr (via the Unofficial Apple Weblog) which looks interesting since, not only is your distribution channel is in plain sight and a broadcast medium at that, your content is disguised amongst a good deal of extraneous noise. Of course now that Flickr knows people are thinking about this, JPEG's which are "just a bit on the large side" for their actual pixel count might start getting flagged automatically by their software. Which isn't necessarily a good thing...

The Conway channel...

OSCON opens with two days of tutorials, and my first is Presentation Aikido by the irrepressible Damian Conway talking about how to talk to people.
If you're really passionate about it, it's mesmerising. Find a way to be interested in your material or your audience won't be... if at all possible, be entertaining. Entertaining always trumps informative, if you're not entertaining people will switch off...
Damian is one of the best public speakers I've ever come across, so I'm sitting here hoping he'll tell me how he does it...

Update: You have to love this quote from Damian,
From then on, whenever they think about the many worlds theory of quantum physics, they think about stoned physicists...
Update: Another great Damian quote with much truth in it,
Having great information on a shoddy looking slide is worse than having shoddy information on a great looking slide...
although I'm a bit unsure about,
...steal from the French!
Update: Damian is recommending that you should neither be arrogant, or nervous, the presentation is not about you, it's the ideas that are important. He admits that this is hard, but...
The goal is not to be there at all... be natural, don't try so hard that it shows...
Update: Brad is also blogging Damian's Presentation Aikido, and he's much better at note taking than I am, although his post comes in two parts...

Update: In closing,
The only way to speak better is to speak...