Heading home from OSCON Español | Deutsch | Français | Italiano | Português

So I'm sitting in the lounge at PDX waiting for my flight to Chicago O'Hare and then onward to Heathrow. However I'm not entirely sure how far I'm going to get after I land back in the Britain. Looking at the news coming out of the UK the rail system doesn't look like its recovered from the heavy flooding last week while I was travelling to OSCON.

The BBC's flood mashup (via Google Lat Long Blog) looks a bit worrying as well. To get from London to Exeter I have to go through Reading, which doesn't seem to be in a good way right now, and with more rain predicted while I'm enroute to Heathrow I'm pretty much reduced to crossing my fingers at this point...

Update: Made it home in one piece, with the trains more or less running to time, unless you wanted to head north out of Paddington towards Oxford. In which case you were out of luck as the lines were still underwater. Although looking at the countryside as I passed through Berkshire, the best description I can come up with is "water-logged". It probably wouldn't put much more rain to start another round of flooding...

OSCON: Final keynote Español | Deutsch | Français | Italiano | Português

The final keynote, and last session of the 2007 Open Source Convention is on Open Source Hardware and it's being given by Philip Torrone, the senior editor of Make: Magazine and Limor Fried who is according to Nat is the only person making money out of open source hardware right now.


Nat Torkington at his last OSCON
We're kicking off with Nat talking about next year, this is his last year working on OSCON, he's been doing this for eleven years and we're going to miss him. Although possibly not his shirts...


Philip and Limor are talking about open hardware
Update: On to the keynote talk and Philip and Limor are talking about open hardware, and how the field is starting to take off...


The Make: Board
interestingly it looks like people are starting to introduce new licences, the Chumby HDK License and the TAPR Open Hardware License because unlike software hardware is mostly based on patents, not copyright, and different licenses may be better.

Update: They've moved on to talking about Fab@Home, which is pretty cool, because once you've built one of these they can actually self-fabricate and duplicate themselves...

Update: Okay, I desperately need this, they're talking about Botanicalls which allows plants to place phone calls for human help, for instance when they need watering...


Botanicalls, plants that ask for help...
Update: They've moved on to talking about WaveBubble an open source RF jammer, which is certainly illegal to sell, and probably illegal to use, but probably not illegal to self-build.

Update: ...and we're done.

Update: Philip and Limor's keynote is now on blip.tv, and the slides are available for download.

Click To Play via Blip.tv

OSCON: OSS Amateur Robotics Español | Deutsch | Français | Italiano | Português

Next up is Amateur Robotics given by Mark Gross from Intel.

CREDIT: Mark Gross

Mark's entry for the SRS Robot Magellan contest
Update: He's talking about the on-board computing system for the robot, which is running a hacked NSLU2, which is pretty cool, as I've got one of those kicking around somewhere.

Unlike the previous talk, which was at about my level of hands-on with hardware, this stuff is well beyond my level of competence. I'm a software guy with delusions of grandeur, but I know my limits...

Update: More on the NSLU2-Linux and the Open Embedded projects.

Update: His slides are available, although they're in Open Office format, which isn't amazingly useful. But there is a lot more information on his website.

OSCON: Hack the Real World Español | Deutsch | Français | Italiano | Português

I'm in Hack the Real World with Open Source and Microcontrollers given by Brian Jepson from Make: Magazine.



Brian Jepson talking about hacking the real world
He's talking about the Make Controller Kit and Arduino boards, and despite being from Make: he's focusing on the Arduino boards because it's "super open". The main difference is that the Arduino doesn't have an operating system, unlike the Make: board, and it's "...a lot like the old days".

Update: He's talking about the different types of 'serial', and just mentioned the MAX232 chip which does conversion between RS-232 and TTL serial.

Update: He's talking about the Parallax RFID module which is a great module, cheap, and the pins plug right into a breadboard.

Update: This is pretty cool, he's pushing through some code examples of how to use the Parallax RFID module with the Arduino board, which apparently come from Making Things Talk (via amazon.com) by Tom Igoe.

Update: He's talking about a bunch of different sensors, including the pretty interesting volatile organic compounds sensor from FigaroSensor.com. Also interesting is force sensing resistors from Trossen Robotics.

Update: He's also talking about a Serial to Wi-Fi widget, with a simple telenet level interface, about US$90 from SemiConductorsStore.com.

Update: There is also a GSM module, with embedded Python, for US$150 from SparkFun.com, and a GPS module for US$70 from Parallax.com.

Update: Apparently Brad was sitting somewhere in the back.

OSCON: Friday Morning Keynote Español | Deutsch | Français | Italiano | Português

The Friday morning keynote kicked off with Philip Rosedale the CEO of Linden Lab, who is here to convince us to go to work on their newly open sourced second life client. He's doing a demo of the new first look viewer with the inbuilt voice client, and oddly he's doing a slide presentation inside Second Life, which is displayed on the projector instead of "real" powerpoint. Which is a bit bizarre...




Philip is arguing that Second Life has to, like the Internet become profoundly open. They're going to open source their server software and allow people to run and host their own server, and they've already open sourced their client.

Update: Philip is talking about the Second Life in a web browser client that turned up on the Teen Grid a few weeks ago...

Update: Next up is Jimmy Wales one of the founders of Wikipedia, who is talking about his new project Wikia. Which, cutting through the hyperbole, looks like a site where you can set up and host your own wiki. He's talking about trying to open source search, and Wikia Search, which he's referring to as the LAMP stack for search. Which is fine as far as it goes, but who's going to pay for the hardware? I think Tim O'Reilly even talked about this earlier in the week, even if you had Google's software, you couldn't rebuild Google's service because you don't have their hardware or their back end distributed database.

One of the thing about modern cryptography is that nobody actually breaks codes anymore, they just prove that in theory it's vulnerable to certain attacks. As far as I can see Jimmy is basically pushing the same sort of agenda here, theoretical search. If we have a billion dollar data centre, we could build a search service?

Update: Interesting, they've just acquired Grub, a distributed crawling service. Oh, that's not useful, the Grub client is Windows only...

Update: The next speaker is Simon Wardley from Fotango, who spent some time apologising for his Britishness, before kicking off and talking about the commoditisation of software.

In the 1800's... electricity engineers were the Spice Girls of their time

Update: You have to give the man credit for mentioning Yak shaving. He's talking about open source standards and providing competitive utility market at all levels of the stack,

...it's not good for the planet, and that annoys me because it's not good for Ducks

Update: Next up is our very own Nat Torkington giving the open source movement some therapy, and gaving a talk that was just too funny to blog properly...

Perl is the middle child that isn't getting any attention any more. "Why don't you love me any more?" says Perl? "Look, I've rewired the car..."

Python should get drunk, get laid, and shut up...

Don't think of open source as projects, think of them as people...

Update: It's all about people,

Most people are morons..

Nat's saying that it's easy to be nasty, and the corollary is that it's hard to be nice. This was a great talk for Nat to close out his last OSCON on...

Update: Our final keynote speaker is James Larsson, who according to Nat embodies the hacker spirit, who is going to Pimp My Garbage...



...who it turns out is obviously clinically insane, but in a great way.

Update: ...and got the best reception of the conference. My guess is that a lot of this was for talking about his project which was intended to be,

A computer vision system that takes the drudgery out of boot fetishism

I guess you had to be there...

Update: ...and we're done!

OSCON: State of the Onion Español | Deutsch | Français | Italiano | Português

So slipping into the back of the room at the tail end of the Perl Lightning Talks, and caught Pudge performing Perl In a Nutshell.



I'm now sitting in the Perl Foundation Auction. Its got to be one of the few auctions you'll ever attend where the auctioneer heckles the goods he's trying to palm off on the unsuspecting public, and boy do they have a lot of books and t-shirts to auction off this year...

...anyway, we have a book on it, who want's it?

...hardbacks! You can hit people with these!

Update: ...and we're done with the auction. Now it's time for this year's State of the Onion given as ever, by Larry Wall, and this year entitled "Programming is hard, let's go scripting!"

All language designers have the idiosynchrosies, I'm just better at it than most... - Larry Wall

...and we're done. Larry's talk was, as always, really funny and entertaining, but totally un-blog'able. Maybe Chris did a better job?

OSCON: Prototype and Object.prototype Español | Deutsch | Français | Italiano | Português

After the afternoon break, and we're back with Prototype and Object.prototype: JavaScript Power Tools given by Amy Hoy. As Mark mentioned earlier in the day, there a bunch of different AJAX toolkits, and Prototype is one of the bigger players.


Amy Hoy talking about prototype
Javascript is a real language, and everything is an object, really, everything. Even strings are objects,

var string = "This is a string"
string.length;

It relies very heavily on functions, while it does have objects, it doesn't have classes. It's a prototype based language...

Update: It's interesting sitting in Javascript talks, the people giving them sound the same way people did giving Perl talks five years ago. They start off justifying their language of choice, reassuring you it's a real language and explain why it can do cool stuff. Thinking about it, there are a bunch of ex-Perl hackers working on AJAX related stuff as well. Interesting, don't you think? These people are definitely drinking the Kool-Aid.

OSCON: Body Hacking Español | Deutsch | Français | Italiano | Português

I'm in Quinn Norton's talk on body hacking and functional body modification. It looks like O'Reilly have toally underestimated how popular this was going to be, it's packed, forget standing room there isn't breathing room.


Quinn Norton talking about body hacking

You are the platform...

This isn't going to be your average OSCON talk, there will be warnings on the slide before if there is going to be blood on the next slide. I've been trying to keep this blog safe for work, so we're going to have to see how much of this I can actually blog.

So what is body hacking? It's "...acting on yourself, with or without assistance, to enhance the function of your body or your perceptions". Stealing Make:'s motto,

If you can't open it, you don't own it

Update: She's talking about getting getting a rare earth magnet embedded inside one of her fingers, near a nerve bundle, to allow her to actually sense magnetic fields. The key factor here is neuroplasticity, allowing parts of your brain to repurpose itself. As time went on she could could sense live wires, spinning hard drives and phone chords.

It's like sticking your hand inside an ultrasonic cleaner

But then, "..who has actually done that?".

Update: Now she's showing us why you don't want to do this, shortly after she had it implanted the sheathing around the magnet breached and bad things happened.

Update: She's talking about Amal Graafstra who had an RFID chip implanted in his arm, and about how there is a theme of control.

Update: Moving on Quinn is talking about CT scans and using the the available open source software to take control, and about the UK expert patient programme, which I hadn't heard of before. She's arguing that there is a fine line between expert patients and body hackers. For instance; glucometers, blood surveillance, brain monitoring and reading all of this live to the internet.

Update: Apparently there are now consumer EEGs coming onto the market, and these are at the point where you can actually figure out facial expressions and emotional responses. This is pretty amazing stuff.

Update: Interesting, she's talking about experiments with a directional sense, integrating a GPS and a buzzer allowing you to always know where north is, comments from the audience indicate that the US airforce have been doing this in their flight jackets since the sixties.

Update: Interesting stuff here Provigil is a stimulant that lets you not sleep. But there is no downside as the side effect profile is minimal. You still want to sleep, and you still feel like you want to, but unlike normal you can actually still function while in sleep deprivation. Then there is CUV1647, which gives a tan, makes you loose weight and increases your sex drive. The company that's made it is desperately looking for a disease, because you can't get a drug approved just because it's good for you.

Update: So whether we call it enhancement or treatment matters for social acceptable, but doesn't actually reflect on the procedure itself. The problem with this is time scale, we used to have generations to adjust, now we have decades at best.

Update: Best quote of the day perhaps?

Amateur brain surgery, sounds like a bad idea...

Update: She's closing out with "the next open vs propriety debate". Apparently there is a gene which can be used to test for breast cancer, but the gene is patented, so there is only one company in the world that can do the test and they charge US$1,200 and the test has to get mailed to Utah to get processed because that's where the only lab that has a license is located.

Update: The question she's asking is "How far is too far?" and "What counts as Human?".

OSCON: Ajax and Web Services Español | Deutsch | Français | Italiano | Português

I'm in Mark Pruett's talk on Ajax and Web Services. He's defining an Ajax application as a web service client that runs inside a web browser, and he's going to be talking about both REST and SOAP services.


Mark Pruett's talk on Ajax and Web Services
Mark argues that people go through an evolutionary path in the way they use Ajax, they start off sending back a chunk of HTML, they move on to sending back delimited text, then XML and finally some people move to JSON.

There a bunch of client side Ajax toolkits: prototype, Dojo, GWT (Google), Atlas (Microsoft), Rico, Zimbra, DWR. Some of these, like GWT and DWR are tightly coupled with the server side language. You probably want to pick a toolkit that doesn't do that...

Update: He's talking about the whole SOAP vs REST debate, and arguing that externally you should expose your services via REST, but internally SOAP might be more appropriate, or at least more frequently used. Which isn't necessarily the same thing.

Update: REST uses GET, PUT, POST and DELETE. Where GET will get data, a PUT will create a new resource, POST will update an existing resource and DELETE will delete an existing service. Although of course there are certain limitations to GET which means that you sometimes have to use POST anyway.

Update: He's just totally dismissed synchronous REST requests, arguing that you should always use asynchronous requests. Which seems like a lot of overhead for simpler services. Although he's also arguing that you shouldn't bother with XML, but you should be using JSON instead, which means you can do this,

var my_json;
my_json = eval ("(" + http_request.respinseText + ")");

which I think is probably officially evil.

Update: He's moving on to talk about the cross-domain problems. This is something we've all run into before, because of course you can only do Ajax calls to the server that delivered the original page. One way around this, which actually hadn't occurred to me before, is to use Apache ProxyPass rules and redirect calls to other servers transparently to your client side Javascript.

OSCON: Error Handling in Ajax Español | Deutsch | Français | Italiano | Português

Next up, I'm in the Error Handling in Ajax session given by Anthony Holdener, who seems to be running late.

Update: Except that I'm now not. Nat just walked in and said that the speaker had mailed them months ago to say that he wasn't going to make it and the session was cancelled, except that nobody had actually got around to cancelling it. He didn't look that happy...

OSCON: wxPerl: cross-platform GUI design Español | Deutsch | Français | Italiano | Português

I'm in wxPerl talk with Eric Wilhelm from Scratch Computing. wxPerl is the binding for the wxWidgets library, and unlike Perl/Tk the standard dialogs and other shiny stuff all look like should do on the platform you're running the application on...

Update: He's talking about how to get wxPerl to play the Perl game, and his wxPerl code for his application dotReader, to actually look like Perl rather than wrapped C++. Since I actually want to listen to this, why don't you go off and read Chris and Brad's coverage, they look like they're typing faster than I am anyway...

Update Eric is pointing us at wxPerl::Constructors and wxPerl::Styles both of which should add a decent Perl-like convenience layer on top of the

Update: He's now pointing us at wxPerl::MenuMaker for a decent menu introspection convenience layer.

Update: Other useful modules are in the Wx::Perl and Wx:: namespace rather than than the wxPerl:: namespace, don't you love CPAN? Wx::Mozilla plugs a Mozilla canvas inside your Perl application, but WebCore and Wx::WebCore is the way of the future.

Update: He's using PAR for distribution on Windows and Linux, although this has some problems. But on a Mac a graphical application has to live inside a .app bundle, so you need to look at PerlWrapper and his own ExtUtils::MacMaker, which he hasn't released to CPAN yet, if you want to release in a sticky lump rather than a scatter of Perl files.

OSCON: Thursday morning keynote Español | Deutsch | Français | Italiano | Português

So I seem to have rolled into the keynote late, which is pretty odd since I arrived at half past eight, and the programme says the keynote doesn't kick off until quarter to nine today. Ben Fry is currently up front he talking about the Processing Development Environment. This looks like a pretty interesting, and is apparrently the library underlying those cool air traffic visualisation I talked about back in March.



Update: Next up is Robin Hanson talking about Overcoming Bias who kicked off by saying,

I'm going to tell you things that you already know...

and did indeed proceed to do so at length, without the aid of power point. This could have been a really interesting talk with the right speaker, Paul Graham perhaps? As it stood, he didn't really set the audience on fire, and that was disappointing, because with this audience and this talk, he really could have...

Update: Next up is Bill Hilf talking about open source at Microsoft, who said the expected things and made the expected justifications, although admittedly in interesting ways.

Update: Now we have Rick Falkvinge, the founder of the Pirate Party in Sweden, talking about Copyright Regime vs. Civil Liberties. Funnily enough, he got a lot bigger round of applause and a big "Arrrr!" from the audience.


Why does the Pirate Party exist? The file sharing debate tends to focus on economic principles. But copyright is a commercial monopoly, and means something very different today than it did when it was introduced. Copyright is commercial, but touches everywhere, and people are now being monitored for violations it's become a civil liberties issue,

...so society must choose, privacy or copyright

If you're being monitored, in comes self censorship. If you don't have the right to private communications when you're in private you loose your identity. The copyright problem isn't about money, it's about viral civil liberties.

Politicians don't understand this...

The solution is to bypass the politicians and go directly to the voters. It's very obvious sitting here in the back that Rick's arguments have a lot more support in this audience than Bill's, and you have to take that seriously because most of the people in this audience make their living of intellectual property. Think about it, if the content creators don't favour copyright and patents any more, why should anyone else?

Update: The final keynote of the day is How to Ignore Marketing and Become Irrelevant in Two Easy Steps by Steve Yegge from Google,

Google will probably fire me for this talk...

He's talking about branding, and the fact that obviously you want people to use your software. He's kicked off with the whole New Coke and Coke Classic marketing disaster. They introduced it because the blind taste tests told them people liked New Coke a lot better, but that flipped around as soon as they saw the label.

...it was like introducing a new God.

He's giving a great talk here, especially since he's facing the ultimate presenter's nightmare as his powerpoint isn't displaying on any of the half dozen screens scattered around the hall. Looks like the technical crew has given up trying to get it to work as well, which is a shame, if he's this good just free wheeling then he probably would have been great with added powerpoint.

Now that's interesting, if your service is awful today, and perfect tomorrow, then apparently studies show that it takes an entire generation to change the perception of your brand.

Update: Both Brad and Chris were covering the keynotes.

OSCON: Machine learning with Perl Español | Deutsch | Français | Italiano | Português

The last talk of the day is Machine Learning Made Easy with Perl with Lino Ramirez. When I noticed this one on the programme I immediately put it in the "must go to" pile as, of course, for my day job I build autonomous agent based systems to manage intelligent robotic telescope networks.

Lino says,

It's all about empowering people

You can think about machine learning as a three phrase process: preparation (definition, gathering, analysis, cleaning, selection), modelling (selection, development, evaluation) and implementation (evaluation, implementation). After implementation there should always be a way for the system to keep learning.

Update: He's currently going through an example using stock quotes, talking about clustering of data. I'm having flashbacks to yesterday's data mining tutorial. If you're interested in following along, apparently the code will be available for download after the talk.

Update: We've been pointed at "Knowledge-Based Clustering" by Witold Pedrycz as a good text on clustering algorithms. I think I've actually got this on my shelf at home, but I'm not sure I've ever done more than flip through it, I'll have to dig it out when I get back into the UK.

Update: Apparently he's a PDL and PGPLOT user. I'm feeling right at home here, this guy is definitely a an academic...

Update: His next (and final) example is based on medical diagnosis, and he's talking about support vector machines classifiers and his PDL implementation of the technique. I'm going to have to mail him at the end of this and get the source code to this, or persuade him to tidy it up and release it onto CPAN.

Update: If you want to code and the slides from this talk you need to email the speaker at ramirez@aranducorp.com.

OSCON: Perl 6 on Parrot Español | Deutsch | Français | Italiano | Português

I'm sitting in Perl 6 on Parrot with Patrick Michaud the pumpking for the Perl 6 compiler. Of course we have to talk about Perl 6 compilers, there are lots; Perl 6 on Parrot ("perl6"), Pugs, v6-alpha and KindaPerl6.

You've probably heard of "perl6" and Pugs, but you might not have heard of v6-alpha which is a Pugs based Perl 6 implementation on top of Perl 5, or KindaPerl6, which is a Perl 6 implementation on top of Perl 6.

But Patirck will be mostly talking about "perl6" today, and if you want to check-out and build Perl 6, you can do it using the following incantation,

$ svn checkout http://svn.perl.org/parrot/trunk parrot
$ cd parrot
$ sperl Configure.pl
$ smake
$ scd languages.perl6
$ smake
$ smake test
$ smake spectest

# run a Perl 6 program
$ parrot perl6.pbc hello.pl

The big focus right now is on the Perl 6 test suite, there is a definition on Synopsis 1 which says the anything that can pass the Perl 6 test suite is an official version of Perl 6. The Perl6/Parrot syncs tests with the Pugs master repository, which continues to host the canonical version of the test suite, which are of course all written in Perl 6.

The tests in 01-sanity/ are the "threshold" to running other tests in the suite, unfortunatley Pugs' Test.pm evolved beyond features in 01-sanity/, so the Perl6 people have now built a simpler Test.pm that only requires these threshold features which allows development to take place on other components.

Commonly asked Perl 6 compiler questions,

Are you really planning to write the (whole) compiler in PIR?

The answer is "No", this has never been the plan. There are a lot of components written in PIR, but most of the compiler is written in Perl 6.

What about Perl6 bootstrapping/self-hosting..?

This is a longer term goal. Patrick's more immediate goal is a stable platform for compiling /running Perl 6 on Parrot, the medium term strategy is to use whatever tools facilitate feature coverage.

Apparently there is a floating garbage collection bug somewhere in the Parrot virtual machine which really needs more eyeballs on the test suite as sometimes it's there, and sometimes it's not...

...and we're done.

OSCON: Perl 6 Update Español | Deutsch | Français | Italiano | Português

I'm in the Perl 6 Update given by Larry Wall and Damian Conway in absentia. There was a lot of material in this talk, as always, so this is a high-lights and low-lights summary only, mostly of the bits and pieces I managed to write down in time before Larry flipped the slides away...

Class and module names can now have adverbial modifiers allowing you to put author (naming authority) and version numbers into the class. Also introduced inline comments, if you follow the # symbol with a bracket, then the comment only goes to the end of the matching bracket.

No more defaulting to $_. Built-in functions no longer default to $_ by, well, default,

#Perl 5
for ( @data )  {
   chomp;
   print;
}

# Perl 6
for( @data ) {
   .=chomp;
   .print;
}

The file test operators -r -w -x are gone from Perl 6 and have been replaced by something that look a lot like this,

if $file ~~ :r { 
   say 'we can read the file';
}

if $file.:w {
   when :r | :w | :x { say "r,w,x" }
   when "w & :!±r { day "write-only" }
}

All global variables now have a star, e.g. @*ARGS, which by-the-by is the replacement for @ARGV...

There is a new loop type,

repeat {
   print "Name: ";
   $name ==<>;
} while $name !~~/\S/

repeat while $name !~~ /\S/ {
   print "Name: ";
   $name ==<>;
}

which fixes the "mess I made of Perl 5 do/while loops" according to Larry.

One of (many) interesting regexp fixes in Perl 6 is match boundaries, which are being added to Perl 5.10 but with far worse syntax,

/ bar < before baz>/
//foo <(bar)> baz/

You can now define fixed size arrays, if you really want to,

my @calendar[12]; # indices are 0..11 only

and it'll give you error if you go outside the range.

It looks like there are lots of new semantic features added into POD, like tables, and a much better implementation of nested lists and link syntax. Can also use DIY mark-up, and add parser and translator support for new block types.

...and we're done.

Update: Both Brad and Chris were covering this session.

OSCON: The Expo Hall Español | Deutsch | Français | Italiano | Português

So I spent the lunch hour wandering around the Expo Hall...

OSCON: Managing technical debt Español | Deutsch | Français | Italiano | Português

I'm in Managing Technical Debt with Andy Lester. He's really rolling, and taking the audience with him and his five step plan to reduce your technical debt...


Andy Lester talking about technical debt

• Identify your debts
• Determine the costs
• Pay the most profitable
• Stop incurring new debt
• Repeat as necessary

Andy is arguing that clutter is probably the biggest indicator of technical debt; failing tests, fragile code, ugly code.

A failing test is a car alarm, you'll just ignore it...

Outdated comments are also a big problem, if the comments don't match the code then you have real problems. "Bus-sensitive knowledge" is any knowledge where if you get hit by a bus, and you loose that knowledge, is technical debt. You might also be missing vital infrastructure; RT, version control, backups and cron jobs to automate tasks.

Update: Andy is making a lot of good points, including advocating giving non-critical projects to your not so smart people to free up the smart ones to do the critical ones. You need to pick one thing to work on, don't pick the easiest, don't pick the most fun either. Think improvement, not perfection.

Land one plane at a time

If you try to do two things, you're going to do both of them poorly.

Update: Technical debt is obviously an issue really close to everyone's heart, Andy is getting a lot audience interaction. I'm half expecting people to start standing up and say, "I'm Bob Smith, and I have technical debt...", followed by a group hug.

OSCON: Wednesday morning keynote Español | Deutsch | Français | Italiano | Português

I'm sitting in the Wednesday morning keynote, listening to Nat warm up the crowd for Tim O'Reilly and the annual O'Reilly Radar. It's good to see that traditional shirt, wouldn't be OSCON without it...


...and here's you're host Nat Torkington

Update: For various reasons I got called out of the keynote and hand to go make a bunch of phone calls, but Brad and Chris blogged it, so just go read their coverage...

Update: Tim O'Reilly and the O'Reilly Radar,

OSCON: Data-mining from Open APIs Español | Deutsch | Français | Italiano | Português

My afternoon tutorial is Data-mining from Open APIs given by Toby Segaran. I'm currently stuck on the end of a row without access to a power socket, so we'll have to see how my battery holds out.


Toby Segaran talking about Data-mining
Having paged through the book I'm a bit unsure I really want to be in this tutorial, Toby has started out with a fairly dry discussion of what data mining actually is, which isn't really that reassuring. I don't think this is going to be the fun fast-paced tutorial on mashups that I was expecting. I think next year I should probably read the tutorial descriptions before picking what I'm going to, rather than waiting till I arrive at the conference.

Apparently he's got an book "Programming Collective Intelligence", which covers a lot of the same ground as this tutorial, due out in August with O'Reilly.

Update: Oh, he's just started to talk about regression trees and the CART algorithim. Moving on from supervised regression trees, he's talking about unsupervised methods.

Update: He's basing his unsupervised data mining example on grouping blogs on hierarchical clusters. He's using Mark Pilgrim's Universal Feed Reader to harvest the data from the Technorati top one hundred blogs,

I've sure you've all seen blogs...

with lots of Python code flipping across the screen he's building up a matrix of word occurrences and determining the distance between two blogs using Euclidean distance between the word counts. I guess that's okay for a simple example but there are other distance metrics; Manhattan, Tanamoto, Pearson Correlation, Chebychev and Spearman.

Update: He's showing how you can use Dendrograms and K-means Clustering to show how the blogs cluster up. I'm a bit frustrated here, this is good stuff, but his code is written in eight point font, black on white, and almost total unreadable.

Update: Okay, we've just broken for fifteen minutes for an unscheduled break...

Update: We just had about three quarters of an hour of network down time. The network crash happened just as I was posting an update, which appears to be toast. I hate it when that happens...

Update: ...okay, he's just finished two hours early. Erm? What on Earth!?

OSCON: Higher-order Perl Español | Deutsch | Français | Italiano | Português

My morning tutorial today is Higher-order Perl, the tutorial of the book, given by Mark-Jason Dominus.



Mark-Jason Dominus talking about Higher-order Perl
After rolling in five minutes late, in typical Mark-Jason style he booted the laptop, stood up the front and said...

Thanks, any questions...

Suffice to say he got a laugh, and got the audience on his side. I've been in a bunch of Mark-Jason's talks before, for instance back at the OSCON in 2005 I went to his Making Programs Faster which was excellent. While he isn't Damian Conway, and you won't get him proving the 2nd law of thermodynamics using the Game of Life, he's a really good speaker.

The point of this tutorial, and the book, is that a lot of people write C programs in Perl, which is a bit of a waste of time. Mark-Jason is trying to persuade him to write Perl programs in Perl.

Update: He's started talking about caching and the Memozie module,

use Memoize;
memoize 'date_to_key';

What Memozie is that it replaces the true function with a wrapper function, and with people looking a bit confused he's dropped back a fair ways to talk about closures, e.g.

sub make_function
  my $val = shift;
  return sub { print "Value is $val.\n"; ++$val; };
}

Which captures the lexical variable at the time they are created, which of course is why they're called closures. Interestingly there are still people looking confused, oh boy, they're so in the wrong tutorial...

Update: Back to Memozie, we pass it a function name then it constructs a new closure with a private reference to that function. Each instance has a real and cached version, and uses a glob assignment to install a reference to the wrapper function into the symbol table. Then simply when the wrapper function is invoked, we grab the identity of the function, and peek in the cache. If it's not in the cache yet, we grab a reference to the real function and stick it in the cache for the next time and then return it. If it is in the cache, then we just return the cached value. Of course, its a bit more complicated than that, you have to take account of scalar and list context for instance.

Update: After more Memozie goodness he has moved on talk about Iterators. For the non Java people in the audience an iterator is an object interface to a list. It supports a next method to generate the next item when it is needed. So why do you want them as a Perl person, well the list might be large, or it might take a long time to come up with list elements, or you might not know in advance how many items you want. Of course Perl file handles are iterators, and the next method is the <...> operator, and iterators turn up everywhere in Perl even though we don't call them that explicitly. Interestingly Python doesn't use this concept for reading directory listings, and Perl does...

Update: Hurrah, we have an "octopus" reference when talking about File::Find, we've all been waiting...

Update: Mark-Jason is making his way through an Interator example re-implementing File::Find, and is talking about classes using bless'ing subroutines rather than hashes. There are few looks of pain in the audience, mostly from the people that didn't seem to follow closures.

Update: Time for coffee, back after these messages...

Update: So unlike yesterday [1, 2] I'm actually having to work to keep up here, which is good because I've got a sneaking suspicion that this afternoon's tutorial on data mining isn't really going to stretch me that much.

Update: ...and we're back, still talking about Iterators and infinite lists.

Update: He's talking about linked lists which he's claiming aren't particularly useful in Perl because the array data type pretty much handles everything you want to do with linked lists. Which is pretty much true in general, although I do have linked lists buried inside my two hundred thousand lines of object-oriented Perl sitting in my project's CVS archive.

Perl loves you, that's why we're all here...

Update: We're moving on to parsing, taking a big pile of unstructured input and turning it in to a data structure. Perl actually hides a lot of this from you, for instance,

while(<$fh>) {
   # do something with $_
}

is actually a parser after all. But at some point your ad-hoc parser and you'll need to go to something like Parse::RecDescent. Of course Parse::RecDescent is a closed system, and you really need to have open architecture to cover your bases, and it looks like he's going to build one in-front of our eyes using Recursive-descent parsing which, funnily enough, is the same algorithm used be Parse::RecDescent.

Update: Okay, that was pretty cool. Moving on he's talking about the book and how Perl is really Lisp...

Update: ...and we're done. Lunch time!