Predicting a Singularity Español | Deutsch | Français | Italiano | Português

I think a lot about the future, and because of that I've gained somewhat of a reputation for making good predictions. This is a characteristic I share with prophets, messiahs and other ne'er-do-wells. I'm not entirely sure what to think about that.

However one of the problems with making predictions about the future, the main problem, is that it's actually not that hard to predict what'll happen next year. Although for some reason this doesn't really seem to help many of the major analysts whose job it is to make such forecasts. Conversely it's also not that hard to make a prediction for the far future, as "…any sufficiently advanced technology is indistinguishable from magic."¹ Why is this a problem? Well if it's easy to make those predictions, making predictions in the sweet spot, both far enough ahead to be ahead to get you ahead of your competition, and close enough that you'll still be around to do something about it is actually almost impossible.

But how far in the future is the sweet spot? The strange thing is that this actually changes, a century ago it was twenty years, or even thirty, but twenty or thirty years ago it was just ten years. Today it's probably five, or less. The rate of technological progress is accelerating, and with it the amount of change we'll experience during our lives is also changing. The time it takes for new technologies to emerge, become mainstream, become dated and then obsolete is falling almost exponentially.

For someone like me, whose career more or less relies on being on, and being seen to be on, the bleeding edge, this is painfully evident. If I'm asked "Have you heard about…?" and I have to answer "No?" you'll generally see a look of pain cross my face, something sort of like constipation, don't worry, it's just my career flashing before my eyes...

At this point having angered both business analysts and science fiction writers I'm going to make a small admission, both professions have the right of it because "...the future is already here — it's just not very evenly distributed."²

While the pace of change has accelerated, it's hard to see how that can be sustained as the size of the install base of existing legacy technology widens. So far we seem to have sustained that pace of change by trickle down economics, the older technology spreads out, and newer technology is dropped in, eagerly seized by early adopters like myself willing to pay the premium, and experience the inevitable problems that come with all new technology, at least until the bugs are shaken out.

Despite that I'm forced to point out that if you extrapolate the current rate of technological progress the view that some sort of technological singularity must almost be inevitable is hard to argue against. Unless of course there is some sort of major catastrophe, something to set us back.

Major catastrophes that could knock us back aren't hard to spot: a global pandemic, climate change, ecological disaster, super volcanoes, mega-tsunami, overpopulation, asteroid impact, a nearby supernovae and of course worldwide thermonuclear war are all favourites. The threat of some of these seems to be fading, but some seem more likely today than ever. There are others, many others, too numerous and depressing to list here.

They're also wildcards, because sometimes the things that should set us back push us forward. It's certainly arguable that two major world wars, so close together, were a major causal factor in the acceleration of the pace of change that is part of our lives today.

Depending on the current news cycle I can swing violently; between a horribly over optimistic view of the future and the inevitability of the rise of trans-humanism, and a view bleak in pessimism, in the inevitability of the abandonment of technology and a slow slide towards narrowing horizons and the eventual extinction of the human race. Doomed as a species that turned its back on space and by having its world view limited to just that, a single world with all the disasters and catastrophes that can result.

Despite this, I'll continue to try and make predictions in the sweet spot, it's fun to be proved right, and sometimes even more fun to be proved wrong.

¹ Arthur C. Clarke
² William Gibson

You don't have to be awesome all the time... Español | Deutsch | Français | Italiano | Português

In her post talking about the public-ness of mourning after the death of Aaron Swartz, Danah Boyd writes "...we’ve created communities connected around ideas and actions, relishing individualistic productivity for collective good. But we haven’t created openings for people to be weak and voice their struggles and demons."

Geek culture is, at least in theory, a meritocracy, and you are measured by your accomplishments. But that means the best of us, those whose work is held up as shining examples, suffer from Impostor Syndrome. Sometimes cripplingly so, even when they are accomplishing awesome things. Because awesome things sometimes look a lot less awesome from the inside, when you know the limitations, flaws and problems with what you've built and shared with the community.

But worse than that, it means when you have your moment of weakness (and we all do), and for a while cannot contribute, cannot accomplish the day-to-day awesomeness that qualifies you as a member of good standing of the community, things can look very bleak. Because we expect the most from those of us that deliver the most, and even the great and the good can fall sometimes, and need support.

We've built a culture where it's hard to acknowledge that you don't know something, because knowing things is intricately linked with the doing of awesome things — which in turn is linked to our stature with our peers.

For someone like me, whose career more or less relies on being on, and being seen to be on, the bleeding edge, this is painfully evident. If I'm asked "Have you heard about…?" and I have to answer "No?" you'll generally see a look of pain cross my face, something sort of like constipation, don't worry, it's just my career flashing before my eyes…

I have no solutions to offer, only the sure and certain knowledge, which I give freely to other geeks, that you are not alone. That the great and the good amongst us suffer as well. That it's okay to be weak and not know the answer to a question. That it's okay to rest and take from others for a while. We'll still be here when you get back, and we'll still remember how awesome you are. You don't have to live your life on Internet time.

Goodbye Aaron Español | Deutsch | Français | Italiano | Português

I heard this morning that Aaron Swartz has committed suicide. He was just twenty six. That's far too young by anyone's measure.

It's unclear how much the pressures of the unreasonably harsh federal prosecution for the JSTOR incident might have weighed on him, because it's been clear that he was depressed for some years. Like many of us that suffer from bouts of depression he had good weeks, and bad weeks. But the legal mess he was in can hardly have been a light weight to bear.

We've had several well known people in the community commit suicide over the last couple of years, and it's jarring. From the outside they look like the best of us, the brightest, sometimes with the most to lose. From the inside it can look much bleaker.

People in our community grew up geeks, many grew up friendless and carry that burden into adulthood. They have real trouble reaching out when they need help; to the friends they're not sure they really have, to the family they often regard as having not been there for them when they were at school. As a result the community is littered with people that suffer depression, that struggle every day with it, and with Impostor Syndrome. No matter how accomplished people look on the outside, and despite past records that should make those accomplishments as evident to them as it is to the rest of us, they suffer. Often in silence. 

I didn't know Aaron well, we had exchanged a few words on a couple of occasions, but I should have had a chance to fix that. He was twenty six and he was at the start of things, not the end.

If you feel like you can't go on, if you feel like it's too much to bear the weight of your life alone. Please, don't do this, please reach out to your friends, your family, to strangers if you must. If you can't face your friends with the news that you hate your life. Because there is always someone that's going to miss you. Always.

Goodbye Aaron.

The inevitability of smart dust Español | Deutsch | Français | Italiano | Português

This post was first published on the O'Reilly Radar.

I've put forward my opinion that desktop computing is dead on more than one occasion, and been soundly put in my place as a result almost every time. "Of course desktop computing isn't dead — look at the analogy you're drawing between the so called death of the mainframe and the death of the desktop. Mainframes aren't dead, there are still plenty of them around!"

Well, yes, that's arguable. But most people, everyday people, don't know that. It doesn't matter if the paradigm survives if it's not culturally acknowledged. Mainframe computing lives on, buried behind the scenes, backstage. As a platform it performs well, in its own niche. No doubt desktop computing is destined to live on, but similarly behind the scenes, and it's already fading into the background.

The desktop will increasingly belong to niche users. Developers need them, at least for now and for the foreseeable future. But despite the prevalent view in Silicon Valley, the world does not consist of developers. Designers need screen real estate, but buttons and the entire desktop paradigm are a hack; I can foresee the day when the computing designers use will not even vaguely resemble today's desktop machines.

For the rest of the world? Computing will almost inevitably diffuse out into our environment. Today's mobile devices are transition devices, artifacts of our stage of technology progress. They too will eventually fade into their own niche. Replacement technologies, or rather user interfaces, like Google's Project Glass are already on the horizon, and that's just the beginning.

People never wanted computers; they wanted what computers could do for them. Almost inevitably the amount computers can do for us on their own, behind our backs, is increasing. But to do that, they need data, and to get data they need sensors. So the diffusion of general purpose computing out into our environment is inevitable.

Everyday objects are already becoming smarter. But in 10 years' time, every piece of clothing you own, every piece of jewelry, and every thing you carry with you will be measuring, weighing and calculating. In 10 years, the world — your world — will be full of sensors.

The sensors you carry with you may well generate more data every second, both for you and about you, than previous generations did about themselves during the course of their entire lives. We will be surrounded by a cloud of data. While the phrase "data exhaust" has already entered the lexicon, we're still essentially at the banging-the-rocks-together stage. You haven't seen anything yet ...

The end point of this evolution is already clear: it's called smart dust. General purpose computing, sensors, and wireless networking, all bundled up in millimeter-scale sensor motes drifting in the air currents, flecks of computing power, settling on your skin, ingested, will be monitoring you inside and out, sensing and reporting — both for you and about you.

Almost inevitably the amount of data that this sort of technology will generate will vastly exceed anything that can be filtered, and distilled, into a remote database. The phrase "data exhaust" will no longer be a figure of speech; it'll be a literal statement. Your data will exist in a cloud, a halo of devices, tasked to provide you with sensor and computing support as you walk along, calculating constantly, consulting with each other, predicting, anticipating your needs. You'll be surrounded by a web of distributed sensors and computing.

Makes desktop computing look sort of dull, doesn't it?

The fifth horseman never gets invited to the good parties Español | Deutsch | Français | Italiano | Português

This article was originally posted on Google+.

Yesterday +MG Siegler argued on +TechCrunch that Samsung is the fifth horseman of technology, filling in for the ailing Microsoft, when the four horsemen: Amazon, Apple, Facebook and Google go riding.

I have to disagree with the underlying assumptions. We're at yet another tipping point in technology. A few years ago we moved from the beige box to the black rectangle, but the black rectangle won't be with us for as long as the beige box.

That black rectangle, the ubiquitous form factor of today's smart phone, is a transition device and it's going to disappear quickly as the speed of technological change is accelerating rapidly. Of the four horsemen only Google seems to be working on alternatives with +Project Glass. It's possible the others, including Samsung, have working hardware, but the successor to today's smart phone is going to be all about context and user interaction.

I've stood up in front of audiences before and argued that our smart phones have our lives on them, the next generation of mobile technology is going to stand between us and our lives and add context. It's hard to do that without a lot of information about the user.

It's also going to be a big leap for the horsemen to make. Despite getting into hardware recently Amazon is about selling content, Facebook has never done hardware and I don't think have this sort of paradigm shift in their corporate bones, Apple has, but without Steve Jobs I don't think they'll have the guts to kill the iPhone and innovate. Samsung, the fifth horsemen that never gets invited to the good parties, is a box shifter. They know hardware, but they don't know design, and they don't know anything about their end users. Their customers are other companies, like Amazon, not you and me, the eventual consumers.

So out of all of them Google  seems to be the only one positioned to move forward, and it'll be a big leap for them even so. The developer release of +Project Glass later this year is going to be crucial. If I had the money to lose making a wager, I'd wager that it'll be some startup you or I haven't heard of yet that makes the leap to the next ubiquitous form factor.

Either way, it's going to be an interesting year...

The rise of the personal space program Español | Deutsch | Français | Italiano | Português

This article was originally posted on Google+.

Just over a year ago now, the first ever project I backed on Kickstarter was Kicksat. A project to put a swarm of small nano-satellites in orbit. The size of a couple of postage stamps each satellite has solar cells, a radio transceiver, and a micro-controller along with memory and sensors.

Today in the mail my souvenir satellite arrived, it's just 3.5 cm square. It's an engineering prototype, presumably one that failed verification, and it doesn't have it's solar cells attached, but otherwise its just like the ones that'll be flown into orbit.

An engineering sample of a flight ready Sprite nano-satellite.

At it's heart is a +Texas Instruments  CC430F5137. It's a whole system on a chip based around an MSP430 CPU, a 16-bit RISC ulta-low power micro-controller, along with an onboard RF transceiver operating in the sub-1GHz bands, a real-time clock and an integrated temperature sensor.

As well as the big solder pads at the top of the board for the solar cell there are a number of unpopulated pads on the board, including space for some through hole components, so there is certainly room for more sensors to be added on the board itself, and the MSP430 has the capacity to handle them.  

However the satellite is going to be entirely reliant on the solar cell for power, there is no battery back up on the board, and the satellite will only be able to operate on the Sun-ward side of its orbit. That means power is the limiting factor. The CC4305137 has good brown-out reset capability, probably one of the reasons it was chosen, however you have to wonder how much more can be crammed onto the board and still reliably operate. 

While it's might not seem much beyond a shrunken down Sputnik at that point, the Kicksat is ground breaking. In just fifty-five years we've advance from the point where it takes the might of one of the world's only super powers to put something like this into orbit, to the point where several hundred of them can be put into orbit by a graduate student with some enthusiastic backers.

Despite the pessimism I often express at the way the space programme is going, this is something that gives me a lot of hope that we aren't going the wrong way. That we aren't starting a march towards the abandonment of technology and a slow fall towards an age of declining possibilities and narrowing horizons.

Kicksat, the CubeSat designed to carry hundreds of these small Sprite nano-satellites aboard, is now on track for launch in the autumn of 2013 onboard the CRS-3/ELaNa-5 mission. This will be the third Commercial Resupply Service (CRS) flight to deliver supplies to the International Space Station (ISS) aboard a +SpaceX Falcon 9 rocket. KickSat, along with 5 other CubeSats, will be hitching a ride as a secondary payload thanks to +NASA's ELaNa  program.

Teardown of Wireless Sensor Tags Español | Deutsch | Français | Italiano | Português

The Wireless Sensor Tag is a smart tag system by CAO Gadgets.

The Wireless Sensor Tag
(Credit: CAO Gadgets)

The system is based around an Ethernet Tag Manager, a small box that connects directly to your home router and manages the all the associated tags. Basically it acts as a bridge between the wireless tags themselves and the Cloud backend which manages the tags.

The Tag Manager
(Credit: CAO Gadgets)

Each tag has motion and temperature sensors and can be configured to notify you via tweet, email, push notification, or via a URL callback (either to an Internet facing host or directly to an internal IP on your home network) when the tag is moved, or when the temperature goes out of a user defined range, or if the tag itself is moved out of range of the tag manager. Each tag also has a piezo electric buzzer and an LED which can be triggered using the same ruleset.

Ordering

I actually ordered my system back in August, only to be told that the sensor tags had not been CE certified, and they weren't quite prepared to do that just yet. I could either cancel my order, or wait until they had enough orders to that it would make "economic sense" for them to obtain CE marking at which point they'd ship me my order.

I have to admit I wasn't that impressed by that. If you're offering something for sale internationally then you should have the items in stock and you should be able to ship it out of the country. If they weren't prepared to ship into the EU, they shouldn't have processed my order, or billed my credit card.

They also seem to be having supply chain problems as their current stock is sold out and their store has a note on the front page saying that the lead time for a new order is currently one to one and half months between order placement to fulfilment.

However it looked like someone had put a lot of thought into this problem space, and despite the number of similar systems that are starting to appear, the system should be a neat solution. So I hung on, and I'm glad I did, as I I'm actually quite impressed with the build quality of the hardware and how it hangs together as a system...

Unboxing

My system arrived in a plain USPS box. Inside was the tag manager, an ethernet cable, a USB power adapter and cable, ten tags with velcro strips, and somewhat oddly, seven spare batteries.

The Wireless Sensor Tag System

Setting up the system is a fairly simple procedure. Plug the tag manager into your router and let it grab a DHCP address and announce itself to the mytaglist.com server. Theoretically the tags included in the tag manager in your shipment should come pre-associated, but at least for me this didn't seem to be the case, but that said associating them is no big deal. Pull the battery tab, and if you have a flashing LED that means the tag is unassociated, go to the web app and click on the button to add a new tag. The tag then shows up in the list on the web backend and you can go ahead and configure the triggers for that tag.

The Software

Unfortunately despite initial promise, I'm not finding the system amazingly reliable as yet. Tags that are within a few feet of the tag manager are being periodically being reported as "Out of Range" by the web interface. However it's possible that the hardware is fine and I'm just not understanding the software.

Update (12 Nov): Just got an email from the manufacturer saying that they've "fixed some bugs on the server" and these spurious "Out of Range" notifications shouldn't happen any more. It's a bit soon to tell one way or the other, but certainly I'm not seeing them at this point, so this might well be fixed now.

The web application used to manage the tags on your system

The web application is confusing, the interface basically looks like someone has exposed the backend database schema in software without much thought as to how the user is going to interact with it. At the moment the interesting hardware is being let down badly by the back end software, there should be preset options, e.g. "this tag is attached to a door", "this tag is attached to a moveable object", that would auto-configure the tag to common presets.

As it is there are numerous settings to go through for each and every tag, and most of these aren't well explained, and the buttons to set the options are not self explanatory or just flat out confusing. 

I'm sure it makes sense to the programmer that built it, as they understand exactly how the system works and how the components interact, to the rest of us, at least initially, it's confusing. 

I've been playing with the system for about an hour now and I still can't figure out how to get a tag to start bleeping when moved, and then keep on bleeping until manually reset. It's something you'd commonly want to do for a tag that's attached to something that might be stolen, and the system should be able to do it, but I can't figure it out. It's probably obvious once you know how, but...

They need to get a good UX person in, along with a designer, to overhaul the backend. That could make a big difference to the software and its usability for the average person. The system itself is really elegant, and easy to get up and working, the software to configure the tags is letting it down. 

Another frustration for me is that the iOS app that allows you to monitor the tags, and receive push notifications to your phone when one of your triggers is tripped, is only available in the US App Store. The ability to get push notifications was one of the major reasons for me to buy this system, so I'm hoping this is going to be resolved quickly. Again, if you're going to ship outside the US you need to make sure your system is ready to go there.

Update (12 Nov): The iOS app is now available in the UK store, which resolves one of my main problems with the system. Although unfortunately the software UX problems extend to the iOS application. It's comprehensive, but not easy to use.

The iOS Tag Manager application running on my iPhone 5

The Hardware

There is actually very little information about how the system hardware works, and I'm not in favour of "just magic", so I wanted to take a closer look at the board to try and figure out what's going on under the hood.

The rubberised enclosures are more-or-less indestructible and amazingly stretchy so they're fairly easy to take off, which is a good thing as you'll need to do that before pulling out the battery tab to activate the tag as they tend to break off if you try and do this with the enclosure still on...

Wireless Sensor Tag

The tag is powered by a single CR2032 button cell Lithium battery which they're claiming will last anywhere between two months and seven years depending on the sensitivity and polling interval you choose for the tag.

The onboard processor is a Microchip PIC16LF720 micro-controller, an interesting choice that draws about a fifth of the current than the Amtel ATmega micro-controllers used in the familiar Arduino boards. Wireless operations for the tags is provided by the Microchip MRF 49XA radio, which operates in the sub-GHz MHz ISM bands. While this can be changed, the tags ship and by default operate at 436 MHz. Unlike the US this is not an ISM band here in the UK, and although it is part of the amateur band, a license to operate is required and the primary user of the band is the MOD.

The choice of the sub-GHz ISM band is an interesting one, most competing products on the market use the 13MHz RFID bands, or more commonly the 2.4GZ band used by both Wi-Fi, Zigbee and Bluetooth LE devices. I'm guessing that they went with the sub-GHz bands to keep power usage to a minimum and extend the battery life of the tags themselves and provide a good range.

Motion sensing is provided by an Honeywell HMC5883L sensor, a three-axis magnetometer, again an interesting choice as most of the competitors are using linear accelerometers. I'm presuming they used a magnetometer to get good angular measurements, which is a perfect fit for one of the main uses cases for the tags; checking whether doors are open or closed. 

Finally I'm guessing temperature measurements are provided by a fairly anonymous chip marked with "AXWB." While that's just a guess, the word "TEMP" is the silkscreen legend next to it, so it's probably a decent one. I haven't been able to track down the data sheet for this chip or the manufacturer, although the specifications page gives an operating range of -40°C  to 85°C with an typical accuracy of ±1°C and a -2 to +4°C maximum deviation, with a sensor quantisation of around 1°C. 

Developer SDK

The system isn't "open hardware" but that's just fine, not everything has to be open, people have to eat and keep a roof over their head and that takes money. However I was pleased to see that there is some developer documentation available for the web service API of the mytaglist.com backend server, along with the the source code for the management software I was complaining about earlier. The source code for the iOS and Android applications is also available if you email the company.

Update (12 Nov): I emailed the manufacturer and they want to know why I want the source code, and for me to sign an NDA before they release it to me, which wasn't exactly what I was expecting from the API documentation. I'm not sure what to think about that...

The system does crucially rely on a back end server provided by the manufacturer, but the FAQ tells us that if the tag manager can be flashed to point at a different server, and it's at least theoretically possible to run your own. The vendor then talks about licensing their own server side software, and promises that if their mytaglist.com server gets shut down that they'll release the code and allow you to update the tag manager to point at your own server. So I'm happy enough at that point.

Summary

Just as I thought when I initially placed my order, someone has thought long and hard about this problem space, and it really shows in the quality and flexibility of the hardware if not the software. 

I'm fairly sure the software issues are going to get ironed out eventually, at least for me the existence of some sort of documentation would probably solve most of the problems I'm having. Although I do think that for the average user the interface needs a good UX expert and a redesign. 

I also hope to have my hands on the iOS app so I can use the tags as I originally intended, and enable push messaging to my phone, as that'll vastly increase the utility of the system for me.

From poking around on the manufacturer's website it seems that there are probably more tag types coming, including current (for home energy monitoring?) and moisture (for detecting water leaks) sensor tags. I'll certainly go ahead and purchase those when they arrive.

Despite some of the criticism above, I am impressed with this system and would recommend it.

Update (23 Nov): My tags were suffering from spurious "out of range" and "reconnected" messages. So I have just sent all my tags back to CAO Gadgets for a firmware update.

With conflicting stories, all we can believe is the data Español | Deutsch | Français | Italiano | Português

This post was originally published on the O'Reilly Radar
under the tile "Digging into the UDID Data"

Over the weekend the hacker group Antisec released one million UDID records that they claim to have obtained from an FBI laptop using a Java vulnerability. In reply the FBI stated:

The FBI is aware of published reports alleging that an FBI laptop was compromised and private data regarding Apple UDIDs was exposed. At this time there is no evidence indicating that an FBI laptop was compromised or that the FBI either sought or obtained this data.

Of course that statement leaves a lot of leeway. It could be the agent's personal laptop, and the data may well have been "property" of an another agency. The wording doesn't even explicitly rule out the possibility that this was an agency laptop, they just say that right now they don't have any evidence to suggest that it was.

This limited data release doesn't have much impact, but the possible release of the full dataset, which is claimed to include names, addresses, phone numbers and other identifying information, is far more worrying.

While there are some almost dismissing the issue out of hand, the real issues here are: Where did the data originate? Which devices did it come from and what kind of users does this data represent? Is this data from a cross-section of the population, or a specifically targeted demographic? Does it originate within the law enforcement community, or from an external developer? What was the purpose of the data, and why was it collected?

With conflicting stories from all sides, the only thing we can believe is the data itself. The 40-character strings in the release at least look like UDID numbers, and anecdotally at least we have a third-party confirmation that this really is valid UDID data. We therefore have to proceed at this point as if this is real data. While there is a possibility that some, most, or all of the data is falsified, that's looking unlikely from where we're standing standing at the moment.

With that as the backdrop, the first action I took was to check the released data for my own devices and those of family members. Of the nine iPhones, iPads and iPod Touch devices kicking around my house, none of the UDIDs are in the leaked database. Of course there isn't anything to say that they aren't amongst the other 11 million UDIDs that haven't been released.

With that done, I broke down the distribution of leaked UDID numbers by device type. Interestingly, considering the number of iPhones in circulation compared to the number of iPads, the bulk of the UDIDs were self-identified as originating on an iPad.

Distribution of UDID by device type

What does that mean? Here's one theory: If the leak originated from a developer rather than directly from Apple, and assuming that this subset of data is a good cross-section on the total population, and assuming that the leaked data originated with a single application ... then the app that harvested the data is likely a Universal application (one that runs on both the iPhone and the iPad) that is mostly used on the iPad rather than on the iPhone.

The very low numbers of iPod Touch users might suggest either demographic information, or that the application is not widely used by younger users who are the target demographic for the iPod Touch, or alternatively perhaps that the application is most useful when a cellular data connection is present.

The next thing to look at, as the only field with unconstrained text, was the Device Name data. That particular field contains a lot of first names, e.g. "Aaron's iPhone," so roughly speaking the distribution of first letters in the this field should give a decent clue as to the geographical region of origin of the leaked list of UDIDs. This distribution is of course going to be different depending on the predominant language in the region.

Distribution of UDID by the first letter of the "Device Name" field

The immediate stand out from this distribution is the predominance of device name strings starting with the letter "i." This can be ascribed to people who don't have their own name prepended to the Device Name string, and have named their device "iPhone," "iPad" or "iPod Touch."

The obvious next step was to compare this distribution with the relative frequency of first letters in words in the English language.

Comparing the distribution of UDID by first letter of the "Device Name" field against the relative frequencies of the first letters of a word in the English language

The spike for the letter "i" dominated the data, so the next step was to do some rough and ready data cleaning.

I dropped all the Device Name strings that started with the string "iP." That cleaned out all those devices named "iPhone," "iPad" and "iPod Touch." Doing that brought the number of device names starting with an "i" down from 159,925 to just 13,337. That's a bit more reasonable.

Comparing the distribution of UDID by first letter of the "Device Name" field, ignoring all names that start with the string "iP", against the relative frequencies of the first letters of a word in the English language

I had a slight over-abundance of "j," although that might not be statistically significant. However, the stand out was that there was a serious under-abundance of strings starting with the letter "t," which is interesting. Additionally, with my earlier data cleaning I also had a slight under-abundance of "i," which suggested I may have been too enthusiastic about cleaning the data.

Looking at the relative frequency of letters in languages other than English it's notable that amongst them Spanish has a much lower frequency of the use of "t."

As the de facto second language of the United States, Spanish is the obvious next choice  to investigate. If the devices are predominantly Spanish in origin then this could solve the problem introduced by our data cleaning. In Spanish you would say "iPhone de Mark" rather than "Mark's iPhone."

Comparing the distribution of UDID by first letter of the "Device Name" field, ignoring all names that start with the string "iP", against the relative frequencies of the first letters of a word in the Spanish language

However, that distribution didn't really fit either. While "t" was much better, I now had an under-abundance of words with an "e." Although it should be noted that, unlike our English language relative frequencies, the data I was using for Spanish is for letters in the entire word, rather than letters that begin the word. That's certainly going to introduce biases, perhaps fatal ones.

Not that I can really make the assumption that there is only one language present in the data, or even that one language predominates, unless that language is English.

At this stage it's obvious that the data is, at least more or less, of the right order of magnitude. The data probably shows devices coming from a Western country. However, we're a long way from the point where I'd come out and say something like " ... the device names were predominantly in English." That's not a conclusion I can make.

I'd be interested in tracking down the relative frequency of letters used in Arabic when the language is transcribed into the Roman alphabet. While I haven't been able to find that data, I'm sure it exists somewhere. (Please drop a note in the comments if you have a lead.)

The next step for the analysis is to look at the names themselves. While I'm still in the process of mashing up something that will access U.S. census data and try and reverse geo-locate a name to a "most likely" geographical origin, such services do already exist. And I haven't really pushed the boundaries here, or even started a serious statistical analysis of the subset of data released by Antisec.

This brings us to Pete Warden's point that you can't really anonymize your data. The anonymization process for large datasets such as this is simply an illusion. As Pete wrote:

Precisely because there are now so many different public datasets to cross-reference, any set of records with a non-trivial amount of information on someone’s actions has a good chance of matching identifiable public records.

While this release in itself is fairly harmless, a number of "harmless" releases taken together — or cleverly cross-referenced with other public sources such as Twitter, Google+, Facebook and other social media — might well be more damaging. And that's ignoring the possibility that Antisec really might have names, addresses and telephone numbers to go side-by-side with these UDID records.

The question has to be asked then, where did this data originate? While 12 million records might seem a lot, compared to the number of devices sold it's not actually that big a number. There are any number of iPhone applications with a 12-million-user installation base, and this sort of backend database could easily have been built up by an independent developer with a successful application who downloaded the device owner's contact details before Apple started putting limitations on that.

Ignoring conspiracy theories, this dataset might be the result of a single developer. Although how it got into the FBI's possession and the why of that, if it was ever there in the first place, is another matter entirely.

I'm going to go on hacking away at this data to see if there are any more interesting correlations, and I do wonder whether Antisec would consider a controlled release of the data to some trusted third party?

Much like the reaction to #locationgate, where some people were happy to volunteer their data, if enough users are willing to self-identify, then perhaps we can get to the bottom of where this data originated and why it was collected in the first place.

Thanks to Hilary Mason, Julie Steele, Irene Ros, Gemma Hobson and Marcos Villacampa for ideas, pointers to comparative data sources, and advice on visualisation of the data.

Update: In response to a post about this article on Google+, Josh Hendrix made the suggestion that I should look at word as well as letter frequency. It was a good idea, so I went ahead and wrote a quick script to do just that...

The top two words in the list are "iPad," which occurs 445,111 times, and "iPhone," which occurs 252,106 times. The next most frequent word is "iPod," but that occurs only 36,367 times. This result backs up my earlier result looking at distribution by device type.

Then there are various misspellings and mis-capitalisations of "iPhone," "iPad," and "iPod."

The first real word that isn't an Apple trademark is "Administrator," which occurs 10,910 times. Next are "David" (5,822), "John" (5,447), and "Michael" (5,034). This is followed by "Chris" (3,744), "Mike" (3,744), "Mark" (3,66) and "Paul" (3,096).

Looking down the list of real names, as opposed to partial strings and tokens, the first female name doesn't occur until we're 30 places down the list — it's "Lisa" (1,732) with the next most popular female name being "Sarah" (1,499), in 38th place.

The top 100 names occurring in the UDID data

The word "Dad" occurs 1,074 times, with "Daddy" occurring 383 times. For comparison the word "Mum" occurs just 58 times, and "Mummy" just 33. "Mom" came in with 150 occurrences, and "mommy" with 30. The number of occurrences for "mum," "mummy," "mom," and "mommy" combined is 271, which is still very small compared to the combined total of 1,457 for "dad" and "daddy."

[Updated: Greg Yardly pointed out on Twitter that I was being a bit British-centric in only looking for the words "mum" and "mummy," which is why I expanded the scope to include "mom" and "mommy."]

There is a definite gender bias here, and I can think of at least a few explanations. The most likely is fairly simplistic: The application where the UDID numbers originated either appeals to, or is used more, by men.

Alternatively, women may be less likely to include their name in the name of their device, perhaps because amongst other things this name is used to advertise the device on wireless networks?

Either way I think this definitively pins it down as a list of devices originating in an Anglo-centric geographic region.

Sometimes the simplest things work better. Instead of being fancy perhaps I should have done this in the first place. However this, combined with my previous results, suggest that we're looking at an English speaking, mostly male, demographic.

Correlating the top 20 or so names and with the list of most popular baby names (by year) all the way from the mid-'60s up until the mid-'90s (so looking at the most popular names for people between the ages of say 16 and 50) might give a further clue as to the exact demographic involved.

Both Gemma Hobson and Julie Steele directed me toward the U.S. Social Security Administration's Popular Baby Names By Decade list. A quick and dirty analysis suggests that the UDID data is dominated by names that were most popular in the '70s and '80s. This maps well to my previous suggestion that the lack of iPod Touch usage might suggest that the demographic was older.

I'm going to do a year-by-year breakdown and some proper statistics later on, but we're looking at an application that's probably used by: English speaking males with an Anglo-American background in their 30s or 40s. It's most used on the iPad, and although it also works on the iPhone, it's used far less on that platform.

Thanks to Josh Hendrix, and again to Gemma Hobson and Julie Steele, for ideas and pointers to sources for this part of the analysis.

Update: A really nice analysis from David Schultz using the frequency of UDID duplicates and the names of those devices to track down the source of the leak. I really should of thought of that...

Interestingly however it does support my own analysis. BlueToad makes apps for magazine publishers, hence the predominance of of the iPad over the iPhone in my results, as those apps are more normally used on the iPad.

Also they seem to mostly market into the U.S., which supports my ethnicity findings, and looking at the list of titles they curate, it does look like my demographics are more-or-less spot on as well. Those look like magazines marketed to men in their 30's and 40's to me...

I'd actually been really confused about what type of app could possibly have that narrow a demographic, and this sort of clears up my confusion. Nice!

Hardware Hacking for iOS Programmers Español | Deutsch | Français | Italiano | Português

This post was originally published on Josetteorama.

The arrival of the iPhone changed the whole direction of software development for mobile platforms, and has had a profound impact on the hardware design of the smart phones that have followed it.

Not only do these devices know where they are, they can tell you how they're being held, they are sufficiently powerful to overlay data layers on the camera view, and record and interpret audio data, and they can do all this in real time. These are not just smart phones, these are computers that just happen to be able to make phone calls.

Alasdair Allan demonstrating an Augmented Reality application

The arrival of the External Accessory Framework was seen, initially at least, as having the potential to open the iOS platform up to a host of external accessories and additional sensors. Sadly, little of the innovation people were expecting actually occurred, and while there are finally starting to be some interesting products arriving on the market, for the most part the External Accessory Framework is being used to support a fairly predictable range of audio and video accessories from big-name manufacturers.

The reason for this lack of innovation is usually laid at the feet of Apple's Made for iPod (MFi) licensing program. To develop hardware accessories that connect to the iPod, iPhone, or iPad, you must be an MFi licensee.

Unfortunately, becoming a member of the MFi program is not as simple as signing up as an Apple Developer, and it is a fairly lengthy process. From personal experience I can confirm that the process of becoming an MFi licensee is not for the faint-hearted. And once you’re a member of the program, getting your hardware out of prototype stage and approved by Apple for distribution and sale is not necessarily a simple process.

However all that started to change with the arrival of Redpark's serial cable. As it's MFi approved for the hobbyist market it allows you to connect your iPhone to external hardware very simply, it also allows you to easily prototype new external accessories, bypassing a lot of the hurt you experience trying to do that wholly within the confines of the MFi program.

Another important part of that change was the Arduino. The Arduino, and the open hardware movement that has grown up with it and to a certain extent around it, is enabling a generation of high-tech tinkers to prototype new ideas with fairly minimal hardware knowledge.

Every so often a piece of technology can become a lever that lets people move the world, just a little bit. The Arduino is one of those levers. While it started off as a project to give artists access to embedded microprocessors for interactive design projects, I think it’s going to end up in a museum as one of the building blocks of the modern world. It allows rapid, cheap prototyping for embedded systems. It turns what used to be fairly tough hardware problems into simpler software problems.

Turning things into software problems makes things more scalable, it drastically reduces development time scales, and up front investment, and as the whole dot com revolution has shown, it leads to innovation. Every interesting hardware prototype to come along seems to boast that it is Arduino-compatible, or just plain built on top of an Arduino.

Controlling an Arduino directly from the iPad

I think the next round of innovation is going to take Silicon Valley, and the rest of us, back to its roots, and that's hardware. If you're a software person the things that are open and the things that are closed are changing. The skills needed to work with the technology are changing as well.

Alasdair demonstrating an Augmented Reality application

At the start of October I'll be running a workshop on iOS Sensors and External Hardware. It's going to be hardware hacking for iOS programmers, and an opportunity for people to get their hands dirty both the internal sensors in the phone, and with external hardware.

The workshop is intended to guide you through the start of that change, and get you hands-on with the hardware in your iPhone you've probably been ignoring until now. How to make use of the on-board sensors and combine them to build sophisticated location aware applications. But also how to extend the reach of these sensors by connecting your iOS device to external hardware.

Blinking the heartbeat LED of a BeagleBone from the iPhone

We'll look at three micro-controller platforms, the Arduino and the BeagleBone and Raspberry Pi, and get our hands dirty building simple applications to control the boards and gather measurements from sensors connected to it, directly from the iPhone. The course should give you the background to build your own applications independently, using the hottest location-aware technology yet for any mobile platform.

The workshop will be on Monday the 8th of October at the Hoxton Hotel in London at the heart of  Tech City, and right next to Silicon Roundabout. I'm extending a discount to readers; 10% off the ticket price with discount code OREILLY10. That makes the early bird ticket price just £449.10 (was £499), or if you miss the early bird deadline (the 1st of September) a full priced ticket still only £629.10 (£699).

Monday 8th October 2012

Hoxton Hotel, London

Early Bird Price: £499 (until 1st Sept.)

Normal Price: £699

Save 10% with code OREILLY10