The often deranged postings of yet another hacker, pretending to be an Astronomer, pretending to be a hacker who has written a book or two for O'Reilly Media.
In a few short years, and from a standing start, Elon Musk and SpaceX has achieved what might otherwise have been thought impossible. Late last year they launched a spacecraft and returned it to Earth safely. Then they launched a second which successfully docked with the International Space Station (ISS) and again returned it to Earth safely.
"The impact of this adventure on the minds of men everywhere, who are attempting to make a determination, of which road they should take...", John F. Kennedy
Working relatively independently of NASA and the other government agencies, and building their technology stack from the ground up, SpaceX has in under a decade already demonstrated Apollo-era capability. However their Dragon capsule is no Apollo, it's a flexible space transport system built with modern technology, whose full capabilities have yet to be demonstrated.
The SpaceX Dragon spacecraft on the end of the Canadarm2.
SpaceX is the first commercial company to send a spacecraft into orbit and recover it successfully, something that only three governments - the United States, Russia and China - have ever done, and with the retirement of the US Space Shuttle they have the capability that only two governments - Russia and China - now posses. The European ATV and Japanese HTV have no return capability and burn up in the Earth's atmosphere, and the US currently has no manned space capability.
The SpaceX Dragon spacecraft floats in the Pacific after returning to Earth from the International Space Station (ISS). Credit: Mike Altenhofen/SpaceX
With the retirement of the Space Shuttle, the Dragon is the only spacecraft in the world capable of returning significant cargo from the station as the Russian Soyuz has only minimal cargo capacity.
Space Stations
The first generation space stations; the Soviet Salyut and Almaz stations, along with the American Skylab station, were all monolithic designs. It wasn't really until Mir was flown with a modular design that we entered the modern era, it was the only second generation station to fly, with the US sitting on the sidelines resting on its Lunar laurels, and the Europeans seemingly uninterested in manned spaceflight.
The ISS and the docked Space Shuttle Endeavour, taken by Expedition 27 crew member Paolo Nespoli from the Soyuz TMA-20 following its undocking on May 23, 2011. It was the first-ever image of a space shuttle docked to the International Space Station. Endeavour at left. European ATV cargo carrier at right. Credit: NASA/Paolo Nespoli
Today's ISS is a bastard child of the follow-on station projects from the various countries involved in space race; the Soviet/Russian Mir-2, the American Freedom project which included the Japanese Kibō Laboratory, and the European Columbus space station. None of which came to fruition separately, mostly due to budgetary considerations but also due to politics.
The first component of the ISS was launched in 1998, and construction began the Russian Mir station was still in orbit. The last manned mission to Mir was a privately funded Soyuz mission by MirCorp, in April 2000, which carried out repair work with the hope of proving that the station could be made safe. There was no return to Mir however which was deorbited the following year following the permanent occupation of the ISS, which began in the November of 2000.
OPSEK and Gateway
Despite being declare "complete" there are two more modules destined for the ISS are due for launch over the next couple of years, both from Russia. The Nakua module will serve as Russia's primary research module on the ISS. It will replace the current Pirs module, When that happens Pirs could become the first permanent ISS module to be decommissioned, and would be destroyed during atmospheric re-entry. The Node Module is intended as the primary core of the Russian OPSEK station. Initially attached to the ISS, it will be detached along with Nakua and some of the other Russian modules before the ISS is decommissioned and deorbitted, and used for the basis of a new station.
Recent proposals by Boeing call for some of the "left over" parts of the ISS program that are still on the ground, notably the unlatched Node 4, to be used to build a Exploration Gateway Platform to be located at one of the Earth-Moon Lagrange points to be used as a launch platform for deep space exploration, robotic relay station for moon rovers, telescope servicing and a deep space practice platform located outside the Earth's protective radiation belts. The new platform would be assembled at the ISS before being boosted towards the Lagrange point. If this came about it could drastically cut the cost of future manned Lunar, Mars or NEO missions, and would represent the first manned presence beyond low-Earth orbit since the Apollo program ended in the 1970's.
Commercial operations
With the launch of first Cygus spacecraft scheduled for October of November, the number of commercial companies with access to low Earth orbit will grow to two, although SpaceX will remain the only company with return capability. There is no immediate expectation that the US government, and NASA, is on track to regain any sort of direct access and commercial operators will therefore remain the only access to space for the US for the foreseeable future.
The decommissioning of the ISS, now scheduled for 2020, would leave no US government presence in space. Of course by then Bigelow Aerospace, already with two pathfinder launches under their belt (Genesis I and Genesis II), plan to have a human-habitable commercial station online. Tentative launch dates for the first modules are around 2014 and 2015, and Bigelow has reserved a 2014 launch slot on SpaceX's Falcon 9, although they have not yet announced the payload.
Unlike the ISS and previous stations Bigelow's station technology is different, and potentially game changing. Based on the TransHub technology and patents, which Bigelow bought from NASA when they were directed to discontinue work on module by the US Congress, Bigelow's inflatable modules will provide large useful volumes for a much smaller launch weight than traditional hard-shell modules.
By the time Bigelow is ready to launch its first station SpaceX should have a fully man-rated Dragon capsule, and possibly a crewed launch to the ISS under their belt. Earlier this month Bigelow and SpaceX teamed up to do joint marketing to international customers of crew transport on SpaceX Falcon 9/Dragon up to the Bigelow BA330 space facility.
Bigelow has agreements with seven sovereign nations to utilize on-orbit facilities of the commercial space station: United Kingdom, Netherlands, Australia, Singapore, Japan, Sweden and the United Arab Emirate of Dubai.
Of course who knows what the Jeff Bezos and his skunk-works company Blue Origin are doing behind heir "cone of silence," beyond their initial test flight back in 2006 we heard very little out of the company until the beginning of September last year where they reported the loss of their second test vehicle during a developmental test at Mach 1.2 at an altitude of 45,000ft. I'm not sure most people were aware they were testing at those altitudes, at least not at the time.
Then there is Excalibur Almaz who are now planning to take customers to the Moon, with a ticket price of $100 million a seat.
History of the Soviet Almaz military program on which the Excalibur Almaz technology is based.
The company relies on the use of decommissioned Salyut-class spacecraft which Excalibur Almaz purchased from Russian. They currently own four reusable reentry vehicles and two station modules, similar to components of the Mir station and the currently flying Zarya module attached to the ISS.
Close to home
Somewhat overshadowed by SpaceX and Dragon, Virgin Galactic has announced that the FAA had given an experimental launch permit for its sub-orbital SpaceshipTwo and air carrier WhiteKnightTwo.
SpaceShipTwo flying with crew for the first time, during a dress rehearsal flight for its first free glide flight in 2010. Credit: Virgin Galactic/Scaled Composities
WIth this permit in hand Scaled Composites and Virgin Galactic are able to press ahead with the testing program and carry out rocket powered test flights of the new craft.
The long duration future
The new commercial space companies have ambitious plans; SpaceX's Red Dragon which may launch as early as 2018 and use a modified Dragon capsule to carry heavy instrumentation for a soft landing on the Martian surface, and as a precursor to a manned mission.
Artist's rendition of a Dragon spacecraft using its SuperDraco thrusters to land on Mars. Credit: SpaceX.
Almost complimentary to the push towards manned spaceflight from the commercial sector, is the arrival of Planetary Resources earlier in the year with a goal of developing a sustainable (and profitable) robotic asteroid mining industry.
The new space race
The US and Russian governments aren't doing planning any novel endeavours in space, and it seems the Chinese are determined to tread the path that the US and Russia has taken before them, their own station seems to be a mix-and-match copy of the historical Russian programme, although the capability of their Shenzhou spacecraft to leave the orbital module behind means that their station might grow incrementally and much more rapidly than the ISS.
The really interesting development work happening in the space industry right now seems to be going on in the private sector. The new space race has begun, it's between SpaceX, Blue Origin, Orbital Sciences and the other commercial companies. The goal isn't national pride, it's part personal pride and ambition, as most of these companies are founded by individuals, and part profit motive.
I'm currently on the way home from O'ReillyOSCON. While I was out in Portland for the convention I was interviewed by Mac Slocum about growth in embedded devices and the arrival of ubiquitous computing, which should be any time now... probably!
In 1984 a game called Lords of Midnight written by Mike Singleton was released for the ZX Spectrum, conversions to the Commodore 64 and Amstrad CPC soon followed. It came to dominate my game playing during my mid teens, games came and went, but always I returned to the War of Solstice and the Lords of Midnight.
The Lord of Blood stands in the Keep of Blood looking north towards the invading armies of Doom Guard as they pour through the Gap of Valethor and onto the Plains of Blood. Things are not going well for the Free.
The game had a well written back story, and for the time an amazing amount of depth to the game play. A unique blend of war game, strategy, and landscape that was ground breaking...
The Lands of Midnight
...and thanks to Chris Wild's bout of nostalgia in the early nineties, and his port to MS-DOS, I went on playing the game. There was even a multi-player version of the game, called Midnight/MU built, allowing you to play online through your browser. It seems I wasn't alone in my nostalgia.
But things changed with the arrival of the iPhone, and even more so with the arrival of the iPad. I thought the iPad was the perfect platform to revive the game. While it was epic in nature, the turn-by-turn nature of the game meant that unlike some other strategy games it was well suited for the dip-in and dip-out nature of gaming on the platform. More so, I wanted to play my favourite game on my new hardware. I stopped playing Chris' port and started to think idly about porting his code, or more likely his Midnight Engine to iOS. I poked around in the source code, but eventually decided against it. Instead I waited. Someone else was going to do it, it was just a matter of time.
My patience was seemingly rewarded, there was going to be an iOS port and Chris Wild and Mike Singleton were going to work on it together...
...but time passed, actually quite a lot of time passed, more than a year, and it started to look like vapourware. Until just a couple of months ago Chris posted some video footage of the game to his blog. It existed, if only in the roughest sense, and it was playable.
The pre-alpha demo of Lords of Midnight for iOS
Content to wait at that point, I sat back. Not only was there going to be an iOS version, but because of the way Chris had ported his Midnight Engine, using the cross-platform Marmalade SDK, there was going to be a port to Android, Mac OS X and MS Windows. This wasn't just an simple iOS port, this was a cross-platform remake of the original game. There was even discussion of finally making the almost legendary missing sequel The Eye of the Moon.
The graphics are still the original imagery taken from the eighties, and the interface is still a bit shaky, and there are a few bugs in artificial intelligence, but I'm enjoying having early access to the game. I'm enjoying wallowing in my eighties nostalgia.
But beyond that I think, that with the bugs and interface problems properly addressed, and the graphics updated to something that looks at home in the twenty first century, that this is still and above all a solid game. That in fact this is a game that appears as if it was always intended to be on the iPad, as if it was always meant to run on touch hardware. The new platform suits it, like a new suit of clothes.
This isn't an ageing rock star coming out of retirement for one more nostalgia tour, this is something bigger. Just like it did the first time around, I think the Lords of Midnight for iOS could change how gaming is done on the platform.
Not bad for a game that's now approaching thirty years old?
It's the MAKE:Hardware Innovation Workshop this week, as well as Maker Faire Bay Area this weekend, and I'll be hanging around both talking about this and other things and well as doing some live demos. So come and talk to me if you see me, and you want to know more about connecting your iPhone to the open hardware world.
Back in November last year I spoke at Øredev in Malmö, Sweden, about location enabled sensors, and the video of the talk has just been put up onto the web by the organisers.
If you're interested in the topics discussed at Øredev, all the talks will eventually make it onto the website including another talk I did at the conference on visualisation which hasn't made it online quite yet.
I forgot to post a pointer to this at the time, but while I was out at O'Reilly's Where conference I was interviewed by Mike Hendrickson about location privacy, data leakage and data exhaust.
While I was out at the O'Reilly offices in Sebastopol earlier in the month I sat down with Dale Dougherty to talk about how to make iPhones and iPads talk to the open source world.
This article was originally published on the O'Reilly Radar.
Big data isn't just about multi-terabyte datasets hidden inside eventually-concurrent distributed databases in the cloud, or enterprise-scale data warehousing, or even the emerging market in data. It's also about the hidden data you carry with you all the time; the slowly growing datasets on your movements, contacts and social interactions.
Until recently, most people's understanding of what can actually be done with the data collected about us by our own cell phones was theoretical. There were few real-world examples. But over the last couple of years, this has changed dramatically. Courting hubris perhaps, but I must admit it's possible some of that was my fault, though I haven't been alone.
You probably think you know how much data you carry around with you on your cell phone. You'll certainly be aware of it if you've ever lost your phone, or had it stolen, or it's just plain stopped working. But there is a large amount of data in the background that isn't surfaced in the user interface.
We know about what I generally call primary data: calendars, address books, photographs, SMS messages and browser bookmarks. These are usually user generated, and we'd be pretty unhappy if we lost them. There is also the secondary data that the phone generates about us: call history, voice mail, usage information and records of our current and past locations. Most of what I'd call secondary data is surfaced to us in our phone's user interface. We generally can't change this sort of information without resetting the phone to a factory fresh condition; it's generated by the device for us, it's not something we generate ourselves.
But there is also what I refer to as tertiary data. This is data that, similar to the examples I mentioned above, is generated about us, rather than by us. Mostly, this data consists of cache files — data that is entirely necessary to you using the device, or significantly improves your user experience, but you don't necessarily know is there. At least until some hole is found in the operating system to expose that data layer to you. That's happened before, after all.
An obvious example is tucked in your photographs. Every picture you take is geotagged and date stamped, and if you publish your pictures to a photo-sharing site without stripping that information, you're leaking data. Back in 2007, when geotagged photographs of newly arrived helicopters at a U.S. Army base in Iraq were published to the Internet, they allowed insurgents to determine the exact location of the helicopters inside the compound and conduct a mortar attack. Four of the AH-64 Apaches on the flight line were destroyed in the attack.
Where Conference 2012 — O'Reilly's Where Conference, being held April 2-4 in San Francisco, is where the people working on and using location technologies explore emerging trends in software development, tools, business strategies and marketing.
Recently, there have been a number of high-profile cases of data leakage, and the one that has raised the most controversy, at least until the next time, is the social network Path.
Upon opening the Path application on your phone, it automatically uploaded your address book to Path's servers so it could find "friends" that you might want to connect to without asking for explicit permission to do so, or even implicit permission for that matter. Path has since apologized and updated its application so that it now asks permission before pushing your address book to its servers.
This was not data theft, but data leakage. You asked the application to accomplish something and didn't really ask yourself how it was doing it. While there are technical solutions that don't involve uploading your address book, the laziest solution is probably what you should have expected. I can almost hear the developers, "... we'll just upload the address book for now and switch to hashing later on when we have time."
There has been a lot of comment that somehow the whole Path thing was unexpected. Realistically, that's not the case. It's not an isolated circumstance, either. To the best of my knowledge Hipster and other apps also tapped your address book behind the scenes without asking permission. Interestingly, there are other, less obvious, culprits. Applications that make use of Chillingo's "Crystal" game service, like Angry Birds, will in some circumstances also upload your address book. While there is a button to push, it is, at least for me, misleadingly labelled and doesn't suggest what's going to happen next.
Data leakage like this is not really a solvable problem at the user level, at least not in real-time. Having multiple permission boxes pop up at regular intervals is a bad design choice; users stop reading them, they lose importance and become ineffective. Just try using Microsoft Windows and you'll understand exactly what I mean. Modal interrupts should be reserved for vital time-critical issues. They're already used far to prolifically in iOS. Run the Mail application with multiple mail accounts configured when you're not connected to the network and that'll become instantly obvious. You'll be bombarded by error messages.
I did have a thought that you might be able to deploy a customized web proxy directly onto your mobile device and have all web requests directed through it. The proxy would sift through the outgoing network connections in a (semi-)Bayesian manner looking for data that you don't want transmitted and stop the application cold before it sends it to the remote server. Basically, it's acting as a reverse spam filter, or a smart firewall, depending on how you want to think about it.
I think that something like this could well be far more effective at stopping data leakage than the current solution, which Google has used on Android: Permissions pages when you initially install an application are all very well, but most people don't read them, and when you're installing an application you're not really thinking about why it might need certain permissions. However, you can be very clear about what data you're interested in not leaving your phone. One configuration page for the proxy, rather than multiple ones, every time you install an application. Like modal dialogs on the iPhone, you subconsciously start to ignore them, to your peril.
Location, location, location
Of course, I can't really talk about data leakage without mentioning the kerfuffle surrounding location and data privacy that happened just about this time last year. Unsurprisingly, the file in question still exists, despite some of the press stories; the existence of the file was never the problem. A cache of that nature is fairly necessary if you want to have reliable and timely location services on your phone. However, the file is now actually just that, a cache, and it is regularly swept clean by the operating system. It's also not included in your usually unencrypted backups to your laptop, which was perhaps more of a problem than the fact it wasn't being cleared out in the first place.
What Apple was doing was taking a piece of tertiary data, generated about you by the device, and then exposing it on a platform (laptop or desktop) where accessing that data was easier. There are a lot of people who know how to navigate a file system on a computer, but a lot fewer who would know how to get the same data directly from the phone itself. It was a classic case of data leakage: data moved from a secure(ish) environment on the phone to a less secure one on the computer.
Data exhaust
Back in the days of floppy disks, the lines of ownership were pretty clear. If you had the disk, the data was yours. If someone else had it, it was theirs. Things these days are much blurrier. That tertiary data — data that's generated about us but not by us — doesn't just build up on your mobile devices of course. Other people are building datasets about our patterns of movement, buying decisions, credit worthiness and other things. The ability to compile these sorts of datasets left the realm of major governments with the invention of the computer.
We're all aware of this, and there's even a provocative buzzword to describe it: data exhaust. It's the data we leave behind us, rather than carry with us.
In the U.S., data from grocery store loyalty schemes has been used by security services to search for terrorist suspects. Turns out the number of toilet rolls you buy can be quite telling.
Which does make me think, instead of being afraid of the data exhaust, perhaps we should embrace it. In the U.K., the biggest retailer is the supermarket Tesco. Like many, I spend a good fraction of my income there, and like almost everyone I know, I have a Tesco Clubcard. This is a loyalty card that has a record of (almost) every purchase I make, from toilet rolls to roast chicken.
I'd actually pay good money for a copy of my own Clubcard data, so long as it was actually in a machine-readable format, not on paper. Although for Tesco, the data is only really interesting in aggregate; it's the fact that they have millions of Clubcard records that makes the dataset useful to the company. To me, a history of my purchases would be useful data.
Of course, people have already started selling our data exhaust back to us. Think about your credit report, for instance.
Keep your friends close, and your enemies closer
It's not just your own data exhaust that you have to worry about. There was an interesting paper recently by Adam Sadilek of the Department of Computer Science at the University of Rochester. It talked about how geotagged tweets could be used to locate individuals, even if they themselves didn't geotag their tweets — it was enough that their friends did so.
Geotagged messages on Twitter during a typical weekday afternoon in New York City.
The paper found that only a couple of weeks' worth of location data on an individual, combined with location data from their two most-sharing friends, was enough to place that person within a 100-meter radius with 77% accuracy. That rises to nearly 85% when you combine information from nine friends.
Even someone who has never shared their location at all can be pinpointed with 47% accuracy from information available from two friends. That goes up to 57% when you include nine friends.
Data sharing
There is a great debate going on right now, which is really only starting to surface into the mainstream press, about how we share data. Despite social networks becoming mainstream, the recent privacy debacles in the mobile space say a lot about how users perceive information privacy. I think Sadilek's paper presents even more compelling evidence.
For instance, I'm finding Google's new Instant Upload feature, where photos taken on my phone are automatically uploaded to Google+ behind the scenes, a lot spookier and more worrying than I thought I would. It's especially interesting that I'm feeling that way, as I'm using Apple's Photo Stream without thinking or worrying about it that much.
I'm trying to figure out whether it's because the privacy trade-off — in Apple's case sharing my photos between all my devices, and in Google's case making my photos more-or-less instantly available for sharing in Google+ — is more obviously in my favor with Photo Stream, or it's for other reasons.
The interesting thing here is that Photo Stream and Instant Upload are, at least behind the scenes, effectively identical. Both are cloud services and your photos are stored in a data center somewhere. The master copies of your photos have essentially been moved to the cloud, rather than residing on your device.
However, because of the context these two services operate in, I have no problems with one, and I'm finding the other an uncomfortable fit. I think there's a big lesson there for people dealing with personal information. When you're sharing someone's information, even with their informed consent, the context is important about how they think about the implications surrounding that sharing.
Building platforms
So, all of this got me thinking. There are large personal datasets about me, and you, and everyone, being built up by large companies. But we're also building up datasets about ourselves, in our own control. What happens if we mash them together? Can we actually do something productive?
I'm currently running an interesting experiment with my credit card and my iPhone. I'm scraping my bank's website to grab transaction data in near real-time onto one of my servers. Each transaction comes with a postcode. This is like a U.S. zip code, but it normally specifies a much smaller neighborhood, perhaps down to a single street or smaller in a major urban area.
Watching my credit card transactions in real-time.
On my iPhone, I'm running an application that continually monitors my location using the Significant Location Change service, so my phone knows my location to better than 1km (perhaps much better in a crowded city) more or less all of the time.
Every time a new transaction occurs, I forward it via push notification from the back-end server to my iPhone. Now, my iPhone knows both the location where the transaction took place and where I actually was at the time. If those locations don't match, then this indicates there might have been a fraudulent transaction and it flags it for me with a notification.
The interesting thing here is that I'm using data that my credit card company doesn't have, and hopefully will never have: my actual physical location when the transaction took place. They couldn't possibly provide this service to me because they simply don't have the data I have.
Of course, there are false positives. Online transactions in particular stand out. Most of these are tagged with a postcode of the headquarters of the company I'm dealing with. However, my next development step will be to give my back-end server code access to my inbox and allow it to scrape for online transaction receipts. This should reduce the false-positive rate down to something vanishingly small, and I should be able to deal with those left over with some sort of machine learning. After all, there's a human-readable string attached to each transaction that details the retailer and sometimes other useful information.
A thought to ponder
A thought to ponder in the dead of night: In the near future, the absence of data is going to be increasingly unusual. If you think the data exhaust you leave behind yourself is wide and varied, then just you wait, because we're at the banging-the-rocks-together stage right now.
If your data exhaust becomes assumed, what happens if you turn your phone off for an hour or two one night? What if you're accused of a murder during that time period, and you can't prove where you were? Perhaps in the future that's going to be sufficiently unusual that it's automatically suspicious. Innocent until proven guilty may underlie our current legal system, but that's because our current legal system was codified in a very different era, one that was data poor rather than data rich. Perhaps in the future, the absence of data will imply guilt.
I spent most of last week at O'Reilly Media's Strata Conference on Big Data in Santa Clara, where I was talking about the data you carry with you. Something I've started to call migratory data.
Big data isn’t just about multi-terrabyte data sets hidden inside eventually-concurrent distributed databases in the cloud, or enterprise-scale data warehousing, or even the emerging market in data. It’s also about the hidden data you carry with you all the time, about the slowly growing data sets on your movements, contacts and social interactions.
Until recently most people’s understanding of what can actually be done with the data collected about us by our own cell phones was theoretical; there were few real-world examples. But over the last couple of years this has changed dramatically.
Interview with Mac Slocum at Strata
Mac Slocum caught up with me about half way through the week to talk about the data on your cell phone, and other mobile devices, along with the possibilities for making use of that hidden data to reveal things about our lives that we might not realise ourselves, about data privacy and data leakage, and about where all this might lead.
The SpaceX Dragon spacecraft on the end of the Canadarm2.
The Bigelow Genesis-I Space Station in orbit
SpaceShipTwo flying with crew for the first time, during a dress rehearsal flight for its first free glide flight in 2010. Credit: Virgin Galactic/Scaled Composities
Artist's rendition of a Dragon spacecraft using its SuperDraco thrusters to land on Mars. Credit: SpaceX.
