At Unixmen, we were lucky enough to score an exclusive interview with former Project Leader of the OpenIndiana, Alasdair Lumsden.
Since Alasdair’s resignation, it has not only sparked discussion on the reasons of Alasdair’s resignation from OpenIndiana, but also generated a lot of interesting discussion on development and the future of OpenIndiana.
I want to really thank Alasdair for taking the time to chat with Unixmen and reveal all in this very intimate and honest view of OpenIndiana from the very inside of the project.
Unixmen: As far as we are aware, you used to work for the Sun Microsystems funded open-source project, OpenSolaris. And when Oracle acquired Sun Microsystems and eventually announced they would discontinue the OpenSolaris project, you obviously felt passionate enough about the project to start the OpenIndiana project?
Alasdair: I never actually worked at Sun, or had any involvement in OpenSolaris prior to OpenIndiana – believe it or not I only started using Solaris in 2008, which does surprise a lot of people. I run a Managed Hosting company in the UK and I saw a lot of promise in technologies such as ZFS, Zones and SMF, which allow us be a lot more agile than our much larger competitors. I got stuck in and learned everything I could. And within 6 months, most of our clients were going into Zones onto Solaris servers.
Then in 2010 when Oracle bought Sun Microsystems and the future of OpenSolaris became very uncertain, I started to worry. Nobody knew what would happen next. Oracle had canceled several open-source projects and there was complete radio silence. I had been attending the London OpenSolaris Users Group (LOSUG) quite a lot. And I decided to investigate the possibility of forking OpenSolaris, via a weekend hackathon I advertised through LOSUG. A surprisingly large number of people attended. Nobody at that point had attempted to assemble the whole distribution outside of Sun. At the end of the hackathon it became apparent that it was entirely possible, with some hard work and dedicated volunteers. So we set to work.
Unixmen: Who was responsible for splitting the project in to two separate entities to include OpenIndiana for an enterprise desktop development and Illumos for the kernel development?
Alasdair: We weren’t kernel engineers – the LOSUG hackathon hackers were all systems administrators. None of us had any experience hacking kernels. I had sent some private emails out to various figureheads in the OpenSolaris community to explain what I was up to and came into contact with Garrett D’Amore.
Garrett was a seasoned kernel programmer at Sun who had left to join Nexenta. Nexenta saw the same threat we did. Garrett’s vision was to fork the kernel specifically. (Actually OS/Net, which includes the core supporting userland tools such as SMF, Dtrace, libc, etc). That made perfect sense to us, so it became quite a symbiotic relationship.
Unixmen: Do you believe splitting the two development branches in to two different entities was the right thing to do at the time? Or do you believe that it is contributing to the problems involved with the development of OpenIndiana?
Alasdair: Oh it was definitely the correct decision – it’s very much like the Linux community – with various distributions based on a shared common kernel. You have Illumos and various Illumos based distributions such as Nexenta, SmartOS, OpenIndiana etc. The kernel+core userland is where all the amazing Solaris technologies live. So it makes sense for them to be maintained by a separate entity.
Having Illumos separate in no way hinders OpenIndiana, and vice versa. In fact, it strengthens Illumos and therefore strengthens all of the distributions, including OpenIndiana.
Unixmen: I think one of the major downsides of the free and open-source software development model is the demand of volunteer developers free time. We all have day jobs and often our FOSS projects are something that we do either as a hobby or just something we do because we enjoy it. It’s been seen time and time again people resigning from such positions in various FOSS projects, was it simply lack of time which prompted your decision to resign from the OpenIndiana project?
Alasdair: Yes, that was the biggest factor contributing to my resignation. Managing an operating system is the equivalent of a full-time job. I already have one of those!
I viewed OpenIndiana as the “Debian distro” of the Illumos world – it was to be a community maintained general purpose distribution that couldn’t disappear because the company maintaining it decides to pull the plug. It’s also the only widely used Illumos based distribution that has a graphical environment that can function as a desktop OS (although desktop was never a space we wanted to compete in – Solaris was always a server operating system).
OpenIndiana was also a continuation of OpenSolaris. A large numbers of OpenSolaris machines (perhaps even most of them) were upgraded to OpenIndiana because it was just a “pkg update” away. This gave OpenIndiana the largest user base of any of the other Illumos based distributions.
I was never paid to work on OpenIndiana. And neither were any of the other contributors. But that’s not true of all the other distributions – Nexenta, a large storage vendor, maintain Nexenta and the bits of Illumos they are interested in. Joyent, a large cloud computing company biting at Amazon’s heels, maintain SmartOS. OmniTI, an IT services company, maintain OmniOS.
I viewed the success of OpenIndiana as being of strategic importance to all the companies invested in Illumos. So I was rather disappointed by the fact that no Illumos developers and none of the commercial companies who depend on Illumos helped us directly. Not even in their spare time! Nexenta to their credit did try, but their staff were too busy to help. We had no support from Joyent, OmniTI or Delphix.
Here I was volunteering large amounts of my personal time with little help from some very large companies with deep pockets, who largely ignored us, even though many of their developers were using OpenIndiana on their workstations. Ultimately if we were to fail, it would be bad for Illumos, and therefore bad for them.
So yes, I did become bitter about it. Working on OpenIndiana was a mammoth task. Sun had left us with a huge mess to clean up (more on that shortly) and none of us working on it had any experience of maintaining operating systems before, or running open-source projects as a whole. It took a huge amount of effort and time, and it just couldn’t continue indefinitely. So there came the point at which I just had to resign.
Unixmen: In your resignation email, you referred to the Linux development model and how the current development model for OpenIndiana just is “not working”. Are you suggesting that OpenIndiana developers need to use Linux as a good example and something to look up to and strive to be?
Alasdair: I don’t think our development model is any different from the Linux one – volunteers hacking away at a distribution. Linux distributions are very healthy because they made it easy to contribute, and OpenIndiana needs to do the same. I was attempting that, but we haven’t got there yet.
Unixmen: There’s no doubt that OpenIndiana consists of some very outdated software. Do you think this is a contributing factor of the rather slow development of OpenIndiana?
Alasdair: The slow development of OpenIndiana is down to one reason and one reason alone – we inherited a gigantic mess from Sun. OpenSolaris was built by a number of large teams spread around the globe, all using different build systems, and Sun had a release engineering team to assemble each bi-weekly build from all these different parts. There are lots of different “consolidations” involved, everything from g11n internationalization, through to JDS (Java Desktop System – Sun did love putting Java in the name of everything whether it contained Java or not), pkg, XNV, vpanels, SFW, userland, it goes on and on!
As a small project, that just doesn’t work – we can’t afford to employ full-time staff to assemble the operating system, nor can we maintain it with so many different (and quite awful) build-systems.
I tried to get the project focused on a single build-system, but it was quite difficult and a slow process. One of the main things that held us back was that most of our developers lacked either the skills necessary to combat the hardest bits of work, or they lacked the time or ability to commit to taking on such large tasks. Many of our volunteers are general system administrators, with no background in the complexities of compilers, linking software, writing makefiles or overhauling build systems! It’s hard work.
So that left me doing a lot of it, with help from others. But I had to drive it. It’s still on-going. Others have stepped up to assist, in particular Jon Tibble, who has been doing our release engineering over the past 6 months. And we’ll get there I’m sure. But until we get there, development will be slow because nobody wants to maintain the old consolidations – it’s hard, and very boring.
Unixmen: And what can be done to bring the software up to date and more relevant to today’s desktop users requirements?
Alasdair: Lots of people would come to us and say “Oh hey, I’d love to help OpenIndiana – I would like to update [some software], how do I get started?”, and there would be no easy answer. We’d point them at the relevant consolidation, which would have an obtuse particular way of being built; they’d need a complex build environment, and most just give up – it’s way too difficult.
Once we’re able to move towards a unified build system and release regular updates, it will be a lot easier for users to contribute. The idea is that anyone should be able to check out our source from a single repo, edit the recipe (a makefile) of the software they want to update, type “make” and off they go.
I hope we’ll get there within the next 12 months, and then things should really get moving.
Unixmen: In my previous articles, I’ve wrote about Unix versus Linux and the future. Do you think that OpenIndiana/Illumos and Unix as a whole has a positive future ahead?
Alasdair: Without a shadow of a doubt, Illumos has a strong future ahead of it. As servers are getting bigger, it is making a lot of sense to consider virtualization. And Solaris comes from a strong history of running on large servers with many CPU’s and a lot of RAM. Zones are such a cool feature. They’re like crack, in fact so many of our unique features are. And going back to Linux, for me feels like stepping back in time. Our userland software is old, but the kernel+core userland is full of amazing technology.
Joyent are gunning for Amazon EC2 and are very well funded – lots of vendors including Dell are pushing their SmartDataCenter service, which is making its way into a lot of public clouds. I’m sure we’ll see a lot more of them over the coming months and years. Nexenta are challenging NetApp and the other enterprise storage vendors. So there is no doubt in my mind that Illumos has a strong future.
OpenIndiana itself could have a bright future as a general purpose distribution, for users who want to play with server features on their desktop. Or perhaps for use as a workstation operating system. But to achieve this, we still have a long way to go to.
If anything, my resignation has been a wake up call to various people. There is a lot of talk going on about what can be done to speed up development. So I’m hopeful for the future and I will continue to help out in my spare time. A project is only dead when there is no interest in it. But there is still plenty of people interested in OpenIndiana.
One thing we all need to do is find out how to reach out to users of other platforms and sell our advantages. Linux users seem to have a mindset that Linux is “the one true path”, and are almost religious in their beliefs. I switched from Linux to Solaris because it had superior technologies – ZFS, Dtrace, SMF, FMA, COMSTAR, Crossbow and Zones, just some examples. So few Linux users are willing to give us a chance and actually make the effort to discover what unique features we have on offer.
Unixmen: If you were staying on as Project Leader of OpenIndiana, looking back at the decisions you’ve made, would you do anything different that could have possibly taken OpenIndiana and Illumos in a different direction?
Alasdair: If I could travel back in time, with the knowledge I have now, I would immediately from day one decree “this mess of build systems must die” and commence work immediately on the unified build system we’re working on at the moment. It took us over 18 months of faffing with the mess we were left with from Sun Microsystems before we concluded that. And that hurt us. But none of us had done anything like this before, so it was all a giant learning experience.
I would also have set down our mission statement a lot more clearly, stating that we were to be the ‘leading general purpose community developed Illumos distribution’, showcasing the amazing technologies Illumos has to offer. Not having a clear mission statement led to confusion about what OpenIndiana represented – Garrett D’Amore’s response to my resignation letter spoke of us as having a policy of following Oracle’s lead with Solaris 11, rather than innovating ourselves. This demonstrated that I failed to convey what we were actually doing and trying to achieve.
I would also have put my foot down a lot more. I was very careful to solicit people’s input and try to please everyone, asking for opinions all the time. This was completely counter to getting anything done and wasted more time than I care to contemplate. As a leader of an open-source project, you have to be brutal – you can’t please everyone, just bash on and get stuff done. Ultimately I wish we’d spent more time doing, and less time talking.
And lastly, I might have picked a different name. Originally “OpenSolaris” referred to the open-sourcing of the Solaris code; it wasn’t a distribution people could download and install. Eventually Sun realized they needed a distribution, so project “Indiana” was born. All the other OpenSolaris distributions felt betrayed by Sun, and it caused a lot of ill-feeling. I also got the impression that Indiana was a sore point within Sun, for a whole host of internal reasons.
As Solaris and OpenSolaris were trademarked, and we were the continuation of “Project Indiana”, OpenIndiana made sense to me as a project name. But I think we ended up embodying in name all the ill feeling ex-Sun engineers had about Solaris/OpenSolaris, and that might have contributed us getting no ex-Sun contributors. Lots of people with no knowledge of “Project Indiana” or Sun’s internal naming conventions also thought OpenIndiana was silly, thinking of Indiana Jones or whatever. It’s also quite long. So perhaps, on reflection, we should have picked something a bit punchier.
