The Trouble With the No Reboot Challenge

Let me start by saying I think it’s great that Mike at Mobile Jaw and Chris from GearDiary are doing this. It’s a neat idea and I’m looking forward to the results. At the same time, while I think this idea is interesting, it also has some inherent flaws. Again, I’m not saying their idea is a bad one or that the results won’t be interesting. I just feel that since our app is one of the ones being used in the experiment, and could thus be blamed if the device still crashes, that I need to say a thing or two about this idea.

So if you want to hear my take on device stability and the “No Reboot Challenge”, read on!

So the theory goes something like this:

Instability in the operating system is not a result of bad design by the OS manufacturer but rather a result of badly designed third party applications.

The goal, as I understand it, is to prove that MS gets way too much grief over the instability of its operating systems (mobile and otherwise.) I think this is true and I think proving this is a laudible goal. Unfortunately, I’m not sure that the methodolgy of the Challenge is sound. Not only might it fail to prove the point, but (and this is what worries me) it could wrongfully implicate a developer in the process.

That said, here are the potential problems that I see. I resisted going into specifics (*cough* WM memory management *cough*) because frankly, a lot of this applies to any OS, not just MS products.

Impossible to Prove
Unfortunately, it will be impossible to pinpoint the source of instability, even with very few applications installed. A flaw in the OS might only rear its ugly head when an application is installed, not due to any failure on the part of the developer. It could be that the flaw remains hidden until an app is run.

It’s like building a bridge and watching it collapse when the first car goes across. Do you blame the car or its driver for driving across the bridge improperly? I mean, the bridge was working fine until the first car drove over it so it must be the car’s fault, right?

It Isn’t Easy to Screw Up the OS
Some will surely disagree, but in my experience it isn’t easy to botch up an operating system accidentally. You’ve got to work really hard to mess up your software so much that the entire OS suffers. Crash your own application? Sure. But screwing up the whole OS is something else entirely.

And let’s be frank about this. The times I’ve seen our applications cause what appeared to be a wide-spread issue (and these were VERY rare and never made it out the door), in the end we discovered that some aspect of the OS was not working as advertised. Perhaps it was a custom method the device had for handling images that failed, an out-dated library, or a missing piece the hardware manufacturer didn’t think was important and cut. In the end, we wrote work arounds to avoid problems with the OS. Had we released the software without addressing this, to anyone on the outside it might have looked like “we broke the OS”, but that simply wasn’t the case.

Failure to Lock-Up the Dogs
But let’s say that some application really IS screwing up the OS. Shouldn’t there be some sort of “police officer” built in to protect the user from this sort of thing? Isn’t it the responsibility of the OS manufacturer to provide customers with a stable user experience? If they are giving applications access to portions of the OS that can cause the entire system to perform badly, is that really the sign of a good OS? I’m all for freedom and flexibility for developers, but someone has to be the boss and say “You can ride in the car but you can’t drive it!”

So there you have it folks…
…my take on the No Reboot Challenge. It’s a cool idea and I hope something cool comes of it, but I’m very concerned about the conclusions folks may come to if things go south.

12 thoughts on “The Trouble With the No Reboot Challenge

  1. Elbrus

    I am going to have to disagree with this statement:

    “Instability in the operating system is not a result of bad design by the OS manufacturer but rather a result of badly designed third party applications.”

    If this was true a Mac running an emulation software for Windows XP should crash as much as Windows XP machine does…. the problem is it doesn’t from personal experience.

    My thought is that the operating system should have safe guards in it to prevenet a third party app from jacking ut up. Of course we could just go back to the days of the old Amiga computer that was more stable than anything out there now.

  2. Andrew

    I agree with your post 100%! I have run Windows, Windows Mobile, Linux, BSD, and Mac for years. Mostly Windows, I will admit, and I do not have a problem with stability of the OS.

    My SP2XP box has almost uptime on it as my friends Linux box, and many Windows servers (at work) also have uptimes that equal Linux and Sun boxes. Most boxes are only rebooted for patches, and some of these we put off for safety sake (Oracle and Sun patches are scariest of all).

    My Windows Mobile devices have been pretty solid. The only issue i ever really had was pocket IE hosing things up, but I can crash a BlackBerry by going to a javascript filled site.

    I would love to see an OS not crash, but as long as we the users expect to have plug and play and other conveniences where user code interacts with kernel, then we have to take some chances on stability.

    I think all OS manufacturers truly strive for a bug free, rock solid OS. But we all know that bugs do happen, and sometimes it takes a user doing something a developer never envisioned to make that bug appear.

    eWallet never once crashed any OS, and, on second thought, has never itself crashed on me.

  3. Adam S

    I enjoy working with the windows mobile kitchens out there trying out different combinations of program being built into the rom. I tend to run each rom about a week before I try again. There are a couple things that I have learned. 1) a complete format really really makes a difference. 2) Order matters (and where you install matters and what goes on the rom vs what is installed later matters, etc). 3) most of the I have a very hard time figuring out what is causing the problem.

  4. Quinton

    I’m with Jay, less geek speak, more getting the eWallet Sync deplyed. I could care about the details. You blew smoke up our rears by deploying a truly half baked application that fails to Sync. I do understand capturing new customers, but you have placed a bad taste in the mouth of the LOYAL CUSTOMER

  5. Marc Post author

    Less Geek Speak: You know, despite the noise that iPhone makes, we have many Loyal Customers who aren’t iPhone users. In fact the vast majority of our customers don’t own iPhones. The article above was needed because it relates to a serious topic regarding applications like ours and Windows Mobile. It might not matter to iPhone folks but to the 90% of our loyal customers that own Windows Mobile devices, this is very important.

    Blew Smoke Up Our Rears: Maybe its because for so many years companies have been taking advantage of their customers that make people assume the worst. We very honestly described what we did and why. That’s the truth whether folks believe it or not. I can’t offer something different. I won’t repeat it all again but it isn’t double speak or BS. I’ve been shooting straight right from the start.

    Truly Half-Baked: There are a ton of applications in the store AppStore that don’t have a desktop component. This includes other secret keeper apps. eWallet as it is right now is an app I’m extremely proud of. It does it’s job extremely well, even without sync. It is a fully functional, complete application. Will it be better when sync comes out? Absolutely! Is it a great app as is? A lot of folks thing so!

    Bad Taste for Loyal Customers: We never promised anything that wasn’t true. In fact we rewarded folks who bought even without sync. And in the end, we’re coming out with sync pretty darn close to our predicted mark. I can tell you’re frustrated and I’m sorry, but there is no way we will release an app that doesn’t do exactly what it should exactly AS it should. If that means a few extra days of development, then we’ll do it. There are many reasons we have loyal customers, and a big one is that when we release something, by god it works and works right…and we don’t manage that by releasing something we aren’t 100% satisfied with just to shave a few days off a calendar date.

  6. David

    I have been a user of eWallet for many years now across several platforms (PC, U3, and PPC). Also, I am a loyal ListPro user who takes that software with me every time I go shopping for CDs and other electronic media.

    My experience with the company has been first rate all these years. In fact I jumped on eWallet for the iPhone just as soon as it was released. (I even did this before I found out about the upgrade allowance on their web site).

    The point is this company has an outstanding track record with their installed base and has consistently well publicized upgrades to the product. Sure, I would love to be able to sync with my Mac today, however, I am absolutely confident Ilium will deliver a first rate product that will work “out of the box”.

  7. Mike Temporale

    Hi Marc,

    Great post! You are correct about what I am hoping to show by doing this challenge. I know it’s going to be hard to prove beyond a doubt, but hopefully this is a solid start. You raised a number of key concerns and issues. There’s just so many factors involved that I can’t possibly rule them all out and create a truly controlled environment for this test.

    Ultimately, it would be best if the OS protected us from unsafe and instable applications. But the truth is that we are dealing with a mobile OS and there just isn’t the resources to handle this correctly. Heck, the full blown desktop OS still doesn’t do an amazing job at this. Although, it’s getting better but there’s a long way to go.

    I didn’t mean to imply that all 3rd party applications are bad. In fact, the reason I selected a few apps to install is because I wanted to show that there are good developers and that you can find good products that won’t compromise the stability of your device. IMHO, Ilium is one of those companies that I know I can trust to build a solid app.

    Cheers!

  8. James Rayner

    I’d have no problem with the no-reboot challenge, except sprite backup needs to reboot to backup.

    Interestingly, since I enabled the nightly backup+reboot, I rarely need to reboot during the day.

    I don’t think it’s reasonable to blame third party applications for needing to reboot. That’s the whole reason why I have a Windows Mobile device — so I can use third party apps and do more with my device. That’s a significant feature.

    The problem is the design of the third party applications. On my Linux box I can install, try and remove applications trivially, and with no affect to the system as a whole. The difference is that there’s features that curb and help avoid poor application design issues. Priviledge separation helps avoid applications abusing system function and resources, and a solid package manager keeps install/removal clean. These aren’t unreasonable expectations for a mobile OS.

    Problems can be avoided with careful selection of applications… but that’s hard when I want to buy a new product. I’ll try out the various competitors – and once in a while I’ll get a dud that causes problems.

    James

  9. Ben

    Wow, Quinton… I mean, wow. Not only was your post off topic, but you flamed the developers that are taking time out of their busy day to contribute to the software community. That is truly amazing. And no, you’re not with Jay. Jay was poking a harmless joke, you flamed the blogger. Iliumsoft is quite obviously one of the few companies left who are interested in communicating with their customers in an open, honest forum. eWallet is a top 1% application written by users, for users. If it takes a few extra days to get a new version out, and that few extra days means regression testing, usability testing, and other testing — then what’s the worry? We’re not talking 6 years (see: Duke Nuke’m Forever)… they are a few days behind. Relax, and please give up your sense of entitlement. The product works [very well] as advertised, and Iliumsoft has been very generous giving it’s new and existing customers the benefit of the doubt.

    Marc — keep up the good work man. Take your time, it’ll be ready when it’s ready, though I am waiting with bated breath.

    As for the experiment you mention in your post I completely agree. There is no purely scientific way they can go about this. It’ll be ‘interesting’, but nothing more. Unless these guys plan to completely dissect the source of each and every anomaly, all they will do is shovel meaningless dreck on the ill-informed masses and give honest developers a black eye (Rupert Murdoch?). I can only hope that they preface any findings with disclaimers about the ambiguity of the results. I think we can all agree that there are substantial shortcomings in all operating systems, and also that there are some very poorly written software packages written by good developers working on unrealistic projects. All you guys can do is keep writing great applications and respond to your customers’ needs. Let the herd cull itself, you guys will be around for years to come.

    Take care,

    -ben

  10. Andrew

    To James above:

    I don’t think Linux is any better then Windows when it comes to testing software and then removing it. I have had plenty of issues with missing dependencies and incorrect package versions. Linux suffers from the same type of “DLL Hell” as windows.

    The real issue is that this is difficult stuff to do. If Application A uses libX and Applicaiton B uses libXv2 you can easily be in a situation with 2 different versions of the library on the same machine and both are used by different applications.

    Also, both Linux and Windows allow the user to install and remove apps outside the package management system, so accurate counts of which apps use which libraries is hard to maintain.

    Sorry to get off topic, but i get Angry when the Linux/Apple/Windows (name your OS) fanboys say “My OS doesn’t have this problem that yours does” and usually it is “Windows sucks, Linux/Mac is better”. Guess what, they all have similar design and similar issues. Apple controls OS hardware and software so they get tighter integration and have to worry less about driver compatibility, but Apple boxes do crash, and people do have issues with shared libraries. All OS have pros and cons, but all get the job done.

  11. Marc Post author

    @Ben: Thanks for the encouragement Ben!

    @Andrew: I think this is a good point. As a guy that works with a program that, while complex, is amazingly simple compared to an operating system, I’m constantly impressed that these OS’s work as well as they do. Trying to create a system that effectively lets anyone hook any device up to it running any sort of driver and accessing any sort of software is a monumental challenge. True, for a commercial product there is a certain expectation of quality, but frankly I abuse my OS so much that I’m surprised it runs at all.

    And on a side note – using Vista here and at home since it came out – one crash ever – every peripheral worked the first time – all my software works – and on top of that, better UI experience than any I’ve had. *shrug* For what it’s worth!

Comments are closed.