Sunday, August 30, 2009

A tribute to Chris Short, resurrector of Palms

I was about ready to give up on my beloved Palm LifeDrive—which was already a refurbed replacement for the original—as it had recently kicked the bucket, refusing to turn on just like the last one and many others I've heard reports of. It was way out of even the extended warranty I'd bought. So I carefully extricated my data and started using an old Sony CLIÉ PEG-S320 to keep my life in order.

But then I heard of a man named Chris Short of Mankato, Minnesota. His reputation preceded him; many people had already sent off all sorts of Palm handhelds to him and found joy in their little computers once more. So I shot him an e-mail—you can reach him yourself at ips@chartermi.net—and, perhaps unsurprisingly due to the rather widespread occurrence of the problem I was experiencing, he knew exactly what was up and how to fix it. (He also offered a MicroDrive-to-Flash conversion for an additional $50, but I declined for now.)

I sent my little LifeDrive off with a check for $49 via Priority Mail and got it back in one week. Chris had fixed it up same day. And, joy of joys, it works again! Chris said in our initial exchange that what happens with the LifeDrive is that the ROM chips have cold solder joints. He resolders the chips in place and applies an adhesive that he says Palm should have applied in the first place that's supposed to protect against a repeat occurrence. Obviously, I've not had it long enough to see if this holds; but I already feel like I'm in better hands.

Amusingly, the night I was working on restoring my data to the LifeDrive, it apparently spontaneously rebooted on me before I'd even had a chance to install my apps back. My heart leapt into my throat. As it turned out later, though, it's karmic's fault. Not Chris's.

It's good to have my constant companion back again.


posted by zigg 6:50 PM 0 Comments

Thursday, August 20, 2009

Getting my data back out of NVBackup

I suppose I've never made this confession here yet, but I'm a Palm addict. It's been going on since the original Handspring Visor came out and my core Palm databases are the same ones that I created nearly nine years ago, brought from system to system.

These days, I've been using Palm's disavowed LifeDrive, and mine just kicked the bucket. This is my second one; the first I got replaced through PalmCare (found an absolute steal on a packaged PalmCare plan.) As part of the LifeDrive lifestyle, I've been using Alexander Pruss' NVBackup to keep my data safe.

A hopefully brief history: traditionally, Palm handhelds have kept everything—programs, data, you name it—in battery-backed RAM. This works a lot better than it sounds, but it does have the disadvantage that it all goes pfft if you don't keep tabs on your Palm's battery. The early models actually ran on standard off-the-shelf batteries; my Visor used AAAs and it lasted a good while unless I was using it constantly. Backup was done by syncing your system to a PC.

Anyway, this battery run-down situation must have irked enough people, because in the last round of Palm handhelds they ever made, they introduced something called NVFS, non-volatile storage for the system's data. This wasn't the first time Palms could write to something other than RAM; over the previous several years they'd been adding storage (usually some form of Flash card) to some models. With those came backup applications, which did sadly have some problems with restoring the system to its original state, owing mostly, as I understand it, to the weird machinations that it took to serialize an in-RAM database to a .pdb (Palm database) file.

NVBackup, by comparison, was as simple as could be. It took these NVFS-based Palms and simply copied their databases straight out of NVFS, where they were already serialized, and dumped them straight to your Flash card. The advantage was that in almost all situations, restoring a backup brought your system back exactly the way it was before, which is great if you still have your system. My LifeDrive was dead, though, and I didn't have another to restore to—so I needed to do something to get these databases into .pdb files so I could then stick them onto one of the other old Palms I have lying around, lest my life become tragically disorganized. Er, more than it is now.

Alex Pruss has done some work on a tool called unbackup which reportedly does a passable job on some of the NVBackup dumps. I tried this tool. It did not do well on my LifeDrive's output at all, sadly. Crashed a lot, failed to make sense of other files... it just wasn't pretty. Facing the prospect of reverting to two-month-old data (my last PC sync), I started scraping together ideas for one last shot.

Palm developers (of which I am one, though I've never released a bit of anything) have access to the Palm Simulator tools, which emulate a Palm device on a (Windows, sigh and boot up the VirtualBox) PC. See where I'm going with this? Now I do have a spare LifeDrive that I can restore my NVBackup set to, and use the onboard tools to make .pdb files out of my core databases. It wasn't quite that simple, but I did eventually get it working.

First problem: getting NVBackup to restore to the simulated LifeDrive in the first place. Palm Simulator sets up an area on your disk to simulate external storage, so I copied the NVBackup set off my SD card and into that area, along with the NVBackup program itself. NVBackup got installed to the virtual LifeDrive and presented me with the backup set I copied. But then I tried to do the restore, and things started to get ugly.

I knew I only needed a small set of databases—AddressDB, DatebookDB, MemoDB, and ToDoDB alongside those from my two key other Palm apps, Keys-Gtkr from Keyring and Months-HkDt from Eat Watch—so I started off trying a selective restore of just those databases. It looked like it was working, but then when it came time to restore the files, I kept getting "error opening" messages. I had no idea what was going on. I tried disabling the Simulator's write-protected storage option (honestly, I have no idea what this actually does) as well as PACE extended checks (I have a slightly better idea but this was last-ditch); neither gave me any joy.

Eventually I even wrote to Alex Pruss, since I saw in one of many Google searches that he had originally started work on NVBackup in the Simulator; unfortunately, he had no idea how to solve my problem. One lead caught my eye, though: "error opening" could also show up if there was trouble dealing with an encrypted backup. Although I'd never had trouble with it on a real device, I wondered if perhaps NVBackup's default gzip compression might be at fault. So I ran gzip -d over the NVBackup set, hopped back into the Simulator, ran the restore, and—success!

Well, partial success, at least. The Simulator crashed irrevocably on reboot, possibly because the Simulator is compiled for x86 and my backup was full of ARM code. So I hard-reset my virtual LifeDrive and went back in again, this time only restoring the databases I needed (as well as their new PalmOS counterparts'—Contacts, Calendar et al.—since the old databases are really just there for compatibility purposes), then used FileZ to make .pdbs out of them on the virtual SD card, and voilà!

Okay, almost voilà. DatebookDB was being stubborn, no matter how many times I tried to restore, convert, and copy it, pilot-xfer was refusing to copy it—likely corrupted somewhere along the line. I eventually ended up HotSyncing that one directly from the simulated LifeDrive over a "network" from within the VirtualBox VM to my host Linux netbook. And now it works. The old CLIÈ has all my important data and apps just as I left them before the last backup.

It was a lot of work, but it was worth it, I think. And hopefully there's some clue in here that might help someone else in the same boat down the road.

Labels: , ,

posted by zigg 11:17 PM 0 Comments

Saturday, August 8, 2009

mustafa downloads to PC at 70 KiB/s

Okay, so 70 KiB/s that's not as good as your broadband. Heck, it's not even as good as my broadband, and I'm a cheapskate.

But I'm pretty thrilled with it. My early TFTP proof-of-concept code was only able to manage a bit over 40 KiB/s—though you'll fall back to near that (maybe slightly faster? I have tuned a number of key things) if your client makes a request without requesting a larger block size. Use one that can ask for a larger blksize, though, and mustafa will shave it down to its largest supported size and the transfer will use that, improving substantially on the speed.

Apart from speed, my current build of mustafa accepts a filename which it passes to libfat and will transmit that file—or, using the newly-supported tsize and toffset options, any part of that file—to any TFTP client. I also built a little fuzzing tool that will send packets full of garbage to test its resiliency; on a 16 MiB payload, it slowed the whole transfer down by only about 30 seconds (the entire transfer took a hair over 4 minutes otherwise). Write support is only literally another screenful of code away, since I worked out most of the groundwork over this weekend in-between viewings of the featurettes on The Lord of the Rings extended edition DVDs.

I think it's still a way off from being able to share it, though. Right now, I think the core TFTP support is stable and probably even good, but the mustafa and "device" layers atop it (the fat device handles libfat; card will eventually handle save chip requests and the like) feel a bit early and messy. I imagine I will be going in and reworking some of those. The good news is that they're a lot smaller than the TFTP core.

But yeah, I feel like I'm in the home stretch now. Very happy to be where I am with this project.

Labels: , , ,

posted by zigg 10:37 PM 0 Comments

Thursday, August 6, 2009

savehost's bizarre metamorphosis

I think this is the first project I've ever actually abandoned—well, sort of—but I don't intend to do any more work on my previously-announced Nintendo DS Save Host.

It isn't that I don't want to create that tool anymore, but I found myself wanting a different approach to the problem. I was spurred down this road by my finding that dswifi's TCP stack isn't quite up to the task of sending large amounts of data reliably and quickly. One bug was found and fixed that was creating hard hangs, but in the process of finding said bug, I decided to try a UDP implementation instead—namely, a very basic TFTP server—and found I really liked the possibilities that opened up.

So now what I'm working on instead is a slightly more-ambitious project. It'll still do everything savehost was going to do, but the new program, mustafa (named after John Ratzenberger's character from Ratatouille), is going to be a little more generic and hopefully useful. Above and beyond savehost's goals, it will (okay, should):
Based on my initial tests, it's doing about 40 KiB/s transfer rates, which is great for most saves and probably most homebrew files. I'm also patching up the last feature-complete version of tftpy so that I can create a few PC-side GUI clients in Python, including one particularly neat idea that I actually half-implemented with savehost: a mass save backup utility that, once started, only requires you to swap cards on the DS as you work through your collection. No keypresses needed.

Hopefully, this will all be done in a few weeks.

Labels: , , , ,

posted by zigg 6:24 AM 0 Comments