Saturday, August 8, 2009
mustafa downloads to PC at 70 KiB/s
Okay, so 70 KiB/s that's not as good as your broadband. Heck, it's not even as good as my broadband, and I'm a cheapskate.But I'm pretty thrilled with it. My early TFTP proof-of-concept code was only able to manage a bit over 40 KiB/s—though you'll fall back to near that (maybe slightly faster? I have tuned a number of key things) if your client makes a request without requesting a larger block size. Use one that can ask for a larger
blksize, though, and mustafa will shave it down to its largest supported size and the transfer will use that, improving substantially on the speed.Apart from speed, my current build of mustafa accepts a filename which it passes to libfat and will transmit that file—or, using the newly-supported tsize and toffset options, any part of that file—to any TFTP client. I also built a little fuzzing tool that will send packets full of garbage to test its resiliency; on a 16 MiB payload, it slowed the whole transfer down by only about 30 seconds (the entire transfer took a hair over 4 minutes otherwise). Write support is only literally another screenful of code away, since I worked out most of the groundwork over this weekend in-between viewings of the featurettes on The Lord of the Rings extended edition DVDs.
I think it's still a way off from being able to share it, though. Right now, I think the core TFTP support is stable and probably even good, but the mustafa and "device" layers atop it (the
fat device handles libfat; card will eventually handle save chip requests and the like) feel a bit early and messy. I imagine I will be going in and reworking some of those. The good news is that they're a lot smaller than the TFTP core.But yeah, I feel like I'm in the home stretch now. Very happy to be where I am with this project.
Labels: code, ds, games, mustafa
posted by zigg 10:37 PM
0 Comments
Thursday, August 6, 2009
savehost's bizarre metamorphosis
I think this is the first project I've ever actually abandoned—well, sort of—but I don't intend to do any more work on my previously-announced Nintendo DS Save Host.It isn't that I don't want to create that tool anymore, but I found myself wanting a different approach to the problem. I was spurred down this road by my finding that dswifi's TCP stack isn't quite up to the task of sending large amounts of data reliably and quickly. One bug was found and fixed that was creating hard hangs, but in the process of finding said bug, I decided to try a UDP implementation instead—namely, a very basic TFTP server—and found I really liked the possibilities that opened up.
So now what I'm working on instead is a slightly more-ambitious project. It'll still do everything savehost was going to do, but the new program, mustafa (named after John Ratzenberger's character from Ratatouille), is going to be a little more generic and hopefully useful. Above and beyond savehost's goals, it will (okay, should):
- add the ability to upload or download files via libfat (meaning, file transfers on or off your homebrew-capable card or cart without shuffling µSD cards around)
- allow any TFTP client—simple clients are included with Windows and Mac OS X and are included in most Linux distro's repositories—to do basic backup/restore of the save chip and basic upload/download of files
- have its TFTP functionality contained within a couple files that can be added to any other homebrew project to give it data transfer functionality (and it's not limited to files!)
Hopefully, this will all be done in a few weeks.
Labels: code, ds, games, mustafa, savehost
posted by zigg 6:24 AM
0 Comments
Wednesday, April 15, 2009
Ubuntu's missing crc32sum tool, in Python
Did I hear you say "I want a crc32sum application for Linux that can not only output CRC32 sums for a list of files, but also handle stdin if I leave its argument list blank, and I don't mind if it's hackish, so long as it works?"#!/usr/bin/env pythonDon't say I've never done anything for you.
#
# A CRC32 summing utility that functions mostly like md5sum or
# sha1sum.
#
# Public domain by Matt Behrens <matt@zigg.com> with NO WARRANTY.
#
# Usage: crc32sum [<filename> [<filename>...]]
#
BLOCKSIZE = 0x10000
import sys
from zlib import crc32
if sys.argv[1:]:
names = sys.argv[1:]
else:
names = [None]
for name in names:
if name is not None:
f = open(name, 'rb')
else:
f = sys.stdin
name = '-'
crc = None
while 1:
data = f.read(BLOCKSIZE)
if data:
if crc is None:
crc = crc32(data)
else:
crc = crc32(data, crc)
else:
break
print '%08x\t%s' % ((crc & 0xffffffff), name)
Labels: code
posted by zigg 10:24 PM
0 Comments
Monday, April 13, 2009
Nintendo DS Save Host up on SourceForge
Almost two weeks ago now, I uploaded the read-only code I had so far for the Nintendo DS Save Host (which I first talked about back in February) to SourceForge. Honestly this is kind of new to me, putting incomplete stuff out there, but I have at least one person I'm talking to already who is interested in early-access stuff.Having an audience helps a little. It doesn't magick time out of thin air, but it's given me a little more impetus to actually do things. What I did this evening was piece together an algorithm for writing to the 256-byte pages (or 64-Kbyte sectors) that make up the Flash chips on many of the larger Nintendo DS games. On these chips, a page must be erased completely, then rewritten completely, which I suppose is a requirement I could have left in the hands of the client authors, but what's the fun in that? They'd have to download entire pages over the wire, modify them, then send them all back.
So I implemented a routine that will simply let them write at any ol' offset, and it will walk as many pages as the write spans, reading out those which aren't slated for complete replacement so they can be patched, then erasing each page and writing its new content back out again.
It's really kind of ugly at the moment, and it's not hooked up to actual write machinery; it's just proof-of-concept that runs entirely on my laptop. I think it could stand a little improvement; when dealing with 256-byte pages it's okay to read the whole original page in, but if we're patching 48 Kbytes of a 64-Kbyte sector we might not really need to read the whole sector in first, so maybe instead of reading then overwriting the in-memory buffer I might just read in the portions I am not going to overwrite (Is that premature optimization? Probably. But I'm a perfectionist.)
It's a nice feeling of accomplishment, though. Maybe I'll finish this up before summer.
Labels: code, ds, games, savehost
posted by zigg 9:12 PM
0 Comments
Thursday, February 12, 2009
Perfectionism slows progress yet again
So, I spent a few feverish nights postponing paying contract work this past weekend to work on a little coding for myself for a change... something I really don't do much of anymore. I get it halfway done, it works great, I just need to implement the other half, and what do I do?I start not refactoring it, but thinking about how I should. I start thinking the protocol needs reworking, so I create about a dozen text files working out how I think the protocol should work. I spend probably a good hour trying to load it into Eclipse and figuring all that out. And I haven't made one whit of progress since as a result.
I'm not very happy with myself right now. This code needs to be done, not endlessly reworked. Why do I do this to myself?
Labels: code
posted by zigg 8:41 PM
0 Comments
Saturday, February 7, 2009
Not so much like riding a bicycle
One of the code projects I've wanted to do for a little while now (at least since the release of Daigasso! Band Brothers DX) is a save host for the Nintendo DS. A brief primer, first: unlike with disc- or online-based game systems, software for the DS comes on game cards with the software itself in ROM and some kind of EEPROM or Flash chip for saving. This is tremendously convenient for me, but sometimes I want to either back the save chip's contents up or—more ambitiously—extract and inject parts of the save chip's contents to enable things like content sharing.I did just this in a fairly roundabout fashion early on in the DS' life with the Daigasso! Band Brothers Save Editor (apologies for the ancient page, there... another one of my someday projects is to update some of that!) But it required you to either use the now-unavailable Action Replay MAX GBA/DS (truly a Frankenstein of devices) or else grab the .sav file from a pirate card—something you can do in at least a morally legitimate fashion if you do own the original game card, though it's still not exactly above-the-level... not to mention I always sort of felt like I was sort of supporting the people who were pirating the game. Plus, if you were using a legitimate game card, you were always rewriting the entire save chip, potentially shortening its life versus the rotating-write strategy that DS games often take with their saves.
My savehost program is going to be a better way of doing this all-around, I think. You still do need to be able to boot my code on your DS, though there are non-piracy-enabling options for getting this done. Once runing, savehost will associate with your local wireless access point, display its IP, and then open up a TCP service. Key this IP address into a PC-side client program, and the client will be able to get details about the onboard chip and read and write portions of it. Barring new and interesting save chips (like we saw with Band Brothers DX, actually), the same savehost should work with all manner of PC-side client applications, allowing people to write tools to deal appropriately with save chips as flexibly as possible.
I did a little bit of this back when Band Brothers DX was new, modifying card.c from devkitPro's libnds to hopefully recognize and be able to work with the new chip. I never finished the server at the time, but recently I decided to jump in and try it again. After a false start a few days back (tip: declaring char buf[65536] is a great way to hang a DS) I stayed up late into the night last night and came up with code that implements at least the "read" portion.
It needs some work, though. I was never particularly good at C, anyway, but not having really used it in about a decade—Python has spoiled me so—I'm very easily tripped up by the little vagaries of being closer to the metal, as it were. I wasted 20 minutes last night on an sscanf call that was missing the s part, for example. Before I turned in, I also posted a plea for help on gbadev in tracking down hangs, and the response I saw this morning is likely absolutely spot-on. I'm just not used to this style of programming anymore; I'm used to code that either does what I ask or throws an exception—not the stuff where I have to check every return.
That said, I'm on the right track, and happy to be there. Recent versions of devkitPro seem to be cleaning up a lot of the really hardware-specific stuff that littered most DS homebrew projects and just served to confuse the heck out of me. Barring any really weird bugs, I think savehost might be something I can formally release before too much longer. I just need to re-learn to pedal this sort of bicycle again.
posted by zigg 9:32 AM
0 Comments
Saturday, December 27, 2008
vorbissort: Reorganizing Vorbis comments for the Sansa Fuze
Part of my holiday haul included a 8 GB Sansa Fuze, something I didn't even really know I wanted till I saw it in the Costco Black Friday flyer for $50. My beloved in-laws honored me with this particular gift, which I've been really enjoying over my trusty but aging LifeDrive with TCPMP.One of the key features I demand from a portable music player is, of course, that it play Ogg Vorbis files. The Fuze recently gained this capability via a firmware update, but it is just a teensy bit rough around the edges yet. Notably, album art, which MusicBrainz Picard will put into the Vorbis comments (if you add the coverart plugin, at least), causes problems.
It's not just that the Fuze doesn't display the album art (which is forgivable, but I hope that gets fixed in a future update; MP3 album art displays just fine), but I found out that where the album art is located within the Vorbis comment stream may break tag parsing—and as the Fuze's library is navigated entirely based on tags, this meant that some music was coming across with unknown artists or missing track numbers... making navigating said music a chore.
Thankfully, it's easy to fix. I found that if I move the album art to the end of the Vorbis comment stream, the Fuze seems to be able to handle everything else. The vorbiscomment utility (part of vorbis-tools, for you Ubuntu/Debianites) can be used to edit the Vorbis comments on your typical Ogg Vorbis file. I automate this process with the following script, vorbissort:
#!/bin/shTypically, you'd cd to the directory containing the Ogg Vorbis files you want to re-sort and type vorbissort *.ogg. (If you're feeling particularly brave, find . -name \*.ogg -print0 | xargs -0 vorbissort can do all your Oggs in one go—but don't you dare try it unless you understand what that entire command line is doing.) I used this on probably a good dozen albums and all came across flawlessly. Once I re-ran my MP3s through Picard, giving them id3v2.3 tags encoded in ISO-8859-1 instead of UTF, the Fuze displayed all my music flawlessly.
#
# Reorganize Vorbis comments for the Sansa Fuze, putting album art last in
# the stream, so that other comments don't get ignored.
#
# Public domain by Matt Behrens <matt@zigg.com> with NO WARRANTY. Back up
# your music first.
#
# Usage: vorbissort <filename> [<filename>...]
#
while [ "x$1" != "x" ]
do
tempfile=`mktemp`
echo $1
vorbiscomment "$1" | grep -v ^coverart >> $tempfile
vorbiscomment "$1" | grep ^coverart >> $tempfile
vorbiscomment -w "$1" < $tempfile
rm -f $tempfile
shift
done
Great little player, when it's all said and done. I hope that future firmware versions solve the tag-reading problems.
posted by zigg 7:08 PM
2 Comments
