RFC: community effort to build portable, scalable debugger protocol

[Lazy Bastard]

I've added a comment about this thread on the main page. Perhaps that will garner a bit more attention.

[Parasyte]

Thanks, Jason.

I setup an IRC channel on Freenode for the protocol project: irc://irc.freenode.net/srdp Anyone is welcome to join and idle or share ideas.

[Parasyte]

Relevant to the topic at hand; The Firebug developers have started "an effort to create a standard remote web-debug protocol" at http://groups.google.com/group/webdebugprotocol?hl=en

While the goals for a web-debugger are slightly different from a low-level-debugger, there is some overlap. What can we do to move forward with our own debugger protocol standardization effort?

[Lazy Bastard]

I suppose one thing that should help soon is that we can begin focusing on the debugging portions of Artemis once the basic hacking features have been established. This should give us a chance to start implementing some concepts.

Other than that, I'm not sure.

[misfire]

I've recently fiddled around with GDB and its serial protocol:
http://sourceware.org/gdb/current/on...cs/gdb_34.html

I'm sure we can take it as an inspiration.

[Parasyte]

That's one of the protocols I suggested as a reference. There are some other interesting ones out there, as well: http://www.kodewerx.org/wiki/index.p...col#References

[misfire]

Oh, sorry, didn't remember that.

By the way, I'd love to implement a virtual filesystem based on SRDP. In the example below, the PS2's memory is mapped to a file in the local filesystem tree, allowing tools like hexedit to directly access the memory. I really like the design.

[Parasyte]

I can't think of any reasons that would not work. It would fit right into the common UNIX environment. And I think your block diagram shows that pretty well.

I was actually curious about I/O and file system support on the target-side of SRDP, on a slightly different note. That could allow mounting memory cards and game discs, for example. But I haven't been entirely sure that it's within the scope of a debugger. (On the one hand, a scalable protocol would still make it possible with various vendor-specific extensions. But I digress.)

[misfire]

Yeah, we could also ignore the filesystem on the target side and use SRDP to provide low-level access to the sectors of a game disc. The debuggee could set up a virtual memory space mapping the disc sectors which allows the debugger to dump the sectors using the standard "read memory" command.

To directly mount the disc, the host must be aware of the used filesystem, of course.

[Parasyte]

I am feeling particularly lazy today, but I wanted to share with you all some recent documentation/proposal work, for historical purposes.

Forgive the long post, and the language. You should also click the links, though, to get a better idea of what I was talking about.

[16:24.33] <Parasytic_> I came up with a few additional ideas this morning; supporting compressed byte arrays/binary blobs
[16:24.59] <Parasytic_> And making the text data type suck a bit less.
[16:31.16] <Parasytic_> I think RLE compression is probably OK for RAM...
[16:33.34] <Parasytic_> But if it's assembly, then there will be different patterns that RLE won't help with.
[16:34.56] <Parasytic_> And considering this is supposed to be generic, I have no intentions to tune any compression algorithms toward any architectures.
[16:51.36] <Viper187> heh
[16:54.10] <Parasytic_> Do you think a floating point data type would be of any use?
[16:55.20] <Parasytic_> I'll put it in the proposal as RFC.
[16:55.48] <Parasytic_> I cannot think of any specific uses, but there may be some.
[17:36.39] <Viper187> any floats we use would more than likely me converted to/from hex, but you never know.
[18:12.40] <Parasytic_> http://www.kodewerx.org/wiki/index.p...tocol#Proposal
[18:12.47] <Parasytic_> Tell me what you think, so far...
[18:14.11] <Parasytic_> As for floats, yes, they will have to be converted. The protocol I'm proposing will have floats written in plain text. Since it is much easier to spit out a text-representation of a non-integer number than it is to attempt to convert between different floating point data storage formats.
[18:23.25] <Parasytic_> That's not the full thing... Obviously {name}:{type}={value}; is very easy to express as a regex.
[18:23.55] <Parasytic_> I haven't started writing about the "object" data type yet, which will confuse things greatly.
[18:25.00] <Viper187> cool
[18:26.28] <Parasytic_> Before anyone bitches about using a text-format data structure... It's a whole lot more scalable than any fixed-binary format. And "scalable" is part of the name, after all.
[18:28.12] <Viper187> That does semm a little odd.
[18:28.16] <Viper187> *seem
[18:28.43] <Viper187> but they'll get used to it, I guess. lol
[18:29.11] <Parasytic_> Well, it _is_ just a proposal. If someone else has a better format, I'm all for it.
[18:31.32] <Viper187> Well, the byte array stuff jimmi started with does have a minor drawback here and there, I guess.
[18:32.34] <Viper187> like it's a char array, but it may as well be int, since we typecast to set addresses to it, and that doesn't work on unaligned bytes
[18:33.24] <Viper187> it's a minor waste of bytes here and there, but not a deal breaker by any means
[18:33.39] <Parasytic_> How is that?
[18:35.17] <Viper187> well, the first 4 bytes are the size of the array, followed by say an address to start dump memory from (4 bytes), and an end address (4 bytes).
[18:35.29] <Parasytic_> Right.
[18:35.33] <Viper187> some things don't always require 4 bytes though
[18:35.48] <Viper187> but th eeasiest way to set those 4 bytes is typecasting the array to int
[18:35.52] <Parasytic_> Yes.
[18:36.06] <Parasytic_> That is one reason I like VLI...
[18:36.32] <Parasytic_> Can an address fit into 16-bit? Sweet ... the VLI is only 16-bits (or at most, 24-bits)
[18:36.48] <Viper187> true
[18:36.54] <Parasytic_> Need a 64-bit number? No problem. 2048-bit? Can do...
[18:37.05] <Viper187> haha
[18:38.55] <Parasytic_> The GCNrd protocol sounds similar to what you're working with, though.
[18:39.29] <Parasytic_> Everything after the "command" field (first four bytes) was more fixed fields.
[18:39.47] <Parasytic_> And those fields changed, depending on the command.
[18:40.03] <Viper187> yep
[18:43.06] <Parasytic_> And some commands were just the 4-byte command itself, with no additional fields. (pause and run commands, come to mind)
[18:43.56] <Parasytic_> It works, but I wouldn't want to use it on everything.
[18:44.48] <Parasytic_> An example of why I wouldn't want to use a fixed-field data structure:
[18:45.42] <Parasytic_> Breakpoints! The MIPS architecture allows using an address mask when setting a breakpoint. I believe it defaults to 0xFFFFFFFC
[18:46.21] <Parasytic_> This allows you to trap on all writes where actual address & mask == breakpoint address.
[18:46.39] <Parasytic_> So you can get writes on half words and bytes, etc...
[18:47.06] <Parasytic_> But ... PowerPC doesn't support a variable mask. Only a fixed mask of 0xFFFFFFF8
[18:48.23] <Parasytic_> So then
[18:48.55] <Viper187> so the reason it was always 4 bytes on N64 was that mask?
[18:49.22] <Parasytic_> If SRDP supports a breakpoint mask when setting breakpoints, using a mask like 0xFFFF0000 is meaningless for some architectures. So putting a fixed "mask" field into the spec is wasteful.
[18:49.37] <Parasytic_> Yeah, pretty much.
[18:50.10] <Viper187> that always pissed me off
[18:51.05] <Viper187> trying to catch a breakpoint on a byte or 2 when the other 2 are being constantly written
[18:51.18] <Parasytic_> Indeed.
[18:51.58] <Parasytic_> I solved it on GCNrd by ignoring the mask and explicitly checking that the actual address == the breakpoint address I wanted.
[18:52.22] <Viper187> haha
[18:52.26] <Parasytic_> Not as flexible, but no where near as annoying.
[18:52.26] <Viper187> nice
[18:53.44] <Parasytic_> But anyway, my point is simply that fixed binary data formats are not scalable enough. If something as silly as a breakpoint mask is a good option to support, but not available everywhere, what other useless crap might you have to cram into it to make the format usable (and useful) everywhere?
[18:55.29] <Parasytic_> I've always kind of felt that one of the problems I will face when trying to convince PS2 hackers of supporting a scalable protocol is that ... they will probably have a slight tunnel vision for PS2. Like, if it exists only outside of PS2, they will likely think of it as "bloat"
[18:56.33] <Viper187> lol
[18:56.39] <Viper187> true
[18:57.14] <Parasytic_> The text format adds a whole bunch of bytes, to be sure ... "bloat" ... but in the end, it also adds so much flexibility, it is mind-bogglingly INSANE
[18:57.58] <Viper187> heheh
[18:58.04] <Parasytic_> Don't need a mask for your breakpoint? Don't fucking put a "mask" argument into the packet. Simple.
[19:01.30] <Parasytic_> Maybe you want to be clever and add a "my-special-super-duper-ultimate-secret-argument" argument into your packets for whatever reason... Do it... clients and servers that don't understand it will just ignore it, like it never even existed in the first place... And you can still reap whatever benefits it might have.
[19:03.46] <Viper187> nice
[19:03.47] <Parasytic_> That kind of flexibility can be done using similar syntax but without "text" per se.
[19:04.03] <Parasytic_> But it would be more difficult to parse and understand.
[19:07.02] <Parasytic_> In fact, if you limit argument names to three bytes or less, you can still do simple 32-bit integer comparisons to check which commands you are parsing. No different than checking a fixed 4-byte binary command.
[19:08.30] <Parasytic_> I don't think the text format seems odd at all, when you think of it like that.
[19:09.34] <Parasytic_> The UTF-8 support is good too, so you can send text strings in any language.
[19:10.10] <Parasytic_> Most games are written by Japanese programmers (that's just the way it is) and many of them like using their own language when they write debug text output.
[19:11.05] <Parasytic_> If you want to send that text through SRDP to a logger or a console ... If it's already in UTF-8, no further processing required.
[19:11.43] <Parasytic_> If it's some other weird format, well, then you might want to send it as a binary blob.
[19:24.55] <Parasytic_> I suppose supporting UTF-16/32 would be a simple matter of requiring a BOM on both of those formats, though. And no BOM just means UTF-8. I'll add that to the spec.

[Parasyte]

I've set up an IRC channel for discussing SRDP and low-level debugging of video games: irc.freenode.org #srdp

Please join the channel if you have any interest at all. I am currently advertising its availability around different game hacking communities to spark interest and discussion. Fir anyone who would like to help, but does not know where to begin: The first step is joining IRC. Start a discussion or join one in progress.

There are a lot of good things going on with this project right now. I've started writing some code for the protocol's reference implementation. I've been blogging about it at http://blog.kodewerx.org/ and I've been sporadically talking about features and trouble areas in IRC. Right now is as good a time as any to actually begin coding, rather than just talking about this whole thing.

[Lazy Bastard]

I've read over the updated documentation here: http://www.kodewerx.org/wiki/index.p...ugger_Protocol

...and re-joined the IRC channel.

Anyway, what are the current (immediate) objectives? In what way (more specifically than garnering support and interest) should people contribute?

[Parasyte]

Current objectives are simple; Write some code!

I have "some code" written, but it's not really useful. It's more of a toy, in the learning phase. I can't decide if it would be helpful to release what I have, even though there is not much to it. I would speculate that it could do more damage than good, though; for example, if it gives me a false sense of accomplishment and everyone else a false sense of progress, then all is lost!

How should people contribute other than showing support and interest?

• Ask questions! The easiest way to get involved with a community project is to ask questions that would would like answered. Questions that spark discussion among members are even better!
• Talk about it! Spread the word. Gather more support and interest from friends, colleagues, random people, hackers everywhere.
• Consider a donation! A donation of code, ideas, hardware, money, are all welcome.
• Join IRC! It's not the world's busiest channel, but some communication does go on there. Please feel free to idle and check in on it every now and again.
• Follow SRDP on Twitter! If you're into that sort of thing, I've set up a Twitter account that I use to dump little tidbits of random info: http://twitter.com/kodewerx

[Lazy Bastard]

Cool; thanks.

I know exactly what you mean, concerning releasing something that isn't well-formed, and the risks of doing so. You have to decide when the turning point is, between raising false expectations and excitement that quickly turns to impatience and disinterest, and waiting way too long to release some, at least slightly, significant progress.

I'll make an effort to contribute some or all of the things you mentioned in the near future.

[Lazy Bastard]

I've stickied this thread.

Parasyte: I'll leave the task to you, but might I suggest a thread-rename (as you've since named this project SRDP), or perhaps even a new thread after much revision...

Anyway, I'll try to actually contribute soon