This is a really interesting idea, and one that I'm not exactly sure how to do. I already have a couple of test client programs I use for various things while working on FSHost, so I have a basis already. But I think we need to work out exactly what this program would do.
I'm trying to think of all the cases I've run across with networking problems, and figure out how I'd solve them with this test program. So help me out here...
First, we have to decide if we're going to support 2002 also, or just 2004. A quick glance at FSHostSpy shows 13 of 43 players using 2002 (30%). But I guess the real question is how often we have to deal with support issues related to 2002. I should also say that 2004 is a lot easier to deal with (programmatically, anyway), since it uses fewer ports, and the ports are more predictable.
Second, we have to figure out which troubleshooting scenarios we'll support. Here are a couple I can think of for 2004 (I'm sure there are others as well)...
1. FS sends a packet from its own port 23456 to FSHost's port 23456, but the response doesn't make it back to FS. Can happen if a router on the FS machine is forwarding the port to a different machine on the local LAN.
How to detect it: send a packet from own 23456 to FSHost's 23456. Find out somehow that FSHost received it and the reply didn't make it back to us. Suggest that the user check that 23456 is not already forwarded to another machine on the local LAN. Or... Find out somehow that FSHost never received the request. Suggest that the user check to see if something (?) is blocking outgoing port 23456.
2. FS sends a packet to FSHost's 23456, but FSHost is hosting on 23455, so FSHost doesn't receive it, and doesn't reply. In this case, FS also sends a packet to FSHost's 6073 (this is the way DirectPlay works in FS2004 -- 6073 is the DirectPlay default port, so it's always used in addition to whatever the FS user specifies). FSHost replies to this packet, but it sends it back to FS's port somewhere in the 2300-2400 range. (This is also because of the way DirectPlay works) If the FS user is in a NAT situation, the incoming packet on port 23xx is blocked by the NAT server. If the NAT is a router, the port range can be forwarded to fix the problem. If the NAT is an NT/2003 server, it can't (not easily anyway), and the solution is to change FS to use the same port as FSHost (23455), so that when the reply is sent back, it'll be sent back to the same port FS originally sent the request from (23455) and the NAT will automatically forward it. (Note that this issue of the reply being sent back on a different port in the 2300-2400 range only happens when 6073 is involved -- if FS sends the request on the same port that FSHost is using, the reply comes back on the same port)
How to detect it: send a packet from own 23456 to FSHost's 23456. Find out somehow that FSHost never received it, and that it's hosting on 23455. Then send a packet from own 23456 to FSHost's 6073. Find out somehow that FSHost replied to it on FS's port 2350, but that the test program never received the reply. Suggest that the user change FS to use 23455, or to forward 2300-2400. Or... find out somehow that FSHost never received the request. Suggest that the user check to see if something (?) is blocking outgoing ports 23456 and 6073.
So the key problem here is how to know if FSHost received the request, and if so, what port it replied to. I think the best way to get any info from FSHost in this situation is probably through a web page, since that usually works when other things don't. But how do we send a request on UDP port 23456 and then ask FSHost on TCP 80 (HTTP) whether or not it received it. The problem is that it's not actually FSHost that's receiving the request -- it's DirectPlay. And by the time DirectPlay notifies FSHost that it received something, FSHost doesn't know which port it actually came in on (23456 or 6073), or which port the reply went back out on (23456 or 2300-2400).
This last part suggests that instead of using FSHost, we really need a separate test server program, and that it needs to be running on a different IP address from FSHost (or on the same IP when FSHost isn't running). That way, we could send non-DirectPlay UDP packets from the test client to the test server, and they could communicate with each other on TCP port 80. So first the client would send a UDP packet on port 23456 saying, "This is test number 42" to the server. Then the client would make a request on TCP port 80 to the server, and ask, "What have you received lately?". And the server would send back, "I received 'test number 42' from IP 184.108.40.206 at 12:34:56." (etc.)
This would mean that for most 24/7 servers, they'd have to have a second machine with a separate IP address, where they could run this test server. Or maybe some kind soul could donate a server for everyone to use (sorry, don't have one, or I'd offer).
Ok, I'm gonna stop talking now, and let you guys come up with all kinds of great ideas that I haven't thought of :-)