Newfound respect for FSHost

jonf · September 21, 2003, 08:59:18 PM

Just thought I'd share this....

A couple of hours ago I had a server crash. The web server, teamspeak and FTP servers are all offline. I can't even PC anywhere in. Trying to access the machine across the LAN gives errors. Can't even force a remote reboot. No choice but to drive into the office and do a manually reboot.

Now for the interesting part.....FSHost is still running and people are still flying. The whole server goes tits up (pardon my French) but FSHost keeps on ticking! Now that's impressive!!!

Russell Gilbert · September 21, 2003, 09:21:09 PM

Oh, did I forget to mention that in the last email? This version has a new feature that crashes all other software on the machine, in order to have the entire machine to itself

Russell

jonf · September 21, 2003, 10:09:29 PM

Writing a program that can crash Windows is easy. Most of the programs I have can already do that. I'd like to see FSHost make Windows better. That would be the feat of the century

jonf · September 22, 2003, 04:45:32 PM

Oh well....got some bad news. Just before I had the server rebooted late last night I logged into the FSHost admin web page and stopped FSHost....Guess what....everything on the machine came back.

The only clue I have is the identical messages I got when I tried to access that PCs hard drive across the network, the message I got when I tries to access the services remotely and when I tried to do a remote reboot...."Server does not have enough storage to perform this command" which means that memory leak I've been seeing reached a point where nothing but FSHost could run on the OS.

Also, the problem with FS2004 pilots not being able to connect. I finally experienced this first hand. There were 4 pilots on the system that I fly with regularly and another that I don't see very often. Anyway, I could not log in. I got the message saying that it couldn't send the info to all pilots. I tried to connect again and while it was trying to connect but just before I would have gotten the error message I asked him (the pilot I don't see often) to disconnect. The instant he disconnected it let me it but then he couldn't log in. He is not using a router but I am. There are also a few other pilots experiencing the same problem now. I restarted FSHost forgetting that it's set to clear the log file so I'll need to go through what we did again to get a log file for you.

PeterR · September 22, 2003, 04:50:50 PM

Way to go, Russell!

Russell Gilbert · September 23, 2003, 12:37:13 AM

Jon,

I did some more research into the memory leak you mentioned...

In your original email a while back, you said you were on NT at that time, but I gather the machine's been upgraded to 2000 Server, based on this post. You also mentioned in that post that you were checking memory and resource usage with the Task Manager.

I tried doing the same thing on my XP Pro machine today, and found the "USER objects" and "GDI objects" columns in Task Manager (never knew they were there before). I think this is what you were probably using as well, but you mentioned in your other post that the resource usage was in percentages, and it doesn't look that way to me (maybe just different in XP). Anyway, I did find a resource leak that causes the GDI objects to go up by 4 every time you click on one of the column headers in the player list or flight plan list. If you keep clicking the headers, it just keeps leaking more and more.

However, I didn't find any other leaks, and I'm thinking it's maybe not the same thing you were seeing (unless you just like clicking column headers a lot while the server is running).

So.. First, can you confirm what I've said here, to make sure we're on the same page. Then could you please try doing a few things in FSHost (running it, connecting, disconnecting, options window, clicking column headers, etc.) and see if you can find something in particular that consistently leaks resources? I'm thinking that maybe something is different in 2000 Server.

Thanks!

Russell

jonf · September 23, 2003, 01:04:04 AM

When I open the task manager and go to Processes there is a column that says CPU. This number is the percent of cpu usage that process is using. Next to it is Mem Usage and it show the memory being used by the process. The server is 2000 server but the columns should be the same on all Win OS's. The symptoms are the same as they were when I was using NT4.

I'll take a look at the interface and see if I can find anything of interest but the FSHost GUI is never used on my server. FSHost starts, minimizes to the task tray and is never touched. The only significant activity other than pilots flying is to the userX.html files that display a list of pilots online and filed flight plans that load on my home page.

Russell Gilbert · September 23, 2003, 01:08:58 AM

Ok, I guess what I was wondering about was how you were watching the resource percentage:

QuoteI'm running 3b8 and had a fly-in today. When the server started running it was using about 4 megs memory and the resource usage was around 0-2%. As people joined up the resources rose to about 50% which was expected but the memory use increased to 80 Megs and when people signed off the memory use and resources didn't go back down. Even with only 3 connections FSHost was still using half the available resources and 80 Megs of memory. I set the process to only use 1 CPU to help free up resources for other processes but eventually had to stop and restart FSHost to free up memory and resources that FS host had taken. After the restart the 3 pilots reconnected and FS Host is back down to 2-5% resources and 6 Megs memory but as people come and go those numbers will continue to increase, regarless of how many people are connected.

Also, the FSHost GUI becomes unusable. If I try to open it from the task bar then the outline of the window appears but it's too busy to refresh itself or allow any control (move, min, max, close, etc)

The only way I've found to do that in Task Manager is by watching the USER and GDI columns. But then it's not a percentage. Did you use some other way?

Russell

jonf · September 23, 2003, 01:26:21 AM

Here's what I'm looking at on my server....

Russell Gilbert · September 23, 2003, 01:39:11 AM

Ah, ok. Here's what I'm using now to find resource leaks:

Notice the two columns on the right. You can add them by going to View / Select Columns. USER and GDI objects are what the Windows 98 "Resource Meter" used to show, if you ever used that. They're basically all of the resources related to buttons, menus, colors, etc. In Win9X, there were a limited amount, and when you ran out, you crashed. That's why I always suggested people run the Resource Meter in those versions, because you could close down some programs when things started getting close to running out. In WinNT/2K/XP, the resources are "unlimited", meaning that they're only limited by your available memory, hard drive space, or whatever.

When you said you were watching the resource % used, I assumed you meant these numbers. But maybe you just meant CPU usage?

The leak I found today was a true resource leak, and the GDI column incremented by 4 each time I clicked on one of the column headers. If that had been in Win9x, it would have eventually crashed the machine, but I think it'd take a lot longer in XP, if it's even possible to crash it that way -- I'm not sure.

So, let's start over now :-) What are you seeing?

Russell

jonf · September 23, 2003, 02:11:17 AM

Up until now I've only been watching the CPU and Mem Usage columns. When FS Starts the CPU column stays around 0-4 and the Mem Usage is between 4 and 10MB.

For a while everything runs fine but eventually the CPU column will show numbers above 70 and the Mem Usage goes up to 80+ Megs. Even when everyone logs off and the server is empty these values don't come back down.

I'll enabled the other columns and see if I can gather more info. It'll be hit or miss trying to get the info, though. There is usually very little time between when the problem starts and when I am still able to get into the server remotely to see it and shut down the process. Usually I don't know about the problem until it's too late. It could happen within a few hours of starting the server or it could be a few days.

Russell Gilbert · September 23, 2003, 02:16:24 AM

Ah, ok, that's what I was starting to figure out... When you said resources, I was thinking user interface resources. Well, I wouldn't worry about those two extra columns then, but I ended up finding a bug there anyway, so it was good I looked into it :-)

As for what you're seeing though, I'm mystified. I've run several multi-hour sessions over the last few weeks, sometimes with around 15-20 players, and have never seen the cpu or memory usage go very much higher than it was when it started. I also have the compiler set to watch for memory leaks, and it never reports anything. I'm definitely starting to suspect something in 2000 server now -- I talked to someone else that said their company has dropped support for the Server products completely, just because of all the headaches they had supporting them. I don't have any plans of doing that, but it does tell me that the Server OSs are quite different in some aspects.

Anyway, let me know if you learn any more...

Russell

Russell Gilbert · September 23, 2003, 02:17:16 AM

Which version of DirectX does that machine have, btw?

Russell

jonf · September 23, 2003, 02:30:45 AM

9.0b

jonf · September 23, 2003, 11:38:11 PM

I don't know how important this is but here goes...

One of the columns I enabled in the Task Manager is "Handles". When FSHost starts there are about 600 handles. I just took a look and that number is now up to over 1,200 on my main server after 4.5 hours. No one is on right now but people have been using it over the past few hours. Our race server has also been running for the same amount of time but no one has been on and the "Handles" for that process are at 630.