Jump to content

Phoenix125

server manager AtlasServerUpdateUtility (ASUU). Server Manager. Blackwood. Backups, Mod Updater, Crash Detection, Discord/Twitch

Recommended Posts

@Phoenix125  Sorry, but I'm still not certain where to obtain the logs you are looking for. I couldn't any logs aside from the _logs folder in ASUU.  Google was not much help in that it directed me to Windows 10 Event viewer but that did not seem to contain usefull logs.

Share this post


Link to post
Share on other sites

@Phoenix125

Thanks for the response. Based on the log file it looks like you may not be freeing some memory if the rcon command fails. I'm running a 6x6, but only the center 4x4 are turned on normally. The outer ring are special grids with the power stones and whatnot. Plan is to turn one or 2 on when we're ready. so every time it tries to check the player count, 20 servers are off. Long story short the log file consists of the following every 3 minutes:

Quote

2020/02/06 16:43:07 [Online Players] Error! No response received from "ListPlayers" RCON command to server (A1).
2020/02/06 16:43:07 [Online Players] Error! No response received from "ListPlayers" RCON command to server (C1).
2020/02/06 16:43:07 [Online Players] Error! No response received from "ListPlayers" RCON command to server (E1).
2020/02/06 16:43:07 [Online Players] Error! No response received from "ListPlayers" RCON command to server (F2).
2020/02/06 16:43:07 [Online Players] Error! No response received from "ListPlayers" RCON command to server (A3).
2020/02/06 16:43:07 [Online Players] Error! No response received from "ListPlayers" RCON command to server (F4).
2020/02/06 16:43:07 [Online Players] Error! No response received from "ListPlayers" RCON command to server (A5).
2020/02/06 16:43:07 [Online Players] Error! No response received from "ListPlayers" RCON command to server (B6).
2020/02/06 16:43:07 [Online Players] Error! No response received from "ListPlayers" RCON command to server (D6).
2020/02/06 16:43:07 [Online Players] Error! No response received from "ListPlayers" RCON command to server (F6).
2020/02/06 16:43:07 [Online Players] Error! No response received from "ListPlayers" RCON command to server (A1).
2020/02/06 16:43:07 [Online Players] Error! No response received from "ListPlayers" RCON command to server (C1).
2020/02/06 16:43:07 [Online Players] Error! No response received from "ListPlayers" RCON command to server (E1).
2020/02/06 16:43:07 [Online Players] Error! No response received from "ListPlayers" RCON command to server (F2).
2020/02/06 16:43:07 [Online Players] Error! No response received from "ListPlayers" RCON command to server (A3).
2020/02/06 16:43:07 [Online Players] Error! No response received from "ListPlayers" RCON command to server (F4).
2020/02/06 16:43:07 [Online Players] Error! No response received from "ListPlayers" RCON command to server (A5).
2020/02/06 16:43:07 [Online Players] Error! No response received from "ListPlayers" RCON command to server (B6).
2020/02/06 16:43:07 [Online Players] Error! No response received from "ListPlayers" RCON command to server (D6).
2020/02/06 16:43:07 [Online Players] Error! No response received from "ListPlayers" RCON command to server (F6).

I read over your source and I think the issue is lines 7725 through 7734

Quote

    Local $mOut = Run($aMCRCONcmd, @ScriptDir, @SW_HIDE, $STDERR_CHILD + $STDOUT_CHILD)
    If $mWaitms > 0 Then
        Local $tTimer1 = TimerInit()
        Local $tExit = False
        While ProcessExists($mOut) And $tExit = False
            Sleep(50)
            If TimerDiff($tTimer1) > $mWaitms Then $tExit = True
        WEnd
        ProcessClose($mOut)
        Local $tcrcatch = StdoutRead($mOut)

It looks like you're opening a handle to STDOUT and STDERR, but only reading STDOUT. StdoutRead() frees the resources for STDOUT, but the resources are never freed for STDERR. I'm guessing that STDERR is being populated when mcrcon.exe is sent SIGKILL from ProcessClose(). However, even if there weren't data in STDERR, just leaving the handle open would cause a slow memory leak.

I think you want StdioClose()

https://www.autoitscript.com/autoit3/docs/functions/StdioClose.htm

I hope this helps.

Share this post


Link to post
Share on other sites

@Neitfall This is the most helpful log for me:
image.png.da534a9738bb7a45b1d578cd1d06bd36.png

@MeatShield Thank you!  I'll certainly give that a try!  That code was so old I had forgotten about it.  As for your grids that are "turned off"... ASUU shouldn't be trying to poll them if you have them off... therefore I'm assuming they're being run on a separate PC?  If so, you could set them to Remote and disable "poll remote": It would stop polling on the running remote grids, too, but it would decrease the log spam and some time.  Maybe someday I'll add a selection for which remote grids to monitor.
image.png.5566ad3e52a2375fcd853baca5a50347.png

Edited by Phoenix125

Share this post


Link to post
Share on other sites

@Phoenix125

Yup, I have 2 physical servers each running half of the grids. The grids on the other servers are marked as remote, but I am polling them. I will disable Poll Remote Servers on one of my 2 and see if that makes an impact.... and 2 minutes later the one I disabled it on stopped growing.... I swapped and the growing swapped. So..... I think we know where the problem lies.... I turned them both to off and I'll let them run for a few hours to see.

 

I had a thought about the hanging. I really don't know much about AutoIT but maybe there is a limit on how many handles it can have open and since they were not being closed it eventually ran out and doesn't handle it gracefully. I think everyone with the hanging issue also have the memory leak issue.

Share this post


Link to post
Share on other sites

@Phoenix125

Awesome! Can we add one more bug?

When you change the setting PlayerDefaultNoDiscoveriesMaxLevelUps it gets written to the game.ini file with a space before the equals sign.

ex:

PlayerDefaultNoDiscoveriesMaxLevelUps =35

 

Share this post


Link to post
Share on other sites
5 hours ago, Grimmy347 said:

So why is windows defender detecting this as a trojan??

False Positive. Mine detected the keepaliveutility as a Trojan when extracting from the zip, but then did not once it was extracted and I ran the scan a second time.

Share this post


Link to post
Share on other sites
7 hours ago, Grimmy347 said:

So why is windows defender detecting this as a trojan??

The keep alive program operates like some of the old school malware. You'd have 2 programs each checking to see if the other was running and restart it if it wasn't. This made it difficult for normal users to kill the process. That's basically what the keep alive program does.

I added an exception for it.

Share this post


Link to post
Share on other sites

@Neitfall Thanks for the log  file.  I'll take a look, although @MeatShield may  have found the cause.. a memory leak.

@MeatShield You found the memory leak!  THANKS MUCH!

After a couple hours of testing using:

Quote

Run(@ComSpec & ' /C DIR "' & @ScriptDir & "*.*" & '" /B /A-D /S', @ScriptDir, @SW_HIDE, $STDOUT_CHILD)    and
Run(@ComSpec & ' /C DIR "' & @ScriptDir & "*.*" & '" /B /A-D /S', @ScriptDir, @SW_HIDE, $STDERR_CHILD + $STDOUT_CHILD)

...  the "fix" wasn't the StdioClose() nor the $STDERR_CHILD, but rather the way I was getting the StdoutRead.

  • The $STDERR_CHILD made no difference in memory. (I was surprised).
  • Executing StdioClose() too early caused AutoIT to freeze.
  • The "standard" way would of getting Stdout would cause ASUU to get stuck in a loop occasionally.
  • So I tried a diff method in ASUU  previously...  Unfortunately, that way caused the memory leak because it never wiped the StdoutRead from memory.

 

So now I have a hybrid that has been working great so far in testing:

Old:

    Local $mOut = Run($aMCRCONcmd, @ScriptDir, @SW_HIDE, $STDERR_CHILD + $STDOUT_CHILD)
    If $mWaitms > 0 Then
        Local $tTimer1 = TimerInit()
        Local $tExit = False
        While ProcessExists($mOut) And $tExit = False
            Sleep(50)
            If TimerDiff($tTimer1) > $mWaitms Then $tExit = True
        WEnd
        ProcessClose($mOut)
        Local $tcrcatch = StdoutRead($mOut)

New:

	Local $mOut = Run($aMCRCONcmd, @ScriptDir, @SW_HIDE, $STDOUT_CHILD)
	Local $tTimer1 = TimerInit()
	While 1
		$tcrcatch &= StdoutRead($mOut)
		If @error Then ExitLoop
		If TimerDiff($tTimer1) > $mWaitms Then
			$tcrcatch = "ERROR! " & $tcrcatch
			StdioClose($mOut)
			Sleep(250)
			ProcessClose($mOut)
			ExitLoop
		EndIf
	WEnd
	Return $tcrcatch

Thanks again!

Unfortunately, a work project took my free time this weekend, so I prob won't get the fix out for a couple more days.
(I'm a hospital nurse. We're upgrading our main software (Cerner) next weekend. I had to develop a program to create an offline dump of all Order Sets used by our physicians during the downtime.  http://www.phoenix125.com/downtime.html )  Thanks for understanding.

Edited by Phoenix125

Share this post


Link to post
Share on other sites

@Phoenix125

Take a look at ProcessWaitClose() It blocks until it's done and it accepts a timeout parameter. You should be able to get rid of the spin wait, and the timer. (Sorry in advance, I don't have an AutoIT IDE, and have never touched it before.)

Quote

Local $mOut = Run($aMCRCONcmd, @ScriptDir, @SW_HIDE, $STDOUT_CHILD)

;ProcessWaitClose returns 1 if it exited, otherwise it timed out
If ProcessWaitClose($mOut, $mWaitms) = 1 Then
  ;Ran successfully
  $tcrcatch = StdoutRead($mOut)
Else
  ;Timedout
  ProcessClose($mOut)
  $tcrcatch = "ERROR! " & StdoutRead($mOut)
EndIf

StdioClose ($mOut)
Return $tcrcatch

Again, no rush.

 

 

Share this post


Link to post
Share on other sites

I am having an issue with the Mod Updater. I had my server up and running with three mods and we decided to add more. I added those mods to the json file after completely exiting the Utility.  When I restarted the utility it detected the new mods downloaded them and then started the servers. About 2 minutes later it said the modlist changed (It did not) and it shut them down updated nothing and started them again. This happened about 5 times  so I bailed out of the program. and closed everything again. Started up it said new mods were added and did the same thing. I am on the third iteration of that now and it is still bringing them up, shutting them down then repeat.

 

Anyone seen this before and know how to resolve it?

 

Thanks!

Share this post


Link to post
Share on other sites

@Blackavar Sorry. I do not know why it's doing that, and I'll try to reproduce your issue to fix it, but in the meantime, here's how you can "fix" it:
ASUU stores the current ModIDs in a temp file.  It scans the ServerGrid.json every 10 seconds and compares that with the stored ModIDs.  If it detects a change, it reboots. (People requested this feature).

For some reason, it doesn't seem to be updating the temp file for ya, so you can do it manually (while ASUU is shut down):
Open       \[ASUU]\AtlasUtilFiles\AtlasServerUpdateUtility_cfg.ini
Put your ModIDs here:      aServerModList=xxx

Another way is to put your ModIDs in using ASUU's Grid Configurator (if you can keep ASUU from rebooting too  quick):Modids2.png

 

@MeatShield Thanks for the suggestion, but ProcessWaitClose() had an unacceptable delay.. that's why I ran the While... End loop in the old code.  And now, after your discovery, it would cause the same memory leak issue. It appears that I have to perform the StdoutRead before the CMD window closes in order to clear the buffer... no idea why, but that's what my testing showed.

Good thinking, though. I certainly am open to other ideas as your discover them. Thanks!

Edited by Phoenix125
  • Thanks 1

Share this post


Link to post
Share on other sites

@PHOENIX125

Thank you! Your explanation of how it handles it  helped me figure it out. Our server is super locked down so it was not able to write the temp file. That was causing it to go in the infinite update loop. I have corrected the file perms and the problem is fixed!

Thanks for the fast response!

Share this post


Link to post
Share on other sites

@Phoenix125  Small QoL request. Would it be possible to add a button to the quick select grid buttons called "ON". The ones called "ALL", "NONE", "INVERT" etc. This button would select any grids currently running.  We typically have 12-20 grids running at once and having a way to quickly select them all to do a shutdown, etc would be great.

Share this post


Link to post
Share on other sites

Anyone have any idea for a cause of this??:::

Server cluster launches fine if either vanilla, or with a single mod.

If I attempt to launch with more than one mod, the servers all hang just after BP Compiles and core numbers are reported, and do not complete the launch procedure. (Using approx 675mb ram each and cpu usuage is either zero or up to 0.2)

This occurs regardless of what mods are tried and order. 1 is fine, more than 1 is a lockout.

Edited by ChupaA

Share this post


Link to post
Share on other sites

New Update! v.2.1.3 (2020-02-23) Memory Leak Fix! Fixed params with spaces. Added Mod Update notes to announcements.
- Fixed: Memory Leak! All output from RCON was not being properly dumped, therefore ASUU was buffering all RCON activity. (Thanks to @MeatShield for reporting and finding cause of memory leak).
- Added: Mod update notes can now also be sent with Discord announcements. (Thanks to Doublee for requesting)
- Fixed: Auto Update would restart the old version instead of the downloaded version.
- Added: "Running" button to main window (in addition to the "All", "None", "Local", etc. (Thanks to @Neitfall for requesting)
- Added: Grid Configurator: AddedEggHatchSpeedMultiplier, BabyMatureSpeedMultiplier values. (Thanks to @Norlinri for requesting)
- Fixed: Grid Configurator: Several parameters were adding a space between param and =. ie. PlayerDefaultNoDiscoveriesMaxLevelUps =35. (Thanks to @MeatShield for reporting)

NOTICE!  Due to an error in v2.1.2, AFTER YOU COMPLETE THE AUTO UPDATE, ASUU will restart the old version. You will have to close ASUU and manually start the new AtlasServerUpdateUtility_v2.1.3.exe in the ASUU folder. Sorry.
image.png.f6e0a25ec26cfcc88e33131f1d880238.png Then run  "AtlasServerUpdateUtility_v2.1.3.exe" or the 64-bit version in the ASUU folder.

v2.1.4 & v2.1.5 (2020-02-24) released.. Hotfixes.

Edited by Phoenix125

Share this post


Link to post
Share on other sites

@Phoenix125  Thanks for the "Running" button. Sure makes live easier.  The left side of the button has the wrong radius for the corners. Minor fix the next time you update something else.

  • Like 1

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×