Jump to content

Phoenix125

server manager AtlasServerUpdateUtility (ASUU). Server Manager. Blackwood. Backups, Mod Updater, Crash Detection, Discord/Twitch

Recommended Posts

@Phoenix125 Hey man! I just came accross your program here! I was looking at the "other guys" program but he up and abandoned his project so i am looking at yours! I have a question. I am wanting to host a server, 1 or 2 or maybe 3 grids, on my pc and play on said server with myself, a friend and my wife. I have configured and played on my own dedicated Conan server, but i was wanting to know what EXACTLY do i need to do to set this up for my use? I know i'm asking for a pretty detailed explanation, but i tried setting this up and the server started (i think) but i could not find my server on neither the internet nor the lan section on steam. When i run a server, what IP and Port should i use to find it? Should i use my local 192 IP or the public ip? Should i not be able to see it under the lan servers in steam (like i was able to on conan)?

 

LOTS of question and i apologize, but i always get the feeling im doing something wrong....Maybe a guide specifically for what im trying to do would help? BTW i REALLY love the layout of your program. It is 100% the easiest to follow server manager i have found while not sacrificing a lot of customization! And i specifically want to thank you for making it able to modify your Game.ini and GUS.ini for the entire grid with a click of a single freakin button!!!!!

But yes, i would appreciate some detailed instructions on what exactly i need to do, beginning AFTER i download/install the server files from steam.

Share this post


Link to post
Share on other sites

Update! v.2.0.6 Fixed Discord Fast Method, added ModIDs & Online Player List to Configurator, Discord "Ready" status announce delay, & more.
- Added: Option to check for mod updates but not install them until "Update Mods" button pressed. Helps prevent mod update from crashing your server without you knowing. (Thanks to @Neitfall  for requesting)
- Added: Added an "Enabled" check box in the ALL GRIDS section to help prevent executing commands on all grids when intending to do select grids only. (Thanks to @Neitfall for requesting)
- Added: ModIDs field to Grid Configurator. Hover mouse over text to see current mod list with names.
- Added: Discord announcement delay after grid reaches Ready status. (Thanks to Nyt & Infiniti for requesting)
- Added: If more than one RCON port entry is in a GUS.ini file, ASUU warns the user and uses the first entry.
- Added: Execute external script before BACK UP feature added. (Thanks to Infiniti fro requesting)
- Added: ModIDs and Names List added to ServerSummary.txt file at end of file. (Thanks to Anorak1313, AceMan, & Infiniti for inspiring)
- Added: Discord: If error 429 (Too many requests), ASUU skips retries.- Added: Online Players list when you hover over grid name/number in Grid Configurator.
- Fixed: Backup: Added DefaultGUS, DefaultEngine, etc to backups. (Thanks to @Neitfall for reporting)

- Fixed: Discord Fast Method: The code was improved. It should work on more systems. (Thanks to AceMan, Linebeck, and others for reporting)
- Fixed: "Error allocating memory" error. I fixed three known causes of this error; Most likely cause was a large log file. (Thanks to AceMan for reporting)
- Changed: Number of Online Player RCON retry attempts (0-3) ###= increased max number to 9 (Thanks to Nyt for requesting)
- Changed: Log: Added more entries/details for server restarts, more details in Discord Fast Method log entries.
- Changed: Batch File folder no longer deletes the entire folder, but batch files still get overwritten every time ASUU is started. (Thanks to @BentoFox for requesting)

image.png.40a6464b471091bd4dae4b0e1d5ae381.pngimage.png.a6cd955049f2fc4482362cd305e96ce9.pngimage.png.48b69f0973fd164b35bdbfb7b174be77.png

  • Like 1

Share this post


Link to post
Share on other sites

@Bovin Did you figure out your issues?  Sorry it took so long for my reply. I occasionally forget to check this forum. Most activity is on my Discord, but I'll try to remember to check here regularly.

Share this post


Link to post
Share on other sites

Hello,When used multiple instances of AtlasServerUpdateUtility When If one instance has a server crash, it will cause another instance's server to crash. Can you look at this? Seems to be caused by PID conflict

Share this post


Link to post
Share on other sites

@Yet Thanks for the report. 

  • I assume you set ASUU to run multiple instances in  the config?  image.png.46997b3ab6ff6cbda911a174522a4a0b.png
  • ASUU shuts down servers by RCON first then PID, as you suspected. 
    • Check to make sure that RCON ports are not duplicated in both instances.
      • This is the likely reason: by default, ASUU assigns the same ports via the wizard. 
      • If server A1 is assigned port 25710 on  both instances, it  would then end up shutting both down.
      • Try changing the RCON port assignment.  Once done, the servers will have to be restarted.
    • If that's no the issue, check to see if ASUU failed to properly detect/assign running grids by comparing a few PIDs from one instance to the other.  This shouldn't happen as long as you have the "Allow multiple instances of AtlasServerUpdateUtility? (yes/no) set to no, but it's the only other thing I can think of.

Let me know how it goes... I'm working on an update right now.
Thanks!

Edited by Phoenix125

Share this post


Link to post
Share on other sites

Rcon port is not duplicate. The conflict seems to come from the allocation of duplicate PIDs. When one instance crashes, the existing PID of another instance will be allocated, resulting in chain crash.

Share this post


Link to post
Share on other sites

Allow multiple instances of AtlasServerUpdateUtility? (yes/no) ###=yes

I'm running an 11 * 11 and a 7 * 7 grid

Share this post


Link to post
Share on other sites
17 hours ago, Yet said:

Rcon port is not duplicate. The conflict seems to come from the allocation of duplicate PIDs. When one instance crashes, the existing PID of another instance will be allocated, resulting in chain crash.

PIDs are assigned by Windows when any process is started.  There can never be duplicate PIDs running on the same PC.
Therefore, I do not know what the issue is.. I was certain it would be the RCON ports.
Do you mind PM or email me the detailed (Full) logs from your two systems for the same day (but not today.. a full 24h would give the best info)?  This will help me significantly. 
kim@kim125.com

Thanks!

Share this post


Link to post
Share on other sites

Update! v.2.0.7 Bug Fixes. Improved ASUU "already running" duplicate detection. Added a total of 140 ini parameters to Grid Configurator.
- Added: grid configurator now has all known (and some untested) parameters from https://docs.google.com/spreadsheets/d/1X4nY0xMzsr65ud5ugvhdF2ZItDXcacbEBxxudfxseyM/edit#gid=607610888 (Inspired by DFS)
- Fixed: Line 43163 Variable Used Without Being Declared Error. (Thanks to msmcpeak for reporting)
- Fixed: MUCH NEEDED! Added a third (and more reliable) ASUU Duplicate Instance check routine to help prevent unintentional multiple instances of ASUU.
- Changed: Removed Underscore _ from filenames in main ASUU folder, except the _Start.bat file. (Thanks to Linebeck for requesting)
- Changed: ServerSummary creation only gets logged in the Detailed log now to prevent it from being broadcast to Discord. (Thanks to Linebeck for requesting)
- Fixed: When closing the Grid Configurator or any of the Wizards, all fields are now saved  by using a new close button. (Thanks to msmcpeak for reporting and helping diagnose issue)
- Added: Web Links to NAT Loopback Help, Redis Desktop Manager, and Fix "Starting" Grid Status notice.

Share this post


Link to post
Share on other sites

@Yet Thanks for showing the screenshots. I now understand what happened.
I've only had it happen once before, although it was NotePad, not an Atlas server, that was "Duplicated": I had forgot about it. The explanation can be confusing, but here goes:
- When ASUU starts Atlas grids, it saves the PIDs to a file.
- When ASUU starts, it reads the PIDs from that file and compares those PIDs with programs already running.
- During a scheduled restart, all grids shut down.  Server C2 was PID 6292. Currently ASUU does not delete the PID from the file, therefore even though C2 is no longer running, the PID file still has C2 assigned with PID 6292.
- So, what happened was one instance of ASUU started the grids.. and coincidentally,  Windows assigned a new Atlas server (A5) with PID 6292.
- When the other instance of ASUU started, it read that PID file and compared all the old PIDs (from the file) with current running processes. 
- It detected a running process with PID 6292 (grid A5), and since PID 6292 was previously assigned to grid C2, it went ahead and assigned it to C2.  Therefore, in reality,  there is no C2 server, even though ASUU says there is.

In the explanation above, I do not know if C2 or A5 was originally assigned PID 6292, so I could have the order reversed, but now I understand what the issue is.

- This only occurs when multiple ASUU instances are allowed.  When only one instance is allowed, ASUU uses a different Atlas server detection process.
- I need to have ASUU unassign PIDs when it shuts down grids!  THANK YOU FOR REPORTING WITH  DETAILS!  I will fix it in next week's update.
 

To fix for now:  (either C2 or A5 is not running, but to resync ASUU, you'll need to ensure both are shut down).
1. Close one instance of ASUU.
2. Shut down the PID 6292 grid (using the running ASUU).
3. Start the other instance of ASUU.. It should detect the one grid not running and restart it.
4. Start the grid again on the "running" ASUU.

That should get everything back into sync.

Edited by Phoenix125

Share this post


Link to post
Share on other sites

The problem is solved by two instances using different settings:

Example 1:

Server AltSaveDirectoryNames Pattern: (1) for 00,01,10,11 (2) for A1,A2,B1,B2 (3) Custom (Enter below) ###=2

Grid naming scheme: Use (1) 00 01 (2) A1 A2 (3) 0,0 0,1 ###=3

Example 2:

Server AltSaveDirectoryNames Pattern: (1) for 00,01,10,11 (2) for A1,A2,B1,B2 (3) Custom (Enter below) ###=1

Grid naming scheme: Use (1) 00 01 (2) A1 A2 (3) 0,0 0,1 ###=2

I hope it will help your update

Share this post


Link to post
Share on other sites

@Phoenix125 Tried to edit a Grid and encountered an error. I do have grinds running that I did not shut down after updating the utility. Not sure if that matters or not.

error.jpg

Share this post


Link to post
Share on other sites

Sorry about that and thanks for reporting!
I have it fixed and will release the fix tomorrow.  (I'm in the middle of adding CPU affinity and it's hard to release a fix while unfinished code exists.. sorry again!)

Share this post


Link to post
Share on other sites

New Update! v.2.0.8 (2020-01-26) New! CPU Affinity! Bug Fixes.
- Added: CPU/Process Affinity added to Grid Configurator. Affinity can be changed without restarting servers. (Thanks to Infiniti for requesting)
(Affinity only works on systems with 64 or fewer logical processors. If >64, it can only assign first half. (ie, if you have 80 processors, it can only assign first 40). I am working on an update.)
- Fixed: Line 43225 Error. (Thanks to JW2020) for reporting.
- Fixed: Line 49621 Error. (Thanks to msmcpeake for reporting)
- Fixed: ASUU was not unassigning PIDs during some shutdowns. This should prevent misidentifying existing servers. (Thanks to @Yet for reporting)

unknown.png
Tip: Move the CPU Calculator next to the Grid Configurator window to quickly go from one grid to the next.

Edited by Phoenix125

Share this post


Link to post
Share on other sites

Found a minor annoying bug. occasionally ASUU goes non responsive and crashes. Often you can still get it to close by clicking the X.  If you do not open Task Manager and then kill the keepaliveutility when you relaunch ASUU, 2 instances of ASUU will end up launching.

Share this post


Link to post
Share on other sites
30 minutes ago, Neitfall said:

Found a minor annoying bug. occasionally ASUU goes non responsive and crashes. Often you can still get it to close by clicking the X.  If you do not open Task Manager and then kill the keepaliveutility when you relaunch ASUU, 2 instances of ASUU will end up launching.

Thanks for reporting.
I reworked the KeepAlive in the latest version (v2.1.2).  The KeepAlive NOW should automatically detect ASUU lockups then task kill the froze instance and start another.
I also worked on ASUU's detection process (to detect if it's already running), but I did that a few versions ago. FYI: If you have "Allow multiple instances of AtlasServerUpdateUtility? (yes/no) ###=yes" (YES), then ASUU doesn't look for existing ASUU instances.

As for it freezing.. I do not know why it's doing that. If you send me your log when it happens, I can try to figure out what it was doing when it froze. kim@kim125.com

Have you had issues since upgrading to v2.1.2?

Thanks again,
Phoenix125

Share this post


Link to post
Share on other sites

Sorry if it's already been covered but is there any way to make this work cross platform? The guys I play with run xbox and currently we are having to rent a nitrado server. Nitrado doesn't allow access to ini file yet to make changes to certain settings and the map editor has been a pain. Also how many servers could I run at once on my system? 

I-7 devils canyon cpu. 32gb ddr3 2400 MHZ ram 2tb SSD(have room to spare for virtual ram)

Edited by Angerrising

Share this post


Link to post
Share on other sites

Hey @Phoenix125

I really like the utility, I'm running it on 2 servers for my cluster. Couple questions/comments.

I have also had the issue with the 2.1.2 64 bit client hanging. I'm able to kill it and restart it without loosing my servers, so no big deal. Next time it happens, what do you need me to collect to help you track it down. If it was linux, I'd attach strace to it... anything I can do under windows to help figure out what it's stuck on?

Do you have plans to multi thread the rcon calls? Maybe let us a maximum number of workers too. It takes a while to churn through all of the grids and 99% of the time is waiting.

Any chance you can modify the zero pop low cpu priority code to look at adjacent grids too. So a grid will go to low priority if it, and it's adjacent grids, have zero population (or are turned off). It would also be nice to be able to white list grids to always run at normal priority. It can be a little painful sailing to a new grid that is running at low priority when the population check only happens every 30 seconds. Also being able always run freeports as normal, and any grid that has permanent player bases too.

When I start the utility, it takes up about 50 MB of memory. It's increasing at a rate of about 1 MB every few minutes. After a day or so it's around 3 GB. I'm running 2.1.2 x64. I'm hesitant to call it a memory leak, because some many people throw that word around without a firm grasp of what it is... but it really looks like a memory leak. Please let me know if I can help track it down. I don't mind restarting the program once a day or so.

Thanks,

Share this post


Link to post
Share on other sites

@Angerrising

I run a low pop server, so your mileage may vary... Without memory compression I'm averaging about 1.9 GB per grid, 2.7 GB high, 1.4 GB low. With compression they average about 675 MB per, 800 MB high, 500 MB low. The compression takes about a day to get settled. It doesn't seem to impact performance in a noticeable way. I'd bet your bottle neck would be CPU. Which i7 do you have, there's a huge difference between a i7-4770TE and a i7-4790K. I'm running 8 grids on a i5-8259U, and that's comparable to a high end i7 devils canyon. Runs without issues, but again, it's just a handful of people spread across all of the grids.

Share this post


Link to post
Share on other sites
7 hours ago, MeatShield said:

@Angerrising

I run a low pop server, so your mileage may vary... Without memory compression I'm averaging about 1.9 GB per grid, 2.7 GB high, 1.4 GB low. With compression they average about 675 MB per, 800 MB high, 500 MB low. The compression takes about a day to get settled. It doesn't seem to impact performance in a noticeable way. I'd bet your bottle neck would be CPU. Which i7 do you have, there's a huge difference between a i7-4770TE and a i7-4790K. I'm running 8 grids on a i5-8259U, and that's comparable to a high end i7 devils canyon. Runs without issues, but again, it's just a handful of people spread across all of the grids.

4790k

Share this post


Link to post
Share on other sites
3 hours ago, Angerrising said:

4790k

So that's on the high end so that's good. It's going to come down to population for you. For us we've got more grids than people so the majority sit idle. If you're in the same boat, you should be able to run 8 ish. However if you're going to have a good sized population you probably want to have at least 1 core per grid, so 4.

Share this post


Link to post
Share on other sites

@Phoenix125 2.1.2 64-bit hung for me today after running for just over 24 hours.

crash.jpg

Which log files do you need? The files in the _log folder don't seem to contain much useful information.

I'm not certain when this instance locked, but it was running around 11pm as I started up another grid for us. So I don't know if it locked while we were still online or after we all logged off. There's 4 of us playing on my server. Its a Dell R710 Dual Xeon X5675's, 120GB of Ram. Atlas runs in a Windows 10 VM with 20 Virtual CPU's and 96 GB of ram assigned. The VM is installed on a dedicated m2 drive.  We typically have 12-16 grids running. I've had it at 22 grids, but that starts getting close to max memory usage. This was with 16 grids up but ASUU has locked even with only 6 grids up. The Atlas servers continue to run normally even when ASUU is locked. I am able to click the X to close ASUU and get the Windows "This program has stopped responding" to shut it down. I do not have to kill the task from Task Manager.

Just restarted ASUU and its sitting at 61MB of memory usage.

Share this post


Link to post
Share on other sites

@Neitfall  re; memory leak: The following is from my 5x5 on my Dual Xeon E5 2680 v2 after running for several days.  You definitely have, what appears to be, a memory leak.

TjWwYsU-c-sg-YjV9KDw3JAV8mDBtLrP9HM12oejh5tJLzowUO6Iq_lhOTryF8tleHvpt1yARpHSWau8TXOe7gP4I3ZHcdwbQQsBMYOLsHEc9Cqk1MdEllirVRZAkJVJMWrHWBuQ

Someone else reported something similar a couple months ago, so I double checked all the document opening-and-closing routines in the code.. I found two that were opening, but not closing, documents and were therefore potentially sucking up RAM.  He hasn't reported any issues since.  I'll take another look for similar things.

The log I want is the most recent Detailed/Full log when the system crashed. It will let me know what ASUU was doing last.  One other person sent me his logs and it was inconsistent, but maybe after a few more logs, I may be able to narrow it down.

I worked hard on that KeepAlive prog. Make sure you're running it, but assuming you are, I'm disappointed that it didn't work for ya.

@MeatShield re; memory leak: Same info as Neitfall above: Could you send me your most recent detailed/full log also? It may help me narrow down what process is causing the lockups.

I thought about "clumping" together RCON message delivery, but they generally send real fast.  I can send a message to my 25 grids in about 2-3 seconds.. and that's while waiting for a response.  If no response needed, such as sending "DoExit", it takes about 0.5 seconds.  If I grouped the messages into a single text file, then sent that using one instance of mcrcon.exe, it might shave .2 seconds.
Generally, the "slowness" is due to using old method of Discord messages. Try changing the config to :Use Fast Method to send Discord messages? (if problems, disable)(yes/no) ###=yes'  I improved the delivery process in v2.09 I think.. it sends MUCH faster and more reliably now.

Atlas has an issue with reporting active players: When a player travels from one grid to another, he/she often shows up on both grids.  Therefore it is difficult to monitor true patient counts and locations.   Therefore I am working to integrate with the redis server to improve accuracy. I had it working for me, but it didn't work for most, so I needed to gather more redis.db files to see where the differences lie. I hope to have this done within a month or so.

Once I have accurate (or lat least MORE accurate) player reporting, then I'll consider adjusting affinity to surrounding grids.  I realize I could do it now, but I feel it will be more useful after I get the redis integration completed.  Just trying to prioritize my efforts a little. Great suggestion... thanks!

-Phoenix125

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×
×
  • Create New...