Nodes slowdown issue: thoughts and conclusions

[Sitra Achara]

Originally Posted by Sitra Achara

Is anyone aware of any precedents where a large company/publisher like Atari has allowed the release of the source code for a relatively recent title like ToEE? Particularly, for a game with a big-name IP?

By the way, iD has recently announced it will release the DooM 3 source code by the end of this year. Makes you go "hmmmm"...

P.S. seems like Nolan Bushnell, a co-founder of the original Atari, is on the board of directors of Atari since April 2010. Getting interesting...

P.P.S. forget it:
1. Nolan quietly slipped away.
2. Reading this, it seems doubtful they have any incentive of releasing the source code.

[Gaear]

Off the bat I'd like to suggest that we first complete the job of identifying the culprit(s) for each case of lag before deciding what to do about them. (I think this would keep us better focused, being as we're relying upon irregular testing here and its easy to get ahead of or behind ourselves and each other.)

Still, my first reactions are:

1. We shouldn't play around with the problem if we really identify it. It needs to be eliminated if possible, not bandaged. Nodes lag has always been nebulous, and who knows what impact half-measures might have across the board? I'd hate to announce "we fixed lag!" and then have the play response ultimately be "not really."
   
2. I'd care far less about the appearance of two efreeti in the fire node than the fact that we may finally kill the single biggest and most widely suffered bug in the history of ToEE (number 2 on the Atari Buglist! ... should have been #1)!

That said, I'm flexible based on further feedback. (Which is why I think this should be done in linear fashion.)

[gazra_1971]

I conducted all of the following tests using 6 pregenerated PCs with a Lawful Good party alignment. In each case, I did not load any game saves when conducting each test (except where noted). Test #2 was done after I had just booted my computer up. All subsequent tests were only done after quitting the game and then starting the game again (not after rebooting my computer).


Fire Node Test #2:

As soon as I started moving my characters in the Fire Node, I experienced a hiccup lasting 2 to 3 seconds. When running my party around while scrolling the screen around at the same time, I experienced many hiccups lasting between approximately 0.1 to 2 seconds. All of these hiccups occurred when my characters were running around the central land loop of the Fire Node, but also part-way up the left path that leads to the Balor Guardian (but only as far up as where the Noble Salamanders are), and on the peninsula where the 2 Efreeti are in the upper right area. When I initiated combat with the 2 Efreeti in the upper right area, I experienced no hiccups or hangs at all.


Fire Node Test #3:

I reverted the game back to its default state before extracting the Fire Node Test #3 file. I experienced no hiccups or hangs when I ran my party up to, fought and killed the 2 Efreeti in the upper right area. At this stage of the test, I should point out that I switched to another task for an hour and a half before quitting that application and returning to the game (which was minimized to the Taskbar during that period). When I was running my party down, I accidentally got into combat with 1 of the Fire Snakes. Everything was fine and running smoothly until I killed the Fire Snake. As soon as I killed that Fire Snake (combat ended), the game hung. I pressed the Ctrl Alt Del keys after waiting 4 and a half minutes (the game was still hung at this stage), which switched tasks to Windows Task Manager. Windows Task Manager reported ToEE as "Running". I quit Windows Task Manager and returned to the game, where-upon the screen scrolled for half a second and then the game hung again. After waiting for 1 minute (the game was still hung at this stage), I pressed the Ctrl Alt Del keys again, but this time it took half a minute for Windows Task Manager to pop up. I returned to the game and this time the game was still hung. I quit the game by ending the ToEE task in Windows Task Manager.
I started the game and loaded my game save from when I first arrived in the Fire Node. I only experienced hiccups lasting between approximately 0.1 to 0.5 seconds when running my characters around or scrolling the screen around. I fought and killed all of the mobs in the Fire Node without any problems. The hiccups became less frequent the more mobs that I killed in the Fire Node. As soon as I had killed all of the Fire Snakes and Large Fire Elementals, I noticed a marked improvement in smooth frame rates when running my characters around and scrolling the screen around the Fire Node. When I killed the Balor Guardian and his 2 Efreeti guards, I noticed another marked improvement in smooth frame rates when running my characters around and scrolling the screen around the Fire Node.
I again quit the game, started the game again and loaded my game save from when I first arrived in the Fire Node. I was only experiencing hiccups lasting between approximately 0.1 to 0.5 seconds when running my characters around or scrolling the screen around. After I had killed all of the Efreeti, Fire Snakes, Large Fire Elementals, and the Balor Guardian, I only experienced occasional hiccups lasting approximately 0.1 seconds from then on when running my characters around or scrolling the screen around.


Sitra's concealed Efreeti test:

I reverted the game back to its default state, then extracted Sitra's modules.rar file. I should point out that during the combat with the Fire Snakes and the Large Fire Elementals in the Fire Node, I switched to another task for an hour before quitting that application and returning to the game (which was minimized to the Taskbar during that period). As soon as I had killed the Fire Snakes and Large Fire Elementals, I experienced a quick succession of about 2 to 4 hiccups that lasted about 1 second each. While running my party around and scrolling the screen around, I only experienced the rare hiccup here and there that lasted between approximately 0.1 to 0.5 seconds. I killed all of the mobs without incident, but the 2 Efreeti in the upper right area weren't there (and they didn't initiate combat with me). I initiated combat mode but I couldn't locate the 2 Efreeti as targets.

[Gaear]

Thanks gazra, food for thought. This is still limited testing, but based on what we've got it looks like we can't completely rule out the KOS flags. I'm dubious about solutions that 'lessen' the problem, because lag seems to always show up differently for everyone, so what's less for us might not be much less for someone else. The heart of the problem is still there.

I've noticed something odd in my initial testing with the Eath Node: the eye particle effects for the gorgons and the dust particle effects for the earth elementals run even if the mobs are turned off. (You see disembodied eyes, et al.) Nice.

[Sitra Achara]

Originally Posted by gazra_1971

Sitra's concealed Efreeti test:

I reverted the game back to its default state, then extracted Sitra's modules.rar file. I should point out that during the combat with the Fire Snakes and the Large Fire Elementals in the Fire Node, I switched to another task for an hour before quitting that application and returning to the game (which was minimized to the Taskbar during that period). As soon as I had killed the Fire Snakes and Large Fire Elementals, I experienced a quick succession of about 2 to 4 hiccups that lasted about 1 second each. While running my party around and scrolling the screen around, I only experienced the rare hiccup here and there that lasted between approximately 0.1 to 0.5 seconds. I killed all of the mobs without incident, but the 2 Efreeti in the upper right area weren't there (and they didn't initiate combat with me). I initiated combat mode but I couldn't locate the 2 Efreeti as targets.

That's weird. Are you able to target them with an Area of Effect spell?
Also, can you verify that the files are ok? They should be 590 and 674 bytes.

Anyway, while the problem isn't completely eliminated, it seems that disabling either the particles or KOS reduces the severity of the problem to a bearable level. While I'd like the lags to be reduced to zero, as they say, "the better is the enemy of the good".

On a related note, I've tried spawning a whole bunch of Efreeti in Hommlet with their KOS unflagged - I get 1 seconds hiccups at most.

P.S. Re. alt+tabbing ToEE for extended periods - in my experience, this can lead to memory leaks and other trouble. Please avoid this in the future.

[Sitra Achara]

Another thing I have noticed -

Swapping out PCs' worn items in the Fire Node seems to lock up the game (for at least 1 minute, perhaps permanently). Has anyone else experienced this?

Edit: it appears to happen only when you load a save game in the node, e.g. if you load a game from Hommlet and then teleport to the node, it's fine.
Also, it's unrelated to the presence of mobs - I killed them all and still got it.
Further, the bug exists in Vanilla ToEE too.

[PKSmith]

I'm not sure how this gels with everything that has been seen in association with the lag issues in these areas (though I've read the whole thread). One thing that struck me is that it might be possible (given the invisibility thing) that the particle engine is trying to calculate the effects of the particles on party members - the reason this might be being done is that spell particle effects might need to be checked against party members for spell effects, if this is hardcoded into the particle engine (maybe to speed up particle using spells) then this would clearly be most visible where there are many particles, regardless of whether they have an effect on the party members of not. This would also mean that the level of graphics displayed would have no effect on the lag since the particles would need to be checked due to the effects they would have regardless of whether they are displayed otherwise particle using spells would not have the same effect across all graphical settings (which would be weird!)

As for the search script, this will have an impact, searching is pretty hard (though it doesn't get to A* levels). Pathfinding is hard. Fog of war probably evens itself out with the things it hides and the things is reveals but at least off-screen things in fog of war are unlikely to be calculated for.

It is possible that these lag problems will never be entirely solved though, even with bugless code since it does seem like it is a combination of many heavy processing things that is causing the lag and each of these might be an issue.

One thought that did give me the chills would be if the particle engine does all particle effects on a map regardless of the area being displayed (without calculating the graphics because they are unneeded and processor expensive). Then when the graphics start being needed the position of all the particles has to be calculated and then graphics matched to that and displayed... this would almost certainly cause an issue when an area scrolls into view and may be worse if the length of time that the particles are not seen for is longer than if they are seen quickly after they are created. It would not be an issue for creatures that appear suddenly on screen since the particles would also start at the same time.

The real nightmare would be if pathfinding was used for particles (which might be a nice touch for horizontally moving particles to flow around things...) - but this seems really, really unlikely since it would combine the two most processor intensive parts of the code (at a guess

Anyway... I hope these ideas might give some fresh ideas as to possible tests and maybe even solutions...

PK

P.S. Has there been any testing of the Fire Node area without an animated environment? - difficult I know (I don't know if it uses the particle engine at all).

[Gaear]

Okay, we're going to gear this operation up again. Are those of you who tested before (gazra) still in? It's good to have at least one other source to verify results.

[Philip]

I'm playing a lot at the moment so while I'm a new face around here I have the time and patience to contribute to some testing for a while.

[Gaear]

Cool. This will be sort of open testing as compared to the jersktop bunch (you can download and test at will), but you will need to be on 7.2.

[Gaear]

Up first is the Air Node. Initial findings for particle effect bearers:

• Ildriss (8) - no
• Vortex (5) - maybe
• Vrock (1) - no
• Large Air Elemental (4) - maybe
• Windwalker (2, concealed) - maybe

As before, I did this by turning all the mobs off and then turning them back on successively by type, keeping the 'nos' on and turning possible offenders back off.

The only other mobs in there are Ashrem, Taki, 8 vapor rats, and a few items.

I wonder ... each of the possibles above has an 'eye' particle effect. I think the only mob in the fire node that had an eye particle effect was the efreeti, the primary offender. The testing I did of the earth node a few months back pointed toward eye-bearing offenders like the gorgons. If this ending up being down to eye particle effects, we could just turn them off specifically (and, for example, restore the effreetis' lower halves).

[Gaear]

Alrighty, more testing shows (on my end) that the air elemental and vortex eye particle effects, alone, are the culprits for the Air Node. (The windwalkers share the mesh and particle effects of air elementals.) Therefore, the fix includes only an adjusted SKA file for each where the references to eye particle effects are broken - very simple. This will mean that the air elementals et al won't have those beady little red eyes.

To test:

1. extract the contents of the attached .rar to your ToEE root directory.
2. start a game.
3. console input
   
   

   from t import *

4. console input
   
   

   t()

   This will send you to Hommlet and make you powerful.
   
5. if you want, console input
   
   

   uberizemajor()

   This will make you more powerful.
   
6. console input
   
   

   tp(5081,477,483)

   This will send you to the center landing point of the Air Node.
   
7. Run around and see what happens. None of the mobs are off or have KOS disabled this time - this is just a hang it all out there test.
8. Let me know what you find.

ALSO, this patch includes a revised SKA file for the efreeti, as described above re: eye particles. Please run around the Fire Node too (per earlier instructions) and see if it works the same. If it does, the efreet can have their lower halves back.

Thanks.

[edit]

Disregard - please see patch in post #76.

[Zoltec]

Sign me up!

I've tested your patch. I'd experience no lockdowns in the fire and air node. ++

Heh! you did a great job here, Gaear!

[Zoltec]

Hmm, OK. At first I did patch it, but I didn't follow your per instructions, I manually run through the fire and air node, all are running smoothly w/ little lags. For the second instance, I've followed your per instructions, I've got the same results "no-lockdown, little-lag". But to be sure, we'll need more testers here.

[Gaear]

More info ...

I searched for all references to 'eyes' particle effects in monster SKA files, and lo and behold, here are the critters that employ them:

• air elementals
• earth elementals (use air particles)
• efreet
• galeb duhr (use air particles)
• vortex (use air particles)

Where do we suffer lag?

• Fire Node - has efreeti
• Air Node - has air elementals and votexes
• Earth Node - has earth elementals and galeb duhr
• Temple Level 4 - has an earth elemental and an air elemental (in Senshock's chambers)
• Temple Level 2, Fire Temple - has air elementals nearby in the Air Temple

Inconsistencies ...

• There are earth elementals in the Earth Temple in Level 1. No idea.
• The air and earth elementals in Senshock's chambers are concealed. Maybe that doesn't actually matter and the game calculates for them anyway. There is at least one earth elemental that is not concealed in the Earth Node though.
• What about the gorgon's beady little eyes? He looked guilty. They actually use the vrock's eye particle effect, which is not problematic for the vrock. The balor also has an eye particle effect that does not appear to be problematic. In both the vrock and balor's cases, their eye particles are described differently - 'Eye Glow,' whereas all the other offenders are described as simply 'eyes.' I have no idea if that's significant, but it could be there is some fundamental flaw with the 'eyes' ones if they all use the same routine (and are hence named the same) that the 'Eye Glow' ones don't suffer.

Notes ...

I'm fairly sure that the general locations on a given map where lag seems to pop up are not directly related to the specific locations of the offending mobs, but are moreso related in general to sectors where the mobs exist. That would explain why you might suffer lag when it seems no offender mob is in the immediate area (like the Fire Temple/Air Temple) - they may belong to the same sector, and the game may calculate effects on the fly as sectors become active.

If this is all actually so, it would mean that the entire concept of 'Nodes Lag' was a f*cking coincidence - that there was nothing wrong with the nodes maps themselves, and that it just so happened that out of 100+ maps, those ones contained the offending mobs. Unbelieveable.

~

Anyway, please do test (thanks Z), and I'll probably put together a comprehensive test patch in the meantime that simply disables the calls to the offending particles for each affected mob. This will cover all 5 lag maps.

[Gaear]

Here is the comprehensive test patch. Breaks references to eye particle effects for air elementals, earth elementals, efreeti, galeb duhr, and vortexes. Intended to fix lag in all locations. Please download and test.

http://files.co8.org/patch/lag_testi...h_05272012.rar

1. extract contents of .rar to ToEE root directory and overwrite
2. start a game
3. console
   

   from t import *

4. console
   

   t()

5. console
   

   uberizemajor()

To test the Air Node:

console

tp(5081,477,483)

To test the Earth Node:

console

tp(5082,484,473)

To test the Fire Node:

console

tp(5083,505,495)

To test Temple Level 4:

console

tp(5080,479,588)

To test Temple Level 2:

console

tp(5067,488,539)

[Gaear]

My own testing results with this are:

Air Node - no lag
Earth Node - one brief blip near the gorgons
Fire Node - significant lag again to the right side
Temple Level 4 - no lag
Temple Level 2 - no lag

Maybe the efreeti have issues with their blue fluffies after all, and maybe the gorgons need to lose their eyes anyway. Will wait for more feedback from you all.

[Zoltec]

What about the gorgon's beady little eyes? He looked guilty. They actually use the vrock's eye particle effect, which is not problematic for the vrock. The balor also has an eye particle effect that does not appear to be problematic. In both the vrock and balor's cases, their eye particles are described differently - 'Eye Glow,' whereas all the other offenders are described as simply 'eyes.' I have no idea if that's significant, but it could be there is some fundamental flaw with the 'eyes' ones if they all use the same routine (and are hence named the same) that the 'Eye Glow' ones don't suffer.

Particle Calculator Queue first-in first-out

Assuming the particle-calculator have a number of queue like [7] and firefrog has the function call of 'eyes', thus if the object is activated, the particle-calculator calls all the sub-processes: "particle effects" who are within the same branch of function parameter 'eyes' limited only to the creatures who have the function call in a particular map, 'IF' there are 9 particle effects for nine different creatures within the map, and does cannot exceed its queue [7] halts down the remaining [2] which creates lockdown to the particular place, 'IF' a particle-effect bearer is activated within your party's range.

Example:

[][][][][][][] Queue Slots for the Particle-Calculator

1 2 3 4 5 6 7 <8,9>

8 and 9 are waiting to be queue'd inside the calculator, which creates temporary lockdown to the nodes. They are the particle effectors w/ a function call of 'eyes'.

Particles effects are the sub-process of the host process.

How will I explain this?

I think if you redistribute the function call 'Eyes Glow' to the other elementals which is labeled as 'eyes' might help the particle-calculator to not exceed the queue slots, thus creating no more lockdowns, maybe by changing the function call 'eyes' of Efreeti to 'Eyes Glow' and other creatures who are particle-bearers within the node.

I don't know if this make any sense to you, but you may try to create a scenario were creatures who have the 'eyes' particle-effect bearers spawned in a map exceeding the # of particle bearers in the nodes, try to move your character. If there is any lockdowns like the one's in the nodes, then my assumption is correct.

[Zoltec]

Here are my test results:

Air Node - little lag
Fire Node - little lag
Earth Node - no lag
Temple Level 2 - little lag
Temple Level 4 - no lag

Very nice results for a low-end computer LoL

[gazra_1971]

Air Node: lag lasting approximately 1 second at worst. Lag only occurred when party was on right side of map, and either on left side of map or when in lower area (I can't remember which).

Earth Node: lag lasting approximately 2 seconds at worst.

Fire Node: lag and temporary lockups lasting approximately 26 seconds at worst.

Greater Temple: lag lasting fractions of a second. 1 temporary lockup lasting approximately 10 seconds when moving along (western) corridor towards room with 3 Greater Temple Ogre Leaders in it (Senshock cast a spell immediately after the lockup). 1 temporary lockup lasting approximately 5 seconds immediately after the combat against Hedrack and his entourage.

Temple of Elemental Evil dungeon level 2: lag lasting fractions of a second.

Conclusion: all of the lag fixes in lag_testing_patch_05272012.rar have significantly improved game performance (i.e. lessened the problem of lag and lockups) in all of the maps except the Fire Node (where the changes made to the mobs have made lag worse in the Fire Node than is experienced in the Circle of Eight Modpack 7.2.0 NC), and I didn't notice any difference in the Temple of Elemental Evil dungeon level 2 compared to 7.2.0 NC (but I'm sure the changes would be an improvement because I have experienced noticeable lag on that map in previous playthroughs of the game).

Note: after I had entered the console codes

t()
uberizemajor()

if I tried to click on an item in a character's inventory, or if I tried to loot a corpse, or if I clicked on the Fire Power Gem (left on the ground after the Balor Guardian died), then the game permanently stopped responding (crashed/hung/locked up). This happened EVERY time that I tried to do any of these things after entering those console codes. Those console codes are seriously bugged!

[Gaear]

Okay, thanks for testing gazra. We will have to simply revert the efreeti to what they were and all should be good again in the Fire Node.

[Sitra Achara]

Re. Efreeti, I'd give the concealment treatment another try.

[Gatts]

I always thought that it's about the animated background (that's why in Fire Note it is worst), I got crashes always when on max resolution + moving screen + party moving. I am not sure if it's based on animated gifs or static jpgs that change so fat that it seems like lava and fire is there. Is there a chance to try fire note with removed graphics? (probably not )

[Gaear]

Every other thing was already tried long ago. It's the particle effects, and happily, lag is now largely gone.

[Gehennis]

Question: is there any game effect to rescuing the folks from the various nodes aside from the usual XP/treasure?

[sirchet]

I know of a few that will join your cause if you free them, but you probably already know that.