Awesome work Suslik! The stuttering has always been the number one reason forcing me to quit playing a modded BG.
This work will make all our playthoughs so much more enjoyable!
Thanks again for your hard work!
[SOLVED]Time to deal with the stuttering
Started by Suslik, Apr 11 2012 06:08 PM
135 replies to this topic
#102
Suslik
-
- Member
- 500 posts
Investigator
Posted 18 April 2012 - 03:53 AM
Yeah, same here. Of course it did not force me to quit, but to start over : DAwesome work Suslik! The stuttering has always been the number one reason forcing me to quit playing a modded BG.
And I have sent my implementation to Asc64. Let's hope he will include it in next public TobEx release.
Edited by Suslik, 18 April 2012 - 05:02 AM.
Howto: mage in Baldur's Gate. My lame guide.
List of issues I have encountered in BWP 10.3 tactical
Stutter-fixer for Infinity Engine
While you were reading this signature, I have probably edited the post above. Twice.
#103
Uranium - 235
-
- Member
- 29 posts
Posted 18 April 2012 - 08:58 PM
WEEEEEEEEEEEEEE!
I've always wanted to play a BWP from start to dramatic end but I've never made it much beyond Athkatla because by that point I'm usually playing the game in slow motion, and end up having to ctrl+J everywhere. At some point I eventually just give up because it's not worth it.
I can't believe it was even possible to fix something like this.
I've always wanted to play a BWP from start to dramatic end but I've never made it much beyond Athkatla because by that point I'm usually playing the game in slow motion, and end up having to ctrl+J everywhere. At some point I eventually just give up because it's not worth it.
I can't believe it was even possible to fix something like this.
Edited by Uranium - 235, 18 April 2012 - 09:04 PM.
#104
-guest-
-guest-
-
- Guest
Posted 18 April 2012 - 10:25 PM
Thanks for the hard work investigating (and fixing!) this, Suslik 
I followed this thread closely in silence since you started it, hoping you would find something out and i'm so damn happy that you did.
I have played (from start to finish) a custom selection of mods in the BWP several times, and I am not much for "megamods" as such. I just use the BWP to mix and match some of my favourite mods, without tossing in hundreds of unknown mods into the game, and sometimes, I have gotten plauged by this kind of stutter. So the game does not even need to be "heavily modded" for this to happen. And when it happens, the game can sometimes be over (needs to be restarted from the beginning), so this is really an absolute bugfix of the century, and I hope people take it seriously.
Ascension64, if you read this, I use your Tobex all the time, and i'd be very happy to see this fix included in it. Just so you know
I followed this thread closely in silence since you started it, hoping you would find something out and i'm so damn happy that you did.I have played (from start to finish) a custom selection of mods in the BWP several times, and I am not much for "megamods" as such. I just use the BWP to mix and match some of my favourite mods, without tossing in hundreds of unknown mods into the game, and sometimes, I have gotten plauged by this kind of stutter. So the game does not even need to be "heavily modded" for this to happen. And when it happens, the game can sometimes be over (needs to be restarted from the beginning), so this is really an absolute bugfix of the century, and I hope people take it seriously.
Ascension64, if you read this, I use your Tobex all the time, and i'd be very happy to see this fix included in it. Just so you know
#105
Ascension64
-
- Modder
-
- 5983 posts
Posted 19 April 2012 - 03:44 AM
There is a fundamental problem with the code provided, due to the fact that the hash value will change every time the hash table is grown.
Take, for example, an initial hash table size of 100, and you manage to fill it to the extent that it needs to grow. Also assume that the variable 'test' is present in a bucket with a hash value of 150 and thus a hash index of 50.
Now, add a new variable called 'stuffed', which forces the hash table to double in size to 200. The hash value of 'test' is still 150, but its hash index would be calculated at 150 instead of 50.
Finally, a SetGlobal() wants to change the value of 'test' to 3. The current CVariableMap::Add() proposed will only search 1/10th of the hash table for the 'test' key, starting from the new hash index of 150. In fact, it will not be found because SetSize() preserves the old hash indices and thus the value of 'test' still sits in the bucket at index 50. Therefore, a new 'test' variable gets added, and the hash tables ends up having two copies of the 'test' key.
In commentary, part of the inefficiency of the vanilla hash table actually lies in the SetSize() proc not re-computing the hash values and storing the variables in the appropriate buckets. In this sense, it may not be necessary to modify the Add() or GetHash() procs at all, but just to modify SetSize() so that the existing elements are placed in the buckets appropriate to the new hash table size. Given this, it is still possible to make a further optimisation so that only 10% of the hash table is searched for a given key before the hash table is grown, which will reduce excessive linear probing.
Take, for example, an initial hash table size of 100, and you manage to fill it to the extent that it needs to grow. Also assume that the variable 'test' is present in a bucket with a hash value of 150 and thus a hash index of 50.
Now, add a new variable called 'stuffed', which forces the hash table to double in size to 200. The hash value of 'test' is still 150, but its hash index would be calculated at 150 instead of 50.
Finally, a SetGlobal() wants to change the value of 'test' to 3. The current CVariableMap::Add() proposed will only search 1/10th of the hash table for the 'test' key, starting from the new hash index of 150. In fact, it will not be found because SetSize() preserves the old hash indices and thus the value of 'test' still sits in the bucket at index 50. Therefore, a new 'test' variable gets added, and the hash tables ends up having two copies of the 'test' key.
In commentary, part of the inefficiency of the vanilla hash table actually lies in the SetSize() proc not re-computing the hash values and storing the variables in the appropriate buckets. In this sense, it may not be necessary to modify the Add() or GetHash() procs at all, but just to modify SetSize() so that the existing elements are placed in the buckets appropriate to the new hash table size. Given this, it is still possible to make a further optimisation so that only 10% of the hash table is searched for a given key before the hash table is grown, which will reduce excessive linear probing.
Edited by Ascension64, 19 April 2012 - 03:46 AM.
--------------
Retired Modder
Note: I do not respond to profile comments/personal messages in regards to troubleshooting my modifications. Please post on the public forums instead.
Baldur's Gate Trilogy-WeiDU and Mods
Throne of Bhaal Extender (TobEx)
Contributions: (NWN2) A Deathstalker (voice acting) - (IWD2) IWD2 NPC Project (soundset editing) - (Misc) SHS PC Soundsets (voice acting)
Legacy: (BG/Tutu/BGT) Beregost Crash Fixer 1.9 (18 Jul 10) - (BG2) Enable conversations with charmed/dominated creatures (18 Jul 10) - (BG2) Experience Corrections (18 Jul 10) - (Misc) Platform Conversion Utility RC2 (13 Feb 10)
#106
Suslik
-
- Member
- 500 posts
Investigator
Posted 19 April 2012 - 06:05 AM
I know. The method "SetSize" will call Add() for every element which was present in the table before - the whole thing will be rehashed. The element with hash 150 will be replaced to bucket 150, even though I do not know how they manage it inside of SetSize, but they do fully rebuild the table upon resize. No problem here.Now, add a new variable called 'stuffed', which forces the hash table to double in size to 200. The hash value of 'test' is still 150, but its hash index would be calculated at 150 instead of 50.
Since the table is rehashed upon resizing, it will not happen.Finally, a SetGlobal() wants to change the value of 'test' to 3. The current CVariableMap::Add() proposed will only search 1/10th of the hash table for the 'test' key, starting from the new hash index of 150. In fact, it will not be found because SetSize() preserves the old hash indices and thus the value of 'test' still sits in the bucket at index 50. Therefore, a new 'test' variable gets added, and the hash tables ends up having two copies of the 'test' key.
Where does this statement come from? SetSize() does recompute each value's hash and fully rebuilds the hashtable with new hashes, using my Add() method.In commentary, part of the inefficiency of the vanilla hash table actually lies in the SetSize() proc not re-computing the hash values and storing the variables in the appropriate buckets.
Beside theoretical speculations, I have checked my implementation by setting the default size to 2 and watching the table grow to 4096. And I used debug output before and after every resize operation to ensure that no elements are lost/duplicated.
Edited by Suslik, 19 April 2012 - 06:29 AM.
Howto: mage in Baldur's Gate. My lame guide.
List of issues I have encountered in BWP 10.3 tactical
Stutter-fixer for Infinity Engine
While you were reading this signature, I have probably edited the post above. Twice.
#107
Uranium - 235
-
- Member
- 29 posts
Posted 19 April 2012 - 08:02 AM
I hope this all works and gets in TobEx soon 
I was just about to undertake the massive job of loading new mods with a fresh install but I don't see the point if such a dramatic and critical breakthrough would be missing!
I was just about to undertake the massive job of loading new mods with a fresh install but I don't see the point if such a dramatic and critical breakthrough would be missing!
#108
Suslik
-
- Member
- 500 posts
Investigator
Posted 19 April 2012 - 08:20 AM
For those willing to test my solution you can grab TobEx.exe/Detoured.dll from the first post of this thread. If you replace normal TobEx.dll of version 23 or newer with this one, it should work fine. No need to start over or reinstall anything.
That's a temporary solution until Asc64 approves my code and includes it to official TobEx release.
That's a temporary solution until Asc64 approves my code and includes it to official TobEx release.
Howto: mage in Baldur's Gate. My lame guide.
List of issues I have encountered in BWP 10.3 tactical
Stutter-fixer for Infinity Engine
While you were reading this signature, I have probably edited the post above. Twice.
#109
Suslik
-
- Member
- 500 posts
Investigator
Posted 19 April 2012 - 10:01 AM
I have come up with a workaround for those who do not have TobEx!
As a workaround you can try this:
If you encounter such kind of stuttering, you can do following steps:
1) Download archive "Stutter Remover.7z" from the first post of this thread
2) Unzip a text file in it anywhere you want. Alternatively you can just copy-paste it from here:
3) Open your baldur.bcs or any other script that is executed with Near Infinity and add contents of "Stutter Remover.txt"(or copy-paste that spoiler) at the end of it. Compile and save your script.
4) Load your stuttering savegame, press ctrl+space and type:
5) If the stuttering disappeared, proceed to step 6). Otherwise try setting other variable blocks:
6) If the stuttering disappeared, DO NOT save your game and DO NOT continue playing it. Re-load the same savegame again instead. If all goes right, after reloading the game there should be no stuttering.
7) If you close the game and launch it again, you'll have to do steps 1-6.
Why this works
The scrips I have attached adds a bunch of dummy variables to the map, forcing to overflow and get rehashed. Each block adds 100 new variables. Script looks like this:
Once the map is enlarged, it will remain with the same size if you load your game again and it will not stutter. However, if you close BGMain, upon launching it again the map will be "stuffed" again and you'll have to repeat this procedure.
Why you should reload your game after executing the script
If you save, all those dummy variables will be added to your savegame globals section. There's nothing lethal in it, but you probably don't want it.
Note, however, that this workaround and solution work only with a particular kind of stuttering, it is not a panacea, and if you have bugged scripts, they will continue being bugged.
As a workaround you can try this:
If you encounter such kind of stuttering, you can do following steps:
1) Download archive "Stutter Remover.7z" from the first post of this thread
2) Unzip a text file in it anywhere you want. Alternatively you can just copy-paste it from here:
Spoiler
3) Open your baldur.bcs or any other script that is executed with Near Infinity and add contents of "Stutter Remover.txt"(or copy-paste that spoiler) at the end of it. Compile and save your script.
4) Load your stuttering savegame, press ctrl+space and type:
If the game is paused, unpause it. Then check if the script worked:CLUAConsole:SetGlobal("AddVariableBlock0", "GLOBAL", 1)
It should return 0. Otherwise you have done wrong some of the steps above.CLUAConsole:GetGlobal("AddVariableBlock0", "GLOBAL")
5) If the stuttering disappeared, proceed to step 6). Otherwise try setting other variable blocks:
In my tests one block was always enough, but who knows.CLUAConsole:SetGlobal("AddVariableBlock1", "GLOBAL", 1)
CLUAConsole:SetGlobal("AddVariableBlock2", "GLOBAL", 1)
...
6) If the stuttering disappeared, DO NOT save your game and DO NOT continue playing it. Re-load the same savegame again instead. If all goes right, after reloading the game there should be no stuttering.
7) If you close the game and launch it again, you'll have to do steps 1-6.
Why this works
The scrips I have attached adds a bunch of dummy variables to the map, forcing to overflow and get rehashed. Each block adds 100 new variables. Script looks like this:
Spoiler
Once the map is enlarged, it will remain with the same size if you load your game again and it will not stutter. However, if you close BGMain, upon launching it again the map will be "stuffed" again and you'll have to repeat this procedure.
Why you should reload your game after executing the script
If you save, all those dummy variables will be added to your savegame globals section. There's nothing lethal in it, but you probably don't want it.
Note, however, that this workaround and solution work only with a particular kind of stuttering, it is not a panacea, and if you have bugged scripts, they will continue being bugged.
Edited by Suslik, 19 April 2012 - 10:11 AM.
Howto: mage in Baldur's Gate. My lame guide.
List of issues I have encountered in BWP 10.3 tactical
Stutter-fixer for Infinity Engine
While you were reading this signature, I have probably edited the post above. Twice.
#110
Ascension64
-
- Modder
-
- 5983 posts
Posted 20 April 2012 - 04:58 AM
Ignore my previous post. I was being tired and confused.
Anyway, why does Find() still search the entire table?
Anyway, why does Find() still search the entire table?
--------------
Retired Modder
Note: I do not respond to profile comments/personal messages in regards to troubleshooting my modifications. Please post on the public forums instead.
Baldur's Gate Trilogy-WeiDU and Mods
Throne of Bhaal Extender (TobEx)
Contributions: (NWN2) A Deathstalker (voice acting) - (IWD2) IWD2 NPC Project (soundset editing) - (Misc) SHS PC Soundsets (voice acting)
Legacy: (BG/Tutu/BGT) Beregost Crash Fixer 1.9 (18 Jul 10) - (BG2) Enable conversations with charmed/dominated creatures (18 Jul 10) - (BG2) Experience Corrections (18 Jul 10) - (Misc) Platform Conversion Utility RC2 (13 Feb 10)
#111
Suslik
-
- Member
- 500 posts
Investigator
Posted 20 April 2012 - 05:04 AM
Well, it actually terminates after the search bumps into an empty bucket OR when a variable with a matching key is found. Provided Add() tries to avoid linear chunks longer than nArraySize / 10, Find() will usually(almost often, actually) bump into an empty bucket before searching any longer.Anyway, why does Find() still search the entire table?
Edited by Suslik, 20 April 2012 - 05:08 AM.
Howto: mage in Baldur's Gate. My lame guide.
List of issues I have encountered in BWP 10.3 tactical
Stutter-fixer for Infinity Engine
While you were reading this signature, I have probably edited the post above. Twice.
#112
Anduin Shadow Mantle
-
- Member
- 102 posts
Posted 21 April 2012 - 03:08 AM
Wow!, I'll be honest followed this thread with interest but actually did not have a clue most of the time what you were talking about.
If this works (and so far its yes all round by the looks of it) this will be an ultimate fix for BG. Stuttering always marrs my games.
I think if added to ToBex (Ascension64 willing
) It will become THE mod to have, mega-mod or otherwise.
Well done Suslik and Ascension64 for your efforts!
Hope players will be able to reap the efforts of your success!
If this works (and so far its yes all round by the looks of it) this will be an ultimate fix for BG. Stuttering always marrs my games.
I think if added to ToBex (Ascension64 willing
) It will become THE mod to have, mega-mod or otherwise.Well done Suslik and Ascension64 for your efforts!
Hope players will be able to reap the efforts of your success!
#113
Ascension64
-
- Modder
-
- 5983 posts
Posted 21 April 2012 - 03:45 AM
Already added to the current test version of TobEx. Has a somewhat different implementation, but similar principle.
--------------
Retired Modder
Note: I do not respond to profile comments/personal messages in regards to troubleshooting my modifications. Please post on the public forums instead.
Baldur's Gate Trilogy-WeiDU and Mods
Throne of Bhaal Extender (TobEx)
Contributions: (NWN2) A Deathstalker (voice acting) - (IWD2) IWD2 NPC Project (soundset editing) - (Misc) SHS PC Soundsets (voice acting)
Legacy: (BG/Tutu/BGT) Beregost Crash Fixer 1.9 (18 Jul 10) - (BG2) Enable conversations with charmed/dominated creatures (18 Jul 10) - (BG2) Experience Corrections (18 Jul 10) - (Misc) Platform Conversion Utility RC2 (13 Feb 10)
#114
Suslik
-
- Member
- 500 posts
Investigator
Posted 21 April 2012 - 05:09 AM
Already added to the current test version of TobEx. Has a somewhat different implementation, but similar principle.
That's great news indeed! I'm somewhat disappointed, that you used only my hashfunction and principle, but that's not a problem lol.
I have a question regarding current implementation:
int nHash = GetHash(sName); int nHashEnd = (nHash + nArraySize / 10 - 1) % nArraySize;Are you sure it works? During my tests if int overflows, hash % nArraySize returned negative number and I got access violation. % actually has compiler-dependent behavior when working with negative numbers. I suggest you change it to unsigned int instead.
Fxd:
Oh, I see what you did there. You compute % nArraySize inside the hashfunction. As for me, it's not philosophically correct, but it should work : )
Edited by Suslik, 21 April 2012 - 08:31 AM.
Howto: mage in Baldur's Gate. My lame guide.
List of issues I have encountered in BWP 10.3 tactical
Stutter-fixer for Infinity Engine
While you were reading this signature, I have probably edited the post above. Twice.
#115
Suslik
-
- Member
- 500 posts
Investigator
Posted 21 April 2012 - 09:14 AM
Damn I should work on my English.Wow!, I'll be honest followed this thread with interest but actually did not have a clue most of the time what you were talking about.
I dunno about other players but I've been playing completely stutter-free fully-modded BWP tactics for a few day already : DHope players will be able to reap the efforts of your success!
Howto: mage in Baldur's Gate. My lame guide.
List of issues I have encountered in BWP 10.3 tactical
Stutter-fixer for Infinity Engine
While you were reading this signature, I have probably edited the post above. Twice.
#116
Daulmakan
-
- Member
- 1065 posts
Comfortably numb
Posted 21 April 2012 - 06:05 PM
Haven't tried it yet, but you deserve a whole box of e-cookies if this works.
#117
Anduin Shadow Mantle
-
- Member
- 102 posts
Posted 22 April 2012 - 04:45 AM
Wow!, I'll be honest followed this thread with interest but actually did not have a clue most of the time what you were talking about.
Damn I should work on my English.
Suslik, your English is fine. (I'm an english teacher... I should know!)
I was referring to your computer knowledge and my lack of. I bow down to your advanced computer skills
Edited by Anduin Shadow Mantle, 22 April 2012 - 04:46 AM.
#118
Ascension64
-
- Modder
-
- 5983 posts
Posted 23 April 2012 - 12:03 AM
I implement usually by completely reversing the existing BioWare-implemented function and then making changes on it. That way, I reduce the possibility of mashing up some code at some far away place without knowing.That's great news indeed! I'm somewhat disappointed, that you used only my hashfunction and principle, but that's not a problem lol.
You are right with the poor type specifications there. I'll change the declarations to be of unsigned int type.Oh, I see what you did there. You compute % nArraySize inside the hashfunction. As for me, it's not philosophically correct, but it should work : )
--------------
Retired Modder
Note: I do not respond to profile comments/personal messages in regards to troubleshooting my modifications. Please post on the public forums instead.
Baldur's Gate Trilogy-WeiDU and Mods
Throne of Bhaal Extender (TobEx)
Contributions: (NWN2) A Deathstalker (voice acting) - (IWD2) IWD2 NPC Project (soundset editing) - (Misc) SHS PC Soundsets (voice acting)
Legacy: (BG/Tutu/BGT) Beregost Crash Fixer 1.9 (18 Jul 10) - (BG2) Enable conversations with charmed/dominated creatures (18 Jul 10) - (BG2) Experience Corrections (18 Jul 10) - (Misc) Platform Conversion Utility RC2 (13 Feb 10)
#119
Suslik
-
- Member
- 500 posts
Investigator
Posted 23 April 2012 - 05:23 AM
I see I see. Reasonable approach, but takes time, yeah? The rest of the implementation seems good to me.I implement usually by completely reversing the existing BioWare-implemented function and then making changes on it. That way, I reduce the possibility of mashing up some code at some far away place without knowing.
Howto: mage in Baldur's Gate. My lame guide.
List of issues I have encountered in BWP 10.3 tactical
Stutter-fixer for Infinity Engine
While you were reading this signature, I have probably edited the post above. Twice.
#120
Suslik
-
- Member
- 500 posts
Investigator
Posted 23 April 2012 - 05:11 PM
I have found a strange bug in my implementation: OpenContainer("ContainerName") does not seem to work with it. Well, luckily Asc64's version works fine with this. That's pretty weird, because algorithms are almost identical. Anyway, I advice for everyone who was sticking to my temporary TobEx, switch to TobEx 0.24.
Howto: mage in Baldur's Gate. My lame guide.
List of issues I have encountered in BWP 10.3 tactical
Stutter-fixer for Infinity Engine
While you were reading this signature, I have probably edited the post above. Twice.
