12 replies to this topic

[The Imp]

... or the moving away from The Bigg's mod to a custom one.

Well, I meant to make a PM topic about this, and post it to a few people, but then had the notion that it might as well be good to take imput from everyone, here.
This will be a long ramble, aren't all my topics , but there aren't that many of them, so read it at your own leisure la... 'K. .. didn't write this post in one go. So it's a proper ramble.

A bit of history. Yes, the ramble starts from here... I try to keep this straight, but if you don't remember it all, I don't, you are likely to unintentionally glance stuff and miss important notions. So try keep up.
In the past, the game and mods have utilized the biffing to make the game run faster, and so it's internal fact that the Mega Moding needs to take it into account, like for example all the big mods use biffing today, list that I have is: BGT-weidu, TDD, SoS, NEJ, CtB, Dark Horizon, Bonehill, NTotSC and so forth. There's a few more.
Now, look at the list and notice that the current BWS removes most of the Biffing from those mods if you install the Lollorian's trimmed Fixpack, aka the BiG World Fixpack upwards from v15. The reasoning is good, as it makes the mod installs generally better, by allowing them to be reinstallable, it saves hard drive space, it removes time expenditure that would go into frivolous biffing and it delivers cookies. Well, ok, it doesn't do the last thing. The bad side of that is that the game then really rreeaallllyy needs to be biffed at the end of the install to speed up the games starting process and running speed.
Now, in the past, this has been resolved by 'forcing' the installation of the tool that does this. In the prime of BWP which was years ago, when the Lollorians Trimpack was still in it's none existing stage, that then got consumed into the Fixpack too, the first mod was the END_BIFF.bat, but that had the humongous problem that it couldn't even be uninstalled in a strait chain weidu uninstall command, as it utilized the old .bat command, which can't restore the removed content.

That was partially solved when The Bigg then added a weidu command that can handle uninstall without .bat's etc and came up with the Generalized Biffing mod. You can actually see in that that his opinion was split there too, as he programmed two distinct forms to the component. And the .tra file reflects this:

~Would you like to biff only wav&tis files, or all files found?
0] only wav&tis&bam (recommended by The Bigg and other Small World Dudes)
1] all files (recommended by the Big World Dudes)~

The zero biffs just the media files, and the other is the only one that the BWS today uses, biffing everything.
It has a small "flaw", at least in the eyes of the BWS install, that doesn't allow the use of both of these modes, as they are not strictly component based, but inside the same component.
Added to that, the BWS today kinda needs to force biffing. And the only other possibility is still the END_BIFF.bat
I propose a small change that will remove the worst parts of that. This is still tested only a limited amount of times, if you can call one successful run a limited amount.

Spoiler

//In all honesty, this is just recycled The Bigg's Generalized Biffing mod v2.2, with the Argent77's EE name addition, and the edit is to allow to utilize both of the two distinct components,
//I changed the author because I take the responsibility of all and any wrong coding, still all credit and praise are due to The Bigg, Valerio Bigiani, but all bug reports and feature requests should go to me, The Imp, Jarno Mikkola.
//tp2 code:

BACKUP ~Improved_generalized_biffing/backup~
AUTHOR ~The Imp~

VERSION ~v1.0~

LANGUAGE ~English~
         ~English~
         ~Improved_generalized_biffing/tra/english/setup.tra~

/*
LANGUAGE ~French (By Isaya)~
         ~french~
         ~Improved_generalized_biffing/tra/french/setup.tra~

LANGUAGE ~Italiano~
         ~italian~
         ~Improved_generalized_biffing/tra/italian/setup.tra~
*/

BEGIN @1 //~Bif all media files~

ACTION_IF ~%WEIDU_OS%~ STRING_COMPARE_CASE ~WIN32~ THEN BEGIN
    AT_UNINSTALL ~rm -rf Improved_generalized_biffing/prod/0~
END ELSE BEGIN
    AT_UNINSTALL ~del /q /f Improved_generalized_biffing\prod\0~ EXACT
END

<<<<<<<< .../list-of-files
>>>>>>>>

// this defines a one to eight character regular expression for matching resrefs
OUTER_TEXT_SPRINT ~char~ ~[^ %TAB%%WNL%]~
OUTER_TEXT_SPRINT ~8char~ ~%char%%char%?%char%?%char%?%char%?%char%?%char%?%char%?~
OUTER_TEXT_SPRINT ~9char~ ~%char%%char%%char%%char%%char%%char%%char%%char%%char%%char%*~
OUTER_TEXT_SPRINT ~ext~ ~\(mve\|wav\|bam\|tis\|wbm\|pvrz\)~
OUTER_TEXT_SPRINT ~myRegExp~ ~^har%\.%ext%$~

MKDIR ~Improved_generalized_biffing/prod/0/biffs/0~
MKDIR ~Improved_generalized_biffing/prod/0/rej~

OUTER_SET currentTotal = 0
OUTER_SET currentFile = 0
ACTION_BASH_FOR ~override~ ~%myRegExp%~ BEGIN
  ACTION_IF ºSH_FOR_SIZE% + currentTotal > 30000000 /* 30M */ && currentTotal > 0 BEGIN
    MAKE_BIFF ~xo#ge%currentFile%~ BEGIN ~Improved_generalized_biffing/prod/0/biffs/%currentFile%~ ~^.*$~ END
    OUTER_SET currentFile = currentFile + 1
    OUTER_SET currentTotal = 0
    MKDIR ~Improved_generalized_biffing/prod/0/biffs/%currentFile%~
  END
  MOVE ~override/ºSH_FOR_FILE%~ ~Improved_generalized_biffing/prod/0/biffs/%currentFile%/ºSH_FOR_FILE%~
  OUTER_SET currentTotal += BASH_FOR_SIZE
END
    
ACTION_IF currentTotal > 0 BEGIN
    MAKE_BIFF ~xo#ge%currentFile%~ BEGIN ~Improved_generalized_biffing/prod/0/biffs/%currentFile%~ ~^.*$~ END
END

OUTER_TEXT_SPRINT ~myRegExp~ ~^har%.%ext%$~
ACTION_BASH_FOR ~override~ ~%myRegExp%~ BEGIN
  OUTER_PATCH_SAVE ~8name~ ~ºSH_FOR_FILE%~ BEGIN
    DELETE_BYTES 8 (INDEX_BUFFER (EXACT_MATCH ~.~) - 8 )
  END
  ACTION_IF FILE_EXISTS_IN_GAME ~%8name%~ THEN BEGIN
    MOVE ~override/ºSH_FOR_FILE%~ ~Improved_generalized_biffing/prod/0/rej/ºSH_FOR_FILE%~
  END
END


BEGIN @2 //~Bif all files~

ACTION_IF ~%WEIDU_OS%~ STRING_COMPARE_CASE ~WIN32~ THEN BEGIN
    AT_UNINSTALL ~rm -rf Improved_generalized_biffing/prod/1~
END ELSE BEGIN
    AT_UNINSTALL ~del /q /f Improved_generalized_biffing\prod\1~ EXACT
END

<<<<<<<< .../list-of-files
>>>>>>>>

// this defines a one to eight character regular expression for matching resrefs
OUTER_TEXT_SPRINT ~char~ ~[^ %TAB%%WNL%]~
OUTER_TEXT_SPRINT ~8char~ ~%char%%char%?%char%?%char%?%char%?%char%?%char%?%char%?~
OUTER_TEXT_SPRINT ~9char~ ~%char%%char%%char%%char%%char%%char%%char%%char%%char%%char%*~
OUTER_TEXT_SPRINT ext ~\(bmp\|mve\|wav\|wfx\|plt\|bam\|wed\|chu\|tis\|mos\|itm\|spl\|bcs\|ids\|cre\|are\|dlg\|2da\|gam\|sto\|wmp\|eff\|chr\|vvc\|vef\|pro\|wbm\|fnt\|gui\|sql\|pvrz\|glsl\|ini\)~
OUTER_TEXT_SPRINT ~myRegExp~ ~^har%\.%ext%$~

MKDIR ~Improved_generalized_biffing/prod/1/biffs/0~
MKDIR ~Improved_generalized_biffing/prod/1/rej~

OUTER_SET currentTotal = 0
OUTER_SET currentFile = 0
ACTION_BASH_FOR ~override~ ~%myRegExp%~ BEGIN
  ACTION_IF ºSH_FOR_SIZE% + currentTotal > 30000000 /* 30M */ && currentTotal > 0 BEGIN
    MAKE_BIFF ~IJ#ge%currentFile%~ BEGIN ~Improved_generalized_biffing/prod/1/biffs/%currentFile%~ ~^.*$~ END
    OUTER_SET currentFile = currentFile + 1
    OUTER_SET currentTotal = 0
    MKDIR ~Improved_generalized_biffing/prod/1/biffs/%currentFile%~
  END
  MOVE ~override/ºSH_FOR_FILE%~ ~Improved_generalized_biffing/prod/1/biffs/%currentFile%/ºSH_FOR_FILE%~
  OUTER_SET currentTotal += BASH_FOR_SIZE
END
    
ACTION_IF currentTotal > 0 BEGIN
    MAKE_BIFF ~IJ#ge%currentFile%~ BEGIN ~Improved_generalized_biffing/prod/1/biffs/%currentFile%~ ~^.*$~ END
END

OUTER_TEXT_SPRINT ~myRegExp~ ~^har%.%ext%$~
ACTION_BASH_FOR ~override~ ~%myRegExp%~ BEGIN
  OUTER_PATCH_SAVE ~8name~ ~ºSH_FOR_FILE%~ BEGIN
    DELETE_BYTES 8 (INDEX_BUFFER (EXACT_MATCH ~.~) - 8 )
  END
  ACTION_IF FILE_EXISTS_IN_GAME ~%8name%~ THEN BEGIN
    MOVE ~override/ºSH_FOR_FILE%~ ~Improved_generalized_biffing/prod/1/rej/ºSH_FOR_FILE%~
  END
END

We should make the component 0 required in a BWS/BWP installs, and component 1 to be pre-selected, but op-out able. The reasoning behind this is the same as the one I said for the current BWS.
Yes, it needs a retweak of the translations as shown in the code, as currently the second component name is rather long... ~IMPROVED_GENERALIZED_BIFFING/IMPROVED_GENERALIZED_BIFFING.TP2~ #0 #1 // Would you like to biff only wav&tis files, or all files found? 0] only wav&tis&bam (recommended by The Bigg and other Small World Dudes) 1] all files (recommended by the Big World Dudes): v1
And currently you of course also need the rest of the the_bigg_biff.rar archives content(.tra files) extracted to the new folder and a renamed weidu.exe to run this .tp2 file.
 
We might as well add a requirement for the later component to first have the the component 0 installed, but that'll wait, for opposite opinion to be raised. The whole reason why to do this is that it could cut the install time of the mods in BWS when it's installed in the correct position, which is after all the sound, animation and area adding mods have been installed, but before the player customization. So it can go before the Widescreen mod and several other tweak mods that the player might wish to customize the install of, yes, I am doing this for my own good.
Unfortunately, I noticed that the benefit of this is somewhat restricted till, but I can't say for sure that we shouldn't just biff the majority of the overrides content.
Opinions ? This kinda depends on where exactly we want to place this in the BWS install order.
 
And yes, the second component(#1) also has the same file types as the first one, in case some one wants to edits or add a mod that adds one of those files, we still want to cover all of them, just in case. In case it's not clear, the component #0 will allow the media files(normal and EE movies: .mve & wbm, normal sound files: .wav, animations: .bam and normal and EE tilesets: .tis & .pvrz) to be biffed. If the player then wants to edit his game by taking the opportunity to remove the component #1 from the BWS install, they don't need to uninstall or op out the bigger files from their game, and those are the files that cause the severe start delays and some lag if not biffed. Yes, the script files(.bcs) can also cause lag, but that's more symptomatic of a bad script or just a very heavy one, which the player should then just install the component #1 to try to get rid of it after they have customized their install.
Comments ?

Edited by The Imp, 20 December 2015 - 01:49 AM.

[The Imp]

I'll place a tactical second post here, so I can edit the first without loosing to unimportant stuff, by placing it here.

And in case it's not obvious, the topic title and the mods name starting with the IMP is a joke. As if  I would approve anything these evil talons type.

Edited by The Imp, 18 December 2015 - 03:23 AM.

[Almateria]

lol if you dont put your name in everything you create

way to let sid meier down

[Roxanne]

Some feedback from a player, tester and modder (trying to keep those aspects apart)

 

1. We do not need to discuss that biffing (#1 generalized biffing) is absolute MUST for any modded game of some significant size (loading times, in-game delay, size on your computer etc....)

2. Biffing was introduced already when most users still did their installations mostly by hand (before BWP/BWS) and that may explain the two existing options.

3. In the very beginning (almost a decade ago) I did option 0 biffing, mostly out of fear for having to reinstall the complete game, I think the option is more or less historic - with all the experience we have today I would advocate to forget about it completely. Do option 1 if you do any biffing (if not just blame yourself for lame gameplay).

4. The question of why and when you should biff the installation is more related to the question of which/why/when do you anticipate to UNINSTALL MODS.

5. From the point of just a player using BWP/BWS to do my install, I see no real reason to ever need to uninstall something. Regardless of having a biffed or non-biffed game, you spend a lot of time and risk a messed installatiion if you try to repair any installation this way. Especially if you have some large game tweak mods near the end or your installation. If you really encounter problems with a file or mod, always try to find the older version of the file(s) in the respective backup folder and restore it to override.

6. With BWP/BWS it is much easier today to just delete a messed installation and let the tools do a new one (all you need is the clean backup you created when first running one of the tools and the tool itself). Rather spend your time in the initial setup and selection of your install, rather than try to repair in reverse.

7. The only real reason to have to constantly install/re-install stuff on a running installation is if you develop mods yourself or you are beta-testing one or more. In this case, the best way for me is to install those unstable mods at the very end (even after biffing) and modify/update/uninstall them easily when needed.

[The Imp]

Roxanne, have you ever installed the NPC component of the Level 1 NPCs mod ? After installing the first 2 out of the 300 other mods with BWS, and went to bed thinking that perhaps you could install the custom mods by hand and then use IGB. Or never having used the BWS and then having to choose the widescreen mods resolution option when you don't know what it looks like in your monitor when you are 70+ years ? It's a havoc to think about all the stumbling things you can fall into .. if you just, ahh cheese, there's always the reinstall option.

 

@Almateria: Well, I can't change my compilation mods name anymore, as the "Impilation" title is already implemented to the folder names structure in the .tp2 file.

[Roxanne]

Roxanne, have you ever installed the NPC component of the Level 1 NPCs mod ? After installing the first 2 out of the 300 other mods with BWS, and went to bed thinking that perhaps you could install the custom mods by hand and then use IGB. Or never having used the BWS and then having to choose the widescreen mods resolution option when you don't know what it looks like in your monitor when you are 70+ years ? It's a havoc to think about all the stumbling things you can fall into .. if you just, ahh cheese, there's always the reinstall option.

Caught me - I NEVER EVER install Level 1 NPCs and I NEVER use widescreen mod because i know how it looks for my 70+ years eyes. And since I NEVER EVER WILL, I cannot comment any further.

 

@Almateria: Well, I can't change my compilation mods name anymore, as the "Impilation" title is already implemented to the folder names structure in the .tp2 file.

Edited by Roxanne, 18 December 2015 - 04:48 AM.

[agb1]

By the by, BWS now installs both Level1NPCs (just the kit change components, not the core tweaks) and NPC_EE after generalized biffing to allow for "oops, didn't want that" partial reinstalls.

I think that Imp is on the right track in questioning which smaller tweaks a user might change his or her mind about, and if we can make small install changes easier without a full reinstall (even if fully automated, it is still time consuming).

Another way to look at this: could WeiDU be improved such that it can selectively extract what it needs from the biffs instead of undoing the entire generalized biffing? Another way of saying it: could we make it seem to WeiDU that generalized biffing actually installed a different component for each file that is generally-biffed, instead of a single gen-biff component covering all files? This would allow WeiDU to unwind biffing on a per file basis when unwinding changes.

Even better would be if biffs could be made transparent to WeiDU so it just modifies the files in the biffs directly without having to unwind.

Of course this wouldn't fully solve the problem because some tweak mods (SCS for example) also change many files in a single component, so the unwinding time is still long even if you want to unwind far enough to change something before SCS. To help with this, we might be able to move more of the small (fewer files modified) tweaks after the big tweaks.

Maybe we could split gen biff component 1 into several components for different types of resource files. Items and creatures and spells and scripts each in separate biffs.

Edited by agb1, 18 December 2015 - 06:18 AM.

[Roxanne]

By the by, BWS now installs both Level1NPCs (just the kit change components, not the core tweaks) and NPC_EE after generalized biffing to allow for "oops, didn't want that" partial reinstalls.

I think that Imp is on the right track in questioning which smaller tweaks a user might change his or her mind about, and if we can make small install changes easier without a full reinstall (even if fully automated, it is still time consuming).

Another way to look at this: could WeiDU be improved such that it can selectively extract what it needs from the biffs instead of undoing the entire generalized biffing? Another way of saying it: could we make it seem to WeiDU that generalized biffing actually installed a different component for each file that is generally-biffed, instead of a single gen-biff component covering all files? This would allow WeiDU to unwind biffing on a per file basis when unwinding changes.

Even better would be if biffs could be made transparent to WeiDU so it just modifies the files in the biffs directly without having to unwind.

Of course this wouldn't fully solve the problem because some tweak mods (SCS for example) also change many files in a single component, so the unwinding time is still long even if you want to unwind far enough to change something before SCS. To help with this, we might be able to move more of the small (fewer files modified) tweaks after the big tweaks.

Maybe we could split gen biff component 1 into several components for different types of resource files. Items and creatures and spells and scripts tend to be targeted most by tweak mods.

I agree to your level1 npc reasoning - in some way this is not really a mod (I always only refer to the second part - the individual NPC tweak) because you may as well use shadow/ee keeper or NI to change your NPCs to whatever (Ajantis the totemic druid meets Minsc the gnomish bard). But what other mod may you just change your mind about halfways through the game? And would it not be faster and safer to just do a new install (we now have efficient tools to do that, it is not like in the -bad- old days anymore)?

Edited by Roxanne, 18 December 2015 - 06:24 AM.

[The Imp]

because you may as well use shadow/ee keeper or NI to change your NPCs to whatever

Yeah, let's see, let's take Minsc(an average NPC) as the example, 5 files in SoA alone, we have 50 BG1 NPCs, and 150 BG2 NPCs. That could be as many as 1000 files you need to change, if you want the change be permanent. I would love to see you do it by hand. Well, no.

Another way to look at this: could WeiDU be improved such that it can selectively extract what it needs from the biffs instead of undoing the entire generalized biffing?

Well no, not really. We can extract files from a .bif easily, but doing that won't remove the need to keep it's presence in the game directory without uninstalling the mod that made it. So you might as well just normally install the mod as that will do the exact same thing, if it's programmed correctly. So no IF FILE_EXISTS, but instead IF FILE_EXISTS_IN_GAME or what ever it was... and then rebif the files.
Yes, the game can have the same file referred in 25 different .bif files and it will know which is the latest. Of course those 24 other files will make the reading some of the 20000 other files relatively a bit slower as it's in 24 other spaces. And it will take 25 times the files space on the hard drive. Don't mistake that to be little stuff, the original ToB game took 2.6 GB's when extracted from 5 CDs, that actually contained just 2.1 GBs actual data. Now, consider the same rate to a game that takes about 30GB's.
Lollorian could probably tell you the amount of space he saved for himself when he made the mods not biff. It was a lots.

And would it not be faster and safer to just do a new install (we now have efficient tools to do that, it is not like in the -bad- old days anymore)?

You somehow lack the basic understanding that if you can't actually install the component to the game in the first place, a reinstall will not fix the problem.

Edited by The Imp, 18 December 2015 - 07:50 AM.

[Lollorian]

Relevant thread for nostalgia and teh feels

[agb1]

Should BWS be using the Borkpack from that thread instead of (or in addition to) the Trimpack?

[Lollorian]

Absolutely not

[The Imp]

The origin of the Borkpack was the trimpack, the yellow smiley monster renamed it as borkpack as it might have borked hard, but it got sorted and then went back to trimpack and then it got trimmed all on it's own into the Fixpack. All hail the yellow smiley monster !