216 replies to this topic

[Lollorian]

Holy what?!? I'm looking at my data folder and the last one is tb#gen180.bif Biffs work on the 8-char limit too?? Well, it's official then, game f00bared

I'm an idiot - biffs are stored as arbitrary-length strings rather than as statically allocated strings, so even long names are OK.

Good to know (but I deleted the 22 GB install after that ) Ah well, might as well try out the new Cleanup.bat

So, there's no way those files can be patched by something later, right?? (doesn't something try to fix bams??)

Well, some mods alter bams, but IMHO only a small portion of them are overwritten, so biffing some of them twice is not that bad (you might lose some ~50 MB by biffing twice, but remove half a gigabyte or so from gen_biff).

Oh, and one more thing, is there a size limit to .bif's?? Can we stuff everything that isn't patched into the .bif's that are required by the megamods for their checks??

The largest bif in my install is 200 MB (BG1MOVIE from BGT) - use that as a thresold. Add MVE to the list of formats to biff per-mod, BTW - the game flat-out crashes if a MVE is in the override.

Great!!! So, the filetypes that can safely be biffed - tis/wed/bmp/mos/bam/wav/mve which we will put into the modcheck.bif's (or into 200 MB chunks ... basically, we leave anything that gets its own .bif (like the TDD-TIS, NEJ-WAV etc) but put in all the stuff that goes with the itm/sto/are/spl/cre/bcs types into the modcheck.bif's.

How does that sound??

Btw, thanks for all the answers, theBigg. You da man!!!

Cheers,
Lol

EDIT: mve's too ...

Edited by Lollorian, 29 December 2009 - 06:19 AM.

[the bigg]

tis/wed/bmp/mos/bam/wav/mve - the latter are especially needed but I forgot them earlier.

[Lollorian]

And that too

Ok, one last teensy-weensy question, why does NeJ have its own separate biffer?? Is there a story behind it?? A conspiracy?? A true tale of love and hate?? Or did somebody just get bored and cook it up??

Cheers,
Lol

Edited by Lollorian, 29 December 2009 - 10:55 AM.

[the bigg]

A true tale of love and hate??

That'd be the shortest explanation possible. If you really want to know all the details, start a new topic and hope you get the answers you want before the lock+ban

Edited by the bigg, 30 December 2009 - 05:52 AM.

[Lollorian]

Err .. umm Hooookkkaaayyy??

So, we'll not go into the details then Just the technicals, is there a special reason NeJ needs its own biffer (aside from the love-hate thing ) Should it be modded to the new standards too??

And most importantly, will I be playing with fire if I mess around with it??

Cheers,
Lol

[the bigg]

NeJ uses a separate biffer because its author doesn't like biffing, not for technical reasons. I'd standardize that too.

And no, you aren't playing with fire - the version of NeJ used with megamods is basically abandonware and not even publicly available (there is a newer version, but it's not for megamods).

[Lollorian]

Ah cool

I looked over the threads and it doesn't seem like Vlad hates the biffer, it does seem to do strange stuff when used without BP, so that's understandable when the new NeJ's supposed to be standalone

(w00t, first post that didn't end in a question )

Cheers,
Lol

[Lollorian]

And finally, I got a much smaller and specific list of exactly which .bif's WON'T be created (so, we delete the lines that create these biffs )
Also, some bifs (like the RULE.bifs) contain patchable stuff along with fixed stuff, so we remove the patchy stuff while keeping the fixed stuff for the bif to be created

• NeJ (BP-NeJAreasSounds.bat from NeJ2Biffer)
  

  NEJ-ARE (remove ARE)NEJ-BCS	NEJ-CRE	NEJ-DLG	NEJ-ITMNEJ-STO	NEJ-RULE (remove 2DA/IDS)NEJ-SFX	(remove EFF)NEJ-SPL

• BGT (EndSetup.bat)
  

  BG1ARE	BG1BCS	BGTCRE	BG1DLG	BG1ITM	BG1MISC	(remove 2DA)BG1SPL	BG1STO

• DH (install.bat in BWFixpack folder)
  

  dhare (remove ARE)

• NTotSC (NTAreas.bat)
  

  NTotSC1 (remove ARE/STO/ITM/CRE/EFF/SPL/2DA/DLG)

• SoBH (BHAreas.bat)
  

  BH-REST (remove DLG/BCS/ARE/CRE/2DA)

• CtB (CtbFinishInstall.bat, ChoresInstall.bat)
  

  CTB-ARECTB-BCS	CTB-CRE	CTB-ITMCTB-RULE (remove 2DA/SPL/STO)CTB-CHRS (remove BCS/DLG)

• DSotSC (DSAreas.bat)
  

  DSOTSC1 (remove ARE/BCS/ITM/STO/SPL/CRE/DLG/2DA/IDS)

• RoT (RoT-Install.bat)
  

  ROT-AREROT-BCS	ROT-CRE	ROT-DLG	ROT-ITMROT-RULE (remove 2DA/EFF/STO)ROT-SPL

• SoS (SOS-TISOGGBIFF.bat)
  

  SOS-ARESOS-BCS	SOS-CRE	SOS-DLG	SOS-ITMSOS-RULE (remove 2DA/STO/EFF)SOS-SPL

• TDD (TDD-TISBIFF.bat)
  

  TDD-ARETDD-BCS	TDD-CRE	TDD-DLG	TDD-ITMTDD-RULE (remove 2DA/EFF/STO)TDD-SPL

• ToA (end.bat)
  

  ToA1 (remove ARE/BCS/CRE/DLG/EFF/ITM/STO)

• TS-BP (TSAreasSounds.bat)
  

  TS-ARETS-BCS	TS-CRE	TS-DLG	TS-ITMTS-RULE (remove 2DA/EFF/SPL/STO)

Well, isn't that just nifty? Now, to actually learn more about the MAKE_BIFF stuff

Amirite in thinking this only involves converting (for instance)

weidu --make-biff NEJ-TIS2

to

MAKE_BIFF ~NEJ-TIS2~ BEGIN ~NEJ-TIS2~ ~^.*$~ END

??
once everything's moved to that directory ofc

Oh, oh, and does biffing actually compress files??

_________________________________________________________________

And I was thinking,
ya know what'd make this stuff a whole lot easier// A simple mod that:

• Unpacks all the files referenced in the chitin.key from the .bif's into the override (while leaving files that are already present in the override)
  
• deletes the contents of the data\ folder (except maybe the vanilla BGII-ToB stuff)
  
• packs all the files into .bif's just like gen_biff does

I know it'll take something like ZOMGWTFPWNZOR hours to finish, and leaves absolutely no chance of uninstalling the mods, but meh ... just throwing out ideas

Cheers,
Lol

Edited by Lollorian, 31 December 2009 - 11:29 AM.

[Jarno Mikkola]

Oh, oh, and does biffing actually compress files?

A bit, but possibly only in the small files significantly enough...

Am I right in thinking this only involves converting (for instance)

weidu --make-biff NEJ-TIS2

to

MAKE_BIFF ~NEJ-TIS2~ BEGIN ~NEJ-TIS2~ ~^.*$~ END

??
once everything's moved to that directory ofc

Erhm, I am pretty sure it's not cause you are thinking of .tp2 code, not cmd.exe code that the .bat's use.

And I was thinking,
ya know what'd make this stuff a whole lot easier// A simple mod that:

• Unpacks all the files referenced in the chitin.key from the .bif's into the override (while leaving files that are already present in the override)
  
• deletes the contents of the data\ folder (except maybe the vanilla BGII-ToB stuff)
  
• packs all the files into .bif's just like gen_biff does

I know it'll take something like ZOMGWTFPWNZOR hours to finish, and leaves absolutely no chance of uninstalling the mods, but meh ... just throwing out ideas

Why would you first pack everything in to .bifs, then unpack them, and then repack them into .bifs? Do you have some sort of masochism?
As you just need to make the mods not biff their files (that can/will be change/overwritten later), then use the Gen_Biff just biff everything at the end. Of course you'll need to make the, NTOTSC1.bif, CTB-RULE.bif, ROT-RULE.bif, SOS-RULE.bif, TDD-RULE.bif, TS-RULE.bif 'images' that hold the 1 file(the biffchec.itm) at most.

Edited by Jarno Mikkola, 31 December 2009 - 12:36 PM.

[Helborn]

RL has a away of interfering with best laid plans...

My apporach has been to:
1 move all processing of ogg, acm, and tiz files to the end of the install (just before biffing)
2 eliminate all biffing during a mega install until the very end
3 eliminate all backups of the override folder
4 eliminate all calls to bat files (except for BGT and I hope to )

#1 - too many mods have an older version of oggdec or tisunpack, with the result that the bat files calling them are larger than they need to be (no Wildcards) and the processing is slower than it needs to be (older version). Replacing all of those calls (and copies + deletes of oggdec/tisunpack) with a single processing call (which can actually be done in a tp2 if the megainstall ever moves from bat to tp2) saves time and space. The latest versions are significantly faster than the older ones.

#2 - space is cheapand getting cheaper. the idea behind delaying until the end is that there is no game started until the megainstall is complete, so everything will be biffed. doing small bifs for those files that will not be changed creates many miniscule bifs for no really good reason. biffing in the middle of the install only slows the process down as well as slowing (slightly) the game because the reference table (chitin.key) becomes larger. (I know, I know, it is unnoticeable - most of the time, - but when you get stuttering from some other cause, this can/will increase the effect)
#3 - this is no longer necessary or helpful
#4 - almost all bat files are used to process ogg and tiz files, create new bifs and delete the directories created for biffing, the exceptions are easy to code around. If we use generalized biffing for everything, no directories need be created only to be deleted.

btw - I have 73 tp2s that require modification

[Jarno Mikkola]

btw - I have 73 tp2s that require modification

Could you upload them in a .zip/.rar/.7z file? Packaged cause they take less space and it's quicker to upload, yes to this thread. And if the version is old/just updated, it doesn't matter.

Edited by Jarno Mikkola, 31 December 2009 - 03:45 PM.

[Lollorian]

Am I right in thinking this only involves converting (for instance)

weidu --make-biff NEJ-TIS2

to

MAKE_BIFF ~NEJ-TIS2~ BEGIN ~NEJ-TIS2~ ~^.*$~ END

??
once everything's moved to that directory ofc

Erhm, I am pretty sure it's not cause you are thinking of .tp2 code, not cmd.exe code that the .bat's use.

Well, yeah!!! (but, the only difference between --make-biff and MAKE_BIFF seems to be that MAKE_BIFF keeps the install state as it was in its backup ... which isn't really necessary, cause people aren't going to tweak their BWP installs ... and it'd also reduce space)

Or, am i gawdawfulworng again and missing something important here??

And I was thinking,
ya know what'd make this stuff a whole lot easier// A simple mod that:

• Unpacks all the files referenced in the chitin.key from the .bif's into the override (while leaving files that are already present in the override)
  
• deletes the contents of the data\ folder (except maybe the vanilla BGII-ToB stuff)
  
• packs all the files into .bif's just like gen_biff does

I know it'll take something like ZOMGWTFPWNZOR hours to finish, and leaves absolutely no chance of uninstalling the mods, but meh ... just throwing out ideas

Why would you first pack everything in to .bifs, then unpack them, and then repack them into .bifs? Do you have some sort of masochism?
As you just need to make the mods not biff their files (that can/will be change/overwritten later), then use the Gen_Biff just biff everything at the end. Of course you'll need to make the, NTOTSC1.bif, CTB-RULE.bif, ROT-RULE.bif, SOS-RULE.bif, TDD-RULE.bif, TS-RULE.bif 'images' that hold the 1 file(the biffchec.itm) at most.

No I'm not saying combine the two methods With the current BWP setup of doing stuff, the above option (while being time-consuming) would save the greatest space

And good to see ya back again, Helborn

1 move all processing of ogg, acm, and tiz files to the end of the install (just before biffing)

#1 - too many mods have an older version of oggdec or tisunpack, with the result that the bat files calling them are larger than they need to be (no Wildcards) and the processing is slower than it needs to be (older version). Replacing all of those calls (and copies + deletes of oggdec/tisunpack) with a single processing call (which can actually be done in a tp2 if the megainstall ever moves from bat to tp2) saves time and space. The latest versions are significantly faster than the older ones.

That's great!!!

2 eliminate all biffing during a mega install until the very end

#2 - space is cheapand getting cheaper. the idea behind delaying until the end is that there is no game started until the megainstall is complete, so everything will be biffed. doing small bifs for those files that will not be changed creates many miniscule bifs for no really good reason. biffing in the middle of the install only slows the process down as well as slowing (slightly) the game because the reference table (chitin.key) becomes larger. (I know, I know, it is unnoticeable - most of the time, - but when you get stuttering from some other cause, this can/will increase the effect)

Now, here,

1) There are files that are not patched by anything else once they are biffed (tis/wed/bmp/mos/bam/wav/mve) along with files that are patchable (cre/itm/sto/spl/are/2da/ids/eff/bcs/dlg) so letting the first group be biffed is not a bad idea
2) Many mods check for other mods and install content accordingly. The megamods have certain .bifs that should be present when they're installed, so removing all instances of biffing in the middle of the install is bad
3) <something I thought of when I started replying, but forgot ... it'll come back ... eventually>

3 eliminate all backups of the override folder

#3 - this is no longer necessary or helpful

YES!!!

4 eliminate all calls to bat files (except for BGT and I hope to )

#4 - almost all bat files are used to process ogg and tiz files, create new bifs and delete the directories created for biffing, the exceptions are easy to code around. If we use generalized biffing for everything, no directories need be created only to be deleted.

While I don't feel it's necessary (come on, with the kinds of processors around, program calls aren't really time-consuming ), well, we're doing everything else, so why not??

btw - I have 73 tp2s that require modification

So, that includes all the NPC/quest/mega mods that include oggdec/tisunpack right?? (I'd be damned if I missed out on 73 mods that biff )

And now, a serious question for theBigg. In the gen_biff download page, it says (quite obviously) that "PLEASE NOTE THAT THIS MOD HAS NOT BEEN EXTENSIVELY TESTED AND MIGHT BREAK YOUR GAME (AND COULD KEEP IT BROKEN EVEN AFTER UNINSTALL)."

How can it keep the game broken even after uninstalls?? It doesn't mod the registry and can't possibly break a full uninstall (unless you count in bad sectors due to the repeated writing ... but that'd be the HDD's fault, non??) Or, is this just your way of saying, "beta version"

Cheers,
Lol

Edited by Lollorian, 01 January 2010 - 12:52 AM.

[Jarno Mikkola]

And now, a serious question for theBigg.

Who is this mysterious theBigg? I know of 'the bigg', but...

In the gen_biff download page, it says (quite obviously) that "PLEASE NOTE THAT THIS MOD HAS NOT BEEN EXTENSIVELY TESTED AND MIGHT BREAK YOUR GAME (AND COULD KEEP IT BROKEN EVEN AFTER UNINSTALL)."

He meant that the mod when installed can leave the game broken even after itself was removed, not the whole game (and then reinstalled), so indeed you would have to reinstall the whole game... and that could indeed be early beta behavior, but the download page just was never updated... well, because you'll never know what happens in new updates, when it's not fully tested with every possible condition, and as there's 2^modcomponent times the possible conditions, basically.

[the bigg]

Well, I guess I could as well remove that warning. The current build uses a lot of OS-dependant code, and was never tested under Windows 9x or a Mac - and it was well possible that uninstalling gen_biff would not restore the files to their original position (leaving you with an unplayable game). The upcoming version uses different techniques (that don't suffer from that problem).

[Helborn]

1) There are files that are not patched by anything else once they are biffed (tis/wed/bmp/mos/bam/wav/mve) along with files that are patchable (cre/itm/sto/spl/are/2da/ids/eff/bcs/dlg) so letting the first group be biffed is not a bad idea
2) Many mods check for other mods and install content accordingly. The megamods have certain .bifs that should be present when they're installed, so removing all instances of biffing in the middle of the install is bad

1) to get tis files you have unpack them - time. to get most wav files you have to unpack the ogg files - also time.
some bmps and bams are modded by specific mods (1pp, etc)
the wed/mos/mve files I agree could be biffed, but why run a bif routine separate from generalized_biffing?

the checks for specific bifs are easy to find and change to specific files:
BGT = BG1ARE.bif ( the one exception so far, still thinking about how to change the BGT process - if I ever get around to it)
TS = TT0010.ARE
TDD = DD1011.ARE
NEJ = either EYEBITE.ITM or TT4000.ARE
BP = BPINVISB.EFF
CtB = CB3661.ARE
RoT = RR4001.ARE
SoS = CBSOUBAR.MVE

the checks need to be changed to "IF_FILE_EXISTS_IN_GAME" but I would suggest that this should be the call anyway for a mega install. I will have mods in my game directory that I am not installing at a particular time ( NeJ & RoT, usually)

I have not found any other installation checks than for the megamods (must admit, haven't looked very hard either, so may have missed some, but other mods don't bif, so I believe all the other checks are for specific files).

I will upload the zip/7z of the tp2s as soon as I have finished some installation tests - that's one of the things that has caused the delay (and I do believe that I need to use the latest or something will be missed).

Cheers

[Lollorian]

He meant that the mod when installed can leave the game broken even after itself was removed, not the whole game (and then reinstalled), so indeed you would have to reinstall the whole game... and that could indeed be early beta behavior, but the download page just was never updated... well, because you'll never know what happens in new updates, when it's not fully tested with every possible condition, and as there's 2^modcomponent times the possible conditions, basically

Ah, sorry bout that

The upcoming version uses different techniques (that don't suffer from that problem).

Oooooh!!! Shiny

1) to get tis files you have unpack them - time. to get most wav files you have to unpack the ogg files - also time.

The only way that time can be reduced is by giving away the unpacked tis'es and wav's, which means increased download size (and we have to remember that everyone don't have 1 MBps ) ... i still agree that updating all the tisunpack/oggdec's are cooler though

some bmps and bams are modded by specific mods (1pp, etc)

Yeah, I thought so, but the bigg guy seems to recommend leaving them be biffed by the mods themselves.

the wed/mos/mve files I agree could be biffed, but why run a bif routine separate from generalized_biffing?

Does biffing together or separately really take different times?? My focus was to reduce space consumption, which is why those patchety-patchy files were biffed at the end.

the checks for specific bifs are easy to find and change to specific files:
BGT = BG1ARE.bif ( the one exception so far, still thinking about how to change the BGT process - if I ever get around to it)
TS = TT0010.ARE
TDD = DD1011.ARE
NEJ = either EYEBITE.ITM or TT4000.ARE
BP = BPINVISB.EFF
CtB = CB3661.ARE
RoT = RR4001.ARE
SoS = CBSOUBAR.MVE

the checks need to be changed to "IF_FILE_EXISTS_IN_GAME" but I would suggest that this should be the call anyway for a mega install. I will have mods in my game directory that I am not installing at a particular time ( NeJ & RoT, usually)

Do you mean to change all the checks themselves?? Isn't that a wee unnecessary work?? The checked .bif's can be easily filled with the non-patchy stuff (didn't we discuss this in page 2 or something??)

I have not found any other installation checks than for the megamods (must admit, haven't looked very hard either, so may have missed some, but other mods don't bif, so I believe all the other checks are for specific files).

I have 2 posts in this thread that list the .bif's (this one tells you the list of ALL biffing mods and this one tells you to remove all the patchy stuff )

I will upload the zip/7z of the tp2s as soon as I have finished some installation tests - that's one of the things that has caused the delay (and I do believe that I need to use the latest or something will be missed).

If you've already done that, great job!!! (less work for me )

Cheers,
Lol

Edited by Lollorian, 01 January 2010 - 10:03 AM.

[the bigg]

Does biffing together or separately really take different times?? My focus was to reduce space consumption, which is why those patchety-patchy files were biffed at the end.

Not noticeably; however, the great advantage of gen_biff (over the earlier end_biff) is that you can tweak your installation without taking any precautionary steps (with end_biff, you had to uninstall end_biff, do your tweaks, and manually reinstall end_biff). By biffing large static resources early, the time taken for biffing mods in gen_biff is drastically reduced, making it feasible to tweak your installation. Another thing is that, by having large resources biffed early, you could install just BGT and it will Just Work (if BGT didn't biff MVE the game would crash, if it didn't biff TIS/WAV/... files it'd be horribly laggy).

PS: I suppose that some BAM and BMP files end up being overwritten; however, the mod-added ones will likely be left alone (meaning they will be biffed only once) - thus, you can have the single mods biff its own BAM/BMP files, whereas the files from vanilla will be left in the override and biffed by gen_biff at the end.

[Lollorian]

PS: I suppose that some BAM and BMP files end up being overwritten; however, the mod-added ones will likely be left alone (meaning they will be biffed only once) - thus, you can have the single mods biff its own BAM/BMP files, whereas the files from vanilla will be left in the override and biffed by gen_biff at the end.

What I'm afraid of is whether anyone knows a mod that patches .bmp and .bam's using the all-patchy regex search (which will probably patch mod files too )

Also, I'd like to point out that folks who use the BWP either tweak it before running the install.bat or don't tweak at all So the tweaking argument doesn't hold much ground atleast for the BWP.

Another thing to note is that, when people uninstall the BWP, they're just deleting their entire BGII-SoA folder and reinstalling from scratch, so the HUGE backup of the override isn't needed by the final biffer (unless it's needed for changelogs ... actually, no backups are needed unless you do a changelog )

And (sorry about these question-rounds ) what's the difference between MAKE_BIFF and --make-biff other than that the former creates a backup for uninstallation (in which case --make-biff would probably be better for disk space)

Cheers,
Lol

Edited by Lollorian, 01 January 2010 - 10:26 AM.

[the bigg]

What I'm afraid of is whether anyone knows a mod that patches .bmp and .bam's using the all-patchy regex search (which will probably patch mod files too )

No mod does that (well, Widescreen does that, but it only does so in PST and plain BG1, so it doesn't count).

Also, I'd like to point out that folks who use the BWP either tweak it before running the install.bat or don't tweak at all So the tweaking argument doesn't hold much ground atleast for the BWP.

Another thing to note is that, when people uninstall the BWP, they're just deleting their entire BGII-SoA folder and reinstalling from scratch, so the HUGE backup of the override isn't needed by the final biffer (unless it's needed for changelogs ... actually, no backups are needed unless you do a changelog )

I do tweak my install after getting through BWP (that was the reason why I wrote gen_biff in the first place). The primary goal of this project (restructuring biffing) should be to make sure that --change-log is useful and that tweaking the install is safe, not to reduce disk consumption (if you want to save disk space and lose tweaking abilities, just have cleanup.bat remove the folder generalized_biffing/prod).

And (sorry about these question-rounds ) what's the difference between MAKE_BIFF and --make-biff other than that the former creates a backup for uninstallation (in which case --make-biff would probably be better for disk space)

The content files are treated identically (I.E. they are left in the source directory and their contents are copied in the resulting bif). The differences are that --make-biff is a command-line switch and MAKE_BIFF is a tp2 action; moreover, MAKE_BIFF will keep a backup of chitin.key (a file that will likely never grow over the 1MB threshold) and will automatically restore and reload it when uninstalling the mod.

[Lollorian]

I do tweak my install after getting through BWP (that was the reason why I wrote gen_biff in the first place). The primary goal of this project (restructuring biffing) should be to make sure that --change-log is useful and that tweaking the install is safe, not to reduce disk consumption (if you want to save disk space and lose tweaking abilities, just have cleanup.bat remove the folder generalized_biffing/prod).

So ... tweaking with minimization?? Keep the backups for tweaking, but do some intelligent biffing (patchy files biffed later ... non-patchy stuff gets biffed with megamod)??

The content files are treated identically (I.E. they are left in the source directory and their contents are copied in the resulting bif). The differences are that --make-biff is a command-line switch and MAKE_BIFF is a tp2 action; moreover, MAKE_BIFF will keep a backup of chitin.key (a file that will likely never grow over the 1MB threshold) and will automatically restore and reload it when uninstalling the mod.

Ah, so MAKE_BIFF = easier tweaking/install/uninstallation (no difference in size/operation time etc right?? )

Now, we wait for Helborn and his mighty package (me only wanna see how it tisunpacks/oggdecs stuff without a .bat)

Cheers,
Lol