216 replies to this topic

[the bigg]

It can't be fixed without applying the obvious ugly hack to chitin.key.

[Lollorian]

Nice catch. Lollorian, can you report this at the BG2 Fixpack forum?

Aya aye cap'n (although I still dunno if it's supposed to "fix" stuff like this, it isn't a problem actually )

Cheers,
Lol

ATTACHMENT ONLY FOR LEOMAR!!! (if you see this, safely ignore this message!!! )

Attached Files

•  Setup-BoneHillv275.7z   6.85K   290 downloads

Edited by Lollorian, 17 April 2010 - 03:35 AM.

[Lollorian]

It looks like NI's Keyfile Cleanup Tool helps with this, I ran it on my test install and it deleted just 1 file (STORES.BIF) and removed some entries from the chitin.key.

[codebox]Deleting E:\Program Files\Black Isle\BGII - SoA\DATA\STORES.BIF
Removing DATA/AREA090A.BIF
Removing DATA/AREA1800.BIF
Removing DATA/AREA200B.BIF
Removing DATA/AREA090B.BIF
Removing DATA/AREA2300.BIF
Removing DATA/AREA280A.BIF
Removing DATA/AREA1400.BIF
Removing DATA/AREA120B.BIF
Removing DATA/AREA150B.BIF
Removing DATA/PROGTES2.BIF
Removing DATA/AREA2600.BIF
Removing DATA/AREA110A.BIF
Removing DATA/PROGTEST.BIF
Removing DATA/AREA150C.BIF
Removing DATA/AREA2200.BIF
Removing DATA/AREA1900.BIF
Removing DATA/AREA1300.BIF
Removing DATA/AREA120A.BIF
Removing DATA/AREA1700.BIF
Removing DATA/AREA020B.BIF
Removing DATA/AREA140B.BIF
Removing DATA/AREA130A.BIF
Removing DATA/AREA160B.BIF
Removing DATA/STORES.BIF
Removing MOVIES/MOVCD3.BIF
Removing DATA/AREA280B.BIF
Removing DATA/AREA210B.BIF
Removing DATA/AREA1500.BIF
Removing DATA/AREA190A.BIF
Removing DATA/AREA1600.BIF
Removing DATA/AREA2800.BIF
Removing DATA/AREA2100.BIF
Removing DATA/AREA210A.BIF
Removing DATA/AREA200A.BIF
Removing DATA/STORES.BIF
Removing DATA/AREA1100.BIF
Removing DATA/AREA160A.BIF
Removing DATA/AREA2900.BIF
Removing DATA/AREA150A.BIF
Removing DATA/STORES.BIF
Removing DATA/AREA1200.BIF
Removing DATA/AREA130B.BIF
Removing DATA/AREA2700.BIF
Removing DATA/AREA260A.BIF
Removing DATA/AREA220A.BIF
Removing DATA/AREA290A.BIF
Removing DATA/AREA2500.BIF
Removing DATA/AREA090C.BIF
Removing DATA/AREA2400.BIF
Removing DATA/AREA140A.BIF
Removing DATA/AREA2000.BIF
Removing DATA/AREA0900.BIF
Removing DATA/AREA020A.BIF[/codebox]
Something I wanna ask here is, is there a tool that can find any unused files (like the MONKTU 8.DLG) and delete its entry from the .key?? (like NI's tool does for .bif's but this one could clean .bam, .dlg, .bcs, etc) This could be good considering almost all the vanilla scripts get overridden by SCS/Tactics/BP leaving them vanilla scripts rotting in the CHITIN.key

I say all this, not knowing (not confirmed exactly ) that all stuff gets added and not overwritten into the chitin.key.

A similar tool to clean up unused strings from the dialog.tlk (after creating a backup ofcourse ) would be great too ... although I don't think it's gonna give as much a performance boost as a keyfile cleanup utility

Another "tool idea" I could think of is something that unpacks any .bif that contains stuff that isn't referenced by the chitin.key and then repackages all of it into a new .bif (without the non-referenced stuff) ... since we have gen_biff for the repackaging bit, only an unpacker/sorter is needed (wouldn't that solve the vanilla .bif's staying in the game even though their contents are already in the other .bifs?? )

Cheers,
Lol

Edited by Lollorian, 03 December 2009 - 09:14 PM.

[Lollorian]

Is someone still working on this?? Thing is, I can offer up my system for some biffjobs for the time The pc's gonna be on all night for a few weeks and I figured I might as well have some testing done while I nap Meanwhile, I got a list of the biffs made by each mod extracted from the .tp2's

• NeJ (BP-NeJAreasSounds.bat)
  [codebox]NEJ-TIS1
  NEJ-TIS2
  NEJ2-TIS
  NEJ-ARE
  NEJ-BAM
  NEJ-BCS
  NEJ-CRE
  NEJ-DLG
  NEJ-ITM
  NEJ-LENA
  NEJ-MELO
  NEJ-PIC
  NEJ-STO
  NEJ-RULE
  NEJ-SFX
  NEJ-SPL
  NEJ-WAV1
  NEJ-WAV2
  NEJ-WAV3
  NEJ-WAV4
  NEJ-WAV5[/codebox]
  [*]BGT (EndSetup.bat) [color="#FF0000"][b][1][/b][/color]
  [codebox]BGTWAV
  BG1MOVIE
  AREA0015
  AREA670A
  AREA720A
  AREA720B
  AREA3100
  AREA3200
  AREA3300
  AREA3400
  AREA3500
  AREA3600
  AREA3700
  AREA3800
  AREA3900
  AREA4100
  AREA4200
  AREA4300
  AREA4400
  AREA4600
  AREA4700
  AREA4800
  AREA4900
  AREA5100
  AREA5300
  AREA5400
  AREA6500
  AREA6600
  AREA6700
  AREA6800
  AREA6900
  AREA7000
  AREA7100
  AREA7200
  AREA7300
  AREA7400
  AREA7500
  AREA7600
  AREA7700
  AREA7800
  AREA7900
  AREA8000
  AREA8100
  AREA8200
  AREA8300
  AREA8400
  AREA8500
  AREA8600
  AREA8700
  AREA8800
  AREA8900
  AREA9000
  AREA9100
  AREA9200
  AREA9300
  AREA9400
  AREA9500
  AREA9600
  AREA9700
  AREA9800
  AREA9900
  AREAA000
  AREAA100
  AREAD000
  AREAU000
  AREAW000
  AREAW500
  AREAAM00
  BG1ARE
  BG1BCS
  BGTCRE
  BG1DLG
  BG1ITM
  BG1MISC
  BG1SPL
  BG1STO[/codebox]
  [*]BGT NPC Sound (Install_BGT-NPCSound.bat)
  [code]BGTGerSndPatch[/code]
  [*]DH (install.bat in BWFixpack folder) [color="#FF0000"][b][2][/b][/color]
  [code]dhare[/code]
  [*]NTotSC (NTAreas.bat)
  [code]NTotSC1
  NTotSC2
  NTotSC3[/code]
  
• SoBH (BHAreas.bat)
  

  BH-TIS1	BH-TIS2	BH-REST

• CtB (CtbFinishInstall.bat, ChoresInstall.bat)
  [codebox]CTB-ARE
  CTB-BAM
  CTB-BCS
  CTB-CRE
  CTB-ITM
  CTB-MOS
  CTB-RULE
  CTB-TIS1
  CTB-TIS2
  CTB-TIS3
  CTB-WAV
  CTB-CHRS[/codebox]
  [*]Drizzt (Areas.bat)
  [code]DZTS-ARE[/code]
  [*]DSotSC (DSAreas.bat)
  [code]DSOTSC1[/code]
  
• JZ (Setup-JZ.tp2)
  

  JZMovie

• RoT (RoT-Install.bat)
  [codebox]ROT-ARE
  ROT-BAM
  ROT-BCS
  ROT-CRE
  ROT-DLG
  ROT-ITM
  ROT-MOS
  ROT-RULE
  ROT-SPL
  ROT-TIS1
  ROT-TIS2
  ROT-TIS3
  ROT-WAV1
  ROT-WAV2 [/codebox]
  [*]SoS (SOS-TISOGGBIFF.bat)
  [codebox]SOS-ARE
  SOS-BAM
  SOS-BCS
  SOS-CRE
  SOS-DLG
  SOS-ITM
  SOS-MOS
  SOS-RULE
  SOS-SPL
  SOS-TIS1
  SOS-TIS2
  SOS-WAV1
  SOS-WAV2
  SOS-MVE [/codebox]
  [*]TDD (TDD-TISBIFF.bat)
  [codebox]TDD-ARE
  TDD-BAM
  TDD-BCS
  TDD-CRE
  TDD-DLG
  TDD-ITM
  TDD-MOS
  TDD-RULE
  TDD-SPL
  TDD-TIS1
  TDD-TIS2
  TDD-TIS3
  TDD-TIS4
  TDD-WAV [/codebox]
  [*]ToA (end.bat)
  [code]ToA1[/code]
  [*]ToD (MovieInstall.bat)
  [code]VA#Movie[/code]
  [*]TS-BP (TSAreasSounds.bat)
  [code]TS-ARE
  TS-BAM
  TS-BCS
  TS-CRE
  TS-DLG
  TS-ITM
  TS-MOS
  TS-RULE
  TS-TIS
  TS-WAV[/code]
  
• Widescreen [3]

[1] - All the make-biff lines have _weidu at the start instead of just weidu, so this probably doesn't do any biffing

[2] - The original mod doesn't do biffing, the BWPFixpack mods it to biff at the end

[3] - Seems to biff stuff using MAKE_BIFF (something that gen_biff uses too)

So, now you know which folders to rename

Cheers,
Lol

Edited by Lollorian, 17 December 2009 - 06:19 PM.

[the bigg]

[1] is because it copies setup-xxx.exe to _weidu.exe rather than weidu.exe. Still suboptimal to MAKE_BIFF, but doesn't break my environment (having weidu.exe in ./ breaks weinstall when c:\bin\weidu.exe is the beta).

[Lollorian]

Updated the list to show which files contain the make-biff lines

So, BGT does biffin too then. Btw, am I right in thinking that you just need to delete all instances of say ... mov override\blah NEJ-TIS1\blah (and variants) and the make-biff lines for the biffing to NOT happen??

Since almost every mod (except JZ which only biffs a movie) does the biffing through .bat, would just deleing the .bat's themselves do the job??

Cheers,
Lol

[Jarno Mikkola]

Since almost every mod (except JZ which only biffs a movie) does the biffing through .bat, would just deleting the .bat's themselves do the job??

I don't think so, cause the files would be in the wrong directory, not in override, and second you would still need parts of the .bat for the other actions, like extending the .tiz'ses to .tis'ses .
Of course replacing the .bat would help as the .bat's usually unpacks and even cuts the old files to a directory which is not the override... say like the SOS-TIS1/-2 has all the SoS's .tis file, while the SOS-ITM has the few items from SoS and the rest of the override folder that are not biffed already by other mods.

Well, as I checked the Setup-CtB.tp2, I saw these 12 lines:

MKDIR ~CTB-ARE~ 
MKDIR ~CTB-BCS~ ...

Which I know will make the directories that will eventually biffed, that should matter much...

I also checked the tiz-to-tis part of the CtbFinishInstall.bat and noticed that it not only unpacks the files but puts them to CTB-TIS/*filename*.tiz, so we just need to correct this and put them to: override/*filename*.tiz

Edited by Jarno Mikkola, 26 December 2009 - 01:23 PM.

[Lollorian]

Since almost every mod (except JZ which only biffs a movie) does the biffing through .bat, would just deleting the .bat's themselves do the job??

I don't think so, cause the files would be in the wrong directory, not in override, and second you would still need parts of the .bat for the other actions, like extending the .tiz'ses to .tis'ses .
Of course replacing the .bat would help as the .bat's usually unpacks and even cuts the old files to a directory which is not the override... say like the SOS-TIS1/-2 has all the SoS's .tis file, while the SOS-ITM has the few items from SoS and the rest of the override folder that are not biffed already by other mods.

That's it I'm leaving this stuff to the experts I'll probably try to be the slave who does the testing

Cheers,
Lol

[Lollorian]

Heehee caught your edit

Well, as I checked the Setup-CtB.tp2, I saw these 12 lines:

MKDIR ~CTB-ARE~ 
MKDIR ~CTB-BCS~ ...

Which I know will make the directories that will eventually biffed, that should matter much...

I also checked the tiz-to-tis part of the CtbFinishInstall.bat and noticed that it not only unpacks the files but puts them to CTB-TIS/*filename*.tiz, so we just need to correct this and put them to: override/*filename*.tiz

So, it's just a matter of replacing the folder names with override and editing the .bat/.tp2 to not biff then??

Sounds too easy ... there must be a catch

PS: Is Helborn (or anyone) working on this in private or something, I'd like to help in this as much as I can from Jan 7 or something (yeah, I am a clean-freak so a smaller data\ folder means a lot to me )

Cheers,
Lol

[Jarno Mikkola]

Heehee caught your edit

The Imp goes on a rampage and beats the lol-sign to the ground. Don't double post.
JJ

So, it's just a matter of replacing the folder names with override and editing the .bat/.tp2 to not biff then??
Sounds too easy ... there must be a catch

Yeah, it does sound too easy, thus the stealth edit, and the joke.
At least on of the catch is; the other big mods assume the previous mods biff their files and so they check for the biffs.
So it's might be better to actually biff the empty folder.

Edited by Jarno Mikkola, 27 December 2009 - 01:50 AM.

[Lollorian]

So it's might be better to actually biff the empty folder.

OMG!! Do size zero .bif's exist?? Then it's easier even

And we'll also need a list of all the files needed by a mod to show that it exists ... (I'll probably extract all the ACTION_IF FILE_EXISTS blocks later ) ... Methinks that's the purpose of those .g3 files

As an example, extracting this will make SoBH not biff (and I checked, and found that there's nothing that checks if SoBH is installed )

Cheers,
Lol

Attached Files

•  BoneHill_NoBiff.7z   5.94K   311 downloads

Edited by Lollorian, 27 December 2009 - 03:20 AM.

[Jarno Mikkola]

So it's might be better to actually biff the empty folder.

OMG!! Do size zero .bif's exist?? Then it's easier even

Don't know, but if one was to put this sort of thing into the .bat file before the biffing, the zero size won't be an issue:

COPY + ~biffingcorrections/sw1h01.itm~ ~*biffed folder*/biffchec.itm~

Where the sw1h01.itm is the normal not modified Bastard Sword from the clean BG2 install. Hmm, did I already named the mod some sad sob needs to build, erhm yes... biffingcorrections.

Edited by Jarno Mikkola, 27 December 2009 - 06:02 AM.

[Lollorian]

Ok, extracted all the FILE_EXISTS things and it seems that only these are actually checked (from the .bif's)

BGT -> BG1ARE.bif
NTotSC -> NTOTSC1.bif
CtB -> CTB-RULE.bif
RoT -> ROT-RULE.bif
SoS -> SOS-RULE.bif
TDD -> TDD-RULE.bif
TS -> TS-RULE.bif

So, only these folders need to be preserved while the others can directly be sent to the override But these RULE.bif's contain .sto's which probably get patched too ... so your idea of putting in a dummy file to create the .bif works

Btw, instead of an .itm, how about putting in something that has no chance of being patched whatsoever (like a .mos, .vvc ... hell, can a .xyz be biffed?? )

Cheers,
Lol

Edited by Lollorian, 27 December 2009 - 05:36 AM.

[Jarno Mikkola]

Btw, instead of an .itm, how about putting in something that has no chance of being patched whatsoever (like a .mos, .vvc ... hell, can a .xyz be biffed?? )

Can those files be biffed? I wouldn't risk it, and besides when we name the file as biffchec.itm(F/&, I forgot it needs to be 8 marks max, ah well), the item will never be used as a real item by anymod, so it will never be in the actual game.

[Lollorian]

Ok, so everything seems easy enough

Now, 2 questions,

1) is someone doing this?? It'd be a waste to start this when someone's already completed 75% of it

And 2) Can gen_biff actually handle such a large volume of files at one time??

If this is successful, it'll probably snip off 2-3 GBs off the BWP And with a good backup-keeping-but-deleting-all-other-files type Cleanup.bat, that's a LOT of space gained

I'm doing an expert install (with the intermediate biffing) tonight for comparison stuffs Then, after new year or something, I could start doing the no-biffer tests

Cheers,
Lol

[Lollorian]

Sorry for the double post, but I finished an expert install (with some errors ... they're probably minor )

Anyway, now, statistics:

Time - 6h 25m
BGII-SoA size - 27.5 GB
data size - 13.7 GB
override size - 24.2 MB

No. of gen_biff .bif's - 180

Now, I used a new (and improved?? ) Cleanup.bat that preserves stuff needed for a changelog, and looks like it did a fairly good job

Statistics (after CleanupX.bat)

BGII-SoA size - 22.1 GB
data size - 13.7 GB
override size - 2.87 MB

So, overall space savings while keeping backups = 5.4 GB ( Bigger than the vanilla BGII-ToB install!!!)

I've attached it here btw, please have a look (it's probably fubared my install without me knowing ... so, it'd help if there's stuff that needs to be changed/removed/added ) Oh, btw, it takes some 2 hours to do the selective cleanup on a full-bells-and-whistles expert install (on my system ), any idea how to optimize it??

Removed because of changelog-fubaring bug because it didn't keep .tp2's and .tra's in the mod folders See next post for new one

Thanks,
Lol

Edited by Lollorian, 28 December 2009 - 06:03 PM.

[Lollorian]

Hmmm, now that I look at it, why does NeJ have its own special biffer?? (NeJ2Biffer)

Next, here's a new CleanupX.bat (*grumble* *grumble* no changelogs *grumble*) Not needed cause the BWP's Cleanup.bat does this much better

So, here's CtB (no Biffin Edition ) ... also made sure that the Modcheck (CTB-RULE.bif) got created. This is the code:

weidu --out CTB-RULE --biff-get-rest sw1h01.itm
weidu --make-biff CTB-RULE
call CtB\deldir CTB-RULE

I figured I'll just leave the itemname as it is (renaming it would create a new copy of the item (yeah, 1 kB means that much!!! )

Cheers,
Lol

Attached Files

•  CtB_NoBiff.7z   28.33K   285 downloads

Edited by Lollorian, 03 January 2010 - 07:50 PM.

[the bigg]

- Gen biff (the publicly available version) can't handle more than 100 bifs (easily fixed by tweaking the tp2 to use less characters than tb#gen):
MAKE_BIFF ~tb#gen%currentFile%~ -> MAKE_BIFF ~tb#gb%currentFile%~ (do this twice in the tp2 file).

- tis/wed/bmp/mos/bam/wav files should still be biffed (using MAKE_BIFF) by the individual mods - this will make the new gen_biff be noticeably faster (which is good, especially since people will be altering tweak components here and there, and gen_biff is always touched by a stack traversal).

Edited by the bigg, 28 December 2009 - 07:12 PM.

[Lollorian]

- Gen biff (the publicly available version) can't handle more than 100 bifs (easily fixed by tweaking the tp2 to use less characters than tb#gen):
MAKE_BIFF ~tb#gen%currentFile%~ -> MAKE_BIFF ~tb#gb%currentFile%~ (do this twice in the tp2 file).

Holy what?!? I'm looking at my data folder and the last one is tb#gen180.bif Biffs work on the 8-char limit too?? Well, it's official then, game f00bared

- tis/wed/bmp/mos/bam/wav files should still be biffed (using MAKE_BIFF) by the individual mods - this will make the new gen_biff be noticeably faster (which is good, especially since people will be altering tweak components here and there, and gen_biff is always touched by a stack traversal).

So, there's no way those files can be patched by something later, right?? (doesn't something try to fix bams??)

Oh, and one more thing, is there a size limit to .bif's?? Can we stuff everything that isn't patched into the .bif's that are required by the megamods for their checks??

This looks like more work than I thought! But it's easy, robotic work (which even I can do ) Will probably start by second week of Jan or something (really hope BWP9 doesn't get out before that though )

Cheers,
Lol

[the bigg]

Holy what?!? I'm looking at my data folder and the last one is tb#gen180.bif Biffs work on the 8-char limit too?? Well, it's official then, game f00bared

I'm an idiot - biffs are stored as arbitrary-length strings rather than as statically allocated strings, so even long names are OK.

So, there's no way those files can be patched by something later, right?? (doesn't something try to fix bams??)

Well, some mods alter bams, but IMHO only a small portion of them are overwritten, so biffing some of them twice is not that bad (you might lose some ~50 MB by biffing twice, but remove half a gigabyte or so from gen_biff).

Oh, and one more thing, is there a size limit to .bif's?? Can we stuff everything that isn't patched into the .bif's that are required by the megamods for their checks??

The largest bif in my install is 200 MB (BG1MOVIE from BGT) - use that as a thresold. Add MVE to the list of formats to biff per-mod, BTW - the game flat-out crashes if a MVE is in the override.