207 replies to this topic

[the bigg]

It can't be fixed without applying the obvious ugly hack to chitin.key.

[Lollorian]

Leomar, on Dec 3 2009, 04:00 AM, said:

Nice catch. Lollorian, can you report this at the BG2 Fixpack forum?

Aya aye cap'n (although I still dunno if it's supposed to "fix" stuff like this, it isn't a problem actually )

Cheers,
Lol

ATTACHMENT ONLY FOR LEOMAR!!! (if you see this, safely ignore this message!!! )

Attached Files

•  Setup-BoneHillv275.7z   6.85K   33 downloads

Edited by Lollorian, 17 April 2010 - 03:35 AM.

[Lollorian]

It looks like NI's Keyfile Cleanup Tool helps with this, I ran it on my test install and it deleted just 1 file (STORES.BIF) and removed some entries from the chitin.key.

Deleting E:\Program Files\Black Isle\BGII - SoA\DATA\STORES.BIF
Removing DATA/AREA090A.BIF
Removing DATA/AREA1800.BIF
Removing DATA/AREA200B.BIF
Removing DATA/AREA090B.BIF
Removing DATA/AREA2300.BIF
Removing DATA/AREA280A.BIF
Removing DATA/AREA1400.BIF
Removing DATA/AREA120B.BIF
Removing DATA/AREA150B.BIF
Removing DATA/PROGTES2.BIF
Removing DATA/AREA2600.BIF
Removing DATA/AREA110A.BIF
Removing DATA/PROGTEST.BIF
Removing DATA/AREA150C.BIF
Removing DATA/AREA2200.BIF
Removing DATA/AREA1900.BIF
Removing DATA/AREA1300.BIF
Removing DATA/AREA120A.BIF
Removing DATA/AREA1700.BIF
Removing DATA/AREA020B.BIF
Removing DATA/AREA140B.BIF
Removing DATA/AREA130A.BIF
Removing DATA/AREA160B.BIF
Removing DATA/STORES.BIF
Removing MOVIES/MOVCD3.BIF
Removing DATA/AREA280B.BIF
Removing DATA/AREA210B.BIF
Removing DATA/AREA1500.BIF
Removing DATA/AREA190A.BIF
Removing DATA/AREA1600.BIF
Removing DATA/AREA2800.BIF
Removing DATA/AREA2100.BIF
Removing DATA/AREA210A.BIF
Removing DATA/AREA200A.BIF
Removing DATA/STORES.BIF
Removing DATA/AREA1100.BIF
Removing DATA/AREA160A.BIF
Removing DATA/AREA2900.BIF
Removing DATA/AREA150A.BIF
Removing DATA/STORES.BIF
Removing DATA/AREA1200.BIF
Removing DATA/AREA130B.BIF
Removing DATA/AREA2700.BIF
Removing DATA/AREA260A.BIF
Removing DATA/AREA220A.BIF
Removing DATA/AREA290A.BIF
Removing DATA/AREA2500.BIF
Removing DATA/AREA090C.BIF
Removing DATA/AREA2400.BIF
Removing DATA/AREA140A.BIF
Removing DATA/AREA2000.BIF
Removing DATA/AREA0900.BIF
Removing DATA/AREA020A.BIF

Something I wanna ask here is, is there a tool that can find any unused files (like the MONKTU 8.DLG) and delete its entry from the .key?? (like NI's tool does for .bif's but this one could clean .bam, .dlg, .bcs, etc) This could be good considering almost all the vanilla scripts get overridden by SCS/Tactics/BP leaving them vanilla scripts rotting in the CHITIN.key

I say all this, not knowing (not confirmed exactly ) that all stuff gets added and not overwritten into the chitin.key.

A similar tool to clean up unused strings from the dialog.tlk (after creating a backup ofcourse ) would be great too ... although I don't think it's gonna give as much a performance boost as a keyfile cleanup utility

Another "tool idea" I could think of is something that unpacks any .bif that contains stuff that isn't referenced by the chitin.key and then repackages all of it into a new .bif (without the non-referenced stuff) ... since we have gen_biff for the repackaging bit, only an unpacker/sorter is needed (wouldn't that solve the vanilla .bif's staying in the game even though their contents are already in the other .bifs?? )

Cheers,
Lol

Edited by Lollorian, 03 December 2009 - 09:14 PM.

[Lollorian]

Is someone still working on this?? Thing is, I can offer up my system for some biffjobs for the time The pc's gonna be on all night for a few weeks and I figured I might as well have some testing done while I nap Meanwhile, I got a list of the biffs made by each mod extracted from the .tp2's

• NeJ (BP-NeJAreasSounds.bat)
  

  NEJ-TIS1	
  NEJ-TIS2	
  NEJ2-TIS	
  NEJ-ARE	
  NEJ-BAM	
  NEJ-BCS	
  NEJ-CRE	
  NEJ-DLG	
  NEJ-ITM	
  NEJ-LENA	
  NEJ-MELO	
  NEJ-PIC	
  NEJ-STO	
  NEJ-RULE	
  NEJ-SFX	
  NEJ-SPL	
  NEJ-WAV1	
  NEJ-WAV2	
  NEJ-WAV3	
  NEJ-WAV4	
  NEJ-WAV5

• BGT (EndSetup.bat) [1]
  

  BGTWAV	
  BG1MOVIE	
  AREA0015	
  AREA670A	
  AREA720A	
  AREA720B	
  AREA3100	
  AREA3200	
  AREA3300	
  AREA3400	
  AREA3500	
  AREA3600	
  AREA3700	
  AREA3800	
  AREA3900	
  AREA4100	
  AREA4200	
  AREA4300	
  AREA4400	
  AREA4600	
  AREA4700	
  AREA4800	
  AREA4900	
  AREA5100	
  AREA5300	
  AREA5400	
  AREA6500	
  AREA6600	
  AREA6700	
  AREA6800	
  AREA6900	
  AREA7000	
  AREA7100	
  AREA7200	
  AREA7300	
  AREA7400	
  AREA7500	
  AREA7600	
  AREA7700	
  AREA7800	
  AREA7900	
  AREA8000	
  AREA8100	
  AREA8200	
  AREA8300	
  AREA8400	
  AREA8500	
  AREA8600	
  AREA8700	
  AREA8800	
  AREA8900	
  AREA9000	
  AREA9100	
  AREA9200	
  AREA9300	
  AREA9400	
  AREA9500	
  AREA9600	
  AREA9700	
  AREA9800	
  AREA9900	
  AREAA000	
  AREAA100	
  AREAD000	
  AREAU000	
  AREAW000	
  AREAW500	
  AREAAM00	
  BG1ARE	
  BG1BCS	
  BGTCRE	
  BG1DLG	
  BG1ITM	
  BG1MISC	
  BG1SPL	
  BG1STO

• BGT NPC Sound (Install_BGT-NPCSound.bat)
  

  BGTGerSndPatch

• DH (install.bat in BWFixpack folder) [2]
  

  dhare

• NTotSC (NTAreas.bat)
  

  NTotSC1	
  NTotSC2	
  NTotSC3

• SoBH (BHAreas.bat)
  

  BH-TIS1	
  BH-TIS2	
  BH-REST

• CtB (CtbFinishInstall.bat, ChoresInstall.bat)
  

  CTB-ARE	
  CTB-BAM	
  CTB-BCS	
  CTB-CRE	
  CTB-ITM	
  CTB-MOS	
  CTB-RULE	
  CTB-TIS1	
  CTB-TIS2	
  CTB-TIS3	
  CTB-WAV	
  CTB-CHRS

• Drizzt (Areas.bat)
  

  DZTS-ARE

• DSotSC (DSAreas.bat)
  

  DSOTSC1

• JZ (Setup-JZ.tp2)
  

  JZMovie

• RoT (RoT-Install.bat)
  

  ROT-ARE	
  ROT-BAM	
  ROT-BCS	
  ROT-CRE	
  ROT-DLG	
  ROT-ITM	
  ROT-MOS	
  ROT-RULE	
  ROT-SPL	
  ROT-TIS1	
  ROT-TIS2	
  ROT-TIS3	
  ROT-WAV1	
  ROT-WAV2	

• SoS (SOS-TISOGGBIFF.bat)
  

  SOS-ARE	
  SOS-BAM	
  SOS-BCS	
  SOS-CRE	
  SOS-DLG	
  SOS-ITM	
  SOS-MOS	
  SOS-RULE	
  SOS-SPL	
  SOS-TIS1	
  SOS-TIS2	
  SOS-WAV1	
  SOS-WAV2	
  SOS-MVE	

• TDD (TDD-TISBIFF.bat)
  

  TDD-ARE
  TDD-BAM	
  TDD-BCS	
  TDD-CRE	
  TDD-DLG	
  TDD-ITM	
  TDD-MOS	
  TDD-RULE	
  TDD-SPL	
  TDD-TIS1	
  TDD-TIS2	
  TDD-TIS3	
  TDD-TIS4	
  TDD-WAV	

• ToA (end.bat)
  

  ToA1

• ToD (MovieInstall.bat)
  

  VA#Movie

• TS-BP (TSAreasSounds.bat)
  

  TS-ARE	
  TS-BAM	
  TS-BCS	
  TS-CRE	
  TS-DLG	
  TS-ITM	
  TS-MOS	
  TS-RULE	
  TS-TIS	
  TS-WAV

• Widescreen [3]

[1] - All the make-biff lines have _weidu at the start instead of just weidu, so this probably doesn't do any biffing

[2] - The original mod doesn't do biffing, the BWPFixpack mods it to biff at the end

[3] - Seems to biff stuff using MAKE_BIFF (something that gen_biff uses too)

So, now you know which folders to rename

Cheers,
Lol

Edited by Lollorian, 17 December 2009 - 06:19 PM.

[the bigg]

[1] is because it copies setup-xxx.exe to _weidu.exe rather than weidu.exe. Still suboptimal to MAKE_BIFF, but doesn't break my environment (having weidu.exe in ./ breaks weinstall when c:\bin\weidu.exe is the beta).

[Lollorian]

Updated the list to show which files contain the make-biff lines

So, BGT does biffin too then. Btw, am I right in thinking that you just need to delete all instances of say ... mov override\blah NEJ-TIS1\blah (and variants) and the make-biff lines for the biffing to NOT happen??

Since almost every mod (except JZ which only biffs a movie) does the biffing through .bat, would just deleing the .bat's themselves do the job??

Cheers,
Lol

[Jarno Mikkola]

Lollorian, on Dec 18 2009, 04:28 AM, said:

Since almost every mod (except JZ which only biffs a movie) does the biffing through .bat, would just deleting the .bat's themselves do the job??

I don't think so, cause the files would be in the wrong directory, not in override, and second you would still need parts of the .bat for the other actions, like extending the .tiz'ses to .tis'ses .
Of course replacing the .bat would help as the .bat's usually unpacks and even cuts the old files to a directory which is not the override... say like the SOS-TIS1/-2 has all the SoS's .tis file, while the SOS-ITM has the few items from SoS and the rest of the override folder that are not biffed already by other mods.

Well, as I checked the Setup-CtB.tp2, I saw these 12 lines:

MKDIR ~CTB-ARE~ 
MKDIR ~CTB-BCS~ ...

Which I know will make the directories that will eventually biffed, that should matter much...

I also checked the tiz-to-tis part of the CtbFinishInstall.bat and noticed that it not only unpacks the files but puts them to CTB-TIS/*filename*.tiz, so we just need to correct this and put them to: override/*filename*.tiz

Edited by Jarno Mikkola, 26 December 2009 - 01:23 PM.

[Lollorian]

Jarno Mikkola, on Dec 18 2009, 01:20 PM, said:

Lollorian, on Dec 18 2009, 04:28 AM, said:

Since almost every mod (except JZ which only biffs a movie) does the biffing through .bat, would just deleting the .bat's themselves do the job??

I don't think so, cause the files would be in the wrong directory, not in override, and second you would still need parts of the .bat for the other actions, like extending the .tiz'ses to .tis'ses .
Of course replacing the .bat would help as the .bat's usually unpacks and even cuts the old files to a directory which is not the override... say like the SOS-TIS1/-2 has all the SoS's .tis file, while the SOS-ITM has the few items from SoS and the rest of the override folder that are not biffed already by other mods.

That's it I'm leaving this stuff to the experts I'll probably try to be the slave who does the testing

Cheers,
Lol

[Lollorian]

Heehee caught your edit

Jarno Mikkola, on Dec 18 2009, 01:20 PM, said:

Well, as I checked the Setup-CtB.tp2, I saw these 12 lines:

MKDIR ~CTB-ARE~ 
MKDIR ~CTB-BCS~ ...

Which I know will make the directories that will eventually biffed, that should matter much...

I also checked the tiz-to-tis part of the CtbFinishInstall.bat and noticed that it not only unpacks the files but puts them to CTB-TIS/*filename*.tiz, so we just need to correct this and put them to: override/*filename*.tiz

So, it's just a matter of replacing the folder names with override and editing the .bat/.tp2 to not biff then??

Sounds too easy ... there must be a catch

PS: Is Helborn (or anyone) working on this in private or something, I'd like to help in this as much as I can from Jan 7 or something (yeah, I am a clean-freak so a smaller data\ folder means a lot to me )

Cheers,
Lol

[Jarno Mikkola]

Lollorian, on Dec 27 2009, 05:53 AM, said:

Heehee caught your edit

The Imp goes on a rampage and beats the lol-sign to the ground. Don't double post.
JJ

Lollorian, on Dec 27 2009, 05:53 AM, said:

So, it's just a matter of replacing the folder names with override and editing the .bat/.tp2 to not biff then??
Sounds too easy ... there must be a catch

Yeah, it does sound too easy, thus the stealth edit, and the joke.
At least on of the catch is; the other big mods assume the previous mods biff their files and so they check for the biffs.
So it's might be better to actually biff the empty folder.

Edited by Jarno Mikkola, 27 December 2009 - 01:50 AM.

[Lollorian]

Jarno Mikkola, on Dec 27 2009, 03:10 PM, said:

So it's might be better to actually biff the empty folder.

OMG!! Do size zero .bif's exist?? Then it's easier even

And we'll also need a list of all the files needed by a mod to show that it exists ... (I'll probably extract all the ACTION_IF FILE_EXISTS blocks later ) ... Methinks that's the purpose of those .g3 files

As an example, extracting this will make SoBH not biff (and I checked, and found that there's nothing that checks if SoBH is installed )

Cheers,
Lol

Attached Files

•  BoneHill_NoBiff.7z   5.94K   43 downloads

Edited by Lollorian, 27 December 2009 - 03:20 AM.

[Jarno Mikkola]

Lollorian, on Dec 27 2009, 12:57 PM, said:

Jarno Mikkola, on Dec 27 2009, 03:10 PM, said:

So it's might be better to actually biff the empty folder.

OMG!! Do size zero .bif's exist?? Then it's easier even

Don't know, but if one was to put this sort of thing into the .bat file before the biffing, the zero size won't be an issue:

COPY + ~biffingcorrections/sw1h01.itm~ ~*biffed folder*/biffchec.itm~

Where the sw1h01.itm is the normal not modified Bastard Sword from the clean BG2 install. Hmm, did I already named the mod some sad sob needs to build, erhm yes... biffingcorrections.

Edited by Jarno Mikkola, 27 December 2009 - 06:02 AM.

[Lollorian]

Ok, extracted all the FILE_EXISTS things and it seems that only these are actually checked (from the .bif's)

BGT -> BG1ARE.bif
NTotSC -> NTOTSC1.bif
CtB -> CTB-RULE.bif
RoT -> ROT-RULE.bif
SoS -> SOS-RULE.bif
TDD -> TDD-RULE.bif
TS -> TS-RULE.bif

So, only these folders need to be preserved while the others can directly be sent to the override But these RULE.bif's contain .sto's which probably get patched too ... so your idea of putting in a dummy file to create the .bif works

Btw, instead of an .itm, how about putting in something that has no chance of being patched whatsoever (like a .mos, .vvc ... hell, can a .xyz be biffed?? )

Cheers,
Lol

Edited by Lollorian, 27 December 2009 - 05:36 AM.

[Jarno Mikkola]

Lollorian, on Dec 27 2009, 03:25 PM, said:

Btw, instead of an .itm, how about putting in something that has no chance of being patched whatsoever (like a .mos, .vvc ... hell, can a .xyz be biffed?? )

Can those files be biffed? I wouldn't risk it, and besides when we name the file as biffchec.itm(F/&, I forgot it needs to be 8 marks max, ah well), the item will never be used as a real item by anymod, so it will never be in the actual game.

[Lollorian]

Ok, so everything seems easy enough

Now, 2 questions,

1) is someone doing this?? It'd be a waste to start this when someone's already completed 75% of it

And 2) Can gen_biff actually handle such a large volume of files at one time??

If this is successful, it'll probably snip off 2-3 GBs off the BWP And with a good backup-keeping-but-deleting-all-other-files type Cleanup.bat, that's a LOT of space gained

I'm doing an expert install (with the intermediate biffing) tonight for comparison stuffs Then, after new year or something, I could start doing the no-biffer tests

Cheers,
Lol

[Lollorian]

Sorry for the double post, but I finished an expert install (with some errors ... they're probably minor )

Anyway, now, statistics:

Time - 6h 25m
BGII-SoA size - 27.5 GB
data size - 13.7 GB
override size - 24.2 MB

No. of gen_biff .bif's - 180

Now, I used a new (and improved?? ) Cleanup.bat that preserves stuff needed for a changelog, and looks like it did a fairly good job

Statistics (after CleanupX.bat)

BGII-SoA size - 22.1 GB
data size - 13.7 GB
override size - 2.87 MB

So, overall space savings while keeping backups = 5.4 GB ( Bigger than the vanilla BGII-ToB install!!!)

I've attached it here btw, please have a look (it's probably fubared my install without me knowing ... so, it'd help if there's stuff that needs to be changed/removed/added ) Oh, btw, it takes some 2 hours to do the selective cleanup on a full-bells-and-whistles expert install (on my system ), any idea how to optimize it??

Removed because of changelog-fubaring bug because it didn't keep .tp2's and .tra's in the mod folders See next post for new one

Thanks,
Lol

Edited by Lollorian, 28 December 2009 - 06:03 PM.

[Lollorian]

Hmmm, now that I look at it, why does NeJ have its own special biffer?? (NeJ2Biffer)

Next, here's a new CleanupX.bat (*grumble* *grumble* no changelogs *grumble*) Not needed cause the BWP's Cleanup.bat does this much better

So, here's CtB (no Biffin Edition ) ... also made sure that the Modcheck (CTB-RULE.bif) got created. This is the code:

weidu --out CTB-RULE --biff-get-rest sw1h01.itm
weidu --make-biff CTB-RULE
call CtB\deldir CTB-RULE

I figured I'll just leave the itemname as it is (renaming it would create a new copy of the item (yeah, 1 kB means that much!!! )

Cheers,
Lol

Attached Files

•  CtB_NoBiff.7z   28.33K   35 downloads

Edited by Lollorian, 03 January 2010 - 07:50 PM.

[the bigg]

- Gen biff (the publicly available version) can't handle more than 100 bifs (easily fixed by tweaking the tp2 to use less characters than tb#gen):
MAKE_BIFF ~tb#gen%currentFile%~ -> MAKE_BIFF ~tb#gb%currentFile%~ (do this twice in the tp2 file).

- tis/wed/bmp/mos/bam/wav files should still be biffed (using MAKE_BIFF) by the individual mods - this will make the new gen_biff be noticeably faster (which is good, especially since people will be altering tweak components here and there, and gen_biff is always touched by a stack traversal).

Edited by the bigg, 28 December 2009 - 07:12 PM.

[Lollorian]

the bigg, on Dec 29 2009, 08:40 AM, said:

- Gen biff (the publicly available version) can't handle more than 100 bifs (easily fixed by tweaking the tp2 to use less characters than tb#gen):
MAKE_BIFF ~tb#gen%currentFile%~ -> MAKE_BIFF ~tb#gb%currentFile%~ (do this twice in the tp2 file).

Holy what?!? I'm looking at my data folder and the last one is tb#gen180.bif Biffs work on the 8-char limit too?? Well, it's official then, game f00bared

the bigg, on Dec 29 2009, 08:40 AM, said:

- tis/wed/bmp/mos/bam/wav files should still be biffed (using MAKE_BIFF) by the individual mods - this will make the new gen_biff be noticeably faster (which is good, especially since people will be altering tweak components here and there, and gen_biff is always touched by a stack traversal).

So, there's no way those files can be patched by something later, right?? (doesn't something try to fix bams??)

Oh, and one more thing, is there a size limit to .bif's?? Can we stuff everything that isn't patched into the .bif's that are required by the megamods for their checks??

This looks like more work than I thought! But it's easy, robotic work (which even I can do ) Will probably start by second week of Jan or something (really hope BWP9 doesn't get out before that though )

Cheers,
Lol

[the bigg]

Lollorian, on Dec 29 2009, 04:31 AM, said:

Holy what?!? I'm looking at my data folder and the last one is tb#gen180.bif Biffs work on the 8-char limit too?? Well, it's official then, game f00bared

I'm an idiot - biffs are stored as arbitrary-length strings rather than as statically allocated strings, so even long names are OK.

Quote

So, there's no way those files can be patched by something later, right?? (doesn't something try to fix bams??)

Well, some mods alter bams, but IMHO only a small portion of them are overwritten, so biffing some of them twice is not that bad (you might lose some ~50 MB by biffing twice, but remove half a gigabyte or so from gen_biff).

Quote

Oh, and one more thing, is there a size limit to .bif's?? Can we stuff everything that isn't patched into the .bif's that are required by the megamods for their checks??

The largest bif in my install is 200 MB (BG1MOVIE from BGT) - use that as a thresold. Add MVE to the list of formats to biff per-mod, BTW - the game flat-out crashes if a MVE is in the override.