Jump to content


Photo

Brainstorming for the BWP


  • Please log in to reply
141 replies to this topic

#61 the bigg

the bigg

    2083 is a prime number.

  • Modder
  • 3331 posts

Posted 12 September 2010 - 02:19 PM

I ran some tests, and it appears that those who are pro-deletion are in the wrong: a hundred marker files don't affect measurably COPY_EXISTING_REGEXP GLOB.

I created 25000 new .mrk in the override on an otherwise clean BG2 installation to simulate a full Expert install.
OUTER_FOR (i = 0; i < 25000; ++i) BEGIN
	COPY_EXISTING ~sw1h01.itm~ ~override/tb#%i%.mrk~
END

I then ran patchless, saveless C_E_R on spl files and measured the times they took under different circumstances.

COPY_EXISTING_REGEXP NOGLOB ~.*\.spl~ ~override~
BUT_ONLY
NOGLOB doesn't read the override folder at all, and processes in about 2.5 seconds.

COPY_EXISTING_REGEXP GLOB ~.*\.spl~ ~override~
BUT_ONLY
This processes in 2.5 seconds without the extra files, and 4.5 seconds with the extra files. Thus, 25k files cost 2 seconds; since Arkenor counted 400 files, they cost 0.02 seconds per C_E_R.

On top of that, WeiDU 221 doesn't check that a file isn't a directory until *after* checking that its name matches against the regexp; with that tweak, the processing times go down to 2.7 seconds with the extra files, meaning that the 400 marker files will cost 0.002 per C_E_R.

Times were taken using Bash `time weinstall test --force-install <x>', so it takes in consideration the time needed to read the tlk, chitin and tp2 file. Even if I were to amend that, it wouldn't change the difference between run times I posted.

Italian users: help test the Stivan NPC!

Author or Co-Author: WeiDU - Widescreen - Generalized Biffing - Refinements - TB#Tweaks - IWD2Tweaks - TB#Characters - Traify Tool - Some mods that I won't mention in public
Maintainer: Semi-Multi Clerics - Nalia Mod - Nvidia Fix
Code dumps: Detect custom secondary types - Stutter Investigator

If possible, send diffs, translations and other contributions using Git.


#62 DavidWallace

DavidWallace
  • Validating
  • 337 posts

Posted 12 September 2010 - 02:40 PM

Thanks! - that seems to clear that one up. I'll go on using marker files with a clear conscience.

PS: I don't think I replied to Jarno's comment about TIS files in the override (and Lollorian's rather more civil one about MVE files).

Yeah, you just forgot the biggest files in the game... the .TIS files.


If you look back at my comment, I didn't actually say that the list was exhaustive. The files listed are illustrative, not complete.

The .TIS are in both Tutu and in IwD-in-BG2... and you didn't notice them ?


You do realise I co-wrote IWD-in-BG2?

TIS files actually get copied, edited, and BIFFed pretty much at the start of the IWD-in-BG2 install process, before any WEIDU level file editing happens. They don't go anywhere near the override. The same is true for TUTU and BGT, I believe (in fact, the IWD-in-BG2 code was based on BGT).

Edited by DavidWallace, 12 September 2010 - 02:40 PM.


#63 Miloch

Miloch

    Barbarian

  • Modder
  • 6573 posts

Posted 12 September 2010 - 05:24 PM

(I also prefer detecting other people's mods via markers because then I'm not hostage to their changes of TP2 structure.)

Not to keep nitpicking this technique, but this bugged me a bit. So you'd rather be hostage to a mod's marker files than its (hopefully) hardcoded component numbers? And what if the marker filenames changed with every version increase (as the bigg seemed to be suggesting was the case for some mod earlier)? I'm not sure if there's a good enough argument here for me (or any other modder) to start installing .xxx files with their components if they don't already.

In principle, there's a natural WEIDU change that gets round that: introduce a LABEL command for components. But in practice, why bother? Marker-based reidentification does not, so far as I am aware, cause any problems whatsoever.

As far as I'm aware, neither does component-based identification, except in some really odd situations where marker-based identification could also fail (as in the case I just mentioned). Rather than another WeiDU command, why not just add an optional VERSION flag to MOD_IS_INSTALLED? Meaning that if you needed to detect whether a particular version of a mod were installed (and couldn't do so otherwise) then that would let you.

C_E_R GLOB checks that every file in the override is a file and not a directory before comparing its name against the search regexps. On OCaml, this amounts to a directory read per file in the override (and zero disk operations per file in the chitin), which is the possible cause of the notorious long waits between C_E_R is called and "Copying and Patching XXX files" is printed.

Yes, and you must have a top-of-the line machine if you can go through 25k files in only a couple seconds. As I've mentioned, it's sometimes taken hours to do some of these calls on what couldn't've been more than a hundred thousand files (if that). Granted, I've sometimes been using an external drive or an older machine, but even on a newer machine's internal drive, I've rarely seen a regexp execute on a BWP install in a matter of seconds (usually we're talking minutes at least). But if you say you've improved that in recent WeiDUs, I guess I'll need to test it again. [Edit: obviously I'm aware the complexity of the patch itself can increase processing time, but I'm talking mainly about the time it takes just to get through all the files in the regexp to the initial patch processing point.]

[Edit @thebigg: now why would someone put subfolders in the override? :huh:]

It's possible that badly-coded mods create subfolders in the override (I recall having an empty ~override/override~ subfolder some years ago, for instance).

Obviously, not functional folders... Mods like that should deserve to fail.

Getting back (somewhat) to the OT, there seems to be other rubbish in the override beyond the marker files, which could point to actual mod install issues.

These don't need to be in the override (certainly not after they've done their conversion work):

sox
tisunpack.exe

A bunch of files without an extension, some of which should possibly be .itm or .eff files: things like ion*, ioun*, otthelm, icarmor, icinhit, etc.

A bunch of uncompiled .baf files (prefixes of d0, dw#, j#, zy and others).

A few uncompressed .tiz and .ogg files.

Sundry junk:

tsuj*.aup - Audacity Project Files?
Thumbs.db - Windows junk file for viewing thumbnails; should not exist in any mod or game folder
.DS_Store - OS X junk file?
fhlplay.fh - unknown

These all appear to be component markers of some sort, I'm guessing (though that's hardly clear, given the mishmash of different extensions or lack thereof):

z#*.#
Gavin*.b
cd*.g3
X#*.g3
dw#*.mrk
dw#*.xxx
b!*.rpgd
r#*.rpgd
rr#*.rr
j#*.txt
tb#*.txt
BG1UB*.UB
ub*.xxx


Edited by Miloch, 12 September 2010 - 05:28 PM.

Infinity Engine Contributions
Aurora * BG1 NPC * BG1 Fixpack * Haiass * Infinity Animations * Level 1 NPCs * P5Tweaks
PnP Free Action * Thrown Hammers * Unique Containers * BG:EE * BGII:EE * IWD:EE
================================================================
Player & Modder Resources
BAM Batcher * Creature Lister * Creature Checker * Creature Fixer * Tutu/BGT Area Map & List * Tutu Mod List
================================================================
"Infinity turns out to be the opposite of what people say it is. It is not 'that which has nothing beyond itself' that is infinite, but 'that which always has something beyond itself'." -Aristotle


#64 Jarno Mikkola

Jarno Mikkola

    The Imp in his pink raincoat.

  • Member
  • 10911 posts

Posted 12 September 2010 - 11:23 PM

PS: I don't think I replied to Jarno's comment about TIS files in the override.

Good, now we are on business then...

Yeah, you just forgot the biggest files in the game... the .TIS files.

If you look back at my comment, I didn't actually say that the list was exhaustive. The files listed are illustrative, not complete.

The .TIS are in both Tutu and in IwD-in-BG2... and you didn't notice them ?

You do realise I co-wrote IWD-in-BG2?

TIS files actually get copied, edited, and BIFFed pretty much at the start of the IWD-in-BG2 install process, before any WEIDU level file editing happens. They don't go anywhere near the override. The same is true for TUTU and BGT, I believe (in fact, the IWD-in-BG2 code was based on BGT).

Well, I do now realize how the EasyTutu is(and probably the IwD-in-BG2 are both) made, it's made by first copying the original game(BG1) content, then reindexing the files to the proper sets so the new game/engine can use them... but it also means that the new engine must overwrite or just ignore a whole lot of files because the originals are on the wrong format... which adds to the space bloat and loading time(because you have bigger mass to load the files from). And considering the file and disc fragmentation is bad thing on any game, this just adds to it... I for one don't want to go and manually retreave the 1 byte from the Saturns third moon to calculate the 1+1=? equation. And the originality of the situation was question if one should .bif the game files or not? And since you already have .bifs from some of the files, why not make one from all of them... after all the modifying is done, which is why I would have made the Generalized Biffing(and the End_bif was made), were I skilled enough.

PS, you wrote your own tone on to my own posts above, I didn't ... well, I was just wondering how you forgot that the biggest files were not included, but the bigger issue does explain this. :ermm:

Edited by Jarno Mikkola, 13 September 2010 - 12:40 AM.

Deactivated account. The user today is known as The Imp.


#65 the bigg

the bigg

    2083 is a prime number.

  • Modder
  • 3331 posts

Posted 13 September 2010 - 12:18 AM

Yes, and you must have a top-of-the line machine if you can go through 25k files in only a couple seconds. As I've mentioned, it's sometimes taken hours to do some of these calls on what couldn't've been more than a hundred thousand files (if that). Granted, I've sometimes been using an external drive or an older machine, but even on a newer machine's internal drive, I've rarely seen a regexp execute on a BWP install in a matter of seconds (usually we're talking minutes at least). But if you say you've improved that in recent WeiDUs, I guess I'll need to test it again.


I was on an one year old laptop's internal hard drive (so 5.4k RPM and probably 16 MB of cache). Of course, if you use external drives you can expect lots of additional slowdown. I was measuring on a warm cache and with files created in sequence (so probably less fragmented). I can send the beta if somebody wants to spend a day measuring install times on a large installation (on which gen_biff was removed).

[Edit: obviously I'm aware the complexity of the patch itself can increase processing time, but I'm talking mainly about the time it takes just to get through all the files in the regexp to the initial patch processing point.]

I can't do much about patch time (since it's mostly caused by either the file loading/saving or the patch complexity itself, which is generally outside of my control and isn't affected by the extraneous files we're discussing), but I think I managed to make the pre-processing much faster.

Italian users: help test the Stivan NPC!

Author or Co-Author: WeiDU - Widescreen - Generalized Biffing - Refinements - TB#Tweaks - IWD2Tweaks - TB#Characters - Traify Tool - Some mods that I won't mention in public
Maintainer: Semi-Multi Clerics - Nalia Mod - Nvidia Fix
Code dumps: Detect custom secondary types - Stutter Investigator

If possible, send diffs, translations and other contributions using Git.


#66 DavidWallace

DavidWallace
  • Validating
  • 337 posts

Posted 13 September 2010 - 02:05 AM

(I also prefer detecting other people's mods via markers because then I'm not hostage to their changes of TP2 structure.)

Not to keep nitpicking this technique, but this bugged me a bit. So you'd rather be hostage to a mod's marker files than its (hopefully) hardcoded component numbers? And what if the marker filenames changed with every version increase (as the bigg seemed to be suggesting was the case for some mod earlier)?


I don't think you're nitpicking - it's a perfectly reasonable question, but it shows that I'm not managing to explain what's going on here properly. So here's another go:

Component numbers are not hardcoded, either in theory or (in complicated mods) in practice. The component number structure of SCS and SCSII have changed at least three times in their four-year history (you can tell when this happens, it's the times when I tell people "you need to uninstall and reinstall"). Sometimes that's because of infelicities in my own organisation of components that I later recognise. More often, it's because I want to rearrange the order in which I present components to the player - for instance, I split the tactics part of both mods into "Tactical challenges" and "AI improvements" a year or two ago, which required the order to change; subsequently, I reclassified a few components ("harder spiders", notably) from one section to another, which again changed the order.

Now, WEIDU component numbers need to be presented in order. (Or at least, not doing so is pretty dangerous at best.) So any time you rearrange so that component A occurs before component B rather than after it, necessarily you'll renumber at least one, and possibly both. So you cannot rely on hardcoded numbers to reidentify components on rearrangement.

For most mods, this isn't an issue. Ascension has six components; Wheels of Prophecy has one. But SCSII has about ninety, and it's crucial to organise them in ways that make them accessible and convenient for installers, and sometimes to change that organisation. That means that the only way I can use MOD_IS_INSTALLED is if I commit to changing all those numbers whenever I change the order. I'm very averse to doing that, because it's an absolute recipe for making mistakes.

Here's a slightly more computer-science way of putting it: the marker approach gets the logical structure of the situation right. Suppose that (say) Improved d'Arnise Keep requires Improved General AI to be installed. The marker approach checks, directly, whether that component is installed. The MOD_IS_INSTALLED approach checks something different: whether component 6000 is installed. Any time I change "Improved General AI" so that it is no longer component 6000, that requires a change in the MOD_IS_INSTALLED requirement even though the actual logic of the task hasn't changed at all.

Now, to be sure: component number changing isn't an everyday occurrence, and I try to do it as little as possible (partly to avoid breaking people's installs part-way through, partly out of a desire not to mess up automated install programs). And so if there really was a relevant downside to the marker approach, I'd find another way around (either by faking up my own version of weidu.log, or by just being very careful updating all the various MOD_IS_INSTALLED lines). But since there really isn't any such downside (of which more below), I don't see the need.

As for detecting other people's mods: well, in general I don't trust either MOD_IS_INSTALLED, or other people's markers, and I try to find a more direct way to check. The concrete case I was thinking of was Spell Revisions, where the mod author has a strong interest in achieving SCS compatibility and added the marker specifically for my convenience.

I'm not sure if there's a good enough argument here for me (or any other modder) to start installing .xxx files with their components if they don't already.

There isn't any such argument. If MOD_IS_INSTALLED works for you in your mods, that's fine with me; if I want to check compatibility between your mod and mine and I can't do it in a way that's satisfactory to us both, we can then liaise. (In the case of IR, I had a faithful promise that component 0 would remain component 0 even unto the end of time; that's fine too.)

Picking up more generally on the install-time issues, what I take from the discussion is that there is a slowdown proportional to the number of files in the override (whether they're markers or not). If that slowdown is bothering you, feel free to BIFF the override, markers and all.

The issue of things like .baf files in the override is slightly more interesting, as these aren't harmless if someone does something hugely stupid (i.e., try to COMPILE the override). My own view, frankly, is rather like Miloch's on creating folders in the override: namely, if your mod does that, it deserves to fail. But I'm somewhat more persuadable here if someone can give me a good reason (bearing in mind that fixing it in SCS would (a) take a nontrivial amount of work, and (b) create a nontrivial risk of introducing bugs into SCS itself). In any case, if install speed is what worries you, then again: just BIFF them. Harmless files sitting in your BIF files won't make any material difference to anything (there are more than a few such Bioware files, as I recall).


Getting back (somewhat) to the OT, there seems to be other rubbish in the override beyond the marker files, which could point to actual mod install issues.


The dw#*.mrk and dw#*.xxx are my marker files (I changed from xxx in SCS to mrk in SCSII because xxx was triggering spam filters). Other dw#* files are bits of dialog and script that I build and compile in the override; they're all harmless (unless someone compiles the override) but equally, they don't serve any function.


Well, I do now realize how the EasyTutu is(and probably the IwD-in-BG2 are both) made, it's made by first copying the original game(BG1) content, then reindexing the files to the proper sets so the new game/engine can use them... but it also means that the new engine must overwrite or just ignore a whole lot of files because the originals are on the wrong format... which adds to the space bloat and loading time(because you have bigger mass to load the files from).


This isn't an issue. Neither IWD-in-BG2 (for sure) nor TUTU (I believe) use any engine-accessible file as the destination for their copied-over pre-modification files. And in IWD-in-BG2, at least, things basically only get copied into the override if that's their final destination. Files that are destined to be BIFFed get put in dedicated folders pre-biffing.

And considering the file and disc fragmentation is bad thing on any game, this just adds to it... I for one don't want to go and manually retreave the 1 byte from the Saturns third moon to calculate the 1+1=? equation.


I have no clue what this means.

And the originality of the situation was question if one should .bif the game files or not? And since you already have .bifs from some of the files, why not make one from all of them... after all the modifying is done


"Why not?" is an incredibly bad design principle in writing half-way complicated computer code. It leads to code bloat, confusion of the internal logic, run-time delays, and general chaos. The right question, always, is "why?" - i.e., before you spend time and effort coding some particular outcome, check if that outcome is actually part of what your code is trying to achieve.

(But since this thread is for BWP brainstorming, and IWD-in-BG2 is one of the few mods which is guaranteed-completely-incompatible with BWP, I'd better not digreess on its design much longer!)

#67 Jarno Mikkola

Jarno Mikkola

    The Imp in his pink raincoat.

  • Member
  • 10911 posts

Posted 13 September 2010 - 04:55 AM

And considering the file and disc fragmentation is bad thing on any game, this just adds to it... I for one don't want to go and manually retreave the 1 byte from the Saturns third moon to calculate the 1+1=? equation.

I have no clue what this means.

Let's say someone had a lot of free energy(the perfect world & all that) would have(about 150 years ago) invented to make a computer from water ducts and pipes... they are a little bigger than say microchips. Now then everything goes well for thousands of years, until... there's a leak and the leak is not in the control room, but... the computers design was very ingenious and it cover the whole solar system. So then, when someone notices the leak on the pipes on the Saturn third moon, the mister fix-it needs to go and fix it... to make even the simplest calculations correct.

"Why not?" is an incredibly bad design principle in writing half-way complicated computer code. It leads to code bloat, confusion of the internal logic, run-time delays, and general chaos. The right question, always, is "why?" - i.e., before you spend time and effort coding some particular outcome, check if that outcome is actually part of what your code is trying to achieve.

Yeah, as does copying unneeded files that already have bloat in them... the main difference is that in my approach the end product is the one that's THE END product, that the BiG World installs the Generalized Biffing after all the files have already been modified and they are in their final composition, so their backups, the mid files and the marker files can all be thrown away as they aren't needed anymore, because the modifying those files is done(unless you wish to start from the beginning), in which case it's faster to delete the whole game directory, and unrar the clean backup you could have made 10 years ago.

Edited by Jarno Mikkola, 13 September 2010 - 05:41 AM.

Deactivated account. The user today is known as The Imp.


#68 the bigg

the bigg

    2083 is a prime number.

  • Modder
  • 3331 posts

Posted 13 September 2010 - 05:09 AM

Let's say someone had a lot of free energy(the perfect world & all that) would have(about 150 years ago) invented to make a computer from water ducts and pipes... they are a little bigger than say microchips. Now then everything goes well for thousands of years, until there's a leek and there was a water leek... the computers design was very engenious and it cover the whole solar system. So then, when someone notices the leek on the pipes on the saturns third moon, the mister fix needs to go and fix it.

Posted Image

Italian users: help test the Stivan NPC!

Author or Co-Author: WeiDU - Widescreen - Generalized Biffing - Refinements - TB#Tweaks - IWD2Tweaks - TB#Characters - Traify Tool - Some mods that I won't mention in public
Maintainer: Semi-Multi Clerics - Nalia Mod - Nvidia Fix
Code dumps: Detect custom secondary types - Stutter Investigator

If possible, send diffs, translations and other contributions using Git.


#69 DavidWallace

DavidWallace
  • Validating
  • 337 posts

Posted 13 September 2010 - 05:23 AM

Jarno, I'm largely no longer able to understand what you're saying, but insofar as I can understand it, it seems now to be related to worries about unnecessary bloat in the install structure of IWD-in-BG2. I don't really want to derail this thread by discussing this further, but I'd welcome any such comments at the appropriate forum.

#70 Jarno Mikkola

Jarno Mikkola

    The Imp in his pink raincoat.

  • Member
  • 10911 posts

Posted 13 September 2010 - 06:29 AM

Jarno, I'm largely no longer able to understand what you're saying, but insofar as I can understand it, it seems now to be related to worries about unnecessary bloat in the install structure...

So far you are doing fine...
And... here's:

...of IWD-in-BG2.

...where you miss, it's not just about your unrelated mod, but the whole BG2 moding, and how the files end up being in the computer that have them. As you can probably conclude from the first post in this... the focus is that the unneeded files aren't in the .bif file, as it saves space, install and loading time. In installs where you have about 1300 individual component installs, it all begins to show up.

Deactivated account. The user today is known as The Imp.


#71 Wisp

Wisp
  • Modder
  • 1353 posts

Posted 13 September 2010 - 06:38 AM

[...]as it saves space, install and loading time. In installs where you have about 1300 individual component installs, it all begins to show up.

Didn't Arkenor say it amounted to 400 files and 7 megabytes? If that's what you're referring to I think you're exaggerating the problem.

#72 the bigg

the bigg

    2083 is a prime number.

  • Modder
  • 3331 posts

Posted 13 September 2010 - 06:47 AM

Bah, while am I wasting time writing researched answers to a village idiot's claims?

space

Files in the biffs take the same space than if they were override, since the files are still on the disk. Besides, assuming almost every one of those 1300 components drops two marker files in the override, 2500 files * 4 kilobyte each (if we count the space wasted by the file system) = 10 MB (out of more than 20 GB taken by the full BWP install). The 'wasted' space is unnoticeable, and fixing currently unnoticeable problems is a recipe for disaster.

install time

2500 files will cost 0.02 per COPY_EXISTING_REGEXP with WeiDU 221. Even if BWP ran 10k of those, it amounts to 200 seconds = 3 extra minutes per a full BWP run, while the full process takes a dozen of hours. Again, unnoticeable.

loading time

Loading times are completely unrelated to the number of files in the override, it's related to the space occupied by the files in the override (and not all the files in the override, just some file types like .BAM, .TIS and .WAV), so having even 10 GB of .txt files in the override won't affect loading times in the slightest.

Edited by the bigg, 13 September 2010 - 06:49 AM.

Italian users: help test the Stivan NPC!

Author or Co-Author: WeiDU - Widescreen - Generalized Biffing - Refinements - TB#Tweaks - IWD2Tweaks - TB#Characters - Traify Tool - Some mods that I won't mention in public
Maintainer: Semi-Multi Clerics - Nalia Mod - Nvidia Fix
Code dumps: Detect custom secondary types - Stutter Investigator

If possible, send diffs, translations and other contributions using Git.


#73 Hoppy

Hoppy

    Mage Hunter

  • Member
  • 2107 posts

Posted 13 September 2010 - 06:51 AM


[...]as it saves space, install and loading time. In installs where you have about 1300 individual component installs, it all begins to show up.

Didn't Arkenor say it amounted to 400 files and 7 megabytes? If that's what you're referring to I think you're exaggerating the problem.


Yeah that is heavy exaggerating. Long load times can also be attributed to the size of the save file as well when you have tons of data in those save files like areas and other necessary items from large installs.
?May God defend me from my friends; I can defend myself from my enemies.? - Voltaire

"If you think that a size of the mod indicates an amount of bugs that it introduces and their severity you're totally wrong...
Try not to use next time a load of shitty "super-mega-improving-tweaking-revising" small mods that you have installed and try to meet Wulfgar once again."
- King Diamond


Posted Image The Definitive Guide to Trolls

"Finding food and a place to sleep is your own business. I imagine Paul the Cat should have some fun with you, too" - Potencius in The Darkest Day
"You have been warned, little bastard!" -Khelben to a young <CHARNAME>in Check the Bodies
There are those who will snivel, and offer nothing in return except criticism, meanwhile never lifting a finger to do other than to cut other peoples labor down simply for the fact that they lack the capability to put anything of their own together. -erebusant

#74 Miloch

Miloch

    Barbarian

  • Modder
  • 6573 posts

Posted 13 September 2010 - 01:37 PM

That means that the only way I can use MOD_IS_INSTALLED is if I commit to changing all those numbers whenever I change the order. I'm very averse to doing that, because it's an absolute recipe for making mistakes.

Ok, well I think there's a simple solution to this apart from using markers - basically (I think) what you suggested for LABEL:
ACTION_IF MOD_IS_INSTALLED ~setup-scs.tp2~ ~Harder Spiders~ BEGIN ...
In other words, MOD_IS_INSTALLED can take a DESIGNATED component # as well as its textual reference (parsing the main .tra file if necessary). Is that doable, the bigg?

(I changed from xxx in SCS to mrk in SCSII because xxx was triggering spam filters).

(Heh.)

Other dw#* files are bits of dialog and script that I build and compile in the override; they're all harmless (unless someone compiles the override) but equally, they don't serve any function.

Any reason, apart from a potential non-trivial amount of work, not to build those elsewhere and then compile them to the override?

And considering the file and disc fragmentation is bad thing on any game, this just adds to it... I for one don't want to go and manually retreave the 1 byte from the Saturns third moon to calculate the 1+1=? equation.

I have no clue what this means.

Not speaking for Jarno again, but he might* be referring to the fact that on some disk systems, even a single-byte file takes up an entire block of disk space, which could be 4MB for example, allowing no other files in that block. This is true for FAT/FAT32 anyway - not sure if it is for uncompressed NTFS. But yes, the presence of many small files can lead to disk fragmentation (a problem which goes away somewhat with biffing, I suppose, though I have to say I'm still rather adverse to biffing files the game does not use).

* This is giving him a huge benefit of doubt, given what he subsequently wrote...

Infinity Engine Contributions
Aurora * BG1 NPC * BG1 Fixpack * Haiass * Infinity Animations * Level 1 NPCs * P5Tweaks
PnP Free Action * Thrown Hammers * Unique Containers * BG:EE * BGII:EE * IWD:EE
================================================================
Player & Modder Resources
BAM Batcher * Creature Lister * Creature Checker * Creature Fixer * Tutu/BGT Area Map & List * Tutu Mod List
================================================================
"Infinity turns out to be the opposite of what people say it is. It is not 'that which has nothing beyond itself' that is infinite, but 'that which always has something beyond itself'." -Aristotle


#75 the bigg

the bigg

    2083 is a prime number.

  • Modder
  • 3331 posts

Posted 13 September 2010 - 01:52 PM

ACTION_IF MOD_IS_INSTALLED ~setup-scs.tp2~ ~Harder Spiders~ BEGIN ...[/code]In other words, MOD_IS_INSTALLED can take a DESIGNATED component # as well as its textual reference (parsing the main .tra file if necessary). Is that doable, the bigg?

It's possible, but it looks like over-engineering over FILE_EXISTS_IN_GAME ~dw#spiders.mrk~. Again, unless you can prove that 20k resource files aren't a problem but 400 non-resource files are, the point is moot.

Not speaking for Jarno again, but he might* be referring to the fact that on some disk systems, even a single-byte file takes up an entire block of disk space, which could be 4MB for example, allowing no other files in that block. This is true for FAT/FAT32 anyway - not sure if it is for uncompressed NTFS.

4kb per file, actually. If it were 4MB, it'd be an issue that needs fixing, and not just a case of people blindly following the Must Be Biffed mantra.

But yes, the presence of many small files can lead to disk fragmentation (a problem which goes away somewhat with biffing, I suppose, though I have to say I'm still rather adverse to biffing files the game does not use).

You can't biff non-resources, actually. The key extension isn't represented as a string in the key file, but rather as an index in an array, thus you can't put whatever.mrk or something.ogg in a biff even if you wanted to.

Italian users: help test the Stivan NPC!

Author or Co-Author: WeiDU - Widescreen - Generalized Biffing - Refinements - TB#Tweaks - IWD2Tweaks - TB#Characters - Traify Tool - Some mods that I won't mention in public
Maintainer: Semi-Multi Clerics - Nalia Mod - Nvidia Fix
Code dumps: Detect custom secondary types - Stutter Investigator

If possible, send diffs, translations and other contributions using Git.


#76 Chevalier

Chevalier

    Knight of the Realms

  • Modder
  • 2405 posts

Posted 13 September 2010 - 01:57 PM

I know there are differences of opinions here but thank you for trying to make the game better!!!!! :clap: :cheers: :clap:

I Ride for the King!


a.k.a. Chev


#77 Miloch

Miloch

    Barbarian

  • Modder
  • 6573 posts

Posted 13 September 2010 - 02:44 PM


ACTION_IF MOD_IS_INSTALLED ~setup-scs.tp2~ ~Harder Spiders~ BEGIN ... In other words, MOD_IS_INSTALLED can take a DESIGNATED component # as well as its textual reference (parsing the main .tra file if necessary). Is that doable, the bigg?

It's possible, but it looks like over-engineering over FILE_EXISTS_IN_GAME ~dw#spiders.mrk~.

So it could be done, right? I sense a certain resistance to change on the part of you and DavidW, understandable to some extent since your mods are already coded with markers, but that shouldn't stop other modders from using a new cruftless technique going forward.

Again, unless you can prove that 20k resource files aren't a problem but 400 non-resource files are, the point is moot.

Ah, but the 20k (or 50k) resource files are a problem. Just an unavoidable one if you want 5000 mods in your game. The 400 non-resource files are just a relatively small (but avoidable) part of that problem. And if, as you say, you can't biff them, they don't even have that benefit that resource files do. Biffed files do take up less diskspace due to the file allocation block issue (even if you've got 4kb blocks and 3kb files).

Infinity Animations, btw, has 45+ components, and there are rare times when I've had to change the DESIGNATED numbers due to what DavidW described. So I'm trying to plan for the future, so to speak. Given that it already installs 1000s of game resource files, I'd like to avoid installing non-resource files, however proportionally small that number may be.

Infinity Engine Contributions
Aurora * BG1 NPC * BG1 Fixpack * Haiass * Infinity Animations * Level 1 NPCs * P5Tweaks
PnP Free Action * Thrown Hammers * Unique Containers * BG:EE * BGII:EE * IWD:EE
================================================================
Player & Modder Resources
BAM Batcher * Creature Lister * Creature Checker * Creature Fixer * Tutu/BGT Area Map & List * Tutu Mod List
================================================================
"Infinity turns out to be the opposite of what people say it is. It is not 'that which has nothing beyond itself' that is infinite, but 'that which always has something beyond itself'." -Aristotle


#78 the bigg

the bigg

    2083 is a prime number.

  • Modder
  • 3331 posts

Posted 13 September 2010 - 02:58 PM

So it could be done, right? I sense a certain resistance to change on the part of you and DavidW, understandable to some extent since your mods are already coded with markers, but that shouldn't stop other modders from using a new cruftless technique going forward.

DW is resistant to change because he is afraid of introducing bugs (a commendable way to act, IMHO). I'm resistant to change because I have to implement it (and I'd rather implement mods that makes the game better and/or make it easier to write mods, rather than optimize 1% of a problem).

Italian users: help test the Stivan NPC!

Author or Co-Author: WeiDU - Widescreen - Generalized Biffing - Refinements - TB#Tweaks - IWD2Tweaks - TB#Characters - Traify Tool - Some mods that I won't mention in public
Maintainer: Semi-Multi Clerics - Nalia Mod - Nvidia Fix
Code dumps: Detect custom secondary types - Stutter Investigator

If possible, send diffs, translations and other contributions using Git.


#79 DavidWallace

DavidWallace
  • Validating
  • 337 posts

Posted 13 September 2010 - 03:08 PM

Other dw#* files are bits of dialog and script that I build and compile in the override; they're all harmless (unless someone compiles the override) but equally, they don't serve any function.

Any reason, apart from a potential non-trivial amount of work, not to build those elsewhere and then compile them to the override?


No, none at all. If I were to rebuild SCS(II) from scratch, knowing what I know now and having WEIDU v221, I'd probably create a new directory, scs_workspace, separate from the scsii file tree, and do all my compilation, SSL construction, markers etc in there. My interest here is more in whether there's any problem with the existing form. (Bear in mind that SCS was written in ~2004, and my example-of-coding was Tactics, which is two or three years before that.)



ACTION_IF MOD_IS_INSTALLED ~setup-scs.tp2~ ~Harder Spiders~ BEGIN ... In other words, MOD_IS_INSTALLED can take a DESIGNATED component # as well as its textual reference (parsing the main .tra file if necessary). Is that doable, the bigg?

It's possible, but it looks like over-engineering over FILE_EXISTS_IN_GAME ~dw#spiders.mrk~.

So it could be done, right? I sense a certain resistance to change on the part of you and DavidW, understandable to some extent since your mods are already coded with markers, but that shouldn't stop other modders from using a new cruftless technique going forward.

That pretty much nails it, actually. My defence of SCS's coding is very much ain't-broke-don't-fix. I'm not necessarily advocating this method to other people. (If ACTION_IF MOD_IS_INSTALLED ~setup-scs.tp2~ ~Harder Spiders~ does actually work, I'd probably use it in future mods. (The WEIDU documentation (dating back to Weimer; this is well before TheBigg took over) implies otherwise.) But what I don't want to do is to rearrange something like this without a positive reason - my experience over the last four years is that it's a good way to generate bugs. (I'm still feeling a bit burned by introducing ADD_CRE_EFFECT for Faster Bears.)

Edited by DavidWallace, 13 September 2010 - 03:09 PM.


#80 the bigg

the bigg

    2083 is a prime number.

  • Modder
  • 3331 posts

Posted 13 September 2010 - 03:18 PM

(The WEIDU documentation (dating back to Weimer; this is well before TheBigg took over) implies otherwise.)

Actually, since you tried to nitpick, MOD_IS_INSTALLED is a BiggFeature, not a WeiFeature (and the absence of MII is what used to make adding marker files proactively a good coding practice).

MII doesn't support component names currently, but might do so in 221 (if somebody can think of a credible way to handle translations).

Italian users: help test the Stivan NPC!

Author or Co-Author: WeiDU - Widescreen - Generalized Biffing - Refinements - TB#Tweaks - IWD2Tweaks - TB#Characters - Traify Tool - Some mods that I won't mention in public
Maintainer: Semi-Multi Clerics - Nalia Mod - Nvidia Fix
Code dumps: Detect custom secondary types - Stutter Investigator

If possible, send diffs, translations and other contributions using Git.