141 replies to this topic

[Jarno Mikkola]

Someone should really rewrite those tp2s so they use stuff like ACTION_IF NOT MOD_IS_INSTALLED ~setup-mod.tp2~ 666 instead of ACTION_IF NOT FILE_EXISTS ~override\marker.txt~. ...

Some TP2s use file markers to keep track of version number or something like that (that's what AoE uses that file marker for, by instance). Also, some pieces of code are installed by about 748930215671203497 different mods, and it's beneficial to have them use file markers so you can FILE_EXISTS ~banter_accelerator~ rather than writing dozens of (MOD_IS_INSTALLED a.tp2 0) OR (MOD_IS_INSTALLED b.tp2 1) OR ...

But the question is, is the cheesed override folder the best place to put the marker on... like I wrote on the above, or at least tried...

Unless you mean that you have to make the effort to not compile the override directory. In that case: sure, I don't accommodate that, but I don't accommodate other ways in which people might make elementary mistakes in WEIDU.

If there isn't anything to compile in the override folder, then one can do it without the care if there was an error in a mod code that results in the compiling the non-files in the folder.

I haven't yet had an explanation of what those "extra efforts" are.

The effort of removing uncompiled BAF files and marker files from the override. Remember, in BiG World, the override must be cleaned because the override must be cleaned!

The whole idea of the clean up is that the override folder is unnecessary after the Generalized Biffing AND the optional clean up is done, the game files are in the .bif's and the optionality of the clean up process makes sure that people do not want to modify the game anymore... so it saves space. But then again there shouldn't be any files there.
Yes, the BWS shouldn't automatically use that clean up process, but then again I have said that the "Further customization of your mod-setting" option should be selected by default, and they haven't corrected that either...

Edited by Jarno Mikkola, 10 September 2010 - 12:18 PM.

[DavidWallace]

Someone should really rewrite those tp2s so they use stuff like ACTION_IF NOT MOD_IS_INSTALLED ~setup-mod.tp2~ 666 instead of ACTION_IF NOT FILE_EXISTS ~override\marker.txt~. ...

Some TP2s use file markers to keep track of version number or something like that (that's what AoE uses that file marker for, by instance). Also, some pieces of code are installed by about 748930215671203497 different mods, and it's beneficial to have them use file markers so you can FILE_EXISTS ~banter_accelerator~ rather than writing dozens of (MOD_IS_INSTALLED a.tp2 0) OR (MOD_IS_INSTALLED b.tp2 1) OR ...

But the question is, is the cheesed override folder the best place to put the marker on... like I wrote on the above, or at least tried...

Arguably, yes. Certainly it's a harmless place to put them. As I note above, in some way it's better than a folder in the mod's own file structure.

Unless you mean that you have to make the effort to not compile the override directory. In that case: sure, I don't accommodate that, but I don't accommodate other ways in which people might make elementary mistakes in WEIDU.

If there isn't anything to compile in the override folder, then one can do it without the care if there was an error in a mod code that results in the compiling the non-files in the folder.

Errors that severe (giving a blanket COMPILE command to a space outside your control, because you don't understand a really elementary bit of WEIDU syntax) should be fixed. I've no sympathy with the idea they should be somehow worked around instead.

I haven't yet had an explanation of what those "extra efforts" are.

The effort of removing uncompiled BAF files and marker files from the override. Remember, in BiG World, the override must be cleaned because the override must be cleaned!

The whole idea of the clean up is that the override folder is unnecessary after the Generalized Biffing AND the optional clean up is done, the game files are in the .bif's and the optionality of the clean up process makes sure that people do not want to modify the game anymore... so it saves space.

On your hard drive?! These are 300-byte files. You can fit three million of them into any given gigabyte of hard drive space. Anyway, the markers have to be somewhere, so you don't actually save any space. It's not as if the override directory itself has a finite volume.

But then again there shouldn't be any files there.

That's question-begging. That's what you're trying to deduce, so just stating it isn't helpful.

[dabus]

Omg! The kungfu in this thread is strong

Let's all just ask the good people at the BWP to just separate the cleanup procedure from the default installer

This is a good idea; however, I have 5$ that say that the choice will never be offered (I mean, in this very thread dabus has declared that 'Further Customization of mod-Setting' shouldn't be checked by default because it offers players too many choices).

What's the context between those requests?
To me, it's zero.

I'm the "BWP Batch to GUI with more options"-guy. So I don't decide if something will be in the batch.
And in most cases I just take that source/batch and try to convert it into the GUI.
Just because I don't do modding nor know or want to know as much about that stuff.

However, anyone is free to either click once to see additional options or not.
It's disabled because it's optional and the request for the GUI was to start an installation with as few clicks as possible (also considering guys new to BG modding). I just don't get why clicking onto some checkbox cause such an fuzz, just like you have during most software-installations.

Btw: If I add that option for the empty_override to the beta, would you pay that bucks to SHS to help them pay their bills?

Edited by dabus, 10 September 2010 - 02:05 PM.

[Jarno Mikkola]

On your hard drive?! These are 300-byte files. You can fit three million of them into any given gigabyte of hard drive space. Anyway, the markers have to be somewhere, so you don't actually save any space. It's not as if the override directory itself has a finite volume.

Well, for some players, the extra space in their hard drive is zero after the installation(or the actual BiG World Setup, customization and completion) is fully complete... as they just wish to play the game, not install the 100 other mods, because if they do that, then they need to restart the whole thing from the beginning. And that's where the optional clean up comes to the play.
And if the markers need to be somewhere, put them into your mod folder... not in the override folder, in a form that can botch the whole game in one nice sweep... that is if you wish the whole game still be playable after your mod is installed.
(yeah, the LStest mod does indeed botch the game, but it's a test mod... yeah, a stupid excuse, but perhaps a valid one as it has it's uses on otherwise already broken game)

But then again there shouldn't be any files there.

That's question-begging. That's what you're trying to deduce, so just stating it isn't helpful.

Well, there wasn't any files in the original override folder that couldn't have been .bif-ed that weren't broken.

And you know what, the whole mod folders can be removed from the game directory after the files in it are installed, .biffed, in the optional clean up. That's 10Gb's free space, easily, unless the install already does that(which it shouldn't) and that includes your three million 300 bit files . As everything will be in the data folder, and .bif's are faster to read too... ok, so the config tools(.ini files etc.) aren't.

[DavidWallace]

@Jarno:

If you are seriously claiming that having one (or even a few dozen) 300-byte marker file(s) in your override directory can either (a) make the slightest amount of relevant difference to your hard drive capacity, or (b) that it's possible to break the game by having such files in place, then either you are being wilfully annoying or you have no idea what you are talking about; I don't think I have the patience to try to work out which one.

[the bigg]

Btw: If I add that option for the empty_override to the beta, would you pay that bucks to SHS to help them pay their bills?

You're on.

[10th]

@Jarno
Those marker files are exactly in the override, because there is the most convenient and practical place to look for them. What if it was done as you suggested? Take SCS II as an example, every marker file in the mod folder, what if DavidW rearranges the folders or renames them? Suddenly every mod and dog which wants to be compatible with SCS II has to receive an update, just because an insignifcant change was made.

@biffing the whole override folder
Unecessary. I'm doing just fine with ~23000 files in the override and only biffing the unneeded ones. Some people have even more and they're not experiencing any serious slowdowns. Most slowdowns are probably due to sloppy coding/too many talking swords/mods which aim to do global changes by cluttering up baldur/25.bcs.

10th

Edited by 10th, 10 September 2010 - 04:23 PM.

[Miloch]

If you are seriously claiming that having one (or even a few dozen) 300-byte marker file(s) in your override directory can either (a) make the slightest amount of relevant difference to your hard drive capacity, or (b) that it's possible to break the game by having such files in place, then either you are being wilfully annoying or you have no idea what you are talking about; I don't think I have the patience to try to work out which one.

Well I can't speak for Jarno (and wouldn't want to ) but it seems to me, the fewer files you can get away with in the override, the better, particularly when doing anything with regexp. Not sure if the mere number of files in the override affects game performance, but I suppose that could be a concern. Still not sure how it's an improvement over MOD_IS_INSTALLED either. If you've moved functionality from one component to another, that alone questions whether any type of component check is valid any longer. If a component doesn't already install a recognisable native game-type file (such as a .cre, .itm., .spl etc.) or change to such, then WeiDU should be able to detect whether that component is installed without resorting to cruft. What with the VERSION command, it should be able to detect that too.

(All this detracts from the topic I suppose, but blame dabus for bringing it up .)

[Edit: I should add that anything that deletes files from an install, particularly if it doesn't really know what its doing and/or uses non-WeiDU techniques to do so, is probably a bad idea. The BWP should at least log such actions to WeiDU.log. Possibly MOVE might be marginally better than deleting them, since at least then they could be restored in theory.]

Edited by Miloch, 10 September 2010 - 06:54 PM.

[DavidWallace]

If you are seriously claiming that having one (or even a few dozen) 300-byte marker file(s) in your override directory can either (a) make the slightest amount of relevant difference to your hard drive capacity, or (b) that it's possible to break the game by having such files in place, then either you are being wilfully annoying or you have no idea what you are talking about; I don't think I have the patience to try to work out which one.

Well I can't speak for Jarno (and wouldn't want to ) but it seems to me, the fewer files you can get away with in the override, the better, particularly when doing anything with regexp.

What slightly bothers me is that it seems to be an article of faith that "the fewer files you can get away with in the override, the better". Whereas I haven't actually seen any evidence at all that a relatively small number of marker files, or uncompiled BAF files for that matter, do anything negative whatsoever. (If by "doing anything with regexp", you mean, "don't COPY all the CRE files into the override just because you're doing a REGEXP change to six of them, that's different, although even then I think it's a matter of space cost, install/uninstall time, and (frankly) aesthetics).

Not sure if the mere number of files in the override affects game performance, but I suppose that could be a concern.

As it happens, I have experimented reasonably extensively with override efficiency, both in trying to optimise SCS script runtime and in writing IWD-in-BG2. My conclusion is that you can have thousands of long scripts in the override without getting any performance hit at all relative to BIFFing them, and that you don't get any speedup by BIFFing the whole of TUTU (which is entirely in-override but consists basically almost entirely of CRE, BCS, DLG, ITM and ARE files), but that having lots of WAV or BAM files in the override does give a major performance hit (IWD-in-BG2 is unplayable without BIFFing). But in any case, override vs BIFF isn't the issue. You're welcome to BIFF SCS's marker files: I detect them (as I recall) by FILE_EXISTS_IN_GAME.

Still not sure how it's an improvement over MOD_IS_INSTALLED either. If you've moved functionality from one component to another, that alone questions whether any type of component check is valid any longer. If a component doesn't already install a recognisable native game-type file (such as a .cre, .itm., .spl etc.) or change to such, then WeiDU should be able to detect whether that component is installed without resorting to cruft. What with the VERSION command, it should be able to detect that too.

Well, three observations:

(i) if I move a component of SCS from, say, the Item Tweaks to the Gameplay Tweaks, or the Tactical Challenges to the Improved AI, the component number unavoidably changes.
(ii) checking for a recognisable file confuses the logical structure of what's going on, isn't always possible, risks breaking if you tweak the component, and only has an advantage if you already accept that there's some disadvantage of using marker files.
(iii) likewise, I haven't actually heard an explanation here for why using MOD_IS_INSTALLED is particularly advantageous. There is an odd, almost aesthetic dislike for putting small numbers of harmless things in the override directory which frankly puzzles me.

[Jarno Mikkola]

@biffing the whole override folder
Unecessary. I'm doing just fine with ~23000 files in the override and only biffing the unneeded ones. Some people have even more and they're not experiencing any serious slowdowns. Most slowdowns are probably due to sloppy coding/too many talking swords/mods which aim to do global changes by cluttering up baldur/25.bcs.

Have you tried to actually start the game ? With big files...

As it happens, I have experimented reasonably extensively with override efficiency, both in trying to optimise SCS script runtime and in writing IWD-in-BG2. My conclusion is that you can have thousands of long scripts in the override without getting any performance hit at all relative to BIFFing them, and that you don't get any speedup by BIFFing the whole of TUTU (which is entirely in-override but consists basically almost entirely of CRE, BCS, DLG, ITM and ARE files), but that having lots of WAV or BAM files in the override does give a major performance hit (IWD-in-BG2 is unplayable without BIFFing). But in any case, override vs BIFF isn't the issue.

Yeah, you just forgot the biggest files in the game... the .TIS files.
The .TIS are in both Tutu and in IwD-in-BG2... and you didn't notice them ?
You should. The biggest notable issue is that without any kind of biffing(in a megamod game)... the game starting(the time between your display starting to show the begin movie and you clicking the quick icon from BGMain.exe in Windows) can be... 10 to 20 minutes, but with biffing, it takes less than 5 seconds, just like I wrote in my FAQs for the Megamod.

[the bigg]

The .TIS are in both Tutu and in IwD-in-BG2... and you didn't notice them ?

Tutu doesn't place TIS files in the override. Will you ever stop talking out of your ass?

Re: COPY_EXISTING_REGEXP GLOB and number of files, I could make it faster, but then C_E_R G will stop working the instant somebody creates a subfolder in the override that matches against the regexp.

[Lollorian]

Is there a reason this discussion went from "why-the-hell-you-deletin-ma-markers" to "why-the-hell-you-biffin-the-entire-override-yo"??

... almost entirely of CRE, BCS, DLG, ITM and ARE files), but that having lots of WAV or BAM files in the override does give a major performance hit (IWD-in-BG2 is unplayable without BIFFing)

Yeah, you just forgot the biggest files in the game... the .TIS files.

Sorry for the half-quote but I'll make a mention of MVE files while we're at it They NEED to be biffed. Period!

Only mods that add an MVE file are CtB, TS-BP and the JZ show .... which already biff them But they're still evil if left in the override!

Edited by Lollorian, 11 September 2010 - 01:38 AM.

[10th]

Have you tried to actually start the game ? With big files... Yeah, you just forgot the biggest files in the game... the .TIS files.
The biggest notable issue is that without any kind of biffing(in a megamod game)... the game starting(the time between your display starting to show the begin movie and you clicking the quick icon from BGMain.exe in Windows) can be... 10 to 20 minutes, but with biffing, it takes less than 5 seconds, just like I wrote in my FAQs for the Megamod.

I'm playing a very customised BWP-install at the moment, using
0] only wav&tis&bam (recommended by The Bigg and other Small World Dudes)
and I'm doing just fine. Never had any startup problems.

10th

[Jarno Mikkola]

Tutu doesn't place TIS files in the override. Will you ever stop talking out of your ass?

Ouh yeah, the data is retrieved from the copied BG1 .bif files, curiously they don't have to be rebiffed... hmm I wonder why. No, I really don't.

[Miloch]

(If by "doing anything with regexp", you mean, "don't COPY all the CRE files into the override just because you're doing a REGEXP change to six of them, that's different, although even then I think it's a matter of space cost, install/uninstall time, and (frankly) aesthetics).

I meant, largely, install/uninstall time. By the time SCS2 (for example) comes along in a BWP-type install, it can take ages to install (depending on your computer speed) due to all the regexp. Same goes for the likes of D0Tweaks and Aurora (I've tried optimising the latter but that only goes so far). Not saying you should avoid regexp (or even markers if you truly feel them necessary) - just saying empirically it takes longer the more files you have in the override (possibly even if they're biffed and you're using GLOB though I haven't tested that extensively).

(iii) likewise, I haven't actually heard an explanation here for why using MOD_IS_INSTALLED is particularly advantageous. There is an odd, almost aesthetic dislike for putting small numbers of harmless things in the override directory which frankly puzzles me.

If it's one mod putting small bits of cruft in the override, that's one thing. But if it's 10 or 50 mods, it can amount to mega-cruft. "Mostly harmless," perhaps, in the words of Douglas Adams, but it's the "mostly" that's the operative word. If there's an alternative method that avoids the cruft, why not use it, unless it's truly impractical? I've found MOD_IS_INSTALLED to be sufficient for everything I do, but no doubt my Wei-Fu isn't on the same par as yours.

[Edit @thebigg: now why would someone put subfolders in the override? )

Edited by Miloch, 11 September 2010 - 06:13 PM.

[DavidWallace]

(If by "doing anything with regexp", you mean, "don't COPY all the CRE files into the override just because you're doing a REGEXP change to six of them, that's different, although even then I think it's a matter of space cost, install/uninstall time, and (frankly) aesthetics).

I meant, largely, install/uninstall time. By the time SCS2 (for example) comes along in a BWP-type install, it can take ages to install (depending on your computer speed) due to all the regexp. Same goes for the likes of D0Tweaks and Aurora (I've tried optimising the latter but that only goes so far). Not saying you should avoid regexp (or even markers if you truly feel them necessary) - just saying empirically it takes longer the more files you have in the override (possibly even if they're biffed and you're using GLOB though I haven't tested that extensively).

I'd be fairly surprised if REGEXPs take longer when files are in the override rather than BIFFed - if anything I'd expect the opposite, since WEIDU has to decompress them. But TheBigg can comment more on this.

As far as I know, all of SCS's REGEXPs are basically unavoidable - the whole idea is to alter every creature having property XYZ. (In some earlier versions that wasn't the case, and I was just using REGEXP to save time, but I think now they're all required.)

(iii) likewise, I haven't actually heard an explanation here for why using MOD_IS_INSTALLED is particularly advantageous. There is an odd, almost aesthetic dislike for putting small numbers of harmless things in the override directory which frankly puzzles me.

If it's one mod putting small bits of cruft in the override, that's one thing. But if it's 10 or 50 mods, it can amount to mega-cruft.

Well, even if it was 10 or 50 mods, that would only be a few hundred 300-byte files that the engine ignores, so I still don't see the problem. But in fact, it isn't: very few mods use markers (the only other one I can think of is BG2Tweaks, which not by coincidence is the only other mod I can think of with a component structure as complicated as SCS has.)

"Mostly harmless," perhaps, in the words of Douglas Adams, but it's the "mostly" that's the operative word.

I'm still holding out for "completely harmless".

If there's an alternative method that avoids the cruft, why not use it, unless it's truly impractical? I've found MOD_IS_INSTALLED to be sufficient for everything I do, but no doubt my Wei-Fu isn't on the same par as yours.

Well, as I say, I don't really find it adequate for what I do, because it's not robust against changes to the ordering and numbering of components. I want to be able to tell other components "don't install unless Smarter Mages is installed" without having to worry about changing lots of lines of code if I change the component number of Smarter Mages. (I also prefer detecting other people's mods via markers because then I'm not hostage to their changes of TP2 structure.)

In principle, there's a natural WEIDU change that gets round that: introduce a LABEL command for components. But in practice, why bother? Marker-based reidentification does not, so far as I am aware, cause any problems whatsoever.

[Miloch]

I'd be fairly surprised if REGEXPs take longer when files are in the override rather than BIFFed - if anything I'd expect the opposite, since WEIDU has to decompress them.

I'd imagine it's roughly the same either way, but I guess what I was saying is that the more files there are in the game (whether they be in the override, portraits, sounds or biffed), the longer regexp operations take. And regexp, unfortunately, does not ignore marker files unless you explicitly tell it to (possibly not even then, as far as the time it takes to determine which files it needs to include/exclude). I think it took over a day to install SCS2 and D0Tweaks on an external hard drive with BWP last time, though using an internal drive shortened this to "only" several hours.

very few mods use markers (the only other one I can think of is BG2Tweaks, which not by coincidence is the only other mod I can think of with a component structure as complicated as SCS has.)

I think a number of the older mods on G3 do (such as DR) unless someone has been updating those to avoid the markers. Ashes of Embers, maybe other older ones on PPG as well.

[the bigg]

C_E_R GLOB checks that every file in the override is a file and not a directory before comparing its name against the search regexps. On OCaml, this amounts to a directory read per file in the override (and zero disk operations per file in the chitin), which is the possible cause of the notorious long waits between C_E_R is called and "Copying and Patching XXX files" is printed.

[Edit @thebigg: now why would someone put subfolders in the override? )

It's possible that badly-coded mods create subfolders in the override (I recall having an empty ~override/override~ subfolder some years ago, for instance).

[Wisp]

but I guess what I was saying is that the more files there are in the game (whether they be in the override, portraits, sounds or biffed), the longer regexp operations take. s. Ashes of Embers, maybe other older ones on PPG as well.

But unless marker files comprise a significant portion of the total number of files, you won't notice much of a difference even if you were to get rid of them. I would hazard a guess and say that marker files are probably relatively few when compared to the total number of files at any given point.

[Arkenor]

Oh. I can probably answer this bit. As you know I got rid of that override emptying batch file.

So, after a pretty full install, and generalised biffing, I have 395 files left in there, weighing 7.06 Megabytes. I'll even zip it up for you and attach it if anyone wants it.

It does not appear to be doing me any great harm.

Edited by Arkenor, 12 September 2010 - 09:01 AM.