24 replies to this topic

[agb1]

Simple idea, probably not the first to think of it or suggest it.

 

Basically, I want a tool that can be run in one of the game directories, after extracting some mods, that will look at all of the setup-??.tp2 files and analyze them to see which ones modify the same files.  The output would be a "??.tp2.conflicts" file for each one, something like this:

 

MyMod component ## designated ## modifies files that are also modified by OtherMod component ## designated ##

 

>>list of first N files (N specified on command line before runnig tool) of each extension type of the files that are "conflicted" in this way << e.g.:

 

17 .cre:  kobold1.cre, kobold2.cre, ...

99 .bcs:  kobold1.bcs, kobold2.bcs, ..., koboldN.bcs [... additional not shown]

 

etc. for each component

 

If it's simpler, the tool could also be designed to work only after all of the mods have been installed, something like the WeiDU --change-log feature except not limited to a single file and with summarized output so we don't get lists of 35565 files.

 

My reason for this request is to make it easier to discover mod component conflicts for BWS.

[Roxanne]

Simple idea, probably not the first to think of it or suggest it.

 

Basically, I want a tool that can be run in one of the game directories, after extracting some mods, that will look at all of the setup-??.tp2 files and analyze them to see which ones modify the same files.  The output would be a "??.tp2.conflicts" file for each one, something like this:

 

MyMod component ## designated ## modifies files that are also modified by OtherMod component ## designated ##

 

>>list of first N files (N specified on command line before runnig tool) of each extension type of the files that are "conflicted" in this way << e.g.:

 

17 .cre:  kobold1.cre, kobold2.cre, ...

99 .bcs:  kobold1.bcs, kobold2.bcs, ..., koboldN.bcs [... additional not shown]

 

etc. for each component

 

If it's simpler, the tool could also be designed to work only after all of the mods have been installed, something like the WeiDU --change-log feature except not limited to a single file and with summarized output so we don't get lists of 35565 files.

 

My reason for this request is to make it easier to discover mod component conflicts for BWS.

Will it be possible to do that based on the changelog - tool? You get the same output but not after the changes are done but as a prognosis

- the problem might be in those tpa scripts and equivalents that change all files of a certain filter, makros that do not list the individual files they change.

[The Imp]

Prediction: This is likely going to be either a nightmare to program, or to use, and even more so both.
Essentially, this can be archived by the existing DOS tools, but to use them you have to extract the mod folders to the game folder, copy that folder somewhere else, install the mod to the old folder, and then make a diff out of the two game folders.

Using the Generalized biffing will also invalidate the whole thing.

[Roxanne]

Prediction: This is likely going to be either a nightmare to program, or to use, and even more so both.
Essentially, this can be archived by the existing DOS tools, but to use them you have to extract the mod folders to the game folder, copy that folder somewhere else, install the mod to the old folder, and then make a diff out of the two game folders.

Using the Generalized biffing will also invalidate the whole thing.

Analysis can be tough enough - prediction?? it is something like a crystal ball.

[agb1]

Well, I am basically trying to automate the process that is today fully manual:

If I want to know what conflicts a particular mod component has for BWS, right now I need to read the tp2, understand what files it modifies, and then find every other mod that modifies the same files, and *then* analyze if those changes are a conflict or if they are patching different parts of the files.

Since WeiDU already has the ability to show what specific components of various mods have affected any particular file, I am thinking that the tool I described is really just an extension of that. In fact I can think of an implementation method now using the backup folders after an installation. Basically search through every backup folder (they are helpfully numbered by component) and make a list of every file. When finding the same file again in a different backup folder, record the additional component number. Effectively a WeiDU --change-log * (all files). The difference is that I want to cross reference the data so that I can view it grouped by component (list of ALL files touched by THIS component) instead of grouped by the file (list of ALL components that touch THIS file). And then I want it to compare the lists of files grouped by component and tell me where they overlap (list of ALL components that touch ANY common file, with a count of how many overlaps there are). Since "how many overlaps there are" can be different from the perspective of each component, this final question produces different results for each one, so I thought of creating a ".conflicts" report for each component.

It actually does not seem so complicated now that I thought of using the backup folders, but this is a fairly large amount of data to collect and process. I would be grateful if someone would volunteer to create a tool to do this efficiently.

[Roxanne]

Well, I am basically trying to automate the process that is today fully manual:

If I want to know what conflicts a particular mod component has for BWS, right now I need to read the tp2, understand what files it modifies, and then find every other mod that modifies the same files, and *then* analyze if those changes are a conflict or if they are patching different parts of the files.

Since WeiDU already has the ability to show what specific components of various mods have affected any particular file, I am thinking that the tool I described is really just an extension of that. In fact I can think of an implementation method now using the backup folders after an installation. Basically search through every backup folder (they are helpfully numbered by component) and make a list of every file. When finding the same file again in a different backup folder, record the additional component number. Effectively a WeiDU --change-log * (all files). The difference is that I want to cross reference the data so that I can view it grouped by component (list of ALL files touched by THIS component) instead of grouped by the file (list of ALL components that touch THIS file). And then I want it to compare the lists of files grouped by component and tell me where they overlap (list of ALL components that touch ANY common file, with a count of how many overlaps there are). Since "how many overlaps there are" can be different from the perspective of each component, this final question produces different results for each one, so I thought of creating a ".conflicts" report for each component.

It actually does not seem so complicated now that I thought of using the backup folders, but this is a fairly large amount of data to collect and process. I would be grateful if someone would volunteer to create a tool to do this efficiently.

Maybe your name conflict analyzer was a bit ambitious. It is a tool to help the *conflict analyzer* which actually is - you.

It filters and collects the files you may want to look up but it will not be able to tell you whether those changes are conflicting.

[Lollorian]

Short answer: Possible but extremely hard to implement properly

[agb1]

I've noticed that sometimes a mod will copy a file (so it appears in backup) without actually making any apparent changes (file size identical before and after, Near Infinity looks identical). That makes this idea somewhat less useful.

Is there any tool for doing a human readable "diff" on all versions of a specific file (like WeiDU --change-log combined with a dump from NI)? Like the TESEdit tool for Bethesda games.

[Lollorian]

Time for a reminder then

[agb1]

So it doesn't exist. +1 for wish list.

Do we have a tool that dumps the contents of any IE file type to a file in human readable format? If yes we could hack together a script to just do that sequentially for each copy of a particular file from the backup folders and then run a diff across all of them.

[Mike1072]

Let's say you made a tool that tells you what mods affect which files (in a particular installation), using WeiDU's backup folders. I don't think it would be an efficient way to find conflicts.

You would get tons of results, and the vast majority of them would be completely innocuous changes that play along nicely with all the other changes. That's the whole point of WeiDU. But, in order to determine if a change played nicely, you'd have to look at the code for that mod component, find the part(s) that modify the file, and figure out what they are doing. Then do that for all the other mods that affect the file and consider if any of those changes might interact poorly with each other.

If you did that tedious work for every file changed by every mod, you'd be able to determine all of the conflicts in your installation. However, you wouldn't be able to say that you know all of the conflicts between the mods you installed. The presence of one mod can mask a conflict between two other mods, so this method is only guaranteed to help you find the conflicts with a particular set of mods installed in a particular order.

[agb1]

@Mike1072 yes, I have come to realize that using copies from backup folders is of limited value in finding conflicts. Thank you for your comments.

[fearless]

I came across this last night and had a think about how it might be implemented. Without knowing many of the weidu commands and how they effect files, and only glancing at one or two mods tp2 files i imagine something like this would need to be done:

 

 

Collect all files names for games folder (recursively walk folders and sub folders to get all files) store in filelist buffer
Whilst doing that collect the tp2 filenames and put into seperate buffer for later processing
for each tp2 filename parse file
for each line in tp2 file search for keyword
    if keyword = COPY then get destination file
        Search filelist buffer for destfile
        if found
            add to filelist.file potential conflict list
        else
            add destfile to 2nd filelist buffer for rechecking later
       endif
    elseif keyword = WRITE_LONG
       add modification to previously found filelist.file conflict list
    elseif keyword = EXTEND_BOTTOM
       add modification to previously found filelist.file conflict list
. 
.
. ; etc for every keyword

at end of processing loop, do another loop of tp2 files, checking 2nd filelist buffer for potential conflicts
loop through filelist, get conflict.list for each file if it has one
output information.

 

Obviously thats just a simple psuedo-code to help arrange my thoughts on how it might be accomplished, probably a lot more to do to get it all working. Id need to have a look at weidu's options to figure out each keyword, and open a few mods to see how to parse the files correctly.

 

Id imagine the output (if found) might be something like:

 

override\spl00024.spl:
   - Copied and overwitten by MySuperMod (Line 17)
   - Copied and overwitten by MEgaCoolMod10 (Line 29)
   - Modified by SimpleMod7: WRITE_LONG x08 "!" (Line 210)
   - Modified by SillyMod: EXTEND_BOTTOM (Line 54)

 

Am i correct in my thinking of how the conflicts might be processed, stored and output? Am i overlooking anything that would potentially throw a wrench into the works? Let me know. Cheers.

[Lollorian]

Pseudocode is nice That would work for a start.

[agb1]

Is WeiDU source code available? Seems to me that it would be ideal to add more logging to WeiDU about the files and sections of files it changes. Sort of a "verbose" logging mode. We could then parse those log files to find conflicts.

[Lollorian]

Wouldn't the DEBUG files it generates on installing suffice?

[The Imp]

Is WeiDU source code available?

Well duh, yeah. The up most link. And the github page has the beta.

[fearless]

The debug files may be of use, but still requires parsing them AND installing each mod. By parsing the tp2 files before installation it would give you the best hope of determining some conflicts.

[The Imp]

By parsing the tp2 files before installation it would give you the best hope of determining some conflicts.

Not necessarily.

Say like with:

BEGIN ~Bow fixes~ 
//Ranger Cleric able to use Long Bows Fix
COPY_EXISTING_REGEXP GLOB ~.*\.itm$~ ~override~
READ_SHORT ~0x001c~ ~type~
  PATCH_IF type = 5 OR type = 15 THEN BEGIN
	READ_BYTE 0x1f unus
	WRITE_BYTE 0x1f (unus BAND 11111011)
  END
BUT_ONLY

Can you make the read from the .tp2 file to notice that even when all the .itm file are read for the copy, the patch_if - but_only, makes it so that only a small portion is actually processed ? -after the first read command of course.

The .debug should give you all the files that are processed, if it's not hidden. Aka if it's ran under the DEBUG mode.

Yes, it needs the mods to be ran ones in the installer... but this tool would be utilized by a few people, not a single one after each mod update. Yeah, there's 1500 mod components, but the tool should be designed to help to take out the burden of install, not the whole thing. As then you have 99% problems..

[fearless]

I had considered the regex / wildcard issue with files, and i was thinking along the lines of not so much parsing files and finding conditionals and evaluating them for possible modifications, but more along the lines of potential conflicts generally.

 

In that instance id only be interested in the commands that potentially modify the files, the WRITE_BYTE in the example above.

 

Sure you could have a lot of files that are touched to modify by multiple mods, but i think it still would be valid to consider all as potential conflicts until resolved by user reviewing logs or by adding in some sort of exception rules for filtering output - that would be based on info from experienced modders and others that know what is ok and what is not, like it may be fine for any mod to modify byte 1f of an .itm file without issue, or maybe only in certain circumstances - i dont know myself.

 

If i was writing the above tool, thats what i would be aiming for. Doing most of the heavy collating work and outputing the data and leaving the review/judgement of what is right/wrong for a possible conflict for others - which would also help refine the tool once i know what is ok.