Config files get changed when running the whole test set

Expected Behavior

Running the test cases does not produce any changes to the config files.