**If you are viewing this file on CRAN, please check [latest news on GitHub](https://github.com/MLopez-Ibanez/irace/blob/master/NEWS.md) where the formatting is also better.**

# irace 4.2

## Major breaking changes

 * Recovery (`--recover-file`) has been completely re-implemented.  This
   version of irace cannot recover files generated by previous versions and
   vice versa.

 * Now `blockSize` multiplies `elitistNewInstances`, which should ensure that
   irace evaluates instances in blocks.

 * Several improvements in `ablation()` may result in different ablation
   results using this version of irace. Also, the log generated by `ablation()`
   has some important changes to record more detailed information and it is
   more consistent with the log generated by `irace()`.

## New features and improvements

 * When `targetRunner` or `targetEvaluator` are implemented in R, using random
   numbers within those functions does not affect the sequence of random
   numbers within irace.

 * `psRace()` gains a `psrace_logFile` argument to avoid overwriting `scenario$logFile`.

 * `psRace()` makes better use of the budget remaining.

 * New scenario option `cappingAfterFirstTest`
   (`--capping-after-first-test`). If set to 1, elimination due to capping only
   happens after `firstTest` instances are seen (issue #78, suggested by Nguyen Dang).

 * `getConfigurationById()` now returns configurations in the same order
   (including repeated values) of the IDs passed as argument.

 * `ablation(type="full")` will evaluate all intermediate configurations on all
   instances at once.  This makes better use of parallel resources but will not
   report any information about progress after each instance.

 * `ablation()` now works better with dependent-domain parameters and with
   complicated conditions.  Ablation will not generate intermediate
   configurations that are forbidden.

 * The ACOTSP example in the user-guide now uses the ACOTSPQAP code and provides more guidance in case of errors.

## Fixes

 * Fix #79: irace.exe did not encode the `libPath` correctly in R >= 4.0 on Windows, producing 'unrecognized escape in character string'. Fixed by replacing backslashes with forward slashes and quoting the arguments that may contain spaces.

 * Fix #76: Recovery (`--recover-file`) is working again with a completely new implementation.

 * Fixed documentation of `psRace()`.

 * Fix `psRace()` not saving `psrace_log` in `scenario$logFile`.

 * Fix bug in `psRace()` using more budget than the maximum given.

 * Handle the case when all elites are rejected before starting the race and
   `sum(alive) <= minSurvival`.

 * Fix bug in `sampleSobol()` with parameters whose domain depends on a conditional parameter.


# irace 4.1

## Major breaking changes

 * Recovery (`--recover-file`) is currently broken (issue #76).

 * The `experiment` list passed to the `targetRunner` R function (see
  `target_runner_default()`) does not contain an element `switches`.
  This element can be obtained from `scenario$parameters$switches`.


## New features and improvements

 * Conditions that are always true because they depend on fixed parameters are
   replaced by a `TRUE` value to speed up their evaluation. This speed-up is only
   measurable with very large and complicated parameter spaces.

 * The options `src` and `target` of `ablation()` (or `--src` and `--target` of
   the command-line `ablation` tool) now accept a character string that points
   to a file containing a configuration that will be read with
   `readConfigurationsFile()`.  This allows performing ablation between
   configurations not explored before.

## Fixes

 * Fixed some typos in the user guide. (@sbomsdorf)

 * Fixed several bugs in post-selection including `Error in if (any(left >= 0L)) { : missing value where TRUE/FALSE needed`.

 * Fix bug in `ablation()` not calling `stop_parallel()`.


# irace 4.0

## Major breaking changes

 * Requires R version >= 4.0

 * Logfiles `*.Rdata` use format version 3, which can only be read by R version >= 3.5.

 * The scenario options `forbiddenFile` and `forbiddenExps` have been removed
   and will give an error if present.  Forbidden configurations are now
   specified in the parameter space description. See the example in
   `readParameters()`.

 * The scenario option `digits` has been removed and will give an error if
   present. The number of `digits` for real-valued parameters is now specified
   in the parameter space description. See the example in `readParameters()`.

 * The default value of the scenario option `trainInstancesDir` is now `""`.
   The previous default value of `"./Instances"` often caused confusion to
   users not using files as training instances.

 * The `scenario` object now includes the `parameters` object. Thus
   functions such as `irace()`, which previously took as arguments both
   `scenario` and `parameters`, now only take `scenario`. This also means that
   the log file `irace.Rdata` does not contain a separate `parameters` element
   since this element can now be found within `scenario`.

 * The default value of the scenario option `softRestartThreshold` is now
   0.0001 and does not depend on `digits`.

 * The command-line executables `irace` and `ablation` (`irace.exe` and
   `ablation.exe` in Windows) will load the version of the `irace` package that
   is found in the same path where the executables are. In earlier versions,
   the executables will always load the version found via `base::.libPaths()`.
   This change allows installing multiple versions of the irace package in
   different locations and each executable will use its corresponding version.
   The correct location can be verified by looking at the line `"installed at:"`
   printed in the output.

 * Adaptive capping is now enabled by default if `maxTime > 0` and `boundMax > 0`.
   It can be disabled with `--capping 0` in the command-line options or `capping=0` in the scenario options. See [Pérez-Cáceres et al. (2017)](https://iridia-ulb.github.io/references/#PerLopHooStu2017:lion) for details.
                                     (Leslie Pérez Cáceres, Manuel López-Ibáñez)

 * The scenario option `targetRunnerLauncherArgs`, introduced in version 3.5,
   was removed and replaced by `targetCmdline`, which is more flexible (fixes #38).
   Please see the user-guide for details.

 * Command-line options in joined form, given as `"--log-file= --check"`, without
   any argument after the `'='` will be interpreted as an empty argument,
   equivalent to using `logFile=""` in `scenario.txt`.

 * irace will now give an error if you try to recover from a file generated by
   a different version of irace, since such attempts typically end up in errors
   that are difficult to understand.

 * irace warns about using `'&&'` and `'||'` instead of `'&'` and `'|'` in
   parameter conditions and forbidden expressions. A future version of irace
   will reject those uses as errors.

 * The internal function `irace.reload.debug()` has been removed.
   Use `devtools::reload()` instead.

 * The column `"instance"` of the `instancesList` data frame stored in the
   logFile has been renamed to `"instanceID"`.  This data frame should not be
   accessed directly. Instead use the new function `get_instanceID_seed_pairs()`.

 * Using `maxTime > 0` with `elitist=0` now gives a clear error rather than fail later.
                                                    (fix #65, reported by @DE0CH)

 * `path_rel2abs()` will not expand symlinks to avoid problems with Python's venv.
                                                    (fix #64, reported by @DE0CH)

 * Expansion of `'~'` in Windows now follows the definition of `fs::path_expand()` rather than `base::path.expand()`.

 * irace is now more strict in enforcing runtime bounds given with `scenario$boundMax`
   and will stop with an error if the `target-runner` reports a runtime larger than the given bound.

 * All functions that contained a period (`'.'`) in the name have been renamed to use `'_'` instead.

 * The periods (`'.'`) in the arguments of `scenario$targetRunnerParallel`
   and `scenario$targetEvaluator` have also been replaced by `'_'`.

 * The environment `.irace` that was available in the log file under
   `iraceResults$state$.irace` is replaced directly by `iraceResults$state`.
   It contains similar information but some entries have been renamed. For
   example, the `experimentLog` data frame is now called `experiment_log`
   and it is a [`data.table`](https://r-datatable.com).

 * The interface of `psRace()` has been simplified.

 * `irace` will automatically execute a post-selection race (`psRace()`) using
   any remaining budget (currently only when `maxTime == 0`). To disable
   this behavior, set the scenario option `postselection` (`--postselection`)
   to `0`.


## New features and improvements

 * `sampleUniform()` and `sampleModel()` are significantly faster thanks to using [`data.table`](https://r-datatable.com).

 * Initial configurations are sampled using Sobol low-discrepancy sequences using `spacefillr::generate_sobol_set()`. This should provide a better initial distribution of parameter values.

 * Parameter spaces can be constructed programmatically using `parametersNew()`. See the documentation for details.

 * Ablation will report configurations that produced the same results, which
   indicates parameter values that have the same effect on the target algorithm,
   possibly indicating a bug in the target algorithm.

 * New option `instancesFile` of `ablation()` for using either the training
   instances, the test instances or reading instances from a given file.

 * New option `nrep` of `ablation()` specifies the number of replications per
   instance used in `"full"` ablation. It replaces the previous parameter
   `n_instances`, whose definition was more difficult to use correctly.

 * Matrix operations are faster thanks to `matrixStats`.

 * New scenario option `blockSize` for defining blocks of instances.
   Configurations are only eliminated after evaluating a complete block and
   never in the middle of a block. This is useful for scenarios when there are
   clearly defined instance classes and the best configuration should be
   balanced among them. In that case, `trainInstancesFile` should be written
   so that each block contains one instance of each class and
   `blockSize` is set to the number of classes.

 * New scenario option `targetRunnerTimeout`: Timeout in seconds of any
   `targetRunner` call (only applies to `target-runner` executables not to R
   functions).

 * `plotAblation()` has several new options:
    - `type='rank'` to plot ranks per instance instead of raw cost values.
    - `n` to limit the number of parameters shown in the plot.
    - `width` replaces `pdf.width`.
    - `height` sets the height of the plot in the PDF file.

 * The previously internal function `check.output.target.runner` is renamed to
   `check_output_target_runner` and exported to allow users who write their own
   `targetRunnerParallel` to check the output and repair it if possible.
   (Deyao Chen)

 * New functions `read_ablogfile()`, `has_testing_data()`, `irace_summarise()`.

 * New functions `get_random_seed()`, `set_random_seed()`,
   `restore_random_seed()` useful for writing `targetRunner` functions in R.

 * New function `get_instanceID_seed_pairs()` to get the pairs of instanceID
   and random seed used during the races (and optionally the actual instances).

 * The `parameters` object now stores the number of `digits` (decimal places
   after the point) for each parameter of type `r`. As a result, the
   `repairConfiguration` function (see `defaultScenario()`) only needs two
   arguments: `configuration` and `parameters`. See examples in the user-guide.

 * `readScenario()` (and command-line irace) do not require a `scenario.txt` file.
                                                         (Contributed by @DE0CH)

 * `read_pcs_file()` now supports forbidden configurations.

 * When testing, `irace` now prints the random seed used for each instance as an additional column.

 * The package provides a new executable `target-runner-dummy` (or
   `target-runner-dummy.exe` in Windows) for the purposes of testing.  It may
   also be useful for understanding the typical setup of `irace`.

 * New scenario option `minExperiments` to set a minimum budget of runs.
                                                (proposed by @Saethox, fixes #58)

 * New function `multi_irace()` for executing multiple runs of irace with the
   same or different scenarios and parameters, possibly in parallel.
                                                       (Contributed by @Saethox)

## Fixes

 * `ablation_cmdline()` and `plotAblation()` no longer create an empty `Rplots.pdf` file when specifying an output PDF file.

 * Fix #66: when using `maxTime > 0`, irace estimates the time per run by
   executing 2 configurations on `firstTest` instances and adjusts `boundMax`
   to not go over `budgetEstimation`. This may result in a smaller `boundMax`
   than before. To reduce this impact, the default value of `budgetEstimation`
   is now `0.05` instead of `0.02`.
                                       (Manuel López-Ibáñez, reported by @DE0CH)

 * Fix #55: Configurations provided may use `<NA>` in addition to `NA` to denote
   the missing value of a disabled parameter.
                                      (Manuel López-Ibáñez, reported by @TheIronBorn)

 * Fix #44: irace now will give an error if the domain of real-valued (r)
   parameters would change depending on the value of `'digits'`. The solution
   is to increase the value of `'digits'` or adjust the domain.
                                       (Manuel López-Ibáñez, reported by @mb706)

 * If scenario option `targetRunnerParallel` is set, irace no longer tries to
   initialize a parallel environment or setup MPI. It is now the responsibility
   of the user to do that before calling irace or within the function assigned
   to `targetRunnerParallel`.

 * irace no longer sets `option(error=utils::recover())` in debug mode to avoid issues
   when calling irace from Python. The user can set this if desired.

 * Fix bug failing to restart with parameters that have dependent domains.

 * Fix bug with `sampleInstances=FALSE` that could re-evaluate the same
   (instance, seed) pair more than once.

 * Fix bug when using `targetRunnerLauncher` and `targetRunner` contain whitespace.

 * Fix bug in `ablation_cmdline()` about missing `scenario` object.

 * `ablation()` will now save and restore the previous random seed.

 * `ablation()` will detect if the logfile (e.g., `irace.Rdata`) is incomplete.

 * `readConfigurationsFile()` now handles parameters with dependent domains.

 * Fix #71: Ensure `".ID."` is the first column in `checkTargetFiles()` (Manuel López-Ibáñez, reported by @ivan1arriola)


# irace 3.5

## New features and improvements

 * Handling of dependent parameter domains: These should be specified in the
   parameter domain definition and, for now, only numerical parameter can
   define dependent domains. A numerical domain can be dependent on one bound,
   e.g. `(1, "param1*2")`, where the dependent bound can include basic
   arithmetic operators.          (Leslie Pérez Cáceres, Manuel López-Ibáñez)

 * The package now provides an `ablation` executable (`ablation.exe` in
   Windows) that makes easier to perform ablation analysis without having any R
   knowledge.

 * The interface to functions `ablation()` and `plotAblation()` has been
   simplified. The `ablation()` function now allows overriding scenario
   settings. The `plotAblation()` function will not create the plot if the
   ablation log does not contain a complete ablation.
                                                    (Manuel López-Ibáñez)

 * The argument `n.instances` of `ablation()` has been renamed to `n_instances`
   and it is now a factor that multiplies `scenario$firstTest`.
                                                         (Manuel López-Ibáñez)

 * New command-line option `--quiet` to run without producing any output
   except errors (also available as a scenario option).
                                                     (Manuel López-Ibáñez)

 * New command-line option `--init` to initialize a scenario. (Deyao Chen)

 * Added support for HTCondor cluster framework to `--batchmode`.
                                                     (Filippo Bistaffa)

 * `--check` now also check the contents of `configurationsFile` and runs
   configurations provided via `initConfigurations`.
                          (Manuel López-Ibáñez, reported by Andreea Avramescu)

 * New scenario options `targetRunnerLauncher` and `targetRunnerLauncherArgs`
   to help in cases where the target-runner must be invoked via another
   software with particular options (such as `python.exe` in Windows).
                                                           (Manuel López-Ibáñez)

 * New scenario option `minMeasurableTime`.
                                                     (Manuel López-Ibáñez)

 * An error is produced if a variable set in the scenario file is not known to
   irace.  If your scenario file contains R code, then use variable names
   beginning with a dot `'.'`, which will be ignored by irace.
                                                    (Manuel López-Ibáñez)

 * Plotting functions have been moved to the new package
   [iraceplot](https://auto-optimization.github.io/iraceplot/).  In particular,
   `configurationsBoxplot()` is replaced by `iraceplot::boxplot_training()` and
   `iraceplot::boxplot_test()`; `parallelCoordinatesPlot()` is replaced by
   `iraceplot::parallel_cat()` and `iraceplot::parallel_coord()`; and
   `parameterFrequency()` is replaced by `iraceplot::sampling_frequency()`.
                                     (Leslie Pérez Cáceres, Manuel López-Ibáñez)

 * The user-guide now contains a detailed section on "Hyper-parameter
   optimization of machine learning methods".
                                                     (Manuel López-Ibáñez)

 * When `testType="F-test"` and only two configurations remain, the elimination
   test now uses the pseudo-median estimated by the Wilcoxon signed-rank test
   to decide which configuration is the best one instead of comparing the
   median difference.
                                                       (Manuel López-Ibáñez)

 * New functions `testing_fromlog()` and `testing_fromfile()` for independently
   executing the testing phase. The function `testing.main()` was removed as it
   is superseded by the new ones.
                                                      (Manuel López-Ibáñez)

 * New function `read_logfile()` to easily read the log file produced by irace.
                                                       (Manuel López-Ibáñez)

 * New function `printParameters()` that prints a parameters R object as a valid input text.
                                                       (Manuel López-Ibáñez)

 * `irace2pyimp` moved to its own R package.
                                                     (Manuel López-Ibáñez)

 * Generating the file `irace.Rdata` may be disabled by setting `logFile=""`.
                               (Manuel López-Ibáñez, reported by Johann Dreo)

 * `path_rel2abs()` and `checkParameters()` are now exported so that other
   packages may use them.
                                                       (Manuel López-Ibáñez)

 * `path_rel2abs()` also searches in system paths.     (Manuel López-Ibáñez)

 * `readConfigurationsFile()` will now detect duplicated configurations and
   error.                                         (Manuel López-Ibáñez)

 * The interface to functions `getFinalElites()`, `getConfigurationById()` and
   `getConfigurationByIteration()` has been simplified.

 * The package provides a `irace.sindef` file that may be used for building a
   standalone container of irace using Singularity. See the `README.md` file
   for instructions.                                (Contributed by Johann Dreo)

 * New example `examples/target-runner-python/target-runner-python-win.bat`
   contributed by Levi Ribeiro.

 * New helper script in `bin/parallel-irace-slurm` to launch `irace` in [SLURM](https://slurm.schedmd.com/) computer clusters.
                                                       (Manuel López-Ibáñez)

 * Rename `scenario.update.paths()` to `scenario_update_paths()`. The old name is deprecated. (Manuel López-Ibáñez)

## Fixes

 * Correctly handle clear out-performance cases despite strong bi-modality.
                                            (Reported by Nguyen Dang,
                                            fixed by Manuel López-Ibáñez)

 * Fix error when recovering from a parallel run on Windows.
                            (Manuel López-Ibáñez, reported by Tarek Gamal)

 * `testNbElites` now controls how many iteration elites are tested when
   `testIterationElites=1`. This is the documented behavior in the user guide.
                           (Manuel López-Ibáñez, reported by Marcelo de Souza)

 * Fixes to the Matlab example. (Manuel López-Ibáñez)

 * The default of `testType` is now set to `t-test` when capping is enabled.
                          (Manuel López-Ibáñez, reported by Jovana Radjenovic)

 * Fix various issues in the user guide.
                          (Manuel López-Ibáñez, reported by Jovana Radjenovic)

 * Remove duplicated elites.
                          (Manuel López-Ibáñez, reported by Federico Naldini)

 * Fix (#7): warnings with partial matched parameters.
                                  (Manuel López-Ibáñez, reported by Marc Becker)

 * Fix (#10): wrong assert with `elitist=0`.             (Manuel López-Ibáñez)

 * Fix (#12): irace can be run with [FastR](https://www.graalvm.org/22.1/docs/getting-started/#run-r).

 * Fix (#13): Maximum number configurations immediately rejected reached.
                                                           (Manuel López-Ibáñez)

 * Fix: when setting the scenario file in the command-line, `scenarioFile` was
   not set correctly. The correct scenario was used, however, the debug output
   and the value stored in the log / recovery file was wrong.
                         (Manuel López-Ibáñez, reported by Richard Schoonhoven)

 * With `sampleInstances = FALSE`, elitist irace does not change the order of
   instances already seen.  However, if you want to make sure that the order of
   the instances is enforced, you also need to set `elitistNewInstances=0`.

 * The function `irace.usage()` was removed. It was not really useful for R
   users as the same result can be obtained by calling
   `irace.cmdline("--help")`.
                                                         (Manuel López-Ibáñez)


# irace 3.4.1  (31/03/2020)

 * `NEWS` converted to markdown.

 * Fix CRAN error on Solaris.


# irace 3.4  (30/03/2020)

 * `irace2pyimp` function and executable (`irace2pyimp.exe` on Windows) to
   convert .Rdata files generated by irace to the input files required by the
   parameter importance analysis tool PyImp
   (https://github.com/automl/ParameterImportance).
                                             (Nguyen Dang, Manuel López-Ibáñez)

 * Initial configurations may also be provided directly in R using
   `scenario$initConfigurations`
                                                   (Manuel López-Ibáñez)

 * Rdata files are saved in version 2 to keep compatibility with older R
   versions.
                                                   (Manuel López-Ibáñez)
 * Fix invalid assert with ordered parameters:        (Leslie Pérez Cáceres)

    ```
    value >= 1L && value <= length(possibleValues) is not TRUE
    ```

 * The `irace` executable (`irace.exe` on Windows) is a compiled binary instead
   of a script. On Windows, `irace.exe` replaces `irace.bat`
                                                   (Manuel López-Ibáñez)

 * `inst/examples/Spear` contains the Spear (SAT solver) configuration scenario.
                                                   (Manuel López-Ibáñez)

 * Fixed bug when reporting minimum `maxTime` required.
                                           (Reported by Luciana Salete Buriol,
                                            fixed by Manuel López-Ibáñez)

 * Fixed bug detected by assert:

    ```R
    all(apply(!is.na(elite.data$experiments), 1, any)) is not TRUE
    ```

     (Reported by Maxim Buzdalov, fixed by Manuel López-Ibáñez)


# irace 3.3 (26/04/2019)

 * Fix buggy test that breaks CRAN.                   (Manuel López-Ibáñez)

 * Do not print "23:59:59" when wall-clock time is actually close to zero.
                                                        (Manuel López-Ibáñez)

# irace 3.2 (24/04/2019)

 * Fix `irace --check --parallel 2` on Windows.         (Manuel López-Ibáñez)

 * Values of real-valued parameter are now printed with sufficient precision to
   satisfy `digits` (up to `digits=15`).
                                                        (Manuel López-Ibáñez)

 * It is possible to specify `boundMax` without capping.
                                    (Leslie Pérez Cáceres, Manuel López-Ibáñez)

 * `irace --check` will exit with code 1 if the check is unsuccessful
                                                        (Manuel López-Ibáñez)

 * Print where irace is installed with `--help`.          (Manuel López-Ibáñez)

 * irace will now complain if the output of `target-runner` or `target-evaluator`
   contains extra lines even if the first line of output is correct. This is to
   avoid parsing the wrong output. Unfortunately, this may break setups that
   relied on this behavior. The solution is to only print the output that irace
   expects.
                                                        (Manuel López-Ibáñez)

 * Completely re-implement `log` parameters to fix several bugs. Domains that
   contain zero or negative values are now rejected.
                                    (Leslie Pérez Cáceres, Manuel López-Ibáñez)

 * New option `aclib=` (`--aclib 1`) enables compatibility with the
   GenericWrapper4AC (https://github.com/automl/GenericWrapper4AC/) used by
   AClib (http://aclib.net/). This is EXPERIMENTAL. `--aclib 1` also sets
   digits to 15 for compatibility with AClib defaults.
                                                         (Manuel López-Ibáñez)

 * Fix printing of output when capping is enabled.
                                                         (Manuel López-Ibáñez)

 * `checkTargetFiles()` (`--check`) samples an instance unless
   `sampleInstances` is FALSE.                           (Manuel López-Ibáñez)

 * Fix symbol printed in elimination test.               (Manuel López-Ibáñez)

 * Use `dynGet()` to find `targetRunner` and `targetEvaluator`.
   As a result, we now require R >= 3.2.
                                                         (Manuel López-Ibáñez)

 * All tests now use `testthat`.                         (Manuel López-Ibáñez)

 * New function `scenario.update.paths()`                (Manuel López-Ibáñez)

 * Fix assert failure that may happen when `elitistNewInstances` is larger than
   `firstTest`. Reported by Jose Riveaux.                (Manuel López-Ibáñez)

 * Fix bug in `checkTargetFiles()` (`--check`) with capping.
                                                        (Leslie Pérez Cáceres)

 * Clarify a few errors/warnings when `maxTime > 0`.
                     (Manuel López-Ibáñez, suggested by Haroldo Gambini Santos)


# irace 3.1  (12/07/2018)

 * Use testthat for unit testing.                        (Manuel López-Ibáñez)

 * Allow instances to be a list of arbitrary R objects (`mlr` bugfix).
                                                         (Manuel López-Ibáñez)

# irace 3.0  (05/07/2018)

 * irace now supports adaptive capping for computation time minimization.
   The default value of the `testType` option is t-test when adaptive capping
   is enabled. Please see the user-guide for details.
                                    (Leslie Pérez Cáceres, Manuel López-Ibáñez)

 * The package contains an `ablation()` function implementing the ablation
   method for parameter importance analysis by Fawcett and Hoos (2016).
                                     (Leslie Pérez Cáceres, Manuel López-Ibáñez)

 * New option `postselection` executes a post-selection race.
                                                        (Leslie Pérez Cáceres)

 * At the end of each race, if the race stops before evaluating all instances
   seen in previous races, then the best overall may be different than the best
   of the race. We now print the best overall (best-so-far). Elites evaluated
   on more instances are considered better than those evaluated on fewer.
                                    (Manuel López-Ibáñez, Leslie Pérez Cáceres)

 * Last active parameter values of numerical parameters (`i` and `r`) are carried
   by the sampling model. When a value must be assigned and the parameter was
   previously not active, the sampling is performed around the last value.
                                   (Leslie Pérez Cáceres, Manuel López-Ibáñez)

 * R help pages are now generated with Roxygen2.
                                   (Leslie Pérez Cáceres, Manuel López-Ibáñez)

 * The user guide documents `--version`, `--help`, and `--check`.
                                                        (Manuel López-Ibáñez)

 * A return value of `Inf` from `targetRunner`/`targetEvaluation` results in
   the immediate rejection of the configuration without any further evaluation.
   This is useful for handling unreliable or broken configurations that should
   not stop irace.                                        (Manuel López-Ibáñez)

 * Numerical parameters may be sampled on a logarithmic scale using `i,log`
   or `r,log`.                                               (Alberto Franzin)

 * New `target-runner.bat` for Windows contributed by André de Souza Andrade.

 * Fixed all shell scripts calling functions before defining them, which is not
   portable.
                                                         (Manuel López-Ibáñez)

 * Fixed `--parallel` bug in Windows that resulted in
   `Error in checkForRemoteErrors(val)`.
                                                          (Manuel López-Ibáñez)

 * Improve error message when no training instances are given.
                                                          (Manuel López-Ibáñez)


# irace 2.4 (03/08/2017)

 * The output of irace now specifies in which order, if any, configurations are
   printed.
                              (Manuel López-Ibáñez, suggested by Markus Wagner)

 * Several fixes for handling paths in Windows.
                                                     (Manuel López-Ibáñez)

 * `readConfigurationsFile()` now has a `text=` argument, which allows reading
   configurations from a string.
                                                     (Manuel López-Ibáñez)

 * User-provided functions (targetRunner, targetEvaluator and
   repairConfiguration) and user-provided conditions for forbidden
   configurations are now byte-compiled when read, which should make their
   evaluation noticeably faster.
                                                      (Manuel López-Ibáñez)

 * The argument `'experiment'` passed to the R function `targetRunner` does not
   contain anymore an element `'extra.params'`. Similarly, the `'scenario'`
   structure does not contain anymore the elements `'instances.extra.params'` and
   `'testInstances.extra.params'`. Any instance-specific parameters values now
   form part of the character string that defines an instance and it is up to
   the user-defined `targetRunner` to parse them appropriately. These changes
   make no difference when targetRunner is an external script, or when
   instances and instance-specific parameter values are read from a file.
                                                        (Manuel López-Ibáñez)

# irace 2.3

 * Fix bug that will cause `iraceResults$experimentLog` to count calls to
   `targetEvaluator` as experiments, even if no call to `targetRunner` was
   performed. This does not affect the computation of the budget consumed and,
   thus, it does not affect the termination criteria of irace. The bug triggers
   an assertion that terminates irace, thus no run that was successful with
   version 2.2 is affected.
                                                     (Manuel López-Ibáñez)

# irace 2.2

 * Command-line parameters are printed to stdout (useful for future
   replications). (Manuel López-Ibáñez, suggested by Markus Wagner)

 * Users may provide a function to repair configurations before being
   evaluated. See the scenario variable repairConfiguration.
                                                     (Manuel López-Ibáñez)

 * The option `--sge-cluster` (`sgeCluster`) was removed and replaced by
   `--batchmode` (`batchmode`). It is now the responsibility of the target-runner
   to parse the output of the batch job submission command (e.g., `qsub` or
   `squeue`), and return just the job ID. Values supported are: "sge", "torque",
   "pbs" and "slurm".                                     (Manuel López-Ibáñez)

 * The option `--parallel` can now be combined with `--batchmode` to limit the
   number of jobs submitted by irace at once. This may be useful in batch
   clusters that have a small queue of jobs.
                                                          (Manuel López-Ibáñez)

 * New examples under `inst/examples/batchmode-cluster/`.
                                                       (Manuel López-Ibáñez)

 * It is now possible to include scenario definition files from other scenario
   files by using:

    ```R
    eval.parent(source("scenario-common.txt", chdir = TRUE, local = TRUE))
    ```

     This feature is VERY experimental and the syntax is likely to change in the
     future.                                             (Manuel López-Ibáñez)

 * Fix a bug that re-executed elite results under some circumstances.
   (Leslie Pérez Cáceres)

 * Restrict the number of maximum configurations per race to 1024.
   (Leslie Pérez Cáceres)

 * Do not warn if the last line in the instance file does not terminate with a
   newline. (Manuel López-Ibáñez)

 * Fix bug when `deterministic == 1`.
   (Manuel López-Ibáñez, Leslie Pérez Cáceres)

 * Update manual and vignette with details about the expected arguments and
   return value of `targetRunner` and `targetEvaluator`. (Manuel López-Ibáñez)

 * Many updates to the User Guide vignette. (Manuel López-Ibáñez)

 * Fix `\dontrun` example in `irace-package.Rd` (Manuel López-Ibáñez)

 * Fix bug: If testInstances contains duplicates, results of testing are not
   correctly saved in `iraceResults$testing$experiments` nor reported correctly
   at the end of a run. Now unique IDs of the form `1t, 2t, ...` are used for
   each testing instance. These IDs are used for the rownames of
   `iraceResults$testing$experiments` and the names of the
   `scenario$testInstances`
   and `iraceResults$testing$seeds` vectors.  (Manuel López-Ibáñez)

 * Fix bug where irace keeps retrying the `target-runner` call even if it
   succeeds. (Manuel López-Ibáñez)

 * New command-line parameter
```
       --only-test FILE
```
   which just evaluates the configurations given in FILE on the testing
   instances defined by the scenario. Useful if you decide on the testing
   instances only after running irace.    (Manuel López-Ibáñez)

 * Bugfix: When using `maxTime != 0`, the number of experiments performed may be
   miscounted in some cases.              (Manuel López-Ibáñez)


# irace 2.1

 * Fix CRAN errors in tests. (Manuel López-Ibáñez)

 * Avoid generating too many configurations at once if the initial time
   estimation is too small. (Manuel López-Ibáñez)

# irace 2.0

 * Minimum R version is 2.15.

 * Elitist irace by default, it can be disabled with parameter `--elitist 0`.
   (Leslie Pérez Cáceres, Manuel López-Ibáñez)

 * The parameter `--test-type` gains two additional values: (Manuel López-Ibáñez)

   - `t-test-bonferroni` (t-test with Bonferroni's correction for multiple
                        comparisons),
   - `t-test-holm` (t-test with Holm's correction for multiple comparisons)

 * MPI does not create log files with `--debug-level 0`.
   (Manuel López-Ibáñez)

 * For simplicity, the `parallel-irace-*` scripts do not use an auxiliary
   `tune-main` script.  For customizing them, make a copy and edit them
   directly.
   (Manuel López-Ibáñez)

 * New parameters: (Manuel López-Ibáñez)
```
--target-runner-retries : Retry target-runner this many times in case of error.
```


 * We print diversity measures after evaluating on each instance:
   (Leslie Pérez Cáceres)

   - Kendall's W (also known as Kendall's coefficient of concordance) If 1,
     all candidates have ranked in the same order in all instances.  If 0, the
     ranking of each candidate on each instance is essentially random.

             W = Friedman / (m * (k-1))

   - Spearman's rho: average (Spearman) correlation coefficient computed on the
     ranks of all pairs of raters. If there are no repeated data values, a
     perfect Spearman correlation of +1 or -1 occurs when each of the variables
     is a perfect monotone function of the other.

 * Many internal and external interfaces have changed. For example, now we
   consistently use 'scenario' to denote the settings passed to irace and
   'configuration' instead of 'candidate' to denote the parameter settings
   passed to the target algorithm. Other changes are:
```R
   parameters$boundary -> parameters$domain
   hookRun             -> targetRunner
   hookEvaluate        -> targetEvaluator
   tune-conf           -> scenario.txt
   instanceDir         -> trainInstancesDir
   instanceFile        -> trainInstancesFile
   testInstanceDir     -> testInstancesDir
   testInstanceFile    -> testInstancesFile
```

 * Minimal example of configuring a MATLAB program
   (thanks to Esteban Diaz Leiva)

 * Paths to files or directories given in the scenario file are relative to the
   scenario file (except for `--log-file`, which is an output file and it is
   relative to `--exec-dir`). Paths given in the command-line are relative to the
   current working directory. Given
```bash
      $ cat scenario/scenario.txt
      targetRunner <- "./target-runner"
      $ irace -s scenario/scenario.txt
```
   irace will search for `"./scenario/target-runner"`, but given
```bash
      $ irace -s scenario/scenario.txt --target-runner ./target-runner
```
   irace will search for `"./target-runner"`.      (Manuel López-Ibáñez)

 * New command-line wrapper for Windows installed at
   `system.file("bin/irace.bat", package="irace")`
   (thanks to Anthony Antoun)

 * Budget can be specified as maximum time (`maxTime`, `--max-time`) consumed by
   the target algorithm. See the documentation for the details about how this
   is handled.
   (Leslie Pérez Cáceres, Manuel López-Ibáñez)


# irace 1.07

 * The best configurations found, either at the end or at each iteration of an
   irace run, can now be applied to a set of test instances different from the
   training instances. See options `testInstanceDir`, `testInstanceFile`,
   `testNbElites`, and `testIterationElites`. (Leslie Pérez Cáceres, Manuel López-Ibáñez)

 * The R interfaces of `hookRun`, `hookEvaluate` and `hookRunParallel` have changed.
   See `help(hook.run.default)` and `help(hook.evaluate.default)` for examples of
   the new interfaces.

 * Printing of race progress now reports the actual configuration and instance
   IDs, and numbers are printed in a more human-readable format.
   (Leslie Pérez Cáceres, Manuel López-Ibáñez)

 * Reduce memory use for very large values of `maxExperiments`.
   (Manuel López-Ibáñez, thanks to Federico Caselli for identifying the issue)

 * New option `--load-balancing` (`loadBalancing`) for disabling load-balancing
   when executing jobs in parallel. Load-balancing makes better use of
   computing resources, but increases communication overhead. If this overhead
   is large, disabling load-balancing may be faster.
   (Manuel López-Ibáñez, thanks to Federico Caselli for identifying the issue)

 * The option `--parallel` in Windows now uses load-balancing by default.
   (Manuel López-Ibáñez)

 * The wall-clock time after finishing each task is printed in the output.
   (Manuel López-Ibáñez, thanks to Federico Caselli for providing an initial
   patch)


# irace 1.06

 * Fix bug that could introduce spurious whitespace when printing the
   final configurations. (Manuel López-Ibáñez)

 * Fix bug if there are more initial candidates than needed for the
   first race. (Leslie Pérez Cáceres, Manuel López-Ibáñez)

 * New configuration options, mainly for R users:

    - `hookRunParallel`: Optional R function to provide custom
      parallelization of `hook.run`.

    - `hookRunData`: Optional data passed to `hookRun`. This is ignored by the
      default `hookRun` function, but it may be used by custom `hookRun` R
      functions to pass persistent data around.  (Manuel López-Ibáñez)

# irace 1.05

 * New option `--version`. (Manuel López-Ibáñez)

 * Terminate early if there is no sufficient budget to run irace with
   the given settings. (Manuel López-Ibáñez)

 * The option `--parallel` (without `--mpi`) now works under Windows.
   (Manuel López-Ibáñez, thanks to Pablo Valledor Pellicer for testing
   it)

 * Improved error handling when running under Rmpi. Now irace will
   terminate as soon as the master node detects at least one failed
   slave node. This avoids irace reporting two times the same error.
   Also, irace will print all the unique errors returned by all slaves
   and not just the first one.
   (Manuel López-Ibáñez)

 * Forbidden configurations may be specified in terms of constraints
   on their values. Forbidden configurations will never be evaluated by irace.
   See `--forbidden-file` and `inst/templates/forbidden.tmpl`.
   (Manuel López-Ibáñez)

 * New option `--recovery-file` (`recoveryFile`) allows resuming a
   previous irace run. (Leslie Pérez Cáceres)

 * The confidence level for the elimination test is now
   configurable with parameter `--confidence`. (Leslie Pérez Cáceres)

 * Much more robust handling of relative/absolute paths. Improved support
   for Windows. (Leslie Pérez Cáceres, Manuel López-Ibáñez)

 * Provide better error messages for incorrect parameter
   descriptions. (Manuel López-Ibáñez)
    Examples:
```
    x "" i (0, 0)       # lower and upper bounds are the same
    x "" r (1e-4, 5e-4) # given digits=2, ditto
    x "" i (-1, -2)     # lower bound must be smaller than upper bound
    x "" c ("a", "a")   # duplicated values
```
 * Print elapsed time for calls to hook-run if `debugLevel >=1`.
   (Manuel López-Ibáñez)

 * `examples/hook-run-python/hook-run`: A multi-purpose `hook-run` written
   in Python. (Franco Mascia)

 * Parallel mode in an SGE cluster (`--sge-cluster`) is more
   robust. (Manuel López-Ibáñez)

# irace 1.04

 * Replace obsolete package multicore by package parallel
   (requires R >= 2.14.0)

 * Use load-balancing (`mc.preschedule = FALSE`) in `mclapply`.

# irace 1.03

 * Use `reg.finalizer` to finish Rmpi properly without clobbering
   `.Last()`.

 * Remove uses of deprecated `as.real()`.

 * Nicer error handling in `readParameters()`.

 * Add hypervolume (multi-objective) example.

 * Fix several bugs in the computation of similar candidates.

# irace 1.02

 * More concise output.

 * The parameters `expName` and `expDescription` are now useless and they
   were removed.

 * Faster computation of similar candidates (Jeremie Dubois-Lacoste
   and Leslie Pérez Cáceres).

 * Fix bug when saving instances in `tunerResults$experiments`.

 * `irace.cmdline ("--help")` does not try to quit R anymore.

# irace 1.01

 * Fix bug caused by file.exists (and possibly other functions)
   not handling directory names with a trailing backslash or slash on
   Windows.

 * Fix bug using per-instance parameters (Leslie Pérez Cáceres).

 * Fix bug when reading initial candidates from a file.
