Refactoring Options Handler & Library Cycle Breaking

What to Know As a User:
A number of files have moved. Users that include files in the public API in more refined ways than using #include <cvc4.h> should consult which files have moved. Note though that some files may move again after being cleaned up. A number of small tweaks have been made to the swig interfaces that may cause issues. Please file bug reports for any problems.

The Problem:
The build order of CVC4 used to be [roughly] specified as:
  options < expr < util < libcvc4 < parsers < main
Each of these had their own directories and their own Makefile.am files. With the exception of the util/ directory, each of the subdirectories built exactly one convenience library. The util/ directory additionally built a statistics library. While the order above was partially correct, the build order was more complicated as options/Makefile.am executed building the sources for expr/Makefile.am as part of its BUILT_SOURCES phase. This options/Makefile.am also build the options/h and options.cpp files in other directories. There were cyclical library dependencies between the first four above libraries. All of these aspects combined to make options extremely brittle and hard to develop. Maintaining these between clang versus gcc, and bazel versus autotools has become increasing unpredictable.

The Solution:
To address these cyclic build problems, I am simplifying the build process. Here are the main things that have to happen:
1. util/ will be split into 3 separate directories: base, util, and smt_util. Each will have their own library and Makefile.am file.
2. Dependencies for options/ will be moved into options/. If a type appears as an option, this file will be moved into options.
3. All of the old options_handlers.h files have been refactored.
4. Some files have moved from util into expr/ to resolve cycles. Some of these moves are temporary.
5. I am removing the libstatistics library.

The constraints that the CVC4 build system will eventually satisfy are:
- The include order for both the .h and .cpp files for a directory must respect the order libraries are built. For example, a file in options/ cannot include from the expr/ directory. This includes built source files such as those coming from */kinds files and */options files.
- The types definitions must also respect the build order. Forward type declarations will be allowed in exceptional, justified cases.
- The Makefile.am for a directory cannot generate a file outside of the directory it controls. (Or call another Makefile.am except through subdirectory calls.)
- One library per Makefile.am.
- No extra copies of libraries will be built for the purpose of distinguishing between external and internal visibility in libraries for building parser/ or main/ libraries and binaries. Any function used by parser/ and main/ will be labeled with CVC4_PUBLIC and be in a public API. (AFAICT, libstatistics was being built exactly to skirt this.)

The build order of CVC4 can now be [roughly] specified as
  base < options < util < expr < smt_util < libcvc4 < parsers < main
The distinction between "base < options < util < expr" are currently clean. The relationship between expr and the subsequent directories/libraries are not yet clean.

More details about the directories:

base/
The new directory base/ contains the shared utilities that are absolutely crucial to starting cvc4. The list currently includes just: cvc4_assert.{h,cpp}, output.{h,cpp}, exception.{h,cpp}, and tls.{h, h.in, cpp}. These are things that are required everywhere.

options/
The options/ directory is self contained.
- It contains all of the enums that appear as options. This includes things like theory/bv/bitblast_mode.h .
- There are exactly 4 classes that handled currently using forward declarations currently to this: LogicInfo, LemmaInputChannel, LemmaOutputChannel, and CommandSequence. These will all be removed from options.
- Functionality of the options_handlers.h files has been moved into smt/smt_options_handler.h. The options library itself only uses an interface class defined in options/options_handler_interface.h. We are now using virtual dispatch to avoid using inlined functions as was previously done.
- The */options_handlers.h files have been removed.
- The generated smt/smt_options.cpp file has been be replaced by pushing the functionality that was generated into: options/options_handler_{get,set}_option_template.cpp . The non-generated functionality was moved into smt_engine.cpp.
- All of the options files have been moved from their directories into options/. This means includes like theory/arith/options.h have changed to change to options/arith_options.h .

util/
The util/ directory continues to contain core utility classes that may be used [almost] everywhere. The exception is that these are not used by options/ or base/. This includes things like rational and integer. These may not use anything in expr/ or libcvc4. A number of files have been moved out of this directory as they have cyclic dependencies graph with exprs and types. The build process up to this directory is currently clean.

expr/
The expr/ directory continues to be the home of expressions. The major change is files moving from util/ moving into expr/. The reason for this is that these files form a cycle with files in expr/.
- An example is datatype.h. This includes "expr/expr.h", "expr/type.h" while "expr/command.h" includes datatype.h.
- Another example is predicate.h. This uses expr.h and is also declared in a kinds file and thus appears in kinds.h.
- The rule of thumb is if expr/ pulls it in it needs to be independent of expr/, in which case it is in util/, or it is not, in which case it is pulled into expr/.
- Some files do not have a strong justification currently. Result, ResourceManager and SExpr can be moved back into util/ once the iostream manipulation routines are refactored out of the Node and Expr classes.
- Note the kinds files are expected to remain in the theory/ directories. These are only read in order to build sources.
- This directory is not yet clean. It contains forward references into libcvc4 such as the printer. It also makes some classes used by main/ and parser CVC4_PUBLIC.

smt_util/
The smt_util/ directory contains those utility classes which require exprs, but expr/ does not require them. These are mostly utilities for working with expressions and nodes. Examples include ite_removal.h, LemmaInputChannel and LemmaOutputChannel.

What is up next:
- A number of new #warning "TODO: ..." items have been scattered throughout the code as reminders to myself. Help with these issues is welcomed.
- The expr/ directory needs to be cleaned up in a similar to options/. Before this happens statistics needs to be cleaned up.

1 files changed, 164 insertions, 0 deletions

diff --git a/src/options/arith_options b/src/options/arith_options
new file mode 100644
index 000000000..9737d5382
--- /dev/null
+++ b/src/options/arith_options
@@ -0,0 +1,164 @@
+#
+# Option specification file for CVC4
+# See src/options/base_options for a description of this file format
+#
+
+module ARITH "options/arith_options.h" Arithmetic theory
+
+option arithUnateLemmaMode --unate-lemmas=MODE ArithUnateLemmaMode :handler CVC4::options::stringToArithUnateLemmaMode :default ALL_PRESOLVE_LEMMAS :handler-include "options/options_handler_interface.h" :include "options/arith_unate_lemma_mode.h"
+ determines which lemmas to add before solving (default is 'all', see --unate-lemmas=help)
+
+option arithPropagationMode --arith-prop=MODE ArithPropagationMode :handler CVC4::options::stringToArithPropagationMode :default BOTH_PROP :handler-include "options/options_handler_interface.h" :include "options/arith_propagation_mode.h"
+ turns on arithmetic propagation (default is 'old', see --arith-prop=help)
+
+# The maximum number of difference pivots to do per invocation of simplex.
+# If this is negative, the number of pivots done is the number of variables.
+# If this is not set by the user, different logics are free to chose different
+# defaults.
+option arithHeuristicPivots --heuristic-pivots=N int16_t :default 0 :read-write
+ the number of times to apply the heuristic pivot rule; if N < 0, this defaults to the number of variables; if this is unset, this is tuned by the logic selection
+
+# The maximum number of variable order pivots to do per invocation of simplex.
+# If this is negative, the number of pivots done is unlimited.
+# If this is not set by the user, different logics are free to chose different
+# defaults.
+expert-option arithStandardCheckVarOrderPivots --standard-effort-variable-order-pivots=N int16_t :default -1 :read-write
+ limits the number of pivots in a single invocation of check() at a non-full effort level using Bland's pivot rule
+
+option arithErrorSelectionRule --error-selection-rule=RULE ErrorSelectionRule :handler CVC4::options::stringToErrorSelectionRule :default MINIMUM_AMOUNT :handler-include "options/options_handler_interface.h" :include "options/arith_heuristic_pivot_rule.h"
+ change the pivot rule for the basic variable (default is 'min', see --pivot-rule help)
+
+# The number of pivots before simplex rechecks every basic variable for a conflict
+option arithSimplexCheckPeriod --simplex-check-period=N uint16_t :default 200
+ the number of pivots to do in simplex before rechecking for a conflict on all variables
+
+# This is the pivots per basic variable that can be done using heuristic choices
+# before variable order must be used.
+# If this is not set by the user, different logics are free to chose different
+# defaults.
+option arithPivotThreshold --pivot-threshold=N uint16_t :default 2 :read-write
+ sets the number of pivots using --pivot-rule per basic variable per simplex instance before using variable order
+
+option arithPropagateMaxLength --prop-row-length=N uint16_t :default 16
+ sets the maximum row length to be used in propagation
+
+option arithDioSolver /--disable-dio-solver bool :default true
+ turns off Linear Diophantine Equation solver (Griggio, JSAT 2012)
+
+# Whether to split (= x y) into (and (<= x y) (>= x y)) in
+# arithmetic preprocessing.
+option arithRewriteEq --enable-arith-rewrite-equalities/--disable-arith-rewrite-equalities bool :default false :read-write
+ turns on the preprocessing rewrite turning equalities into a conjunction of inequalities
+/turns off the preprocessing rewrite turning equalities into a conjunction of inequalities
+
+
+option arithMLTrick miplib-trick --enable-miplib-trick/--disable-miplib-trick bool :default false
+ turns on the preprocessing step of attempting to infer bounds on miplib problems
+/turns off the preprocessing step of attempting to infer bounds on miplib problems
+
+option arithMLTrickSubstitutions miplib-trick-subs --miplib-trick-subs=N unsigned :default 1
+ do substitution for miplib 'tmp' vars if defined in <= N eliminated vars
+
+option doCutAllBounded --cut-all-bounded bool :default false :read-write
+ turns on the integer solving step of periodically cutting all integer variables that have both upper and lower bounds
+/turns off the integer solving step of periodically cutting all integer variables that have both upper and lower bounds
+
+option maxCutsInContext --maxCutsInContext unsigned :default 65535
+ maximum cuts in a given context before signalling a restart
+
+option revertArithModels --revert-arith-models-on-unsat bool :default false
+ revert the arithmetic model to a known safe model on unsat if one is cached
+
+option havePenalties --fc-penalties bool :default false :read-write
+ turns on degenerate pivot penalties
+/turns off degenerate pivot penalties
+
+option useFC --use-fcsimplex bool :default false :read-write
+ use focusing and converging simplex (FMCAD 2013 submission)
+
+option useSOI --use-soi bool :default false :read-write
+ use sum of infeasibility simplex (FMCAD 2013 submission)
+
+option restrictedPivots --restrict-pivots bool :default true :read-write
+ have a pivot cap for simplex at effort levels below fullEffort
+
+option collectPivots --collect-pivot-stats bool :default false :read-write
+ collect the pivot history
+
+option useApprox --use-approx bool :default false :read-write
+ attempt to use an approximate solver
+
+option maxApproxDepth --approx-branch-depth int16_t :default 200 :read-write
+ maximum branch depth the approximate solver is allowed to take
+
+option exportDioDecompositions --dio-decomps bool :default false :read-write
+ let skolem variables for integer divisibility constraints leak from the dio solver
+
+option newProp --new-prop bool :default false :read-write
+ use the new row propagation system
+
+option arithPropAsLemmaLength --arith-prop-clauses uint16_t :default 8 :read-write
+ rows shorter than this are propagated as clauses
+
+option soiQuickExplain --soi-qe bool :default false :read-write
+ use quick explain to minimize the sum of infeasibility conflicts
+
+option rewriteDivk rewrite-divk --rewrite-divk bool :default false :read-write
+ rewrite division and mod when by a constant into linear terms
+
+option trySolveIntStandardEffort --se-solve-int bool :default false
+ attempt to use the approximate solve integer method on standard effort
+
+option replayFailureLemma --lemmas-on-replay-failure bool :default false
+ attempt to use external lemmas if approximate solve integer failed
+
+option dioSolverTurns --dio-turns int :default 10
+ turns in a row dio solver cutting gets
+
+option rrTurns --rr-turns int :default 3
+ round robin turn
+
+option dioRepeat --dio-repeat bool :default false
+ handle dio solver constraints in mass or one at a time
+
+option replayEarlyCloseDepths --replay-early-close-depth int :default 1
+ multiples of the depths to try to close the approx log eagerly
+
+option replayFailurePenalty --replay-failure-penalty int :default 100
+ number of solve integer attempts to skips after a numeric failure
+
+option replayNumericFailurePenalty --replay-num-err-penalty int :default 4194304
+ number of solve integer attempts to skips after a numeric failure
+
+option replayRejectCutSize --replay-reject-cut unsigned :default 25500
+ maximum complexity of any coefficient while replaying cuts
+
+option lemmaRejectCutSize --replay-lemma-reject-cut unsigned :default 25500
+ maximum complexity of any coefficient while outputing replaying cut lemmas
+
+option soiApproxMajorFailure --replay-soi-major-threshold double :default .01
+ threshold for a major tolerance failure by the approximate solver
+
+option soiApproxMajorFailurePen --replay-soi-major-threshold-pen int :default 50
+ threshold for a major tolerance failure by the approximate solver
+
+option soiApproxMinorFailure --replay-soi-minor-threshold double :default .0001
+ threshold for a minor tolerance failure by the approximate solver
+
+option soiApproxMinorFailurePen --replay-soi-minor-threshold-pen int :default 10
+ threshold for a minor tolerance failure by the approximate solver
+
+option ppAssertMaxSubSize --pp-assert-max-sub-size unsigned :default 2
+ threshold for substituting an equality in ppAssert
+
+option maxReplayTree --max-replay-tree int :default 512
+ threshold for attempting to replay a tree
+
+
+option pbRewrites --pb-rewrites bool :default false
+ apply pseudo boolean rewrites
+
+option pbRewriteThreshold --pb-rewrite-threshold int :default 256
+ threshold of number of pseudoboolean variables to have before doing rewrites
+
+endmodule