Why does my simulation fail in GEMPACK 12, although it worked OK in earlier GEMPACK?

Some users have reported that a simulation that seemed to work OK in GEMPACK 11 (GP11) or earlier cannot be rerun using GEMPACK 12 (GP12). Instead the simulation ends badly with an arithmetic error (overflow or zero-divide), an illicit sign change, an assertion error, or a fatal loss of accuracy. Of course this is very annoying — we feel your pain!

To avoid such problems, the GEMPACK developers maintain a huge library of simulations, using many different models. Before each new GEMPACK release, we check that all these simulations still run OK, and produce the same results as before (or very nearly the same results). Hence, we are pretty confident that stable, well-behaved simulations which used to run will still run OK.

As hinted above, we cannot guarantee that results will be identical in new releases of GEMPACK, for various reasons:

Expressions may be evaluated in a different order, due to choices made by newer TABLO or a newer Fortran compiler. For example, A*(B+C) might be evaluated as (A*B + A*C).
Newer GEMPACK may be performing some calculations more accurately than previously (perhaps by using double precision).
Newer Fortran may use features available in more modern CPUs (eg, SSE2 arithmetic).
The Harwell linear solver used by GEMPACK makes various choices about the order in which it eliminates individual rows and columns of the (probably huge and sparse) LHS matrix. And it chooses when to move from a sparse to a dense solution technique. In GP12 the choices made by the solver have been re-tuned to greatly speed up the linear solution phase. However, this slightly affects numerical results. We feel that, in general, the new LU strategy yields more accurate results than the old; but it is hard to be definitive.

In normal circumstances, the above effects are too small to raise concern, and we cannot say whether results computed the new way are more or less accurate than those computed in the old way — they are merely slightly different approximations to the true solution.

But a problem arises if the small differences (in GP12 relative to GP11) are sufficient to trigger some error condition. For example:

A flow which in GP11 became close to zero might in GP12 actually become negative.
Suppose our TABLO code contained an assertion ABS(A-B)<0.001. Under GP11 this might be satisfied, but only just. Under GP12 the (A-B) difference might slightly exceed the 0.001 tolerance.
An LHS matrix which in GP11 was close to singular might in GP12 actually become singular, or so nearly singular that endogenous variable changes become huge.

Problems like those above often occur in the initial stages of developing a simulation. The modeller may take various steps to make a simulation run, such as: loosening the tolerance in assertions; choosing a more accurate solution method; or editing the initial database to alleviate problems with zero or tiny flows. Probably, after some fine-tuning, the simulation completes, the modeller sighs with relief, and then gets on with other tasks. However, if the adjustments were only just sufficent to run the simulation under GP11, it is quite possible that under GP12 (with its slightly different numerical properties) the simulation fails. The modeller might need to again fine-tune the simulation.

We have examined several examples of simulations which completed under GP11 but crashed under GP12. In most cases the GP11 log files contained numerous warnings, suggesting that the original simulation was problematic, and was perhaps teetering on the edge of failure.

In some cases, GEMPACK will terminate a simulation if logged warnings are too numerous or severe: for example, if a linear system is solved inaccurately, or if extrapolation reveals that calculations are not converging to a solution. The criterion for termination may differ between GEMPACK versions — another reason why a simulation might run under GP11 not under GP12.

Our advice, to avoid or minimize such problems, is to examine all log file warnings carefully, and to take steps to reduce or eliminate such warnings. If you do this, it is far less likely that changes to GEMPACK will cause your previously-working simulations to fail.