Running simulations

The remage executable drives the simulation from the command line. A short overview of all available options can be obtained with remage --help.

$ remage [OPTIONS] [macros or command string...]

Command line options

Typical invocations supply one or more GDML geometry files and a list of macros to execute:

$ remage -g setup.gdml -- run.mac

Note

The -- is required here to disambiguate since -g can be supplied with multiple arguments.

The most useful options include:

  • macros – macro files or inline command strings. If the given argument is not an existing file it is interpreted as a newline-separated list of commands.

  • -g, --gdml-files – include one or more GDML geometry files (see Experimental geometry).

  • -o, --output-file – detector hits output file (see Output).

  • -i, --interactive – keep the application open after executing macros and present a Geant4 prompt.

  • -t, --threads – number of worker threads to use (this cannot be combined with -P/--procs).

  • -P, --procs – number of worker processes to use (this cannot be combined with -t/--threads).

  • --output-group – store the whole output in the given group of the LH5 file, instead of its root (see Output).

  • -w, --overwrite – overwrite an existing output file.

  • -a, --append – add the output to an existing output file, keeping the objects that the run does not write (see Output).

  • -q, --quiet/-v, --verbose/-l, --log-level – control the verbosity. Logging levels are debug, detail, summary, warning, error, fatal, and nothing (e.g. -l debug).

  • --version/--version-rich – print version information and exit.

  • -m, --merge-output-files – merge thread-specific output files at the end of execution (see Output).

  • --flat-output – store each Geant4 step as its own row in the output (see Output).

  • --time-window-in-us – time window used when reshaping hits (see Output).

  • -s, --macro-substitutions – provide key=value pairs that will be expanded as Geant4 aliases in macros.

Parallel execution

remage supports two ways of parallelising simulations, each with its own advantages and limitations.

In short

If memory is the limiting factor, prefer --threads; otherwise, try --procs for better throughput. See below for important notes.

The output from remage running with multiple processes or threads can be merged as if they were written by a single remage process. This is described in Output in more detail.

Warning

Beware that the performance of the merging --merge-output-files together with --procs/--threads can take a lot of time as these operations are run on a single core at the moment.

Geant4 multithreading

This mode is enabled by passing --threads INTEGER to the command line. remage does not implement its own multithreading but delegates it to Geant4: a single process spawns several worker threads sharing the same memory space. This approach is memory-efficient because shared objects (such as the geometry and the physics tables) are instantiated only once.

Warning

The performance in multithreaded mode does not scale linearly with the number of threads (see issue #287 for details).

Multiple processes

This mode is enabled by passing --procs INTEGER to the command line. It is implemented by remage itself, independently of Geant4: the Python wrapper (see the Developer’s guide for details) launches several independent remage instances, each running a single process with its own private memory. This mode is markedly more resource-hungry, because every process carries a full memory footprint. In exchange, it usually provides near 1:1 performance scaling up to the number of available physical CPU cores.

As a rule of thumb, the threading mode is the better choice when memory is the limiting resource, while the multi-processing mode gives better overall throughput otherwise.

../_images/speedup_log.png

Speedup factor of the simulated event rate as a function of the number of parallel threads or processes, for a germanium box in liquid argon with both volumes registered as sensitive. Primaries were 1 MeV electrons (left) and gammas (right), generated with the general particle source without confinement, with the same number of primaries per worker. The benchmark ran on a node with two AMD EPYC 7763 CPUs (128 physical cores) writing to flash storage. Both modes scale linearly at first: multi-processing slows down around the number of physical cores, when entering the hyper-threaded regime, while multithreading saturates earlier, at a point that depends on the setup and the primary particle.

Warning

In the current design, a macro executed with multiple processes will simulate more events than a single/multi-threaded mode with the same macro. The total event count is increased by a factor of the number of processes. Example: with 5 processes and /run/beamOn 1000 in the macro, remage will simulate 5000 events.

Batch versus interactive mode

By default remage runs all specified macro commands in batch mode and then exits. When -i is given, or the /RMG/Manager/Interactive command is used in a macro, a command prompt is opened after macro execution:

$ remage -i -g setup.gdml -- run.mac
remage>

Tip

The type of Geant4 user interface is chosen automatically, and a graphical session might be started if DISPLAY is set. To always get a plain terminal prompt, set the G4UI_USE_TCSH environment variable:

$ G4UI_USE_TCSH=1 remage -i -g setup.gdml -- run.mac

Executing commands in batch mode

Inline macro commands can also be executed directly from the command line, useful feature when generating macros on the fly (in e.g. a Python script). Multiple commands may be separated by a newline or passed in multiple arguments:

$ remage -i -g setup.gdml -- "/run/initialize \n /run/beamOn 100"
$ remage -i -g setup.gdml -- "/run/initialize" "/run/beamOn 100"

Macro substitutions also work with inline commands. Values are expanded where {name} placeholders appear in the command string:

$ remage -s EVENTS=100 -- "/run/initialize \n /run/beamOn {EVENTS}"

Running from Python

The remage Python package provides the remage.remage_run() function, which accepts the same options as the command line interface. This lets you drive the simulation directly from a Python script:

from remage import remage_run

exit_code, info = remage_run(
    macros="run.mac",
    gdml_files="setup.gdml",
    output="hits.lh5",
    threads=2,
    merge_output_files=True,
)

The returned info object contains metadata from the C++ executable, including logging output and locations of temporary files.