remage package

Subpackages

Submodules

remage.cli module

remage.cli._cleanup_tmp_files(ipc_info, extra_tmp_files)

Remove temporary files created by the C++ application, that might not have been cleaned up.

Parameters:
Return type:

None

remage.cli._run_remage_cpp(args=None, is_cli=False, num_procs=0)

run the remage-cpp executable and return the exit code as seen in bash.

Parameters:
Return type:

tuple[list[int], list[Signals | None], IpcResult]

remage.cli.remage_cli()
Return type:

int

remage.cli.remage_run(macros=(), *, gdml_files=(), output=None, threads=1, procs=1, overwrite_output=False, merge_output_files=False, flat_output=False, time_window=None, macro_substitutions=None, log_level=None, raise_on_error=True, raise_on_warning=False)

Run the remage simulation utility with the provided args.

This is the main entry point for users wanting to run remage from Python code.

Parameters:

  • macros (Sequence[str] | str) – one or more remage/Geant4 macro command listings to execute.

  • gdml_files (Sequence[str | Path | g4.Registry] | str | Path | g4.Registry) – supply one or more GDML files describing the experimental geometry.

  • output (str | Path | None) – output file for detector hits.

  • threads (int) – set the number of threads used by remage. This cannot be combined with procs.

  • procs (int) – set the number of processes used by remage. This cannot be combined with threads.

  • overwrite_output (bool) – overwrite existing output files.

  • merge_output_files (bool) – merge output files created by individual remage threads.

  • flat_output (bool) – if False, perform a reshaping of the output files so that each row in the output table contains data about all steps in a physical interaction in a detector, based on the time-window. This results in each column being a VectorOfVectors. If True, the output table will be flat with each row holding information about a single Geant4 step.

  • time_window (float | None) – time window to group together steps into hits, in microseconds.

  • macro_substitutions (Mapping[str, str] | None) – key-value-pairs that will be substituted in macros as Geant4 aliases.

  • log_level (str | None) – logging level. One of debug_event, debug, detail, summary, warning, error, fatal, nothing.

  • raise_on_error (bool) – raise a RuntimeError when an error in the C++ application occurs. This applies to non-fatal errors being logged as well as fatal errors. If false, the function only returns the error code, the Python-based post-processing will be skipped in any case.

  • raise_on_warning (bool) – raise a RuntimeError when a warning (or error) is logged in the C++ application. If false, warnings are only logged and the python-based post-processing will be run normally.

Return type:

tuple[int, IpcResult]

remage.cli.remage_run_from_args(args=None, *, raise_on_error=True, raise_on_warning=False, extra_tmp_files=())

Run the remage simulation utility with the provided args.

Parameters:
Return type:

tuple[int, IpcResult]

remage.cli.watchdog_thread_fn(proc)

In multiprocess mode, ensures failure from one child process propagates to the others.

Important

This function runs in a dedicated thread in remage.cli._run_remage_cpp() and should not be called directly by the user.

Parameters:

proc (list[Popen]) – The subprocess(es) running remage-cpp.

Return type:

None

remage.find_remage module

remage.find_remage._find_remage_from_config()
Return type:

tuple[Path, str] | None

remage.find_remage.find_remage_cpp()

Find the remage executable to wrap.

It uses (in order of precedence) the following methods: - the REMAGE_CPP_PATH environment variable pointing to the remage-cpp executable, - the config stored into the package at build-time, or - the system PATH.

Return type:

Path

remage.ipc module

IPC message receiver implementation for remage-cpp.

Note

The C++ IPC sender implementation can be found in RMGIpc.

Binary message format

Messages are encoded as UTF-8 strings; transmitting binary (non-string) data with this IPC mechanism is not possible.

Message parts are separated using ASCII control characters. Each message ends with GS (group separator, 0x1D) which may be optionally preceded by ENQ (enquiry, 0x05) to indicate that the C++ process expects an acknowledgement with a POSIX signal before continuing.

Records within a message are delimited by RS (record separator, 0x1E). Each message must contain at least three records. The first is the process index, for typical (non-multiprocessing) uses, this is always 0. The second is treated as the message’s key, whereas the second one is the associated value:

proc_id

key

RS

value

[ENQ]

GS

More values can follow afterwards, again delimited by RS.

Each record may contain multiple units split by US (unit separator, 0x1F). Records with more then one unit are returned as tuples on the python side.

Example: A message

proc_id

key

RS

value0

RS

value1

US

value2

[ENQ]

GS

would be decoded to ["value0", ("value1", "value2")].

Blocking messages

Blocking messages need to be acknowledged by sending ASCII ACK over the second pipe (per-process file descriptor) to the remage-cpp process, after performing the associated action (example: checking version equality of python and C++ IPC sides, pre-processing files).

Transmitting additional response data with the acknowledgement is not possible.

class remage.ipc.IpcResult(ipc_info)

Bases: object

Storage structure for the IPC messages returned by remage-cpp.

get(name: str, expected_len: Literal[1] = 1) → list[str]
get(name: str, expected_len: int) → list[list[str]]

Return all messages of a given key name from the IPC message list.

Parameters:

expected_len – only return messages that have the expected number of records. Other messages are silently skipped.

get_as_dict(name)

Same as get() but return a dictionary keyed by record key.

Operates on two-record messages: the first record is used as the key and the second as a value appended to that key’s list.

Parameters:

name (str)

Return type:

dict[str, list[str]]

get_single(name, default=None)

Return the single single value for the key name or default if not present.

Note

if more then one value for the key had been submitted, this function will throw.

Parameters:

  • name (str)

  • default (str | None)

Return type:

str | None

remove(name)

Remove all messages with the given key name from the IPC info.

Parameters:

name (str)

Return type:

None

set(name, values)

Replace existing stored messages for they key name with values.

Note

This will create a single-record message for each value in values. Setting more complex messages is not implemented yet.

Parameters:
Return type:

None

remage.ipc.handle_ipc_message(msg, proc)

Parse a already UTF-8 decoded IPC message from remage-cpp.

This function should directly handle all known blocking IPC messages, which will not be returned for subsequent processing.

Parameters:

  • msg (str) – The raw message bytes decoded to UTF-8, still including the trailing separator(s).

  • proc (list[Popen])

Returns:

(is_blocking, parsed_message, is_fatal) where is_blocking is True when the sender waits for a reply, parsed_message is the decoded message or None if it was consumed internally, and is_fatal signals that the application should terminate. proc_num is the index of the subprocess.

Return type:

tuple[bool, list | None, bool, int]

remage.ipc.ipc_thread_fn(pipe_r, pipes_o_w, proc, unhandled_ipc_messages)

Read and handle IPC messages coming from remage-cpp.

Important

This function runs in a dedicated thread in remage.cli._run_remage_cpp() and should not be called directly by the user. The parameter unhandled_ipc_messages acts as a return value, as thread functions cannot directly return.

The function reads from the pipe file descriptor pipe_r and dispatches each complete IPC message to handle_ipc_message() for parsing and handling of the associated action. Any message parts will be decoded as UTF-8 before parsing.

Blocking messages are acknowledged by sending ASCII ACK over the second pipe (per-process file descriptor).

Parameters:

  • pipe_r (int) – File descriptor for the read end of the IPC pipe.

  • proc (list[Popen]) – The subprocess(es) running remage-cpp.

  • pipes_o_w (list[int]) – File descriptor(s) for the write end of the IPC pipe(s) to the subprocess(es).

  • unhandled_ipc_messages (list) – List that will receive messages which were not directly handled (i.e., for further processing)

Return type:

None

remage.logging module

remage.logging.set_logging_level(logger, rmg_log_level)
remage.logging.setup_log()

Setup a colored logger for this package.

Return type:

Logger

remage.logging.supports_color()
Return type:

bool

remage.post_proc module

Parameters:
Return type:

None

remage.post_proc.deduplicate_table(file, table_name, unique_col, to_struct)
Parameters:
Return type:

None

remage.post_proc.make_tmp(files)

Hide files.

Prepend a . to their name and rename them on disk.

Parameters:

files (list[str] | str)

Return type:

list[str]

remage.post_proc.post_proc(ipc_info, flat_output, merge_output_files, time_window_in_us)
Parameters:
Return type:

None

remage.post_proc.tmp_renamed_files(remage_files)

Temporarily rename files, restoring originals on error and deleting temps on success.

remage.post_proc.un_make_tmp(files)

Un-hide files.

Remove . from name and rename on disk.

Parameters:

files (list[str] | str)

Return type:

list[str]

remage.post_proc.update_number_of_simulated_events(original_files, output_files, lh5_event_number_name)
Parameters:
Return type:

None

remage.reshaping module

remage.reshaping._calorimeter_hit_table(stps)

Build a flat hit table from an already per-hit detector table.

Calorimeter-like output accumulates exactly one hit per event. It must not be step-grouped; we only rename the time column to t0 for the TCM.

Parameters:

stps (Table)

Return type:

Table

remage.reshaping._get_wo_mode(first_write, chunk_idx, new_file, overwrite)

LH5 write mode for a detector table chunk.

For the first chunk of the first detector in a new hit file we (over)write the whole file; for the first chunk of a further detector we append a new column to the parent struct; otherwise we append rows to the existing table.

Parameters:
Return type:

str

remage.reshaping._get_wo_mode_forwarded(written_tables, new_file, overwrite)

LH5 write mode for a table that is copied through unchanged.

Parameters:
Return type:

str

remage.reshaping._group_by_time(obj, *, window_us)

Group steps into hits by evtid and a coincidence time window.

A new hit starts whenever evtid changes, or time jumps by more than window_us microseconds within the same evtid. Returns an lgdo.Table of VectorOfVectors, one per field of obj, preserving each field’s units parameter.

time is assumed to be in nanoseconds (the unit remage writes).

Parameters:

  • obj (Array)

  • window_us (float)

Return type:

Table

remage.reshaping._iter_chunks(stp_file, table, buffer)

Yield buffer-sized row chunks of table.

Used for flat hit tables (e.g. calorimeter) where every row is an independent hit, so no evtid alignment is needed and the table can be streamed in fixed-size blocks to bound memory.

Parameters:
remage.reshaping._iter_event_aligned_chunks(stp_file, table, buffer)

Yield buffer-sized chunks of table aligned to evtid boundaries.

All steps of a given evtid stay in the same chunk, so each chunk can be grouped into hits independently. The full evtid column is read once up front to compute the split points.

This relies on remage writing evtids monotonically increasing.

Parameters:
remage.reshaping._move_units_to_flattened_data(data)

Recursively move a units attr from a VectorOfVectors to its flattened_data.

Parameters:

data (LGDO)

Return type:

None

remage.reshaping._write_lh5(hit_table, file, out_field, out_detector, wo_mode)

Write a hit table at out_field/out_detector.

On the first write the table is wrapped in a Struct so the parent group is created with the right type; subsequent appends target the table directly.

Parameters:
Return type:

None

remage.reshaping.reshape_output(stp_files, hit_files, *, reshape_tables, forward_tables, out_field, time_window_in_us, flat_hit_tables=(), overwrite=False, buffer=5000000)

Post-process remage step files into reshaped hit files.

For each detector listed in reshape_tables, steps are grouped by event id and time (window in microseconds), producing per-hit t0 and evtid columns. Detectors in flat_hit_tables already hold one hit per detector per event and are written alongside the reshaped detectors with their time column renamed to t0 (no step-grouping). Tables listed in forward_tables are copied unchanged.

Parameters:

  • stp_files (list[str]) – list of remage step files.

  • hit_files (list[str] | str) – list of output hit files (one per stp file) or a single file when all inputs should be merged into the same output.

  • reshape_tables (Sequence[str]) – detector tables to reshape via time-grouping.

  • forward_tables (Sequence[str]) – auxiliary tables to forward to the output unchanged.

  • out_field (str) – lh5 group under which reshaped detector tables are written.

  • time_window_in_us (float) – coincidence time window in microseconds for hit grouping.

  • flat_hit_tables (Sequence[str]) – detector tables that already contain one hit per detector per event (e.g. calorimeter output) and must not be step-grouped. They are written next to the reshaped detectors under out_field with time renamed to t0 so they can join the time-coincidence map.

  • overwrite (bool) – whether to overwrite the output files.

  • buffer (int) – buffer size (in rows) for the lh5 iterator.

Return type:

None

remage.utils module

remage.utils._to_list(thing)
remage.utils.sanitize_macro_cmds(text)
Parameters:

text (str | Iterable[str])

Return type:

list[str]