Batch Engine

The batch engine provides the following functionality:

  1. Attempt to log-in to each IP address in a network, and identify it as a TG radio. Where successful:

  2. Execute the get RPC and parse the results. This includes obtaining a list of remote Client-Nodes which may not have an IP address.

  3. Perform one or more interactions with the radio, parsing and saving the results.

  4. If so configured, tunnel into the remote IP-less Client Nodes and repeat the interactions.

  5. Repeat for the next IP address

  6. Excute all above once, or automatically repeat multiple times every specified time period.

The main API is TgCrawler. A typical usage would be to instantiate the class (with all parameters are arguments), and then run batch_run() for single network sweep; or poll_run() for polling mode, indicating the iteration period and number of iterations.

The batch engine typically connects to several radios concurrently utilising multi-threading. The number of threads should not exceed 10, as this is the maximum number of tunnels currently supported by the TG radios. It is possible to run the batch engine sequentially, but this is much slower (typically used for debugging only).

The interaction with each radio is managed as an instance of a ‘commander’ (TgCommander). Outputs are then saved to files in a thread-safe way.

Class Information:

class tgtools.crawler.batch.TgCrawler(credentials: Credentials, /, action_local: bool = True, action_remote_cns: bool = False, action_remote_cns_list: list[str] | None = None, get_status: bool = True, get_status_xml: bool = False, get_status_xml_max_files: int = 3, get_configs: bool = False, get_configs_db: str = 'startup', get_configs_max_files: int = 3, get_events: bool = True, get_tech_logs: bool = False, get_tech_log_max_files: int = 20, send_cmds: bool = False, send_cmds_script: list[str] | None = None, set_tod: bool = False, set_tod_shift: float = 0, concurrency: bool = True, concurrency_threads: int = 10, dir_output_root: str = 'output', dir_get_configs: str = 'configs', dir_get_events: str = 'events', dir_get_status: str = 'status', dir_get_status_xml: str = 'status_xml', dir_get_tech_logs: str = 'tech_logs', filename_cmdlog: str = 'cmdlog.txt', filename_errlog: str = 'errlog.txt', debug_print: bool = True)
A batch manager for ‘crawling’ over a TG network and performing one or more actions:

  • get_status (get RPC)

  • get_configs (get-configs RPC)

  • get_events (get-events RPC)

  • get_tech_logs (via HTTP)

  • send_cmds (run-cli-commands RPC)

  • set_tod (set-system-time and set-system-date RPCs)

Actions are executed on local radios (with network IP addresses) and/or remote Client Nodes (via tunneling).

Results are saved to text files. Errors and commands sent to radios are logged to text files.

Functionality is controlled via instantiation arguments.

__init__(credentials: Credentials, /, action_local: bool = True, action_remote_cns: bool = False, action_remote_cns_list: list[str] | None = None, get_status: bool = True, get_status_xml: bool = False, get_status_xml_max_files: int = 3, get_configs: bool = False, get_configs_db: str = 'startup', get_configs_max_files: int = 3, get_events: bool = True, get_tech_logs: bool = False, get_tech_log_max_files: int = 20, send_cmds: bool = False, send_cmds_script: list[str] | None = None, set_tod: bool = False, set_tod_shift: float = 0, concurrency: bool = True, concurrency_threads: int = 10, dir_output_root: str = 'output', dir_get_configs: str = 'configs', dir_get_events: str = 'events', dir_get_status: str = 'status', dir_get_status_xml: str = 'status_xml', dir_get_tech_logs: str = 'tech_logs', filename_cmdlog: str = 'cmdlog.txt', filename_errlog: str = 'errlog.txt', debug_print: bool = True)
Parameters:

  • credentials (Credentials) – Description of network and login credentials over which actions are taken.

  • action_local (bool) – Execute actions on the local radio (directly connected to via its IP address). Default: True.

  • action_remote_cns (bool) – Execute actions on some/all remote radios (via tunnelling from local radio(s)). Default: True.

  • action_remote_cns_list (list[str]|None) – Applicable only if action_remote_cns is True: List of remote radio names for actions. If None or empty, remote radios are automatically discovered and actions are applied to them all. Default: None

  • get_status (bool) – Actions to include fetching comprehensive telemetry (get RPC), parsing, and saving the parsed results. Default: True.

  • get_status_xml (bool) – Applicable only if get_status is True: Save the raw XML response from radio. Default: False.

  • get_status_xml_max_files (int) – Applicable only if get_status_xml is True: The maximum number of files to save before rolling over. Default: 3.

  • get_configs (bool) – actions to include fetching configuration file of type get_configs_db. Default: False.

  • get_configs_db (str) – Applicable only if get_configs is True: indicates type of configuration file to fetch: ‘candidate’, ‘running, or ‘startup’. Default: ‘candidate’.

  • get_configs_max_files (int) – Applicable only if get_configs is True: The maximum number of files to save before rolling over. Default: 3.

  • get_events (bool) – actions to include fetching and parsing events. Default: True

  • get_tech_logs (bool) – actions to include fetching the tech_files. Default: False

  • get_tech_log_max_files (int) – Applicable only if get_tech_logs is True: The maximum number of files to save before rolling over. Default: 20.

  • send_cmds (bool) – actions to include sending commands (script). Default: False.

  • send_cmds_script (list[str]) – Applicable only if send_cmds is True: script contents. Default: None.

  • set_tod (bool) – actions to include setting the time of day based on the computer’s clock. Default: False.

  • set_tod_shift (float) – Applicable only if set_tod is True: a time shift (in hours) for cases where the computer is in a different timezone relative to the radios. Default: 0.

  • concurrency (bool) – if True: access radios concurrently (multiple threads); else: sequentially (much slower, handy for debugging). Default: True.

  • concurrency_threads (int) – Applicable if concurrency is True: maximum number of concurrent threads (exceeding 10 may lead to trouble due to TG embedded CPU limitations). Default: 10.

  • dir_output_root (str) – Root directory for all output files. Default: ‘output’.

  • dir_get_configs (str) – Applicable if get_configs is True: Directory for saving config files. Default: ‘configs’.

  • dir_get_events (str) – Applicable if get_events is True: Directory for saving files containing events. Default: ‘events’.

  • dir_get_status (str) – Applicable if get_status is True: Directory for saving files containing parsed get telemetry. Default: ‘status’.

  • dir_get_status_xml (str) – Applicable if get_status_xml is True: Directory for saving files containing the raw XML response to get RPC. Default: ‘status_xml’.

  • dir_get_tech_logs (str) – Applicable if get_tech_logs is True: Directory for saving tech_files. Default: ‘tech_logs’.

  • filename_cmdlog (str) – Filename for logging commands sent to radios. Default: ‘cmdlog.txt’.

  • filename_errlog (str) – Filename for logging errors. Default: ‘errlog.txt’.

  • debug_print (bool) – Run with additional output to console.

batch_run()

Batch-run do_action() for multiple radios, either concurrently (if concurrency`==`True`), or else sequentially.

If action_remote_cns is True, then do_action() is automatically called for remote CNs: those listed in action_remote_cns_list (if populated), or otherwise all remote CNs (automatically discovered).

do_actions(credential: Credential, remote_cn: str = '') → TgCommander

Executed actions on a single radio, pointed to by credential (or one of its remote CNs named remote_cn). Results, configuration-altering commands, and errors are written to files.

Parameters:

  • credential (Credential) – radio’s IP addresses, username, password

  • remote_cn (str) – name of remote radio (optional)

Return type:

TgCommander

poll_run(iteration_period: timedelta = 0, num_iterations: int = 1)

Run batch_run() every iteration_period, for a maximum of num_iterations.

Parameters:

  • iteration_period (datetime.timedelta) – Time in between iterations. If iteration executes for longer, then start next iteration immediately.

  • num_iterations (int) – The maximum number of iterations. Default: 1. A value of zero means ‘forever’.