md_docs_ug1main(1)

NAME
md_docs_ug1main - User guide The ROCm Validation Suite (RVS) is a system validation and diagnostics tool for monitoring, stress testing, detecting and troubleshooting issues that affect the functionality and performance of AMD GPU(s) operating in a high-performance/AI/ML computing environment. RVS is enabled using the ROCm software stack on a compatible software and hardware platform.

RVS is a collection of tests, benchmarks, and qualification tools each targeting a specific sub-system of the ROCm platform. The tools are implemented in software and share a common command-line interface. Each set of tests is implemented in a “module” which is a library encapsulating the functionality specific to the tool. The CLI can specify the directory containing modules to use when searching for libraries to load. Each module may have a set of options that it defines and a configuration file that supports its execution.

  Installing RVS

RVS can be obtained by building it from source code base or by installing from pre-built package.

  Building from source code

RVS has been developed as open source solution. Its source code and belonging documentation can be found at AMD's GitHub page. In order to build RVS from source code, refer ROCm Validation Suite GitHub site and follow instructions in README file.

  Installing from package

Based on the OS, use the appropriate package manager to install the rocm-validation-suite package. For more details, refer to the ROCm Validation Suite GitHub site.

RVS package components are installed in /opt/rocm. Package contains:

•   executable binary (located in _install-base_/bin/rvs)

•   public shared libraries (located in _install-base_/lib)

•   module specific shared libraries (located in _install-base_/lib/rvs)

•   default configuration files (located in _install-base_/share/rocm-validation-suite/conf)

•   GPU specific configuration files (located in _install-base_/share/rocm-validation-suite/conf/<GPU folder>)

•   testscripts (located in _install-base_/share/rocm-validation-suite/testscripts)

•   user guide (located in _install-base_/share/rocm-validation-suite/userguide)

•   man page (located in _install-base_/share/man)

  Running RVS

  Run version built from source code

cd <source folder>/build/bin

Command examples

./rvs --help                       # List all options
./rvs -g                           # List supported GPUs available in the machine
./rvs -c conf/gst_single.conf      # Run GST module default test configuration
./rvs -m gst                       # Run GST module using platform-detected config
./rvs -r 3                         # Run predefined level 3 tests (range: 1–5, 5 = highest stress)

  Run version pre-compiled and packaged with ROCm release

cd /opt/rocm/bin

Command examples

./rvs --help                                                                      # List all options
./rvs -g                                                                          # List supported GPUs available in the machine
./rvs -c ../share/rocm-validation-suite/conf/gst_single.conf                     # Run GST default test configuration
./rvs -m gst                                                                      # Run GST module using platform-detected config
./rvs -r 3                                                                        # Run predefined level 3 tests (range: 1–5, 5 = highest stress)

To run a GPU-specific test configuration, use configuration files from the GPU subfolders under /opt/rocm/share/rocm-validation-suite/conf/:

./rvs -c ../share/rocm-validation-suite/conf/MI300X/gst_single.conf  # Run MI300X-specific GST test configuration
./rvs -c ../share/rocm-validation-suite/conf/nv32/gst_single.conf    # Run Navi 32-specific GST test configuration

If present, always use GPU specific configurations instead of default test configurations.

  Basic concepts

  RVS architecture

RVS is implemented as a set of modules each implementing particular test functionality. Modules are invoked from one central place (aka Launcher) which is responsible for reading input (command line and test configuration file), loading and running appropriate modules and providing test output. RVS architecture is built around concept of Linux shared objects, thus allowing for easy addition of new modules in the future.

  Available modules

  GPU Properties – GPUP module

The GPU Properties module queries the configuration of a target device and returns the device’s static characteristics. These static values can be used to debug issues such as device support, performance and firmware problems.

  GPU Monitor – GM module

The GPU monitor tool is capable of running on one, some or all of the GPU(s) installed and will report various information at regular intervals. The module can be configured to halt another RVS module's execution if one of the quantities exceeds a specified boundary value.

  PCI Express State Monitor – PESM module

The PCIe State Monitor tool is used to actively monitor the PCIe interconnect between the host platform and the GPU. The module will register a “listener” on a target GPU’s PCIe interconnect, and log a message whenever it detects a state change. The PESM will be able to detect the following state changes:

1.  PCIe link speed changes

2.  GPU power state changes

  ROCm Configuration Qualification Tool - RCQT module

The ROCm Configuration Qualification Tool ensures the platform is capable of running ROCm applications and is configured correctly. It checks the installed versions of the ROCm components and the platform configuration of the system. This includes verifying that the dependencies corresponding to the ROCm meta-packages are installed correctly.

  PCI Express Qualification Tool – PEQT module

The PCIe Qualification Tool is used to qualify the PCIe bus on which the GPU is connected. The qualification test will be capable of determining the following characteristics of the PCIe bus interconnect to a GPU:

1.  Support for Gen 3 atomic completers

2.  DMA transfer statistics

3.  PCIe link speed

4.  PCIe link width

  SBIOS Mapping Qualification Tool – SMQT module

The GPU SBIOS mapping qualification tool is designed to verify that a platform’s SBIOS has satisfied the BAR mapping requirements for VDI and Radeon Instinct products for ROCm support.

Refer to the “ROCm Use of Advanced PCIe Features and Overview of How BAR Memory is Used In ROCm Enabled System” web page for more information about how BAR memory is initialized by VDI and Radeon products.

  P2P Benchmark and Qualification Tool – PBQT module

The P2P Benchmark and Qualification Tool is designed to provide the list of all GPUs that support P2P and characterize the P2P links between peers. In addition to testing for P2P compatibility, this test will perform a peer-to-peer throughput test between all P2P pairs for performance evaluation. The P2P Benchmark and Qualification Tool will allow users to pick a collection of two or more GPUs on which to run. The user will also be able to select whether or not they want to run the throughput test on each of the pairs.

Please see the web page “ROCm, a New Era in Open GPU Computing” to find out more about the P2P solutions available in a ROCm environment.

  PCI Express Bandwidth Benchmark – PEBB module

The PCIe Bandwidth Benchmark attempts to saturate the PCIe bus with DMA transfers between system memory and a target GPU card’s memory. The maximum bandwidth obtained is reported to help debug low bandwidth issues. The benchmark should be capable of targeting one, some or all of the GPUs installed in a platform, reporting individual benchmark statistics for each.

  GPU Stress Test - GST module

The GPU Stress Test runs various GEMM computations as workloads to stress the GPU FLOPS performance and check whether it meets the configured target GFLOPS. GEMM workloads shall be configured as either operation type or data type. GEMM based on operation types include SGEMM, DGEMM and HGEMM (Single/Double/Half-precision General Matrix Multiplication) - configured using operation parameter. GEMM based on data types include fp8, i8, fp16, bf16, fp32 and tf32 (xf32) - configured using data type parameter. The duration of the test is configurable, both in terms of time (how long to run) and iterations (how many times to run).

  Input EDPp Test - IET module

The Input EDPp Test runs GEMM workloads to stress the GPU power (that is, TGP). This test is used to verify if the GPU is capable of handling max. power stress for a sustained period of time. Also checks whether GPU power reaches a set target power.

  GPU Power Pulse Test - PULSE module

The Pulse test drives repeating high-power (GEMM compute) and low-power (idle, minimum clocks) phases at a configurable rate so that GPU power swings over time. That pattern stresses the power supply and voltage regulators with transients rather than a single sustained power level. With parallel: true on multiple GPUs, the module uses a CPU-side barrier and a GPU-side fine-grained barrier so that devices tend to enter the heavy phase together, increasing aggregate current steps. Power and temperature are read through AMD SMI. GEMM execution uses the same rvs_blas stack as GST/IET (rocBLAS or hipBLASLt).

This is a beta feature and is not intended for production use. Pass/fail criteria are still being refined.

  Memory Test - MEM module

The Memory module tests the GPU memory for hardware errors and soft errors using HIP. It consists of various tests that use algorithms like Walking 1 bit, Moving inversion and Modulo 20. The module executes the following memory tests [Algorithm, data pattern]

1.  Walking 1 bit

2.  Own address test

3.  Moving inversions, ones & zeros

4.  Moving inversions, 8 bit pattern

5.  Moving inversions, random pattern

6.  Block move, 64 moves

7.  Moving inversions, 32 bit pattern

8.  Random number sequence

9.  Modulo 20, random pattern

10. Memory stress test

  BABEL Benchmark Test - BABEL module

The Babel module executes BabelStream (synthetic GPU benchmark based on the original STREAM benchmark for CPUs) benchmark that measures memory transfer rates (bandwidth) to and from global device memory. Various benchmark tests are implemented using GPU kernels in HIP (Heterogeneous Interface for Portability) programming language.

  Command line options

Command line options are summarized in the table below:

Command line options take precedence over the same parameters set in the configuration file. For example, if `parallel: false` is specified in the configuration file but `-p true` is passed on the command line, parallel execution will be enabled.

 Short optionLong optionDescription -a--appendLogWhen generating a debug logfile, do not overwrite the content of the current log. Use in conjuction with -d and -l options.

-c--configSpecify the test configuration file to use. This is a mandatory field for test execution.

-r--runSpecify the predefined test level to run (1–5). Each level selects a GPU-specific configuration file without requiring a -c argument. See Test Levels for details.

-d--debugLevelSpecify the debug level for the output log. The range is 0-5 with 5 being the highest verbose level.

-g--listGpusList all the GPUs available in the machine, that RVS supports and has visibility.

-i--indexesComma-separated list of GPU IDs or SMI-based indexes to run tests on. Overrides the device and device_index values specified in all actions of the configuration file, including the all value.

-s--selectActionsComma separated list of action names or 0-based action index numbers to run from the configuration file. Only the matching actions will be executed. All other actions are skipped.

-j--jsonGenerate output file in JSON format. if a path follows this argument, that will be used as json log file; else a file created in /var/tmp/ with timestamp in name.

-l--debugLogFileGenerate log file with output and debug information.

-m--moduleSpecify a module name to run the corresponding platform-specific (MI-series GPUs) module configuration file. Valid modules: babel, gpup, gst, iet, mem, pebb, peqt, pbqt, rcqt.

-t--listTestsList the test modules present in RVS.

-v--verboseEnable detailed logging. Equivalent to specifying -d 5 option.

-p--parallelEnables or Disables parallel execution across multiple GPUs. Use this option in conjunction with -c option. Accepted Values: true – Enables parallel execution. false – Disables parallel execution. If no value is provided for the option, it defaults to true.

-n--numTimesNumber of times the test repeatedly executes. Use this option in conjunction with -c option.

--quietNo console output given. See logs and return code for errors.

--versionDisplays the version information and exits.

-h--helpDisplay usage information and exit.

  Test Levels

When using the -r option, RVS automatically selects a predefined configuration file based on the detected GPU platform (for example, conf/MI300X/levels/rvs_level_N.conf). No -c argument is needed. The five levels represent progressively increasing test coverage and duration:

 LevelTestDescriptionModulesTypical Duration 1Software validationVerifies that all required ROCm packages and metapackages are correctly installed.rcqtShort (seconds) 2Hardware capability checksChecks PCIe link speed and width, performs a quick HBM read/write pass, and verifies basic PCIe and XGMI interface connectivity.peqt, babel, pebb, pbqtShort (seconds) 3Basic performanceMeasures HBM bandwidth across common operations, PCIe host-to-device and device-to-host throughput, bidirectional XGMI bandwidth, and limited gemm compute performance runs.babel, pebb, pbqt, gstMinutes 4Extended performanceSame coverage as level 3 with longer test durations, full HBM operation set, bidirectional PCIe transfers, a basic memory test pass, and full gemm compute performance runs.babel, pebb, pbqt, mem, gst, ietTens of minutes 5Full stress testRuns all tests from level 4 at maximum duration, adds high-iteration memory stress testing, and includes a sustained power stress test targeting peak GPU TDP. Intended for thorough pre-production validation.babel, pebb, pbqt, mem, gst, ietHours

The exact tests and thresholds within each level vary by GPU platform.

  Examples

Print version information and exit:

./rvs --version

List all available test modules:

./rvs -t

 List all GPUs visible to RVS:

./rvs -g

Run a specific configuration file:

./rvs -c conf/gst_single.conf

Run a platform-specific configuration file (for example, MI350X GST):

./rvs -c conf/MI350X/gst_single.conf

Repeat a test 5 times in sequence (soak/stability testing):

./rvs -c conf/gst_single.conf -n 5

Run only the GST module using the built-in platform configuration:

./rvs -m gst

Run the predefined level 3 test suite (basic performance):

./rvs -r 3

Run the predefined level 5 (full stress) test suite on specific GPUs only:

./rvs -r 5 -i 0,1

Run a configuration file on specific GPUs (by index) with parallel execution enabled:

./rvs -c conf/gst_single.conf -i 0,1 -p true

Run only selected actions from a configuration file (by name or 0-based index):

./rvs -c conf/iet_single.conf -s action_1,action_2
./rvs -c conf/iet_single.conf -s 0,2

Generate JSON output to a specific file:

./rvs -c conf/gst_single.conf -j /tmp/rvs_results.json

Run with verbose logging and save the log to a file:

./rvs -c conf/gst_single.conf -v -l /tmp/rvs_debug.log

Suppress console output and write results to a JSON file (useful in CI pipelines):

./rvs -c conf/gst_single.conf --quiet -j /tmp/rvs_results.json

The configuration files in the top-level conf/ folder are generic samples and may not reflect the correct thresholds or parameters for your hardware. For accurate results, use the platform-specific configuration files under conf/<GPU>/ (for example, conf/MI350X/), or use the -r or -m option, which automatically selects the appropriate platform configuration.

  GPU-Specific Configurations

RVS includes optimized test configurations for a range of GPU families, organized under conf/<GPU>/. Level-based configurations (usable with -r) are available for: MI300X, MI300X-HF, MI308X, MI308X-HF, MI325X, MI350X, MI355X.

AMD Instinct Series:

•   MI210, MI250X, MI300A, MI300X, MI308X, MI325X, MI350X, MI355X

•   High-frequency variants: MI300X-HF, MI308X-HF

AMD Radeon Series:

•   nv21, nv31, nv32, gfx1200, gfx1201

•   RX9060, RX9070, RX9070GRE, R9600D

  Configuration files

The RVS tool will allow the user to indicate a configuration file, adhering to the YAML 1.2 specification, which details the validation tests to run and the expected results of a test, benchmark or configuration check.

The configuration file used for an execution is specified using the --config option. The default configuration file used for a run is rvs.conf, which will include default values for all defined tests, benchmarks and configurations checks, as well as device specific configuration values. The format of the configuration files determines the order in which actions are executed, and can provide the number of times the test will be executed as well.

Configuration file is, in YAML terms, mapping of 'actions' keyword into sequence of action items. Action items are themselves YAML keyed lists. Each list consists of several key:value pairs. Some keys may have values which are keyed lists themselves (nested mappings).

Action item (or action for short) uses keys to define nature of validation test to be performed. Each action has some common keys -- like 'name', 'module', 'deviceid' -- and test specific keys which depend on the module being used.

An example of RVS configuration file is given here:

actions:
- name: action_1
  device: all
  module: gpup
  properties:
    mem_banks_count:
  io_links-properties:
    version_major:
- name: action_2
  module: gpup
  device: all
  properties:
    mem_banks_count:
- name: action_3

  Common configuration keys

Common configuration keys applicable to most module are summarized in the table below:

 Config Key TypeDescription nameStringThe name of the defined action. deviceCollection of String This is a list of device indexes (gpu ids), or the keyword “all”. The defined actions will be executed on the specified device, as long as the action targets a device specifically (some are platform actions). If an invalid device id value or no value is specified the tool will report that the device was not found and terminate execution, returning an error regarding the configuration file.

deviceidIntegerThis is an optional parameter, but if specified it restricts the action to a specific device type corresponding to the deviceid. parallelBoolIf this key is false, actions will be run on one device at a time, in the order specified in the device list, or the natural ordering if the device value is “all”. If this parameter is true, actions will be run on all specified devices in parallel. If a value isn’t specified the default value is false.

countIntegerThis specifies number of times to execute the action. If the value is 0, execution will continue indefinitely. If a value isn’t specified the default is 1. Some modules will ignore this parameter.

waitIntegerThis indicates how long the test should wait between executions, in milliseconds. Some modules will ignore this parameter. If the count key is not specified, this key is ignored.

durationIntegerThis parameter overrides the count key, if specified. This indicates how long the test should run, given in milliseconds. Some modules will ignore this parameter.

moduleStringThis parameter specifies the module that will be used in the execution of the action. Each module has a set of sub-tests or sub-actions that can be configured based on its specific parameters.

log_intervalIntegerThis specifies how often, in milliseconds, the module emits a progress or status log message during a running test. If a value isn't specified the default is 1000 ms. Some modules will ignore this parameter.

  GPUP module

The GPU properties module provides an interface to easily dump the static characteristics of a GPU. This information is stored in the sysfs file system for the kfd, with the following path:

/sys/class/kfd/kfd/topology/nodes/<node id>

Each of the GPU nodes in the directory is identified with a number, indicating the device index of the GPU. This module will ignore count, duration or wait key values.

  Module specific keys

Config Key

Type

Description </thead> <tbody>

properties

Collection of Strings

 The properties key specifies what configuration property or properties the query is interested in. Possible values are:
•   all - collect all settings
•   gpu_id
•   cpu_cores_count
•   simd_count
•   mem_banks_count
•   caches_count
•   io_links_count
•   cpu_core_id_base
•   simd_id_base
•   max_waves_per_simd
•   lds_size_in_kb
•   gds_size_in_kb
•   wave_front_size
•   array_count
•   simd_arrays_per_engine
•   cu_per_simd_array
•   simd_per_cu
•   max_slots_scratch_cu
•   vendor_id
•   device_id
•   location_id
•   drm_render_minor
•   max_engine_clk_fcompute
•   local_mem_size
•   fw_version
•   capability
•   max_engine_clk_ccompute

io_links-properties
Collection of Strings
 The properties key specifies what configuration property or properties the query is interested in. Possible values are:
•   all - collect all settings
•   count - the number of io_links
•   type
•   version_major
•   version_minor
•   node_from
•   node_to
•   weight
•   min_latency
•   max_latency
•   min_bandwidth
•   max_bandwidth
•   recommended_transfer_size
•   flags
</tbody>

  Output
Module specific output keys are described in the table below:
 Output Key TypeDescription properties-valuesCollection of Integers The collection will contain a positive integer value for each of the valid properties specified in the properties config key. io_links-propertiesvaluesCollection of Integers The collection will contain a positive integer value for each of the valid properties specified in the io_links-properties config key.
Each of the settings specified has a positive integer value. For each setting requested in the properties key a message with the following format will be returned:
[RESULT][<timestamp>][<action name>] gpup <gpu id> <property> <property value>
For each setting in the io_links-properties key a message with the following format will be returned:
[RESULT][<timestamp>][<action name>] gpup <gpu id> <io_link id> <property> <property value>

  Examples
Example 1:
Consider action:
actions:
- name: action_1
  device: all
  module: gpup
  properties:
    all:
  io_links-properties:
    all:
Action will display all properties for all compatible GPUs present in the system. Output for such configuration may be like this:
[RESULT] [597737.498442] action_1 gpup 3254 cpu_cores_count 0
[RESULT] [597737.498517] action_1 gpup 3254 simd_count 256
[RESULT] [597737.498558] action_1 gpup 3254 mem_banks_count 1
[RESULT] [597737.498598] action_1 gpup 3254 caches_count 96
[RESULT] [597737.498637] action_1 gpup 3254 io_links_count 1
[RESULT] [597737.498680] action_1 gpup 3254 cpu_core_id_base 0
[RESULT] [597737.498725] action_1 gpup 3254 simd_id_base 2147487744
[RESULT] [597737.498768] action_1 gpup 3254 max_waves_per_simd 10
[RESULT] [597737.498812] action_1 gpup 3254 lds_size_in_kb 64
[RESULT] [597737.498856] action_1 gpup 3254 gds_size_in_kb 0
[RESULT] [597737.498901] action_1 gpup 3254 wave_front_size 64
[RESULT] [597737.498945] action_1 gpup 3254 array_count 4
[RESULT] [597737.498990] action_1 gpup 3254 simd_arrays_per_engine 1
[RESULT] [597737.499035] action_1 gpup 3254 cu_per_simd_array 16
[RESULT] [597737.499081] action_1 gpup 3254 simd_per_cu 4
[RESULT] [597737.499128] action_1 gpup 3254 max_slots_scratch_cu 32
[RESULT] [597737.499175] action_1 gpup 3254 vendor_id 4098
[RESULT] [597737.499222] action_1 gpup 3254 device_id 26720
[RESULT] [597737.499270] action_1 gpup 3254 location_id 8960
[RESULT] [597737.499318] action_1 gpup 3254 drm_render_minor 128
[RESULT] [597737.499369] action_1 gpup 3254 max_engine_clk_ccompute 2200
[RESULT] [597737.499419] action_1 gpup 3254 local_mem_size 17163091968
[RESULT] [597737.499468] action_1 gpup 3254 fw_version 405
[RESULT] [597737.499518] action_1 gpup 3254 capability 8832
[RESULT] [597737.499569] action_1 gpup 3254 max_engine_clk_ccompute 2200
[RESULT] [597737.499633] action_1 gpup 3254 0 count 1
[RESULT] [597737.499675] action_1 gpup 3254 0 type 2
[RESULT] [597737.499695] action_1 gpup 3254 0 version_major 0
[RESULT] [597737.499716] action_1 gpup 3254 0 version_minor 0
[RESULT] [597737.499736] action_1 gpup 3254 0 node_from 4
[RESULT] [597737.499763] action_1 gpup 3254 0 node_to 1
[RESULT] [597737.499783] action_1 gpup 3254 0 weight 20
[RESULT] [597737.499808] action_1 gpup 3254 0 min_latency 0
[RESULT] [597737.499830] action_1 gpup 3254 0 max_latency 0
[RESULT] [597737.499853] action_1 gpup 3254 0 min_bandwidth 0
[RESULT] [597737.499878] action_1 gpup 3254 0 max_bandwidth 0
[RESULT] [597737.499902] action_1 gpup 3254 0 recommended_transfer_size 0
[RESULT] [597737.499927] action_1 gpup 3254 0 flags 1
[RESULT] [597737.500208] action_1 gpup 50599 cpu_cores_count 0
[RESULT] [597737.500254] action_1 gpup 50599 simd_count 256
[RESULT] [597737.501603] action_1 gpup 50599 0 recommended_transfer_size 0
[RESULT] [597737.501626] action_1 gpup 50599 0 flags 1
[RESULT] [597737.501877] action_1 gpup 33367 cpu_cores_count 0
[RESULT] [597737.501921] action_1 gpup 33367 simd_count 256
[RESULT] [597737.503258] action_1 gpup 33367 0 recommended_transfer_size 0
[RESULT] [597737.503282] action_1 gpup 33367 0 flags 1
Example 2:
Consider action:
actions:
- name: action_1
  device: all
  module: gpup
  properties:
    simd_count:
    mem_banks_count:
    io_links_count:
    vendor_id:
    device_id:
    location_id:
    max_engine_clk_ccompute:
  io_links-properties:
    version_major:
    type:
    version_major:
    version_minor:
    node_from:
    node_to:
    recommended_transfer_size:
    flags:
This action explicitly lists some of the properties. Output for such configuration may be:
[RESULT] [597868.690637] action_1 gpup 3254 device_id 26720
[RESULT] [597868.690713] action_1 gpup 3254 io_links_count 1
[RESULT] [597868.690766] action_1 gpup 3254 location_id 8960
[RESULT] [597868.690819] action_1 gpup 3254 max_engine_clk_ccompute 2200
[RESULT] [597868.690862] action_1 gpup 3254 mem_banks_count 1
[RESULT] [597868.690903] action_1 gpup 3254 simd_count 256
[RESULT] [597868.690950] action_1 gpup 3254 vendor_id 4098
[RESULT] [597868.691029] action_1 gpup 3254 0 flags 1
[RESULT] [597868.691053] action_1 gpup 3254 0 node_from 4
[RESULT] [597868.691075] action_1 gpup 3254 0 node_to 1
[RESULT] [597868.691099] action_1 gpup 3254 0 recommended_transfer_size 0
[RESULT] [597868.691119] action_1 gpup 3254 0 type 2
[RESULT] [597868.691138] action_1 gpup 3254 0 version_major 0
[RESULT] [597868.691158] action_1 gpup 3254 0 version_minor 0
[RESULT] [597868.691425] action_1 gpup 50599 device_id 26720
[RESULT] [597868.691469] action_1 gpup 50599 io_links_count 1
[RESULT] [597868.691517] action_1 gpup 50599 location_id 17152
[RESULT] [597868.692159] action_1 gpup 33367 device_id 26720
[RESULT] [597868.692204] action_1 gpup 33367 io_links_count 1
[RESULT] [597868.692252] action_1 gpup 33367 location_id 25344
[RESULT] [597868.692619] action_1 gpup 33367 0 version_minor 0
Example 3:
Consider this action:
actions:
- name: action_1
  device: all
  module: gpup
  deviceid: 267
  properties:
    all:
  io_links-properties:
    all:
Action lists deviceid 267 which is not present in the system. Output for such configuration is:
RVS-GPUP: action: action_1  invalid 'deviceid' key value

  GM module
The GPU monitor module can be used monitor and characterize the response of a GPU to different levels of use. This module is intended to run concurrently with other actions, and provides a start and stop configuration key to start the monitoring and then stop it after testing has completed. The module can also be configured with bounding box values for interested GPU parameters. If any of the GPU’s parameters exceed the bounding values on a specific GPU an INFO warning message will be printed to stdout while the bounding value is still exceeded.

  Module specific keys

Config Key
Type
Description </thead> <tbody>

monitor
Bool
 If this key is set to true, the GM module will start monitoring on specified devices. If this key is set to false, all other keys are ignored and monitoring of the specified device will be stopped.

metrics
 Collection of Structures, specifying the metric, if there are bounds and the bound values. The structures have the following format:
{String, Bool, Integer, Integer}

 The set of metrics to monitor during the monitoring period. Example values are:
•   {'temp', 'true', max_temp, min_temp}
•   {'clock', 'false', max_clock, min_clock}
•   {'mem_clock', 'true', max_mem_clock, min_mem_clock}
•   {'fan', 'true', max_fan, min_fan}
•   {'power', 'true', max_power, min_power}
The set of upper bounds for each metric are specified as an integer. The units and values for each metric are:
•   temp - degrees Celsius
•   clock - MHz
•   mem_clock - MHz
•   fan - Integer between 0 and 255
•   power - Power in Watts

sample_interval
Integer
 If this key is specified metrics will be sampled at the given rate. The units for the sample_interval are milliseconds. The default value is 1000.

log_interval
Integer
 If this key is specified informational messages will be emitted at the given interval, providing the current values of all parameters specified. This parameter must be equal to or greater than the sample rate. If this value is not specified, no logging will occur.

terminate
Bool
 If the terminate key is true the GM monitor will terminate the RVS process when a bounds violation is encountered on any of the metrics specified.

force
Bool
 If true and terminate key is also true the RVS process will terminate immediately. Note: this may cause resource leaks within GPUs. </tbody>

  Output
Module specific output keys are described in the table below:
 Output Key TypeDescription metric_valuesTime Series Collection of Result IntegersA collection of integers containing the result values for each of the metrics being monitored. metric_violationsCollection of Result Integers A collection of integers containing the violation count for each of the metrics being monitored. metric_averageCollection of Result Integers
When monitoring is started for a target GPU, a result message is logged with the following format:
[RESULT][<timestamp>][<action name>] gm <gpu id> started
In addition, an informational message is provided for each metric being monitored:
[INFO ][<timestamp>][<action name>] gm <gpu id> monitoring <metric> bounds min:<min_metric> max: <max_metric>
During the monitoring informational output regarding the metrics of the GPU will be sampled at every interval specified by the sample_rate key. If a bounding box violation is discovered during a sampling interval, a warning message is logged with the following format:
[INFO ][<timestamp>][<action name>] gm <gpu id> <metric> bounds violation <metric value>
If the log_interval value is set an information message for each metric is logged at every interval using the following format:
[INFO ][<timestamp>][<action name>] gm <gpu id> <metric> <metric_value>
When monitoring is stopped for a target GPU, a result message is logged with the following format:
[RESULT][<timestamp>][<action name>] gm <gpu id> gm stopped
The following messages, reporting the number of metric violations that were sampled over the duration of the monitoring and the average metric value is reported:
[RESULT][<timestamp>][<action name>] gm <gpu id> <metric> violations <metric_violations>
[RESULT][<timestamp>][<action name>] gm <gpu id> <metric> average <metric_average>

  Examples
Example 1:
Consider action:
actions:
- name: action_1
  module: gm
  device: all
  monitor: true
  metrics:
    temp: true 20 0
    fan: true 10 0
  duration: 5000
- name: another_action
This action will monitor temperature and fan speed for 5 seconds and then continue with the next action. Output for such configuration may be:
[RESULT] [694381.521373] [action_1] gm 33367 started
[INFO  ] [694381.531803] action_1 gm 33367  monitoring temp bounds min:0 max:20
[INFO  ] [694381.531817] action_1 gm 33367  monitoring temp bounds min:0 max:20
[INFO  ] [694381.531828] action_1 gm 33367  monitoring fan bounds min:0 max:10
[RESULT] [694381.521373] [action_1] gm 3254 started
[INFO  ] [694381.532257] action_1 gm 3254  monitoring temp bounds min:0 max:20
[INFO  ] [694381.532276] action_1 gm 3254  monitoring temp bounds min:0 max:20
[INFO  ] [694381.532293] action_1 gm 3254  monitoring fan bounds min:0 max:10
[RESULT] [694381.521373] [action_1] gm 50599 started
[INFO  ] [694381.534471] action_1 gm 50599  monitoring temp bounds min:0 max:20
[INFO  ] [694381.534487] action_1 gm 50599  monitoring temp bounds min:0 max:20
[INFO  ] [694381.534502] action_1 gm 50599  monitoring fan bounds min:0 max:10
[INFO  ] [694381.534623] action_1 gm 33367 temp  bounds violation 22C
[INFO  ] [694381.534822] action_1 gm 3254 temp  bounds violation 22C
[INFO  ] [694381.534946] action_1 gm 50599 temp  bounds violation 22C
[INFO  ] [694382.535329] action_1 gm 33367 temp  bounds violation 22C
[INFO  ] [694385.537777] action_1 gm 50599 temp  bounds violation 21C
[RESULT] [694386.538037] [action_1] gm 3254 stopped
[RESULT] [694386.538037] [action_1] gm 50599 stopped
[RESULT] [694386.538037] [action_1] gm 33367 stopped
[RESULT] [694386.521449] [action_1] gm 3254 temp violations 1
[RESULT] [694386.521449] [action_1] gm 3254 temp average 19C
[RESULT] [694386.521449] [action_1] gm 3254 fan violations 0
[RESULT] [694386.521449] [action_1] gm 3254 fan average 0%
[RESULT] [694386.521449] [action_1] gm 50599 temp violations 5
[RESULT] [694386.521449] [action_1] gm 50599 temp average 21C
[RESULT] [694386.521449] [action_1] gm 50599 fan violations 0
[RESULT] [694386.521449] [action_1] gm 50599 fan average 0%
[RESULT] [694386.521449] [action_1] gm 33367 temp violations 5
[RESULT] [694386.521449] [action_1] gm 33367 temp average 22C
[RESULT] [694386.521449] [action_1] gm 33367 fan violations 0
[RESULT] [694386.521449] [action_1] gm 33367 fan average 0%
Example 2:
Consider action:
actions:
- name: action_1
  module: gm
  device: all
  monitor: true
  metrics:
    temp: true 20 0
    fan: true 10 0
    power: true 100 0
  sample_interval: 1000
  log_interval: 1200
  terminate: false
  duration: 5000
This configuration is similar to that in Example 1 but has explicitly given values for sample_interval and log_interval. Output is similar to the previous one but averaging and the printout are performed at a different rate.
Example 3:
Consider action with syntax error ('temp' key is missing lower value):
actions:
- name: action_1
  module: gm
  device: 33367 50599
  monitor: true
  metrics:
    temp: true 20
    fan: true 10 0
    power: true 100 0
  sample_interval: 1000
  log_interval: 1200
Output for such configuration is:
RVS-GM: action: action_1 Wrong number of metric parameters
Example 4:
Consider action with logical error:
actions:
- name: action_1
  module: gm
  device: all
  monitor: true
  metrics:
    temp: false 20 0
    clock: true 1500 852
    power: true 100 0
  sample_interval: 5000
  log_interval: 4000
  duration: 8000
Output for such configuration is:
RVS-GM: action: action_1 Log interval has the lower value than the sample interval

  PESM module
The PCIe State Monitor (PESM) tool is used to actively monitor the PCIe interconnect between the host platform and the GPU. The module registers “listener” on a target GPUs PCIe interconnect, and log a message whenever it detects a state change. The PESM is able to detect the following state changes:
1.  PCIe link speed changes
2.  GPU device power state changes
This module is intended to run concurrently with other actions, and provides a ‘start’ and ‘stop’ configuration key to start the monitoring and then stop it after testing has completed. For information on GPU power state monitoring, see PCI Power Management Capability Structure, Gen 3 spec, device states D0-D3. For information on link status changes see Status Register (Offset 12h), Gen 3 spec.
Monitoring is performed by polling respective PCIe registers roughly every 1ms (one millisecond).

  Module Specific Keys
 Config Key TypeDescription monitorBoolThis key is set to true, the PESM module will start monitoring on specified devices. If this key is set to false, all other keys are ignored and monitoring will be stopped for all devices. debugwaitIntegerDebug wait period in milliseconds inserted before monitoring begins. Intended for development and debugging use. The default value is 0 (no wait).

  Output
Module specific output keys are described in the table below:
 Output Key TypeDescription stateStringA string detailing the current power state of the GPU or the speed of the PCIe link.
When monitoring is started for a target GPU, a result message is logged with the following format:
[RESULT][<timestamp>][<action name>] pesm <gpu id> started
When monitoring is stopped for a target GPU, a result message is logged with the following format:
[RESULT][<timestamp>][<action name>] pesm all stopped
When monitoring is enabled, any detected state changes in link speed or GPU power state will generate the following informational messages:
[INFO ][<timestamp>][<action name>] pesm <gpu id> power state change <state>
[INFO ][<timestamp>][<action name>] pesm <gpu id> link speed change <state>

  Examples
Example 1
Here is a typical check utilizing PESM functionality:
actions:
- name: action_1
  device: all
  module: pesm
  monitor: true
- name: action_2
  device: 33367
  module: gst
  parallel: false
  count: 2
  wait: 100
  duration: 18000
  ramp_interval: 7000
  log_interval: 1000
  max_violations: 1
  copy_matrix: false
  target_stress: 5000
  tolerance: 0.07
  matrix_size: 5760
- name: action_3
  device: all
  module: pesm
  monitor: false
•   action_1 will initiate monitoring on all devices by setting key monitor to true
•   action_2 will start GPU stress test
•   action_3 will stop monitoring
If executed like this:
sudo rvs -c conf/pesm8.conf -d 3
output similar to this one can be produced:
[RESULT] [497544.637462] [action_1] pesm all started
[INFO  ] [497544.648299] [action_1] pesm 33367 link speed change 8 GT/s
[INFO  ] [497544.648299] [action_1] pesm 33367 power state change D0
[INFO  ] [497544.648733] [action_1] pesm 3254 link speed change 8 GT/s
[INFO  ] [497544.648733] [action_1] pesm 3254 power state change D0
[INFO  ] [497544.650413] [action_1] pesm 50599 link speed change 8 GT/s
[INFO  ] [497544.650413] [action_1] pesm 50599 power state change D0
[INFO  ] [497545.170392] [action_2] gst 33367 start 5000.000000 copy matrix:false
[INFO  ] [497547.36602 ] [action_2] gst 33367 Gflops 6478.066983
[INFO  ] [497548.69221 ] [action_2] gst 33367 target achieved 5000.000000
[INFO  ] [497549.101219] [action_2] gst 33367 Gflops 5189.993529
[INFO  ] [497550.132376] [action_2] gst 33367 Gflops 5189.993529
[INFO  ] [497563.569370] [action_2] gst 33367 Gflops 5174.935520
[RESULT] [497564.86904 ] [action_2] gst 33367 Gflop: 6478.066983 flops_per_op: 382.205952x1e9 bytes_copied_per_op: 398131200 try_ops_per_sec: 13.081952 pass: TRUE
[INFO  ] [497564.220311] [action_2] gst 33367 start 5000.000000 copy matrix:false
[INFO  ] [497566.70585 ] [action_2] gst 33367 Gflops 6521.049418
[INFO  ] [497567.99929 ] [action_2] gst 33367 target achieved 5000.000000
[INFO  ] [497568.143096] [action_2] gst 33367 Gflops 5130.281235
[INFO  ] [497582.683893] [action_2] gst 33367 Gflops 5135.204729
[RESULT] [497583.130945] [action_2] gst 33367 Gflop: 6521.049418 flops_per_op: 382.205952x1e9 bytes_copied_per_op: 398131200 try_ops_per_sec: 13.081952 pass: TRUE
[RESULT] [497583.155470] [action_3] pesm all stopped
Example 2:
Consider this file:
actions:
- name: act1
  device: all
  deviceid: xxx
  module: pesm
  monitor: true
This file has an invalid entry in deviceid key. If execute, an error will be reported:
RVS-PESM: action: act1  invalide 'deviceid' key value: xxx

  RCQT module
RCQT ensures the platform is capable of running ROCm applications and is configured correctly. It checks the installed versions of the ROCm components and the platform configuration of the system. This includes checking the dependencies corresponding to the ROCm meta-packages are installed correctly. The purpose of the RCQT is to provide an extensible, OS independent and scriptable interface capable for performing the configuration checks required for ROCm support. The checks in this module do not target a specific device.
Two types of actions are performed by RCQT. 1) Metapackage Check metapackage-validation: This will check the installation of the mentioned metapackages and their dependencies and their respective versions as required by metapackage. List of metapackages are provided with key package
2) Packages installation check packagelist-install-validation: This action checks if the package is installed. Packages are provided against key rpmpackagelist and debpackagelist
This feature is used to check installed packages on the system. It provides checks for installed packages and the currently available package versions, if applicable.

  Metapackage Check Specific Keys
Input keys are described in the table below:
 Config Key TypeDescription packageCollection of Strings Specifies the list of metapackages to check. This key is required.

  Output
Output keys are described in the table below for each metapackage along with versions of each sub package:
 Output Key TypeDescription Total packages validatedInteger total dependency packages under the said metapackage Installed packagesInteger installed dependency packages under the said metapackage Missing packages Integer missing packages under the said metapackage Version mismatch packagesInteger installed dependency packages but with wrong versions
The check will emit a result message with the following format:
Meta package <metapackage-name> :
Package <dep-package1> installed version is <version>
Package <dep-package2> installed version is <version>
Package <dep-package3> installed version is <version>
Meta package validation complete :
    Total packages validated     : <3>
    Installed packages           : <3>
    Missing packages             : <0>
    Version mismatch packages    : <0>

  Examples
Example 1:
In this example, given package has all dependencies installed.
actions:
- name: metapackage-validation
  module: rcqt
  package: rocm-ml-sdk
The output for such configuration is:
[RESULT] [3648664.1164  ] Action name :metapackage-validation
[RESULT] [3648664.1363  ] Module name :rcqt
Meta package rocm-ml-sdk :
Package miopen-hip-dev installed version is 3.3.0.60300
Package rocm-core installed version is 6.3.0.60300
Package rocm-hip-sdk installed version is 6.3.0.60300
Package rocm-ml-libraries installed version is 6.3.0.60300
Meta package validation complete :
    Total packages validated     : 4
    Installed packages           : 4
    Missing packages             : 0
    Version mismatch packages    : 0
For other cases, we will see mismatched/missing packages printed with respective count

  Packages installation check
This action checks if the package is installed. Packages are provided against key rpmpackagelist and debpackagelist

  Packages installation Specific Keys
Input keys are described in the table below:
 Config Key TypeDescription rpmpackagelistCollection of Strings Specifies the packages checked if installed on system for rhel family. debpackagelistCollection of Strings Specifies the packages checked if installed on system for ubuntu family.

  Output
Output keys are described in the table below:
 Output Key TypeDescription PackageString Name of checked package versionFloating Number Installed version of the package Missing packagesInteger Number of packages not installed. Installed packagesInteger Number of packages installed.

  Examples
Example 1:
In this example, all given packages are installed.
actions:
- name: packagelist-install-validation
  device: all
  module: rcqt
  rpmpackagelist: rocm-hip-libraries rocm-core
The output for such configuration is:
[RESULT] [496559.219160] Action name :packagelist-install-validation
[RESULT] [496559.219161]  Module name :rcqt
Package rocm-hip-libraries installed version is 6.3.0.60300
Package rocm-core installed version is 6.3.0.60300
Packages install validation complete :
    Missing packages      : 0
    Installed packages    : 2

  PEQT module
PCI Express Qualification Tool module targets and qualifies the configuration of the platforms PCIe connections to the GPUs. The purpose of the PEQT module is to provide an extensible, OS independent and scriptable interface capable of performing the PCIe interconnect configuration checks required for ROCm support of GPUs. This information can be obtained through the sysfs PCIe interface or by using the PCIe development libraries to extract values from various PCIe control, status and capabilities registers. These registers are specified in the PCI Express Base Specification, Revision 3. Iteration keys, i.e. count, wait and duration will be ignored for actions using the PEQT module.

  Module specific keys
Module specific output keys are described in the table below:

Config Key
Type
Description </thead> <tbody>

capability
 Collection of Structures with the following format:
{String, String}

 The PCIe capability key contains a collection of structures that specify which PCIe capability to check and the expected value of the capability. A check structure must contain the PCIe capability value, but an expected value may be omitted. The value of all valid capabilities that are a part of this collection will be entered into the capability_value field. Possible capabilities, and their value types are:
•   link_cap_max_speed
•   link_cap_max_width
•   link_stat_cur_speed
•   link_stat_neg_width
•   slot_pwr_limit_value
•   slot_physical_num
•   bus_id
•   atomic_op_32_completer
•   atomic_op_64_completer
•   atomic_op_128_CAS_completer
•   atomic_op_routing
•   dev_serial_num
•   kernel_driver
•   pwr_base_pwr
•   pwr_rail_type
•   device_id
•   vendor_id
</tbody>

The expected value String is a regular expression that is used to check the actual value of the capability.

  Output
Module specific output keys are described in the table below:
 Output Key TypeDescription capability_valueCollection of Strings For each of the capabilities specified in the capability key, the actual value of the capability will be returned, represented as a String. passString 'true' if all of the properties match the values given, 'false' otherwise.
The qualification check queries the specified PCIe capabilities and properties and checks that their actual values satisfy the regular expression provided in the ‘expected value’ field for that capability. The pass output key will be true and the test will pass if all of the properties match the values given. After the check is finished, the following informational messages will be generated:
[INFO  ][<timestamp>][<action name>] peqt <capability> <capability_value>
[RESULT][<timestamp>][<action name>] peqt <pass>
For details regarding each of the capabilities and current values consult the chapters in the PCI Express Base Specification, Revision 3.

  Examples
Example 1:
A regular PEQT configuration file looks like this:
actions:
- name: pcie_act_1
  module: peqt
  capability:
    link_cap_max_speed:
    link_cap_max_width:
    link_stat_cur_speed:
    link_stat_neg_width:
    slot_pwr_limit_value:
    slot_physical_num:
    device_id:
    vendor_id:
    kernel_driver:
    dev_serial_num:
    D0_Maximum_Power_12V:
    D0_Maximum_Power_3_3V:
    D0_Sustained_Power_12V:
    D0_Sustained_Power_3_3V:
    atomic_op_routing:
    atomic_op_32_completer:
    atomic_op_64_completer:
    atomic_op_128_CAS_completer:
  device: all
- When setting the `device` configuration key to `all`, the RVS will detect all the AMD compatible GPUs and run the test on all of them
- There are no regular expression for this `.conf` file, therefore RVS will report `TRUE` if at least one AMD compatible GPU is registered within the system. Otherwise it will report `FALSE`.
The Power Budgeting capability is a dynamic one, having the following form:
<PM_State>_<Type>_<Power rail>
Where:
PM_State = D0/D1/D2/D3
Type=PMEAux/Auxiliary/Idle/Sustained/Maximum
PowerRail = Power_12V/Power_3_3V/Power_1_5V_1_8V/Thermal
When the RVS tool runs against such a configuration file, it will query for the all the PCIe capabilities specified under the capability list (and log the corresponding values) for all the AMD compatible GPUs. For those PCIe capabilities that are not supported by the HW platform were the RVS is running, a 'NOT SUPPORTED' message will be logged.
The output for such a configuration file may look like this:
[INFO ] [177628.401176] pcie_act_1 peqt D0_Maximum_Power_12V NOT SUPPORTED
[INFO ] [177628.401229] pcie_act_1 peqt D0_Maximum_Power_3_3V NOT SUPPORTED
[INFO ] [177628.401248] pcie_act_1 peqt D0_Sustained_Power_12V NOT SUPPORTED
[INFO ] [177628.401269] pcie_act_1 peqt D0_Sustained_Power_3_3V NOT SUPPORTED
[INFO ] [177628.401282] pcie_act_1 peqt atomic_op_128_CAS_completer FALSE
[INFO ] [177628.401291] pcie_act_1 peqt atomic_op_32_completer FALSE
[INFO ] [177628.401303] pcie_act_1 peqt atomic_op_64_completer FALSE
[INFO ] [177628.401311] pcie_act_1 peqt atomic_op_routing TRUE
[INFO ] [177628.401317] pcie_act_1 peqt dev_serial_num NOT SUPPORTED
[INFO ] [177628.401323] pcie_act_1 peqt device_id 26720
[INFO ] [177628.401334] pcie_act_1 peqt kernel_driver amdgpu
[INFO ] [177628.401342] pcie_act_1 peqt link_cap_max_speed 8 GT/s
[INFO ] [177628.401352] pcie_act_1 peqt link_cap_max_width x16
[INFO ] [177628.401359] pcie_act_1 peqt link_stat_cur_speed 8 GT/s
[INFO ] [177628.401367] pcie_act_1 peqt link_stat_neg_width x16
[INFO ] [177628.401375] pcie_act_1 peqt slot_physical_num #0
[INFO ] [177628.401396] pcie_act_1 peqt slot_pwr_limit_value 0.000W
[INFO ] [177628.401402] pcie_act_1 peqt vendor_id 4098
[INFO ] [177628.401656] pcie_act_1 peqt D0_Maximum_Power_12V NOT SUPPORTED
[INFO ] [177628.401675] pcie_act_1 peqt D0_Maximum_Power_3_3V NOT SUPPORTED
[INFO ] [177628.401692] pcie_act_1 peqt D0_Sustained_Power_12V NOT SUPPORTED
[INFO ] [177628.401709] pcie_act_1 peqt D0_Sustained_Power_3_3V NOT SUPPORTED
[INFO ] [177628.401719] pcie_act_1 peqt atomic_op_128_CAS_completer FALSE
[INFO ] [177628.401728] pcie_act_1 peqt atomic_op_32_completer FALSE
[INFO ] [177628.401736] pcie_act_1 peqt atomic_op_64_completer FALSE
[INFO ] [177628.401745] pcie_act_1 peqt atomic_op_routing TRUE
[INFO ] [177628.401750] pcie_act_1 peqt dev_serial_num NOT SUPPORTED
[INFO ] [177628.401757] pcie_act_1 peqt device_id 26720
[INFO ] [177628.401771] pcie_act_1 peqt kernel_driver amdgpu
[INFO ] [177628.401781] pcie_act_1 peqt link_cap_max_speed 8 GT/s
[INFO ] [177628.401788] pcie_act_1 peqt link_cap_max_width x16
[INFO ] [177628.401794] pcie_act_1 peqt link_stat_cur_speed 8 GT/s
[INFO ] [177628.401800] pcie_act_1 peqt link_stat_neg_width x16
[INFO ] [177628.401806] pcie_act_1 peqt slot_physical_num #0
[INFO ] [177628.401814] pcie_act_1 peqt slot_pwr_limit_value 0.000W
[INFO ] [177628.401819] pcie_act_1 peqt vendor_id 4098
[RESULT] [177628.403781] pcie_act_1 peqt TRUE
Example 2:
Another example of a configuration file, which queries for a smaller subset of PCIe capabilities but adds regular expressions check, is given below
actions:
- name: pcie_act_1
  module: peqt
  capability:
    link_cap_max_speed: '^(2.5 GTs|5 GTs|8 GTs)$'
    link_cap_max_width:
    link_stat_cur_speed: '^(2.5 GTs|5 GTs|8 GTs)$'
    link_stat_neg_width:
    slot_pwr_limit_value: '[a-b][d-'
    slot_physical_num:
    device_id:
    vendor_id:
    kernel_driver:
  device: all
For this example, the expected PEQT check result is TRUE if:
•   At least one AMD compatible GPU is registered within the system and:
•   All <link_cap_max_speed> values for all AMD compatible GPUs match the given regular expression and
•   All <link_stat_cur_speed> values for all AMD compatible GPUs match the given regular expression
Please note that the <slot_pwr_limit_value> regular expression is not valid and will be skipped without affecting the PEQT module's check result, however, an error will be logged out.
Example 3:
Another example with even more regular expressions is given below. The expected PEQT check result is TRUE if at least one AMD compatible GPU having the ID 3254 or 33367 is registered within the system and all the PCIe capabilities values match their corresponding regular expressions.
actions:
- name: pcie_act_1
  module: peqt
  deviceid: 26720
  capability:
    link_cap_max_speed: '^(2.5 GTs|5 GTs|8 GTs)$'
    link_cap_max_width: ^(x8|x16)$
    link_stat_cur_speed: '^(8 GTs)$'
    link_stat_neg_width: ^(x8|x16)$
    kernel_driver: ^amdgpu$
    atomic_op_routing: ^((TRUE|FALSE){1})$
    atomic_op_32_completer: ^((TRUE|FALSE){1})$
    atomic_op_64_completer: ^((TRUE|FALSE){1})$
    atomic_op_128_CAS_completer: ^((TRUE|FALSE){1})$
  device: 3254 33367

  SMQT module
The GPU SBIOS mapping qualification tool is designed to verify that a platform’s SBIOS has satisfied the BAR mapping requirements for VDI and Radeon Instinct products for ROCm support. These are the current BAR requirements:
BAR 1: GPU Frame Buffer BAR
In this example it happens to be 256M, but typically this will be size of the GPU memory (typically 4GB+). This BAR has to be placed < 2^40 to allow peer- to-peer access from other GFX8 AMD GPUs. For GFX9 (Vega GPU) the BAR has to be placed < 2^44 to allow peer-to-peer access from other GFX9 AMD GPUs.
BAR 2: Doorbell BAR
The size of the BAR is typically will be < 10MB (currently fixed at 2MB) for this generation GPUs. This BAR has to be placed < 2^40 to allow peer-to-peer access from other current generation AMD GPUs.
BAR 3: IO BAR
This is for legacy VGA and boot device support, but since this the GPUs in this project are not VGA devices (headless), this is not a concern even if the SBIOS does not setup.
BAR 4: MMIO BAR
This is required for the AMD Driver SW to access the configuration registers. Since the reminder of the BAR available is only 1 DWORD (32bit), this is placed < 4GB. This is fixed at 256KB.
BAR 5: Expansion ROM
This is required for the AMD Driver SW to access the GPU’s video-BIOS. This is currently fixed at 128KB.
Refer to the ROCm Use of Advanced PCIe Features and Overview of How BAR Memory is Used In ROCm Enabled System page for more information about how BAR memory is initialized by VDI and Radeon products. Iteration keys, for example, count, wait and duration will be ignored.

  Module specific keys
Module specific output keys are described in the table below:
 Config Key TypeDescription bar1_req_sizeInteger This is an integer specifying the required size of the BAR1 frame buffer region. bar1_base_addr_minInteger This is an integer specifying the minimum value the BAR1 base address can be. bar1_base_addr_maxInteger This is an integer specifying the maximum value the BAR1 base address can be. bar2_req_sizeInteger This is an integer specifying the required size of the BAR2 frame buffer region. bar2_base_addr_minInteger This is an integer specifying the minimum value the BAR2 base address can be. bar2_base_addr_maxInteger This is an integer specifying the maximum value the BAR2 base address can be. bar4_req_sizeInteger This is an integer specifying the required size of the BAR4 frame buffer region. bar4_base_addr_minInteger This is an integer specifying the minimum value the BAR4 base address can be. bar4_base_addr_maxInteger This is an integer specifying the maximum value the BAR4 base address can be. bar5_req_sizeInteger This is an integer specifying the required size of the BAR5 frame buffer region.

  Output
Module specific output keys are described in the table below:
 Output Key TypeDescription bar1_sizeIntegerThe actual size of BAR1. bar1_base_addrIntegerThe actual base address of BAR1 memory. bar2_sizeIntegerThe actual size of BAR2. bar2_base_addrIntegerThe actual base address of BAR2 memory. bar4_sizeIntegerThe actual size of BAR4. bar4_base_addrIntegerThe actual base address of BAR4 memory. bar5_sizeIntegerThe actual size of BAR5. passString 'true' if all of the properties match the values given, 'false' otherwise.
The qualification check will query the specified bar properties and check that they satisfy the give parameters. The pass output key will be true and the test will pass if all of the BAR properties satisfy the constraints. After the check is finished, the following informational messages will be generated:
[INFO  ][<timestamp>][<action name>] smqt bar1_size <bar1_size>
[INFO  ][<timestamp>][<action name>] smqt bar1_base_addr <bar1_base_addr>
[INFO  ][<timestamp>][<action name>] smqt bar2_size <bar2_size>
[INFO  ][<timestamp>][<action name>] smqt bar2_base_addr <bar2_base_addr>
[INFO  ][<timestamp>][<action name>] smqt bar4_size <bar4_size>
[INFO  ][<timestamp>][<action name>] smqt bar4_base_addr <bar4_base_addr>
[INFO  ][<timestamp>][<action name>] smqt bar5_size <bar5_size>
[RESULT][<timestamp>][<action name>] smqt <pass>

  Examples
Example 1:
Consider this file (sizes are in bytes):
actions:
- name: action_1
  device: all
  module: smqt
  bar1_req_size: 17179869184
  bar1_base_addr_min: 0
  bar1_base_addr_max: 17592168044416
  bar2_req_size: 2097152
  bar2_base_addr_min: 0
  bar2_base_addr_max: 1099511627776
  bar4_req_size: 262144
  bar4_base_addr_min: 0
  bar4_base_addr_max: 17592168044416
  bar5_req_size: 131072
Results for three GPUs are:
[INFO  ] [257936.568768] [action_1]  smqt bar1_size      17179869184 (16.00 GB)
[INFO  ] [257936.568768] [action_1]  smqt bar1_base_addr 13C0000000C
[INFO  ] [257936.568768] [action_1]  smqt bar2_size      2097152 (2.00 MB)
[INFO  ] [257936.568768] [action_1]  smqt bar2_base_addr 13B0000000C
[INFO  ] [257936.568768] [action_1]  smqt bar4_size      524288 (512.00 KB)
[INFO  ] [257936.568768] [action_1]  smqt bar4_base_addr E4B00000
[INFO  ] [257936.568768] [action_1]  smqt bar5_size      0 (0.00 B)
[RESULT] [257936.568920] [action_1]  smqt fail
[INFO  ] [257936.569234] [action_1]  smqt bar1_size      17179869184 (16.00 GB)
[INFO  ] [257936.569234] [action_1]  smqt bar1_base_addr 1A00000000C
[INFO  ] [257936.569234] [action_1]  smqt bar2_size      2097152 (2.00 MB)
[INFO  ] [257936.569234] [action_1]  smqt bar2_base_addr 19F0000000C
[INFO  ] [257936.569234] [action_1]  smqt bar4_size      524288 (512.00 KB)
[INFO  ] [257936.569234] [action_1]  smqt bar4_base_addr E9900000
[INFO  ] [257936.569234] [action_1]  smqt bar5_size      0 (0.00 B)
[RESULT] [257936.569281] [action_1]  smqt fail
[INFO  ] [257936.570798] [action_1]  smqt bar1_size      17179869184 (16.00 GB)
[INFO  ] [257936.570798] [action_1]  smqt bar1_base_addr 16C0000000C
[INFO  ] [257936.570798] [action_1]  smqt bar2_size      2097152 (2.00 MB)
[INFO  ] [257936.570798] [action_1]  smqt bar2_base_addr 1710000000C
[INFO  ] [257936.570798] [action_1]  smqt bar4_size      524288 (512.00 KB)
[INFO  ] [257936.570798] [action_1]  smqt bar4_base_addr E7300000
[INFO  ] [257936.570798] [action_1]  smqt bar5_size      0 (0.00 B)
[RESULT] [257936.570837] [action_1]  smqt fail
In this example, BAR sizes reported by GPUs match those listed in configuration key except for the BAR5, hence the test fails.

  PBQT module
The P2P Qualification Tool is designed to provide the list of all GPUs that support P2P and characterize the P2P links between peers. In addition to testing for P2P compatibility, this test will perform a peer-to-peer throughput test between all unique P2P pairs for performance evaluation. These are known as device-to-device transfers, and can be either uni-directional or bi-directional. The average bandwidth obtained is reported to help debug low bandwidth issues.

  Module specific keys
 Config Key TypeDescription peersCollection of Strings This is a required key, and specifies the set of GPU(s) considered being peers of the GPU specified in the action. If ‘all’ is specified, all other GPU(s) on the system will be considered peers. Otherwise only the GPU ids specified in the list will be considered. peer_deviceidInteger This is an optional parameter, but if specified it restricts the peers list to a specific device type corresponding to the deviceid. test_bandwidthBool If this key is set to true the P2P bandwidth benchmark will run if a pair of devices pass the P2P check. bidirectionalBool This option is only used if test_bandwidth key is true. This specifies the type of transfer to run:
•   true – Do a bidirectional transfer test
•   false – Do a unidirectional transfer test from one node to another.
parallelBool This option is only used if the test_bandwidth key is true.
•   true – Run all test transfers in parallel.
•   false – Run test transfers one by one.
durationInteger This option is only used if test_bandwidth is true. This key specifies the duration a transfer test should run, given in milliseconds. If this key is not specified, the default value is 10000 (10 seconds). log_intervalInteger This option is only used if test_bandwidth is true. This is a positive integer, given in milliseconds, that specifies an interval over which the moving average of the bandwidth will be calculated and logged. The default value is 1000 (1 second). It must be smaller than the duration key.
if this key is 0 (zero), results are displayed as soon as the test transfer is completed. block_sizeCollection of Integers Optional. Defines list of block sizes to be used in transfer tests.
If 'all' or missing list of block sizes used in rocm_bandwidth_test is used:
•   1 * 1024
•   2 * 1024
•   4 * 1024
•   8 * 1024
•   16 * 1024
•   32 * 1024
•   64 * 1024
•   128 * 1024
•   256 * 1024
•   512 * 1024
•   1 * 1024 * 1024
•   2 * 1024 * 1024
•   4 * 1024 * 1024
•   8 * 1024 * 1024
•   16 * 1024 * 1024
•   32 * 1024 * 1024
•   64 * 1024 * 1024
•   128 * 1024 * 1024
•   256 * 1024 * 1024
•   512 * 1024 * 1024
b2b_block_sizeInteger This option is only used if both 'test_bandwidth' and 'parallel' keys are true. This is a positive integer indicating size in Bytes of a data block to be transferred continuously ('back-to-back') for the duration of one test pass. If the key is not present, ordinary transfers with size indicated in 'block_size' key will be performed. link_typeInteger This is a positive integer indicating type of link to be included in bandwidth test. Numbering follows that listed in hsa_amd_link_info_type_t in hsa_ext_amd.h file.
b2bBool If true, transfers are run back-to-back continuously for the full test duration. Only applicable when using the native transfer method. If not specified the default value is false.
hot_callsInteger Number of timed transfer iterations used to measure bandwidth. If not specified the default value is 1.
warm_callsInteger Number of warm-up transfer iterations run before timing begins. Warm-up iterations are not included in the bandwidth measurement. If not specified the default value is 1.
transfer_methodString Transfer backend to use. Accepted values: native – Use the ROCm native SDMA transfer path (default). transferbench – Use the TransferBench library as the transfer backend. If not specified the default is native.
transferbench_testString TransferBench test type. Only applicable when transfer_method: transferbench. Accepted values: p2p – Point-to-point transfer test (default). alltoall – All-to-all transfer test across all participating GPUs. If not specified the default is p2p.
executorString Execution engine used by TransferBench. Only applicable when transfer_method: transferbench. Accepted values: gfx – Use GPU shader kernels (default). dma – Use the DMA engine. If not specified the default is gfx.
subexecutorInteger Number of sub-executors (wavefronts or threads) per transfer. Only applicable when transfer_method: transferbench. If not specified the default value is 1.
gfx_unrollInteger Unroll factor for the GFX shader kernel. Only applicable when transfer_method: transferbench and executor: gfx. If not specified the default value is 4.
use_remote_readInteger If set to 1, transfers use remote read instead of remote write. If not specified the default value is 0 (remote write).
a2a_modeInteger All-to-all transfer pattern mode. Only applicable when transferbench_test: alltoall. If not specified the default value is 0.
a2a_directInteger If set to 1, enables direct peer-to-peer transfers in the all-to-all test. Only applicable when transferbench_test: alltoall. If not specified the default value is 1.
a2a_localInteger If set to 1, includes local (same-GPU) transfers in the all-to-all test. Only applicable when transferbench_test: alltoall. If not specified the default value is 0.
a2a_num_gpusInteger Number of GPUs to include in the all-to-all test. A value of 0 means all detected GPUs are included. Only applicable when transferbench_test: alltoall. If not specified the default value is 0.
Suitable values for log_interval and duration depend on your system.
•   log_interval, in sequential mode, should be long enough to allow all transfer tests to finish at least once or '(pending)' and '(*)' will be displayed (see below). Number of transfers depends on number of peer NUMA nodes in your system. In parallel mode, it should be roughly 1.5 times the duration of single longest individual test.
•   duration, regardless of mode should be at least, 4 * log_interval.
You may obtain indication of how long single transfer between two NUMA nodes take by running test with '-d 4' switch and observing DEBUG messages for transfer start/finish. An output may look like this:
[DEBUG ] [183940.634118] [action_1] pbqt transfer 6 5 start
[DEBUG ] [183941.311671] [action_1] pbqt transfer 6 5 finish
[DEBUG ] [183941.312746] [action_1] pbqt transfer 4 5 start
[DEBUG ] [183941.990174] [action_1] pbqt transfer 4 5 finish
[DEBUG ] [183941.991244] [action_1] pbqt transfer 4 6 start
[DEBUG ] [183942.668687] [action_1] pbqt transfer 4 6 finish
[DEBUG ] [183942.669756] [action_1] pbqt transfer 5 4 start
[DEBUG ] [183943.340957] [action_1] pbqt transfer 5 4 finish
[DEBUG ] [183943.342037] [action_1] pbqt transfer 5 6 start
[DEBUG ] [183944.17957 ] [action_1] pbqt transfer 5 6 finish
[DEBUG ] [183944.19032 ] [action_1] pbqt transfer 6 4 start
[DEBUG ] [183944.700868] [action_1] pbqt transfer 6 4 finish
From this printout, it can be concluded that single transfer takes on average 800ms. Values for log_interval and duration should be set accordingly.

  Output
Module specific output keys are described in the table below:
 Output Key TypeDescription p2p_resultBool Indicates if the gpu and the specified peer have P2P capabilities. If this quantity is true, the GPU pair tested has p2p capabilities. If false, they are not peers. distanceInteger NUMA distance for these two peers hop_typeString Link type for each link hop (e.g., PCIe, HyperTransport, QPI, ...) hop_distanceInteger NUMA distance for this particular hop transfer_idString String with format '<transfer_index>/<transfer_number>' where
•   transfer_index - is number, starting from 1, for each device-peer combination
•   transfer_number - is total number of device-peer combinations
interval_bandwidthFloat The average bandwidth of a p2p transfer, during the log_interval time period.
 This field may also take values:
•   (pending) - this means that no measurement has taken place yet.
•   xxxGBps (*) - this means no measurement within current log_interval but average from previous measurements is displayed.
bandwidthFloat The average bandwidth of a p2p transfer, averaged over the entire test duration of the interval. This field may also take value:
•   (not measured) - this means no test transfer completed for those peers. You may need to increase test duration.
durationFloat Cumulative duration of all transfers between the two particular nodes
If the value of test_bandwidth key is false, the tool will only try to determine if the GPU(s) in the peers key are P2P to the action’s GPU. In this case the bidirectional and log_interval values will be ignored, if they are specified. If a gpu is a P2P peer to the device the test will pass, otherwise it will fail. A message indicating the result will be provided for each GPUs specified. It will have the following format:
[RESULT][<timestamp>][<action name>] p2p <gpu id> <peer gpu id> peers:<p2p_result> distance:<distance> <hop_type>:<hop_dist>[ <hop_type>:<hop_dist>]
If the value of test_bandwidth is true bandwidth testing between the device and each of its peers will take place in parallel or in sequence, depending on the value of the parallel flag. During the duration of bandwidth benchmarking, informational output providing the moving average of the transfer’s bandwidth will be calculated and logged at every time increment specified by the log_interval parameter. The messages will have the following output:
[INFO  ][<timestamp>][<action name>] p2p-bandwidth [<transfer_id>] <gpu id> <peer gpu id> bidirectional: <bidirectional> <interval_bandwidth>
At the end of the test the average bytes/second will be calculated over the entire test duration, and will be logged as a result:
[RESULT][<timestamp>][<action name>] p2p-bandwidth [<transfer_id>] <gpu id> <peer gpu id> bidirectional: <bidirectional> <bandwidth> <duration>

  Examples
Example 1:
Here all source GPUs (device: all) with all destination GPUs (peers: all) are tested for p2p capability with no bandwidth testing (test_bandwidth: false).
actions:
- name: action_1
  device: all
  module: pbqt
  peers: all
  test_bandwidth: false
Possible result is:
[RESULT] [1656631.262875] [action_1] p2p 3254 3254 peers:false distance:-1
[RESULT] [1656631.262968] [action_1] p2p 3254 50599 peers:true distance:56 HyperTransport:56
[RESULT] [1656631.263039] [action_1] p2p 3254 33367 peers:true distance:56 HyperTransport:56
[RESULT] [1656631.263103] [action_1] p2p 50599 3254 peers:true distance:56 HyperTransport:56
[RESULT] [1656631.263151] [action_1] p2p 50599 50599 peers:false distance:-1
[RESULT] [1656631.263203] [action_1] p2p 50599 33367 peers:true distance:56 HyperTransport:56
[RESULT] [1656631.263265] [action_1] p2p 33367 3254 peers:true distance:56 HyperTransport:56
[RESULT] [1656631.263321] [action_1] p2p 33367 50599 peers:true distance:56 HyperTransport:56
[RESULT] [1656631.263360] [action_1] p2p 33367 33367 peers:false distance:-1
From the first line of result, we can see that GPU (ID 3254) can't access itself. From the second line of result, we can see that source GPU (ID 3254) can access destination GPU (ID 50599).
Example 2:
Here all source GPUs (device: all) with all destination GPUs (peers: all) are tested for p2p capability including bandwidth testing (test_bandwidth: true) with bidirectional transfers (bidirectional: true) and with immediate output for each completed transfer (log_interval: 0)
actions:
- name: action_1
  device: all
  module: pbqt
  log_interval: 0
  duration: 0
  peers: all
  test_bandwidth: true
  bidirectional: true
When run with '-d 3' switch, possible result is:
[RESULT] [1657122.364752] [action_1] p2p 3254 3254 peers:false distance:-1
[RESULT] [1657122.364845] [action_1] p2p 3254 50599 peers:true distance:56 HyperTransport:56
[RESULT] [1657122.364917] [action_1] p2p 3254 33367 peers:true distance:56 HyperTransport:56
[RESULT] [1657122.364985] [action_1] p2p 50599 3254 peers:true distance:56 HyperTransport:56
[RESULT] [1657122.365037] [action_1] p2p 50599 50599 peers:false distance:-1
[RESULT] [1657122.365094] [action_1] p2p 50599 33367 peers:true distance:56 HyperTransport:56
[RESULT] [1657122.365157] [action_1] p2p 33367 3254 peers:true distance:56 HyperTransport:56
[RESULT] [1657122.365221] [action_1] p2p 33367 50599 peers:true distance:56 HyperTransport:56
[RESULT] [1657122.365270] [action_1] p2p 33367 33367 peers:false distance:-1
[INFO  ] [1657123.644203] [action_1] p2p-bandwidth  [1/6] 3254 50599  bidirectional: true  7.013 GBps
[INFO  ] [1657123.644376] [action_1] p2p-bandwidth  [2/6] 3254 33367  bidirectional: true  6.615 GBps
[INFO  ] [1657123.644453] [action_1] p2p-bandwidth  [3/6] 50599 3254  bidirectional: true  2.367 GBps
[INFO  ] [1657123.644522] [action_1] p2p-bandwidth  [4/6] 50599 33367  bidirectional: true  7.504 GBps
[INFO  ] [1657123.644590] [action_1] p2p-bandwidth  [5/6] 33367 3254  bidirectional: true  8.207 GBps
[INFO  ] [1657123.644673] [action_1] p2p-bandwidth  [6/6] 33367 50599  bidirectional: true  7.680 GBps
[INFO  ] [1657124.926221] [action_1] p2p-bandwidth  [1/6] 3254 50599  bidirectional: true  6.646 GBps
[INFO  ] [1657124.926368] [action_1] p2p-bandwidth  [2/6] 3254 33367  bidirectional: true  8.418 GBps
[INFO  ] [1657124.926438] [action_1] p2p-bandwidth  [3/6] 50599 3254  bidirectional: true  7.402 GBps
[INFO  ] [1657124.926506] [action_1] p2p-bandwidth  [4/6] 50599 33367  bidirectional: true  6.161 GBps
[INFO  ] [1657124.926573] [action_1] p2p-bandwidth  [5/6] 33367 3254  bidirectional: true  9.024 GBps
[INFO  ] [1657124.926640] [action_1] p2p-bandwidth  [6/6] 33367 50599  bidirectional: true  8.740 GBps
[INFO  ] [1657126.208742] [action_1] p2p-bandwidth  [1/6] 3254 50599  bidirectional: true  5.680 GBps
[INFO  ] [1657126.208905] [action_1] p2p-bandwidth  [2/6] 3254 33367  bidirectional: true  8.011 GBps
[INFO  ] [1657126.208990] [action_1] p2p-bandwidth  [3/6] 50599 3254  bidirectional: true  3.918 GBps
[INFO  ] [1657126.209066] [action_1] p2p-bandwidth  [4/6] 50599 33367  bidirectional: true  6.058 GBps
[INFO  ] [1657126.209140] [action_1] p2p-bandwidth  [5/6] 33367 3254  bidirectional: true  6.650 GBps
[INFO  ] [1657126.209213] [action_1] p2p-bandwidth  [6/6] 33367 50599  bidirectional: true  0.000 GBps
[RESULT] [1657126.742128] [action_1] p2p-bandwidth  [1/6] 3254 50599  bidirectional: true  5.767 GBps  duration: 0.368453 sec
[RESULT] [1657126.743287] [action_1] p2p-bandwidth  [2/6] 3254 33367  bidirectional: true  6.013 GBps  duration: 0.498944 sec
[RESULT] [1657126.744411] [action_1] p2p-bandwidth  [3/6] 50599 3254  bidirectional: true  5.278 GBps  duration: 0.380393 sec
[RESULT] [1657126.745534] [action_1] p2p-bandwidth  [4/6] 50599 33367  bidirectional: true  4.160 GBps  duration: 0.484577 sec
[RESULT] [1657126.746684] [action_1] p2p-bandwidth  [5/6] 33367 3254  bidirectional: true  5.219 GBps  duration: 0.407190 sec
[RESULT] [1657126.747827] [action_1] p2p-bandwidth  [6/6] 33367 50599  bidirectional: true  4.001 GBps  duration: 0.562350 sec
We can see that on this particular machine there are three GPUs and six possible device-to-peer transfers.
Example 3:
Here some source GPUs (device: 50599) are targeting some destination GPUs (peers: 33367 3254) with specified log interval (log_interval: 1000) and duration (duration: 5000). Bandwidth is tested (test_bandwidth: true) but only unidirectional (bidirectional: false) without parallel execution (parallel: false).
actions:
- name: action_1
  device: 50599
  module: pbqt
  log_interval: 1000
  duration: 5000
  count: 0
  peers: 33367 3254
  test_bandwidth: true
  bidirectional: false
  parallel: false
Possible output is:
[RESULT] [1657218.801555] [action_1] p2p 50599 3254 peers:true distance:56 HyperTransport:56
[RESULT] [1657218.801655] [action_1] p2p 50599 33367 peers:true distance:56 HyperTransport:56
[INFO  ] [1657219.871532] [action_1] p2p-bandwidth  [1/2] 50599 3254  bidirectional: false  4.517 GBps
[INFO  ] [1657219.871717] [action_1] p2p-bandwidth  [2/2] 50599 33367  bidirectional: false  4.475 GBps
[INFO  ] [1657220.940263] [action_1] p2p-bandwidth  [1/2] 50599 3254  bidirectional: false  4.476 GBps
[INFO  ] [1657220.940461] [action_1] p2p-bandwidth  [2/2] 50599 33367  bidirectional: false  4.601 GBps
[INFO  ] [1657222.7589  ] [action_1] p2p-bandwidth  [1/2] 50599 3254  bidirectional: false  4.488 GBps
[INFO  ] [1657222.7760  ] [action_1] p2p-bandwidth  [2/2] 50599 33367  bidirectional: false  4.470 GBps
[INFO  ] [1657223.74647 ] [action_1] p2p-bandwidth  [1/2] 50599 3254  bidirectional: false  4.666 GBps
[INFO  ] [1657223.74810 ] [action_1] p2p-bandwidth  [2/2] 50599 33367  bidirectional: false  4.576 GBps
[RESULT] [1657224.181106] [action_1] p2p-bandwidth  [1/2] 50599 3254  bidirectional: false  4.539 GBps  duration: 1.321909 sec
[RESULT] [1657224.182255] [action_1] p2p-bandwidth  [2/2] 50599 33367  bidirectional: false  4.551 GBps  duration: 1.318517 sec
From the last line of result, we can see that source GPU (ID 50599) can access destination GPU (ID 33367) and that the bandwidth is 4.495 GBps.
Example 4:
Here, all GPUs are targeted with bidirectional transfers and parallel execution of tests:
actions:
- name: action_1
  device: all
  module: pbqt
  log_interval: 1200
  duration: 4000
  peers: all
  test_bandwidth: true
  bidirectional: true
  parallel: true
Possible output is:
[RESULT] [1657295.937184] [action_1] p2p 3254 3254 peers:false distance:-1
[RESULT] [1657295.937267] [action_1] p2p 3254 50599 peers:true distance:56 HyperTransport:56
[RESULT] [1657295.937324] [action_1] p2p 3254 33367 peers:true distance:56 HyperTransport:56
[RESULT] [1657295.937379] [action_1] p2p 50599 3254 peers:true distance:56 HyperTransport:56
[RESULT] [1657295.937429] [action_1] p2p 50599 50599 peers:false distance:-1
[RESULT] [1657295.937482] [action_1] p2p 50599 33367 peers:true distance:56 HyperTransport:56
[RESULT] [1657295.937543] [action_1] p2p 33367 3254 peers:true distance:56 HyperTransport:56
[RESULT] [1657295.937607] [action_1] p2p 33367 50599 peers:true distance:56 HyperTransport:56
[RESULT] [1657295.937655] [action_1] p2p 33367 33367 peers:false distance:-1
[INFO  ] [1657297.216212] [action_1] p2p-bandwidth  [1/6] 3254 50599  bidirectional: true  4.972 GBps
[INFO  ] [1657297.216351] [action_1] p2p-bandwidth  [2/6] 3254 33367  bidirectional: true  8.183 GBps
[INFO  ] [1657297.216423] [action_1] p2p-bandwidth  [3/6] 50599 3254  bidirectional: true  8.911 GBps
[INFO  ] [1657297.216490] [action_1] p2p-bandwidth  [4/6] 50599 33367  bidirectional: true  7.690 GBps
[INFO  ] [1657297.216558] [action_1] p2p-bandwidth  [5/6] 33367 3254  bidirectional: true  7.768 GBps
[INFO  ] [1657297.216642] [action_1] p2p-bandwidth  [6/6] 33367 50599  bidirectional: true  4.589 GBps
[INFO  ] [1657298.487427] [action_1] p2p-bandwidth  [1/6] 3254 50599  bidirectional: true  8.778 GBps
[INFO  ] [1657298.487593] [action_1] p2p-bandwidth  [2/6] 3254 33367  bidirectional: true  7.921 GBps
[INFO  ] [1657298.487730] [action_1] p2p-bandwidth  [3/6] 50599 3254  bidirectional: true  8.164 GBps
[INFO  ] [1657298.487807] [action_1] p2p-bandwidth  [4/6] 50599 33367  bidirectional: true  8.921 GBps
[INFO  ] [1657298.487878] [action_1] p2p-bandwidth  [5/6] 33367 3254  bidirectional: true  8.487 GBps
[INFO  ] [1657298.487956] [action_1] p2p-bandwidth  [6/6] 33367 50599  bidirectional: true  7.648 GBps
[INFO  ] [1657299.760175] [action_1] p2p-bandwidth  [1/6] 3254 50599  bidirectional: true  7.210 GBps
[INFO  ] [1657299.760249] [action_1] p2p-bandwidth  [2/6] 3254 33367  bidirectional: true  4.274 GBps
[INFO  ] [1657299.760284] [action_1] p2p-bandwidth  [3/6] 50599 3254  bidirectional: true  0.000 GBps
[INFO  ] [1657299.760318] [action_1] p2p-bandwidth  [4/6] 50599 33367  bidirectional: true  5.942 GBps
[INFO  ] [1657299.760349] [action_1] p2p-bandwidth  [5/6] 33367 3254  bidirectional: true  0.001 GBps
[INFO  ] [1657299.760381] [action_1] p2p-bandwidth  [6/6] 33367 50599  bidirectional: true  5.490 GBps
[RESULT] [1657300.293126] [action_1] p2p-bandwidth  [1/6] 3254 50599  bidirectional: true  6.964 GBps  duration: 0.287248 sec
[RESULT] [1657300.294334] [action_1] p2p-bandwidth  [2/6] 3254 33367  bidirectional: true  3.960 GBps  duration: 0.536554 sec
[RESULT] [1657300.295528] [action_1] p2p-bandwidth  [3/6] 50599 3254  bidirectional: true  5.442 GBps  duration: 0.368977 sec
[RESULT] [1657300.296691] [action_1] p2p-bandwidth  [4/6] 50599 33367  bidirectional: true  4.187 GBps  duration: 0.477756 sec
[RESULT] [1657300.297840] [action_1] p2p-bandwidth  [5/6] 33367 3254  bidirectional: true  4.942 GBps  duration: 0.607009 sec
[RESULT] [1657300.299016] [action_1] p2p-bandwidth  [6/6] 33367 50599  bidirectional: true  3.828 GBps  duration: 0.523495 sec
It can be seen that transfers [2/6] and [5/6] did not take place in the second log interval so average from the previous cycle is displayed instead and marked with '(*)'

  PEBB module
The PCIe Bandwidth Benchmark attempts to saturate the PCIe bus with DMA transfers between system memory and a target GPU card’s memory. These are known as host-to-device or device- to-host transfers, and can be either unidirectional or bidirectional transfers. The maximum bandwidth obtained is reported.

  Module specific keys
 Config Key TypeDescription host_to_deviceBool This key indicates if host to device transfers will be considered. The default value is true. device_to_hostBool This key indicates if device to host transfers will be considered. The default value is true. parallelBool This option is only used if the test_bandwidth key is true.
•   true – Run all test transfers in parallel.
•   false – Run test transfers one by one.
durationInteger This option is only used if test_bandwidth is true. This key specifies the duration a transfer test should run, given in milliseconds. If this key is not specified, the default value is 10000 (10 seconds). log_intervalInteger This option is only used if test_bandwidth is true. This is a positive integer, given in milliseconds, that specifies an interval over which the moving average of the bandwidth will be calculated and logged. The default value is 1000 (1 second). It must be smaller than the duration key.
if this key is 0 (zero), results are displayed as soon as the test transfer is completed. block_sizeCollection of Integers Optional. Defines list of block sizes to be used in transfer tests.
If 'all' or missing list of block sizes used in rocm_bandwidth_test is used:
•   1 * 1024
•   2 * 1024
•   4 * 1024
•   8 * 1024
•   16 * 1024
•   32 * 1024
•   64 * 1024
•   128 * 1024
•   256 * 1024
•   512 * 1024
•   1 * 1024 * 1024
•   2 * 1024 * 1024
•   4 * 1024 * 1024
•   8 * 1024 * 1024
•   16 * 1024 * 1024
•   32 * 1024 * 1024
•   64 * 1024 * 1024
•   128 * 1024 * 1024
•   256 * 1024 * 1024
•   512 * 1024 * 1024
b2b_block_sizeInteger This option is only used if both 'test_bandwidth' and 'parallel' keys are true. This is a positive integer indicating size in Bytes of a data block to be transferred continuously ('back-to-back') for the duration of one test pass. If the key is not present, ordinary transfers with size indicated in 'block_size' key will be performed. link_typeInteger This is a positive integer indicating type of link to be included in bandwidth test. Numbering follows that listed in hsa_amd_link_info_type_t in hsa_ext_amd.h file.
b2bBool If true, transfers are run back-to-back continuously for the full test duration. Only applicable when using the native transfer method. If not specified the default value is false.
hot_callsInteger Number of timed transfer iterations used to measure bandwidth. If not specified the default value is 1.
warm_callsInteger Number of warm-up transfer iterations run before timing begins. Warm-up iterations are not included in the bandwidth measurement. If not specified the default value is 1.
transfer_methodString Transfer backend to use. Accepted values: native – Use the ROCm native SDMA transfer path (default). transferbench – Use the TransferBench library as the transfer backend. If not specified the default is native.
executorString Execution engine used by TransferBench. Only applicable when transfer_method: transferbench. Accepted values: gfx – Use GPU shader kernels (default). dma – Use the DMA engine. If not specified the default is gfx.
subexecutorInteger Number of sub-executors (wavefronts or threads) per transfer. Only applicable when transfer_method: transferbench. If not specified the default value is 1.
gfx_unrollInteger Unroll factor for the GFX shader kernel. Only applicable when transfer_method: transferbench and executor: gfx. If not specified the default value is 4.
source_memoryString Memory type for the transfer source. Only applicable when transfer_method: transferbench. Accepted values: cpu – System (host) memory. gpu – GPU device memory. null – No explicit memory allocation; let TransferBench decide (default). If not specified the default is null.
destination_memoryString Memory type for the transfer destination. Only applicable when transfer_method: transferbench. Accepted values: cpu – System (host) memory. gpu – GPU device memory. null – No explicit memory allocation; let TransferBench decide (default). If not specified the default is null.
Suitable values for log_interval and duration depend on your system.
•   log_interval, in sequential mode, should be long enough to allow all transfer tests to finish at least once or '(pending)' and '(*)' will be displayed (see below). Number of transfers depends on number of peer NUMA nodes in your system. In parallel mode, it should be roughly 1.5 times the duration of single longest individual test.
•   duration, regardless of mode should be at least, 4 * log_interval.
You may obtain indication of how long single transfer between two NUMA nodes take by running test with '-d 4' switch and observing DEBUG messages for transfer start/finish. An output may look like this:
[DEBUG ] [187024.729433] [action_1] pebb transfer 0 6 start
[DEBUG ] [187029.327818] [action_1] pebb transfer 0 6 finish
[DEBUG ] [187024.299150] [action_1] pebb transfer 1 6 start
[DEBUG ] [187029.473378] [action_1] pebb transfer 1 6 finish
[DEBUG ] [187023.227009] [action_1] pebb transfer 1 5 start
[DEBUG ] [187029.530203] [action_1] pebb transfer 1 5 finish
[DEBUG ] [187025.737675] [action_1] pebb transfer 3 5 start
[DEBUG ] [187030.134100] [action_1] pebb transfer 3 5 finish
[DEBUG ] [187027.19961 ] [action_1] pebb transfer 2 6 start
[DEBUG ] [187030.421181] [action_1] pebb transfer 2 6 finish
[DEBUG ] [187027.41475 ] [action_1] pebb transfer 2 5 start
[DEBUG ] [187031.293998] [action_1] pebb transfer 2 5 finish
[DEBUG ] [187027.71717 ] [action_1] pebb transfer 0 5 start
[DEBUG ] [187031.605326] [action_1] pebb transfer 0 5 finish
From this printout, it can be concluded that single transfer takes on average 5500ms. Values for log_interval and duration should be set accordingly.

  Output
Module specific output keys are described in the table below:
 Output Key TypeDescription CPU nodeInteger Particular CPU node involved in transfer distanceInteger NUMA distance for these two peers hop_typeString Link type for each link hop (e.g., PCIe, HyperTransport, QPI, ...) hop_distanceInteger NUMA distance for this particular hop transfer_idString String with format '<transfer_index>/<transfer_number>' where
•   transfer_index - is number, starting from 1, for each device-peer combination
•   transfer_number - is total number of device-peer combinations
interval_bandwidthFloat The average bandwidth of a CPU-to-GPU or GPU-to-CPU transfer, during the log_interval time period.
 This field may also take values:
•   (pending) - this means that no measurement has taken place yet.
•   xxxGBps (*) - this means no measurement within current log_interval but average from previous measurements is displayed.
bandwidthFloat The average bandwidth of a CPU-to-GPU or GPU-to-CPU transfer, averaged over the entire test duration. This field may also take value:
•   (not measured) - this means no test transfer completed for those peers. You may need to increase test duration.
durationFloat Cumulative duration of all transfers between the two particular nodes
At the beginning, the test will display link info for every CPU/GPU pair:
[RESULT][<timestamp>][<action name>] pcie-bandwidth [<transfer_id>] <cpu node> <gpu node> <gpu id> distance:<distance> <hop_type>:<hop_dist>[ <hop_type>:<hop_dist>]
During the execution of the benchmark, informational output providing the moving average of the bandwidth of the transfer will be calculated and logged. This interval is provided by the log_interval parameter and will have the following output format:
[INFO ][<timestamp>][<action name>] pcie-bandwidth [<transfer_id>] <cpu node> <gpu id> h2d: <host_to_device> d2h: <device_to_host> <interval_bandwidth>
At the end of test, the average bytes/second will be calculated over the entire test duration, and will be logged as a result:
[RESULT][<timestamp>][<action name>] pcie-bandwidth [<transfer_id>] <cpu node> <gpu id> h2d: <host_to_device> d2h: <device_to_host> <bandwidth> <duration>

  Examples
Example 1:
Consider action:
actions:
- name: action_1
  device: all
  module: pebb
  log_interval: 0
  duration: 0
  device_to_host: false
  host_to_device: true
  parallel: false
This will initiate host to device transfer to all GPUs with immediate output (parallel: false, log_interval: 0)
Output from this action might look like:
[RESULT] [1658774.978614] [action_1] pcie-bandwidth 0 4 3254  distance:36 HyperTransport:36
[RESULT] [1658774.978664] [action_1] pcie-bandwidth 1 4 3254  distance:20 PCIe:20
[RESULT] [1658774.978695] [action_1] pcie-bandwidth 2 4 3254  distance:36 HyperTransport:36
[RESULT] [1658774.978728] [action_1] pcie-bandwidth 3 4 3254  distance:36 HyperTransport:36
[RESULT] [1658774.978763] [action_1] pcie-bandwidth 0 5 50599  distance:36 HyperTransport:36
[RESULT] [1658774.978795] [action_1] pcie-bandwidth 1 5 50599  distance:36 HyperTransport:36
[RESULT] [1658774.978825] [action_1] pcie-bandwidth 2 5 50599  distance:20 PCIe:20
[RESULT] [1658774.978856] [action_1] pcie-bandwidth 3 5 50599  distance:36 HyperTransport:36
[RESULT] [1658774.978889] [action_1] pcie-bandwidth 0 6 33367  distance:36 HyperTransport:36
[RESULT] [1658774.978922] [action_1] pcie-bandwidth 1 6 33367  distance:36 HyperTransport:36
[RESULT] [1658774.978952] [action_1] pcie-bandwidth 2 6 33367  distance:36 HyperTransport:36
[RESULT] [1658774.978982] [action_1] pcie-bandwidth 3 6 33367  distance:20 PCIe:20
[INFO  ] [1658774.983743] [action_1] pcie-bandwidth  [1/12] 0 3254  h2d: true  d2h: false  12.233 GBps
[INFO  ] [1658774.988272] [action_1] pcie-bandwidth  [2/12] 1 3254  h2d: true  d2h: false  12.227 GBps
[INFO  ] [1658774.993197] [action_1] pcie-bandwidth  [3/12] 2 3254  h2d: true  d2h: false  11.770 GBps
[INFO  ] [1658774.998105] [action_1] pcie-bandwidth  [4/12] 3 3254  h2d: true  d2h: false  11.313 GBps
[INFO  ] [1658775.4457  ] [action_1] pcie-bandwidth  [5/12] 0 50599  h2d: true  d2h: false  12.218 GBps
[INFO  ] [1658775.9589  ] [action_1] pcie-bandwidth  [6/12] 1 50599  h2d: true  d2h: false  10.292 GBps
[INFO  ] [1658775.14627 ] [action_1] pcie-bandwidth  [7/12] 2 50599  h2d: true  d2h: false  10.456 GBps
[INFO  ] [1658775.19664 ] [action_1] pcie-bandwidth  [8/12] 3 50599  h2d: true  d2h: false  10.614 GBps
[INFO  ] [1658775.26210 ] [action_1] pcie-bandwidth  [9/12] 0 33367  h2d: true  d2h: false  12.222 GBps
[INFO  ] [1658775.31188 ] [action_1] pcie-bandwidth  [10/12] 1 33367  h2d: true  d2h: false  12.215 GBps
[INFO  ] [1658775.36137 ] [action_1] pcie-bandwidth  [11/12] 2 33367  h2d: true  d2h: false  12.219 GBps
[INFO  ] [1658775.41117 ] [action_1] pcie-bandwidth  [12/12] 3 33367  h2d: true  d2h: false  12.219 GBps
[RESULT] [1658775.42219 ] [action_1] pcie-bandwidth  [1/12] 0 3254  h2d: true  d2h: false  12.233 GBps  duration: 0.000780 sec
[RESULT] [1658775.42235 ] [action_1] pcie-bandwidth  [2/12] 1 3254  h2d: true  d2h: false  12.227 GBps  duration: 0.000780 sec
[RESULT] [1658775.42246 ] [action_1] pcie-bandwidth  [3/12] 2 3254  h2d: true  d2h: false  11.770 GBps  duration: 0.000810 sec
[RESULT] [1658775.42256 ] [action_1] pcie-bandwidth  [4/12] 3 3254  h2d: true  d2h: false  11.313 GBps  duration: 0.000843 sec
[RESULT] [1658775.42271 ] [action_1] pcie-bandwidth  [5/12] 0 50599  h2d: true  d2h: false  12.218 GBps  duration: 0.000781 sec
[RESULT] [1658775.42286 ] [action_1] pcie-bandwidth  [6/12] 1 50599  h2d: true  d2h: false  10.292 GBps  duration: 0.000927 sec
[RESULT] [1658775.42297 ] [action_1] pcie-bandwidth  [7/12] 2 50599  h2d: true  d2h: false  10.456 GBps  duration: 0.000912 sec
[RESULT] [1658775.42309 ] [action_1] pcie-bandwidth  [8/12] 3 50599  h2d: true  d2h: false  10.614 GBps  duration: 0.000898 sec
[RESULT] [1658775.42321 ] [action_1] pcie-bandwidth  [9/12] 0 33367  h2d: true  d2h: false  12.222 GBps  duration: 0.000780 sec
[RESULT] [1658775.42332 ] [action_1] pcie-bandwidth  [10/12] 1 33367  h2d: true  d2h: false  12.215 GBps  duration: 0.000781 sec
[RESULT] [1658775.42344 ] [action_1] pcie-bandwidth  [11/12] 2 33367  h2d: true  d2h: false  12.219 GBps  duration: 0.000780 sec
[RESULT] [1658775.42355 ] [action_1] pcie-bandwidth  [12/12] 3 33367  h2d: true  d2h: false  12.219 GBps  duration: 0.000780 sec
Example 2:
Consider action:
actions:
- name: action_1
  device: all
  module: pebb
  log_interval: 500
  duration: 5000
  device_to_host: true
  host_to_device: true
  parallel: true
Here, although parallel execution of transfers is requested, log_interval is to short for some transfers to complete. For them, cumulative average is displayed and marked with (*):
[RESULT] [1659672.517170] [action_1] pcie-bandwidth 0 4 3254  distance:36 HyperTransport:36
[RESULT] [1659672.517222] [action_1] pcie-bandwidth 1 4 3254  distance:20 PCIe:20
[RESULT] [1659672.517257] [action_1] pcie-bandwidth 2 4 3254  distance:36 HyperTransport:36
[RESULT] [1659672.517290] [action_1] pcie-bandwidth 3 4 3254  distance:36 HyperTransport:36
[RESULT] [1659672.517324] [action_1] pcie-bandwidth 0 5 50599  distance:36 HyperTransport:36
[RESULT] [1659672.517357] [action_1] pcie-bandwidth 1 5 50599  distance:36 HyperTransport:36
[RESULT] [1659672.517388] [action_1] pcie-bandwidth 2 5 50599  distance:20 PCIe:20
[RESULT] [1659672.517419] [action_1] pcie-bandwidth 3 5 50599  distance:36 HyperTransport:36
[RESULT] [1659672.517452] [action_1] pcie-bandwidth 0 6 33367  distance:36 HyperTransport:36
[RESULT] [1659672.517483] [action_1] pcie-bandwidth 1 6 33367  distance:36 HyperTransport:36
[RESULT] [1659672.517515] [action_1] pcie-bandwidth 2 6 33367  distance:36 HyperTransport:36
[RESULT] [1659672.517546] [action_1] pcie-bandwidth 3 6 33367  distance:20 PCIe:20
[INFO  ] [1659673.49782 ] [action_1] pcie-bandwidth  [1/12] 0 3254  h2d: true  d2h: true  1.489 GBps
[INFO  ] [1659673.49814 ] [action_1] pcie-bandwidth  [2/12] 1 3254  h2d: true  d2h: true  2.701 GBps
[INFO  ] [1659673.582639] [action_1] pcie-bandwidth  [1/12] 0 3254  h2d: true  d2h: true  1.489 GBps (*)
[INFO  ] [1659673.582686] [action_1] pcie-bandwidth  [2/12] 1 3254  h2d: true  d2h: true  16.367 GBps
[INFO  ] [1659673.582700] [action_1] pcie-bandwidth  [3/12] 2 3254  h2d: true  d2h: true  17.300 GBps
[INFO  ] [1659677.851697] [action_1] pcie-bandwidth  [1/12] 0 3254  h2d: true  d2h: true  16.793 GBps
[INFO  ] [1659677.851727] [action_1] pcie-bandwidth  [2/12] 1 3254  h2d: true  d2h: true  16.872 GBps (*)
[INFO  ] [1659677.851741] [action_1] pcie-bandwidth  [3/12] 2 3254  h2d: true  d2h: true  14.796 GBps (*)
[INFO  ] [1659677.851754] [action_1] pcie-bandwidth  [4/12] 3 3254  h2d: true  d2h: true  20.358 GBps
[INFO  ] [1659677.851770] [action_1] pcie-bandwidth  [5/12] 0 50599  h2d: true  d2h: true  15.632 GBps (*)
[INFO  ] [1659677.851828] [action_1] pcie-bandwidth  [6/12] 1 50599  h2d: true  d2h: true  14.541 GBps (*)
[RESULT] [1659678.148280] [action_1] pcie-bandwidth  [1/12] 0 3254  h2d: true  d2h: true  16.309 GBps  duration: 0.061316 sec
[RESULT] [1659678.148318] [action_1] pcie-bandwidth  [2/12] 1 3254  h2d: true  d2h: true  16.871 GBps  duration: 0.118547 sec
[RESULT] [1659678.148332] [action_1] pcie-bandwidth  [3/12] 2 3254  h2d: true  d2h: true  13.360 GBps  duration: 0.149705 sec
[RESULT] [1659678.148349] [action_1] pcie-bandwidth  [4/12] 3 3254  h2d: true  d2h: true  15.371 GBps  duration: 0.130115 sec
[RESULT] [1659678.148363] [action_1] pcie-bandwidth  [5/12] 0 50599  h2d: true  d2h: true  15.631 GBps  duration: 0.127954 sec
[RESULT] [1659678.148377] [action_1] pcie-bandwidth  [6/12] 1 50599  h2d: true  d2h: true  14.185 GBps  duration: 0.140989 sec
[RESULT] [1659678.148390] [action_1] pcie-bandwidth  [7/12] 2 50599  h2d: true  d2h: true  15.242 GBps  duration: 0.131245 sec
[RESULT] [1659678.148404] [action_1] pcie-bandwidth  [8/12] 3 50599  h2d: true  d2h: true  16.071 GBps  duration: 0.124452 sec
[RESULT] [1659678.148418] [action_1] pcie-bandwidth  [9/12] 0 33367  h2d: true  d2h: true  16.505 GBps  duration: 0.121178 sec
[RESULT] [1659678.148432] [action_1] pcie-bandwidth  [10/12] 1 33367  h2d: true  d2h: true  16.720 GBps  duration: 0.059807 sec
[RESULT] [1659678.148445] [action_1] pcie-bandwidth  [11/12] 2 33367  h2d: true  d2h: true  15.604 GBps  duration: 0.128168 sec
[RESULT] [1659678.148458] [action_1] pcie-bandwidth  [12/12] 3 33367  h2d: true  d2h: true  16.193 GBps  duration: 0.123525 sec
Please note that in link information results, some records could be marked with (R). This means, that communication is possible if initiated by the destination NUMA node HSA agent.

  GST module
The GPU Stress Test drives and measures the specified GPU(s) performance (GFLOPS) - by means of large matrix multiplications using GEMM operation types based computations like SGEMM/DGEMM/HGEMM (Single/Double-precision/Half-precision General Matrix Multiplication) or GEMM data types based computations like fp8, i8, fp16, bf16, fp32 or tf32 (xf32) via BLAS libraries like rocBLAS or hipBLASLt. The GPU stress module may be configured so it does not copy the host source matrix array to the GPU before every matrix multiplication. This allows the GPU performance to not be capped by device to host bandwidth transfers. The module calculates the GFLOPS performance for configured GEMM computation and checks if it meets configured performance target. The test passes if it achieves the target performance GFLOPS number during the duration of the test else reported as fail.
This module should be used in conjunction with the GPU Monitor, to watch for thermal, power and related anomalies while the target GPU(s) are under realistic load conditions. By setting the appropriate parameters a user can ensure that all GPUs in a node or cluster reach desired performance levels. Further analysis of the generated stats can also show variations in the required power, clocks or temperatures to reach these targets, and thus highlight GPUs or nodes that are operating less efficiently.

  Module specific keys
Module specific keys are described in the table below:
 Config Key TypeDescription target_stressFloat The maximum relative performance the GPU will attempt to achieve in gigaflops. This parameter is required. copy_matrixBool This parameter indicates if each operation should copy the matrix data to the GPU before executing. The default value is true. ramp_intervalInteger This is a time interval, specified in milliseconds, given to the test to reach the given target_stress gigaflops. The default value is 5000 (5 seconds). This time is counted against the duration of the test. If the target gflops, or stress, is not achieved in this time frame, the test will fail. If the target stress (gflops) is achieved the test will attempt to run for the rest of the duration specified by the action, sustaining the stress load during that time. toleranceFloat A value indicating how much the target_stress can fluctuate after the ramp period for the test to succeed. The default value is 0.05 (5%). max_violationsInteger The number of tolerance violations that can occur after the ramp_interval for the test to still pass. The default value is 0. log_intervalInteger This is a positive integer, given in milliseconds, that specifies an interval over which the moving average of the bandwidth will be calculated and logged. matrix_sizeInteger Sets all three matrix dimensions (M, N, and K) to the same value. Equivalent to setting matrix_size_a, matrix_size_b, and matrix_size_c to the same value. The default value is 5760. matrix_size_aInteger Number of rows of matrix A (the M dimension in GEMM). Overrides matrix_size for this dimension. The default value is 5760. matrix_size_bInteger Number of columns of matrix B (the N dimension in GEMM). Overrides matrix_size for this dimension. The default value is 5760. matrix_size_cInteger Inner (shared) dimension K of the GEMM operation (columns of A / rows of B). Overrides matrix_size for this dimension. The default value is 5760.
ops_typeString GEMM operation type. Accepted values: sgemm, dgemm, hgemm. If neither ops_type nor data_type is set, the module defaults to sgemm. Mutually exclusive with data_type.
data_typeString Data type for the GEMM computation. Accepted values include: fp4_r, fp6_r, fp8_r, bf16_r, fp16_r, fp32_r, tf32_r (xf32_r), i8_r. If not specified the module falls back to the type implied by ops_type.
out_data_typeString Output (result) data type, e.g. fp16_r, fp32_r. Only applicable when using hipBLASLt. If not specified the default matches the compute type.
compute_typeString Accumulation/compute type used internally by the BLAS library. If not specified the default is fp32_r.
blas_sourceString BLAS library backend to use. Accepted values: rocblas (default), hipblaslt.
hot_callsInteger Number of GEMM kernel invocations per measurement window used to amortise launch overhead. The default value is 1.
matrix_initString Matrix initialization method. Accepted values: default – Initialize with default pattern (default). trig – Initialize with trigonometric (sine/cosine) values. random – Initialize with random values.
transaInteger Transpose operation applied to matrix A before the GEMM call. 0 = no transpose (default), 1 = transpose.
transbInteger Transpose operation applied to matrix B before the GEMM call. 0 = no transpose, 1 = transpose (default).
alphaFloat Scalar multiplier applied to the product of matrices A and B in the GEMM operation (C = alpha * A * B + beta * C). The default value is 1.
betaFloat Scalar multiplier applied to matrix C in the GEMM operation. The default value is 1.
ldaInteger Leading dimension offset added to the computed leading dimension of matrix A. The default value is 0.
ldbInteger Leading dimension offset added to the computed leading dimension of matrix B. The default value is 0.
ldcInteger Leading dimension offset added to the computed leading dimension of matrix C. The default value is 0.
lddInteger Leading dimension offset added to the computed leading dimension of matrix D (output). The default value is 0.
scale_aString Scaling mode applied to matrix A. Used with low-precision types such as fp8. Accepted values include block. If not specified no scaling is applied.
scale_bString Scaling mode applied to matrix B. Used with low-precision types such as fp8. Accepted values include block. If not specified no scaling is applied.
rotatingInteger Size of the rotating buffer (in elements) used to prevent data from residing in cache between iterations, enabling cache-cold benchmarking. A value of 0 disables rotating buffers. The default value is 0.
gemm_modeString GEMM execution mode. Accepted values: '' or unset – Standard (single) GEMM (default). batched – Batched GEMM; use with batch_size. strided_batched – Strided batched GEMM; use with batch_size and stride_* keys.
batch_sizeInteger Number of GEMM operations in a batched or strided-batched call. Only applicable when gemm_mode is batched or strided_batched. The default value is 0.
stride_aInteger Stride (in elements) between consecutive matrices A in a strided-batched GEMM. Only applicable when gemm_mode: strided_batched. The default value is 0.
stride_bInteger Stride (in elements) between consecutive matrices B in a strided-batched GEMM. The default value is 0.
stride_cInteger Stride (in elements) between consecutive matrices C in a strided-batched GEMM. The default value is 0.
stride_dInteger Stride (in elements) between consecutive output matrices D in a strided-batched GEMM. The default value is 0.
self_checkBool If true, validates the GEMM result for correctness after each operation. Adds overhead; intended for debugging. The default value is false.
accuracy_checkBool If true, runs a numerical accuracy check after each GEMM operation. The default value is false.
error_injectBool If true, enables error injection mode to deliberately introduce errors into the computation for testing error detection. The default value is false.
error_freqInteger Frequency of error injection; specifies how often (every N operations) an error is injected. Only applicable when error_inject: true. The default value is 0.
error_countInteger Number of errors to inject per injection event. Only applicable when error_inject: true. The default value is 0.

  Output
Module specific output keys are described in the table below:
 Output Key TypeDescription target_stressTime Series Floats The average gflops over the last log interval. max_gflopsFloat The maximum sustained performance obtained by the GPU during the test. stress_violationsInteger The number of gflops readings that violated the tolerance of the test after the ramp interval. flops_per_opInteger Flops (floating point operations) per operation queued to the GPU queue. One operation is one call to SGEMM/DGEMM. bytes_copied_per_opInteger Number of bytes copied to the GPU per GEMM operation when copy_matrix is true. try_ops_per_secFloat Calculated number of ops/second necessary to achieve target gigaflops. passBool 'true' if the GPU achieves its desired sustained performance level.
An informational message will be emitted when the test starts execution:
[INFO ][<timestamp>][<action name>] gst <gpu id> start <target_stress> copy matrix: <copy_matrix>
During the execution of the test, informational output providing the moving average the GPU(s) gflops will be logged at each log_interval:
[INFO ][<timestamp>][<action name>] gst Gflops: <interval_gflops>
When the target gflops is achieved, the following message will be logged:
[INFO ][<timestamp>][<action name>] gst <gpu id> target achieved <target_stress>
If the target gflops, or stress, is not achieved in the ramp_interval provided, the test will terminate and the following message will be logged:
[INFO ][<timestamp>][<action name>] gst <gpu id> ramp time exceeded <ramp_time>
In this case the test will fail.
 If the target stress (gflops) is achieved the test will attempt to run for the rest of the duration specified by the action, sustaining the stress load during that time. If the stress level violates the bounds set by the tolerance level during that time a violation message will be logged:
[INFO ][<timestamp>][<action name>] gst <gpu id> stress violation <interval_gflops>
When the test completes, the following result message will be printed:
[RESULT][<timestamp>][<action name>] gst <gpu id> Gflop: <max_gflops> flops_per_op:<flops_per_op> bytes_copied_per_op: <bytes_copied_per_op> try_ops_per_sec: <try_ops_per_sec> pass: <pass>
The test will pass if the target_stress is reached before the end of the ramp_interval and the stress_violations value is less than the given max_violations value. Otherwise, the test will fail.

  Examples
When running the GST module, users should provide at least an action name, the module name (gst), a list of GPU IDs, the test duration and a target stress value (gigaflops). Thus, the most basic configuration file looks like this:
actions:
- name: action_gst_1
  module: gst
  device: all
  target_stress: 3500
  duration: 8000
For the above configuration file, all the missing configuration keys will have their default values (for example: copy_matrix=true, matrix_size=5760). For more information about the default values, see Common configuration keys and Configuration keys.
When the RVS tool runs against such a configuration file, it will do the following:
•   run the stress test on all available (and compatible) AMD GPUs, one after the other
•   log a start message containing the GPU ID, the target_stress and the value of the copy_matrix:
```
[INFO  ] [164337.932824] action_gst_1 gst 50599 start 3500.000000 copy matrix:true
```
•   emit, each log_interval (for example: 1000ms), a message containing the gigaflops value that the current GPU achieved:
```
[INFO  ] [164355.111207] action_gst_1 gst 33367 Gflops 3535.670231
```
•   log a message as soon as the current GPU reaches the given target_stress:
``` [INFO ] [164350.804843] action_gst_1 gst 33367 target achieved 500.000000 ```
•   log a ramp time exceeded message if the GPU was not able to reach the target_stress in the ramp_interval time frame (for example: 5000). In such a case, the test will also terminate:
```
[INFO  ] [164013.788870] action_gst_1 gst 3254 ramp time exceeded 5000
```
•   log the test result, when the stress test completes. The message contains the test's overall result and some other statistics according in Output keys:
```
[RESULT] [164355.647523] action_gst_1 gst 33367 Gflop: 4066.020766 flops_per_op: 382.205952x1e9 bytes_copied_per_op: 398131200 try_ops_per_sec: 9.157367 pass: TRUE
```
•   log a stress violation message when the current gigaflops (for the last log_interval, for example: 1000ms) violates the bounds set by the tolerance configuration key (for example: 0.1). Note that this message is not logged during the ramp_interval time frame:
```
[INFO  ] [164013.788870] action_gst_1 gst 3254 stress violation 2500
```
If a mandatory configuration key is missing, the RVS tool will log an error message and terminate the execution of the current module. For example, the following configuration file will cause the RVS to terminate with the following error message: `RVS-GST: action: action_gst_1 key 'target_stress' was not found`
actions:
- name: action_gst_1
  module: gst
  device: all
  duration: 8000
A more complex configuration file looks like this:
actions:
- name: action_1
  device: 50599 33367
  module: gst
  parallel: false
  count: 12
  wait: 100
  duration: 7000
  ramp_interval: 3000
  log_interval: 1000
  max_violations: 2
  copy_matrix: false
  target_stress: 5000
  tolerance: 0.07
  matrix_size: 5760
For this configuration file, the RVS tool will:
•   run the stress test only for the GPUs having the ID 50599 or 33367. To get all the available GPU IDs, run RVS tool with -g option
•   run the test on the selected GPUs, one after the other
•   run each test, 12 times
•   only copy the matrices to the GPUs at the beginning of the test
•   wait 100ms before each test execution
•   try to reach 5000 gflops in maximum 3000ms
•   if target_stress (5000) is achieved in the ramp_interval (3000 ms) it will attempt to run the test for the rest of the duration, sustaining the stress load during that time
•   allow a 7% target_stress tolerance (each target_stress violation will generate a stress violation message as shown in the first example)
•   allow only 2 target_stress violations. Exceeding the max_violations will not terminate the test, but RVS will mark the test result as 'fail'.
The output for such a configuration key may look like this:
[INFO  ] [172061.758830] action_1 gst 50599 start 5000.000000 copy
matrix:false
[INFO  ] [172063.547668] action_1 gst 50599 Gflops 6471.614725
[INFO  ] [172064.577715] action_1 gst 50599 target achieved 5000.000000
[INFO  ] [172065.609224] action_1 gst 50599 Gflops 5189.993529
[INFO  ] [172066.634360] action_1 gst 50599 Gflops 5220.373979
[INFO  ] [172067.659262] action_1 gst 50599 Gflops 5225.472000
[INFO  ] [172068.694305] action_1 gst 50599 Gflops 5169.935583
[RESULT] [172069.573967] action_1 gst 50599 Gflop: 6471.614725 flops_per_op:
382.205952x1e9 bytes_copied_per_op: 398131200 try_ops_per_sec: 13.081952 pass:
TRUE
[INFO  ] [172069.574369] action_1 gst 33367 start 5000.000000 copy
matrix:false
[INFO  ] [172071.409483] action_1 gst 33367 Gflops 6558.348080
[INFO  ] [172072.438104] action_1 gst 33367 target achieved 5000.000000
[INFO  ] [172073.465033] action_1 gst 33367 Gflops 5215.285895
[INFO  ] [172074.501571] action_1 gst 33367 Gflops 5164.945297
[INFO  ] [172075.529468] action_1 gst 33367 Gflops 5210.207720
[INFO  ] [172076.558102] action_1 gst 33367 Gflops 5205.139424
[RESULT] [172077.448182] action_1 gst 33367 Gflop: 6558.348080 flops_per_op:
382.205952x1e9 bytes_copied_per_op: 398131200 try_ops_per_sec: 13.081952 pass:
TRUE
When setting the parallel to true, RVS will run the stress tests on all selected GPUs in parallel and the output may look like this:
[INFO  ] [173381.407428] action_1 gst 50599 start 5000.000000 copy
matrix:false
[INFO  ] [173381.407744] action_1 gst 33367 start 5000.000000 copy
matrix:false
[INFO  ] [173383.245771] action_1 gst 33367 Gflops 6558.348080
[INFO  ] [173383.256935] action_1 gst 50599 Gflops 6484.532120
[INFO  ] [173384.274202] action_1 gst 33367 target achieved 5000.000000
[INFO  ] [173384.286014] action_1 gst 50599 target achieved 5000.000000
[INFO  ] [173385.301038] action_1 gst 33367 Gflops 5215.285895
[INFO  ] [173385.315794] action_1 gst 50599 Gflops 5200.080980
[INFO  ] [173386.337638] action_1 gst 33367 Gflops 5164.945297
[INFO  ] [173386.353274] action_1 gst 50599 Gflops 5159.964636
[INFO  ] [173387.365494] action_1 gst 33367 Gflops 5210.207720
[INFO  ] [173387.383437] action_1 gst 50599 Gflops 5195.032357
[INFO  ] [173388.401250] action_1 gst 33367 Gflops 5169.935583
[INFO  ] [173388.421599] action_1 gst 50599 Gflops 5154.993572
[RESULT] [173389.282710] action_1 gst 33367 Gflop: 6558.348080 flops_per_op:
382.205952x1e9 bytes_copied_per_op: 398131200 try_ops_per_sec: 13.081952 pass:
TRUE
[RESULT] [173389.305479] action_1 gst 50599 Gflop: 6484.532120 flops_per_op:
382.205952x1e9 bytes_copied_per_op: 398131200 try_ops_per_sec: 13.081952 pass:
TRUE
It is important that all the configuration keys will be adjusted/fine-tuned according to the actual GPUs and HW platform capabilities. For example, a matrix size of 5760 should fit the VEGA 10 GPUs while 8640 should work with the VEGA 20 GPUs.

  IET module
The Input EDPp Test can be used to characterize the peak power capabilities of a GPU (that is, TGP) for a sustained duration of time. This tool leverage GEMM workload to drive the compute load on the GPU and check whether the power consumed meets configured target power in watts. The GEMM compute workloads are also pre-configured. This verifies that the GPUs can sustain a power level for a reasonable amount of time without problems like thermal violations arising. The test passes if GPU power meets or crosses the target power during the duration of the test else reported as fail.
This module should be used in conjunction with the GPU Monitor, to watch for thermal, power and related anomalies while the target GPU(s) are under realistic load conditions. By setting the appropriate parameters a user can ensure that all GPUs in a node or cluster reach desired performance levels. Further analysis of the generated stats can also show variations in the required power, clocks or temperatures to reach these targets, and thus highlight GPUs or nodes that are operating less efficiently.

  Module specific keys
Module specific keys are described in the table below:
 Config Key TypeDescription target_powerFloat This is a floating point value specifying the target sustained power level for the test. ramp_intervalInteger This is a time interval, specified in milliseconds, given to the test to determine the compute load that will sustain the target power. The default value is 5000 (5 seconds). This time is counted against the duration of the test. toleranceFloat A value indicating how much the target_power can fluctuate after the ramp period for the test to succeed. The default value is 0 (any violation immediately fails the test). max_violationsInteger The number of tolerance violations that can occur after the ramp_interval for the test to still pass. The default value is 0. sample_intervalInteger The sampling rate for target_power values given in milliseconds. The default value is 1000 (1 second). log_intervalInteger This is a positive integer, given in milliseconds, that specifies an interval over which the moving average of the bandwidth will be calculated and logged.
cp_workloadBool If true, enables the GEMM compute workload to drive GPU power. This is the primary workload used to reach the target power level. The default value is true.
bw_workloadBool If true, enables a memory bandwidth kernel workload in addition to or instead of the GEMM workload. The default value is false.
wg_countInteger Number of GPU workgroups used for the bandwidth kernel. Only applicable when bw_workload: true. The default value is 80.
nt_loadsBool If true, the bandwidth kernel uses non-temporal loads that bypass the L2 cache, exercising memory bandwidth more directly. Only applicable when bw_workload: true. The default value is false.
matrix_sizeInteger Sets all three GEMM matrix dimensions (M, N, and K) to the same value. Equivalent to setting matrix_size_a, matrix_size_b, and matrix_size_c to the same value. The default value is 5760.
matrix_size_aInteger Number of rows of matrix A (the M dimension in GEMM). Overrides matrix_size for this dimension. Default is 0 (uses matrix_size).
matrix_size_bInteger Number of columns of matrix B (the N dimension in GEMM). Overrides matrix_size for this dimension. Default is 0 (uses matrix_size).
matrix_size_cInteger Inner (shared) dimension K of the GEMM operation. Overrides matrix_size for this dimension. Default is 0 (uses matrix_size).
ops_typeString GEMM operation type. Accepted values: sgemm, dgemm, hgemm. If neither ops_type nor data_type is set, the module defaults to sgemm. Mutually exclusive with data_type.
data_typeString Data type for the GEMM computation. Accepted values include: fp4_r, fp6_r, fp8_r, bf16_r, fp16_r, fp32_r, tf32_r (xf32_r), i8_r. If not specified the module falls back to the type implied by ops_type.
out_data_typeString Output (result) data type, e.g. fp16_r, fp32_r. Only applicable when using hipBLASLt. If not specified the default matches the compute type.
compute_typeString Accumulation/compute type used internally by the BLAS library. If not specified the default is fp32_r.
blas_sourceString BLAS library backend to use. Accepted values: rocblas (default), hipblaslt.
hot_callsInteger Number of GEMM kernel invocations per measurement window used to amortise launch overhead. The default value is 1.
matrix_initString Matrix initialization method. Accepted values: default – Initialize with default pattern (default). trig – Initialize with trigonometric (sine/cosine) values. random – Initialize with random values.
transaInteger Transpose operation applied to matrix A before the GEMM call. 0 = no transpose (default), 1 = transpose.
transbInteger Transpose operation applied to matrix B before the GEMM call. 0 = no transpose, 1 = transpose (default).
alphaFloat Scalar multiplier applied to the product of matrices A and B (C = alpha * A * B + beta * C). The default value is 1.
betaFloat Scalar multiplier applied to matrix C in the GEMM operation. The default value is 1.
ldaInteger Leading dimension offset for matrix A. The default value is 0.
ldbInteger Leading dimension offset for matrix B. The default value is 0.
ldcInteger Leading dimension offset for matrix C. The default value is 0.
lddInteger Leading dimension offset for matrix D (output). The default value is 0.
gemm_modeString GEMM execution mode. Accepted values: '' or unset – Standard (single) GEMM (default). batched – Batched GEMM; use with batch_size. strided_batched – Strided batched GEMM; use with batch_size and stride_* keys.
batch_sizeInteger Number of GEMM operations in a batched or strided-batched call. Only applicable when gemm_mode is batched or strided_batched. The default value is 0.
stride_aInteger Stride (in elements) between consecutive matrices A in a strided-batched GEMM. The default value is 0.
stride_bInteger Stride (in elements) between consecutive matrices B in a strided-batched GEMM. The default value is 0.
stride_cInteger Stride (in elements) between consecutive matrices C in a strided-batched GEMM. The default value is 0.
stride_dInteger Stride (in elements) between consecutive output matrices D in a strided-batched GEMM. The default value is 0.

  Output
Module specific output keys are described in the table below:
 Output Key TypeDescription current_powerTime Series Floats The current measured power of the GPU. power_violationsInteger The number of power readings that violated the tolerance of the test after the ramp interval. passBool 'true' if the GPU achieves its desired sustained power level in the ramp interval.

  Examples
Example 1:
A regular IET configuration file looks like this:
actions:
- name: action_1
  device: all
  module: iet
  parallel: false
  count: 2
  wait: 100
  duration: 10000
  ramp_interval: 5000
  sample_interval: 500
  log_interval: 500
  max_violations: 1
  target_power: 135
  tolerance: 0.1
  matrix_size: 5760
``{note}
•   when setting thedeviceconfiguration key toall`, the RVS will detect all the AMD compatible GPUs and run the test on all of them
•   the test will run 2 times on each GPU (count = 2)
•   only one power violation is allowed. If the total number of violations is bigger than 1 the IET test result will be marked as failed
When the RVS tool runs against such a configuration file, it will do the following:
- run the test on all AMD compatible GPUs
- log a start message containing the GPU ID and the `target_power`, for example:
 [INFO ] [167316.308057] action_1 iet 50599 start 135.000000 ```
•   emit, each log_interval (for example: 500ms), a message containing the power for the current GPU
``` [INFO ] [167319.266707] action_1 iet 50599 current power 136.878342 ```
•   log a message as soon as the current GPU reaches the given target_power
``` [INFO ] [167318.793062] action_1 iet 50599 target achieved 135.000000 ```
•   log a 'ramp time exceeded' message if the GPU was not able to reach the target_power in the ramp_interval time frame (for example: 5000ms). In such a case, the test will also terminate
``` [INFO ] [167648.832413] action_1 iet 50599 ramp time exceeded 5000 ```
•   log a 'power violation message' when the current power (for the last sample_interval, for example: 500ms) violates the bounds set by the tolerance configuration key (for example: 0.1). Please note that this message is never logged during the ramp_interval time frame
``` [INFO ] [161251.971277] action_1 iet 3254 power violation 73.783211 ```
•   log the test result, when the stress test completes.
``` [RESULT] [167305.260051] action_1 iet 33367 pass: TRUE ```
The output for such a configuration file may look like this:
[INFO ] [167261.27161 ] action_1 iet 33367 start 135.000000
[INFO ] [167263.516803] action_1 iet 33367 current power 136.934479
[INFO ] [167263.521355] action_1 iet 33367 target achieved 135.000000
[INFO ] [167264.16925 ] action_1 iet 33367 current power 138.421844
[INFO ] [167264.517018] action_1 iet 33367 current power 138.394608
[INFO ] [167271.518402] action_1 iet 33367 current power 139.231918
[RESULT] [167272.67686 ] action_1 iet 33367 pass: TRUE
[INFO ] [167272.68029 ] action_1 iet 3254 start 135.000000
[INFO ] [167274.552026] action_1 iet 3254 current power 139.363525
[INFO ] [167274.552059] action_1 iet 3254 target achieved 135.000000
[INFO ] [167275.52168 ] action_1 iet 3254 current power 138.661453
[INFO ] [167275.552241] action_1 iet 3254 current power 138.857635
[INFO ] [167282.553983] action_1 iet 3254 current power 140.069687
[RESULT] [167283.95763 ] action_1 iet 3254 pass: TRUE
[INFO ] [167283.96158 ] action_1 iet 50599 start 135.000000
[INFO ] [167285.532999] action_1 iet 50599 current power 137.205032
[INFO ] [167285.543084] action_1 iet 50599 target achieved 135.000000
[INFO ] [167286.33050 ] action_1 iet 50599 current power 136.137115
[INFO ] [167293.534672] action_1 iet 50599 current power 139.753464
[RESULT] [167294.131420] action_1 iet 50599 pass: TRUE
Example 2:
Another configuration file, which may raise some 'power violation' messages (due to the small tolerance value) looks like this
- name: action_1
  device: all
  module: iet
  parallel: false
  count: 1
  wait: 100
  duration: 8000
  ramp_interval: 5000
  sample_interval: 700
  log_interval: 700
  max_violations: 1
  target_power: 80
  tolerance: 0.06
  matrix_size: 5760
The output for such a configuration file may look like this:
[INFO ] [161236.677785] action_1 iet 33367 start 80.000000
[INFO ] [161239.350055] action_1 iet 33367 current power 84.186142
[INFO ] [161239.354542] action_1 iet 33367 target achieved 80.000000
[INFO ] [161241.450517] action_1 iet 33367 current power 77.001945
[INFO ] [161241.459600] action_1 iet 33367 power violation 75.163689
[INFO ] [161242.150642] action_1 iet 33367 current power 82.063576
[RESULT] [161245.698113] action_1 iet 33367 pass: TRUE
[INFO ] [161245.698525] action_1 iet 3254 start 80.000000
[INFO ] [161248.394003] action_1 iet 3254 current power 78.842796
[INFO ] [161248.418631] action_1 iet 3254 target achieved 80.000000
[INFO ] [161249.94149 ] action_1 iet 3254 current power 79.938454
[INFO ] [161249.794201] action_1 iet 3254 current power 76.511711
[INFO ] [161249.818803] action_1 iet 3254 power violation 74.279594
[INFO ] [161250.494263] action_1 iet 3254 current power 74.615120
[INFO ] [161254.117386] action_1 iet 3254 power violation 73.682312
[RESULT] [161254.738939] action_1 iet 3254 pass: FALSE
[INFO ] [161254.739387] action_1 iet 50599 start 80.000000
[INFO ] [161257.374079] action_1 iet 50599 current power 81.560165
[INFO ] [161257.392085] action_1 iet 50599 target achieved 80.000000
[INFO ] [161258.774304] action_1 iet 50599 current power 75.057304
[INFO ] [161262.974833] action_1 iet 50599 current power 80.200668
[RESULT] [161263.771631] action_1 iet 50599 pass: TRUE
•   All the missing configuration keys (if any) will have their default values. For more information about the default values please consult the dedicated sections (3.3 Common Configuration Keys and 13.1 Module specific keys).
•   If a mandatory configuration key is missing, the RVS tool will log an error message and terminate the execution of the current module. For example, if the target_power is missing, the RVS to terminate with the following error message: 'RVS-IET: action: action_1 key 'target_power' was not found'
•   All the configuration keys will be adjusted/fine-tuned according to the actual GPUs and HW platform capabilities.
•   For example, a matrix size of 5760 should fit the VEGA 10 GPUs while 8640 should work with the VEGA 20 GPUs
•   For smalltarget_power values (for example: 30-40W), the sample_interval should be increased, otherwise the IET may fail either to achieve the given target_power or to sustain it (for example: ramp_interval = 1500 for target_power = 40). In case there are problems reaching/sustaining the given target_power:
•   Increase the ramp_interval and/or the tolerance value(s) and try again (in case of a 'ramp time exceeded' message)
•   Increase the tolerance value (in case too many 'power violation message' are logged out)

  Pulse Module
This module is in beta and is not intended for production use. Pass/fail criteria (especially around power-delta enforcement, clock-pinning verification, and throttle detection) are still being tuned and may change between releases.
The Pulse module is intended for time-varying GPU power stress: it alternates high phases (maximum clocks plus continuous GEMM) and low phases (minimum clocks plus idle/sleep), repeating for the action duration. That produces periodic power swings useful for exercising PSU transient response and platform power delivery, complementing IET (which targets a sustained power level).
GEMM type, matrix size, and BLAS backend follow the same concepts as GST / IET (see those sections and rvs_blas). Sample reference configuration: rvs/conf/pulse_single.conf.

  Behavior summary
•   Each pulse cycle is one high phase followed by one low phase. Phase lengths derive from pulse_rate (Hz) and high_phase_ratio; each phase is at least 10 ms after rounding.
•   High phase: sets GPU clocks high, runs GEMM in a loop (workload_iterations per inner batch) until the high-phase time budget elapses, samples power, checks junction temperature (failure if above 105°C).
•   After the high phase, the module calls hipDeviceSynchronize so work drains before the low phase (clearer separation of high vs. low power).
•   Low phase: sets clocks low, sleeps briefly between sample_interval-style sampling (5 ms sleep steps in the implementation), samples power.
•   With parallel: true and more than one GPU, threads coordinate with a std::barrier and a small GPU kernel using fine-grained coherent host memory so all GPUs synchronize across the pulse loop (avoids deadlock on shutdown via a shared done flag).
•   Pass: for primary MCM dies, the action passes if at least one pulse completed, there was no BLAS enqueue/sync failure (unless halt_on_error stops earlier), and no thermal violation; secondary MCM GPUs are treated as pass by default. Fail otherwise.

  Prerequisites
•   ROCm with hipBLASLt and rocBLAS available as for the rest of RVS (the rvs binary links hipblaslt; GEMM code lives in rvslib).
•   AMD SMI initialized for power and temperature queries (same family of requirements as other SMI-based modules). Elevated privileges are often required for clock control and power metrics.
•   For hipBLASLt, data_type must be valid (for example, fp32_r with sgemm, fp16_r with hgemm and compute_type such as fp32_r). If data_type is omitted with blas_source: hipblaslt, the module infers fp32_r/fp64_r/fp16_r from ops_type/sgemm/dgemm/hgemm respectively; for dgemm with hipBLASLt, if compute_type is still the default fp32_r, it is adjusted to fp64_r.

  Module specific keys
Keys below are in addition to common keys (name, module, device, duration, parallel, log_interval, count, wait and so on. For more information, see Common configuration keys.
Config Key TypeDescription pulse_rateIntegerPulse frequency in Hertz (cycles per second). Default 2. Use roughly 1–5 Hz if you want power readings to show clear high/low separation; very high rates shorten phases below typical GPU power-state and SMI averaging windows, so reported averages may converge. high_phase_ratioFloatFraction of each cycle spent in the high (GEMM) phase, 0.0–1.0. Default 0.5. Lower values spend more time in the low phase. matrix_sizeIntegerSquare GEMM dimension M = N = K. Default 4096. Larger matrices increase compute and power draw during the high phase. ops_typeStringGEMM operation flavor (e.g. sgemm, dgemm, hgemm). Default sgemm. Must match data_type / compute_type when using hipBLASLt. data_typeStringMatrix data type string for hipBLASLt (e.g. fp32_r, fp16_r). Default empty (rocBLAS can infer from ops_type alone). out_data_typeStringOptional output type for GEMM; default empty (same as input type where applicable). compute_typeStringhipBLASLt compute type (e.g. fp32_r). Default fp32_r. blas_sourceStringrocblas or hipblaslt. Default rocblas. alphaFloatGEMM scalar α. Default 2.0. betaFloatGEMM scalar β. Default -1.0. transaIntegerTranspose A: 0 no transpose, 1 transpose. Default 0. transbIntegerTranspose B: 0 no transpose, 1 transpose. Default 1. ldaIntegerLeading dimension offset A (0 = use minimum). Defaults 0. ldbIntegerLeading dimension offset B. Default 0. ldcIntegerLeading dimension offset C. Default 0. lddIntegerLeading dimension offset D. Default 0. matrix_initStringHost matrix initialization (e.g. default, hiprand). Default default. workload_iterationsIntegerGEMM calls per inner batch in the high phase before re-checking time and power. Default 128. Increase for heavier per-iteration work; tune with pulse_rate and phase length. sample_intervalIntegerReserved sampling interval (ms) for pulse-specific logic; values below 50 are raised to 50. Default 100. toleranceFloatDefault 10.0. Parsed and passed to the worker; not currently used in pass/fail logic (reserved for future checks). verify_modeStringe.g. diff / crc. Default diff. Parsed; not currently used in pass/fail logic (reserved for future GEMM verification). halt_on_errorBoolIf true, stop the GPU thread on first BLAS or thermal error. Default false. hot_callsIntegerBLAS “hot call” / warmup-related parameter forwarded to rvs_blas. Default 1. gpu_sync_waitIntegerDefault 10000. Parsed from configuration; not referenced by the current barrier implementation (placeholder for future timeout behavior). max_temp_cFloatJunction temperature ceiling in degrees Celsius. If the GPU junction temperature exceeds this threshold during the run, the worker logs a thermal-violation error and, when halt_on_error is true, terminates that GPU thread. Default 105.0.

  Output
Log lines use the action name, module tag pulse, and GPU ID.
Output / log Description Start[INFO] ... pulse <gpu_id> start pulse_rate=<hz> Parameters[INFO] ... pulse_rate=... Hz period=...ms high=...ms low=...ms Thermal error[ERROR] ... thermal violation: <temp>C (junction > 105°C) Periodic summaryAt each log_interval, moving averages and extrema: pulse #N avg_high=...W avg_low=...W max_high=...W min_low=...W delta=...W CompletionSummary over the run: pulse count, average high/low power, delta, max high, min low. pass[RESULT] ... pass: TRUE or FALSE per GPU.
With JSON logging enabled, per-pulse records can include power_high_w, power_low_w, power_delta_w, phase durations, temp_c, gemm_count, and a final summary with pass.

  Example
Minimal illustration (see pulse_single.conf for full examples including hipblaslt + fp16_r):
actions:
- name: pulse_stress_basic
  device: all
  module: pulse
  parallel: true
  duration: 60000
  log_interval: 5000
  pulse_rate: 2
  high_phase_ratio: 0.5
  ops_type: sgemm
  matrix_size: 8192
  blas_source: hipblaslt
  data_type: fp32_r
  compute_type: fp32_r
  workload_iterations: 128
  halt_on_error: false
Run from the build or package bin directory, for example:
- Tune `matrix_size`, `pulse_rate`, and `high_phase_ratio` to match GPU class and the transient behavior you want to stress.
- hipBLASLt often delivers higher GEMM throughput than rocBLAS on supported GPUs; `fp16_r`/`hgemm` with `compute_type`: `fp32_r` is a common high-throughput choice (as in the `pulse_stress_fp16` action in `pulse_single.conf`).

  MEM module
The Memory module tests GPU memory for hardware errors and soft errors using HIP. It executes a configurable suite of memory test algorithms that exercise various data patterns and access sequences. Each test can be individually included or excluded. The module reports errors found per test and passes only if no memory errors are detected.
The following tests are available (referenced by index in exclude):
Index 0 Walking 1 bit 1 Own address test 2 Moving inversions, ones & zeros 3 Moving inversions, 8-bit pattern 4 Moving inversions, random pattern 5 Block move, 64 moves 6 Moving inversions, 32-bit pattern 7 Random number sequence 8 Modulo 20, random pattern 9 Bit fade test 10 Memory stress test

  Module specific keys
Config Key TypeDescription mem_blocksInteger Number of GPU memory blocks used per test iteration. The default value is 256. num_passesInteger Number of passes (repeats) per block in each test. The default value is 1. thrds_per_blkInteger Number of HIP threads per block launched for each test kernel. The default value is 128. stressBool If true, enables the memory stress test (Test 10) in addition to the standard tests. The default value is false. mapped_memoryBool If true, uses host-mapped (pinned) memory instead of device memory for the test buffers. The default value is false. num_iterInteger Number of iterations to run per test. The default value is 1. excludeCollection of Integers Space-separated list of test indices (0–10) to skip. All tests not listed here will be executed. For example, exclude: 9 10 skips the bit fade and memory stress tests.

  Output
Output Key TypeDescription TestString Name of the memory test that was executed. Time TakenFloat Time in seconds taken to complete the test on this GPU. errorsInteger Number of memory errors detected during the test. A value of 0 indicates no errors. passBool True if no memory errors were detected for this test.

  Examples
Example 1:
Run all tests in parallel on all GPUs, excluding bit fade (9) and stress (10):
actions:
- name: action_1
  device: all
  module: mem
  parallel: true
  count: 1
  wait: 100
  mapped_memory: false
  mem_blocks: 128
  num_passes: 500
  thrds_per_blk: 64
  stress: true
  num_iter: 50000
  exclude: 9 10
Run with:
The test passes if no memory errors are detected on any GPU. A typical result message looks like:
[RESULT][<timestamp>][action_1] mem <gpu_id> Test 1  [Walking 1 bit] : pass: true  errors: 0 Time Taken: 0.652 s
[RESULT][<timestamp>][action_1] mem <gpu_id> Test 2  [Own address test] : pass: true  errors: 0 Time Taken: 0.423 s

  BABEL module
The BABEL module executes BabelStream benchmark tests that measure GPU memory bandwidth. BabelStream is a synthetic benchmark based on the STREAM benchmark for CPUs. It runs configurable memory kernels (Copy, Mul, Add, Triad, Dot, Read, Write) using HIP and reports the achieved bandwidth in GB/s or GiB/s for each kernel. The benchmark is useful for characterizing peak memory bandwidth and detecting bandwidth regressions.

  Module specific keys
Config Key TypeDescription array_sizeInteger Size of the test buffer in bytes (or in mebibytes if mibibytes: true). The default value is 33554432 (32 MB). test_typeInteger Data precision used for the benchmark. Accepted values: 1 – Float (32-bit, default). 2 – Double (64-bit). 3 – Triad float. 4 – Triad double. num_iterInteger Number of kernel launch iterations. Used when duration is 0. The default value is 100. durationInteger Duration of the test in milliseconds. When greater than 0, the test runs for this time instead of a fixed number of iterations. A value of 0 means use num_iter. The default value is 0. mibibytesBool If true, array_size is interpreted in mebibytes (MiB) and bandwidth is reported in GiB/s. If false, bytes and GB/s are used. The default value is false. o/p_csvBool If true, outputs results in CSV format in addition to the standard log. The default value is false. readBool If true, enables the Read kernel (measures read-only bandwidth). The default value is false. writeBool If true, enables the Write kernel (measures write-only bandwidth). The default value is false. copyBool If true, enables the Copy kernel (a[i] = b[i]). The default value is false. mulBool If true, enables the Mul kernel (a[i] = scalar * b[i]). The default value is false. addBool If true, enables the Add kernel (a[i] = b[i] + c[i]). The default value is false. dotBool If true, enables the Dot kernel (sum of a[i] * b[i]). The default value is false. triadBool If true, enables the Triad kernel (a[i] = b[i] + scalar * c[i]). The default value is false. data_initString Data initialization method for the test arrays. Accepted values: default – Initialize with default constant values (default). Other values may be supported depending on the build. nontemporalString Non-temporal (streaming) memory access mode for the kernels. Non-temporal stores bypass the cache and can be useful for measuring true memory bandwidth. Accepted values: all – Apply non-temporal accesses to all kernels (default). none – Disable non-temporal accesses. read – Apply only to read accesses. write – Apply only to write accesses. dwords_per_laneInteger Number of 32-bit words processed per GPU lane per kernel invocation. The default value is 4. chunks_per_blockInteger Number of data chunks processed per GPU thread block. The default value is 2. tb_sizeInteger Thread block size (number of threads per block). The default value is 1024.

  Output
Output Key TypeDescription Array sizeInteger Size of the test array used for the measurement, in bytes or MiB depending on the mibibytes setting. FunctionString Name of the BabelStream kernel executed (e.g., Copy, Mul, Add, Triad, Dot, Read, Write). bandwidthFloat Measured memory bandwidth for this kernel in GB/s (or GiB/s if mibibytes: true). passBool True if the kernel completed successfully.

  Examples
Example 1:
Run all BABEL kernels with a 32 MB buffer, 5000 iterations, double precision:
actions:
- name: action_1
  device: all
  module: babel
  parallel: true
  count: 1
  num_iter: 5000
  duration: 0
  array_size: 33554432
  test_type: 2
  mibibytes: false
  o/p_csv: false
  copy: true
  read: true
  write: true
  mul: true
  add: true
  dot: true
  triad: true
Run with: