.. _testing:

CLI Tests
========================

Preparation
-----------

In order to run the CLI tests, the ``sim`` and ``test`` dependencies should be installed with icicle, usually like so::

    python3 -m pip install ".[sim,test]"

Don't forget to include ``-e`` if developing.

In order to run the ``pyvisa_sim`` simulations for the ``MP1`` instrument, ``pyvisa_sim`` must be patched to allow injection into setter responses. To do so, run::

    cd deps/
    bash patch-pysica-sim.sh

Or patch the current ``virtualenv`` installation of ``pyvisa-sim`` manually to match the patch files in ``deps``. An MR is open on ``pyvisa-sim`` and this should not be necessary once the targeted version is updated.


Running tests locally
---------------------

The CLI tests are run from the ``test/cli`` directory using ``make``::

    cd test/cli
    make test

In order to run a single test, each test file can be run using the ``*.cli-test`` target::

    make keithley2410.cli-test

Tests are run by the ``cli-test.py script``::

    python3 cli_test.py cli/keithley2410

A test can also be run in `parallel-mode` manually, which is a bit faster due to multiprocessing and also checks locking as a side-effect::

    cd test/
    python3 cli_test.py -j cli/keithley2410

Tests are generally run sequentially, but one may add ``-j`` to a ``make test`` call to run multiple test suites in parallel. Printing order is guaranteed - test outputs will be printed together when all tests in a suite are completed.

CI Integration
--------------

Tests are run automatically by the CI on any ``git push`` instruction to the main ``ICICLE`` gitlab repository.

Test Files
----------

Test files contain test commands and expected output and return value in the following format::

    ##
    {test name}
    # COMMAND
    {test command} {test args...}
    # OUTPUT
    {expected output}
    # RETURNS {expected return value}
    ##
    ...

The separator ``##`` is used between multiple tests in the same file.

For example, the Keitley2410 ``identify`` test looks like this::

    ##
    identify
    # COMMAND
    keithley2410 -S identify
    # OUTPUT
    Keithley Instruments Inc., Model 2410, 000000, SIM1.0
    # RETURNS 0
    ##

There is also a utility to generate a test file from the current state of a CLI built into ``cli_test.py``. In order to use it, write a ``<intrument>.gen`` file in the following format::

    {test name} | {test command}

For the Keithley2410 ``identify`` test this looks like::

    identify | keithley2410 -S identify

Then a test file may be generated by running ``cli_test.py`` with the generate file passed to the ``-g`` flag, for example::

    python3 cli_test.py -g keithley2410.gen keithley2410.test


Test Writing Good Practise
--------------------------

When submitting merge requests, in general test files should be updated in a separate commit just before submitting the request, to allow the changes to the test cases to the reviewed. In addition, test files should not be regenerated after initial submission, but only updated manually in most cases, since changes should be small and well-defined. An exception may be made here for the addition of large numbers of new test cases.

When submitting changes, any changes to CLI behaviour should be well-motivated, and changes in test case expected output for unrelated instruments will cause the request to be paused until the errant behaviour is removed or justified.

Tests cases should check valid and invalid, expected and unexpected inputs, and should ensure validity of the entirity of new CLI interfaces submitted in merge requests. Exceptions may be made for non-exiting commands (e.g. ``monitor``) as these are not currently supported by the test system.