How to Upload Project on Workstation in Python

NeoLoad CLI

Overview

This command-line interface helps you launch and notice performance tests on the Neotys Web Platform. Since NeoLoad is very flexible to many deployment models (SaaS, self-hosted, cloud or local containers, etc.), configuration and exam execution parameters depend on your licensing and infrastructure provisioning options.

Property	Value
Maturity	Stable
Author	Neotys
License	BSD 2-Clause "Simplified"
NeoLoad Licensing	License Complimentary edition, or Enterprise edition, or Professional
Supported versions	Tested with NeoLoad Spider web from version 2.3.2
Download Binaries	Come across the latest release on pypi

TL;DR ... What

The goal of this guide is to demonstrate how you can:

create API load tests using lawmaking (YAML)
run them from any environment
visualize test results in web dashboards

TL;DR ... How

              pip3 install neoload neoload login $NLW_TOKEN \         test-settings --zone $NLW_ZONE_DYNAMIC --lgs 5 --scenario sanityScenario createorpatch NewTest1 \         project --path tests/neoload_projects/example_1 upload NewTest1 \         run

Note: For Windows control line, supplant the '\' multi-line separators to a higher place with '^'

Prerequisites
Installation
Login to Neoload Spider web
Setup a test
- Setup resources in Neoload Spider web
- Define a test settings
- Upload a Neoload projection
  - [Excluding files from the project upload] (#excluding-files-from-the-project-upload)
Run a test
- Finish a running test
Reporting
- View results
- Exporting Transaction CSV data
View zones
Create local docker infrastructure to run a test
Continuous Testing Examples
- Support for fast-fail based on SLAs
Packaging the CLI with Build Agents
IDE Integrations
Contributing

Prerequisites

The CLI requires Python3

Download and install python3 for Windows x from Python.org
- Make sure y'all check the selection 'Add Python to the environment variables' option
- Install pip: python -m pip install -U pip
Download and install python3 for Mac OS Ten from Python.org - Python3 on Mac OS X

Optional: Install Docker for hosting the test infra on your car (this feature does non work with Docker for Windows).

Installation

              pip3 install neoload neoload --assistance

Annotation: if y'all receive SSL download errors when running the in a higher place control, you lot may besides need to install sources using the post-obit command:

Login to Neoload Web

NeoLoad CLI defaults to using the NeoLoad Web APIs for most operations. That's why y'all demand to login.

              neoload login [TOKEN] neoload login --url http://your-onpremise-neoload-api.com/ --workspace "Default Workspace" your-token

The CLI volition connect by default to Neoload Spider web SaaS to charter license.
For self-hosted enterprise license, you must specify the Neoload Web Api url with --url.

The CLI stores data locally like api url, token, the workspace ID and the examination ID yous are working on. The commands can exist chained !

              neoload status          # Displays stored data

Setup a exam

Optionally Cull a workspace to work with

              Usage: neoload workspaces [OPTIONS] [[ls|use]] [NAME_OR_ID] neoload workspaces use "Default Workspace"

Since Neoload Spider web 2.v (Baronial 2020), assets are scoped to workspaces. The CLI allows you to cull your workspace at login or with the "utilize" sub-command, otherwise the "Default Workspace" is used.
/!\ The zones are shared between workspaces.

Setup resources in Neoload Web

Run a exam requires an infrastructure that is defined in Neoload Web Zones section (encounter documentation how to manage zones) Y'all must at to the lowest degree have either a dynamic or a static zone with one controller and i load generator. At First, y'all could add resource to the "Default zone" since the CLI use it by default.

Ascertain a test settings

Exam settings are how to run a examination, a sort of template. Tests are stored in Neoload Web.

              Usage: neoload test-settings [OPTIONS] [[ls|create|put|patch|delete|utilize]] [Name] neoload test-settings --zone defaultzone --lgs 5 --scenario sanityScenario create NewTest1

You must define :

Which scenario of the Neoload projection to utilise

You can optionally define :

The test-settings description
The controller and load generator's zone to use (defaultzone is set up past default)
How many load generators to use for the zone (1 LG on the defaultzone is set by default)
Advanced users who already have several zones with available resource in it can write : --zone my_controller_zone --lgs lg_zoneA:2,lg_zoneB:three

To work with a specific test already created and be able to chain commands

              neoload test-settings use NewTest1 neoload examination-settings employ 4a5e7707-75c0-4106-bbd4-68962ac7f2b3

Upload a Neoload project

Encounter bones projects examples on github tests/neoload_projects folder To upload a NeoLoad projection zero file or a standalone equally lawmaking file into a test-settings

              Usage: neoload project [OPTIONS] [up|upload|meta] NAME_OR_ID neoload projection --path tests/neoload_projects/example_1/ upload

You lot must specify in which exam the projection will be uploaded:

either by doing this control first

                  neoload test-settings employ NewTest1

or past adding the name or id of the test to the project command

                  neoload project --path tests/neoload_projects/example_1/ upload NewTest1

To Validate the syntax and schema of the every bit-code project yaml files

              neoload validate sample_projects/example_1/default.yaml

Excluding files from the project upload

If you are uploading a project directory that contains not NeoLoad as-code YAML files (such every bit .gitlab-ci.yml) y'all volition need to create a .nlignore file (exactly the same as .gitignore) that excludes these files from the project upload process so that NeoLoad Spider web does non parse them and neglect them as if they should exist the NeoLoad DSL.

Delight encounter Gitlab and Azure pipeline examples for more item.

Run a exam

This command runs a exam, it produces blocking, unbuffered output about examination execution process, including readout of current information points. At the terminate, displays the summary and the SLA passed & failed.

              Usage: neoload run [OPTIONS] [NAME_OR_ID] neoload run \         # Runs the currently used examination-settings (see neoload condition and neoload test-settings use)      --every bit-lawmaking default.yaml,slas/uat.yaml \      --proper noun "MyCustomTestName_${JOB_ID}" \      --description "A custom test description containing hashtags like #latest or #issueNum"

disassemble selection kick off a exam and returns immediately. Logs are displayed in Neoload Web (follow the url).
as-lawmaking option specify every bit-code yaml files to use for the examination. They should already be uploaded with the project.
Examination consequence proper name and description can be customized to include CI specific details (e.thousand. CI job, build number...).
Reservations tin be used with either the reservationID or a reservation duration and a number of Virtual users.

If you are running in interactive panel fashion, the NeoLoad CLI volition automatically open the organisation default browser to your live exam results.
When hitting Ctrl+C, the CLI volition try to stop the test gracefully

Cease a running test

              neoload terminate             # Send the stop signal to the test and wait until information technology ends.

Reporting

There is basic back up in the NeoLoad CLI for viewing and exporting results.

View results

              Usage: neoload exam-results [OPTIONS] [[ls|summary|junitsla|put|patch|delete|use]] [NAME] neoload test-results ls                 # Lists examination results                                            . neoload test-results use                # Call up the test result yous want to work on.                           . neoload exam-results summary            # The Json result summary, with SLAs neoload test-results junitsla           # Output the summary in a JUnit xml file

Metadata on a examination tin can exist modified subsequently the test is complete, such equally name, description, and status.\

To filter test results based on projection, scenario, or condition:

              neoload test-results --filter "project=MyProject;scenario=fullTest" ls neoload test-results --filter "status=TERMINATED|qualityStatus=FAILED" ls

NOTE: you can use either a semicolon OR a pipe, but non both interchangeably in the same filter.

To work with a specific examination outcome and be able to concatenation commands

              neoload test-results use 4a5e7707-75c0-4106-bbd4-68962ac7f2b3

Detailed logs and results are available on Neoload Web. To get the url of the electric current event :

              neoload logs-url                        # The URL to the examination in Neoload Web

The test-results vs. study subcommands

The 'test-results' subcommand is intended for direct operational queries against high-level API data.

The 'study' subcommand is intended to simplify not only common information exporting needs, but also provide templating capabilities over a standard, correlated data model. In contrast to the test-results subcommand, 'study' can be used to generate equally well as transform test outcome data.

Exporting Transaction CSV data

              Usage: neoload written report [OPTIONS] neoload report --template builtin:transactions-csv > temp.csv

Filtering export data past timespan

In many load tests, ramp-up and spin-downwards time is considered irrelevant to calculate into amass statistics, such as how when warming up, systems may produce college-than-expected latencies until a steady state is reached.

Therefore, the NeoLoad CLI allows for export of particular fourth dimension ranges by providing a timespan filter.

              neoload study --template builtin:transactions-csv --filter "timespan=5m-95%" neoload study --template builtin:transactions-csv --filter "timespan=15%" neoload written report --template builtin:transactions-csv --filter "timespan=-ninety%"

Timespan format is [Fourth dimension], so '-' representing to, then some other [Time]. Time format tin be either a human readable duration or percentage of overall examination duration.

Human being readable fourth dimension duration format is hr|minute|2d such as '1h5m30s' or a sub-portion such as '5m'.

Omitting the end [Time] segment will filter results outset with the time specified to the finish of the exam.

Similarly, ommiting the start [Time] segment volition filter results showtime with the kickoff of the test to the end fourth dimension specified.

Filtering export data past chemical element

It is often useful to narrow analysis and statistics to a particular grouping of activities, such every bit Login processes across multiple workflows (user paths) or other mutual cardinal business concern transactions.

Therefore, the NeoLoad CLI allows for exports of specific transcations whose proper name, parent, or User Path name matches specific values or patterns.

              neoload report --template builtin:transactions-csv --filter "elements=Login"

You tin filter to specific transactions or requests by specifying 'elements' then a pipe-delimited list of element GUIDs, full names, or partial proper name matches. This can likewise include python-compliant regular expressions.

Combining timespan and element filters

              neoload report --template builtin:transactions-csv --filter "timespan=l%-95%;elements=AddToCart"

Both timespan and elements filters can be combined in order to get statistics for specific elements within a precise portion of the test elapsing. Per the instance to a higher place, transaction information will be computed for elements that have 'AddToCart' somewhere in their name, user path, or parent chemical element and summate aggregates based on data starting from halfway through the test upward to just virtually the very end.

Exporting All Test Data and Using Custom Templates

If yous would similar to employ multiple templates to create divide output files for specific test information, you should dump the examination result information using the standard JSON scheme showtime:

              neoload written report --out-file ~/Downloads/temp.json

NOTE: past default, this queries all entity information in test results and may crusade multiple API calls to occur depending on the structure of the user paths and monitoring data in the test result set.

So you can produce multiple output files from a single information snapshot:

              neoload report --json-in ~/Downloads/temp.json \                --template builtin:transactions-csv \                --out-file ~/Downloads/temp.csv  neoload report --json-in ~/Downloads/temp.json \                --template /path/to/a/jinja/template.j2 \                --out-file ~/Downloads/temp.html

Annotation: born reports produce a reduced-scope JSON data model and are therefore faster that exporting all test data for various templates and output specs.

View zones

Brandish in a human-readable manner the list of all static and dynamic zones registered on Neoload Spider web, and the resources attached (controllers and load generators).

Create local docker infrastructure to run a examination [EXPERIMENTAL]

Warning: Docker features are non officially supported by Neotys as they rely heavily on your own Docker setup and environs. This command is but for local/dev test scenarios to simplify infrastructure requirements.

In certain environments, such equally on a local dev workstation or in a Docker-in-Docker CI build node, it is useful to "bring your ain infrastructure". In other words, when yous don't already have a controller and load generators available in a zone, you can spin some up using Docker before the test starts. An example of an all-on-one approach:

              neoload docker install  neoload login $NLW_TOKEN \         test-settings --lgs two --scenario sanityScenario create NewTest1 \         project --path tests/neoload_projects/example_1 upload \         run

What the 'docker install' CLI add step in run command. This stride is triggered when zone of controller the test-settings is same than docker.zone (default is defaultzone). When it is triggered, it launches 1 controller with number of LG test-settings in docker.zone. At the end of the run the containers are removed.

The docker container tin can be launched manually with neoload docker up command and removed with neoload docker downwardly command. In this case the number of controller and lg is defined by configuration respectively docker.controller.default_count (default: one) and docker.lg.default_count (default: two).

              Usage: neoload docker [OPTIONS] [up|down|clean|forget|install|uninstall|status]   neoload docker upwards / down         # start or delete container depend on configuration neoload docker install/uninstall # add/remove hooks on run command to up when the controller zone is same and zone is empty. Shut downward at the cease of test running. neoload docker forget            # remove container from the launched list. That avert to be removed with down control. neoload docker clean             # remove all container created past neoload-cli even if information technology was forgotten. neoload docker status            # brandish configuration and general condition.  Options:   --no-wait  Do not wait for controller and load generator in zones api   --aid     Testify this message and exit.  Configuration:   - docker.controller.image (default:  neotys/neoload-controller:latest)   - docker.controller.default_count (default: 1)   - docker.lg.image (default: neotys/neoload-loadgenerator:latest)   - docker.lg.default_count (default: two)   - docker.zone (default: defaultzone)

Note: Docker CLI must be installed on the system using these commands. This will apply the Docker daemon, however it is configured. In a Docker-in-Docker context, this is inferred. For local workstations, it is sufficient to install Docker Desktop or Docker for Mac.

Continuous Testing Examples

The main goal of the NeoLoad-CLI is to standardize the semantics of how load tests are executed across evolution, non-prod, and production environments. While the above instructions could be run from a contributor workstation, they can easily exist translated to various continuous build and deployment orchestration environments, every bit exampled:

Jenkins
Azure DevOps
Gitlab
Bamboo
AWS
CircleCI, TBD when @punkdata gets back to @paulsbruce :)

NB: When chaining commands, the return lawmaking of the whole command is the return code of the last command. That's why you lot should not chain the two commands "run" and "test-results junitsla".

Note: When combining NeoLoad projects and YAML-based pipeline declarations, please encounter [Excluding files from the project upload] (#excluding-files-from-the-project-upload) to ensure that unecessary artifacts aren't included in the project upload procedure.

Support for fast-fail based on SLAs

Non all tests succeed. Sometimes environments are down. Sometimes 3rd parties are surprisingly deadening. You don't want to wait for your build pipelines to conduct the whole test duration if it'southward possible to identify these issues early. Applying proper SLAs to your tests allows you to monitor for errors and latency during the test.

Consider the post-obit SLA:

              sla_profiles: - name: geo_3rdparty_sla   description: Avg Resp Fourth dimension >=100ms >= 250ms for cached queries   thresholds:   - avg-resp-fourth dimension warn >= 100ms fail >= 250ms per interval   - mistake-rate warn >= five% fail >= ten% per test

If you want to neglect the pipeline if either of these thresholds are exceeded over a certain percent of their times, you must:

run the test in 'detached' mode to permit for non-blocking execution of a test
utilize the fastfail command to monitor for early signals to stop the exam if SLAs are violated
finally wait for the examination results

To run the test in detached mode:

And so immediately subsequently, use the fastfail command:

              neoload fastfail --max-failure 25 slas cur

In the above example, '25' represents the percent of times where the SLA was violated, such as 'on a item asking with an SLA applied, x out of 50 times it was executed, the SLA failed'.

Finally, because the test was executed in non-blocking mode, you should wait for the concluding test event.

An example for Jenkins pipeline is constitute here.

Packaging the CLI with Build Agents

Many of the above CI examples include a step to explicitly install the NeoLoad CLI every bit part of the build steps. Withal, if you want the CLI baked into some build amanuensis directly so that information technology is set up for employ during a job, here'due south a Docker instance:

For Docker builds See the test harness Tall-based Dockerfile

IDE Integrations

Since almost of what we do in an IDE is create/edit code, we're mostly interested in how to:

make it piece of cake to write API tests in YAML (automatic syntax validation)
validate that tests do not contain unanticipated errors even at small scale
dry out-run small (smoke) load tests locally so that lawmaking check-ins volition work in CI/pipeline tests

Since the latter 2 cases are already covered by command-line semantics, our main focus is to accelerate exam authoring by providing NeoLoad as-code DSL (Domain-specific Language) validation and in some cases editor car-consummate.

Status of IDE / editor integrations

IDE / Editor	Syntax checks	Auto-complete	Setup steps
Visual Studio Code	[x]	[ten]	see instructions
PyCharm	[x]	[x]	Mark 'neoload' directory as "Sources Root"

Contributing

Feel gratis to fork this repo, brand changes, test locally, and create a pull request.

Local Verification

Tests

As part of your testing, y'all should run the built-in test suite with the following command:
Annotation: for testing from Mac, delight change the PYTHONPATH separators beneath to colons (:) instead of semicolons (;).

              pytest -5 pytest -v -m "not slow"          # Skip slow tests that run tests  # Run on a existent Neoload. Mocks are disabled pytest -v --token <your_personal_token> --url https://neoload-api.saas.neotys.com/ --makelivecalls  # Run integration tests. This volition run scripts with real neoload commands and assert json output with jq. # This require at to the lowest degree ane controller and one LG on the provided zone ./tests/integration/runAllScripts.sh <your_personal_token> --url https://neoload-api.saas.neotys.com/ defaultzone

Additionally, any contributions to the DSL validation functionality, such as on the JSON schema or the validate command, should execute the following tests locally earlier pushing to this repo:

              ./tests/neoload_projects/yaml_variants/validate_all.sh

This command executes a number of NEGATIVE tests to prove that changes to the JSON schema or validation process produce failures when their input is malformed in very specific means (common mistakes).

Release Process (managed by Neotys team)

Auto-generating Changelog

Before tagging a release, merged PRs should update the CHANGELOG.medico via the post-obit:

              github_changelog_generator -u Neotys-Labs -p neoload-cli --token $GIT_CHANGELOG_GEN --exclude-tags-regex ".*(dev|rc).*" --add-sections '{"documentation":{"prefix":"**Documentation updates:**","labels":["documentation"]}}'

This utility is a Ruby-based Jewel that tin exist installed (as well used in CI/Actions) as follows:

              precious stone install github_changelog_generator

Version direction on pypi

Suppose Ten, Y, Z and Due north are integers, versions volition exist named as following on pypi:
Final release version = X.Y.Z Example ane.4.0 Install it with pip install neoload
Release candidate version = X.Y.ZrcN Case one.v.0rc1 for the next candidate version. Install it with pip install neoload --pre
Development versions = 10.Y.Z.devN Case 1.four.0.dev1 for a development version based on the final release 1.iv.0. Install it with pip install neoload==1.four.0.dev1

Release candidate version contains all features planned and in testing by Quality Assurance team.

Evolution versions may contains work not planned by R&D and not tested by Quality Assurance team. They should always be based on an official release, non on the adjacent release.

Increase policy:

Minor version increase when major feature, for example new top-level control
Fix version increment when executable changes, for example fixing an existing feature, or update a subcommand to an existing height-level command or update options to an existing command
No release needed when the executable is non modified, for example when updating the following: automated CI tests, unit tests, README, Pipeline examples, written report templates...