Setting Up Execution Servers to Run Commands
This article explains how to configure execution servers to run commands in CloudShell. Note that associating execution servers to automation suites is done on the job level, as explained in Add jobs to an automation suite.
In this article:
Managing execution servers
The Execution Servers management pages in the web portal allow you to both manage and troubleshoot your execution servers, providing critical, real time data about the status of your execution servers, command and job executions, and troubleshooting information and options. For additional information, see Managing Execution Servers.
Optimizing execution capacity
Available capacity for a job's selected execution servers, license usage, resource availability, and the number of other jobs in the queue can all influence how long it takes to schedule and execute an enqueued job.
In theory, the number of TestShell clients in use determines how many execution servers you can install and your overall execution capacity. In practice, the number of installed execution servers is only part of the equation that determines your maximum test execution capacity.
Actual execution capacity is also determined according to the maximum capacity of each execution server (how many tests it can run in parallel), by the number of available execution licenses, and by the offline | online and included | excluded status of each server at the time of execution.
A shortage of execution servers – or execution server licenses – can create a backlog in the execution of scheduled jobs.
You can maximize available execution capacity by installing execution servers on every available TestShell client, by adding additional concurrent execution licenses, and by configuring the execution servers for optimal performance.
- To optimize run-time performance for individual tests, Quali recommends setting server capacity to 1. This setting ensures that maximum resources are always available to the running instance.
- If the time it takes to execute each test is less important than the time it takes to run a series of tests, you can set execution server capacity according to the number of cores or the amount of RAM on each machine.
Note: Setting the maximum execution server capacity above the recommended settings may cause performance failures.
Distributed command execution
Multiple execution servers can be deployed in order to scale out the provisioning and resource commands tasks. By default, blueprint and resource commands will be distributed between the execution servers according to their capacity. It is possible, however, to specify more explicit rules to control the execution server selection for commands. In this section we'll review some of those options and understand how distributed provisioning is implemented.
Resource commands
When a resource command is executed, the system first checks whether a driver for that resource is already active on one of the execution servers. If it is, the command is automatically sent to that driver to be queued and handled. In case no driver is currently active, the resource driver is started on an available execution server.
Controlling execution server selection for resource commands
Attributes can be used to match resource commands to the right execution server based on geographical location of the server and resource, execution server capabilities or other concerns. In a multi-site deployment, for example, there is an advantage in ensuring that overseas lab resources are only controlled by an on-premise execution server. This will help reduce the network latency and improve performance.
For additional information, see Associating Resources to Specific Execution Servers.
Controlling execution server selection for App deployments
To learn how to do this, see the appropriate article:
- For private cloud Apps, see Managing Private Cloud Apps in Domains.
- For public cloud Apps, see Managing Public Cloud Apps in Domains.
Controlling execution server selection for blueprint commands
Blueprint driver commands are also distributed between the execution servers based on availability and capacity. In order to restrict blueprint commands to a specific group of execution servers, a configuration key needs to be added specifying the relevant selector attribute value.
Create a selector attribute and assign it to an execution server, as explained in Associating Resources to Specific Execution Servers.
- Specify an attribute value for blueprint commands. Add the following configuration key to the Quali server customer.config file:
EnvironmentCommandsESRestrictions
The value should be the attribute name and value given the syntax Name=Value. For example:
Configuring the Execution Server to Run as a Process by Default
See this article in the CloudShell Suite Installation Guide.
Working with local tests
If you are using a source control tool and wish to configure CloudShell to work with your local tests, see Source Control: Using Local Tests in Automation Suites or contact Quali support or your Customer Success representative.
Configuring Execution Servers to Deploy vCenter Apps
Number of execution slots for VM deployments
Take the following considerations into account when setting the number of Execution Server slots for the deployment of vCenter VMs.
| App deployment | Number of slots required |
| Apps deployed manually | 3 slots per App |
| Apps deployed automatically by CloudShell setup | 2n + 1 (n = total number of Apps to be deployed at the same time) For example, deploying 5 Apps requires at least 11 execution slots. |
Deployments from OVF image files
The following configurations should be performed on each execution server machine that will be used by the vCenter resource to deploy VMs from OVF image files during App deployments.
- Configure access to vCenter OVF image file path
- Install the OVF tool
Configuring Execution Servers to support python 3 shells and scripts
Make sure to install Microsoft Visual C++ Redistributable 2015 on the Execution Servers. Without this installation, CloudShell will not create the python virtual environment for the shell or script's execution on the Execution Server.
Enabling custom execution servers
You can enable or disable the use of your own custom-built execution server. This feature is enabled by default.
To enable custom execution servers:
- Go to C:\Program Files (x86)\QualiSystems\CloudShell\Portal\customer.config file, and add the following key:
- Restart the CloudShell Portal IIS service.
<add key="EnableCustomServerTypes" value="True"/>
Setting the logging level for python drivers and scripts running on an Execution Server
It is possible to set the logging level for automation processes running on a specific Execution Server or python driver (for drivers, see CloudShell Dev Guide's Tips and Tricks article). The logging levels are: DEBUG, INFO, WARN and ERROR.Only messages that are greater than the specified log level are saved to the logs.
Logs are organized in different files according to resource and sandbox. The folder location will be relative to the driver in the virtual environment location at: %venv%\[drivername]\Lib\site-packages\cloudshell\Logs (e.g. C:\ProgramData\QualiSystems\venv\Deployment_Orchestrator_5_2\Lib\site-packages\cloudshell\Logs). Under windows, [venv] will be located at %programdata%\qualisystems\venv.
To set the logging level for python drivers and scripts:
- Go to C:\Program Files (x86)\QualiSystems\TestShell\ExecutionServer\customer.config file, and add the following key:
Replace
<log-level>with the desired level.For example:
<add key="DefaultPythonEnvrionmentVariables" value="LOG_LEVEL=DEBUG"/>- Restart the TestShell Execution Server service.
<add key="DefaultPythonEnvrionmentVariables" value="LOG_LEVEL=<log-level>"/>
Setting environment variables to be used by python driver instances
Using the DefaultPythonEnvrionmentVariables key, you can define environment variables to be used by the driver. The environment variable is defined on the Execution Server and will be used by the appropriate driver instances that are running on that Execution Server.
To set environment variables for python driver instances:
- Go to C:\Program Files (x86)\QualiSystems\TestShell\ExecutionServer\customer.config file, and add the following key:
Replace each
<variable-pairs>with a semicolon (;) separated list of the appropriate variable name-value pairs.For example:
<add key="DefaultPythonEnvrionmentVariables" value="name1=value1;name2=value2,"/>- Restart the TestShell Execution Server service.
<add key="DefaultPythonEnvrionmentVariables" value="<variable-pairs>"/>
Troubleshooting
What happens if the Execution Server is offline while I run the update process?
The execution server will still change to a state of Waiting for update and once it gets back online, it will start the update process first, and only then it will get jobs to run.
What happens if I (or someone else) start another update process before the previous one is done?
You cannot start another process until the update on the QualiServer is done.
Once its done, and the system is updating or waiting to update the execution servers, you can start another update process and even provide different blueprint parameters.
Idle stations that were already updated by the previous process and stations that are currently updating will start the update again. Each station that is in a Waiting for update state will remain in that state and will execute the batch file with the new parameters when it becomes idle
If the batch files fails, how can I check what happened?
On each machine, you define the batch file to launch and a folder in which we save the outputs from these batch files. If your batch files outputs any information about the process, then you'll be able to see it in these files and check what might have gone wrong. If the process stopped on TestShell Studio, you can check the logs of the portal for other relevant details. If it failed on the execution server, check the logs of it to see if there is more information.
