PyPi Server - Managing Python Driver and Script Dependencies

CloudShell manages driver and script dependencies using PyPi Server. PyPi Server is a process that is installed with CloudShell on the Quali Server computer. PyPi Server was implemented in CloudShell 8.2.

Note: If you are upgrading to CloudShell 8.3 and above, make sure to migrate your offline and locally developed python packages to the PyPi Server repository. For additional information, see this article on the CloudShell Suite Installation Guide.

The PyPi server allows easy and organized development of Python packages for local use. A developer can remotely upload or update their own local Python packages to the local PyPi server, thereby making it available to all the Execution Servers, without having to expose these packages in a public repository (such as public PyPi) or a testing repository (such as a local TestPi).

When a Python script or driver is launched, the execution server contacts the PyPi Server to get the required packages. Then, the PyPi Server searches for the packages in its repository, searches for any missing packages in the public PyPi repository, and sends the packages or a link to the package residing in public PyPi to the execution server.

It is also possible to configure PyPi Server behind a reverse proxy for secure communication between the PyPi Server and the execution servers. For details, see this article in the CloudShell Suite Installation Guide.

The following configuration modes are available for Python driver and script dependencies:

Automatic online mode

Note: The PyPi Server and Execution Servers must have access to the Internet in order for CloudShell to work in this mode.

This is the default mode. When a Python driver or script that has dependencies is running, the execution server running the driver or script automatically creates a new virtual environment for the driver or script's instance and PyPi Server sends the dependencies (i.e. links to download the dependencies) to the execution server, which downloads the missing dependencies and loads all the required dependencies to the virtual environment.

If any of the dependencies are not in the local repository, PyPi Server searches for them online in public PyPi and sends links to their online locations for download to the local PyPi server. To learn how to block PyPi Server from downloading certain packages in online mode, see the section below.

Important: Packages you add to the local PyPi Server repository will constrict all Python scripts and drivers that use any version of these packages to use the specific versions residing in the repository (if they require a different version, the execution will result in error). Therefore, for packages that are published in the public PyPi repository, the best practice is to specify the package versions in the requirements.txt instead of adding them to the local repository.

Blocking PyPi Server from downloading packages from public PyPi

If you have internet access but want to block PyPi Server from downloading packages that reside in the public PyPi repository, place all the Python dependencies in your local PyPi Server repository, including the dependencies of Shells you are using and any other dependencies required by custom scripts and drivers you have in CloudShell.

However, we do not recommend to place CloudShell dependencies in the repository (packages that come out-of-the-box with CloudShell and packages that are released by Quali's Shells team), as this will force your scripts and drivers to use those package versions even if newer ones are in public PyPi (as explained in the note above). Therefore, we recommend to leave out any CloudShell dependencies (packages that include "cloudshell" in their name) to make sure you get backwards-compatible CloudShell fixes and updates we publish from time to time on public PyPi.

Manual offline mode

This mode is used when access to the Internet is unavailable for Quali Server and the execution servers. To make dependencies available in offline mode, the admin needs to download the dependencies and place them in the local PyPi Server repository residing on the Quali Server machine when installing or upgrading CloudShell.

For additional information, see Configuring CloudShell to Execute Python Commands in Offline Mode.

Loading packages to your PyPi Server repository

This procedure explains how to place specific packages in the local PyPi Server repository.

Note that this applies to both online and offline CloudShell deployments.

This is useful for development and testing of drivers and scripts that require the use of packages, which are not published in the public PyPI Python repository.

Special considerations

Packages you add to the local PyPi Server repository will constrict all Python scripts and drivers that use any version of these packages to use the specific versions residing in the repository (if they require a different version, the execution will result in error). Therefore, for packages that are published in the public PyPi repository, the best practice is to specify the package versions in the requirements.txt instead of adding them to the local repository.
If you are upgrading to version 8.3, you will need to add many packages at once, for example, when migrating to the PyPi Server method for managing Python packages, copy the Python dependencies residing in the dependencies folder(s) defined by the <PythonOfflineRepositoryPath> key to the new PyPi Server repository.
Before you start, make sure you have the access credentials to the PyPi Server repository, which were defined when running the Quali Server configuration wizard. If you do not have them, contact your IT administrator. By default, the username and password are "pypiadmin".

To load a package to the PyPi Server repository:

Unzip the package. Make sure the package contains a setup.py file.
In the unzipped package, create a file named .pypirc with the following data (common practice when working with Python packages and uploading them to PyPi):
[distutils]
index-servers =
pypiserver

[pypiserver]
repository=http://<CloudShell server hostname/ip>:<pypi server port (default is 8036)>
username=<pypi server username>
password=<pypi server password>

For example:
[distutils]
index-servers =
pypiserver

[pypiserver]
repository=http://192.168.54.37:8036
username=pypiuser
password=pypipassword
Save the file.
Open command-line from the unzipped package folder location and upload using the suitable option: Setup or Twine:
- To upload using Setup:
  set HOME=<unzipped package folder path>
  python setup.py sdist upload -r pypiserver
- To upload using Twine:
  (If the dist folder is missing from the package, run python setup.py sdist)
  pip install twine
  python -m twine upload dist/* --config-file .pypirc -r pypiserver
The package is uploaded to the local PyPi Server repository.