Python Run-Time Environment

~~~ PRE-RELEASE ~~~

Version 2.0.0, 10th September 2020
Version 3.0.0, 10th September 2020

Copyright © 2007-2020 Mark G. Daniel
This program, comes with ABSOLUTELY NO WARRANTY.
This is free software, and you are welcome to redistribute it under the
conditions of the GNU GENERAL PUBLIC LICENSE, version 3, or any later version.
https://www.gnu.org/licenses/gpl.txt

Contents




PyRTE is an interface to a Python interpreter engine and environment for the WASD VMS Web server. It is designed to be able to be used in standard CGI and CGIplus/RTE persistent scripting modes. The persistent modes (default) provide a ~35x (yes, an approximate thirty-five times) improvement in script activation times (compared to CGI/command-line) and comparably reduced load on both server and system. Note that this package does not contain the Python interpreter or kit, that has to be obtained and installed separately (a simple matter as it is provided via a PCSI package).

PyRTE is linked against the Python interpreter shareable image and so shares a set of Python capabilities in common with command-line OpenVMS Python. Alpha (AXP) and Itanium object modules are provided for the VMS platforms available for Python.

PyRTE has been developed and tested on V8.4, Alpha and Itanium, using WASD v11.n and VMS Python kits available from and documented at https://www.vmspython.org/. JFP's Python kits are kept very up-to-date as Python updates or implementation problems arise.

FYI: Beginning August 2020, PyRTE underwent significant rework prompted by the pre-release of JFP's Python v3.10 port. The core PyRTE code now supports both Python 2.7 and 3.10 as independent builds and executables. PyRTE reports itself as v2.n.n for the Python 2.7 build, and as v3.n.n for Python 3.10.

Installation

Configure WASD

Although there are experimentation modes built into PyRTE basically it will be used as a WASD Run-Time Environment providing persistent Python interpreter(s) and environment(s) for Python scripts and applications. Benchmarking indicates the CGIplus/RTE use reduces activation time to 3% of CGI (yes, that's correct, by a factor 35 - a python interpreter is quite expensive to instantiate). There are subtle differences in the way CGIplus and RTE parse and provide the PATH_INFO data (see the "WASD Scripting Overview" for detail).

Configuration and Mapping

One or more of the following approaches can be implemented.

Remember to

$ HTTPD /DO=MAP
after changing mapping rules.

Supporting Multiple Python Versions

It is sometimes useful, sometimes necessary, to maintain multiple versions of Python on a system, and also support multiple versions to web applications (scripts). It is not difficult to have PyRTE activate a specific version by wrapping the executable in a procedure that sets up the appropriate Python environment in a process-local context.

$! CGI_BIN:PYRTE.COM
$! activate JFP Python 2.7
$ @disk$jfplib0020i:[000000]lib_logicals.com "/process"
$ @disk$jfppy1400i:[000000]python_logicals.com "/process"
$ @python_root:[vms]setup
$ pyrte == "$cgi_exe:pyrte.exe"
$ pyrte
$! CGI_BIN:PYRTE3.COM
$! activate JFP Python 3.10
$ @disk$jfplib0020i:[000000]lib_logicals.com "/process"
$ @disk$jfppy3100i:[python3100.vms]logicals.com "/process"
$ @python3_root:[vms]setup
$ pyrte3 == "$cgi_exe:pyrte3.exe"
$ pyrte3

Each of these is mapped for independent activation based on the request path.

# WASD_CONFIG_MAP
exec /py-bin/* (@cgi_bin:pyrte.com)/wasd_root/src/python/scripts/* \
     script=syntax=unix map=once script=query=none ods=5
exec /py3-bin/* (@cgi_bin:pyrte3.com)/volume**/wasd_root/src/python/scripts/* \
     script=syntax=unix map=once script=query=none ods=5
** Note that the scripts activated are the same on-disk files, located in the PyRTE source directory.
Also note that /py3-bin/ mapping result is different to that of the /py-bin/ mapping. There is a /volume/ present.
This is only required where the same on-disk files are being scripted. This is a kludge. Due to the way WASD caches script information (on the disk file path), including any associated RTE, it was necessary to differentiate the two activations and the chosen solution was to include the physical volume (on which [WASD_ROOT] resides) in the mapping. Another solution might be to use an alternate concealed logical device name.

This is one approach. A little creativity would yield others.

wasd.vsm.com.au

At the time of writing the demonstration site uses this approach to default the examples below to (JFP) Python 2.7, while optionally allowing them to be activated under 3.10 – just modify the browser location field to contain /py3-bin/ rather than the default /py-bin/. Several of the scripts display the underlying Python version confirming the activations.

Example Scripts

After configuration (as described immediately above) the following scripts may be used to confirm the environment is functioning.

Some of these examples also support CGIplus mode for comparison purposes.

Of course the Python code in the script may be inspected from the source directory:

/wasd_root/src/python/scripts/

Web Server Gateway Interface

PyRTE contains a Web Server Gateway Interface v1.0 (WSGI). This is a Python standard specification for web servers and application servers to communicate with web applications (though it can also be used for more than that).

The WSGI code has been developed under the sponsorship of SysGroup

http://www.sysgroup.fr/
and generously made available to the wider WASD community.

The following is an example of a simple (and classic) WSGI application being activated using the PyRTE wasd.wsgi_run() function.

def hello_world (environ, start_response):
    status = '200 OK'
    response_headers = [('Content-type','text/plain')]   
    start_response(status, response_headers)
    return ['Hello world!\n']

import wasd
wasd.wsgi_run(hello_world)

WSGI Output Buffering

PEP 333 specifies that WSGI should not buffer output and PyRTE complies but some tools (like Mercurial) don't buffer internally themselves and end up sending many very small records. This is obviously inefficient. If buffered the throughput boost for Mercurial is about x4. Well worth having for specific Web applications.

This forced buffering may be enabled on a per-application basis using a mapping rule, for example

set /mercurial/* script=param=PYRTE=/WSGI=BUFFER
or by defining the logical name
$ DEFINE /JOB PYRTE_WSGI_FORCE_BUFFERING TRUE
in a wrapper procedure.

Example WSGI Scripts

The following scripts may be used to confirm the environment is functioning.

The Python code in the script may be inspected from the source directory:

/wasd_root/src/python/scripts/

There are also equivalent wsgiref (the Python reference implementation) scripts available for comparison. Just add ref to the wsgi in the script name.

Leveraging PyRTE

Python scripts activated using the default persistent engine generally have low-latency, low-system-impact characteristics and seem to perform reliably and efficiently. The PyRTE generally handles all the WASD server interaction requirements. However there may be occasions where the application itself benefits from being persistent (many Python Web application are written with this in mind) or where more control needs to be exercised by the application designer. The WASD PyRTE provides a WASD Python module that provides an API for some of these aspects. Rather than clutter this document with such arcane detail the description may be found in the prologue to the PyRTE source code:

/wasd_root/src/python/pyrte.c

An example of where an application benefits from an explicitly CGIplus activation is MoinMoin:

MoinMoin is generally run independently to a system's primary Web server using the embedded Web server available for Python. Reverse proxy is often then use to provide and control access through to the persistent MoinMoin server.

Using the same principle of keeping MoinMoin persistent (for the obvious latency and efficiency benefits) it can be run and controlled directly by the WASD Web server using a CGIplus wrapper DCL procedure and a little Python glue (directly derived from persistence code already present in the MoinMoin package).

/wasd_root/src/python/moincgiplus.com

The same principles may be applied to any application environment (third-party or in-house) with similar characteristics. Python applications running as CGIplus scripts using the PyRTE must be activated via DCL wrappers as illustrated above (in part due to mapping requirements).

With CGIplus persistent applications configuration or code changes will in all probability require a 'restart' of the application using one or other of:

$ HTTPD/DO=PURGE
$ HTTPD/DO=DELETE

WATCHing Scripts

Debugging is always fun 

PyRTE provides some execution data via the WATCH report [X]Script item. For example (note the SCRIPTs):

|00:30:50.08 SERVICE  1735 093001 CONNECT    VIRTUAL klaatu.local:443|
|00:30:50.08 REQUEST  4318 093001 REQUEST    GET /py-bin/wsgi_test3.py|
|00:30:50.09 CACHE    0604 093001 RESPONSE   CACHE search path 5F1A99EFCB98C60126B8F244110E5BDD|
|00:30:50.09 DCL      1572 093001 RESPONSE   SCRIPT as HTTP$NOBODY RTE /py-bin/wsgi_test3.py wasd_root:[src.python.scripts]wsgi_test3.py ($cgi_exe:pyrte.exe)|
|00:30:50.10 DCL      7962 125001 SCRIPT     PYRTE AXP-2.0.0 Python 2.7.18 (default, Jul 23 2020, 17:40:05) [DECC]|
|00:30:50.10 DCL      7945 093001 SCRIPT     RTE caching /py-bin/wsgi_test3.py /WASD_ROOT/src/PYTHON/SCRIPTS/wsgi_test3.py|
|00:30:50.10 DCL      7945 093001 SCRIPT     CACHE new 1/1|
|00:30:50.10 DCL      7945 093001 SCRIPT     LOAD /py-bin/wsgi_test3.py /WASD_ROOT/src/PYTHON/SCRIPTS/wsgi_test3.py|
|00:30:50.10 DCL      7945 093001 SCRIPT     CODE /WASD_ROOT/src/PYTHON/SCRIPTS/wsgi_test3.py|
|00:30:50.11 DCL      7945 093001 SCRIPT     EVAL /py-bin/wsgi_test3.py|
|00:30:50.12 DCL      7945 093001 SCRIPT     PYRTE AXP-2.0.0 USAGE:1/19 REAL:00:00:00.02 CPU:0.02 DIO:3 BIO:36 FAULTS:0 PGFL:463040/8%|
|00:30:50.12 GZIP     0608 093001 RESPONSE   DEFLATE 1289->507 bytes, 39% (261kB)|
|00:30:50.12 REQUEST  1438 093001 REQUEST    STATUS 200 (OK) rx:78 tx:2415 bytes 1.045832 seconds 2,383 B/s|

Any reportable script error is indicated.

|00:33:50.17 DCL      7962 265001 SCRIPT     EVAL /py-bin/pyrte_test3.py|
|00:33:56.82 DCL      7962 265001 SCRIPT     ERROR <type 'exceptions.RuntimeError'> 'logical name PYRTE_CACHE_ENTRY not defined'|

It is also possible to add items from within the script Python code using the wasd.WATCH() function. The example /py-bin/pyrte_test1.py script contains the statement  wasd.WATCH('and hello [x]Script')

|00:35:15.05 DCL      7945 057001 SCRIPT     RTE caching /py-bin/pyrte_test1.py /WASD_ROOT/src/PYTHON/SCRIPTS/pyrte_test1.py|
|00:35:15.05 DCL      7945 057001 SCRIPT     CACHE new 1/1|
|00:35:15.05 DCL      7945 057001 SCRIPT     LOAD /py-bin/pyrte_test1.py /WASD_ROOT/src/PYTHON/SCRIPTS/pyrte_test1.py|
|00:35:15.05 DCL      7945 057001 SCRIPT     CODE /WASD_ROOT/src/PYTHON/SCRIPTS/pyrte_test1.py|
|00:35:15.06 DCL      7945 057001 SCRIPT     EVAL /py-bin/pyrte_test1.py|
|00:35:15.07 DCL      7945 057001 SCRIPT     WATCH and hello [x]Script|
|00:35:15.07 DCL      7945 057001 SCRIPT     PYRTE AXP-2.0.0 USAGE:1/1 REAL:00:00:05.10 CPU:5.06 DIO:65 BIO:1060 FAULTS:870 PGFL:476448/5%|

Problems?

Unfortunately the author of the PyRTE interface is such a Python novice he is not in any position to answer queries about Python "programming" or usage. There is the VMS Python Wiki

https://www.vmspython.org/

and support forum

https://forum.vmspython.org/

for VMS Python issues. If there's an obvious behavioural problem with PyRTE then contact the author or use the info-WASD mailing list.

Releases

v3.0.0  10-SEP-2020 ~~~ PRE-RELEASE ~~~
•  Python version 3.10.0a0
v2.0.0  10-SEP-2020 ~~~ PRE-RELEASE ~~~
•  Python version 2.7.18
v1.1.12  30-JUL-2017
•  wasd.cgi_callout()
•  bugfix
v1.0.0  22-APR-2007
•  initial

Acknowlegements

Many thanks to Jean-François Piéronne for his initial port of Python to VMS and his energetic, ongoing maintenance of the product.