Copyright © 2007-2020 Mark G. Daniel
This program, comes with ABSOLUTELY NO WARRANTY.
This is free software, and you are welcome to redistribute it under the
conditions of the GNU GENERAL PUBLIC LICENSE, version 3, or any later version.
https://www.gnu.org/licenses/gpl.txt
PyRTE is an interface to a Python interpreter engine and environment for the WASD VMS Web server. It is designed to be able to be used in standard CGI and CGIplus/RTE persistent scripting modes. The persistent modes (default) provide a ~35x (yes, an approximate thirty-five times) improvement in script activation times (compared to CGI/command-line) and comparably reduced load on both server and system. Note that this package does not contain the Python interpreter or kit, that has to be obtained and installed separately (a simple matter as it is provided via a PCSI package).
PyRTE is linked against the Python interpreter shareable image and so shares a set of Python capabilities in common with command-line OpenVMS Python. Alpha (AXP) and Itanium object modules are provided for the VMS platforms available for Python.
PyRTE has been developed and tested on V8.4, Alpha and Itanium, using WASD v11.n and VMS Python kits available from and documented at https://www.vmspython.org/. JFP's Python kits are kept very up-to-date as Python updates or implementation problems arise.
FYI: Beginning August 2020, PyRTE underwent significant rework
prompted by the pre-release of JFP's Python v3.10 port. The core PyRTE code
now supports both Python 2.7 and 3.10 as independent builds and executables.
PyRTE reports itself as v2.n.n for the Python 2.7 build, and as
v3.n.n for Python 3.10.
Installation
https://www.vmspython.org/and install according to the instructions available.
$ RENAME WASD_ROOT:[SRC]PYTHON.DIR WASD_ROOT:[SRC]PYTHON_nnn.DIR
https://wasd.vsm.com.au/wasd/
$ SET DEFAULT WASD_ROOT:[000000] $ UNZIP "-V" location:PYRTEnnn.ZIP
$ SET DEFAULT WASD_ROOT:[SRC.PYTHON] $ @BUILD_PYRTE LINKFor Python 3:
$ @BUILD_PYRTE3 LINK
$ COPY WASD_EXE:PYRTE.EXE CGI_EXE:For Python 3:
$ COPY WASD_EXE:PYRTE3.EXE CGI_EXE:
Although there are experimentation modes built into PyRTE basically it will be used as a WASD Run-Time Environment providing persistent Python interpreter(s) and environment(s) for Python scripts and applications. Benchmarking indicates the CGIplus/RTE use reduces activation time to 3% of CGI (yes, that's correct, by a factor 35 - a python interpreter is quite expensive to instantiate). There are subtle differences in the way CGIplus and RTE parse and provide the PATH_INFO data (see the "WASD Scripting Overview" for detail).
# WASD_CONFIG_GLOBAL [DclScriptRunTime] .PY $CGI-BIN:[000000]PYRTE [AddType] .PY text/plain Python source .PYC application/octet-stream Python byte-code .PYO application/octet-stream Python optimised byte-codeFor Python 3:
# WASD_CONFIG_GLOBAL [DclScriptRunTime] .PY $CGI-BIN:[000000]PYRTE3
# WASD_CONFIG_MAP
exec /py-bin/* (cgi_exe:pyrte)/cgi-bin/* \
script=syntax=unix script=query=none map=once ods=5
For Python 3:
# WASD_CONFIG_MAP
exec /py-bin/* (cgi_exe:pyrte3)/cgi-bin/* \
script=syntax=unix script=query=none map=once ods=5
# WASD_CONFIG_MAP for automatic RTE usage
map /cgi-bin/*.py* /py-bin/*.py*
exec /py-bin/* (cgi_exe:pyrte)/cgi-bin/* \
script=syntax=unix script=query=none map=once ods=5
For Python 3:
exec /py-bin/* (cgi_exe:pyrte3)/cgi-bin/* \
script=syntax=unix script=query=none map=once ods=5
# WASD_CONFIG_MAP
exec /web/**.py* /web/*.py* \
script=syntax=unix script=query=none map=once ods=5
# WASD_CONFIG_MAP
exec /appx/**.py* /appx_root/*.py* \
script=syntax=unix script=query=none map=once ods=5 \
script=as=USERX
# WASD_CONFIG_MAP
exec /py-bin/* (cgi_exe:pyrte)/wasd_root/src/python/scripts/* \
script=syntax=unix script=query=none map=once ods=5
For Python 3:
exec /py-bin/* (cgi_exe:pyrte3)/wasd_root/src/python/scripts/* \
script=syntax=unix script=query=none map=once ods=5
Note that the location of the above example scripts has changed to [.SCRIPTS]
since the v1.0 release.
Remember to
$ HTTPD /DO=MAPafter changing mapping rules.
It is sometimes useful, sometimes necessary, to maintain multiple versions of Python on a system, and also support multiple versions to web applications (scripts). It is not difficult to have PyRTE activate a specific version by wrapping the executable in a procedure that sets up the appropriate Python environment in a process-local context.
$! CGI_BIN:PYRTE.COM $! activate JFP Python 2.7 $ @disk$jfplib0020i:[000000]lib_logicals.com "/process" $ @disk$jfppy1400i:[000000]python_logicals.com "/process" $ @python_root:[vms]setup $ pyrte == "$cgi_exe:pyrte.exe" $ pyrte
$! CGI_BIN:PYRTE3.COM $! activate JFP Python 3.10 $ @disk$jfplib0020i:[000000]lib_logicals.com "/process" $ @disk$jfppy3100i:[python3100.vms]logicals.com "/process" $ @python3_root:[vms]setup $ pyrte3 == "$cgi_exe:pyrte3.exe" $ pyrte3
Each of these is mapped for independent activation based on the request path.
# WASD_CONFIG_MAP
exec /py-bin/* (@cgi_bin:pyrte.com)/wasd_root/src/python/scripts/* \
script=syntax=unix map=once script=query=none ods=5
exec /py3-bin/* (@cgi_bin:pyrte3.com)/volume**/wasd_root/src/python/scripts/* \
script=syntax=unix map=once script=query=none ods=5
| ** |
Note that the scripts activated are the same on-disk files, located in
the PyRTE source directory.
Also note that /py3-bin/ mapping result is different to that of the /py-bin/ mapping. There is a /volume/ present. This is only required where the same on-disk files are being scripted. This is a kludge. Due to the way WASD caches script information (on the disk file path), including any associated RTE, it was necessary to differentiate the two activations and the chosen solution was to include the physical volume (on which [WASD_ROOT] resides) in the mapping. Another solution might be to use an alternate concealed logical device name. |
This is one approach. A little creativity would yield others.
At the time of writing the demonstration site uses this approach to
default the examples below to (JFP) Python 2.7, while optionally allowing them
to be activated under 3.10 – just modify the browser location field to
contain /py3-bin/ rather than the default /py-bin/. Several
of the scripts display the underlying Python version confirming the activations.
Example Scripts
After configuration (as described immediately above) the following scripts may be used to confirm the environment is functioning.
Some of these examples also support CGIplus mode for comparison purposes.
Of course the Python code in the script may be inspected from the source directory:
/wasd_root/src/python/scripts/
PyRTE contains a Web Server Gateway Interface v1.0 (WSGI). This is a Python standard specification for web servers and application servers to communicate with web applications (though it can also be used for more than that).
The WSGI code has been developed under the sponsorship of SysGroup
http://www.sysgroup.fr/and generously made available to the wider WASD community.
The following is an example of a simple (and classic) WSGI application being activated using the PyRTE wasd.wsgi_run() function.
def hello_world (environ, start_response):
status = '200 OK'
response_headers = [('Content-type','text/plain')]
start_response(status, response_headers)
return ['Hello world!\n']
import wasd
wasd.wsgi_run(hello_world)
PEP 333 specifies that WSGI should not buffer output and PyRTE complies but some tools (like Mercurial) don't buffer internally themselves and end up sending many very small records. This is obviously inefficient. If buffered the throughput boost for Mercurial is about x4. Well worth having for specific Web applications.
This forced buffering may be enabled on a per-application basis using a mapping rule, for example
set /mercurial/* script=param=PYRTE=/WSGI=BUFFERor by defining the logical name
$ DEFINE /JOB PYRTE_WSGI_FORCE_BUFFERING TRUEin a wrapper procedure.
The following scripts may be used to confirm the environment is functioning.
The Python code in the script may be inspected from the source directory:
/wasd_root/src/python/scripts/
There are also equivalent wsgiref (the Python reference
implementation) scripts available for comparison. Just add ref to the
wsgi in the script name.
Leveraging PyRTE
Python scripts activated using the default persistent engine generally have low-latency, low-system-impact characteristics and seem to perform reliably and efficiently. The PyRTE generally handles all the WASD server interaction requirements. However there may be occasions where the application itself benefits from being persistent (many Python Web application are written with this in mind) or where more control needs to be exercised by the application designer. The WASD PyRTE provides a WASD Python module that provides an API for some of these aspects. Rather than clutter this document with such arcane detail the description may be found in the prologue to the PyRTE source code:
/wasd_root/src/python/pyrte.c
An example of where an application benefits from an explicitly CGIplus activation is MoinMoin:
MoinMoin is generally run independently to a system's primary Web server using the embedded Web server available for Python. Reverse proxy is often then use to provide and control access through to the persistent MoinMoin server.
Using the same principle of keeping MoinMoin persistent (for the obvious latency and efficiency benefits) it can be run and controlled directly by the WASD Web server using a CGIplus wrapper DCL procedure and a little Python glue (directly derived from persistence code already present in the MoinMoin package).
/wasd_root/src/python/moincgiplus.com
The same principles may be applied to any application environment (third-party or in-house) with similar characteristics. Python applications running as CGIplus scripts using the PyRTE must be activated via DCL wrappers as illustrated above (in part due to mapping requirements).
With CGIplus persistent applications configuration or code changes will in all probability require a 'restart' of the application using one or other of:
$ HTTPD/DO=PURGE $ HTTPD/DO=DELETE
Debugging is always fun ☹
PyRTE provides some execution data via the WATCH report [X]Script item. For example (note the SCRIPTs):
|00:30:50.08 SERVICE 1735 093001 CONNECT VIRTUAL klaatu.local:443| |00:30:50.08 REQUEST 4318 093001 REQUEST GET /py-bin/wsgi_test3.py| |00:30:50.09 CACHE 0604 093001 RESPONSE CACHE search path 5F1A99EFCB98C60126B8F244110E5BDD| |00:30:50.09 DCL 1572 093001 RESPONSE SCRIPT as HTTP$NOBODY RTE /py-bin/wsgi_test3.py wasd_root:[src.python.scripts]wsgi_test3.py ($cgi_exe:pyrte.exe)| |00:30:50.10 DCL 7962 125001 SCRIPT PYRTE AXP-2.0.0 Python 2.7.18 (default, Jul 23 2020, 17:40:05) [DECC]| |00:30:50.10 DCL 7945 093001 SCRIPT RTE caching /py-bin/wsgi_test3.py /WASD_ROOT/src/PYTHON/SCRIPTS/wsgi_test3.py| |00:30:50.10 DCL 7945 093001 SCRIPT CACHE new 1/1| |00:30:50.10 DCL 7945 093001 SCRIPT LOAD /py-bin/wsgi_test3.py /WASD_ROOT/src/PYTHON/SCRIPTS/wsgi_test3.py| |00:30:50.10 DCL 7945 093001 SCRIPT CODE /WASD_ROOT/src/PYTHON/SCRIPTS/wsgi_test3.py| |00:30:50.11 DCL 7945 093001 SCRIPT EVAL /py-bin/wsgi_test3.py| |00:30:50.12 DCL 7945 093001 SCRIPT PYRTE AXP-2.0.0 USAGE:1/19 REAL:00:00:00.02 CPU:0.02 DIO:3 BIO:36 FAULTS:0 PGFL:463040/8%| |00:30:50.12 GZIP 0608 093001 RESPONSE DEFLATE 1289->507 bytes, 39% (261kB)| |00:30:50.12 REQUEST 1438 093001 REQUEST STATUS 200 (OK) rx:78 tx:2415 bytes 1.045832 seconds 2,383 B/s|
Any reportable script error is indicated.
|00:33:50.17 DCL 7962 265001 SCRIPT EVAL /py-bin/pyrte_test3.py| |00:33:56.82 DCL 7962 265001 SCRIPT ERROR <type 'exceptions.RuntimeError'> 'logical name PYRTE_CACHE_ENTRY not defined'|
It is also possible to add items from within the script Python code using the wasd.WATCH() function. The example /py-bin/pyrte_test1.py script contains the statement wasd.WATCH('and hello [x]Script')
|00:35:15.05 DCL 7945 057001 SCRIPT RTE caching /py-bin/pyrte_test1.py /WASD_ROOT/src/PYTHON/SCRIPTS/pyrte_test1.py| |00:35:15.05 DCL 7945 057001 SCRIPT CACHE new 1/1| |00:35:15.05 DCL 7945 057001 SCRIPT LOAD /py-bin/pyrte_test1.py /WASD_ROOT/src/PYTHON/SCRIPTS/pyrte_test1.py| |00:35:15.05 DCL 7945 057001 SCRIPT CODE /WASD_ROOT/src/PYTHON/SCRIPTS/pyrte_test1.py| |00:35:15.06 DCL 7945 057001 SCRIPT EVAL /py-bin/pyrte_test1.py| |00:35:15.07 DCL 7945 057001 SCRIPT WATCH and hello [x]Script| |00:35:15.07 DCL 7945 057001 SCRIPT PYRTE AXP-2.0.0 USAGE:1/1 REAL:00:00:05.10 CPU:5.06 DIO:65 BIO:1060 FAULTS:870 PGFL:476448/5%|
Unfortunately the author of the PyRTE interface is such a Python novice he is not in any position to answer queries about Python "programming" or usage. There is the VMS Python Wiki
https://www.vmspython.org/
and support forum
https://forum.vmspython.org/
for VMS Python issues. If there's an obvious behavioural problem with
PyRTE then contact the author or use the info-WASD mailing list.
Releases
Many thanks to Jean-François Piéronne for his initial port of Python to VMS and his energetic, ongoing maintenance of the product.