|1Server Performance|
|0These Are v12.1 X86 Results|
|^ The server has a single-process, multi-threaded, asynchronous I/O design. On
a single-processor system this is the most efficient approach. On a
multi-processor system it is limited by the single process context (with
scripts executing within their own context). For I/O constrained processing
(the most common in general Web environments) the AST-driven approach is quite
efficient.
|^ The test-bench system was
|^+ |*Dell Optiplex 9020 4 core i7 3.4Ghz 16GB Win 10 Pro 22H2|
|^- |*innotek GmbH VirtualBox with 2 CPU and 7574MB running VMS V9.2|
|^- |*Approximate System VUPs Rating : 284.4 ( min: 281.4 max: 286.6 )|
|note|
|0... not the fastest kid on the block|
|^ but as with previous benchmarks, these are intended as relative not absolute
comparisons. Importantly, it has recent releases of system software, courtesy
of VSI's ISV support programme. For performance purposes, this allows
comparison with recent releases of CSWS (VMS Apache).
|!note|
|^ This performance data (WASD v12.1) has been collected very differently to
those over a decade ago (WASD v10.0). Earlier benchmarking tools were
WASD-in-house, ApacheBench (AB) and WASDbench (WB), executing on the same
system as the server, eliminating network traffic |/on-the-wire||.
|0Benchmark Setup|
|^ VMS V9.2 with VSI TCP/IP Services for OpenVMS x86_64 Version X6.0
|^ The data have been collected using the |/h2load|| utility
(|link%|https://nghttp2.org/documentation/h2load.1.html||) from the HTTP/2
C Library (|link%|https://nghttp2.org||). This utility can be used to
configurably load |*HTTP, HTTPS and HTTP/2| servers. Note that the number of
client threads (|=.-t||) is explicitly set to the connection concurrency
(|=.-c||) to maximise |/h2load|| processing. The |/h2load|| utility maintains
connection persistence between HTTP/1.1 requests so the network connection
setup is generally only a factor for the first of many.
|^ The |/h2load|| utility is running on a 6 core Intel Core i7 3.2 GHz 16GB Mac
Mini, across a 1Gbps LAN to the Dell.
|^ Shell scripts for |/h2load|| benchmarking runs are included in the
|link%|/wasd_root/exercise/perf*.|WASD_ROOT:[EXERCISE]PERF*.|
directory.
|^ Clear text HTTP (port 80) data is collected to measure internal server
processing without the CPU-intensive overhead of encryption. Encrypted HTTP
(port 443) data provides more real-world scenarios (especially now clear-text
is largely deprecated). Both WASD and Apache were using OpenSSL 3.0 and
negotiated TLS v1.3.
|^ Output from |/h2load|| benchmarking runs are included in the
|link%|/wasd_root/exercise/perf*1210*.txt|WASD_ROOT:[EXERCISE]PERF*1210*.TXT|
directory and is summarised below.
|0These results are indicative only!|
|^ Every endeavour has been made to ensure the comparison is as equitable as
possible. Both servers run on the same machine in the same relatively
quiescent environment, access logging and name lookup enabled. Each test run
was interleaved between each server to try and distribute any environment
variations. Those runs that are very high throughput use a larger number of
requests to improve sample period validity. Both servers were configured
pretty-much "out-of-the-box", minimal changes (generally just enough to get the
test environment going). Multiple data collections have yielded essentially
equivalent relative results.
|^ For the test-bench WASD v12.1 is present on ports 80 and 443.
|0Apache Comparison||
|^ The Apache comparison used the latest VSI X86VMS CSWS V2.4-54A kit (based on
Apache 2.4.54 and OpenSSL 3.0).
|^ Apache is present on ports 7780 and 7443.
|2Simple File Request Turn-Around|
|^ A series of tests using batches of accesses. The first test returned an
empty file measuring response and file access time, without any actual
transfer. The second requested a file of 64k characters, testing performance
with a more realistic load. All were done using one and ten concurrent
requests.
|block><|
|0_HTTP/1.1 clear|
|0Concurrency 1|
|tabular|
|~ |: |:2 Requests/Second|:2 Data Rate MBps
|~ |: Response|: WASD|: Apache|: WASD|: Apache
|~ |. 0k |. 1409 |. 104 |. 0.465 |. 0.026
|~ |. 64k |. 124 |. 78 |. 7.59 |. 4.8
|!tabular|
|0Concurrency 10|
|tabular|
|~ |: |:2 Requests/Second|:2 Data Rate MBps
|~ |: Response|: WASD|: Apache|: WASD|: Apache
|~ |. 0k |. 205 |. 12.5 |. 0.674 |. 0.032
|~ |. 64k |. 29 |. 8 |. 17.9 |. 5.0
|!tabular|
|0_HTTP/1.1 encrypted|
|0Concurrency 1|
|tabular|
|~ |: |:2 Requests/Second|:2 Data Rate MBps
|~ |: Response|: WASD|: Apache|: WASD|: Apache
|~ |. 0k |. 869 |. 91 |. 0.319 |. 0.023
|~ |. 64k |. 116 |. 59 |. 7.14 |. 3.6
|!tabular|
|0Concurrency 10|
|tabular|
|~ |: |:2 Requests/Second|:2 Data Rate MBps
|~ |: Response|: WASD|: Apache|: WASD|: Apache
|~ |. 0k |. 606 |. 135 |. 0.223 |. 0.034
|~ |. 64k |. 97 |. 79 |. 5.95 |. 4.83
|!tabular|
|0_HTTP/2 (encrypted)|
|^ (VMS Apache currently does not support HTTP/2)
|0Concurrency 1|
|tabular|
|~ |: |:2 Requests/Second|:2 Data Rate MBps
|~ |: Response|: WASD|: Apache|: WASD|: Apache
|~ |. 0k |. 496 |. - |. 0.068 |. -
|~ |. 64k |. 107 |. - |. 6.53 |. -
|!tabular|
|0Concurrency 10|
|tabular|
|~ |: |:2 Requests/Second|:2 Data Rate MBps
|~ |: Response|: WASD|: Apache|: WASD|: Apache
|~ |. 0k |. 437 |. - |. 0.061 |. -
|~ |. 64k |. 90 |. - |. 5.52 |. -
|!tabular|
|!block|
|^ Data file (extraneous output snipped):
|simple#|
|item| |link%|/wasd_root/exercise/perf_files_1210.txt|\
WASD_ROOT:[EXERCISE]PERF_FILES_1210.TXT|
|!simple|
|0File Transfer Rate|
|^ Requests for a large |/binary| file (5.69MB - 11673 blocks) indicate a
|*potential transfer rate of multiple tens of Mbytes per second||.
|block><|
|0Data Rate - MBytes/Second|
|^ (VMS Apache currently does not support HTTP/2)
|tabular|
|~ |. |: Concurrent|: WASD|: Apache
|~ |:12 HTTP/1.1\
(clear) |. 1 |. 33.71 |. 20.54
|~ |. 10 |. 35.51 |. 32.61
|~ |:12 HTTP/1.1\
(encrypted) |. 1 |. 11.03 |. 9.16
|~ |. 10 |. 11.82 |. 12.25
|~ |:12 HTTP/2\
(encrypted) |. 1 |. 11.93 |. -
|~ |. 10 |. 12.00 |. -
|!tabular|
|!block|
|^ Data file (extraneous output snipped):
|simple|
|& |link%|/wasd_root/exercise/perf_xfer_1210.txt|\
WASD_ROOT:[EXERCISE]PERF_XFER_1210.TXT|
|!simple|
|0File Record Format||
|^ The WASD server can handle STREAM, STREAM_LF, STREAM_CR, FIXED and UNDEFINED
record formats very much more efficiently than VARIABLE or VFC files. With
STREAM, FIXED and UNDEFINED files the assumption is that HTTP carriage-control
is within the file itself (i.e. at least the newline (LF), all that is required
required by browsers), and does not require additional processing. With
VARIABLE record files the carriage-control is implied and therefore each record
requires additional processing by the server to supply it. Even with variable
record files having multiple records buffered by the HTTPd before writing them
collectively to the network improving efficiency, stream and binary file reads
are by Virtual Block and are written to the network immediately making the
transfer of these very efficient indeed!
|2Scripting|
|^ A simple performance evaluation shows the relative merits of WASD scripting
and Apache in CGI and persistent environments, using
|link%|/wasd_root/src/cgiplus/cgiplustest.c|\
WASD_ROOT:[SRC.CGIPLUS]CGIPLUSTEST.C|
which executes in standard CGI, CGIplus and Apache loadable module
environments. CGIplus and Apache modules are somewhat analagous. A series of
accesses were made. The first test returned only the HTTP header, evaluating
raw request turn-around time. The second test requested a body of 64k
characters, again testing performance with a more realistic load.
|block><|
|0Concurrency 1 - Requests/Second|
|tabular|
|~ |: Response|: WASD CGI|: WASD CGIplus|: Apache CGI|: Apache module
|~ |. 0kB |. 19 |. 143 |. 7 |. 77
|~ |. 64kB |. 12 |. 18 |. 6 |. 45
|!tabular|
|0Concurrency 10 - Requests/Second|
|tabular|
|~ |: Response|: WASD CGI|: WASD CGIplus|: Apache CGI|: Apache module
|~ |. 0kB |. 28 |. 337 |. 10 |. 51
|~ |. 64kB |. 22 |. 65 |. 9 |. 73
|!tabular|
|!block|
|^ Data file (extraneous output snipped):
|simple#|
|& |link%|/wasd_root/exercise/perf_scripts_1210.txt|\
WASD_ROOT:[EXERCISE]PERF_SCRIPTS_1210.TXT|
|!simple|
|0Persistent Scripting|
|^ CGI scripting is notoriously slow (as above), hence the effort
expended by designers in creating persistent scripting environments - those
where the scripting engine (and perhaps other state) is maintained between
requests. Both WASD and Apache implement these as integrated features,
the former as |*CGIplus/RTE||, and in the latter as |*loadable modules||.
|^ The |/CGIplus| and |/Apache module| data from the above CGIPLUSTEST.EXE
table show the benefits of having scripts persist, reducing activation latency,
thereby increasing throughput, and potentially retaining state, including the
scripts themselves in local caches. Both WASD and VMS Apache use their
respective |*persistence technologies| to provide common scripting
environments, including |*Perl||, |*PHP| and |*Python||.
|^ The WASD CGIplus/RTE technology used to implement its persistent scripting
environments are available for general use and based on CGI principles offer a
ready adaptation of well-known principles. Most site-specific scripts can also
be built using the libraries, code fragments, and example scripts provided
with the WASD package, and obtain similar efficiencies and low latencies.
See |link%|../../scripting/scripting.html|WASD Scripting Environment| document.