/*****************************************************************************/ /* screper.c ... a terminal screen screper, er scréper, um scraper. HOW IT WORKS ------------ Getting the VT100-style terminal output of a "regular" command-line application onto a browser HTML page. The parent application (script) using an XMLHttpRequest() activates a wrapper for the routines in this module. This spawns a subprocess which initiates the commnd-line application and a screen of terminal output is generated. This output is parsed with VT100/ANSI cursor, character attributes, and other sequences, building a virtual, in-memory screen. This generates HTML markup to represent the screen in a browser. The HTML is streamed as a response to the XMLHttpRequest() with the "snapshot" representing each terminal screen. A stream is a series of time-based snapshots. The idea is that a screen is built from output fairly quickly, and may have a well-defined update rate. The snapshotter ticks ten times a second and when the virtual screen has not received new display data for the specified number of ticks a snapshot of that virtual screen (in HTML markup) is sent to the parent running in the browser. For example, the MONITOR update rate is roughly 6 seconds. So a snapshot value of one-tenth second ticks of 55 will snapshot after 5.5 seconds of MONITOR quiesence. Just enough not to update too frequently but not to update in the middle of new data being received. There is probably a sweetspot for each update interval and application. This time-based approach is not suitable for continuously or randomly updating screens, and congested systems can result in partial or broken displays. An alternative to a time-based update is to trigger a snapshot on the detection of unique octet sequence. For example, if the clear-screen CSI sequence is used to begin a fresh screen display then the octet sequence, 0x1b 0x5b 0x32 0x4a, then detecting this in the terminal output can be used to generate a snapshot before the output is sent to to scraper for processing. Such a sentinal must be terminated by 2 null characters. Multiple sentinals can be inside the string by separating each with a single null. Obviously a null character cannot be used as part of the sentinal. To be detected a sentinal must be self-contained in a single output record. Here is an example os a single sentinal string. uchar example [] = { 0x1b, 0x5b, 0x32, 0x4a, 0x00, 0x00 }; SCREPER DOES NOT PRETEND ------------------------ to be an exhaustive VT terminal interpreter. The development baseline application was the MONITOR utility. Screper seems to handle MONITOR with considerable aplomb. However, no little wonder that terminal emulator authors seldom get it 100% and quickly move on to something else. SCREPER API ----------- ScreperInit() returns a pointer to an opaque structure (void*). ScreperDone() deallocates memory used during processing and returns a NULL. ScreperDo() is supplied a string which is processed to set parameters and/or perform a DCL command. Directives must be lower case, include at least the first three characters, and have no white space between the directive, equate symbol, and value. Some can only be used standlone. A single or multiple calls may be made to set up the virtual terminal session. The final call must contain the DCL commands. -7bit terminal 7 bits -8bit terminal 8 bits (default) -dcl= DCL command(s) (to end of line) -device= e.g. value from ttdef.h (default TT$_VT100) -noerror do not detect -E- status and exit -ctrlz supply a ^Z to the DCL command (also see -input) -eot force send an EOT (subprocess not restarted) -ext force send an EXT (subprocess is restarted) -exit="" a string when matched in output triggers an exit -noeot do not send an EOT -noext do not send an EXT -nofatal do not detect -F- status and exit -input="" supply this string as input to the command -inspect= inspect the data stream at various levels -page=[+] number of lines on page (24..255) -noecho do not echo the command -noprogress= if output stalls for this many seconds (default 30) -repeat= repeat every seconds -sentinal="" when output contains this it triggers a snapshot -scroll .scrollIntoView() -snapshot= tenths of a second before snapshot (-n, 0, 5..600) -timestamp[=""] begin the display with a timestamp -utility="" name of the utility displayed with the pause button -wait[=] hibernate (sleep) seconds (default is forever) -nowarn do not detect -W- status and exit -width= width of line, 80 or 132 ... 255 chars (default 80) -css return style sheet -javascript return JavaScript -pause return HTML for a "pause all" button -resize return JavaScript to resize an iframe -screen return HTML providing virtual screen -version return version as string (standalone) The -input string, along with the -sentinal and -timestamp strings, must be quotation character delimited, and can include control codes and other "exotic" characters using the ^xx escape syntax. So inputing a control-Z would be ^1a, a literal hat character ^3e, and literal quotation ^22. The -timestamp has a default VMS date-time. A format string may be supplied allowing VMS date/time formatting (e.g. "!8%T", "!20%D") or strftime() formatting (e.g. "%c", "%A, %B %d, %Y"). Other characters and HTML markup may lead and trail the string formatting but the characters '!' and '%' are considered reserved and introduce the format string. HTML-entify if required as literals. See example applications for usage. INTERNAL SENTINALS ------------------ Various "states" are communicated from the DCL commands strings using shebang-commented sentinal strings that are detected and acted upon by spawned output stream processor, as well as in the reverse direction, from the HTML-ified terminal output stream to the JavaScript browser page "hidden" in HTML-comment braces. All DCL sentinals begin with the longword "!~!~", in part making them cheaper to detect in the stream. All HTML-embedded sentinals occur as HTML-commented abbreviations (e.g. ). Not all entinals are present in both DCL and HTML streams. EOT, ETX, STX and SYN are used for various purposes. STX sentinal marks the start of meaningful output. Up until the STX sentinal is detected output from the spawned terminal session is ignored. EOT sentinal indicates the spawning script should exit gracefully leaving the HTML page intact on the browser. ETX sentinal indicates the end of the current screen's output and the spawning script should wait for another screenful. The terminal session periodically send SYN sentinals (as ) to keep the script output from no-progress timeout. OPCOM ----- Sometimes it is difficult to get meaningful (debug) data from this sort of application. The ScreperOpcom() function may be used to generate messages containing basically anything useful. The application executable must be installed with /PRIVILEGE=OPER in addition to anything else required. RESOURCES --------- https://vt100.net/ https://vt100.net/docs/vt100-ug/chapter3.html https://vt100.net/docs/vt510-rm/chapter7.html http://fileformats.archiveteam.org/wiki/DEC_Special_Graphics_Character_Set https://en.wikipedia.org/wiki/VT100_encoding https://developer.mozilla.org/en-US/docs/Web/HTTP/CSP https://en.wikipedia.org/wiki/Content_Security_Policy COPYRIGHT --------- Copyright (C) 2021-2023 Mark G.Daniel Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. VERSION LOG ----------- 06-OCT-2022 MGD v1.1.0, CSP same-origin 'strict-dynamic' constraints noprogress and error processing means LOTS more timers -exit is now -ctrlz major changes to internal subprocess processing 01-AUG-2021 MGD v1.0.0, initial */ /*****************************************************************************/ #define SOFTWAREVN "v1.1.0" #define SOFTWARENM "SCREPER" #ifdef __ALPHA # define SOFTWAREID SOFTWARENM " " SOFTWAREVN " AXP" #endif #ifdef __ia64 # define SOFTWAREID SOFTWARENM " " SOFTWAREVN " IA64" #endif #ifdef __x86_64 # define SOFTWAREID SOFTWARENM " " SOFTWAREVN " X86" #endif #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include "screper.h" #define FI_LI "SCREPER",__LINE__ #define DC$_TERM 6 #ifndef UINT64PTR /* mainly to allow easy use of the __unaligned directive */ #define UINTPTR __unaligned unsigned int* #define ULONGPTR __unaligned unsigned long* #define USHORTPTR __unaligned unsigned short* #define UINT64PTR __unaligned uint64* #define INT64PTR __unaligned int64* #endif /* Each cursor position in the virtual screen is 32 bits deep. 0 7 8 15 16 23 24 31 |................|................|................|................| | 8 bit char | SGR bits | G0 alpha | G1 alpha | (these two fields are extravagant) */ #define SGR_BOLD 0x00000100 #define SGR_UNDER 0x00000200 #define SGR_BLINK 0x00000400 #define SGR_REVER 0x00000800 #define SGR_RESET 0x00004000 #define SGR_G1 0x00008000 #define CHAR_MASK 0x000000ff #define SGR_MASK 0x0000ff00 #define G0_MASK 0x00ff0000 #define G1_MASK 0xff000000 #define MAX_PAGE 255 #define MAX_WIDTH 255 #define BUTTON_PAUSE "| |" #define BUTTON_UNPAUSE ">>" #define BUTTON_PAUSEALL "Pause All" #define BUTTON_UNPAUSEALL "Resume All" #define DEFAULT_NO_PROGRESS 30 #define DELTA64_100_MSEC ((int64)-1000000) #define DELTA64_ONE_SEC ((int64)-10000000) static ulong SentLong; char SentEOT [16]; char SentETX [16]; char SentSTX [16]; static ulong PromptLong; static int CliPromptLength; static char CliPrompt [16]; static int64 etxTimer64 = DELTA64_ONE_SEC; static int64 noProgTimer64 = DELTA64_ONE_SEC; static int64 snapTimer64 = DELTA64_100_MSEC; static int64 synTimer64 = DELTA64_ONE_SEC * 60; static char SoftwareId [] = SOFTWAREID; # define CSAVE_MAX 16 #pragma __member_alignment __save #pragma __member_alignment static struct ScreperStruct { int AnsiMode, CharTableG0, CharTableG1, CurSave, EfnRead, FontSize, InputLength, InspectStream, LibSpawnStatus, NoEcho, NoEOT, NoETX, NoError, NoFatal, NoPrompt, NoProgressSecs, NoWarning, PtdBits8, PtdDevice, PtdPageLength, PtdPagePlus, PtdPageWidth, PtdReadAst, PtdReadStatus, PtdWriteStatus, RepeatCount, RepeatSecs, RespondLength, ScreenBufferSize, ScreenCol, ScreenRow, ScrollIntoView, ScrolledCount, SendEOT, SendETX, SentDetected, SgrBlink, SgrBold, SgrEoj, SgrGrone, SgrReverse, SgrG1, SgrShot1, SgrUnder, SnapshotDataCount, SnapshotSentinalLength, SnapshotTenths, SpecGraphMode, StxDetected, TimeStampLength, UtilityLength, WaitSecs; int ColSave [CSAVE_MAX], RowSave [CSAVE_MAX], SgrSave [CSAVE_MAX]; int64 EotScrPtr, SnapScrPtr, SynScrPtr, TmoScrPtr; ulong LibSpawnPid, ScriptJpiPid; ulong inadr [2]; ulong chbuf [3]; ushort PtdChan; char DclBuffer [512], InputBuffer [64], PrcNam [16], PtdDevName [64], ExitBuffer [128], ScriptPidString [16], SgrBuffer [128], SnapshotSentinal [64], TimeStamp [64], Utility [64]; char *CspHeader, *CspNonce, *JavaScriptPage, *JavaScriptPause, *JavaScriptResize, *NextWritePtr, *PtdReadBuffer, *PtdWriteBuffer, *RepeatCmdPtr; FILE *ScrOut; ulong *ScreenBuffer; } ScreperStruct; typedef struct ScreperStruct SCREPER_STRUCT; #pragma __member_alignment __restore #define PTD_READ_SIZE 18 * 512 #define PTD_WRITE_SIZE 2 * 512 #define PTD_BUFFER_PAGES 20 /* private prototypes */ static int PtdDollarWrite (ushort, char*, int); static void PtdNoProgress (int64*); static void PtdPoof (SCREPER_STRUCT*, int); static void PtdRead (SCREPER_STRUCT*); static void PtdScreperEot (int64*); static void PtdScreperSyn (int64*); static int PtdSpawn (SCREPER_STRUCT*); static void PtdSpawnAst (SCREPER_STRUCT*); static int PtdWrite (SCREPER_STRUCT*); static char* ScreenChar (SCREPER_STRUCT*, ulong); static char* ScreperCSP (void*); static void ScreenCursor (SCREPER_STRUCT*, char*, ...); static void ScreenDump (SCREPER_STRUCT*, char*, int); static void ScreenIndex (SCREPER_STRUCT*, int*, int*); static void ScreenNextLine (SCREPER_STRUCT*, int*, int*); static void ScreenReverseIndex (SCREPER_STRUCT*, int*, int*); static char* ScreperBegin (SCREPER_STRUCT*); static void ScreperData (SCREPER_STRUCT*, char*, int); static void ScreperDump (SCREPER_STRUCT*, char*, int); static char* ScreperHtmlEncode (char*); static void ScreperKnown (SCREPER_STRUCT*, char*, char*, char*); static void ScreperMore (SCREPER_STRUCT*, char*, int); static void ScreperPrint (SCREPER_STRUCT*, char*, int); static void ScreperUnknown (SCREPER_STRUCT*, char*, char*, char*); static void ScreenOutput (SCREPER_STRUCT*); static char* ScreperPauseAll (); static char* ScreperScreen (); static void ScreenReport (SCREPER_STRUCT*, char*, ...); static void ScreenReset (SCREPER_STRUCT*); static void ScreenScrape (SCREPER_STRUCT*, char*, int); static int ScreenSentinal (SCREPER_STRUCT*, char*, int); static void ScreenSnapshot (int64*); static void ScreenTimeStamp (SCREPER_STRUCT*); static char* ConfigString (char*, char*, int, int*); /*************/ /* code page */ /*************/ static char *CharTable_0_31 [32] = { ".", ".", ".", ".", ".", ".", ".", ".", ".", ".", ".", ".", ".", ".", ".", ".", ".", ".", ".", ".", ".", ".", ".", ".", ".", ".", ".", ".", ".", ".", ".", "." }; static char *CharTable_32_94 [63] = { " ", "!", "\"", "#", "$", "%", "&", "'", "(", ")", "*", "+", ",", "-", ".", "/", "0", "1", "2", "3", "4", "5", "6", "7", "8", "9", ":", ";", "<", "=", ">", "?", "@", "A", "B", "C", "D", "E", "F", "G", "H", "I", "J", "K", "L", "M", "N", "O", "P", "Q", "R", "S", "T", "U", "V", "W", "X", "Y", "Z", "[", "\\", "]", "^" }; static char *CharTable_95_127 [33] = { "_", "`", "a", "b", "c", "d", "e", "f", "g", "h", "i", "j", "k", "l", "m", "n", "o", "p", "q", "r", "s", "t", "u", "v", "w", "x", "y", "z", "{", "|", "}", "~", "" }; /* special graphics */ static char *CharTable_SG_95_127 [33] = { " ", "♦", "░", "⇥", "⇊", "⇥", "⇊", "°", "±", "↤", "↧", "┘", "┐", "┌", "└", "├", "─", "─", "─", "─", "─", "├", "┤", "┴", "┬", "│", "≤", "≥", "π", "≠", "£", "•" }; static char *CharTable_128_255 [128] = { "Ä", "Å", "Ç", "É", "Ñ", "Ö", "Ü", "á", "à", "â", "ä", "ã", "å", "ç", "é", "è", "Ý", "°", "¢", "£", "§", "¸", "¶", "ß", "®", "©", "™", "´", "¨", "≠", "Æ","Ø", "×", "±", "≤", "≥", "¥", "µ", "¹", "²", "³", "π", "¦", "ª", "º", "▒", "æ", "ø", "¿", "¡", "¬", "½", "ƒ", "¼", "¾", "«", "»", "…", "�", "À", "Ã", "Õ", "Œ", "œ", "–", "—", "┘", "┐", "┌", "└", "÷", "◆", "ÿ", "Ÿ", "┼", "€", "Ð", "ð", "Þ", "þ", "ý", "·", "⎺", "⎻", "─", "Â", "Ê", "Á", "Ë", "È", "Í", "Î", "Ï", "Ì", "Ó", "Ô", "", "Ò", "Ú", "Û", "Ù", "⎼", "⎽", "├", "ড়", "┴", "┬", "│", "", "", "", "" }; /**************/ /* JavaScript */ /**************/ static char PageJavaScript [] = "\n"; /* ~~~~~~~~~~~~~~~~~~~~ Iframe meta characteristics can only be managed via the parent page where multiple terminals are displayed. */ static char ResizeJavaScript [] = "\n"; /* ~~~~~~~~~~~~~~~~~~~~ Pause-all button on parent page used where multiple terminals displayed. */ static char ButtonPauseAll [] = "\n\ \n"; /*******/ /* CSS */ /*******/ /* ensure monospace is explcitly specified!! */ static char PageCSS [] = "\ \n\ "; /*****************************************************************************/ /* Allocate the structure and set some defaults. */ void* ScreperInit () { char *cptr; SCREPER_STRUCT *scrptr; /*********/ /* begin */ /*********/ scrptr = calloc (1, sizeof(SCREPER_STRUCT)); if (!scrptr) exit (vaxc$errno); scrptr->PtdBits8 = TT$M_EIGHTBIT; scrptr->PtdDevice = TT$_VT100; scrptr->PtdPageLength = 24; scrptr->PtdPageWidth = 80; scrptr->UtilityLength = sprintf (scrptr->Utility, "-utility"); scrptr->ScrOut = fdopen (dup (fileno (stdout)), "w"); if (!(cptr = getenv ("WWW_CSP_NONCE"))) cptr = getenv ("CSP_NONCE"); if (cptr) { scrptr->CspNonce = calloc (1, strlen(cptr)+1); strcpy (scrptr->CspNonce, cptr); } return ((void*)scrptr); } /*****************************************************************************/ /* Release resources. */ void* ScreperDone (void *vscrptr) { SCREPER_STRUCT *scrptr; scrptr = (SCREPER_STRUCT*)vscrptr; if (scrptr->ScreenBuffer) free (scrptr->ScreenBuffer); if (scrptr->JavaScriptPage) free (scrptr->JavaScriptPage); if (scrptr->JavaScriptPause) free (scrptr->JavaScriptPause); if (scrptr->JavaScriptResize) free (scrptr->JavaScriptResize); if (scrptr->CspHeader) free (scrptr->CspHeader); if (scrptr->CspNonce) free (scrptr->CspNonce); fflush (scrptr->ScrOut); fclose (scrptr->ScrOut); free (scrptr); return (NULL); } /*****************************************************************************/ /* Parse the string of screper directives to set up the virtual terminal session. When the -dcl directive is encountered the rest of the string is considered and the terminal session is created and the DCL executed. Returns a pointer to a string which should begin with a DCL status value (e.g. %X00000001). */ char* ScreperDo (void *vscrptr, char *this) { int length, status; char *aptr, *cptr, *sptr, *zptr; SCREPER_STRUCT *scrptr; /*********/ /* begin */ /*********/ if (!strncmp (this, "-csp", 4)) return (ScreperCSP(vscrptr)); if (!strncmp (this, "-version", 5)) return (SoftwareId); scrptr = (SCREPER_STRUCT*)vscrptr; if (!scrptr) return ("%X00000014 hurrump"); if (!strncmp (this, "-css", 4)) return (PageCSS); if (!strncmp (this, "-javascript", 4)) { if (!scrptr->CspNonce) return (PageJavaScript); scrptr->JavaScriptPage = calloc (1, sizeof(PageJavaScript) + 64); sprintf (scrptr->JavaScriptPage, "