DMX Change Log
This is a running summary of changes made in the last couple of years to the DMX programs. These may, or may not, be reflected in the issue of the documentation that you have. Some of the changes may be highly technical, and may not necessarily have significance for the user. For the benefit of those browsing online the most recent changes are first.
Use of Regular Expressions in code conversion
A special version of DMXEFAX has been produced which allows the use of regular expression matching to perform code conversion. The presence of an initial open square bracket ( [ ) on a line in a code file signals that the Regex engine is to be invoked; this is a marker to switch to the comparison method and is not part of the Regex itself. In the absence of the initial [ the matching is performed using absolute or wildcard matching as in previous DMXEFAX versions. A simple example could be:
Since this is a very specialised requirement, and its use also requires installation of the matching PCRE.DLL file, a separate version of DMXEFAX has been generated which can be obtained from us, together with the DLL, if needed.
Update 2014-05-19: We have found a work-around for the problem which prevented us from loading PCRE dynamically. The standard DMXEFAX now supports regex code conversion; you only need PCRE.DLL to be installed if you want to use it.
Relaxed key comparisons
When the key fields in master and secondary records are compared, they are now stripped of leading and trailing blanks first and a case-insensitive match is made.
WINTASKS, WEBKGEN 2014-04-16
File timestamps & DST
Unfortunately the way the timestamp on a file is modified for DST (Summer Time) varies across the many operating systems that DMX can be run on. The IFOLDER command in Wintaks can fall foul of this. The only reliable universal way of handling this is to create a file, and then use its timestamp immediately as the current time for a DST compatible comparison. We have introduced this rather clumsy method into Wintasks, and also into the WEBKGEN utility used by agents.
!LEN Function available for !SET
The ability to use functions in a !SET directive has been introduced Currently there is only one.
will place the length of the string contained in %variable into %dest. You can only use it in a simple !SET. No code conversion. No concatenation. Not supported in any other statement. In the future we will be generalising the use of functions, and the range of them available.
Bugfix and import mailbox flags
A bug which meant that EDI extracted from mail body on import would not trigger a specified external task has been corrected.
The import mailbox definition grid has been enhanced with a further field flags. This allows you to overwrite the settings of the edi_in_body and content_check states for each mailbox polled. The field can for example contain body=Yes check=Yes to turn on the processing for that mailbox. Allowable settings are Yes True On No False Off. In the absence of a flags definition the .ini file settings are used.
To use this enhancement you will require a new dmxspop3.mdb database with the flags field defined in it.
We have introduced a new command line parameter /Z=Q which will set a 'quiet mode' in which no error message is generated if an undefined variable is encountered in a template, and the field will be output as an empty string rather than as the text UNDEFINED_VARIABLE. This is a convenience when working with boilerplate templates for which the .fxs for a particular implementation may be under-defined.
We have also corrected a bug in the handling of the ^b escape sequence. It will now only be transformed to "" if part of doublequote enclosed field. In other cases it will merely transform to " See CSV handling from 2013-10-29.
DMXEFAX DMXGWAY 2013-11-14
Although DMX has had the capability to support multiple independent users sharing the same system, but with independent views, for many years; the logging capability of DMXEFAX was introduced more recently. We have modified the system so that it is possible to run DMXEFAX as a defined user, and have the resultant logs imported into the Gateway database only for that user. To preserve backward compatibility the DMXEFAX /L command line parameter has been extended rather obscurely. Where you would have used /L you can now use /LUnnn. and where you would have used /LR you can use /LRnnn, where nnn is the 3-digit user number.
DMXGWAY has been modified so that when it is run in User mode (Nnnn command line parameter) it will only import DMXEFAX logs if they have been generated using the same user number.
Variable interpolation change
This is a small and rather technical change which will probably affect few users. The internal routine dotbuf(), which is used by the SAY, DISPLAY, and external command invocation logic to interpolate variables into the string provided, has been modified so that %!nnn variables do not have to be followed by one or more blanks. This means that, for example:
SET 0 "PicFile"
MYPROG %!0in.jpg %!0out,jpg
will now resolve to
MYPROG Picfilein.jpg PicFileout.jpg
Internal variables such as %!DATE still have to be blank terminated, since otherwise the parser will not recognise them. Other internal commands, which use resolved arguments, still expect internal variable designations to be blank terminated.
Wildcard code conversion
You can now use wildcards (* and ?) in the from fields of code files. If you specify an * in any section of a code file, that section will be marked as needing pattern matching rather than direct lookup. A section without any asterisks is processed as before, since this has a lower computational load. The matching is case-insensitive, unlike conventional code lookup. Two successive asterisks are not allowed in a matching pattern, and thus will not trigger the wildcard mode for the section. This enhancement was introduced for ecommerce transformations where several similar phrases need to be resolved to the same category code, and thus reduces the effort required to create the code file..
Simple JOIN utility
A utility which gives a simplified database Join capability using two key linked CSV files. It reads a master and secondary file and, if the designated ley fields in the two files match, appends designated fields from the secondary file record to the master file record and outputs it. If there is no key match the master record is output with a matching number of empty fields appended. Any doublequotes in key fields are stripped out before the comparison.
EZEJOIN /M=masterfile,keyfield /S=secondaryfile,keyfield /O=outputfile /T=transferfield,..... [/J] [/D=x]
fields are given as decimal numeric numbers, where 1 is the first field in a record. You can have any number of transfer fields comma separated in the specification, but they must be in ascending order (/T=1,9,13 is OK; /T=1,13,9 is not). Examination of the secondary file will stop after the first match unless the /J command line parameter is present, since normally there is only one secondary record per master record and this speeds up processing.
If, in a /T= specification, you specify a non-numeric character, then that character will be appended to the output record if a match is found. For example /T=3,5,@ would mean that any output record for which a match had been found would conclude with ,@. This can act as a flag to identify records for which a match has been found.
The default field delimiter is the comma. You can vary this by giving the /D=delimiter_character parameter. Caret escaping is recognised, so for a tab delimited file you would specify /D=^I
The CSV handling on input and output has been modified to match RFC 4180 more closely; particularly in the handling of double-quote enclosed fields which themselves contain double-quote or linefeed. Internally the characters are converted to their caret escaped equivalents.
A minor bug introduced as a consequence of long field logic on 2013-03-17, which prevented a Permute target from being being updated if it had previously been used, has been corrected.
Bugfix: EXPN Flag
The EXPN=N flag was not working for incoming Tradacoms messages. Since the only Tradacoms messages which do not use explicit nesting are Polaris insurance ones this has only just come to light during internal testing. We have, hopefully, corrected the logic.
DMXEFAX DMXPRINT 2013-10-03
Nested IF constructs
You can now nest IF/THEN/ELSE/ENDIF constructs. Previously this was not supported. You can follow the commands by a space and then a comment if you wish to keep track of what belongs to what!
By default DMXCONST did not remove empty fixed length fields from the generated EDI. Fixed length fields were rare in EDIFACT, and this behaviour undoubtedly had a reason, although after 20 years we cannot quite remember what it was. In some message standards fixed length fields are much more common, and so we have introduced the CFIX flag. If you set FLAG CFIX=Y then empty fixed length fields will be compressed away unless the No-Vanish flag is set in the table. By default, or with CFIX=N, DMXCONST behaves as it did previously.
MOVE changes, and more
The MOVE command has been modified so that a hash parameter after the command verb will cause it to select only matching folders:
MOVE # foldername older$$$
will rename 'foldername' to 'older' followed by three unique random characters. This mirrors the capability introduced for the FOR command recently.
The IF ~~ pattern matching operator is now case insensitive.
A bug in the IF command which failed to take account of blanks in quoted strings when positioning the pointer for the next command to be executed has been corrected.
Escape character processing
At the point of output DMXEFAX will now handle the caret escape character convention for conventional and delimited output. SDF output and @....@ constants in delimited output are not processed. This is a slightly controversial change introduced to make it easier to use CSS in templates which generate HTML. We shall monitor it closely and, if necessary, introduce a command line switch to toggle the behaviour.
FOR and IF changes
You can now nest FOR and IF/THEN/ENDIF statements. FOR can be nested up to a depth of ten levels. IF/THEN/ELSE/ENDIF can have any level of nesting, but the logic is in terms of their physical order in the task, and could be confused by GOTO or GOSUB which terminate an IF/THEN construct elsewhere. FOR has always worked on files only. If instead of FOR whatever you now put FOR # whatever then only subdirectories will match (note that even an empty folder always has the two subdirectories "." and ".."). Some internal cleanup has also been done, including removing a bug in the CD command which could cause script lines to 'vanish' if the CD was in a subroutine.
Changes for 64-bit systems
DMX programs are 32-bit. They access the registry. Windows has a special registry node for 32-bt programs. It is supposed to handle the use of this node seamlessly. There can, however, be glitches. We have modified the security module used by all DMX programs in order to bypass these glitches when establishing the machine code to be used for enabling key validation. Substituting the new D32ALLOW for the old one on a 64-bit system where the modules are already registered may, or may not, require you to obtain and apply new enabling keys.
Mail Menu enhancement
You now have a chance to cancel a Mail request made inadvertently, and if you have failed to select a document you will now be informed and no further action will be taken.
DMX WebView2 2013-08-01
XML files now displayed correctly
The cgi will now examine a file before it is displayed, and if it contains XML it will be sent with a text/xml header to enable modern browsers to display it satisfactorily.
PARSE destination source separator count
This is used to extract individual fields from a delimited string. The source is the string to be analysed, the separator is the delimiter character (typically a comma), count is the field number to extract (1 will pick up the first field), and destination is the string buffer position into which the extracted string is to be stored. For example:
set 0 123,"234,xxx",345,456
parse 100 %!0 , 2
say 20 %!100
will display "234,xxx". Had we specified a count of 4 it would have displayed 456 .
Escape characters in /U= definition
If you wish to specify odd characters then the caret escape convention is now honoured. ^^ is interpreted as a single ^, otherwise decimal 64 is subtracted from the character following the caret. Escaped characters are physically two characters but they only 'count' as one. For example to process a tab delimited file you could use /U=#^I.### .
Increased variable pool size
We have increased the size of the variable buffer pool from 2048 to 16000 characters to cope with operations on massive fields.
PREVIEW command bugfix
The PREVIEW command was unable to 'find' a section whose level tag was defined using advanced matching (this can happen for XML). This has been corrected. This was a very obscure bug, which is why it has taken ten years to identify.
Utility to identify files by content pattern matching
This is a utility which reads the first part of a file, compares it to a wildcarded match string, and returns True (1) or False (0) to indicate whether there has been a match. It is surprisingly useful. The command line is:
PNMATCH file_to_examine "pattern match string" [seek_depth]
The wildcard characters you can use in the match string are the conventional * to match zero or any number of characters, and ? to match any one character. The string has an escape character, the caret (^), such that if you precede any character by a caret then it will be transformed to the character which has an ASCII value decimal 64 less. This means, for example, that ^J is a newline, ^b is a doublequote, ^M is a carriage return, ^I is a tab character. There are three exceptions to this rule; ^^ will be interpreted as a single caret character, and ^* and ^? cause matching on asterisk and question mark as data characters which are not considered to be wildcards. It is documented in UTILITY.RTF
Utility to merge command line data into a report
This is a utility which allows you to merge data from its command line into an HTML template (or other files) to generate an output page for web display or as the body/attachment for an email. The command line is
htmblend template_file output_file param0 param1 param2 ......... param9
The template contains HTML comments of the form <!--$$N--> where N is 0 through 9. Whenever such a comment is encountered it is removed from the output but the value of paramN is substituted in the output. Remember that any paramN which contains spaces should be enclosed in quotes e.g. "Format Error". If you specify a $$N for which there is no corresponding paramN then the word MISSING will be used. It is documented in UTILITY.RTF.
When processing incoming XML DMXEFAX can now extract from CDATA tags. In the fxs you define somrthing like:
%mytag "![CDATA[*'" 0:1 A ;extract data from a CDATA tag
The 0:1, rather than the 0:0 usual for XML, indicates that the data is to be extracted from the tag itself. No escape processing is applied to the data.. You will probably need to use offset indication in the TPL since there may well be more than one CDATA occurrence at different parts of the XML hierarchy e.g. %mytag<anothertag.
Long data fields re-visited
A month ago we posted about changes to DMXEFAX to cope with data fields up to 4kb in length. It seems we were living in the past; this is an era of Big Data, and now fields of 500kb and greater can be handled. A new command line switch /B=nnn has been introduced, where nnn is the maximum field size in kb, for fields longer than 4kb. When this switch is used you do not have to worry about padding template lines to cope with long field interpolation.
Update: On 2013-03-17 we changed the way global variables were handled so that long fields no longer used the internal memory pool but were allocated independent memory regions. Again this was to support the use of very long data fields.
DMX Generic 2013-02-12
Many DMX programs feature the ability to compare a field against a wildcard pattern (this does not apply to Advanced Matching capabilities, which use a special 'EDI Aware' algorithm). The subroutine used to implement this is being changed to a non-recursive form which is considerably faster. The new implementation will not handle multiple successive asterisks in the pattern but, since these were purposeless anyway, this should not have any practical consequences. As before, matching is case-sensitive. The change will be applied when affected programs are re-compiled for other reasons
Additional argument options for external task
In addition to the %F parameter on the arguments field in the mailbox grid, a number of other %x parameters are now available to pass header information from the email to the called program. Some internal changes have been made to allow connection for secure mail through the Stunnel proxy, and to use case-insensitive matching when checking for allowable strings.
Long data fields
In the good old days DMXEFAX was never expected to encounter a data field longer than 70 characters, and so it had 'generous' allocation of internal buffers to cover longer fields 'just in case'. When used to process delimited files it can potentially encounter fields much, much larger. It has been modified so that it should be OK for fields up to 4k. There is a directive, buried deep in the documentation, telling you that if you use a %nnn on a template line that can represent a large variable then the line should be blank extended until it can accommodate the variable, and ended with a non-blank character to prevent blank stripping when the template is read. This was generally ignored with impunity, but for a data field of over 200 characters you HAVE to do it! We also brought DMXEFAX into line with the other re-formatters to handle delimiters in quoted strings correctly; we must have forgotten it fifteen years ago!
Select trailing variable characters
The reference %!-4R100 will return the final four characters of the string variable stored at location 100. The general expression is %!-nRy where n is the number of characters at the right of the string to select, and y is the offset into the variable storage area. The 'R' can actually be any non-numeric character.
DMX Enabling Keys 2012-12-31
Possible problem establishing Machine Code
DMX programs use a security module in which a 'machine code' is established for the PC they run on, and then checks to see if enabling keys have been registered which match the program and the machine code. Amongst other things this requires reading some registry values from HKLM\Software\Microsoft\Windows NT\CurrentVersion. If the account under which DMX is run does not have permissions to read these values then the machine code may be indeterminate, and appear to vary. This means that the programs cannot have stable enabling keys. This is an unusual problem, but if you have problems applying working enabling keys then you should check using Regedit that the DMX account has the necessary Read permission.
SIZE command introduced
The command "SIZE n filename" will now place the size of the named file into the string buffer at offset n
GROSS_ERROR flag changed
This is a very minor alteration. Previously when the Gateway processed a converter log which contained no message information it created an entry in the traffic view with the UDID defined as GROSS_ERROR. This is rather too dramatic, so such entries are now defined as NO_CONTENT, which is a more accurate description.
DMXEFAX can be set with filters, so that only EDI which matches its templates will be processed. If there was no match in a file then when the log was imported by the Gateway it would be flagged as a GROSS_ERROR, since there were no messages logged. This could be unsightly when multiple passes were made over the same file to extract different possible message types. The logging has been altered so that no record is kept in the log if no matching messages are found. When used for non-EDI files (using the /X or /U modes) the log is not suppressed, since the data in it is under template script control.
Weird apparent 'bug'
We recently had a very strange 'bug'. Wintasks refused to run any external process if the original task had been specified with command line arguments (which is unusual); the CreateProcess call failed with a memory access violation. After many hours of investigation it turned out that the behaviour was almost certainly caused by a DLL which had been injected by an external process. We looked at the usual suspects (ZoneAlarm and Avast antivirus) and it was caused by Avast. Setting the particular copy of Wintasks as a trusted program in the Avast behaviour shield cured the problem. Not really a DMX issue, but worth recording. We only switched to Avast from AVG recently, so we had not seen this previously.
MTOK flag bug fixed
Setting the flag MTOK=N (default =Y) caused non-zero input files to be moved to Trashcan after processing, since the file length for input files was not set and defaulted to zero. This has been corrected. This bug had been around for many years, since in our installations we never set MTOK=N because even an empty file can carry information in a Gateway environment (name, location and timestamp).
WEB KEY GENERATOR 2012-10-08
Bug in .reg file name fixed
If agents used Customer Names which contained special characters an anomaly in Windows stopped the stored .reg file being accessed from the key generator. The cgi for the Key Generator has been modified to create acceptable names when the file is stored locally at EDIMatrix.
DMX WebView 2012-08-15
DMXEFAX DMXPRINT 2012-06-04
Interactive interface change
The interactive interface for DMXPRINT and DMXEFAX does not have input fields for some of the more specialised command line parameters, such as /U= and /X=. The operational kludge was to give an output filename followed by a blank and then the parameter. This meant that output could not be to screen. Now if the 'output filename' is given as a single exclamation mark then no output file wil be opened, but the following parameter is still accepted.
DMXCONST DMXTRANS 2012-05-30
Data logging for X12
X12 segments may be only two characters long. When elements from such segments are logged the log now contains an underscore instead of a null for the third character, thus facilitating its identification as a UDID by DMXGWAY. If B1:0145 is logged then the Gateway should search for B1_0145.
Postbox List spurious logs
The change made on 2011-09-22 stopped DMXTRADA from recording the spurious logs, or trying to reconcile them into the traffic database, but until very recently they were still being sent, and thus creating traffic volume charges. To alert users if this is occurring we now post an entry in the network log saying how many spurious logs have been ignored.
A bug which caused quoted strings containing blanks to be incorrectly parsed by the JOIN command has, hopefully, been fixed
More Grid Views
The Database Functions panel has been enhanced with buttons to give grid views of the Messages, Interchanges, Files and Admin database tables. Changes made in these grid views are not validated, and if saved may make the Gateway unusable. Use with extreme caution.
Bugfix: Archiving problem on 64-bit systems
DMXGWAY used to run the LHA archiving program using an ancient PIF file from Win16 days! It turned out it referenced COMMAND.COM, which cannot run on 64-bit Windows. We have altered the Gateway to use CMD.EXE directly.
LCD implemented as parsed command
Until now local directory change (LCD) was handled as an external command, and was performed by code in the FTP client. In the old DMXIFTP the client code was part of the network interface, and the interface 'knew' about the new local folder. In DMXIFTPS the FTP client is external, and whilst the client changed its local folder, the interface did not. This made no difference until you used an unmirror command. We have now introduced "lcd,pathname" which changes directory in the interface and then sends an equivalent external command to the FTP client. LCD without a following comma is still a valid external command.
The Gateway was originally designed so that its main display would fit onto a VGA monitor. Nowadays this is a bit restrictive, so we have daringly altered the size of the form so that it will fit on an SVGA monitor, and used the extra screen real estate to give a larger traffic view window.
Based on suspicions that some Windows versions under some circumstances could object to a filename ending with a period, we now create destination filenames ending in ".ok" or ".bad" when moving files. If a file move fails the error code is now logged. The seeding of the random number generator was also rationalised to avoid risk of using the same seed if the converters are run at very short intervals.
DMXPCOMP was also changed slightly to avoid an internal error if there were more than a thousand partners - a situation which has not yet occurred to our knowledge - the most we have heard of is around 650.
Auxiliary file linking
Since the Gateway can legitimately create multiple file table entries for a single file in some circumstances, the linking of an auxiliary file to a destination file now searches for all instances of the destination file name, rather than just the first.
Profile read-ahead buffer increased
DMXTRANS now pre-reads 512 rather than 256 characters into an interchange/message when performing an advanced matching multiple segment search. This enables you to drill deeper for difficult matches.
Trading Grid compatability
We can now provide a full interface to GXS Trading Grid when accessed using conventional FTP over a VPN. If the SSH SFTP sccess method is used then, whilst files can be sent and received successfully, we currently can find no mechanism for feeding back sent file status information to the Gateway database. This limitation is imposed by the access method.
DMX WebView 2008-05-03
The TinyWeb server (TINY.EXE) from RitLabs is now available in the external folder of installation CD's to allow simple enabling of the Web Traffic Viewing capability. The cgi programs have been altered very slightly to be compatible with this server. An installation text is available as Tinyweb.txt.
Import bug corrected
A bug which caused the reading of the list of mailboxes to terminate if one with no mail available was encountered has been corrected.
DMXEFAX DMXPRINT 2008-04-18
Nested !CALL directives
!CALL directives can now be nested without losing the memory of the original calling section. Bug in !PERMUTE $$$ handling in DMXEFAX corrected.
Menu Item to mail files to EDIMatrix
The menu item Maintain - Mail has been added. Providing the Mailer has been installed this will create an archive of the files associated with the currently selected document and email it to firstname.lastname@example.org.
New network interface for secure FTP
This is a new network interface which supports both conventional and secure (SSL/TLS) FTP. It is functionally very similar to the original DMXIFTP interface, but certain script commands have changed syntax. It uses the MOVEitFreely FTPS client from Standard Networks, which will now be part of a DMX distribution.
Zones for export scripts
Export scripts can now contain zone identifiers to enable excecution of different sections of the script for different types of export file to the saame partner. A zone identifier is an exclamation mark followed by one or more characters which must match the initial characters of the export filename for the script commands following it to be executed.
Enhanced scope for fstatus FTP script command
The fstatus script command can now be used to set a file status which will be recorded in the Gateway database even if the script subsequently fails. You can now stop a file from reverting to NEW status if the error occurred after the main file PUT stage. In the absence of any fstatus commands the file status behaviour of DMXIFTP is as before (Mark as submitted if script is error-free, New if it is not).
Pattern matching operator
Wintasks now implements the "~~" pattern matching operator introduced into DMXEFAX/PRINT on 2007-07-05
XML attribute handling
By setting a non-zero subelement counter in the .fxs specification you can now pick up values of element attributes as well as the actual element value.
/NOX command line flag
If an initial /NOX command line parameter is present the requirement for an initial XML declaration is suppressed. If one is not found he first thing that looks like a tag is used as document start.
Bugfixes - Advanced Matching, !SET Code Conversion
Advanced mathing now works correctly in @PARTNER and @MESSAGE lines. !SET code conversion now works in DMXEFAX (and is now documented for both DMXEFAX and DMXPRINT),
No paging option for traffic list view
By default the Gateway will only list 800 documents at a time, with a paging option to see more. In the Options Menu we have added a checkable 'No Page' item. If this is checked then all available documents will appear in the traffic list view. The default is unchecked, which corresponds to the previous Gateway behaviour.
DMXEFAX DMXPRINT 2007-07-05
Pattern matching operator
The comparison operators in the !IF statement have been extended to include a double tilde (~~) operator. This is used with a %nnn to the left of the operator and a quoted string to the right. The quoted string can contain the normal wildcard characters '?' and '*'. If the variable matches the string then the condition is true. For example !IF %invno~~"*X*" will be true if ther is an 'X' anywhere in %invno
If you wish to clear the message status markers (the + or - in the gateway main view) you can place a line in FILERROR.UPD of the form:
where 'filename' is the name of the destination file they appear in. The file status of 'filename' is not altered.
DMXEFAX DMXPRINT 2006-08-27
STREAM keyword for !XMLMODE
If the keyword STREAM appears on the !XMLMODE line then output will be generated without linefeeds, and leading blanks will be atripped from output lines.
Multiple mailbox support
The DMX internet mail interface now supports import from multiple mailboxes, with mail handling options specifiable for each mailbox. The features are implemented using the data grid approach common to many of the other network interfaces.
Tradanet script change
TRADANET.SCR now allows dial up PPP connection to AT&T to allow TCP/IP connection for those not using the VPN software
XML input files CFJ07
DMXEFAX can now process XML input files. A command line parameter of format /X=<>,.## will turn on this behaviour. The approach used involves advanced matching on tags and is described in the DMXEFAX documentation.
We now support the use of the $ wildcard character in extraction filemasks. This reduces the risk that a file downloaded in a previous session, but not processed, be overwritten by fresh data. Each occurrence of $ will be replaced by a random digit when the fileid is submitted to IBM IE.
When using Expedite Base for Windows 4.7 the normal way to communicate with IBM IE is over the Internet. This no longer requires the SM password management facilities present in DMXIBMIE. It does require that the file BASEIN.PRO be present for each connection. This file was previously used for dynamic changes and deleted after each session. When using iebase 4.7 you will be issued with an EXPEDITE.TSK file which copies BASEIN.RAW to BASEIN.PRO before iebase is invoked. Your original BASEIN.PRO should therefore be copied to BASEIN.RAW before you first use DMXIBMIE.
XML closure tag CFJ06
In XMLMODE where the data field is absent, the XML convention of expressing <tag></tag> as <tag/> is now applied. This takes effect both for template tags and AUTOTAG tags. When using template tags this assumes that the opening and closing tags are on the same template line.
Bugfix: A bug very similar to that corrected in DMXXNET2 on 2005-06-20 has been identified and removed. It could cause interchanges in a mixed EDI standard file to be skipped.
CSV file handling, Message skipping
Bugfix: When used to analyse CSV files (/U= mode) DMXEFAX was failing to recognise field delimiters in a quote-enclosed data field as being part of the data. This has been corrected. An entirely unrelated bug was also corrected; when in message skip mode the mode was not being reset when a new interchange was detected, leading to potential loss of envelope information.
Tradanet Script 2005-06-27
This is a very technical change. The script TRADANET.SCR has been modified to use MODE 8 (substitute for incoming NULL characters) except when receiving a binary file (revert to MODE 9 for the duration of the receive). This follows the discovery that the TIP File Header can contain an illegal NULL, which was never a problem until very fast connection over a VPN combined with a short received file caused the NULL to act as a stop character for the normal WAITFOR trigger string handling.
Short Interchange problem 2005-06-20
Bugfix: A bug has been identified which could affect the processing of short interchanges (<2000 bytes) in a file which contained a mixture of different EDI standards. If the forward search logic intended to pass over non-EDI data encountered two interchanges in its buffer it chose one by a priority (Tradacoms > EDIFACT > X12) rather than taking the first one which occurred. This could lead to the short initial interchange being treated as noise. The program has now been altered to detect the first interchange present in the buffer.
Installation Package changed
The DMX installation is now distributed as a single executable rather than several 1.44Mb files. Few people now use diskettes!
CSVF/ignore string mismatch fixed *137
Bugfix: Previously if ignore strings were used actively with a delimited inhouse file then warnings about missing tags would be generated. This was an uncommon situation and there was a work-around. Recognition of tagged records to ignore has now been introduced at the level of delimited to SDF record resolution.
Icon in SysTray for DOZE mode *136
If the converters are run in DOZE mode they will be minimised to the system tray. Right clicking on the icon there will toggle the normal taskbar presence.
Icon in SysTray
DMXALARM now places an icon on the SysTray by default instead of appearing minimised on the taskbar. The taskbar presence can be toggled by right-clicking on the SysTray icon. The Tooltip text can be toggled by left-clicking on the SysTray icon. The tooltip text corresponds to the taskbar legend in earlier versions. This change is purely cosmetic; there is no change to the programs functionality.
Empty file deletion
BUGFIX: A situation in which empty files were not deleted when the flag MTOK=N had been set has been corrected.
DMXGWAY/Network Interfaces 2004-10-06
Problem with Win XP Service Pack 2
Application of SP2 broke the logical linkage amongst the .dlls which support database access, thus rendering the Gateway and Network Interfaces unusable. A quick fix is to move the affected executables from dmx\bin to dmx, and ensure that the required VB5 runtime modules are installed in the Windows system32 folder. We have a separate installation available for the VB5 runtimes required.
DMX WebView Released 2004-10-01
Browser access to EDI Gateway database
A CGI interface to allow web browser access to traffic information in the DMXGWAY database. The facilities are similar to the Gateway 'Criteria Select' form. Requires that you have a web server either on, or with network access to, the EDI PC. Documented in DMXWEBVU.RTF.
DMX Online Key Generator 2004-09-07
Browser access to Enabling Keys
Agents and OEMs can now link to a URL on our technical support web server to generate their own .reg files to register the client installations they make. Access to this facility requires a password.
Advanced Matching Profile bug
A bug introduced in Nov. 2002 has been identified and corrected. It stopped advanced matching in a profile at message level from searching beyond the UNH/MHD/ST segment.
Socket Error handling MXC029
When using TCP/IP if DMXCOMMS now encounters persistent winsock problems during data transmission (100 or more successive attempts fail) it will simulate a carrier loss condition. TCP/IP scripts may need modification to take advantage of this. Winsock problems are now logged, and a possible overrun problem removed. Previously DMXCOMMS did not attempt to monitor the status of an internet connection which it had not established itself.
Datagram sent when Registration screen appears
The security module D32allow will now send a short datagram to EDIMatrix when the registration 'nag' screen appears. This allows us to have your enabling keys ready for you before you request them! The datagram contains no information which is not visible on the registration form (which is concerned entirely with the DMX installation) and does not compromise the security of your system in any way. We use UDP protocol on port 3048. If the EDI machine has no internet connectivity, or your firewall blocks the outgoing message, no harm is done.
Minor enhancement for partner names
The Seqnum utility now supports an additional column in which you can put a name to identify the trading partner for a .sqn file. The names are stored in SEQTITLE.TXT in the DMX folder as lines of the form nnnnnn.sqn=nickname.
The BASP21 object is now deleted and re-created for each FTP connection. This gives greater reliability when a session requires multiple connections.
Tradacoms File Generation Number
Traditionally we have mirrored the Interchange Reference number as the Tradacoms File Generation Number. This simplifies Gateway tracking, since we track by ICRF, not FLGN. Unfortunately when the ICRF exceeded 9999 this gave problems, since FLGN is only 4 digits. For Tradacoms interchanges only we have made two modifications to behaviour. If the field length of any data element is exceeded then it will be left truncated if numeric and right truncated if alphanumeric. In addition, if a General Sequence Number (table *SQn!) is used to maintain a data element then, if the data element is called FLGN, the sequence number will flip over to 1 if it was previously 9999. If you continue to mirror ICRF for FLGN (table *CA3) then 9999 will increment to 0, which some partners do not accept as a File Generation Number.
DMXEFAX DMXPRINT 2004-01-23
This command, which should be used as the first command of the first executed section (i.e. ####1, or the section called by a !INIT) will change the way DMXEFAX interpolates data into the template. Conventionally the template is assumed to represent a printed report, and fixed text strings in it will appear in the output starting at the same column as in the template. This means data can overwrite fixed text. If %data was 12345678 then <tag>%data</tag> would be written as <tag>12345678ag> . When !XMLMODE is used the output appears as <tag>12345678</tag> . This command also causes escaping of the XML syntax characters so that if %data was Marks&Spencer the out put would appear as <tag>Marks&Spencer</tag> .
If the parameter AUTOTAG is given, then if you terminate the %nnn tag with a greater than (>) sign it will be automatically wrapped by XML tags. In the example above, in AUTOTAG mode, %data> in the template will appear as <data>Marks&Spencer</data> . This requires that the %nnn names that you use for this feature must be the XML tag names. You can mix tags which you specify yourself and automatically generated tags by the presence and absence of the terminating > character. If you wanted to put in attributes you could write <data id="supplier">%data</data> to generate
<data id="supplier">Marks&Spencer</data> .
This command is ignored if you are in CD or SDF mode. DMXREFRM will also soon be enhanced with this command.
Irrespective of the expiry date in the D32ALLOW module, unregistered DMX programs will run for a period after initial DMX installation without generating a nag screen. This 'period of grace' has now been increased to two months. This is primarily intended for clients who re-install the system on a new PC. During the final fortnight of this period a reminder box is displayed for the duration of the program run, but no operator acknowledgement is required (since DMX programs are fast you may not be aware of the reminder for many of them). Thereafter the normal nag screen, which does require operator intervention, will appear.
File Numbers & Bookmarks
The Gateway assigns a file number to each file it keeps track of. Previously this number was reset to 1 when it exceeded 99,990 (thus assuming that a user would archive before he had a hundred thousand files stored locally in the DMX subdirectories. This has worked fine for normal usage, but it gave problems if a high-volume user was using the Bookmark function to generate reports, because the presence of earlier files with a higher file number upset the simple bookmark logic. We have now increased the reset value to 9,999,990. This should mean many years of operation before Bookmarking could potentially work anomanously. If it occurs the cure is to archive the whole database, which means that new entries always will have a higher file number than existing ones. Bookmarking is so rarely used that the problem has not been noticed before.
The last vestiges of the old yyyy.mm.dd date format have now been changed. They were lurking in little-used functions and causing sorting problems.
DMXGWAY/Network Interfaces 2004-01-02
Screen Saver Problem
DMXGWAY and the Network Interfaces are written in Visual Basic. Unfortunately this can give problems under some operating system configurations if the programs are run in background mode as part of a scheduled task. When a screen saver (or lock out) is active the VB programs cannot complete their loading process. This is a characteristic of the language, and we cannot program around it. If you MUST run on a locked server or workstation then consider running the scheduler as a service; this seems to avoid the problem. Note: on 2004-09-24 the utility KILLSSVR was introduced (documented in Utility.rtf) which can often turn off a screen saver programmatically,
Large Compiled Profiles *135
The compiled profile files generated by DMXPCOMP now use 32-bit integers internally for their linkage pointers, thus allowing a much larger profile. The source profile is now pre-read by DMXPCOMP to allocate its internal arrays dynamically. There are thus no longer any restrictions on numbers of partners and messages, or profile size. DMXCONST and DMXTRANS have been modified to expect the new .CPR format, and will reject any not generated in the new format.
This corresponds to version 1.9 of the converters and version 2 of DMXPCOMP. You must install all 3 of the new modules if you want any one of them. Old versions of Const and Trans will crash if used with the new compiled profile. These changes are still in Beta mode.
** OLD PROFILES MUST BE RE-COMPILED WHEN THIS UPDATE IS MADE **
The Extracts type field has an added purpose. If the first two characters in it are !x (where 'x' is a character of your choice) then these will be removed for the traditional evaluation of types B and ?, but they will specify an extraction class. If on the command line for DMXTRADA you have placed a !xyz parameter, then any extract tagged as !x will only be performed if x is present in !xyz. Extracts not tagged as !xyz will always be performed, as will all extracts unless the command line has a !xyz on it. This facility requires that a new compatible version of TRADA32X.MDB is installed, or that the length of the type field in the extracts table is increased to at least 3.
The normal collection of postbox and mailbox information from Tradanet (this occurs automatically; you do not need to specify any extraction for them) can be suppressed if the files POSTBOX.NOT and MAILBOX.NOT exist in the DMX home directory. Users with high traffic volume and frequent Tradanet connections may wish to set and clear the MAILBOX.NOT semaphore file by external logic to reduce the traffic volume, and consequent network charges, associated with fetching this list at every connection.
If the .INI file parameter del_after_days is set to -1 then all files in your mailbox which have been previously fetched will now be deleted at the end of the tradanet session. We do not recommend this under normal circumstances. This may have worked with previous versions of DMXTRADA; we just never dared try it!
TCP/IP support in Tradanet script
The standard script for Tradanet connection has been modified to allow connection over TCP/IP, following successful testing of this facility using the Securemote VPN support provided by GXS.
Explicit Nesting for Tradacoms
If an EDI message is using the TDI syntax then explicit nesting is automatically enabled. There is no longer any need to flag table entry points or use the EXPN flag.
Several years ago the script processor used by DMXCONST and DMXTRANS had a minor change made to the way the script pointer was advanced. We have just discovered that three commands had not been modified to reflect this change. This has been corrected. The commands were obscure and the delay reflects how seldom they have been used!
An error was made copying the code supporting the PREVIEW command from DMXEFAX to DMXPRINT. This has been corrected, and PREVIEW now works as documented in DMXPRINT.
Under very specialised circumstances encountering a segment with invalid syntax characters could cause the converter to loop processing the same bad segment. We have applied a temporary fix which cures the problem, but might have other unforeseen consequences under other specialised circumstances (although we think this unlikely). DMXTRANS will be monitored carefully in the near and medium-term future.
We recently, rashly, changed the date format used within the Gateway database from CCYY.MM.DD to the ISO CCYY-MM-DD after complaints from clients when trying to import reports into a database. This has led to problems with the selection of views by date, particularly in a database with a mixture of old and new date forms. The Other Functions panel now has an extra temporary button 'Date Format'. Clicking this will convert all dates in the message table to the new format.
IF operations on absent files
Previously the IFOLDER and IFLARGER commands could give an unpredictable result if the file tested did not exist. Now the result is always FALSE (even if the ! negation prefix is used) if a non-existent file is tested.
Externally defined files archived.
During archiving DMXGWAY will now search for the file in the DMX area called ARCHIVE.ADD. If found it will add its contents to the list of files to be archived, and then delete ARCHIVE.ADD.
This behaviour is intended to aid those who generate a number of supplementary files (reports etc.) which are not known to the Gateway. If the EDI task contains lines of type:
SAY ARCHIVE.ADD reports\RPT12345.txt
then RPT12345.txt in the folder C:\DMX\REPORTS will be archived (and deleted) when next archiving is run. The filenames should all be relative to the DMX area. You may wish to elaborate on this approach to introduce a delay before the files will be archived.
This is highly technical. DMXTRADA will now still warn when the 'owner' of the file, as marked in the database, does not match the recipient in the EDI, but will no longer flag the file as invalid. This is to deal with an unusual situation in which the recipient is a clearing house, but the true recipient is marked somewhere else in the message. If you want the Gateway to track the true recipient, then the construction table must be binary edited to clear the '5' action flag from the first S003/UNTO data element and insert it on the desired data element in the UNB/STX (it is the final byte in the data element array entry). The action flags are set programmatically by DMXTCOMP, and hence the modification must be re-applied whenever the table has been re-compiled.
MODE modifiers for FSEND
A file sent using the FSEND command is by default opened in text mode, which converts CR/LF pairs to LF and treats <Ctl>Z as an EOF marker. Where this behaviour is undesirable MODE 18 will cause subsequent FSEND commands to read the file in binary mode (no conversions). MODE 17 will revert subsequent FSEND's to the default text mode reading.
DMXPRINT DMXEFAX 2003-04-24
A command which allows you to look ahead in the message hierarchy to use, and store as globals, data from an input section which has not yet been processed. The section number given is located, and its data appended to that from the current section. The subsequent normal handling of the previewed section is not affected. This is very useful when trailer information needs to be included in the output header.
Internet Dial-Up and Disconnect
In preparation for the close-down of the IBM IE async access method, DMXIBMIE now has 2 extra buttons to enable Internet Dial-Up and Disconnect. They are invoked by the command line letters D and H .
DMXCONST DMXTRANS 2003-02-20
Profile behaviour; ICRF
A bug in DMXCONST has been fixed so that ICRF data can be read from control file DATA lines.
DMXTRANS no longer needs a script to mark buffer 15 as clean when processing the envelope segment under a profile; this will occur automatically. If the TRIG MSG tag starts with an exclamation mark no message trigger information will be written to inhouse.
Network Interfaces 2003-01-27
The network interfaces now create a semaphore file of form Interface_Name.XXX if any errors were detected during the session. The file is cleared when the interface starts up. It contains the error messages. All error messages have !! as a prefix and warnings have !. A separate log of the session (as LASTxxxx.LOG) is kept as well as the updating of network.log.
DMXPRINT DMXEFAX DMXREFRM 2002-11-30
%INFILE %OUTFILE Functions
Two new internal symbols have been added. For DMXPRINT and DMXEFAX %INFILE allows you to reference the input file declared by the command line /I parameter, and %OUTFILE references the output filename declared by the /O or /A parameter. %OUTFILE has the same effect in DMXREFRM, but %INFILE will return the name of the input file opened for SECTION 1.
A new command has been added to the script language, intended primarily for import scripts.
This will resolve the filemask into filenames, and then send an explicit deletion request for a file of that name in the current working directory on the server. Used to delete explicitly files you have successfully received, typically into an empty local directory. e.g.
!SHELL directive, enhanced !SET
A new directive SHELL has been added. It has form:
!SHELL n %command_line
n is a digit which indicates the window type (e.g. 5 for Normal or 7 for Minimised, No-focus). If it is preceded by a minus sign the external process is run synchronously (no wait for completion) otherwise DMXREFRM will not continue till it has completed. The second argument is a variable which has been set up to contain the string which you might enter in, for example, the Start->Run program field.
Source variables in the !SET command can now use the !n suffix to apply code conversion.
DMXCONST DMXTCOMP 2002-10-25
X12 modifications DMX133
The control file flag X12S has been modified to use caret as a release character so that control codes can be specified as syntax characters. The default X12 separator characters have been changed so that newline is used as a segment delimiter (this makes for readability but we do not recommend it for use in Europe) There has been some slight trimming of X12 construction. The DMXSEF and MAKTAB32 utilities have been changed slightly to support X12 SEF's (which meant a slight change to the X12 action element designators in DMXTCOMP).
A completely unrelated bug in CSVF processing which could cause the final field in the inhouse record to be ignored has been, hopefully, eliminated.
Trim no more than 7/8ths
DMXLTRIM will now always leave at least the last eigth of a trimmed file.
DMXREFRM DMXPRINT DMXEFAX 2002-10-13
DMXREFRM now has the PERMUTE command as implemented in DMXPRINT/EFAX
The PERMUTE command has been augmented for alphabetic month conversion. If a dollar character is used in the From=To mask then it will be assumed to mask a month, and will be matched from alpha to numeric and vice-versa. For example, if %date is "13-Oct-2002" then
!PERMUTE %date "Dd-$$$-CcYy=CcYy$$Dd" %newdate
will make %newdate contain "20021013"
!PERMUTE %newdate " CcYy$$Dd=Dd-$$$-CcYy" %date3
will leave "13-Oct-2002" in %date3. The numeric side should always be represented by 2 dollar characters, the alpha may have as many as you need to represent the month. If we had used
!PERMUTE %newdate " CcYy$$Dd=Dd-$$$$$$$-CcYy" %date3
then %date3 would be "13-October-2002". If the month is shorter than the number of $ characters then it is blank padded to the required length.
MULTIPLY & DIVIDE
Two new maths commands. Usage the same as ADD and SUBTRACT.
DMXPRINT DMXPLCC 2002-09-20
Mandatory field checking
If the /V command line parameter is used then any %nnn variable used in an interpolation record will generate a warning if it is marked MM in the .PLS, but is absent or blank in the incoming data.
Field overrun limiting
When reading from delimited files the CDL to SDF conversion now removes any extra characters in a field if the defined length is exceeded. This prevents 'bleeding' of data into following fields, which was not always overwritten by the following field content if the length was less than the maximum defined.
The previous limits for number of partners and messages were 100 & 200 respectively. Both are now 300, and warnings are issued if these numbers are exceeded. The size of the profile data buffer has been increased to 32kb. This latter is an upper limit if backwards compatibility in the converters is to be retained, whereas we can increase the 300 limit readily if required.
Rewind and sequence numbers DMX132
Previously when an interchange was rewound (aborted) the interchange reference number was not incremented. When using .SQN files to keep the ICRF this meant that the next valid interchange constructed took the next ICRF in sequence, and there were no gaps. However, no attempt was made to reset other sequence numbers (*SQn in table). This behaviour has now been changed. If an interchange is aborted the .SQN contents is reset to the values before the processing of the failed interchange. This change should simplify handling of Tradacoms interchanges where a *SQn is used to create the file generation number.
The directive !DELAY nnn will cause DMXPRINT to sleep for nnn tenths of a second. It was put in as a quick fix for a client who was generating multiple output files via !OUTFILE and needed the file timestamps to be at least a second apart!
Budget .SQN file Editor
Allows the user to modify the binary .SQN sequence number files. An equivalent to SEQNUM for Budget systems. See the Utility.rtf document.
DMXIFTP, DMXSPOP3 2002-06-21
Previously the Connect and Hangup functions were implemented using a third-party freeware component RASDIAL.OCX. This has been shown to give registration problems on Win2k. The functions are now implemented using our own D32RAS.DLL, which will be distributed as part of a standard installation. It does not require registration!
Import of PRINT sessions into Gateway
DMXPRINT, like DMXEFAX, is now capable of generating a converter compatible log if the /L command line parameter is specified. This DMXEFAX.LOG can be analysed by DMXERRAN, and will be imported into the Gateway during Update to generate incoming traffic information. A new directive !LOG in the template can be used to generate Log information for Gateway display. Unlike EFAX, the template must be explicitly set up using !LOG to generate logs for trading partner, 'interchange', and 'message', since there is no standard covering the files which PRINT processes. The Gateway will interpret the log as if it was incoming EDI! This capability has been added to allow atypical formats such as BISAC and MFG/PRO to generate meaningful entries in the Gateway.
Asynchronous communications have previously accessed the modem directly. If a client has software resident which uses TAPI to access the modem then it may not be directly accessible even when it is unused. Two new commands have been introduced which allow scripts to use TAPI to link to the modem.
TAPIDIAL device_number "dialstring" timeout_in_seconds
Device number is the index into the TAPI device list as shown in Control Panel -> Telephony lists, where the first device is 1 (and for most clients is the one to use). Dialstring is the number to dial, and timeout dictates how long to wait before aborting when no connection is made. TAPICLOSE will explicitly close down the TAPI connection and disconnect DMXCOMMS from TAPI.
TAPIDIAL replaces SETCOMM and script sections concerned with initialising the modem, dialling, and checking for connection. Some fairly fundamental changes have been made to the way we read and write to the modem to accommodate both it and conventional modem access, so these changes are still in beta mode.
.TPD partner data DMX131
Whenever a trading partner .SQN file is opened and read then, if a file with the same name and extension .TPD exists, it will be read into the buffer accessed by table *N5 and *P5 parameters. This allows automatic reference to partner specific data without having to use a profile. Up to 400 characters of information can be provided. We recommend comma delimited parameters to be accessed by *N5. Please note that the *N5 buffer is now blanked out automatically when a new .SQN file is read. If you are using *N5 or *P5 for control file data then you should switch to another DATA line.
The SEQNUM utility will be enhanced soon to provide a user interface to maintain the .TPD files.
Bugfix - Tradanet file send
A typo in buffer overrun detection logic introduced at the end of Nov. 2001 disabled the Tradanet file send capability. This has only just been discovered since the binary issue directory for installations was not updated until fairly recently. DMXCOMMS executables with a file date from 27/11/2001 to 14/04/2002 are flawed and should be updated (if you start using Tradanet!)
This directive will only be processed if it appears before the first #### line in the template. It is an instruction to the program to run section ####n before the first real section is processed, and thus functions like a call. It is a way to initialise working variables and also, if required, use the inbuilt functions in a header page for reports.
This directive will cause the processing of a section iteration to be concluded immediately if no number follows the skip. Typically used to skip irrelevant instances in conjunction with !IF evaluation. If n is non-zero then the re-formatter will skip that number of lines in the section before processing continues. You need to be careful here, particularly if editing an existing template or using backwards jumps (negative number). In particular !SKIP -1 will cause an infinite loop!
An optional first argument of form:
has been introduced, where HHH is a 3 digit number for font height, WWW is a 3 digit number for font width, and B is a single digit for font weight (boldness) from 0 for 'don't care' through 1 for light to 9 for heavy.. If present WinPrint will ask Windows to use a fixed pitch font whose type size is as close as available to the size given. We cannot say for certain what the result will be. Try /0800400 to start with and then experiment if that is not what you want. This feature is not officially supported, and never will be!
Setting initial Page Number
When you start up DMXPRINT the initial page number is 0, and for DMXEFAX it is 1. It is now possible using a command line parameter to set this initial page number.
will set the initial page number to 1. Remember that a !EJECT command will increment page number and generate a formfeed before any header section is called.
DMXCONST, DMXTRANS 2002-02-08
DISK Flag DMX130
Because of problems with the operation of lazy writes under NT4/Win2k the converters now default to using direct commitment to disk for output data files. This prevents data loss, or data being written to the wrong file under a profile change, but it is perceptibly slower. For Win 9x systems, which do not seem to suffer from this problem, a new control file flag has been added.
DISK=N will turn off commit-to-disk, DISK=Y will turn it back on again
IF THEN ELSE ENDIF & CALL
Beta versions of DMXPRINT and DMXEFAX are now available which implement the same IF THEN [ELSE] ENDIF logic as DMXREFRM. In addition a !CALL ####n directive is available to execute the numbered section as a 'subroutine' of the current one.
%!HANGUP Internal Variable
It is sometimes desirable to know if an external process called from a Wintask has 'hung'. Previously you could specify a timeout period in parentheses before invoking the process. Wintasks would then continue its processing after the timeout period even if the process had not completed, but you had no easy way in the task script of knowing when this occurred.
A new internal variable %!HANGUP has been introduced. When an external process is run this is set to zero. If the wintask continues because the timeout period expired, rather than because the process completed, this will be set to one. As an example:
if %!HANGUP <> 0 display Print spooling still active after 30 secs; may have hung up