ComCap5 Data Capture Utility
===========================

Readme for Beta Release 5.4 - 8th June 2024
-------------------------------------------

ComCap5 is a Windows application designed to capture data received on PC serial
communications ports or using network TCP and UDP streams and write it to text
files.  Captured data is shown in scrolling windows, and may be printed,
written to SQL database tables or echoed to other PCs using network protocols
or serial ports. Captured data can have text added such as date and time, a
serial number and remote IP address. Data from up to 999 serial ports and 2,000
network streams can be captured simultaneously, in separate or mixed files,
with various file rotation schemes to start new files periodically. ComCap will
capture to files on two separate disk drives for redundancy and will send email
and SMS alerts if problems occur. ComCap is both a system tray and background
service application that can be set to start automatically when windows starts,
and remain unobtrusive.  When using the background service, captured data may
be still be viewed as it arrives.

Major changes between v4 and v5

- Single channel will accept hundreds of simultaneous remote clients with 'TCP
  Multi Server', simplifies set-up and operation.
- Zip compression of capture logs during rotation to save disk space.
- Improved data Filtering including  'Required Phrases', one of which must
  exist for record to be captured.
- Capture alerts to different email addresses or SMS numbers for differing
  phrases.
- Searching for phrases including wildcard characters or complex regular
  expressions.
- Capture of XML and Json data formats, more flexible CSV formats.
- Capture of HTTP protocol POST and PUT requests.
- Reformat captured data, for instance from Fixed Width Columns to CSV, saving
  in new format.
- New SMS bureau, The SMS Works.
- Automatic free SSL/TLS certificate acquisition and installation from Let's
  Encrypt.
- Improved SSL/TLS certificate support, more flexible configuration.
- Capture time format can now be UTC or local time without summer time.
- Hexadecimal capture data converted to ASCII.
- New Line or Record Start option, as well as Line End.
- Manually close a remote TCP connection if stalled.
- Windows Defender Firewall support.

ComCap features include:

- Capture from hundreds of serial COM ports or TCP/IP Server, TCP/IP Client and
UDP/IP network protocol streams simultaneously, with suitable hardware.  Many
network appliances output log information, typically using the UDP/IP 'syslog'
protocol, and as telephone switches become network aware they are offering
network logging instead of the serial port. A TCP/IP Multiple Server channel
will capture up to 2,000 simultaneous SSL streams saved to a single log file.

- Serial COM ports are dynamically detected as they are installed and removed
from Windows, so capture will automatically start if a USB serial device is
plugged in, and stop if it's unplugged.

- Capture from serial ethernet device servers that ‘convert’ serial port data
to network protocols, from Brainboxes, StarTek, Lantronix, Lavalink and RE
Smith for instance, easing data capture distances, Audon in the UK sells
various converters.

- ComCap supports IPv4 and IPv6 network standards, and SSL/TLS network capture
and echo, to provides encryption of data sent over the internet, and to confirm
the identity of the other party with X509 SSL/TLS certificates.  It includes
support for the latest TLSv1.3 protocol and automatic SSL/TLS certificate
acquisition and installation from Let's Encrypt.

- Email may be captured, such as the alert emails sent by internet aware
appliances, such as firewalls, security monitors, power distribution units,
uninterruptible power supplies, remote sensors, transponders, etc. The emails
may be written to a SQL database or used to trigger alerts.

- GPS NMEA 0183 sentences and various vehicle tracker outputs can be parsed
into simple comma separated records, allowing location information received
from a serial or network connected GPS sensor to be easily saved to a SQL
database.  A map window shows a GPS track to ensure everything is working.

- GPS location data may be captured on Windows tablets and high end laptops
that have a GPS location sensor built-in.

- Captured data is optionally shown in scrolling windows as it arrives, and
earlier data can be viewed as well.  Coloured tabs indicate which channels have
started capturing data and which are stopped.

- Data from multiple serial or network capture channels may be merged or
consolidating, allowing all data to be displayed in a single window, written to
a single log file, and added to a database using a single connection.

- Capture files may be in separate directories for each capture port and new
files may be created daily or multiple per day (at specified times), weekly
(Monday), monthly, hourly, every few minutes, after an inactivity period, when
a new page character is received, each new record, or a fixed file name may be
used.  Multiple channels may optionally capture data into a common file, to
reduce the number of separate log files.

- When a capture file is closed and rotated for a new file, it may be
optionally zip compressed to save space, moved to another directory for further
processing or emailed to multiple recipients.

- Capture file names (and optionally paths) are automatically generated, with
file name format customised with date and time in various ways, numeric or
alphabetic.

- Capture data may be written to a database, such as Microsoft SQL Server or
MySQL.  Data formats may be created to identify fields within each line of
data, as fixed width columns, character separated columns (CSV), variable named
columns, Json or XML.  ComCap examines the SQL tables or stored procedures for
column names and types, and allows mapping of which field of data is written to
which SQL column.  ComCap protects data that can not be immediately written to
the database due to network problems, and will write it once the database
becomes available again.

- Capture data may be reformatted and saved in a different format, for instance
fixed width lines of data, Json or XML may be saved as comma separated quoted
columns for easier processing, or CSV could be saved as Json.

- ComCap can echo or proxy captured data directly to serial communications
ports or parallel ports to drive printers, or echo to the network using UDP/IP
(syslog), TCP/IP Server or TCP/IP Client protocols.  Network echo can be used
to allow capture of the same data to a maximum of five PCs simultaneously for
redundancy or for remote capture with one PC near the data source echoing data
to a centralised location.

- ComCap has options to safeguard captured data, closing the log after each
line to force it to write to disk, or after an inactivity period or
periodically every few seconds or minutes.

- A separate information log file is maintained showing when capture starts and
stops and other ComCap events, it will log capture status hourly with the
number of lines received from each port or stream and to which logs it is being
written.  The information log may be sent to a remote PC using network
protocols, perhaps to another copy of ComCap, to ease central monitoring of
remote capture.

- A sound file may be played when each new line of data is captured (with a
minimum gap between sounds, in case of frequent data).

- Raw data may be captured unchanged from the COM port or network stream, or
the data may be cleaned up with non-printing characters removed and trailing
spaces removed. Hexadecimal capture data can be converted to ASCII

- When capture is started or stopped, command strings may be optionally
transmitted and periodically repeated, perhaps to trigger a remote appliance to
start or stop. Optionally, extra commands may be transmitted on demand, perhaps
to configure a remote capture device.

- Captured lines may have text added at the start or end, that can include
escape sequences to add a serial number (of specified length), date and time in
various formats, PC name, local or remote IP address.

- A comment may be added manually to any capture channel, using a pop-up
dialog, this is intended to assist in documenting batch captures, perhaps from
laboratory instruments.

- Continuous data streams may be selectively captured by filtering to reduce
the amount of data saved, for instance from GPS locators or environmental
sensors. Alerts may be sent according to filtered data. Filters may discard
records with specific phrases or only save records such records (ie specific
mobile IMEI or IP address).

- Alerts for problems are presented in a pop-up window and may be sent by
email, by SMS to mobile telephones, using either a GSM modem or SMS internet
gateway or to a remote PC using network protocols, perhaps to another copy of
ComCap, to ease central monitoring of remote capture. Alerts may also be
triggered for phrases in captured data.

- Data loss checking, to detect if other windows applications caused ComCap to
possibly lose data.  An alert may be triggered if new data is not captured
after a period, configurable by time and day of week, or if the PC appears to
hang for a short period.

- For redundancy, ComCap will capture logs on two different disk drives at the
same time, and continue logging if one of the disk drives is lost, the drives
may be across a network with remote logon details specified.  ComCap makes
multiple attempts to open files, in case of conflict problems such as backup
and protects data until it can be written to the capture file.  If disk space
runs low, ComCap will send an alert.

ComCap usually runs as a Windows service that starts immediately the PC starts
without needing a user to log-on.  There is also a system tray application
which can configure, monitor and control the service displaying data currently
being captured, but which will also capture data if the service is not used.

ComCap is supported on Windows Vista, 7, 8, 10, 2008, 2012, 2016 and 2019, both
32-bit and 64-bit editions. Note Windows XP, 2003 and 2000 are no longer
supported. ComCap may be run with a higher priority than other windows
applications to ensure stable data capture, although a dedicated PC is
recommended for valuable information.

ComCap was originally designed to capture telephone call logging data from the
serial port provided on most telephone switching systems (PABXs), typically
called Call Data Records (CDR).  The saved data may then be used as input to
telephone call management applications that will cost calls and produce reports
on telephone usage or for security audit purposes.

ComCap is application non-specific and will capture any data that arrives on a
serial port or using network protocols.  It has been used for a wide variety of
purposes, such as logging output from test, alarm, GPS and monitoring equipment
and serial printer data (but it can not process printer control characters).

Please note that ComCap is not designed to monitor data between modems and PC
applications, it requires exclusive access to the serial port so no other
application can use the port at the same time.

The ComCap distribution includes the 'ComGen Data Stream Generator' application
which is designed to generate various types of test streams using any or all of
the PC serial COM ports and potentially dozens of network streams, UDP Client,
TCP Client or TCP Server, and has been used extensively for testing ComCap
capturing multiple channels.  There is also a small ‘Test Serial Ports and
Hardware Events’ utility that may be useful in testing communication port and
cabling problems.

If the requirement is to capture serial data from another application on the
same PC, ComCap includes a Null Modem Emulator (com0com) from
http://com0com.sourceforge.net/ that installs a linked pair of virtual serial
ports, instead of needing to use a physical pair of COM ports and a null modem
cable.


Installation
------------

ComCap is usually distributed as a ZIP file COMCAP5.ZIP, which should
first be unzipped into a temporary folder.

The program CM5SETUP.EXE should then be run to install ComCap.  About 20
MBytes of disk space is required for applications, help and supporting
files.

If you have received an executable with the name CM5SETUP.EXE, this
should be run to install the software.

ComCap is normally installed into the program files directory, in the
comcap sub-directory.  A user with administrative level privileges must
install ComCap, and configure the service version.

ComCap may be accessed by users without administrator access to the
PC, provided file permissions are set-up accordingly for the directories
being used by ComCap.  Note that only administrators may start and stop
Windows services.  If ComCap is configured to run as a service, non-
administrators will be able to run the system tray application and
monitor the service, but not start or stop it (the buttons are hidden).
The PC name, user name and security level are logged on start-up, for
the service it will say: User: SYSTEM (Administrator).

In Windows 10, it is not possible to have desktop start-up programs with
administrator rights, they simply don't run when you logon on  (presumably
a new anti-malware measure).  But the ComCap tray application needs
administrator rights in order to start and stop the ComCap Service.
Normally, the ComCap Service will start automatically when Windows boots,
and the ComCap Tray application can still monitor the service, change
settings and pause/resume single channels, but it can not stop the ComCap
Service or Start it again without being given administrator rights through
Properties, Compatibility, 'Run this program as an administrator'. So the
ComCap Service will need to be controlled by the Services Windows
Administrator Tool, if you want the ComCap Tray to start automatically.
Workarounds for this are still being investigated.

All ComCap settings are stored in files, rather than in the registry, to
ease sharing between users and back-up.  Actual configuration settings
are stored in the Windows common application path:

C:\Users\All Users\Magenta-Systems\ComCap5

The actual files are comcap5.config and comcap5.current, so the complete
file name is C:\Users\All Users\Magenta-Systems\ComCap5\comcap5.config.
The comcap5.config file is only updated when configuration changes occur,
the  comcap5.current file contains dynamic information updated ever few
seconds, such as log names and serial numbers.  A sub-directory
certificates is used to save SSL/TLS certificates.

User specific information such as windows sizes and positions, fonts,
etc, for the tray application is stored separately for each user in a
single file:

C:\Users\(Administrator)\AppData\Roaming\Magenta-Systems\comcap5.forms

where (Administrator) is replaced by the user logon name. If access to
directories is blocked, or they don't exist, then the configuration files
are stored in the ComCap program directory.

ComCap is run from Start, Programs, Magenta Systems, then click ComCap5.
The ComCap icon will appear in the System Tray, which should be clicked
to restore the main application window.  ComCap may be set-up to auto
start each time windows is booted through Common.

Although the ComCap Background Service may be used logged-on to the
‘local system account’, it is better to specify a specific user account
with a password.  This ensures that dates and times in ComCap will be
formatted according to that user’s preferences, rather than the American
default, and allows access to network resources.  Windows Echo printing
is only possible when using the account of a user that has the required
printer installed.

Note that the ComCap PC must be running continually for data to be
captured.  Any power saving features, such as the hard disk being
powered down, should be disabled, and suspending the PC will also stop
anything being captured.


Uninstalling
------------

ComCap may be removed from the Control Panel, Add/Remove Programs, or by
running the program unins000.exe in the installation directory.  The
uninstaller will remove all installed programs and files.  It will not
remove any capture or log files that ComCap has created.


Release History
---------------

This is now in the help file.


Ordering ComCap
---------------

Until registered, ComCap will only run for one hour before capture is
automatically stopped.  It will be necessary to restart ComCap to
continue capture.  ComCap may be registered using a secure order form at
the web site https://www.magsys.co.uk/comcap/.  Once the order has been
processed, a small file userreg.txt is returned as an email attachment,
which should be copied into the ComCap program directory.  If ComCap is
already running, it should be exited and restarted to see the new file.
When started, ComCap will show the name of the individual or company
that registered the software and will run continually.  The registration
status is shown in the information log on start-up and when capture
starts.  The Order form may also be used to request a temporary one week
license key for extended testing.


Distribution
------------

ComCap may be freely distributed via web pages, FTP sites, BBS and
conferencing systems or on CD-ROM in unaltered zip format, but no
charge may be made other than reasonable media or bandwidth cost.  Any
notifications must state that ComCap is shareware until registered
with Magenta Systems Ltd.  Please email Magenta Systems Ltd if you
distribute ComCap in some way, so you can be notified of upgrades or
other important changes.


Copyright Information
---------------------

Copyright Magenta Systems Ltd, England, 2024.

Magenta Systems Ltd
9 Vincent Road
Croydon
CR0 6ED
United Kingdom

Phone 020 8656 3636, International Phone +44 20 8656 3636

Email: comcap@magsys.co.uk
Web: https://www.magsys.co.uk/comcap/
HTTP Download: https://www.magsys.co.uk/download/software/comcap5.zip
FTP Download: ftp://www.magsys.co.uk/software/comcap5.zip


