Introduction:
How to consolidate HL7 messages into an hourly data file.
When an application receives a continuous stream of HL7 messages, the way those messages are stored can have a significant impact on downstream processing.
The default behavior of the CORE HL7 TCP/IP Listener is deliberately straightforward: each successfully received HL7 message is written to its own HL7 data file. This provides a simple, reliable way to persist incoming messages and makes each message independently accessible.
For many applications, however, individual message files are not the most efficient format for subsequent processing.
Consider a system receiving several thousand HL7 messages per hour. If every message is written to a separate file, a downstream application may need to open, read, process, and close thousands of files simply to process the messages received during that hour.
This is where the embedded CORE HL7 C# Script Engine can provide a significant advantage.
A C# script can process each HL7 message in memory as it is received, and instead of allowing the Listener's normal individual-file output to occur, the script can redirect the messages into a consolidated hourly data file.
The result is a much simpler workload for the downstream application:
Thousands of individual HL7 messages → one hourly HL7 data file
The accompanying diagram illustrates this architecture.
The CORE HL7 TCP/IP Listener default behavior
At its most basic, the CORE HL7 TCP/IP Listener performs three fundamental tasks:
- Accept an incoming HL7 connection.
- Receive and validate the HL7 message.
- Write the received message to the configured Data Folder.
With the default configuration, each HL7 message is written to a separate data file.
For example, if three messages arrive, the Data Folder would contain 3 files:
Message.l16.20260914144540743.hl7
Message.l16.20260914144540812.hl7
Message.l16.20260914144540903.hl7
The filename contains useful information that identifies the message and its origin. In this example:
Message.l16.20260914144540743.hl7
can be interpreted as:
|
Component
|
Meaning
|
|
Message
|
Literal filename prefix
|
|
l16
|
HL7 Listener Profile ID
|
|
20260914144540743
|
Date/time to milliseconds plus a rolling number
|
|
.hl7
|
HL7 file extension
|
This default behavior has an important benefit: every message is independently persisted.
If a downstream application needs to process one message at a time, this will be exactly what is required.
But what happens when the downstream application needs to process messages in batches?
The downstream processing problem
Suppose an HL7 interface receives 10,000 messages during an hour.
With the default behavior, the Data Folder could contain approximately:
10,000 individual HL7 files
A downstream application could certainly process these files, but it now has to perform a considerable amount of file-system activity:
Find file
↓
Open file
↓
Read HL7 message
↓
Process message
↓
Close file
↓
Move/delete/archive file
↓
Repeat thousands of times
The number of files can also become a consideration for file-system management, monitoring, backup, archival, and downstream application design.
In some environments, it would be much more efficient for the downstream application to process a single file containing all of the messages received during a particular hour.
That is the problem addressed by this scenario.
Using the embedded C# Script Engine
The CORE HL7 TCP/IP Listener includes the CORE HL7 C# Script Engine as an integrated module within the Listener application.
It is important to understand that the Script Engine is not a separate server or network component.
The processing takes place within the Listener's message-processing workflow.
Conceptually, the process becomes:
Incoming HL7 Message
↓
CORE HL7 TCP/IP Listener
↓
Embedded C# Script
↓
Modify / Filter / Enhance
↓
Write to Hourly Data File
The significant point is that the script gets an opportunity to work with the HL7 message before the Listener performs its normal default file output.
This makes the Script Engine useful not only for transforming messages, but also for implementing an alternative output strategy.
Overriding the default output behavior
The C# script can take control of what happens to the message after it has been received.
Instead of allowing the Listener to create:
Message 1 → Individual file
Message 2 → Individual file
Message 3 → Individual file
...
Message 10,000 → Individual file
the script can append or otherwise redirect the messages into a file associated with the current hour:
Message 1 ─┐
Message 2 ─┤
Message 3 ─┤
Message 4 ─┤──→ 2026091414.CS012.hl7
... │
Message N ─┘
At the beginning of the next hour, the script starts using the next hourly file:
2026091414.CS012.hl7
2026091415.CS012.hl7
2026091416.CS012.hl7
The exact implementation of the script is under the developer's control. The script can determine the appropriate filename, open or create the file, and write the incoming message to it.
The result is that the downstream application can process one file per hour instead of potentially thousands of individual files.
The message can also be transformed before it is stored
Consolidating the messages is only one possible use of the C# Script Engine.
Because the script operates on the message before the output is generated, it can also perform processing such as:
- Filtering unwanted messages
- Removing unnecessary segments or fields
- Modifying field values
- Adding information required by the downstream application
- Converting or normalizing values
- Routing messages according to their content
- Enriching the HL7 message
- Applying customer-specific business rules
For example, the script could examine the content of segments like:
MSH, PID, PV1, OBR, OBX, etc.
and determine whether the message should be included in the downstream hourly file.
This means the architecture is not simply:
HL7 Listener → File
It becomes:
HL7 Listener → In-memory processing → Customized output
That distinction is important.
Why process the message in memory?
One of the advantages of this approach is that the transformation occurs before an unnecessary intermediate file is created.
Without scripting, the process would potentially look like:
HL7 Message
↓
Individual HL7 File
↓
Read individual file
↓
Modify / filter message
↓
Write consolidated file
The script-based approach can instead be:
HL7 Message
↓
CORE HL7 TCP/IP Listener
↓
C# Script
↓
Modify / filter / enhance in memory
↓
Hourly output file
The second approach eliminates the need for the downstream process—or another intermediate process—to first retrieve thousands of individual files simply to combine them again.
A practical example
Imagine a laboratory system sending approximately 5,000 ORU messages per hour. The default Listener behavior would result in 5000 separate HL7 message files per hour.
A downstream laboratory analytics process, however, only runs once per hour.
Rather than having that application scan thousands of individual files, a C# script can organize the messages into hourly files:
2026091413.CS012.hl7
2026091414.CS012.hl7
2026091415.CS012.hl7
The downstream application can then process the completed hourly file at a convenient time and only has to make sure that they ignore any file for the current hour (it’s still being created).
For example:
13:00–13:59
↓
2026091413.CS012.hl7
↓
Downstream processing
14:00–14:59
↓
2026091414.CS012.hl7
↓
Downstream processing
This creates a natural boundary between real-time HL7 acquisition and batch-oriented downstream processing.
Real-time acquisition, batch downstream processing
This is perhaps the most useful way to think about the architecture.
The Listener remains a real-time HL7 interface.
Messages do not need to wait for the downstream application. They are received and acknowledged immediately and processed by the embedded script as they arrive.
The downstream application can operate independently according to its own schedule.
This separation can be particularly useful when the receiving system must remain continuously available but the consuming application does not need to process every HL7 message immediately.
An important distinction: receiving vs. consuming
The Listener and the downstream application have different responsibilities.
The CORE HL7 TCP/IP Listener is responsible for reliable message acquisition. It handles the incoming HL7 connection and receives the messages as they are transmitted.
The C# Script Engine provides the customization layer between acquisition and storage.
The downstream application is responsible for consuming the resulting data according to its own requirements.
This separation allows each component to do what it does best:
|
Component
|
Responsibility
|
|
HL7 Sending System
|
Transmit HL7 messages
|
|
CORE HL7 TCP/IP Listener
|
Receive HL7 messages
|
|
Embedded C# Script
|
Filter, modify, enhance, and route messages
|
|
Hourly Data File
|
Consolidate messages for downstream consumption
|
|
Downstream Process
|
Consume and process the consolidated data
|
Default behavior remains available
An important aspect of this approach is that the Listener's default behavior remains useful.
If no custom scripting is required, the Listener can simply write each incoming message to its individual HL7 data file.
For example:
Message.l16.20260914144540743.hl7
Message.l16.20260914144540812.hl7
Message.l16.20260914144540903.hl7
There is no requirement to implement scripting merely to receive HL7 messages.
The embedded Script Engine becomes valuable when the application's requirements go beyond simple message reception and individual-file storage.
In this scenario, the script effectively provides an alternative output mechanism tailored to the downstream application's requirements.
When is this approach appropriate?
Consolidating messages into hourly files can be particularly useful when:
- Large numbers of HL7 messages are received.
- The downstream application processes messages in batches.
- The downstream application runs periodically rather than continuously.
- File-system overhead from large numbers of individual files is undesirable.
- Messages need to be filtered before reaching the downstream process.
- Messages need to be transformed or enriched.
- A defined time-based processing boundary is useful.
- The downstream application is designed around file-based ingestion.
It is not necessarily the best approach for every integration.
If a downstream application needs to react to every HL7 message immediately, individual message processing—or another real-time integration mechanism—may be more appropriate.
The important point is that the Listener does not force the downstream application into one particular processing model.
Conclusion
The default behavior of the CORE HL7 TCP/IP Listener provides a simple and reliable mechanism for persisting incoming HL7 messages: one message per HL7 data file.
For applications receiving relatively small message volumes, this may be all that is required.
For higher-volume interfaces, however, a downstream application may benefit from processing consolidated data rather than thousands of individual files.
The embedded CORE HL7 C# Script Engine provides a way to change that workflow.
By processing each message in memory as it arrives, the script can filter, modify, enhance, and ultimately redirect the message into an hourly data file. The Listener continues to perform the real-time HL7 acquisition, while the downstream application receives data in a format optimized for its own batch-processing requirements.
The resulting architecture is straightforward:
Receive → Process → Consolidate → Consume
And because the scripting capability is embedded directly within the CORE HL7 TCP/IP Listener, this can be accomplished without introducing another server or intermediate network service into the HL7 interface.
