An in-line probe produces a spectrum every five seconds. Over a fourteen-day culture, that is two hundred and forty thousand spectra. Where they go is rarely asked before the installation, and always after.
In-line measurement projects are discussed on the technology, the performance of the model and the price of the instrument. Then the measurement works, and a second question arrives that had gone unasked. This value: who stores it, who reads it again in three years, and what is shown to the auditor.
Data that is kept, readable and attributed is data that counts, for quality and in front of an inspection. It is what decides whether your installation is still in use in five years.
What is worth preparing on your side
- Who owns the raw data, and in which format it can leave the supplier’s software. A question for the consultation stage.
- The retention period applicable to your sector and your product, which sets the sizing of the archive.
- The target system: LIMS, MES, historian, or none of the three to begin with. And the person who decides.
- The position of computerised systems validation, obtained before the trials rather than after.
Four stages, and each has its cost
The lifecycle of a process datum is simple to describe. What is less simple is that each stage commits a decision that cannot be taken back afterwards.
1 · Acquisition
Automatic, robust, traceable. Automatic because manual re-entry is a break in traceability. Robust because a network outage must not create a silent gap in the series. Traceable because you will have to say which instrument, which setting and which software version produced that value.
The question to settle: what does the system do when the measurement is missing or doubtful? A blank, a zero value and a value flagged as invalid are not handled the same way downstream.
2 · Storage
Secure, validated, compliant. This is where the question of raw data arises: do you keep the spectrum, or only the value the model drew from it?
Keeping the result alone costs a thousand times less. But the day the model is revised, only the raw spectra make it possible to recalculate the history. And therefore to compare before and after. Without them, a model revision creates a discontinuity that nothing makes up for.
3 · Processing
With qualified tools, with the transformations tracked. A spectral pre-processing step is a calculation: it has a version, it produces a reproducible result, and it must be possible to replay it identically.
Processing done in a spreadsheet, on one person’s computer, with a formula known only to them. It is a frequent breaking point. It does not show as long as that person is there.
4 · Archiving
Long term, retrievable, in a durable format. It is the easiest stage to postpone, and the only one whose deadline is certain.
A proprietary format tied to a piece of software is data whose lifetime is that of the maintenance contract. The question to put to the supplier, at purchase and not ten years later: in which open format can the raw data come out.
An order of magnitude, to make the subject concrete. A thousand-point spectrum recorded every five seconds represents a few kilobytes. Over a two-week operation, that makes about a gigabyte of raw data for a single batch, and a single measurement point. Multiply by the number of measurement points, by the number of batches per year, then by the retention period your sector imposes.
The exact figure depends on the instrument and the rate, and it is established at scoping. What counts is the order of magnitude: the question becomes architectural, and it arises before the purchase.
The FDA PAT guidance makes exactly this point, and has done since 2004. The text, word for word: “Process analyzers typically generate large volumes of data. […] In a PAT environment, batch records should include scientific and procedural information indicative of high process quality and product conformance. For example, batch records could include a series of charts depicting acceptance ranges, confidence intervals, and distribution plots (inter- and intrabatch) showing measurement results. Ease of secure access to these data is important for real time manufacturing control and quality assurance. Installed information technology systems should accommodate such functions.“
The nuance between the two verbs is worth holding on to: scientific and procedural information in the batch record is expected, charts of intervals and distributions are given as an example. And the document carries the mention “Contains Nonbinding Recommendations“. That does not weaken the point: data architecture is within the scope of the text, not beside it.
LIMS and MES do different jobs
The two acronyms circulate as if they were interchangeable. They are not, and the choice of connection decides what the measurement will be able to do.
| What is compared | LIMS | MES |
|---|---|---|
| What it manages | Samples: registration, analysis workflow, archiving of results, issuing certificates | Manufacturing execution: orders, batch genealogy, real-time process data |
| Its clock | The laboratory’s. The sample exists, it is analysed, a result is returned | The workshop’s. The operation is running and the datum arrives during it |
| What a connection brings | The in-line result joins the analytical history of the batch and can feed a certificate | The batch record gains the measurement at the moment it is made, which opens the way to parametric release |
| The limit | A LIMS is not built to receive a measurement every five seconds | An MES does not necessarily keep raw data long enough for later analytical use |
In most installations the answer combines the two: an acquisition layer that keeps the detail, and two derived flows. A consolidated result to the LIMS, real-time data to the MES. That complication saves you from choosing between fineness and traceability.
What the connection changes for the people doing the work: the re-keying disappears, and the result is there when the decision is taken rather than when the laboratory reports back.
The protocols, and what they actually settle
Carrying the value up is the easy part of the subject. OPC UA is the reference in process industry. It transports the value, its time stamp and a data quality indicator, and that last point is missing from most hand-built links. Depending on the sector you also meet ASTM on analytical instruments, and HL7 in health environments.
A standard protocol is separate from a standard data model
The confusion that weighs most in the field. That an instrument speaks OPC UA means its variables can be read. It says nothing about how they are named, organised and documented. With no standardised information model and no published namespace, the integration is built on a structure specific to the manufacturer. It works, it costs to maintain, and it moves at each major version.
The question to ask at consultation, rather than after the order: is there documentation of the namespace, and is its stability across versions committed to? An evasive answer is no bar to buying. It puts a figure on a maintenance line that had gone unplanned.
Where the measurement stops and automation begins
This is the question asked in supplier qualification, and it deserves a written answer before the project rather than during it.
Five layers, each with an owner. The sensor produces a signal. The PAT acquisition system turns it into a value and a diagnostic. The SCADA or the DCS receives that value, displays it, archives it and decides. The PLC acts on the equipment. Above them, the MES holds the batch record, the LIMS the analytical results, the ERP the material flows.
Between acquisition and the control system there is often an orchestration layer — PAT orchestration software. It calls a method at the right moment, checks that every instrument is online, synchronises the start on the process event, applies the models, sends the prediction back to the control system and produces the batch quality report. On a single measurement it is not needed. As soon as there are several measurement points, several linked steps or a release to prepare, it is the piece that holds the whole together.
We deploy ProaXesS, from our partner KAx Group, which we have experience of deploying — and we work with the platforms already in place at our clients, synTQ among them. What is expected of this layer does not change from one publisher to the next: an audit trail, model version control, a clean separation between the development sandbox and GMP mode, and a proper interface to the SCADA.
Our responsibility is the data, and it reaches into your supervisory system. We deliver a value, a timestamp, a data quality indicator and an out-of-domain diagnostic, on a standard interface — OPC UA most often, sometimes OPC DA or Modbus depending on the installed base. And we support your teams on the part of the configuration that carries those data: the name and unit of each tag, the frequency, the threshold that means something, what the system should do with an out-of-domain diagnostic. The full SCADA configuration belongs to your IT or your integrator, whose trade it is — and we work alongside them.
| Regime | What the measurement does | What it takes |
|---|---|---|
| Display | The value is shown and archived. Nobody acts on it automatically. | A qualified installation and a periodic verification. This is the regime of the large majority of measurements in service. |
| Operator advisory open loop | The value or an alarm reaches the operator, who decides and acts. | A justified threshold, a written course of action, and an operator trained to tell a doubtful value from a sound one. |
| Automatic control closed loop | The value triggers an action with no human in between: stopping a drying, adjusting a flow rate, diverting a segment. | One step up. The measurement enters the control strategy, a fault in it becomes a process fault, and the behaviour on loss of signal is settled before the wiring. |
Closed loop is decided at scoping, and designed backwards
The useful question is not whether the measurement can control, it is what the installation does when the measurement goes quiet. A fouled probe, an aberrant spectrum, an out-of-domain sample, a broken link: each calls for a defined behaviour, and automation carries it rather than the model.
Hence the order of work: write the fallback first, the control second. A model that can say “I do not know” is what makes a closed loop defensible, which is why out-of-domain diagnostics matter as much as the prediction.
What goes in writing before the instrument is ordered. Who supplies the connection point and the power. Who writes the control logic and who validates it. Which protocol, which frequency, which tag format. Where the data is archived and for how long. Who is called, and within what time, when the measurement becomes unavailable.
Five lines in the requirement specification, and most integration disputes cease to exist. That is the work of scoping.
Integrity, in one sentence and one link
The requirements are known and they are not specific to in-line measurement. ALCOA+ principles for the datum itself, 21 CFR Part 11 and Annex 11 of the European GMP for the system carrying it. Access traceability, audit trail, rights control, encryption in transit and at rest, periodic review.
Worth repeating once more: an instrument announced as compliant is separate from a validated system. Compliance is demonstrated on the whole, instrument, workstation, network, database, procedures, people. What "21 CFR Part 11 ready" means, and what it leaves with you.
What to display, and the indicator rarely on the dashboard
An in-line measurement dashboard is built with five indicators, displayed in real time and read as control charts rather than as isolated figures. Four are expected. The fifth is the one that decides the lifetime of the installation.
- Availability of the measurement system. The percentage of time the measurement was really usable, rather than the time the instrument was powered.
- The deviation rate, and within it the share attributable to the measurement itself rather than to the process.
- The alerts, with what followed. An alert left unacknowledged is an alert that will soon be acknowledged by reflex.
- The return on investment achieved, against the one calculated at scoping. That is what funds the next measurement point.
- Model drift. Almost always absent, and the only one that anticipates rather than records.
Watching the drift means watching the residuals
A drifting model does not start displaying absurd values. It carries on returning plausible ones, which is precisely the point. What moves first are the diagnostic indicators: distance of the sample to the learnt domain, spectral residual, Mahalanobis distance. They rise before the result moves.
Carrying them onto the dashboard alongside the measured value costs little. Revising the model then becomes a planned operation rather than a reaction to a non-conformity. What keeping a model alive means.
Three points outside the technology
Treating the measurement as an instrumentation subject
IT and computerised systems validation are consulted at deployment, when the architecture is frozen. At that point they can no longer steer, only stop. Consulted at scoping, they choose with you.
Leaving training and support out
A curve the operators cannot interpret ends up set aside. The training budget is the easiest line to cut and the one that costs most to have cut. What to prepare on the organisation side.
Under-sizing the infrastructure
Storage, network and backup are sized on the first measurement point. The third arrives eighteen months later, and the architecture follows with difficulty. It costs little to plan generously at the start, and a great deal to rebuild later.
And artificial intelligence in all this
The volume, the rate and the diversity of process data are increasing, and the methods that exploit that material are progressing quickly. Predictive models, anomaly detection, digital twins: real subjects, and we practise them.
The order matters, though. None of those methods compensates for data poorly acquired, poorly time-stamped or incomplete. They amplify what they are given. An installation whose four steps above are clean is ready for those methods on the day they serve. An installation that has left them aside is not, whatever the algorithm.
There is a further reason to take the subject in that order in a regulated environment. A model whose output cannot be explained is harder to defend in front of an inspector. Performance is one half. Being able to say why is the other.
Tell us where your measurement will have to land. You will know what that implies.
Forty-five minutes is enough: locating the target system, estimating the volume of data your rate will produce, identifying who needs to be consulted and when, and knowing whether the architecture envisaged will hold at the third measurement point. IT and computerised system qualification have their place in this exchange: that is where they steer, instead of blocking later.