To evaluate manufacturing operations platforms, run a requirements workshop before talking to vendors, test connectivity against your actual machine park, verify integration with your existing ERP, WMS and CMMS, demand a concrete time-to-value commitment, check security certifications such as ISO 27001 and GDPR compliance, structure a scored proof of concept on your own lines, and read the pricing model for hidden scaling costs. Score every vendor against the same weighted criteria.
Why is this a hard purchase to get right?
An operations platform touches the shop floor, the planning office, maintenance, quality and IT at the same time, and it will sit in the middle of your stack for years. The market does not make it easier: products with very different centres of gravity - OEE monitoring, frontline apps, edge data, predictive maintenance, full operations suites - all answer to the same search terms.
Demos make every platform look finished, because demos run on the vendor's data against the vendor's machines. The evaluation methods below are designed to replace demo impressions with evidence from your own environment.
What does a structured evaluation look like?
- Run a requirements workshop before contacting vendors. Get operations, maintenance, planning, quality and IT in one room for half a day. List the decisions and workflows the platform must improve, rank them must-have versus nice-to-have, and write down your KPI baseline. The output is a one-page requirements sheet that keeps the process anchored when vendor roadmaps start sounding attractive. Vendors should respond to your requirements, not define them.
- Check connectivity against YOUR machine park. Export your asset list with make, model, age, controller type and available interfaces, and ask each vendor to map it line by line: native OPC UA or MQTT here, Modbus to the PLC there, serial or retrofit sensors for the legacy presses. Be suspicious of the answer "we connect everything" without a per-asset method. The honest answer names the protocol per machine and flags the hard cases up front.
- Verify integration with your existing systems. The platform must exchange data with the ERP (orders, materials, confirmations), the WMS (stock movements) and the CMMS (work orders, asset history) you already run - not the ones the vendor prefers. Ask for a reference customer on your specific ERP version. A platform that cannot read and write to your stack becomes another silo with a nicer interface.
- Ask the time-to-value question explicitly. "How long until our first line is live and our first KPI moves?" should get an answer in weeks, with a project plan behind it. Multi-quarter implementation projects were normal a decade ago; modern platforms connect devices in minutes and stand up first dashboards in days. Long deployments also have a hidden cost: the team that championed the project loses momentum before seeing results.
- Demand the right security certifications. Minimum bar for an EU or UK plant: ISO 27001 certification for the vendor's information security management, GDPR compliance with a signed data processing agreement, and clarity on where your data is hosted. Cyber Essentials is a reasonable additional signal for UK operations. Ask how OT network access is secured and what happens to your data if you leave. Treat missing certifications as disqualifying, not negotiable.
- Structure a proof of concept, not a pilot that drifts. Agree in writing: one or two real lines, your machines, a fixed duration of four to six weeks, named success criteria from your requirements sheet, and a price (free PoCs are fine, but unscoped ones are how evaluations drift for a year). At the end, score against the criteria and decide. A vendor who resists measurable PoC criteria is telling you something useful.
- Read the pricing model for scaling traps. Common pitfalls: per-user pricing that punishes you for rolling out to the whole shop floor, per-data-point or per-tag pricing that turns connectivity success into a cost problem, mandatory professional services for every new line, and integration connectors priced as separate products. Model the cost at full deployment across all plants, not at pilot scope, and ask what the price does in year three.
How do you score the shortlist?
Use one weighted scorecard for every vendor, completed by the same evaluation team, with PoC evidence outranking demo impressions. Adjust the weights to your situation - the discipline matters more than the exact percentages.
| Criterion | Weight | What to score |
|---|---|---|
| Fit to must-have requirements | 25% | Coverage of the workshop's must-have list, evidenced in the PoC |
| Connectivity to your machine park | 15% | Per-asset connection method, legacy coverage, effort per device |
| Integration with ERP / WMS / CMMS | 15% | Live bidirectional integration with your versions, reference proof |
| Time-to-value | 10% | Weeks to first live line and first measured KPI movement |
| Security and compliance | 10% | ISO 27001, GDPR with DPA, hosting location, OT access model |
| Usability for shop-floor teams | 10% | Operator adoption during the PoC, training effort |
| Total cost at full scale | 10% | Three-year cost modelled at full rollout, all fees included |
| Vendor viability and support | 5% | Customer base, support model, product cadence |
Where do specific platforms fit?
The shortlist should follow your requirements, not the other way round. Frontline-app platforms, OEE specialists, edge data tools and enterprise IIoT suites each win on different profiles. KFactory is one option that meets the criteria above for mid-market operations teams: connectivity across OPC UA, Modbus, MQTT and serial, integration with ERP, WMS and CMMS systems, ISO 27001 and Cyber Essentials certification with GDPR compliance, and an agent-based approach - described in the Operate and Plan modules - where AI agents act on the data rather than only displaying it. Evaluate it with the same scorecard as everyone else.
Whichever way the scoring lands, keep the artefacts: the requirements sheet, the asset mapping and the PoC scores become the implementation plan for the winner, so none of the evaluation effort is wasted.
Frequently asked questions
How many vendors should we evaluate?
Long-list six to eight from desk research, shortlist two or three for the proof of concept. Running more than three PoCs exhausts the internal team and degrades the quality of every evaluation.
How long should the whole evaluation take?
Roughly one quarter: two to three weeks for requirements and desk research, two to three weeks for structured demos and reference calls, four to six weeks for the PoC, then decision. Longer than that usually means the requirements were never agreed.
Should IT or operations own the evaluation?
Operations should own the outcome and the requirements; IT must own the security, integration and architecture criteria with veto power on them. Evaluations run by either side alone produce platforms the other side rejects.
Is a free proof of concept better than a paid one?
Not necessarily. The scope and the success criteria matter more than the price. A paid, tightly scoped PoC with named criteria typically gets more vendor attention and a faster, clearer result than an open-ended free trial.
What is the most common evaluation mistake?
Choosing from demos without testing connectivity against your own machine park. Connection effort against real, mixed-age equipment is where platforms differ most, and it is invisible in a demo.