We’re experiencing critical workflow failures in our quality management module during batch release operations. When our QA team attempts to release inspection lots in batch mode (typically 15-20 lots at once), the workflow randomly fails for 3-5 lots with a generic status error.
The inspection lot status transitions seem inconsistent - some lots move from ‘In Review’ to ‘Released’ successfully, while others stay stuck in ‘In Review’ despite passing all quality checks. We’ve verified workflow permission checks are configured correctly for the QA role, but the batch operation still fails.
This is causing significant shipment delays as we have to manually investigate each failed lot and retry the release individually. Our custom handler registration for quality events was working fine in TC 12.2, but after upgrading to 12.3, these batch failures started occurring. Has anyone encountered similar inspection workflow issues with batch operations?
Also worth checking the workflow template itself. In TC 12.3, there were changes to how status transitions are validated in quality workflows. The batch processor might be hitting a validation rule that wasn’t enforced in 12.2. Look at your workflow exit conditions - specifically any custom conditions that check inspection lot attributes. If these conditions aren’t thread-safe, they can fail intermittently during batch processing. The fact that it’s random suggests a race condition rather than a configuration error.
I’d recommend a hybrid approach here. Keep critical validation handlers synchronous but move notification and secondary update handlers to asynchronous. For quality workflows specifically, the inspection lot status transition must complete synchronously to maintain data integrity. However, updating related change objects or sending notifications can happen asynchronously without impacting the core workflow.
Before changing to asynchronous, understand the implications. Asynchronous handlers don’t guarantee immediate execution, which might affect your downstream change object updates. A better approach might be to implement proper transaction handling within your synchronous handler. Use separate transactions for each inspection lot update and implement retry logic with exponential backoff. This way you maintain synchronous behavior but avoid the lock contention issues. Also verify your workflow permission checks include the batch processor service account.
The root cause of your batch release failures is a combination of factors introduced in TC 12.3. Let me address each focus area systematically:
Inspection Lot Status Transitions:
TC 12.3 introduced stricter validation for status transitions in quality workflows. The batch processor now enforces atomic status changes - either all lots in the batch succeed or none do. Your intermittent failures suggest that 3-5 lots are hitting validation errors that weren’t caught in 12.2. Check the workflow audit logs for specific status transition errors. The status must progress through defined states: Draft → In Review → Approved → Released. Skipping states or having invalid intermediate statuses will cause batch failures.
Workflow Permission Checks:
The permission model changed subtly in 12.3. Batch operations now require explicit ‘Batch Release’ permission on the Quality Administrator role, not just standard ‘Release’ permission. Navigate to Organization > Role > Quality Administrator and verify this permission exists. Additionally, the batch processor service account needs ‘Modify All’ privilege on inspection lots, which is separate from user-level permissions. This is often overlooked during upgrades.
Custom Handler Registration:
Your synchronous handler is the primary culprit. In TC 12.3, batch quality operations use parallel processing threads (configurable via wt.quality.batch.threadPoolSize). Synchronous handlers block these threads, creating contention. Here’s the fix:
-
Re-register your handler as asynchronous for batch contexts:
- Use HandlerRegistry.registerHandler with ExecutionMode.ASYNC
- Implement QueuedEventHandler interface instead of EventHandler
- Add batch context detection: if (isBatchOperation()) use async path
-
For change object updates, implement a separate batch-aware handler:
- Collect all inspection lot IDs in the batch
- Update related change objects in a single transaction after batch completes
- Use workflow completion event rather than status transition event
-
Add transaction isolation:
- Wrap inspection lot updates in SERIALIZABLE transaction level
- Implement optimistic locking with version checking
- Add retry logic for deadlock exceptions (max 3 retries)
Configuration Changes:
Update your quality.properties:
- wt.quality.batch.transactionTimeout=300 (increase from default 120)
- wt.quality.batch.threadPoolSize=5 (reduce if you have many custom handlers)
- wt.quality.workflow.validateTransitions=true (ensure this is enabled)
Restart the method server after these changes. Test with small batches (5 lots) first, then scale up. This should resolve your shipment delays and stabilize the batch release process.
Thanks for the insights. I checked our custom handler and you’re right - it’s registered as synchronous. We have a handler that updates related change objects when inspection lots are released. Should I simply change the registration to asynchronous, or are there other considerations for quality workflows?
Let me add one more critical point about TC 12.3 quality workflows. There’s a known issue with batch operations when custom exit conditions reference related objects through multiple hops. If your workflow checks something like InspectionLot → Part → ChangeOrder status, the batch processor can hit database deadlocks.