We’re implementing demand planning in D365 10.0.39 and need to sync forecast data from our external demand planning tool to D365 Supply Chain Management. The sync includes historical sales data, statistical forecasts, and demand plans across 15,000 SKUs and 200 customer locations. Currently syncing twice daily, but we’re seeing data consistency issues where forecast versions get out of sync between systems, and some batch processes fail silently without proper error notification. Looking for recommendations on batch processing approaches and error handling patterns that others have successfully used for demand planning data synchronization.
This is really helpful. We’ve started implementing batch processing by product category and added basic error logging. Already seeing improvement - we can now identify which specific SKUs are causing issues rather than having the entire sync fail mysteriously. Question on the incremental sync - how do you track which SKUs have changed? Are you comparing checksums or using timestamp-based change detection?
We’re currently overwriting the active forecast each sync, which might be causing the version conflicts. Your approach with separate streams makes sense. How do you handle the validation between stages? And what’s your batch size for 15,000 SKUs - are you processing all SKUs in one batch or breaking it down?
Demand planning sync is tricky because of the versioning complexity. Are you syncing forecast versions as separate entities or overwriting the active forecast? We maintain separate sync streams for baseline forecasts versus adjusted forecasts, processing them in sequence with validation between stages. This prevents version conflicts.
On the silent failure issue - implement comprehensive logging and monitoring. We created a sync monitoring dashboard showing: last successful sync timestamp, records processed/failed by batch, average sync duration, error rate trending. Set up automated alerts when error rate exceeds 5% or when sync hasn’t completed within expected timeframe. Also log every batch operation with start/end times, record counts, and success/failure status.
Great question on change detection. We use a hybrid approach combining both methods for reliability. The primary mechanism is timestamp-based - our external demand planning tool maintains a LastModifiedDate on each forecast record. Our sync process tracks the last successful sync timestamp and queries for records where LastModifiedDate > LastSyncTimestamp. However, we also maintain a checksum column calculated from key forecast fields (SKU, Location, ForecastDate, Quantity, Version). This catches cases where the forecast was modified but the timestamp wasn’t updated properly (we’ve seen this happen during bulk updates in the planning tool).
For batch processing with 15,000 SKUs, here’s our detailed approach: First, identify changed SKUs using timestamp and checksum comparison. Group these into batches of 2,000 SKUs, organized by product category to maintain logical data boundaries. Process each batch within explicit transaction boundaries with commit points. If a batch fails, log detailed error information (batch ID, SKU list, error message, timestamp) and continue with next batch rather than failing the entire sync.
For error handling, implement multi-level notifications: Batch-level errors (transaction failures, database connectivity issues) trigger immediate alerts to the integration team. Record-level errors (validation failures, data quality issues) are logged to an error queue table and summarized in a daily report to the planning team. Maintain error history for trending analysis - if specific SKUs fail repeatedly, it indicates a systemic data quality issue that needs business process correction.
For incremental sync specifically, maintain a sync state table tracking: SyncJobID, EntityType (baseline/adjusted forecast), LastSyncTimestamp, RecordsProcessed, RecordsUpdated, RecordsInserted, ErrorCount. This gives you full visibility into sync performance over time. When errors occur, implement retry logic with exponential backoff for transient issues (network timeouts, temporary locks) but don’t retry data validation errors - those need human review.
One critical point on data consistency: Implement a reconciliation process that runs after each sync comparing record counts and aggregate forecast quantities between source and target systems. Any variance exceeding 1% should trigger investigation. This catches silent data loss scenarios where records are skipped without generating errors.
This comprehensive approach should eliminate your version conflicts and silent failures while providing full visibility into sync operations.
Tom’s incremental approach is critical. For validation between stages, we check: forecast quantities are within min/max bounds for each SKU, forecast dates fall within active planning horizon, customer-location combinations exist in master data, and forecast versions are sequential. Any validation failure triggers an alert email to the planning team with details of which SKUs failed and why. Don’t let failures go silent - that’s how you end up with phantom inventory issues weeks later.
For that volume, definitely break into batches. We process demand planning data in batches of 2,000 SKUs, grouped by product category to maintain logical boundaries. Each batch is wrapped in a transaction with explicit error handling. If any SKU in the batch fails validation, we log it to an error table and continue with the next batch. This prevents one bad forecast from blocking 15,000 SKUs. We also implement incremental sync - only SKUs with forecast changes since last sync are processed, reducing batch size by 70-80% on average.