Building an External Scheduling Engine for D365 Project Operations | Part 6: Making It Fast

Making it fast: why the plugin cascade deadlocks, and the pattern that replaces it

Everything in Parts 1 to 5 can be implemented as Dataverse plugins, and the first version usually is. It works beautifully on a demo project with forty tasks. Then a planner moves the status date on a program with a few hundred unfinished tasks, waits, and gets a timeout. Or worse, gets a deadlock error naming two tasks they never touched.

This post is about that wall: what actually causes it, the two changes that remove it, and the architecture we ended up with. None of it is exotic. Most of it is Microsoft’s own guidance, applied.

1) The shape of the problem

Follow one status-date change through a naive implementation:

1. The program’s status date moves forward. A plugin on the program finds every unfinished task that now starts in the past (Part 3) and updates each one.
2. Each task update fires the task plugin, which recalculates that task’s fields, and, because dates changed, runs the critical path for the whole project and writes early, late and slack values to every task in it.
3. Each of those writes fires the task plugin again.
4. Each task change also has to reach the resource assignments (contours, Part 2), roll dates up to the project, and from the project to the program.

One user action, synchronous, inside a single transaction, with a two-minute ceiling on the whole thing. Cost scales with the number of pushable tasks multiplied by the length of the cascade behind each one. It does not scale with program size, which is why the demo never shows it: the demo program has no unfinished work in the past.

2) Anatomy of the deadlock

The timeout is annoying. The deadlock is the one that tells you the design is wrong.

Status date moves
  └─ Program plugin: batch-update Task A, Task B, Task C
       ├─ Task A update → task plugin → project-wide CPM
       │     └─ writes A, B, C           (locks B and C)
       └─ Task B update → task plugin → project-wide CPM
             └─ writes A, B, C           (locks A ... waits for B)

A's CPM waits for B. B's CPM waits for A. SQL picks a victim.

The root cause is a per-record plugin doing project-wide writes. Two records in the same batch each try to rewrite the whole project. There is no ordering that makes this safe, and adding retries only makes it slower.

3) Fix one: separate the task’s math from the project’s math

The per-task plugin has two jobs mixed together. Splitting them removes the deadlock outright.

The program plugin becomes two phases: batch the date writes (each one now cheap and lock-free), then call the critical path calculation directly, once for each affected project, on committed data.

Two details make this hold up:

• Tag your own writes. Every write the engine makes carries a marker (in Dataverse, a request parameter that surfaces as a shared variable in the child pipeline). The task-level trigger checks for the marker and exits. This is how the critical path run’s own writes do not re-trigger the critical path.
• Depth is not a substitute for tags. It is tempting to write “if pipeline depth > 1, exit” and call it loop protection. It does stop loops. It also stops your engine from ever running on anything created by another process, which is everything that arrives from an integration. That story is Part 7. Keep a depth ceiling as a runaway brake, well above anything legitimate, and use tags for the real check.

With this change the deadlocks are gone and the sweep is faster. It is still synchronous, and on a large program it still runs into the timeout. That needs the second fix.

4) Fix two: stop persisting inside the plugin

Microsoft’s guidance for Dataverse plugins is explicit: they are for validation and light logic, and bulk writes belong outside the platform, in something like an Azure Function. So the second change is to make the plugin stop writing tasks at all.

The plugin still validates the input and runs the full calculation, so a bad change is still rejected on the spot with the user watching. But instead of committing hundreds of task rows, it writes one record: the set of pending changes for the project. Then it returns. The save completes in the time it takes to calculate, which is the part that was always fast.

Four parts, and only one of them is new infrastructure:

5) What the user sees in the gap

Asynchronous persistence creates a window, usually well under a minute, where the stored schedule and the displayed schedule differ. Design for it explicitly or it will be reported as a bug.

• Lock on In progress, not on Pending. A pending row is a queued intent; the persist has not started and a second change can safely merge into it. Once the persist is running, the scheduling fields on the project and task forms go read-only with a notice, and the Gantt refuses drags. Server-side, the orchestrator can reject writes for that project with a clear message.
• Time out the lock. If a row stays In progress for several multiples of the timer interval, the function died. Do not leave the project locked forever; surface it as Failed and let the next change retry.
• Refresh is not automatic. A form save does not re-render an open Gantt, and a program-level change does not refresh the project Gantts underneath it. Either push a notification to open clients or tell the user plainly that the view needs refreshing. Silence here is what generates “the status date didn’t do anything” tickets.
• Show pending state. A small indicator on the Gantt that pending changes exist, and when they were persisted, removes most of the confusion on its own.

6) Loop safety in the new design

Moving the writes out of the plugin changes the loop-safety argument, so restate it:

1. The orchestrator’s own writes are tagged; the trigger recognises the tag and exits.
2. Nothing is registered on the pending-change table, so writing it cannot re-enter the engine.
3. The persist function writes tasks with the same tag, so the task-level trigger exits for those too.
4. The persist is idempotent: re-running a Completed row produces no changes. Verify this directly; a persist that computes zero deltas on its second run is the convergence proof.
5. A depth ceiling remains as a hard brake, at a level that no legitimate cascade reaches.

7) Measuring it honestly

Two timers, measured separately, because they have different causes and different fixes:

• Timer A: save click to save confirmation. Server time, plugin time, and after fix two, calculation time only.
• Timer B: refresh to rendered Gantt. Client load and render, plus the API merge.

And a test set that actually exercises the cascade. Demo data does not. What does:

• A program of realistic size. We use one of roughly 250 projects with 20 tasks each, about 5,000 tasks and 4,500 dependencies.
• Unfinished tasks scheduled in the past, so the status-date sweep has something to push. This is the variable that drives cost, not task count.
• Some of those tasks carrying constraints, because the constraint adjustment is an extra write per task.
• Forward and backward status-date moves measured separately. A backward move should not rewrite dates at all; if it costs the same as a forward move, something is doing unnecessary work.
• Plugin trace logs on, so you can see tag exits, depth exits and per-step durations rather than inferring them.

8) What it changes

For the planner: a status-date change on a large program comes back in seconds, the Gantt shows the result at once, and there is a visible processing state instead of a spinner followed by an error. For the platform: no deadlocks, no timeouts, and task writes that happen in controlled batches from one place.

For the engineering team, the more important change is that the calculation and the persistence are now separate components with separate tests. The scheduling math can be unit tested against an in-memory project. The persist can be tested against a pending row. Neither needs the other running.

Next in the series: work that arrives from outside. Why integration-created projects silently never get scheduled, what a due date and a duration have to become before the engine can place anything, and how to seed a schedule for an order that came across with no dates at all.