TICKET HX-1402

Workflow runs are taking 4x longer than they used to

Status: Resolved Category: Performance Priority: P2 Customer: Wei Zhang (Aspen Robotics) Agent: Sandra Liu
Wei Zhang Customer 2024-04-14 10:08 UTC

Our 'Daily ops sync' workflow used to run in about 90 seconds. Over the last 2 weeks it's been creeping up — yesterday it took 6 minutes. We didn't change anything. Are there platform issues?

Sandra Liu Helix Support 2024-04-14 10:22 UTC

Hi Wei — no platform issues on our end. Looking at your workflow's run history, I can confirm the slowdown — average duration went from 95s to 380s over 14 days. The increase is concentrated in step 4 (the Salesforce SOQL query), which went from 8s to 290s.

Everything else is fast. Can you tell me about the SOQL query in that step? Is it a query you wrote, and does the size of the result set vary day to day?

Wei Zhang Customer 2024-04-14 10:35 UTC

Yes, custom SOQL. It pulls all opportunities updated in the last day. We did a backfill in late March that updated about 50K opportunities at once. Could that be related?

Sandra Liu Helix Support 2024-04-14 10:48 UTC

Probably yes. If your query is `WHERE LastModifiedDate >= YESTERDAY` and the backfill bumped LastModifiedDate on 50K records all on the same date, your query is now matching all 50K every single day until 24 hours after the backfill rolls past your window. So you have about 14-21 days of slow runs depending on exact timing.

Two options:

  1. Wait it out: the issue self-resolves once your YESTERDAY window moves past the backfill date.
  2. Tighten the query: add a filter that excludes the backfill records, e.g., `AND LastActivityDate >= YESTERDAY` or filter on a specific field that the backfill didn't touch.

Option 2 brings runtime back to baseline immediately.

Wei Zhang Customer 2024-04-14 11:02 UTC

Going with option 2. Updated the SOQL. Test run finished in 84s. Beautiful.

RESOLUTION

Workflow slowdown was caused by a customer-side Salesforce backfill that updated 50K opportunities, causing the workflow's SOQL query (which filtered by LastModifiedDate) to match all 50K records daily for ~21 days. Solution: tightened SOQL with an additional filter that the backfill didn't touch. Runtime returned to ~85s.