TICKET HX-1189

Getting 401 errors with what should be a valid API key

Status: Resolved Category: API Priority: P1 Customer: Jakob Andersen (Skylane Aviation) Agent: Devon Park
Jakob Andersen Customer 2024-03-22 08:11 UTC

My production integration started failing at 3am UTC with 401 unauthorized. Key hasn't been rotated. Nothing changed on our side. We're partly down.

Devon Park Helix Support 2024-03-22 08:19 UTC

Hi Jakob — looking into this immediately. What's the prefix of the key (just the first 12 chars, don't share the full key)?

Jakob Andersen Customer 2024-03-22 08:21 UTC

hlx_live_8a3f4...

Devon Park Helix Support 2024-03-22 08:32 UTC

Found it. That key was deleted at 02:58 UTC by user erin@skylane.com. The deletion is in your audit log. Once a key is deleted it can't be undeleted; you'll need to generate a new one.

Jakob Andersen Customer 2024-03-22 08:35 UTC

WHAT. That's our production key. Erin had no business deleting that. Can you restore it?

Devon Park Helix Support 2024-03-22 08:42 UTC

I understand the frustration. We genuinely cannot restore deleted keys — that's a security guarantee, not a tooling limitation. The fastest path back to working: (1) generate a new key in Settings → API Keys, (2) deploy it to your production integration. The integration should be back up within minutes of the new key being installed.

Going forward, two things I'd suggest: (1) set up scoped keys so production uses a key that's narrower than full-access — limits blast radius if it happens again. (2) Restrict who can delete API keys via a role policy. Owner/Admin-only is our recommendation.

Jakob Andersen Customer 2024-03-22 09:14 UTC

New key deployed. Production back up. Will follow up with Erin. Closing this out — thanks for the fast triage.

RESOLUTION

Customer's API key returning 401 because it had been deleted by another user in their workspace at 02:58 UTC. Audit log confirmed the deletion. Deleted keys are unrecoverable by design. Resolved by customer generating a new key. Recommended scoped keys and restricting deletion permissions to Admin/Owner only.