hiapi /v1/tasks Job Hangs or Times Out: Diagnosis and Retry Playbook

Why async jobs hang and time out

Async jobs on /v1/tasks are a contract between three parties: your client, the HiAPI platform, and whatever compute actually runs the work. A hang is almost always one of those layers waiting on another, and the first job of debugging is to work out which one — because the fix for each is different, and a blind retry can make things worse.

Here is the lifecycle you should hold in your head:

The lifecycle, stage by stage

1. Submit — your client sends the request and the platform accepts it, returning a job identifier.
2. Enqueue — the job is scheduled. This is the stage most likely to be slow under load and the most invisible to you.
3. Run — compute processes the job. Duration here depends entirely on the workload, not on your HTTP client.
4. Complete — a terminal state is recorded and becomes readable via the same job identifier.

A "hang" is rarely a single event. It is usually one of these:

• The job is genuinely still running. Long inputs, large outputs, or a busy queue all extend stage 3. Nothing is broken; you are just impatient.
• The job is queued but not started. Stage 2 can stretch, and from your side it looks identical to a hang.
• Your client gave up too early. You set an HTTP timeout on the status check rather than on the job, so your process aborted while the platform was fine.
• The job reached a terminal state you did not detect. Your polling loop stopped, crashed, or exited before it saw the answer.
• A genuine platform-side failure. Rarer, but real, and the only case where a platform-level retry is the right instinct.

The core mistake

The most common self-inflicted timeout is conflating two clocks. Your HTTP request to check status should be short — it is just a read. The job has its own, much longer deadline. If you use one timeout value for both, you will cancel healthy jobs and then retry them, duplicating work that was about to finish.

Before you touch anything, decide which of the five situations above you are actually in. Everything else in this playbook depends on that answer.

First moves: gather the job evidence

Resist the urge to change code or fire a retry in the first minute. A hang is a state you can only debug after the fact if you recorded enough about it. Capture the following before you intervene.

What to record for every job

• The job identifier returned at submission. This is the key to everything; without it you are guessing.
• The status string you last observed, verbatim, not paraphrased.
• Timestamps: when you submitted, when each status check fired, and when each returned.
• The number of status checks you performed and their intervals.
• The request payload shape — not necessarily the full body, but enough to reproduce: input size, parameters, and anything that affects runtime.
• Your client-side deadline, if you set one, and whether it fired.

Re-check the same identifier once, manually

Before concluding anything, issue a single fresh status check for the same job identifier. Two outcomes:

• The status advanced. The job is alive; you had a polling or timeout bug, not a job bug. Go to the polling section.
• The status is unchanged and the job is old. Now you have a real candidate for a stuck job. Note how long it has been in that state relative to your expectations.

Write it down

Keep a small log line per attempt: identifier, status, elapsed time, decision taken. Hangs that recur are almost always diagnosed from history, not from the live incident. If you cannot reconstruct what a job did an hour ago, you will keep re-debugging the same failure.

One more piece of evidence that is easy to forget: the time of day and the workload type. If hangs cluster on a particular input class, you have found a pattern worth reporting rather than a transient hiccup worth retrying.

Reading job status correctly

Status is the source of truth for every retry decision. Treat it as a small state machine and map each state to exactly one action, so you never retry something that is still alive.

The states you will see, and what each means

• Queued / pending — accepted, not started. A long time here means scheduling pressure. Do not retry; the job already exists.
• Running / processing — compute is working. Nothing is wrong. Waiting is the correct action.
• Succeeded / completed — terminal. Read the result and be done.
• Failed / error — terminal, and the error payload is your best debugging material. Read it before deciding anything.
• Cancelled / expired — terminal by policy or by your own action. Retrying is a fresh decision.
• Unknown or missing — the identifier is not recognized. This is not a failure state; it is an evidence state. Reread the identifier and check that you are looking at the right one.

The two questions to ask of any status

1. Is this terminal? If not, the only correct move is to keep waiting within your budget.
2. If terminal, is it retryable? A success is not. A failure may or may not be — read the error. An unknown identifier needs investigation, not a retry.

Do not infer state from elapsed time

"It has been running for ten minutes, so it must be stuck" is a guess, not a diagnosis. Elapsed time tells you when to check, never what the state is. A job can sit in a queue for a long stretch and then complete perfectly. Let the status string draw the conclusion; your clock only decides how often you ask.

Designing a sane polling cadence

Most "hangs" are polling bugs wearing a disguise. Two failure modes dominate: polling so aggressively you degrade your own reliability, and polling so naively you never notice a terminal state. A bounded cadence fixes both.

Start slow, back off gently

Do not poll in a tight loop. Begin with a short interval so fast jobs resolve quickly, then increase it as the job ages:

import time

def poll(job_id, check_status, deadline_s=600):
    start = time.monotonic()
    interval = 1.0
    while True:
        status = check_status(job_id)
        if status in ("succeeded", "failed", "cancelled", "expired"):
            return status
        if time.monotonic() - start > deadline_s:
            return "client_deadline_exceeded"
        time.sleep(interval)
        interval = min(interval * 1.5, 30.0)  # cap the ceiling

The important properties: an initial interval short enough to catch quick jobs, a growth factor that keeps long jobs cheap, and a cap so you never end up sleeping for an hour and missing the result.

Always set a client-side deadline

Never assume the platform will signal completion for you. Set your own deadline — the maximum time you are willing to wait — and treat exceeding it as a distinct outcome, not as a failure. That distinction matters: a client_deadline_exceeded result means "I stopped looking," not "the job died." Record it as such.

Two practical rules

• Bound the total number of checks. A loop with no maximum iteration count is a hang waiting to happen. Compute it from your deadline and interval ceiling.
• Separate read timeouts from job deadlines. The HTTP call to fetch status should have a short timeout of its own. If a status read times out, retry the read — never the job.

Respect the endpoint

Polling is cheap but not free. If you are checking hundreds of jobs, stagger them and reuse a single client. When in doubt, slower is almost always the safer direction: a job worth waiting six minutes for is worth waiting six minutes plus thirty seconds for.

Deciding between retry and re-submit

This is the decision where correctness and cost collide. Retrying the same job and submitting a new one are not interchangeable, and picking wrong either duplicates work or strands a healthy job.

The decision rule

Work through these in order:

1. Is the job terminal? If no, do not retry and do not re-submit. Wait. A retry on a live job duplicates work.
2. If terminal, what does the status say? A success means you are done. A failure carries an error payload — read it first.
3. Is the failure plausibly transient? Timeouts under load, brief unavailability, rate limiting — these are retry candidates.
4. Is the failure plausibly permanent? Invalid input, unsupported parameters, malformed payload — these will fail identically no matter how many times you try. Fix the request.

Retry the same job when…

The job exists and is terminal-failed for a reason that might not recur, and the platform's semantics for that job allow another attempt. You reuse the identifier, you keep one line of history, and you avoid creating a second job that does the same thing.

Re-submit as a new job when…

• The identifier is unknown or no longer retrievable — there is nothing left to retry.
• The original job has definitively expired by policy, so a retry has nothing to attach to.
• You need to change the input. A retry is the same request; a correction is a new request. Never try to make a retry mean something different from the original.

The cost asymmetry

A retry that duplicates a live job doubles the work done for one result. A re-submit is a knowingly fresh job — a new identifier, a new lifecycle, and a new bit of spend. Both are legitimate tools; both should be exercised deliberately. The rule of thumb: retry on uncertainty about infrastructure, re-submit on certainty about input.

Making retries safe

A retry that quietly duplicates work is worse than a hang, because it fails silently and costs you. Build the safety in before you need it.

Think in idempotency terms

Ask, for every job type you run: if this runs twice, is the outcome still correct? If the answer is no — say the job appends a row or triggers a downstream side effect — then your retry logic must be gated on status, never on a timer. Retry only terminal failures, and only after a fresh status check confirms that state.

Cap the attempts

Every retry loop needs a maximum attempt count and a maximum total elapsed time, both checked explicitly. An unbounded retry loop is just a slower hang with extra billing. When the budget is exhausted, surface the failure — do not silently swallow it.

Add backoff between attempts

Retry attempts should themselves be spaced out, growing with each failure. An instant retry on a failing dependency usually fails again and adds load to whatever is already struggling.

Log the attempt, not just the error

For every attempt record: the job identifier, the status at the moment of decision, the attempt number, the interval used, and the outcome. This is the difference between "it hangs sometimes" and "it hangs on Tuesday afternoons with large inputs." The second is fixable; the first is not.

Never mask the real failure

If a job fails three times and your wrapper returns a generic error, you have thrown away the error payloads that explained why. Preserve the last status and error detail all the way up to your caller. Retries are a recovery mechanism, not a concealment mechanism — they should give the underlying problem fewer chances to surface, not fewer chances to be seen.

A postmortem checklist for recurring hangs

When a hang happens once, fix the immediate job. When it happens twice, run this checklist — otherwise you are just paying for retries forever.

The checklist

• Classify the incident. Was it a live job you stopped watching, a genuinely stuck job, a client timeout, or a status you failed to read? Naming it is half the fix.
• Check the polling math. Did your interval cap, deadline, and attempt cap actually apply? Was any loop unbounded?
• Confirm the timeout separation. Was your status-read timeout ever confused with your job deadline?
• Review the retry decisions. Did any retry fire on a non-terminal job? Did any re-submit duplicate a live job?
• Inspect the failure payloads. Were errors preserved, or swallowed by the wrapper?
• Look for input patterns. Do failures cluster by payload size, parameter, or workload type?
• Re-examine expectations. Is the job taking longer than your deadline because the work is genuinely that big? If so, the deadline is the bug.

Turn findings into defaults

Every item above should convert into a default in your client: a bounded poller, a distinct deadline outcome, a retry policy driven by status, and structured logs keyed on the job identifier. Do that, and the next hang becomes a line in a log rather than an afternoon.

Async jobs are a contract, and the contract holds as long as you respect its states. Separate still running from never finishing, wait with a bounded cadence, and retry only what is safe to retry — and a hang stays an inconvenience instead of an outage.