Reliability regression
shell failures increased
4.6% now vs 3.1% previously across 371 calls; Timeout is the leading code.
Inspect recent failures, then check the tool handler and upstream dependency logs.
InvestigateBidlo Status Frontend
Chat tool telemetry
Find slow, failing, and repetitive agent tools without exposing prompts, arguments, or outputs.
Detailed production timing active
6101 of 6166 saved tool calls include execution duration.
Tool calls
371
+94% vs previous 7d52 saved production turns
p50 latency
1.1 s
+152% vs previous 7d362 timed calls
p95 latency
26 s
+2% vs previous 7d362 timed calls
Tool failures
4.6%
+1.4 pts vs previous 7d17 excluding NoResults
No result
0.0%
No change vs previous 7d0 product outcomes
Calls / turn
7.1
+27% vs previous 7d98.1% of turns at 5+ calls
What changed
Material regressions ranked by severity and affected call volume.
Agent loop pressure
More turns are using five or more tools
98.1% now vs 88.2% previously (51 turns).
Inspect the affected turn sequences for repeated or avoidable tool selection.
InvestigateRegression checks require at least 3 calls in both windows. Latency is flagged at +20%; failure and no-result rates are flagged at +1 and +3 points respectively.
Execution volume
Successful calls, no-results, and failures combined.
Calls
Oct 1Oct 4Oct 7
Payload pressure
How often results exceed the 20 KB inline cap.
5.0%
handle rows / all calls
309 stored handles in this period. This is a payload-size proxy, not a one-to-one call classification.
Tool comparison
Each change is measured against the previous equal window; latency appears only where detailed timing exists.
| Tool | Calls | p50 | p95 | Failures | No result |
|---|---|---|---|---|---|
| shell | 371+94% vs prior | 1.1 s+152% vs prior | 26 s+2% vs prior | 4.6%+1.4 pts vs prior | 0.0%No change vs prior |
Failure codes
Operational failures only; NoResults is excluded.
Timeout17
Interpretation notes
The data has known edges that change what a chart means.
- Blocked loops are invisible
- Rate-limited and identical-argument calls are rejected before telemetry fires.
- NoResults is not downtime
- It remains a failed tool outcome, but is separated from runtime reliability here.
- Preview volume lives elsewhere
- Preview and threadless chats reach PostHog but are intentionally absent from Postgres.
- Latency is partially instrumented
- Saved tool parts provide production outcomes and volume; duration appears only on calls with detailed telemetry.
- Partial turns are retained
- They remain in volume counts but are excluded from failures and latency percentiles.
- Comparisons use equal windows
- Every delta compares the selected period with the immediately preceding period under the same tool and team filters.
Recent sanitized calls
Arguments, outputs, user IDs, and message content are never rendered.
shellHJC2.3 sOct 6, 6:08 PM UTC
Outcome
Success
Thread
50ac87df…
Tool call
call_VUY…
Turn sequence
11 of 28
Turn state
Completed
shellHJC2.2 sOct 6, 6:08 PM UTC
Outcome
Success
Thread
50ac87df…
Tool call
call_57v…
Turn sequence
13 of 28
Turn state
Completed
shellHJC1.9 sOct 6, 6:08 PM UTC
Outcome
Success
Thread
50ac87df…
Tool call
call_eV3…
Turn sequence
14 of 28
Turn state
Completed
shellHJC4.9 sOct 6, 6:08 PM UTC
Outcome
Success
Thread
50ac87df…
Tool call
call_vBr…
Turn sequence
15 of 28
Turn state
Completed
shellHJC6.6 sOct 6, 6:09 PM UTC
Outcome
Success
Thread
50ac87df…
Tool call
call_dOm…
Turn sequence
16 of 28
Turn state
Completed
shellHJC3.0 sOct 6, 6:09 PM UTC
Outcome
Success
Thread
50ac87df…
Tool call
call_LXw…
Turn sequence
18 of 28
Turn state
Completed
shellHJC1.5 sOct 6, 6:09 PM UTC
Outcome
Success
Thread
50ac87df…
Tool call
call_lAx…
Turn sequence
20 of 28
Turn state
Completed
shellHJC6.0 sOct 6, 6:09 PM UTC
Outcome
Success
Thread
50ac87df…
Tool call
call_OXI…
Turn sequence
21 of 28
Turn state
Completed
shellHJC524 msOct 6, 6:09 PM UTC
Outcome
Success
Thread
50ac87df…
Tool call
call_tw5…
Turn sequence
23 of 28
Turn state
Completed
shellHJC2.7 sOct 6, 6:10 PM UTC
Outcome
Success
Thread
50ac87df…
Tool call
call_h99…
Turn sequence
24 of 28
Turn state
Completed
shellTRP 22.6 sOct 6, 1:59 PM UTC
Outcome
Success
Thread
09a6f0d4…
Tool call
call_QM9…
Turn sequence
20 of 58
Turn state
Completed
shellTRP 22.3 sOct 6, 1:59 PM UTC
Outcome
Success
Thread
09a6f0d4…
Tool call
call_ZJR…
Turn sequence
26 of 58
Turn state
Completed
shellTRP 210 sOct 6, 1:59 PM UTC
Outcome
Success
Thread
09a6f0d4…
Tool call
call_TB3…
Turn sequence
28 of 58
Turn state
Completed
shellTRP 2196 msOct 6, 2:00 PM UTC
Outcome
Success
Thread
09a6f0d4…
Tool call
call_MJF…
Turn sequence
30 of 58
Turn state
Completed
shellTRP 2158 msOct 6, 2:00 PM UTC
Outcome
Success
Thread
09a6f0d4…
Tool call
call_ZY9…
Turn sequence
31 of 58
Turn state
Completed
shellTRP 24.5 sOct 6, 2:00 PM UTC
Outcome
Success
Thread
09a6f0d4…
Tool call
call_VLp…
Turn sequence
33 of 58
Turn state
Completed
shellTRP 2140 msOct 6, 2:00 PM UTC
Outcome
Success
Thread
09a6f0d4…
Tool call
call_5cd…
Turn sequence
34 of 58
Turn state
Completed
shellTRP 2140 msOct 6, 2:00 PM UTC
Outcome
Success
Thread
09a6f0d4…
Tool call
call_ZKh…
Turn sequence
35 of 58
Turn state
Completed
shellTRP 2340 msOct 6, 2:01 PM UTC
Outcome
Success
Thread
09a6f0d4…
Tool call
call_djk…
Turn sequence
36 of 58
Turn state
Completed
shellTRP 2132 msOct 6, 2:01 PM UTC
Outcome
Success
Thread
09a6f0d4…
Tool call
call_zBp…
Turn sequence
37 of 58
Turn state
Completed
shellTRP 2107 msOct 6, 2:01 PM UTC
Outcome
Success
Thread
09a6f0d4…
Tool call
call_qi0…
Turn sequence
38 of 58
Turn state
Completed
shellTRP 21.4 sOct 6, 2:01 PM UTC
Outcome
Success
Thread
09a6f0d4…
Tool call
call_Rbp…
Turn sequence
39 of 58
Turn state
Completed
shellTRP 215 sOct 6, 2:02 PM UTC
Outcome
Success
Thread
09a6f0d4…
Tool call
call_hoT…
Turn sequence
44 of 58
Turn state
Completed
shellTRP 27.6 sOct 6, 2:03 PM UTC
Outcome
Success
Thread
09a6f0d4…
Tool call
call_Yru…
Turn sequence
45 of 58
Turn state
Completed