One line per request, per proxy
With access logging on, every proxy writes one line per request it handled. A call from curl to httpbin in the mesh is logged twice: by the client sidecar (outbound) and by the server sidecar (inbound).
What you need to know already: the previous lessons of this part, the ingress-nginx access log (18.19) for comparison, HTTP status codes (11.21).
[2026-10-07T10:12:01.400Z] "GET /status/503 HTTP/1.1" 503 URX via_upstream - "-" 0 0 54 1 "-" "curl/8.16.0" "963585ef-092e-cf6b-b305-fb8ddaf44fa0" "httpbin:8000" "10.244.2.229:8080" outbound|8000||httpbin.demo.svc.cluster.local 10.244.2.192:45011 10.96.204.113:8000 10.244.2.192:37740 - default
Istio's default format, field by field:
[2026-10-07T10:12:01.400Z] START_TIME
"GET /status/503 HTTP/1.1" method, path, protocol
503 RESPONSE_CODE sent to the downstream (the caller)
URX RESPONSE_FLAGS: why, in Envoy's words ("-" = nothing special)
via_upstream RESPONSE_CODE_DETAILS: who made the code (the upstream answered)
- CONNECTION_TERMINATION_DETAILS
"-" UPSTREAM_TRANSPORT_FAILURE_REASON (TLS errors, connect errors)
0 0 BYTES_RECEIVED, BYTES_SENT
54 DURATION (ms, the whole request incl. retries)
1 X-ENVOY-UPSTREAM-SERVICE-TIME (ms the last upstream try took)
"-" "curl/8.16.0" X-Forwarded-For, User-Agent
"963585ef-..." X-REQUEST-ID: the same id on every hop - grep for it
"httpbin:8000" authority (Host)
"10.244.2.229:8080" UPSTREAM_HOST: the pod that was tried last
outbound|8000||httpbin.demo.svc... UPSTREAM_CLUSTER: direction|port|subset|host
10.244.2.192:45011 UPSTREAM_LOCAL_ADDRESS
10.96.204.113:8000 DOWNSTREAM_LOCAL_ADDRESS (what the app dialled: the ClusterIP)
10.244.2.192:37740 DOWNSTREAM_REMOTE_ADDRESS (the caller)
- REQUESTED_SERVER_NAME (SNI: outbound_.8000_._.httpbin... on inbound mTLS)
default ROUTE_NAME (the VirtualService route's name, or default)
Duration 54 ms for a 1 ms upstream answer: two retries with 25 ms backoff. URX says the retries ran out. Default retries, visible in one field.
Response flags
UH no healthy upstream: no endpoints, all ejected, or a subset matching no pod 503 "no healthy upstream"
UF upstream connection failure: connect refused / timed out 503 "upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused"
UO upstream overflow: a circuit breaker (connectionPool) said no 503
NR no route: no VirtualService rule matched, or (inbound) no filter chain 404 / reset
URX retries (or connect attempts) exhausted - usually with UF or after 5xx
UT upstream request timeout: the route's timeout (or perTryTimeout) expired 504 "upstream request timeout"
UC upstream connection termination: the server closed it (mTLS mismatch...) 503
DC downstream connection termination: the CALLER went away (its timeout)
DI a fault delay was injected
FI a fault abort was injected "fault filter abort"
NC no cluster: the route points at a host or subset istiod never sent 503
RL rate limited locally 429
The pairing that saves hours: a 503 from the app has flag "-" and via_upstream; a 503 the proxy made itself has a flag. Look at the flag before you look at the app.
A UF's details carry both reasons, spaces written as _: upstream_reset_before_response_started{remote_connection_failure|delayed_connect_error:_Connection_refused} (the reset reason, then after the | the transport failure reason). Which proxy logs it depends on who failed to connect. With no sidecar on the server, the client's own connect is refused: the client logs UF (and URX once its retries ran out). With sidecars on both ends, the client's connection lands on the server's sidecar, which then fails to reach the app: the UF is in the server proxy's log (cluster inbound|PORT||), and the client logs that 503 as URX via_upstream. Read both logs, matched by the x-request-id.
What one proxy was told: proxy-config
Logs show what happened; istioctl proxy-config (pc) shows the configuration that made it happen, for one proxy:
istioctl pc clusters deploy/web -n shop --fqdn reviews SERVICE FQDN, PORT, SUBSET, DIRECTION, TYPE, DESTINATION RULE
istioctl pc endpoints deploy/web -n shop --cluster "outbound|9080|v2|reviews.shop.svc.cluster.local"
ENDPOINT, STATUS, OUTLIER CHECK, CLUSTER
istioctl pc routes deploy/web -n shop --name 9080 NAME, VHOST NAME, DOMAINS, MATCH, VIRTUAL SERVICE
istioctl pc listeners deploy/web -n shop --port 9080 ADDRESSES, PORT, MATCH, DESTINATION
istioctl pc secret deploy/web -n shop the workload certificate
The cluster name format outbound|9080|v2|reviews.shop.svc.cluster.local appears everywhere: logs, endpoints, stats. A subset in a VirtualService that no DestinationRule defines has no cluster in pc clusters; a subset whose labels match nothing has a cluster with no endpoints.
Envoy's own counters
The admin API listens on localhost:15000 inside the pod. The proxy image has no curl, so use pilot-agent:
# in a mesh where web calls a flaky reviews service (an illustration, no ▶)
$ kubectl exec deploy/web -c istio-proxy -- pilot-agent request GET stats | grep 'reviews.*upstream_rq_retry'
cluster.outbound|9080||reviews.shop.svc.cluster.local.upstream_rq_retry: 112
upstream_rq_retry, upstream_rq_pending_overflow (circuit breaker), upstream_rq_timeout, outlier_detection.ejections_active - each one a behaviour from the previous lesson, counted. Port 15020 serves the merged metrics of the app and the proxy at /stats/prometheus (istio_requests_total with source, destination, response code, flags and connection_security_policy).
What you can now do:
- read every field of an Istio access log line and follow a request id across hops
- name the cause behind UH, UF, UO, NR, URX, UT, UC, DC, DI, FI and NC
- use proxy-config and Envoy stats to see what one proxy was told and did