application-integration-troubleshooting

Files (6)

• SKILL.md
• common-errors.md · references/
• connectors-health.md · references/
• execution-states.md · references/
• logging-and-monitoring.md · references/
• suspensions.md · references/

SKILL.md (first lines)

# Application Integration Troubleshooting

Diagnose and recover Google Cloud Application Integration executions that fail, stall, or appear to do nothing. This skill walks through execution state verification, per-task failure inspection, Cloud Logging and Cloud Monitoring checks, Integration Connectors health probes, suspension resolution, and IAM or quota checks.

## Attribution

**MUST READ**: Before executing any `gcloud` command, REST API call, or Python
discovery-client call, you MUST read
[Usage Attribution](references/attribution.md). For commands executed directly
in the terminal (never in generated scripts, Terraform, ADK agents, or user
code), prefix `gcloud` with
`CLOUDSDK_METRICS_REQUEST_ATTRIBUTION="agent-skills/1.0 (skill:application-integration-troubleshooting)"`
and add `agent-skills/1.0 (skill:application-integration-troubleshooting)` to
the `User-Agent` header of `curl` requests.

## Tooling and Scope Boundaries

- **REST is the interface.** There is no `gcloud integrations` command group and no Google-managed MCP server for Application Integration. Use `curl` with `gcloud auth print-access-token`, or the discovery-based Python client (`googleapiclient.discovery.build("integrations", "v1")` and `build("connectors", "v1")`; install with `pip install "google-api-python-client>=2.100.0,<3" "google-auth>=2.20.0,<3"`) (changes host or credential state; get the user's approval first).
- **Generic `gcloud` commands apply** for authentication (`gcloud auth`), enabling APIs (`gcloud services`), IAM checks (`gcloud projects get-iam-policy`, `gcloud iam service-accounts get-iam-policy`), and log queries (`gcloud logging read`).
- **Regional scope.** Integrations, executions, connections, and execution logs live in a single Google Cloud region (`LOCATION`), which is the `LOCATION` in every REST path; no `gcloud` default-region setting applies. Ask the user which region the integration is deployed in. If they don't know, start with `us-central1` and, if `executions.list` is empty there, check the other [Application Integration locations](https://docs.cloud.google.com/application-integration/docs/locations.md.txt) before concluding nothing ran (Step 2).
- **Sibling skills:**
  - Use `application-integration-basics` for product concepts, enabling APIs, provisioning a region, initial IAM roles, and service limits.
  - Use `application-integration-authoring` for creating drafts, editing tasks and triggers, config variables, publishing, `:execute` and `:schedule` payloads, and CI/CD export/import.

## Safety and Confirmation Rules

- **Read-only checks run freely:** `executions.list`, `suspensions.list`, `connections.get`, `connectionSchemaMetadata` get calls, runtime v2 `actions.list` / `entityTypes.list`, `gcloud logging read`, and `v2` `executions:troubleshoot`.
- **State-changing actions require explicit user confirmation.** Before canceling or replaying an execution, lifting or resolving a suspension, refreshing connector schema metadata, repairing connector eventing, publishing or unpublishing a version, or changing IAM bindings, show the exact command and wait for user approval.
- **Host changes require approval.** Before installing packages (`pip install`) or changing global gcloud or credential state (`gcloud config set ...` such as `auth/impersonate_service_account`, `gcloud auth application-default login` / `set-quota-project`), show the exact command and wait for the user's approval.
- **Never bundle destructive scripts** or delete integrations, versions, connections, or auth profiles during troubleshooting unless the user specifically asks.

## Prerequisites and Setup Check

Before querying APIs, confirm the environment has credentials, active APIs, and the right target region:

```bash
# Verify active identity and project
gcloud config get-value account
gcloud config get-value project

# Check that required APIs are enabled on PROJECT_ID
gcloud services list --enabled --project="PROJECT_ID" \
  --filter="config.name:(integrations.googleapis.com OR connectors.googleapis.com OR secretmanager.googleapis.com)"
```

- If an API is missing, ask the user before running `gcloud services enable integrations.googleapis.com connectors.googleapis.com secretmanager.googleapis.com --project="PROJECT_ID"`.
- Ensure an active billing account is attached to `PROJECT_ID` (required even when usage stays within the monthly free tier; see the [Application Integration pricing page](https://cloud.google.com/application-integration/pricing)).
- For Python discovery clients (install with `pip install "google-api-python-client>=2.100.0,<3" "google-auth>=2.20.0,<3"`) or ADK (`ApplicationIntegrationToolset`), authenticate with `gcloud auth application-default login` and `gcloud auth application-default set-quota-project PROJECT_ID` (changes host or credential state; get the user's approval first).

## Diagnostic Decision Flow

Follow these steps in order whenever a user reports that an integration failed, did nothing, or is stuck.

### Step 1: Confirm Whether an Execution Ran and Check Its True State

Do not rely on an HTTP 200 status from `:execute` as proof that every task succeeded (note: `integrations:test` also exists in the v1 REST API, but its required `TestIntegrationsRequest.clientId` value is not publicly documented, so do not invent one; test by publishing to a non-production project and calling `:execute`). As a diagnostic rule, always check the response payload and execution record because:
- The required verification is the execution record from `executions.list?filter=execution_id="EXECUTION_ID"`—inspect `executionDetails.state`, each task's `taskExecutionState`, and `ErrorInfo` in `responseParameters` (do not rely on the legacy/deprecated `executionFailed` field, which is not a supported public signal).
- If the caller passed `"doNotPropagateError": true` on `:execute`, the API returns HTTP 200 with an `executionId` and places error details inside event parameters instead of returning an HTTP error.
- A duplicate `requestId` on `:execute` or `:schedule` returns the previous execution's response without running a new execution.
- Task failure policies (`IGNORE` and `NONE`) allow an integration to continue or finish as `SUCCEEDED` down an alternate branch even when an individual task failed.

List executions for the integration (use `-` as `INTEGRATION_NAME` to search across all integrations in the region; the discovery doc does not specify whether `orderBy=create_time` sorts ascending or descending, so add a `CreateTimestamp` filter or paginate with `pageToken` if needed):

```bash
curl -sS -H "Authorization: Bearer $(gcloud auth print-access-token)" \
  "https://integrations.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION/integrations/INTEGRATION_NAME/executions?pageSize=10&orderBy=create_time&snapshotMetadataWithoutParams=true"
```

Once you have the `EXECUTION_ID`, fetch its full `ExecutionDetails` and per-task states (use `filter`, `snapshotMetadataWithoutParams=true`, and `readMask` on `executions.list`—never the deprecated `truncateParams` [UI-only] or `filterParams.*` [internal-only] parameters):

```bash
curl -sS -H "Authorization: Bearer $(gcloud auth print-access-token)" \
  "https://integrations.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION/integrations/INTEGRATION_NAME/executions?filter=execution_id%3D%22EXECUTION_ID%22"
```

To filter `executions.list` by state, use the `state` field (`=` or `!=`) with unquoted `ExecutionDetails.state` values (for example, `filter=state=FAILED`, `filter=state!=SUCCEEDED`, `filter=state=CANCELLED`, or `filter=state=FAILED OR state=CANCELLED`, URL-encoded as `filter=state%3DFAILED` or `filter=state%3DFAILED%20OR%20state%3DCANCELLED`); use `state` (not `eventExecutionState`, as shown on the REST reference page, or `event_execution_state` in the API reference's filter description), and use the supported filter keys (`execution_id`, `state`, `update_time`, `workflow_name`, `trigger_id`, `CreateTimestamp`, `parameter_type`)—camelCase variants such as `executionId` are not supported even though some docs examples show them.

### Step 2: What If No Execution Record Exists?

If `executions.list` returns no matching run, check these five causes before assuming the trigger never fired:

| Cause | Symptom | What to verify |
|---|---|---|
| **Wrong region** | Empty list in `LOCATION` | Check `https://console.cloud.google.com/integrations` or list integrations in other regions. |
| **API trigger 403** | Caller got HTTP 403 `PERMISSION_DENIED` | API trigger auth denials fail before execution creation; nothing is written to execution logs. Check caller IAM and `roles/iam.serviceAccountUser` on the run-as service account. |
| **API trigger payload > 8 MB** | Caller got an error | Cumulative API trigger input variables exceed the 8 MB limit, so the integration does not start. |
| **Local logging disabled** | Integration ran, but logs are empty | Check `IntegrationVersion.databasePersistencePolicy` (`DATABASE_PERSISTENCE_DISABLED` = `Disable`; `DATABASE_PERSISTENCE_POLICY_UNSPECIFIED` = `Enable (sync)`, the API default; `DATABASE_PERSISTENCE_ASYNC` = `Enable (async)`, the console default). Also, if total integration data exceeds 10 MB (up to the 30 MB limit), local logging is disabled automatically. |
| **Version not `ACTIVE`** | Trigger never fires | Only an `ACTIVE` (published) version runs from triggers. After publishing, an older version can keep serving for up to 10 minutes due to eventual consistency. |

### Step 3: Branch by Execution State (`ExecutionDetails.state`)

See [Execution States](references/execution-states.md) for the full state reference, task-level states (`TaskExecutionDetails.taskExecutionState`), filter syntax, and cancel/replay rules.

| `ExecutionDetails.state` | Console status | Meaning and next step |
|---|---|---|
| `FAILED` | Failed | Check `ErrorInfo` (`message`, `code`) in `responseParameters`, inspect the failing task's `taskExecutionState` (`FAILED` or `FATAL`), and run Step 4 (`:troubleshoot` and Cloud Logging). |
| `SUCCEEDED` | Succeeded | If expected work did not happen, inspect `taskExecutionDetails` for tasks in `SKIPPED` (edge condition false) or `FAILED` (masked by `IGNORE` or `NONE` retry strategy). |
| `CANCELLED` | Canceled | User canceled the run, a new integration version was published while the run was suspended, a SYNC run hit the 2-minute timeout, or a replayed run hit the 10-minute deadline. |
| `SUSPENDED` | Suspended | Waiting on an Approval task (resolvable in the Cloud console UI or via `suspensions:resolve`) or a Suspend task (resumable via `suspensions:lift`). See [Suspensions](references/suspensions.md). |
| `RETRY_ON_HOLD` | Retry on hold | A task or integration failed with a temporary error and is waiting in the backoff retry queue. |
| `PROCESSING` | In process | Currently running. Check whether an ASYNC run is approaching the 10-minute timeout. |
| `PENDING` | On hold | Scheduled or temporarily paused and waiting to be triggered. |

### Step 4: Inspect Logs, AI Root Cause, and Subsystem Health

1. **Run AI Troubleshoot (v2 REST API)** on a failed execution to get a structured root-cause summary (requires `integrations.executions.get`):
   ```bash
   curl -sS -H "Authorization: Bearer $(gcloud auth print-access-token)" \
     "https://integrations.googleapis.com/v2/projects/PROJECT_ID/locations/LOCATION/integrations/INTEGRATION_NAME/executions/EXECUTION_ID:troubleshoot"
   ```
2. **Query Cloud Logging** if Cloud Logging was enabled on the version. Note that Cloud Logging is **disabled by default** and writes nothing if local logging is disabled. See [Logging and Monitoring](references/logging-and-monitoring.md) for how to enable it, severity behavior, and exact queries.
   ```bash
   gcloud logging read \
     'resource.type="integrations.googleapis.com/IntegrationVersion" AND jsonPayload.executionId="EXECUTION_ID"' \
     --project="PROJECT_ID" --limit=20 --format=json
   ```
3. **If a Connectors task failed**, do not guess whether the connector or IAM is at fault. Inspect the connection resource (`connections.get`) and its schema metadata (`GET .../connections/CONNECTION_NAME/connectionSchemaMetadata` for `state` and `errorMessage`, and runtime v2 `GET .../v2/.../connections/CONNECTION_NAME/actions` and `.../entityTypes` for listing—v1 `connectionSchemaMetadata:listActions` and `:listEntityTypes` require a `filter` and reject unfiltered calls). See [Connectors Health](references/connectors-health.md) for connection states, schema metadata checks, and node throttling (HTTP 429).
   ```bash
   curl -sS -H "Authorization: Bearer $(gcloud auth print-access-token)" \

From google/skills, Apache-2.0.

More from google/skills