aidp-object-storage

Files (2)

• LICENSE
• SKILL.md

SKILL.md (first lines)

# `aidp-object-storage` — OCI Object Storage native (`oci://`)

Read and write Object Storage data directly from Spark. The AIDP cluster's IAM identity is used automatically — no `OCI_CONFIG`, no API keys, no inline PEM.

## When to use
- Land or read CSV / Parquet / JSON / Avro / Delta files in an OCI bucket from Spark.
- Register an **External Volume** (`/Volumes/...`) backed by an OCI bucket.
- Define an **External Table** (`USING CSV/PARQUET/...`) over an `oci://` path.
- Mentioned: "oci://", "Object Storage bucket", "external volume", "external table".

## When NOT to use
- For **Iceberg** tables on OCI Object Storage → use [`aidp-iceberg`](../aidp-iceberg/SKILL.md).
- For **AWS S3** → use [`aidp-aws-s3`](../aidp-aws-s3/SKILL.md).
- For **Azure ADLS Gen2** → use [`aidp-azure-adls`](../aidp-azure-adls/SKILL.md).

## URI form
```
oci://<bucket>@<namespace>/<path>
```
The namespace is the tenancy's Object Storage namespace (OCI Console > Object Storage > Bucket Details).

## Direct read/write
```python
oci_path = "oci://my-bucket@mynamespace/folder/file"

# Write
df.write.mode("overwrite").option("header", True).format("csv").save(oci_path)

# Read
df_read = spark.read.option("header", True).format("csv").load(oci_path)
df_read.show()
```

Same pattern with `format("parquet")`, `format("json")`, `format("delta")`.

## External Volume (DDL)
Mount a bucket once, reference by Volume path forever:

```sql
CREATE EXTERNAL VOLUME IF NOT EXISTS default.default.ext_volume
LOCATION 'oci://my-bucket@mynamespace/';
```
Then:
```python
volume_path = "/Volumes/default/default/ext_volume/folder/file"
df.write.format("csv").option("header", True).save(volume_path)
spark.read.option("header", True).format("csv").load(volume_path).show()
```
Drop with `DROP VOLUME default.default.ext_volume`.

## External Table (DDL)
Register a table whose data lives in `oci://`:

```sql
CREATE TABLE IF NOT EXISTS default.default.ext_table (name STRING, age INT)
USING CSV
OPTIONS (path='oci://my-bucket@mynamespace/folder/file', delimiter=',', header='true');
```
Query like any Spark table:
```python
spark.sql("SELECT * FROM default.default.ext_table").show()
```
Drop with `DROP TABLE default.default.ext_table`.

## Gotchas
- **Auth is implicit** — the AIDP cluster's IAM identity is used. The user never types OCI keys. If reads fail with 404/403, the workspace identity lacks bucket privileges; fix in OCI IAM.
- **Namespace ≠ tenancy name.** The Object Storage namespace is a separate, immutable string. Find it in `OCI Console > Profile > Tenancy: <tenancy_name>` — the `object_storage_namespace` field.
- **External volume path is `/Volumes/<catalog>/<schema>/<volume>/...`**, NOT `oci://...`. Once a volume is registered, address files via the Volume path.
- **External table `path` option uses `oci://` directly**, not the Volume path. Both work; choose based on whether you want a re-mountable abstraction (Volume) or a simple direct reference (Table).
- **`/Workspace/...` is NOT for data.** It's a FUSE-mounted file system intended for notebooks/configs. For data files use `oci://` or `/Volumes/...`.

## References
- Official sample: [oracle-samples/oracle-aidp-samples → `getting-started/Access_Object_Storage_Data.ipynb`](https://github.com/oracle-samples/oracle-aidp-samples/blob/main/getting-started/Access_Object_Storage_Data.ipynb)

From Kilo-Org/kilo-marketplace, Apache-2.0.

More from Kilo-Org/kilo-marketplace