fsi-insurance-analytics

Files (5)

• SKILL.md
• SKILL_ja.md
• overview.png · assets/images/
• reference-architecture.png · assets/images/
• architecture.md · references/

SKILL.md (first lines)

# Insurance Workload: Data Analytics Platform

## Workload Overview

This case study describes a reference architecture for an **insurance data
analytics platform** on AWS. It is modeled on the architecture of a data
analytics platform currently operated by real insurance companies, referencing
the data-processing and security requirements typical of insurance workloads.
Published examples of insurers running insurance-workload data analytics
platforms include **New York Life** and **Sun Life**.

**Purpose.** The architecture builds a data platform that realizes the data
processing required for insurance operations efficiently and economically. It
provides an analytics foundation that supports the delivery of diverse services
to agents and policyholders.

**Current challenges (business drivers).** The insurer's existing on-premises
data platform faces insufficient processing capacity and escalating cost due to a
rapid increase in data volume. Functional constraints of the legacy system make it
unable to respond to increasingly diverse user needs. Security boundaries are
ambiguous, making it difficult to satisfy compliance requirements, and failures or
changes at each processing stage risk impacting other systems.

**What the architecture aims to achieve.** Four goals:

- **Cost-efficient scalability** — automatically scale resources up and down in
  response to data-volume fluctuation, realizing a pay-as-you-go model that pays
  only for what is needed when it is needed.
- **High-performance data processing** — provide processing capacity that handles
  high-speed processing of large data volumes and concurrent access from multiple
  users.
- **High availability** — achieve redundancy so business can continue even during
  system failures.
- **Strengthened security and governance** — clarify security boundaries, address
  compliance requirements, strengthen separation of duties and governance, and
  ensure operational independence so that failures or changes at each processing
  stage do not affect others, with appropriate access control and privilege
  management by role (data engineers, analysts, security teams, etc.).

## Best Practices and Key Components

### Best Practices

- SHOULD: Center the data platform on an integrated data lake so that ingestion
  through analytics of policyholder information, health information, and financial
  data for insurance operations is systematized on the cloud.
- SHOULD: Adopt managed services with auto-scaling and a pay-as-you-go model tied
  to data-volume fluctuation to achieve cost-efficient scalability and resolve the
  fixed-cost and over-provisioning problems of on-premises environments.
- SHOULD: Combine a data lake and data warehouse into an integrated processing
  foundation to handle high-speed processing of large data volumes and concurrent
  access from multiple users; run automated ETL/ELT pipelines with parallel
  execution of data transformation.
- SHOULD: Achieve redundancy for business continuity through automatic replication
  of the data lake, multi-node cluster configuration, and automatic job retry.
- MUST: Introduce a multi-account strategy so that ingestion, data lake,
  transformation, data warehouse, and analytics are each executed in separated
  accounts, incorporating PII/PHI tokenization, account isolation, and
  least-privilege access control to satisfy insurance-industry data privacy and
  regulatory requirements.
- SHOULD: Use an Amazon Aurora metadata repository together with an AWS Glue
  framework so that Glue jobs are auto-generated according to the pattern of the
  data source, eliminating point-to-point per-source ingestion solutions and
  enabling efficient ingestion from 100+ heterogeneous sources.
- SHOULD: Use Amazon Redshift Data Sharing to fully separate ETL/ELT processing
  from analytics processing, providing data scientists a serverless cluster with
  safe access to production data while avoiding performance contention.
- MUST: Land data in the landing bucket in exactly the same state as the source
  system and perform no transformation there; run standardization and PII/PHI
  tokenization only in the unified pipeline afterward.
- MUST: Store only changed, standardized, and tokenized data in the Raw bucket of
  the ingestion account before applying changes to the data lake account.
- SHOULD: Grant least-privilege, role-specific permissions — the data-acquisition
  layer gets write-only access to the landing bucket and the processing layer gets
  read-only access — and use trust-policy-based cross-account access to enable
  rapid isolation on compromise.
- MUST: Apply least-privilege IAM role-based cross-account access so that resources
  are accessible only from specific accounts with an established trust relationship.
- SHOULD: Store sensitive information encrypted in AWS Secrets Manager and AWS
  Systems Manager Parameter Store (SecureString), with key management by AWS KMS.
- SHOULD: Audit all API calls with AWS CloudTrail and track resource configuration
  changes with AWS Config.
- MAY: Use the Detect PII transform in an AWS Glue Studio visual ETL job to
  automatically detect personal information (configuring detection patterns,
  thresholds, and sampling rate) and mask sensitive data; combine with AWS Glue
  DataBrew recipes for date-format conversion, and use AWS Glue regex to create
  Japanese-capable sensitive-data detection entities.
- MAY: Build a generic ingestion framework for structured and unstructured data
  from sources such as Salesforce, Adobe, and SharePoint.
- SHOULD: Migrate eligible tables from Parquet to Apache Iceberg where it yields
  benefits (identified for 1,200–1,400 of roughly 4,000 tables).
- WILL: Converting about 500 tables to Iceberg achieved a 25–40% performance
  improvement for update and write processing.
- MAY: Defer building a full disaster-recovery (DR) environment for analytics and
  reporting workloads when several hours of downtime is acceptable, relying on S3
  durability and Multi-AZ placement, and implement DR incrementally later as
  business requirements change.

### Key Components

#### Data Lake

- **Component Overview**: The central store of the lake house. Ingests data from
  roughly 100 heterogeneous sources and provides 99.999999999% (11 nines)
  durability and virtually unlimited scalability, suited to the 30–40 year
  long-term data retention required in insurance. The data lake account is the
  first layer where end users consume data, maintaining 30–40 years of
  policyholder history as Type 2 (history-retaining) slowly changing dimensions.
- **Assumed AWS Services**: Amazon S3, AWS Lake Formation.

#### Metadata Repository and Ingestion Framework

- **Component Overview**: A unified, metadata-driven ingestion framework. The main
  metadata repository stores source patterns; loading metadata auto-generates
  Glue jobs, eliminating point-to-point ingestion and enabling efficient ingestion
  from 100+ sources (SFTP, DMS, CDC, message-based, etc.).
- **Assumed AWS Services**: Amazon Aurora, AWS Glue, AWS Lambda.

#### ETL/ELT and Transformation

From aws-samples/sample-fsi-reference-architecture-jp, MIT-0.

More from aws-samples/sample-fsi-reference-architecture-jp