fsi-capital-market-data

Files (71)

• SKILL.md
• SKILL_ja.md
• cdk-nag-test.template.ts · assets/
• marketdata-arch.png · assets/images/
• SECURITY_NOTICE.md · assets/sample-cdk/
• bleafsi-guest-market-data-sample.ts · assets/sample-cdk/bin/
• parameter.ts · assets/sample-cdk/bin/
• cdk.json · assets/sample-cdk/
• jest.config.js · assets/sample-cdk/
• bleafsi-guest-market-data-stack.ts · assets/sample-cdk/lib/
• build-container.ts · assets/sample-cdk/lib/
• container-app.ts · assets/sample-cdk/lib/
• container-distributor-app.ts · assets/sample-cdk/lib/
• dynamodb.ts · assets/sample-cdk/lib/
• monitor-alarm.ts · assets/sample-cdk/lib/
• package-lock.json · assets/sample-cdk/
• package.json · assets/sample-cdk/
• Dockerfile · assets/sample-cdk/sample-ecs-app/
• buildspec.yml · assets/sample-cdk/sample-ecs-app/
• bleafsi-guest-market-data.test.ts · assets/sample-cdk/test/
• tsconfig.json · assets/sample-cdk/
• architecture.md · references/
• cdk-nag.md · references/
• deployment.md · references/
• fisc-mapping.md · references/
• .gitignore · resources/bleafsi-shared-constructs/v1.0.0/
• cdk.json · resources/bleafsi-shared-constructs/v1.0.0/
• Bucket.png · resources/bleafsi-shared-constructs/v1.0.0/doc-image/
• PrivateVpc.png · resources/bleafsi-shared-constructs/v1.0.0/doc-image/
• PublicVpc.png · resources/bleafsi-shared-constructs/v1.0.0/doc-image/
• doc-readme.md · resources/bleafsi-shared-constructs/v1.0.0/
• .nojekyll · resources/bleafsi-shared-constructs/v1.0.0/doc/
• README.md · resources/bleafsi-shared-constructs/v1.0.0/doc/
• Bucket.md · resources/bleafsi-shared-constructs/v1.0.0/doc/classes/
• Ecr.md · resources/bleafsi-shared-constructs/v1.0.0/doc/classes/
• IamRole.md · resources/bleafsi-shared-constructs/v1.0.0/doc/classes/
• KmsKey.md · resources/bleafsi-shared-constructs/v1.0.0/doc/classes/
• PrivateVpc.md · resources/bleafsi-shared-constructs/v1.0.0/doc/classes/
• PublicVpc.md · resources/bleafsi-shared-constructs/v1.0.0/doc/classes/
• Waf.md · resources/bleafsi-shared-constructs/v1.0.0/doc/classes/
• VpcEndpointTypeName.md · resources/bleafsi-shared-constructs/v1.0.0/doc/enums/
• BucketProps.md · resources/bleafsi-shared-constructs/v1.0.0/doc/interfaces/
• EcrProps.md · resources/bleafsi-shared-constructs/v1.0.0/doc/interfaces/
• IamRoleProps.md · resources/bleafsi-shared-constructs/v1.0.0/doc/interfaces/
• KmsKeyProps.md · resources/bleafsi-shared-constructs/v1.0.0/doc/interfaces/
• VpcProps.md · resources/bleafsi-shared-constructs/v1.0.0/doc/interfaces/
• WafProps.md · resources/bleafsi-shared-constructs/v1.0.0/doc/interfaces/
• Bucket.png · resources/bleafsi-shared-constructs/v1.0.0/doc/media/
• PrivateVpc.png · resources/bleafsi-shared-constructs/v1.0.0/doc/media/
• PublicVpc.png · resources/bleafsi-shared-constructs/v1.0.0/doc/media/
• modules.md · resources/bleafsi-shared-constructs/v1.0.0/doc/
• bleafsi-ecr.ts · resources/bleafsi-shared-constructs/v1.0.0/lib/
• bleafsi-iam-role.ts · resources/bleafsi-shared-constructs/v1.0.0/lib/
• bleafsi-kms-key.ts · resources/bleafsi-shared-constructs/v1.0.0/lib/
• bleafsi-s3-bucket.ts · resources/bleafsi-shared-constructs/v1.0.0/lib/
• bleafsi-vpc.ts · resources/bleafsi-shared-constructs/v1.0.0/lib/
• bleafsi-waf.ts · resources/bleafsi-shared-constructs/v1.0.0/lib/
• index.ts · resources/bleafsi-shared-constructs/v1.0.0/lib/
• package-lock.json · resources/bleafsi-shared-constructs/v1.0.0/
• package.json · resources/bleafsi-shared-constructs/v1.0.0/
• …and 11 more

SKILL.md (first lines)

# Market Data Reference Architecture (BLEA for FSI)

## Workload Overview

A financial institution's market data distribution system generates a wide
variety of market-related data and distributes it to internal and external
clients. The distributed data is typically calculated from data collected from
internal and external data providers. Communication with clients and providers
uses protocols such as FIX, REST, and WebSocket.

Market data is classified into three broad patterns based on how frequently the
data is generated and how fresh it must be by the time it reaches the client:

- **Real-time** — stock prices, FX rates, etc.; more than several thousand
  updates per minute; often latency-sensitive.
- **Near real-time** — market news, non-real-time price distribution, etc.;
  updated once every few minutes.
- **Batch** — interest-rate indices, corporate credit ratings, foreign-exchange
  published rates, etc.; updated once every several hours or more.

This reference architecture targets the **real-time** use case.

The architecture is built on top of the BLEA for FSI governance base (the shared
governance/security account foundation) and is deployed into a guest account.

## Best Practices and Key Components

### Best Practices

- MUST: Compose the system of three application tiers — Handler App, Composer
  App, and Distributor App — each with a distinct responsibility.
- MUST: Have the Handler App write received input data into Amazon Kinesis Data
  Streams as a producer; the protocol and authentication method depend on the
  interface defined by the provider.
- MUST: Have the Composer App calculate the distributed data by consuming the
  Streams provided by the Handler App (or by other Composers) as a consumer.
- MUST: Have the Distributor App consume the Stream provided by the Composer App
  and perform format conversion, distribution-frequency and granularity
  adjustment according to client requirements.
- SHOULD: Make the Distributor App provide the most recent known-good value as a
  fixed value when an upstream Composer App or its Stream becomes unavailable due
  to a failure.
- MUST: Route FIX consumers through an NLB to the Distributor App and perform
  authentication during the FIX Logon process.
- MUST: Route REST and WebSocket consumers through an ALB to the Distributor App;
  the ALB validates the HTTP Authorization header and routes between the
  authentication Lambda and the Distributor App.
- SHOULD: Select the network connectivity to providers and clients (Direct
  Connect, Internet, PrivateLink, etc.) according to requirements.
- MUST: Deploy each container application in the same configuration across
  Multiple AZs and run them Active-Active; present two active endpoints to
  clients and guide failover on failure.
- SHOULD: Adopt Streams for inter-component integration to keep components
  loosely coupled, so producers (Handler, Composer) need not be aware of the
  existence or location of downstream components and the number of consumers can
  vary easily.
- MAY: Extend this configuration by adding a new consumer (for example a new
  database for analytics) as an additional member of the consumer set.
- MUST: Encrypt data with AWS KMS and set appropriate access permissions on the
  encryption key; set appropriate IAM policies on the AWS resources that store
  data to prevent data leakage (FISC 実 3 / 実 30 / 実 69).
- MUST: Access each resource using temporary credentials obtained through IAM
  roles, following the principle of least privilege (FISC 実 9 / 実 25).
- MUST: Restrict communication protocols at the connection endpoints (Amazon ELB,
  Amazon API Gateway, etc.) to prevent unauthorized intrusion from external
  networks, and perform peer authentication on the application side (FISC 実 14 /
  実 15).
- SHOULD: Verify that container images and libraries are trusted at build time
  and run vulnerability scanning of containers in Amazon ECR (FISC 実 20 / 実 21
  / 実 32).
- SHOULD: Monitor compute resources, network bandwidth, ALB, Aurora, DynamoDB and
  Amazon ECS resources with Amazon CloudWatch, and enable VPC Flow Logs and ALB
  access logs (FISC 実 46 / 実 72).
- WILL: The Amazon Kinesis Data Stream stores data in physically isolated
  locations and retains data for up to 24 hours by default (up to 7 days maximum).
- SHOULD: Build a multi-region / multi-AZ configuration and use highly available
  AWS managed services for availability (FISC 実 85–実 88).
- SHOULD: For large-scale failures beyond AZ resilience, decide whether to wait
  for service recovery or switch to a secondary region, accounting for the time
  and cost of switching connection targets and creating resources on both the
  producer and consumer sides (FISC 実 71 / 実 73 / 実 74).
- MUST: Apply cdk-nag's AwsSolutionsChecks (via Aspects) to every CDK stack built from this skill, and gate CI (jest etc.) on zero unsuppressed AwsSolutions-* findings; suppress exceptions only with an explicit reason via NagSuppressions (see references/cdk-nag.md).

### Key Components

#### Handler App

- **Component Overview**: Receives the input that becomes the source of the
  distributed data. The protocol and authentication method depend on the
  interface defined by the provider. It writes the received data into a Kinesis
  Data Stream as a producer. In the CDK sample it is realized as an ECS on
  Fargate service (`sample_handler` container) plus a dedicated Kinesis Data
  Stream.
- **Assumed AWS Services**: Amazon ECS (AWS Fargate), Amazon Kinesis Data
  Streams, Amazon ECR.

#### Composer App

- **Component Overview**: Calculates the distributed data. In most cases the
  distributed data is calculated from upstream Market Data Provider data or
  internal reference data. It consumes the Stream provided by the Handler App as
  input, and may also consume Streams provided by other Composers (for example,
  Composer C computing GBPJPY may consume the Streams of Composer A computing
  GBPUSD and Composer B computing USDJPY). In the CDK sample it is an ECS on
  Fargate service (`sample_composer` container) with its own Kinesis Data Stream,
  and computed results can be stored in DynamoDB.
- **Assumed AWS Services**: Amazon ECS (AWS Fargate), Amazon Kinesis Data
  Streams, Amazon DynamoDB, Amazon ECR.

#### Distributor App

- **Component Overview**: Distributes data toward clients by consuming the Stream
  provided by the Composer App. It performs format conversion and adjusts
  distribution frequency and granularity per client requirements, and provides
  the most recent known-good value as a fixed value when the upstream Composer or
  its Stream is unavailable. FIX consumers connect via an NLB; REST/WebSocket
  consumers connect via an ALB that validates the HTTP Authorization header. In
  the CDK sample it is an ECS on Fargate service (`sample_distributor` container)
  fronted by an ALB with AWS WAF and ALB access logging enabled.

From aws-samples/sample-fsi-reference-architecture-jp, MIT-0.

More from aws-samples/sample-fsi-reference-architecture-jp