Aws - Click Event Rest API (Capture user behavior)

Card Puncher Data Processing

About

Adapted from the Aws tutorial - Building a Modern app (module 5).

Architecture

Click Event records will be processed in real-time by a serverless code function, aggregated, and stored for any future analysis that you may want to perform.

Aws User Click Event Processing Architecture

Steps

Create the code and save it to a repository

aws codecommit create-repository --repository-name MythicalMysfitsStreamingService-Repository
{
    "repositoryMetadata": {
        "accountId": "REPLACE_ME_ACCOUNT_ID",
        "repositoryId": "32941d10-66e1-4230-b3d6-eeb55e64a60b",
        "repositoryName": "MythicalMysfitsStreamingService-Repository",
        "lastModifiedDate": 1554965738.429,
        "creationDate": 1554965738.429,
        "cloneUrlHttp": "https://git-codecommit.eu-central-1.amazonaws.com/v1/repos/MythicalMysfitsStreamingService-Repository",
        "cloneUrlSsh": "ssh://git-codecommit.eu-central-1.amazonaws.com/v1/repos/MythicalMysfitsStreamingService-Repository",
        "Arn": "arn:aws:codecommit:eu-central-1:REPLACE_ME_ACCOUNT_ID:MythicalMysfitsStreamingService-Repository"
    }
}

git clone https://git-codecommit.eu-central-1.amazonaws.com/v1/repos/MythicalMysfitsStreamingService-Repository
Cloning into 'MythicalMysfitsStreamingService-Repository'...
warning: You appear to have cloned an empty repository.

  • Copy the source
cd MythicalMysfitsStreamingService-Repository
cp ..\aws-modern-application-workshop\module-5\cfn .
cp ..\aws-modern-application-workshop\module-5\app\streaming .
:: A CFN template for creating the full stack.
..\aws-modern-application-workshop\module-5\cfn\real-time-streaming.yml
        1 file(s) copied.
:: A Python script that contains the code for our Lambda function: streamProcessor.py
..\aws-modern-application-workshop\module-5\app\streaming\streamProcessor.py
        1 file(s) copied.

  • package all of library dependencies together with the Lambda code function
:: Install the request package into the local directory
pip install requests -t .
Collecting requests
  Using cached https://files.pythonhosted.org/packages/7d/e3/20f3d364d6c8e5d2353c72a67778eb189176f08e873c9900e10c0287b84b/requests-2.21.0-py2.py3-none-any.whl
Collecting chardet<3.1.0,>=3.0.2 (from requests)
  Using cached https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl
Collecting certifi>=2017.4.17 (from requests)
  Downloading https://files.pythonhosted.org/packages/60/75/f692a584e85b7eaba0e03827b3d51f45f571c2e793dd731e598828d380aa/certifi-2019.3.9-py2.py3-none-any.whl (158kB)
    100% |                                        | 163kB 163kB/s
Collecting urllib3<1.25,>=1.21.1 (from requests)
  Using cached https://files.pythonhosted.org/packages/62/00/ee1d7de624db8ba7090d1226aebefab96a2c71cd5cfa7629d6ad3f61b79e/urllib3-1.24.1-py2.py3-none-any.whl
Collecting idna<2.9,>=2.5 (from requests)
  Using cached https://files.pythonhosted.org/packages/14/2c/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl
Installing collected packages: chardet, certifi, urllib3, idna, requests
Successfully installed certifi-2019.3.9 chardet-3.0.4 idna-2.8 requests-2.21.0 urllib3-1.24.1

  • List
ls -A1
.git
bin
certifi
certifi-2019.3.9.dist-info
chardet
chardet-3.0.4.dist-info
idna
idna-2.8.dist-info
real-time-streaming.yml
requests
requests-2.21.0.dist-info
streamProcessor.py
urllib3
urllib3-1.24.1.dist-info

  • Replace REPLACE_ME_API_ENDPOINT in the Lambda Function Code streamProcessor.py by the ApiEndpoint of the Mysfits service API (this is the same service ApiEndpoint that you created for the website frontend).
def retrieveMysfit(mysfitId):
    apiEndpoint = 'REPLACE_ME_API_ENDPOINT' + '/mysfits/' + str(
        mysfitId)  # eg: 'https://ljqomqjzbf.execute-api.us-east-1.amazonaws.com/prod/'
    mysfit = requests.get(apiEndpoint).json()
    return mysfit
  • Package
git add .
git commit -m "New stream processing service."
git push

Create the streaming service stack with SAM

The creation of the streaming stack is done using the AWS SAM CLI to:

  • package all of our function code,
  • upload it to S3,
  • and create the deployable CloudFormation template

Create a S3 bucket

Create a bucket named real-time-streaming

aws s3 mb s3://real-time-streaming

Package the code with SAM

With the Sam cli

cd aws-modern-application-workshop\module-5
sam package ^
  --template-file .\cfn\real-time-streaming.yml ^
  --output-template-file .\cfn\transformed-streaming.yml ^
  --s3-bucket real-time-streaming
Uploading to 31ae6f83bb35657cc2aad3d168246ada  2629 / 2629.0  (100.00%)
Successfully packaged artifacts and wrote output template to file .\cfn\transformed-streaming.yml.
Execute the following command to deploy the packaged template
aws cloudformation deploy --template-file d:\code-external\aws-modern-application-workshop\module-5\cfn\transformed-streaming.yml --stack-name <YOUR STACK NAME>

  • The CodeUri parameter of the serverless Lambda function has been updated in the transformed-streaming.yml file with the object location where the SAM CLI has uploaded your packaged code
.........
MysfitsClicksProcessor:
    Type: AWS::Serverless::Function
    Properties:
      Handler: streamProcessor.processRecord
      Runtime: python3.6
      CodeUri: s3://real-time-streaming/31ae6f83bb35657cc2aad3d168246ada
      ..........

S3 Lambda Sam Uploaded Code

Deploy The Stack Using AWS CloudFormation

Returned by the SAM CLI command is the CloudFormation command needed to be executed to create our new full stack. But because our stack creates IAM resources, you'll need to add one additional parameter to the command.

  • deploy the streaming stack (deploy the function)
aws cloudformation deploy ^
  --template-file .\cfn\transformed-streaming.yml ^
  --stack-name MythicalMysfitsStreamingStack ^
  --capabilities CAPABILITY_IAM

Once this stack creation is complete, the full real-time processing microservice will be created.

Documentation / Reference

Support

AccessDenied when calling the CreateChangeSet operation

An error occurred (AccessDenied) when calling the CreateChangeSet operation: User: arn:aws:iam::09000999845148:user/cli is not authorized to perform: cloudformation:CreateChangeSet on resource: arn:aws:cloudformation:eu-central-1:094473333148:stack/MythicalMysfitsStreamingStack/*





Discover More
Aws Moder App Mysfits Home Page
Aws - Modern Web App Workshop

Note the Aws project Build a Modern Web Application Code: aws-samples/aws-modern-application-workshop/tree/python...



Share this page:
Follow us:
Task Runner