=======================

From a wide range of AWS Partner success stories to new partner programs launched at AWS re:Invent, here’s a look at the most popular APN Blog posts we shared in 2021.

Our goal with this blog has always been to share timely and relevant news, technical solutions, partner stories, and more from Amazon Web Services (AWS) and the AWS Partner Network (APN) specifically.

The APN is a global community of partners that leverages programs, expertise, and resources to build, market, and sell customer offerings. Together, partners and AWS can provide innovative solutions, solve technical challenges, win deals, and deliver value to our mutual customers.

For customers, the APN helps you identify companies with deep expertise on AWS that can deliver on core business objectives. The robust AWS Partner community features over 100,000 partners from more than 150 countries. This vibrant, diverse network of partners can help you drive greater value for your business, increase agility, and lower costs.

Thank you for reading the APN Blog, and Happy New Year to all our AWS Partners and customers!

These were the most viewed APN Blog posts published in 2021 that focused on AWS Partner successes and how-tos:

1. Deploying IBM Mainframe z/OS on AWS with IBM ZD&T
2. Mainframe Data Migration to AWS with Infosys (guest post by Infosys)
3. Palantir Foundry Helps You Build and Deploy AI-Powered Decision-Making Applications (guest post by Palantir)
4. Amazon S3 Malware Scanning Using Trend Micro Cloud One and AWS Security Hub (guest post by Trend Micro)
5. Design Considerations for Disaster Recovery with VMware Cloud on AWS
6. Taming Machine Learning on AWS with MLOps: A Reference Architecture (guest post by Reply)
7. How to Get Logs from Amazon S3 Using Filebeat and Metricbeat in Elastic Stack (guest post by Elastic)
8. Implementing Multi-Factor Authentication in React Using Auth0 and AWS Amplify
9. Amazon AppFlow for Bi-Directional Sync Between Salesforce / Amazon RDS for PostgreSQL (guest post by Trantor)
10. Data Tokenization with Amazon Redshift and Protegrity (guest post by Protegrity)

Read all of our AWS Partner success stories >>

These posts were the most viewed APN Blog posts in 2021 about our AWS partner programs:

1. 2021 Japan APN Ambassadors / 2021 APN Top Engineers
2. Architecting Successful SaaS: Understanding Cloud-Based Software-as-a-Service Models
3. Building a Multi-Tenant SaaS Solution Using Amazon EKS
4. Announcing the AWS Partner Network’s Regional 2021 Partners of the Year
5. AWS Partner Paths: A New Way to Accelerate Engagement with AWS
6. AWS Mainframe Migration Competency Featuring Validated AWS Partners
7. Introducing New AWS Solution Provider Program Incentives for AWS Partners
8. AWS Control Tower Best Practices for AWS Solution Providers
9. Getting Out of Your Own Way: How to Avoid Common SaaS Pitfalls
10. Introducing the AWS Energy Competency Partners

Read all of our APN program posts >>

These posts that we have published since the APN Blog’s inception were the most viewed in 2021:

1. The 5 Pillars of the AWS Well-Architected Framework
2. Terraform: Beyond the Basics with AWS
3. Using Terraform to Manage AWS Programmable Infrastructures (guest post by Cloudsoft)
4. Now You Can Take the AWS Certified Cloud Practitioner Exam at Your Home or Office 24/7
5. AWS Lambda Custom Runtime for PHP: A Practical Example (guest post by Rackspace)
6. Making Application Failover Seamless by Failing Over Your Private Virtual IP Across Availability Zones
7. Connecting AWS and Salesforce Enables Enterprises to Do More with Customer Data
8. How to Get AWS Certified: Tips from a DevOps Engineer (guest post by SQUADEX)
9. Using GitLab CI/CD Pipeline to Deploy AWS SAM Applications
10. Getting Started with Ansible and Dynamic Amazon EC2 Inventory Management

Bookmark the APN Blog for the latest updates >>

Each month on the APN Blog, we highlight AWS Partners that received new designations for our global AWS Competency, AWS Managed Service Provider (MSP), AWS Service Delivery, and AWS Service Ready programs.

These designations span workload, solution, and industry, and help AWS customers identify top AWS Partners that can deliver on core business objectives.

See the running list of new AWS Partners >>

Follow the AWS Partner Network (APN) on social media for all the latest updates:

=======================

By Priya Bains, Sr. Product Manager – AWS Partner Network
By Christine Linthacum, Principal Product Manager – AWS Partner Network

We are excited to introduce new AWS Partner Paths, a flexible way to accelerate engagement with Amazon Web Services (AWS) starting January 28, 2022.

Partners have shared with us that they need more ways to work with AWS faster as their business evolves to meet customer needs—Partner Paths are a response to this evolution.

This framework provides a curated journey through partner resources, benefits, and programs. Partner Paths replace technology and consulting partner type models—evolving to an offering type model.

Last year, we launched the first ISV Partner Path. The new Paths support more customer offerings and provide partners flexibility in all the ways they engage with AWS. We now offer five Partner Paths—Software Path, Hardware Path, Training Path, Distribution Path, and Services Path—which represents consulting, professional, managed, or value-add resale services.

Partners can enroll in one or multiple Paths based on their unique customer offerings.

As part of this announcement, we are also renaming the ISV Path to Software Path to better reflect offering types, rather than partner types.

AWS Partner Paths simplify the engagement model for partners, expand partner access to benefits, and provide a more intuitive and streamlined experience.

AWS Partner Paths are purpose-built to support and strengthen a range of partners’ customer offerings. We provide organizations that register with the AWS Partner Network (APN) access to a dedicated partner portal, training discounts, business and technical enablement content, programs, and benefits relevant to their offerings.

As AWS Partners progress along each Path, they can leverage enablement resources, unlock funding benefits, and tap into a broad set of programs to innovate, expand, and differentiate their customer offerings.

There are no tier requirements to validate software and hardware offerings with AWS. Partners can earn a qualified software and/or hardware badge, and directly access several AWS resources specifically designed for the Software or Hardware Paths.

For organizations engaged with the Services and Training Path—tiers continue to remain important to customers to understand the depth of their experience.

For example, a partner may enroll in the Software Path and utilize the AWS SaaS Factory program to help optimize their solution and complete a Foundational Technical Review (FTR) to achieve the “AWS Qualified Software” badge for their offering, while also enrolling in the Services Path to achieve partner tier recognition for their AWS-aligned professional capabilities and managed services experience.

We also have a dedicated Partner Path for distribution, an invitation-only Path to help Authorized Distributors engage faster with AWS.

In support of this new framework, we have streamlined the AWS Partner Central experience to empower partners with the right tools that make it easier to self-serve.

This new user experience—another response to partner feedback—will help AWS Partners navigate through enablement resources, structured content, benefits, and programs to help them better showcase customer expertise.

As with everything we do at AWS, customer obsession is a core part of our business. We’re obsessed with giving our partners the flexibility and agility to thrive in the business areas of their choice, and we look forward to rolling out these Partner Paths in early 2022.

Learn more about the new AWS Partner Paths.

Join the AWS Partner Network (APN) to build, market, and sell with AWS.

=======================

By James Devine, VP Product Management – Aviatrix
By Jacob Cherkas, Principal Architect – Aviatrix
By Mandar Alankar, Sr. Networking Solutions Architect – AWS

Aviatrix

It can be difficult to gain security insights into your cloud infrastructure, especially as architectures grow to encompass multiple availability zones, regions, and clouds.

Aviatrix Systems is an AWS Partner with Competency designations in both Networking and Security. Aviatrix is uniquely positioned to provide deep insights into network traffic.

This includes security insights that can augment Amazon Web Services (AWS) native security capabilities, such as those found by Amazon GuardDuty, for example.

In a previous blog post, we discussed the advanced visibility and troubleshooting capabilities that Aviatrix brings to AWS customers. In this post, we’re excited to detail new capabilities that were recently added to the Aviatrix Secure Network Platform—ThreatIQ and ThreatGuard.

Aviatrix ThreatIQ is a capability that detects malicious traffic based on IP address reputation and known bad actors. It serves as an intrusion detection system (IDS) to alert on malicious traffic throughout your cloud architecture.

ThreatIQ uses a threat feed of known malicious IP addresses that is continuously updated. This ensures you always have an up-to-date detection capability.

Since Aviatrix Gateways export data flow summaries as NetFlow, it’s possible to have a view of threats across an entire AWS infrastructure as well as external connections and clouds.

Alerts are aggregated from all Aviatrix Gateways within and across regions, and even other clouds. This capability embeds detection inside the data plane, rather than limiting threat visibility to the edge of the network.

All traffic is analyzed in real-time as it goes across the Aviatrix Platform and becomes visible in the Aviatrix CoPilot web console.

Configuring an overly permissive security group on an Amazon Elastic Compute Cloud (Amazon EC2) instance quickly shows the power of ThreatIQ. Aviatrix has done just that and left the instance running; as you can see below, a holistic view of all the threats that were detected.

Figure 1 – ThreatIQ dashboard.

You can also view a line-by-line summary of detected threats. This allows you to drill down into individual findings to get more details, including access to raw NetFlow data and topology information.

Figure 2 – ThreatIQ threat details.

Each of the threats can be further examined to see exactly where in the environment each finding was detected.

Such a view is quite difficult to get from traditional tools that don’t have an end-to-end topology view and understanding of your cloud network. This provides an invaluable tool for incident response to allow security teams to pinpoint the exact source of a malicious traffic flow. Root cause analysis becomes a breeze.

Figure 3 – ThreatIQ threat details.

The capability doesn’t stop there. You can double-click on the Aviatrix Gateway to drill all the way down to the individual instance that is the source/destination of the identified malicious traffic. This powerful capability is difficult to impossible to accomplish without Aviatrix.

Figure 4 – ThreatIQ instance view.

ThreatIQ provides a compelling IDS capability that augments Amazon GuardDuty findings; however, the Aviatrix platform takes these capabilities further.

You can enable automatic remediation to turn the IDS capability into an intrusion prevention system (IPS). This feature, called ThreatGuard, takes the ThreatIQ findings and programs deny rules in Aviatrix Gateways to block the traffic at the source virtual private cloud (VPC) of the identified host.

It can also send webhooks to external services like Slack and email notifications so your Security Operations Center (SOC) gets notified as soon as malicious traffic is detected.

Figure 5 – Enabling ThreatGuard.

Once enabled, you see the stateful firewall rules that get programmed on Aviatrix Gateways to block malicious traffic. Below, you can see where stateful firewall rules have been programmed into Aviatrix Gateways to block malicious traffic flows.

Figure 6 – ThreatGuard block rules.

Aviatrix enables you to deploy secure-by-default, best-practice network architectures. This enables networking teams to easily codify and provision repeatable network architectures that security teams have full flow-level visibility into.

Application developers and application owners have full autonomy to deploy their application within the guardrails that the Aviatrix Secure Networking Platform provides.

Aviatrix is already helping customers detect, identify, and remediate malicious traffic flows in their cloud deployments. Even customers that thought they had a strong security perimeter are discovering malicious traffic patterns using ThreatIQ.

In this post, we discussed ThreatIQ with ThreatGuard and how each can help you augment cloud-native capabilities.

In addition to the Aviatrix Secure Networking Platform bringing a single-tenant data plane to AWS, it also serve as a distributed security inspection and remediation mechanism. With end-to-end control and visibility, the network becomes more intelligent and aware of the data traversing it. This allows you to secure critical business workloads in real-time right at the source of malicious traffic.

There is no configuration or additional costs to enable ThreatIQ on the Aviatrix Secure Networking Platform. Any Aviatrix customers with CoPilot can take advantage of ThreatIQ with ThreatGuard.

To get started, launch the Sandbox Starter Tool or deploy directly from AWS Marketplace. If you’d like to learn more and see what Aviatrix can do for your cloud networking, reach out and set up a demo.

.

.

Aviatrix is an AWS Competency Partner that is uniquely positioned to provide deep insights into network traffic.

Contact Aviatrix | Partner Overview | AWS Marketplace

*Already worked with Aviatrix? Rate the Partner

*To review an AWS Partner, you must be a customer that has worked with them directly on a project.

=======================

By Saar Tochner, Team Leader – Lumigo

Lumigo

As serverless architectures start to grow, finding the right troubleshooting approach becomes a business-critical aspect.

In this post, I will dive into the “instrumentation approach” and how to keep track of internal events within an AWS Lambda function, and how to export processed telemetry data.

In my role at Lumigo, an AWS DevOps Competency Partner whose software-as-a-service (SaaS) platform helps companies monitor and troubleshoot serverless applications, I lead a team of extremely talented developers that builds the instrumentation tools of Lumigo.

The goal of my team is to provide frictionless instrumentation methods that users can instantly use in their serverless production environments. Here, I’ll share the knowledge on extensions that we gathered while pursuing these methods.

As with any real-life project, we will handle legacy code, multiple code owners, and a huge stack of serverless technologies. Our goal is to write as little code as possible, avoid changing existing code, support cross-runtime Lambda functions, and have no latency impact.

Below is a practical guide on how to use AWS Lambda Extensions. We’ll follow a storyline of extracting internal events from Lambda functions, processing them, and sending telemetry data to external services.

Big ambitions require great technology. Extensions to the rescue!

In a nutshell, a blindfolded programmer cannot achieve greatness solve bugs.

While working on serverless architectures, asynchronous operations happen all the time. A Lambda function triggered Amazon API Gateway, and an Amazon Simple Queue Service (SQS) message was written to Amazon Simple Storage Service (Amazon S3), while an Amazon DynamoDB stream triggers a Lambda function, and so on.

Troubleshooting a serverless architecture means being able to track down all of these events into a single “flow,” where the exception (when occurred) is on one end of the flow and the cause is on the other.

The main issue here is the data. We need to:

Considering a real-world approach, we should be able to do it everywhere:

We’ll use internal and external extensions to achieve our goals, as shown in this Lumigo blog post. Following, I’ll discuss two interesting features that are the core of our implementation: post-execution processing and pre-execution processing with wrappers.

External extensions are hooks the Amazon Web Services (AWS) infrastructure provides inside the Lambda container. The code of the extension runs independently of the Lambda runtime process and is integrated into the Lambda lifecycle.

By communicating with the Lambda function, the extension can gather information and process it in the post-execution time. This occurs after the Lambda function has returned its response and before the extensions are finished running.

More information about the lifecycle of a Lambda function can be found in the AWS documentation.

Being an external process allows us to write it in our favorite language, and it will work on any Lambda runtime (assuming the extension is wrapped as an executable binary).

Figure 1 – AWS Lambda function lifecycle.

During the post-execution phase, we can process and export the telemetry data without interfering with the core invocation phase. Thus, we can avoid latency impact on the Lambda function, and the response will return as fast as it would without the extension.

At Lumigo, we use this phase to enforce privacy guarantees on the exported data (such as masking secrets and domains), and to ensure no private data escapes the Lambda environment. We use this timespan to limit data size and execute the exporting request itself.

Wrappers, or internal extensions, are executable files that are executed during the container’s cold initialization, just before the runtime is initialized. It can be used to alter the runtime, modify environment variables, and execute code at the beginning of the process.

In our case, we use a wrapper to define the environment hooks that communicate with the external extension and transfer data. The external extension will later (in the post-execution time) take this data, process it, and export it.

This is a critical phase where we add some code lines that run in the Lambda function itself. More information about the communication methods between the internal and external extensions will be discussed in the next section.

At Lumigo, we’re wrapping all the HTTP requests of the Lambda function. When a Lambda function creates a request, we catch it and send it to the external extension. Using these sensors, we can reconstruct the full “flow” of the serverless architecture.

To “move” the CPU or time-extensive logic from the execution phase to the post-execution phase, we must transfer the event’s data from the runtime’s process to the extension process.

There are different architectures that fit this use case and each has its own pros and cons.

Following, we’ll compare the most popular approaches: server-client, file system, and plain IPC.

In the performance tests, we transferred 1MB of data and checked it with Lambda function memory sizes: 128MB, 256MB, and 512MB.

1. Server-client communication: The external extension serves as the server, and the hooks post data to it. An obvious pro here is that this is the most simple and elegant method. Another pro is reliability (the extension can return an acknowledgment, or ACK) which is achieved using TCP.
   .
   On the other hand, the performances are low: 80ms (128MB), 38ms (256MB), and 10ms (512MB). Slow or malfunctioned extensions may damage the execution time even more (due to the wait for an ACK).
   .
2. File-system communication: The hooks write data to a special directory, and the extension reads from it only during the post-execution time. The biggest pro here is the lack of coupling between the two processes. The Lambda function’s main process will never be affected if the extension malfunctions.
   .
   On the other hand, the API should be implemented with care (in order to avoid miscommunication that may cause timeouts) and there is no reliability. The performances are much better here, with: 5.56ms (128MB), 3ms (256MB), and 1ms (512MB).
   .
3. Intra-process-communication (IPC): The extensions and hooks communicate over operating system (OS) syscalls. This is a very raw method and, thus, very fast (2-3 microseconds), but it’s hard to implement and doesn’t have any out-of-the-box reliable communication.

To sum it up, there are many different architectures with their own pros and cons. Your specific use case should inform the choice of one over the other. A brief summary can be found in the following table:

At Lumigo, we decided to use the file-system approach, as it’s relatively clean to implement and ensures high decoupling between the processes. This way, we answer to both performance and isolation requirements, and also write elegant code that can be maintained better.

External Extension

An external extension is an executable file that contains two important parts: the register and the extension loop.

We decided to write our extension in a higher-level language (Node.js 14.x) and compiled it into an executable that could be run in the Lambda runtime environment.

In order for the Lambda service to identify this file as an extension, it needs to be executable and packed inside a layer, under a directory named ‘extensions’.

Register happens in the bootstrap time of the container and is the first communication with the Lambda service:

Note that a common fallback here is the “extension-name” should be equal to the name of the file. More information can be found in the AWS documentation.

Next, the extension loop queries for the next event from the Lambda service:

This ‘urlopen’ request blocks the execution of the program when there’s no waiting invocation.

Note there is no need to define a timeout here; the container is being halted when there is no invocation, so this call will not get enough CPU to reach its timeout.

More in-depth code examples can be found in AWS official samples.

Internal Extension

In general, a do-nothing internal extension is much simpler and looks like this:

In order for the Lambda service to use this wrapper script, we need to add the environment variable AWS_LAMBDA_EXEC_WRAPPER with the path to this file.

At Lumigo, we use different methods to wrap the HTTP requests in different runtimes. For example, in the pythonic wrapper, we use the library wrapt to keep track of the most commonly-used function for HTTP communication (used by AWS SDK, requests, and many others):

Using the advanced tools that AWS Lambda provides, you can create robust mechanisms that answer core demands in serverless architectures.

In this post, I explored how to use external and internal extensions to create an instrumentation tool that extracts data from the Lambda function, processes, and exports it, and with almost no latency hit.

I also showed how most of the code could be written just once, in the external extension part, thus avoiding unnecessary code repetition and allowing fast adoption across different runtimes in the project.

Check out Lumigo’s serverless solution to learn more. We already implemented all of the above, and more, to provide you with a full monitoring and troubleshooting solution, tailored to fit your serverless architectures.

The content and opinions in this blog are those of the third-party author and AWS is not responsible for the content or accuracy of this post.

.

.

Lumigo is an AWS Competency Partner whose SaaS platform helps companies monitor and troubleshoot serverless applications.

Contact Lumigo | Partner Overview | AWS Marketplace

*Already worked with Lumigo? Rate the Partner

*To review an AWS Partner, you must be a customer that has worked with them directly on a project.

=======================

By Rana Dutt, Sr. Solutions Architect – AWS

Plaid

Open Finance initiatives have been gaining momentum across the world. These initiatives require that banks provide access to customer data through a common, open API for third-party applications, which are referred to as fintech apps.

The fintech app providers are generally not banks, but they offer users a variety of convenient payment and finance features on smartphone apps. These apps enhance the customer experience and foster greater choice and innovation. Users simply need to link the app to their bank and brokerage accounts, and grant the necessary permissions.

Fintech apps offer users benefits such as:

In this post, I will show you how to build and deploy a basic fintech app on Amazon Web Services (AWS) in under an hour by using the Plaid Link API. This app allows users to sign up, log in, select their bank from a list, connect to that bank, and display the latest transactions.

Plaid is a San Francisco-based financial services company and AWS Partner that helps fintech providers connect users safely to their bank accounts.

The Plaid Link acts as a secure proxy between a fintech app and a bank. With Plaid, application developers no longer need to worry about implementing scores of different ways to access data in myriad financial institutions.

Plaid is currently able to connect to more than 10,000 banks and financial institutions throughout the world. It provides a single API to connect to them. Currently, about 3,000 fintech apps use Plaid’s API to enable their users to access their bank accounts.

Through this post, we will build a demo fintech app on AWS using the AWS Amplify framework and Plaid Link. AWS Amplify helps us quickly build a serverless web app with a React frontend, user sign-up and sign-in using Amazon Cognito, an Amazon API Gateway-based REST API, and an Amazon DynamoDB database for storage.

Figure 1 – Architecture of demo fintech app.

AWS Amplify generates the code for signing up and authenticating users who are then stored in a Cognito user pool. It also helps create a REST API invoked by the React frontend and implemented by an AWS Lambda function behind Amazon API Gateway. The backend Lambda function sets up the Plaid Link which allows the end user to interact with a selected bank.

AWS Amplify also helps store the Plaid API key securely in AWS Secrets Manager so that it never needs to appear in the code or in a file. Plaid access tokens (described in the next section) are stored in the DynamoDB database.

This is a completely scalable and secure architecture which does not require the user to manage any server instances.

To build an app using Plaid Link, you first need to go to Plaid.com, click on the Get API Keys button, and create an account. You can create a free sandbox account to start.

You can then log into your dashboard and find your sandbox API key under the menu for Team Settings – Keys.

The following diagram shows what our demo Web app needs to implement.

Figure 2 – Plaid Link flow.

All API calls are made through a Plaid client object. The message flow is as follows:

1. The app first creates a Plaid client object by passing in the Plaid API key and Plaid client ID. It then calls the client’s createLinkToken method to obtain a temporary link token.
2. When the user selects a bank, the app uses the link token to open a Plaid Link to the bank and obtain a temporary public token.
3. The app then calls the client object’s exchangePublicToken method to exchange the public token for a permanent access token and an item ID that represents the bank.
4. The app stores the access token in DynamoDB for subsequent requests pertaining to that item. For example, the app can pass the access token to the client object’s getTransactions method to obtain a list of transactions within a specific date range.

Prerequisites

Building the App

Clone the repo and run npm install:

Initialize a new Amplify project. Hit Return to accept the defaults.

Add authentication:

Add the API:

Copy the Lambda source file, install dependencies, and push:

Add a database:

Update the Lambda function to add permissions for the database:

Deploying the App

Add hosting for the app:

Deploy the app:

Testing the App

Go to the URL displayed by the amplify publish command, and sign up as a new user. After logging in, select a bank from the list displayed.

If you are using the sandbox environment, use the credentials user_good / pass_good to access the bank and display the transactions.

The walkthrough in this post demonstrates how easy it is to use AWS Amplify to create a secure, scalable, and completely serverless fintech app on AWS that allows users to sign up, select from among the 10,000 banks that Plaid Link connects to, and obtain the transaction history for a particular account.

From here, you can add features such as making payments to friends or vendors, displaying balances across multiple accounts, sending low balance alerts and helping set a budget.

.

.

Plaid is an AWS Partner that  helps fintech providers connect users safely to their bank accounts.

Contact Plaid | Partner Overview

*Already worked with Plaid? Rate the Partner

*To review an AWS Partner, you must be a customer that has worked with them directly on a project.

=======================

By Naeem Asghar, Sr. Partner Solution Architect – AWS
By Vamsi Yanamadala, Sr. Partner Solution Architect – AWS
By Zoe Tomkins, Principle Partner Development Manager – AWS
By Jasbir Singh, IBM Global Lead, Accelerated Move Centre (AMC) – IBM

IBM

This post explains the challenges faced by customers running SAP ECC environments on premises and how IBM can accelerate their S/4 HANA transformation journey to Amazon Web Services (AWS).

IBM Accelerated Move Centre (AMC) delivers a cost-effective approach to SAP S/4HANA transformations:

IBM is an AWS Premier Consulting Partner with 10 AWS Competencies including the AWS SAP Competency and 13 AWS service validations.

IBM also participates in AWS Partner programs such as AWS Migration Acceleration Program (MAP), which offsets the migration costs of cloud adoption for customers and accelerates migration with additional AWS resources.

Customers face challenges when migrating and modernizing legacy SAP systems, and that was the case for a large process manufacturing company based in India which was experiencing issues with old hardware due for a refresh.

This included 13 years of legacy data, old versions of Oracle DB, complex interfaces and customer objects, performance issues with ABAP objects, material requirement planning, and Batch Management functionality for finished goods that was not performant.

The IBM Accelerated Move Centre (AMC) solution is a brownfield transformation to SAP S/4 HANA in a single step. By utilizing this solution, IBM was able to migrate the customer to SAP S/4HANA quickly and efficiently, while also upgrading Solution Manager and activating Batch Management to monitor and track finished goods.

The customer saw 100% capital cost avoidance on new on-premises infrastructure, 30% faster application response times for business users, shorter approval duration to due to workflow implementation, overall data footprint reduction, and an overall reduction in storage costs.

The organization was also able to automate 85% of their data conversion processes during the SAP transformation. To learn more, read the full case study.

IBM and AWS have the experience and knowledge to streamline complex SAP migrations and enable organizations to fully leverage the breadth of AWS tools and services that modernize SAP workloads.

AWS is the most reliable cloud to run your SAP workloads, providing the longest track record of cloud success and supporting more than 5,000 active SAP customers. Get the assistance you need to execute migrations, undergo transformation, or improve how you run SAP by leveraging IBM and their proven expertise.

With AWS, businesses can confidently run SAP systems with SAP-certified infrastructure that can support even the most demanding HANA-based production deployments.

Together, IBM and AWS offer a practical and cost-effective process for transformations to SAP S/4HANA, pre-scoped to fit your business requirements and drive future adoption of innovative SAP solutions.

Figure 1 – Accelerated S/4 HANA conversion with AWS services.

1. Design the AWS landing zone based on security and compliance needs of the customer.
2. Deploy the landing zone as infrastructure as code (IaC) with AWS CloudFormation.
3. Size out SAP system based on SAP HANA Sizing and business needs.
4. Understand and design high availability and disaster recovery (DR) requirements based recovery point objective (RPO) and recovery time objective (RTO) requirements.
5. Utilize the AWS Launch Wizard for SAP to deploy the SAP S/4 HANA target system in just over two hours.
6. Start Software Update Manager (SUM) with the system update and database migration process.
7. Finish the system update and database migration procedure on the target system.
8. Install and configure the AWS Backint Agent to manage and save SAP HANA backups to Amazon Simple Storage Service (Amazon S3). AWS Backint Agent is SAP-certified and integrates with SAP management tools.
9. Configure the SAP HANA database with Amazon CloudWatch Application Insights to monitor the health of your of the HANA database.

IBM AMC is a next-generation factory migration solution designed to migrate and modernize SAP landscapes to SAP S/4HANA on AWS with pre-scoped solutions to fit the specific requirements of each business and its SAP S/4HANA transformation goals.

Keeping in line with customer requirements and market trends, the IBM AMC solution is pre-scoped into T-shirt sizes to accommodate all types of customers. Each T-shirt size comes with a pre-defined system architecture, delivery model, and project plan.

Figure 2 – Pre-scoped T-shirt sizes.

Regardless of the T-shirt size, all IBM AMC projects are scoped with basis and security, development, functional, and testing capabilities.

IBM AMC has developed key differentiators with their tooling and accelerators, including:

These tools have been assessed by SAP and declared as market differentiators and help reduce project timelines and costs, while ensuring business continuity.

The value proposition to customers is quantifiable. With benchmarks based on different projects starting from the preparation phase, actual execution of the project, and then moving to the new system.

Some of the benefits realized are:

Figure 3 – IBM tooling value proposition.

The IBM AMC offering also supports RISE with SAP, a business transformation as a service to help clients shift to the intelligent enterprise.

IBM AMC is a central part of the strategic evolution partnership between IBM and SAP. As a global leader in SAP transformations, IBM offers in-depth industry and process experience to make your business smarter.

Together with SAP, IBM and AWS can help make your digital transformation a reality, bringing intelligence to your entire enterprise.

IBM’s benefits include:

Figure 4 – IBM’s experience and benefits.

Post deployment, there are several AWS services available to help automate DevOps-based management for SAP.

IBM delivers this automation for infrastructure provisioning, SAP installation including SAP clusters by leveraging AWS Launch Wizard, auto scaling, SAP kernel patch automation, HANA upgrades using AWS Systems Manager, and monitoring SAP systems via Amazon CloudWatch.

To learn more, see the Get Started with SAP on AWS page.

IBM provides the experience, tooling, methodology, and best practices to streamline your SAP transformation, combining SAP S/4HANA conversion and migration to AWS in a single step using the IBM’s Accelerated Move Centre (AMC).

IBM has nearly 50 years of experience in the industry, where no other SAP partner has more Global Partner Program Certifications. IBM has also completed over 6,500 SAP projects including hundreds of SAP S/4HANA implementations.

.

.

IBM is an AWS Premier Consulting Partner and MSP that offers comprehensive service capabilities addressing both business and technology challenges that clients face today.

Contact IBM | Partner Overview

*Already worked with IBM? Rate the Partner

*To review an AWS Partner, you must be a customer that has worked with them directly on a project.

=======================

By Rafal Krol, Cloud SRE – Chaos Gears

Chaos Gears

At Chaos Gears, an AWS Advanced Consulting Partner, we help customers and companies of all sizes utilize Amazon Web Services (AWS) to its full potential so they can focus on evolving their business.

One of our customers, a startup from the medical industry, has gained a global reach and serves its clients by operating in multiple AWS regions (currently 10 with more scheduled to come) spanning many time zones.

At the center of each region, there’s an Amazon Elastic Compute Cloud (Amazon EC2) instance that, by design, occasionally maxes out on the CPU. When this happens, a member of the customer’s operations team runs a set of checks to determine whether the instance is reachable; if it’s not, it gets restarted.

Once restarted and back online, which takes a few minutes, the same round of checks recommences. More often than not, this approach proves sufficient and the on-call engineer handling the issue can get back to work.

Being a startup and lacking the resources to man a follow-the-sun operations team, the customer came to Chaos Gears requesting a simple, adjustable, and cost-effective solution that would relieve their engineers from such an operational burden.

This post looks at the multi-regional first-line of support solution Chaos Gears built for the customer. I will also discuss how we automated the incident response duties that would typically engage at least one of the first-line of support engineers.

Before launching this solution, the on-call engineers needed to identify the affected instances, spread among different AWS regions, manually run a set of pre-defined checks on each one, and, based on the outcome, either do nothing or restart the pertinent machines and rerun the checks.

In today’s world of agile software development, our team at Chaos Gears treats everything as code, or at least we should be doing that.

Hence, the first decision we made for the customer was to leverage the AWS Cloud Development Kit (AWS CDK), a multi-language software development framework for modelling cloud infrastructure as reusable components as our infrastructure as code (IaC) tool.

Our customer’s software engineers were already familiar with TypeScript, the language we chose to build out the infrastructure with, which meant they’d comprehend the final solution quickly.

Moreover, we avoided the steep learning curve of mastering a domain-specific language (DSL) and the additional burden of handling an unfamiliar codebase.

The recent introduction of ASW CDK integration with the AWS Serverless Application Model (SAM), allows for developing serverless applications seamlessly within a CDK project.

On top of all of that, we could reuse the existing software tooling like linters and apply the industry’s coding best practices.

The adage says that “no server is better than no server,” and with that in mind we turned our heads towards AWS Step Functions, a serverless orchestrator for AWS Lambda and other AWS services.

The challenge at hand was perfect for an event-driven architecture, and we had already envisioned the subsequent steps of the verification process:

We needed the glue, and with AWS Step Functions we effortlessly combined all of those pieces without worrying about server provisioning, maintenance, retries, and error handling.

We had the backbone figured out, but we still had to decide how to monitor the CPU usage on the Amazon EC2 instances and pass the knowledge of a breach to AWS Step Functions state machine.

It screamed of Amazon CloudWatch alarms for the metric monitoring bit and Amazon EventBridge for creating a rule for routing the alarm event to the target (a state machine, in our case).

When the ‘CPUUtilization’ metric for a given instance reaches 100%, a CloudWatch alarm enters the ‘alarm’ state. This change gets picked up by an EventBridge rule that triggers the AWS Step Functions state machine.

Upon receiving the event object from the EventBridge rule, the state machine orchestrates the following workflow:

1. Three checks run, one after another—URL check, Route 53 check, and SSH check.
2. If all checks succeed during the first run, the execution ends silently (the ‘All good’ step followed by the ‘End’ field).
3. When a check fails, the EC2 instance is restarted and we recommence from the beginning with a second run.
4. If all checks succeed during the second run, a Slack notification is sent and the execution ends (the ‘Slack’ step followed by the ‘End’ field).
5. When a check fails during the second run, an OpsGenie alert is created and the execution ends (the ‘OpsGenie’ step followed by the ‘End’ field).

Here’s the diagram depicting the complete solution:

Figure 1 – State machine.

All of the above-mentioned resources, plus the Lambda functions, an Amazon Simple Storage Service (Amazon S3) bucket for the Lambda code packages, and the necessary AWS Identity and Access Management (IAM) roles and policies are created and managed by AWS CDK and AWS SAM.

Furthermore, this solution can be deployed effortlessly to multiple regions using AWS CDK environments.

A public repository is available on GitHub with a full working solution and a detailed README. I won’t dissect all of the code here, but let me draw your attention to some of the more interesting elements.

In the project’s root directory, we keep a ‘tsconfig.json’ file responsible for configuring the TypeScript’s compiler, and an ‘.eslintrc.json’ file holding the configuration for ES Lint, a popular JavaScript linter.

These two configuration files serve the entire project since we use TypeScript for both the infrastructure and application layers.

AWS CDK’s support for many popular general-purpose languages (TypeScript, JavaScript, Python, Java, C#, and Go, which is in developer preview) enables and encourages the DevOps culture by making the end-to-end development experience more uniform, as you can use familiar tools and frameworks across your whole stack.

Now, let’s take a closer look at the ‘bin/cpu-check-cdk.ts’ file, the point of entry to our CDK app, whence all stacks are instantiated.

We imported all of the necessary dependencies one library at a time, but in AWS CDK v2 all of the CDK libraries are consolidated in one package.

Next, we check whether all of the necessary environment variables have been set.

Then, we initialize the CDK app construct.

We grab the regions to which to deploy, along with corresponding instance IDs to monitor from AWS CDK’s context.

Next, we create a tags object with the app’s version and repo’s URL taken directly from the package.json file.

Finally, we loop through the map of regions and corresponding instance IDs we grabbed earlier.

In each region, we produce eight stacks: one for every Lambda function, one for the state machine, and one for the metric, alarm, and rule combo.

Thanks to using a basic programming concept of the for loop, we saved ourselves from unnecessary duplication by keeping things DRY (Don’t Repeat Yourself).

Nice and easy, and all in one go, regardless of the number of regions to which we would want to deploy, and mind you, there are 25 available (with more to come).

I won’t be going through all of the AWS CDK and Lambda files in this post, though I strongly encourage you to give the code a thorough review.

Notwithstanding, let’s see how easy it is to define a stack class in AWS CDK looking at the `lib/lambda-stack.ts` file. First, we import the dependencies:

You’ll notice there is a helper function called `capitalizeAndRemoveDashes` amongst the CDK libs. Since AWS CDK uses a GPL, we can introduce any amount of custom logic, as we could do with a _regular_ application.

The `lib/helpers.ts` file looks as follows:

Next, we extend the default stack properties like description with ours, setting some as mandatory and some as optional.

We start a declaration of the `LambdaStack` class with a `lambdaFunction` read-only property and a constructor.

We then create a resource name out of the mandatory name property that will be passed in during the class initialization.

We create an IAM role that the Lambda service can assume. We add the `service-role/AWSLambdaBasicExecutionRole` AWS-managed policy to it and, if provided, a custom user-managed policy.

Next, we initialize a construct of the Lambda function using the role defined earlier and stack properties, or arbitrary defaults if stack properties were not provided.

Finally, we expose the Lambda function object as the class’s read-only property we defined earlier. We’re also sure to close our brackets to avoid the implosion of the universe.

In this post, I showed how our team at Chaos Gears put together a serverless application running AWS Lambda under the baton of AWS Step Functions to relieve our customer’s engineers from some of their operational burdens. This enables them to focus more on evolving their business.

The approach described here can be adapted to serve other needs or cover different cases, as AWS Step Functions’ visual workflows allow for a quick translation of business requirements to technical ones.

By using AWS CDK as the infrastructure as code (IaC) tool, we were able to write all of the code in TypeScript, which puts us in an excellent position for future improvements.

We avoided the trap of introducing unnecessary complexity and kept things concise with codebase that was approachable and comprehensive to all team members.

Check out the GitHub repository and visit the Chaos Gears website to learn more about collaborating with us.

The content and opinions in this blog are those of the third-party author and AWS is not responsible for the content or accuracy of this post.

.

.

Chaos Gears is an AWS Partner that helps customers and companies of all sizes utilize AWS to its full potential so they can focus on evolving their business.

Contact Chaos Gears | Partner Overview

*Already worked with Chaos Gears? Rate the Partner

*To review an AWS Partner, you must be a customer that has worked with them directly on a project.

=======================

By Kalyan Purkayashta, Sr. Manager – Capgemini
By Rakesh Porwad, Manager – Capgemini
By Vikas Nambiar, Manager, Partner Solutions Architect – AWS

Capgemini

Mutual or insurance companies are often governed by members and have business and/or membership rules that govern membership grants. Such rules are classified as Inclusion rules and Exclusion rules.

Inclusion rules are business rules that reward customers who meet certain criterion, such as purchase of bonds or high-value financial instruments, and tenure or association with the company.

Conversely, Exclusion rules are defined to update/grant/revoke membership based on member activities, such as purchasing a certain products or product types, or having a secondary role on the product (particularly products that have joint holdings).

When implemented in legacy on-premises systems, such rules tend to be rigid in nature and impact business agility. Business users who depend on the IT department to make rule changes and updates triggered by changing business or regulatory requirements need to plan for development resource availability, change management, rules development lead times, and associated project costs.

This inhibits an organization’s ability to be agile and serve their members efficiently, expand memberships, and adapt quickly to changing business or regulatory requirements.

In this post, we will explore how Capgemini uses Amazon Web Services (AWS) to build a simple, agile, and configurable solution that implement business and membership rules on customer or master data. This is further improved using metadata to introduce or amend additional business rules.

Such a design pattern can be easily customized to additional business use case; not just customers or members.

Capgemini is an AWS Premier Consulting Partner and Managed Service Provider (MSP) with a multicultural team of 220,000 people in 40+ countries. Capgemini has more than 12,000 AWS accreditations and over 4,900 active AWS Certifications.

The diagram below depicts the high-level architecture of a rules-based engine on AWS. The main components of this architecture are:

Figure 1 – High-level solution overview.

This architecture caters for cleaning and deduplicating customer data, and for creating master data where an organization may have multiple sources for customer data.

The rules engine by design is configurable, allowing business rules that can be configurable and maintained with rule-specific metadata for instance type of rule, category, and so on. The rules evaluation results are enriched and captured with details to maintain history to understand the member journey.

Figure 2 shows how such an architecture can be realized using AWS-native managed services that reduce spend on operational maintenance activities, such as patching and capacity management, while providing programmatic access to the service feature. This enables operational excellence via automation.

Figure 2 – Solution overview using AWS managed services.

The first step of implementing this solution is to get all of the customer and related data needed to run the business rules into your data lake. The data lake acts as the central location for all data sources on which the rules engine will run.

Additional important points to consider while implementing the data lake are:

Figure 3 – Agile business rules engine implementation design on AWS.

The implementation uses the following AWS services:

Amazon Simple Storage Service (Amazon S3)

Amazon S3 is used to create a data lake of all data required for rules engine processing. As the data store aspect of the architecture, data lakes on S3 benefits from its 99.999999999% (11 nines) durability and object store nature, allowing the data lake to hold multiple data formats, datasets, and be able to be consumed by a variety of services in the AWS and third-party applications.

All customer information from the source environment is extracted and placed onto the data lake for subsequent processing by rules engine.

Amazon Aurora

Amazon Aurora is used to configure and capture the business rules with the metadata and reference data (like rule and membership types) that are needed to provision a simple relational database.

Aurora is a database engine for Amazon Relational Database Service (Amazon RDS) and meets requirements. This also provides a relational model that can be extended to include additional metadata like version of rule, active, or inactive rules.

These attributes make it easier to understand the customer/member journey and how and when they attained memberships, as well as which rules were in effect at that point in time and such.

Note that depending on the complexity of the rules and business requirements on how often these rules would change, you may want to configure the reference data accordingly.

Amazon EMR

Amazon EMR is used to create a data processer that can easily join data from different data stores and execute SQL in memory. It can also run rules in parallel, scale to changing business load, and be cost optimized by enabling transient features so you can save costs when the system is not in use.

Amazon EMR provides these features on AWS to rapidly process, analyze, and apply machine learning (ML) to big data using open-source frameworks.

The architecture uses Apache Spark on Amazon EMR due to its flexibility in being able to configure conditions and filters on the rules repository. Using SQL enables easier build and maintenance of rules.

Spark executes these SQL-based rules in parallel to identify the inclusions and exclusions against all of the customer data, and can be used to join with the other critical data elements (like product and service) and to validate against reference data configured in the rules repository.

Under the hood, the rules engine will:

Additional design considerations for the rules engine inclusions and exclusions modules:

The rules engine generates a set of customer inclusions and exclusions tagged with additional metadata like types of memberships. As a next step, these inclusions and exclusions are evaluated to determine a binary outcome that will tag only those customers that qualify under the various membership types.

Amazon DynamoDB

Amazon DynamoDB stores the results from the rules engine, as it provides a fully managed, serverless, key-value NoSQL database designed to run high-performance applications at any scale for end system consumption.

The membership records are is inserted into a DynamoDB database that maintains the history of results. The records can be inserted into DynamoDB using Python. This will be an insert only table to capture all the history.

Amazon DynamoDB can be partitioned on the master customer ID and sorted using a history created date. The benefit of partitioning the data on the same key used in the customer master makes it easier to combine these datasets easily. The history created date makes the sorting easier to generate the customer or membership journey.

Optionally, these results can be streamed using DynamoDB streams into another DynamoDB table to maintain a current member register with the latest details for easy access or current view of the member register. Often, businesses may have another key like membership number, so the benefit of using another current DynamoDB table with the current version is able to use an alternate partition key and enable faster access.

Amazon Managed Workflows for Apache Airflow (MWAA)

Amazon MWAA is used to orchestrate workflow sequencing of events required to ingest, transform, and load data using a managed service. This enables development teams to focus on defining the workflow sequences and not worry about the underlying infrastructure capacity and availability.

By using a combination of AWS managed services, Capgemini can build a cloud-based rules engine that can be scaled to increasing data volume, easily configured, and quickly accessed. This enables organizations to be completely agile in developing new business rules, or updating/retiring business rules, without relying on IT.

Capgemini, with its global experience, best-of-breed technology, process, and people, can partner with you to help design and build solutions that can be tailored to handle other medium to complex business rules for any industry, using AWS cloud-native services.

.

.

Capgemini is an AWS Premier Consulting Partner and MSP with a multicultural team of 220,000 people in 40+ countries. Capgemini has more than 12,000 AWS accreditations and over 4,900 active AWS Certifications.

Contact Capgemini | Partner Overview

*Already worked with Capgemini? Rate the Partner

*To review an AWS Partner, you must be a customer that has worked with them directly on a project.

=======================

By Bharani Subramaniam, Head of Technology – ThoughtWorks
By Mageswaran Muthukumar, Lead Consultant, Data Engineering – ThoughtWorks
By Nagesh Subrahmanyam, Partner Solutions Architect – AWS

Thoughtworks

In the digital economy, customers expect immediate, relevant, and frictionless experience when they want to find, do, or buy something.

To meet these expectations, businesses need to deliver hyper-personalization for every one of their customers. This demands a comprehensive view of customers’ behavioral patterns across direct and indirect touchpoints.

Such a solution requires an elastic infrastructure, as you can’t predict the amount of compute and storage a solution will need up front. Moreover, a regulated industry like banking puts additional constraints on dealing with personally identifiable information (PII).

This post describes how ThoughtWorks, a global technology consultancy company, handled infrastructure and regulatory challenges and delivered a cloud-native, hyper-personalized data platform on top of Amazon EMR for a leading private sector bank in India.

ThoughtWorks is as AWS Competency Partner and leading global technology consultancy that enables enterprises and technology disruptors to thrive as modern digital businesses.

Digital platforms are disrupting the traditional business models across many industries. Though there are a number of factors that make a digital platform successful, we have observed the strategy to sustain and promote usage of the platform is the one of most critical factors.

Traditionally, businesses focused most of their marketing efforts on acquiring new customers, but in the digital era they also need to continuously engage existing customers at the same time.

To achieve continuous engagement, some organizations have a dedicated “growth team” while others treat this as a part of their general marketing efforts. No matter how a business is set up, you need a good infrastructure to derive insights and make data-driven decisions.

Years of research point to the fact customers value contextual and personalized content over generic ads. Before we go further, let’s unpack the different levels of personalization and explore why hyper-personalization is so effective.

Assume your business has one million consumers and your marketing department wants to promote 10 offers, each with 10 variants, to keep these customers engaged.

Though there can be many ways to achieve this, we can broadly categorize communication mechanisms into three distinct categories, as shown in the levels of personalization chart below: generic messaging, segmentation, and hyper-personalization. The effectiveness of communication increases as the focus of personalization increases.

Figure 1 -Levels of personalization.

The following sections describe the levels of personalization.

Generic Messaging

In this approach, you would simply send all 100 offers (10 offers with 10 variants) to every one of your million customers. You may overlay the customer name or gender to make it look personalized, but this kind of generic messaging is mostly ineffective and there’s a high probability your customers would treat it as spam.

Segmentation

In this approach, you’d leverage the knowledge gained about your customers through their purchase history and preferences to effectively cluster or segment them into distinct personas. You can also filter the offers according to each segment. Segmentation leads to better conversion rate as it’s more targeted than generic messaging.

Hyper-Personalization

With hyper-personalization, you would go a step further than just segmenting your user base. You don’t send the offers at random intervals, but rather watch for signals from customer behavior and promote valid offers that address the needs of your customer in a timely manner.

Hyper-personalization is far more effective than other approaches, but it does require sufficient data to infer behavioral patterns and the ability to process them in near real-time.

When we analyzed the volume and nature of transactional data required to create the behavioral model of our customers, we realized a few key insights:

With this insight, it was clear we can’t depend only on the on-premises data center capacity; rather, we need a hybrid-cloud model. For security considerations, the data preparation and sanitization steps are executed from on-premises data centers.

ThoughtWorks decided to leverage AWS for the big data pipeline, and this hybrid cloud approach enabled them to focus on quickly testing business hypotheses rather than operating the big data infrastructure.

Figure 2 – Expand and contract nature of data.

The following table describes our approach at a high level.

To illustrate the five stages of hyper-personalization as described in the table, let’s consider a fictional user named Saanvi Sarkar and see how the model gets progressively built as we go through each stage.

Integration with System of Records (SoRs)

We carefully mapped our requirements with hundreds of data sources from within and outside the bank. Details on such integrations and mapping of data sources are beyond the scope of this post.

In this stage, we’d have all transactional details of Saanvi Sarkar from their existing banking account. We also have access to credit and other scoring details from regulatory sources.

Personally Identifiable Information (PII) Scrubber

The first stage in the privacy preserving layer of the solution is the PII scrubber. Depending on the nature of the attributes, we apply a few strategies to protect the end user’s privacy:

At this stage, Saanvi Sarkar details look like this:

<Key Identifier> <Necessary PII Data> <Non-PII Data>

Tokenization and Encryption

In the next stage, we tokenize the key identifier so that even if there is a breach it will be difficult for bad actors to make sense of the records.

After tokenization and encryption, the output record of Saanvi Sarkar will look like <Token>< Non-PII Data>. At this stage, the data is anonymized and safe for processing in the cloud.

Big Data Processing in Amazon EMR

The bulk of the data processing happens in this stage. Cleansed data that’s safe for computation in the cloud is fed into data pipelines in Amazon EMR Spark clusters.

We have a mix of batch and streaming workloads as described in the architecture diagram in Figure 3 below. The entire pipeline can be broken down into two stages: modelling customer behavior, and the matching engine.

Modelling Customer Behavior

Data arrives either via Kafka topics or via files from Amazon Simple Storage Service (Amazon S3). We utilize Apache Airflow to schedule jobs in Amazon EMR clusters.

Since we have hundreds of data sources, we have to support both continuous streams of updates and batches arriving in hourly, daily, or even weekly intervals.

We can now infer that Saanvi Sarkar regularly deposits ₹50,000 within the first two weeks of every month and their average balance never went below ₹300,000. We also know she extensively travels by two wheeler based on fuel expenses.

Figure 3 – Hyper-personalization architecture.

Matching Engine

We wanted a flexible way to express the changing needs of the marketing team, so we created a custom domain specific language (DSL) that’s easy to understand but at the same time expressive enough to capture the changing needs.

The DSL takes the customer model, inventory of offers along with its variants, and the real-time behavioral data to effectively match suitable offers.

Users from the campaign management team continuously come up with offers and variants and capture the matching logic in the DSL. Once executed, the DSL takes care of creating necessary jobs in Airflow, which in turn orchestrates the matching pipelines in Amazon EMR Spark clusters.

In this example, there is a high probability that Saanvi Sarkar is looking to buy a car. Given this context, the matching engine will pick the car loan offer with attractive interest percentage based on Saanvi Sarkar’s credit history.

De-Tokenize and Deliver

The matching engine communicates the match as <Token> <Offer> to the on-premises data center. Since the data in the cloud is tokenized for protecting customer privacy, we need to de-tokenize before delivering the offer.

The matching engine predicted the need of the Saanvi Sarkar to own a car and came up the relevant offer for the car loan with customized interest rate based on credit history. This approach significantly transformed the customer relationships for the bank.

Instead of designing a group of offers and sending them to millions at a time, the bank offers immediate, relevant, and frictionless experiences to its diverse customers.

Digitally savvy customers want to feel delighted and they expect organizations to add value to their lives with relevant and timely information. ThoughtWorks and AWS have seen hyper-personalization help in shifting the focus from static promotions to address the dynamic needs of their customers.

From our experience, some key factors to consider while implementing hyper-personalization initiatives are:

If you are interested in knowing more about hyper-personalization, here’s a good collection of articles addressing the dynamic micro-moments of customers.

If you’re interested in the implementation details of streaming processing using Apache Spark and Apache Kafka, learn more in this AWS Big Data blog post.

.

.

ThoughtWorks is an AWS Competency Partner and leading global technology consultancy that enables enterprises and technology disruptors to thrive as modern digital businesses.

Contact ThoughtWorks | Partner Overview

*Already worked with ThoughtWorks? Rate the Partner

*To review an AWS Partner, you must be a customer that has worked with them directly on a project.

=======================

By Deepika Giri, Sr. Director, Insights and Data – Capgemini
By Vikas Nambiar, Manager, Partner Solutions Architect – AWS

Capgemini

As organizations evolve and embrace digital transformation, they realize customer value delivered is a function of quality of engagement through various channels: digital, online, and in-person.

Customer experience is at its best when a customer perceives the experience offered is unique and aligns to their preferences. The need to engage, at a very personal level, becomes key.

Organizations have realized that knowledge of data around customer behaviors are crucial for a differentiated customer experience.

Traditionally, customer relationship management (CRM) tools were considered the mainstay for first-party customer data, whereas now there are a plethora of solutions to choose from.

This ranges from a standalone tool or service to a complete ecosystem that deliver superior customer experience platforms, including customer data platforms (CDP) and marketing data management platforms (DMP), to name a few.

Businesses, meanwhile, have realized there is no single product or service that provides all of the features, and building such a platform requires a complex process. A robust and reliable customer platform has to integrate and aggregate first-, second- and third-party data seamlessly and in real-time in a cookie-less paradigm.

In this post, we’ll look at how Capgemini’s data and analytics practice implements Customer Intelligence Platforms on Amazon Web Services (AWS) to help companies a unified data hub. The platforms enable organizations to convert customer data into insights that can be used for reporting, building artificial intelligence (AI) and machine learning (ML) predictive analytics capabilities, and more.

Capgemini is an AWS Premier Consulting Partner and Managed Service Provider (MSP) with a multicultural team of 220,000 people in 40+ countries. Capgemini has more than 12,000 AWS accreditations and over 4,900 active AWS Certifications.

Capgemini has worked with clients globally, across multiple industry sectors, and helped customers in their data transformation journey. This rich and deep industry experience and best practice knowledge has helped Capgemini simplify and industrialize the approach of building a modern data estate, designed to deliver business value and be future proof.

The Capgemini reference architecture outlined in this post serves as a blueprint to build next-generation data platforms that are robust, scalable, and drive innovation. This isn’t just a technology platform but a framework that brings together people, processes, and technology. The architecture covers all aspects of data, from ingestion to consumption.

Capgemini’s reference architecture is designed to be modular and decoupled, covering all major aspects of a data platform including:

Figure 1 – Capgemini customer data and AI platform reference architecture.

Given the need to create a differentiated customer experience, major CRM system vendors have built systems with capabilities to integrate customer data. The goal is to help drive insights and activation to marketing and sales, customer care, and ecommerce systems.

There are many customer data platforms that serve to integrate all customer data and enrich them through AI/ML. These share actions to be delivered via marketing data management platforms or existing customer marketing or sales platforms.

However, the data capabilities of such implementations can be limiting, albeit they enable organizations to build a unified customer profile.

Such implementations are incomplete without the capability to provide a unique enterprise-wide customer ID that is derived from all parts of customer data spread across their applications. In addition, they are not specialized in data quality and privacy, golden record generation, and management and sector specific solutions.

This implementation also needs to be further integrated with customer behavior data to generate consumable products such as customer segmentation, propensity models, cognitive services and measurements, and insights.

Thus, Capgemini’s Customer Intelligence Platform on AWS serves as a robust and reliable source of harmonized customer-related data.

A great degree of personalization is possible when all behavior and customer knowledge is deep and complete. A large variety of use cases can be built around this design using the customer intelligence gathered.

Below are some of the relevant use cases:

Capgemini’s Customer Intelligence Platforms on AWS enable superior omnichannel customer experiences, empowering businesses to be innovative, agile, and pioneers in the way they operate. The platforms serve as a foundational data layer that delivers robust customer data and insights to applications like marketing, sales, customer care, and ecommerce.

Please note that the following architecture addresses common data ingestion, curation, and analytics foundation aspects of a typical solution. This can be further expanded to include platform and data governance aspects based on an organization’s operational requirements.

Figure 2 – Capgemini customer data and AI platform implementation design overview.

Key AWS services leveraged in the architecture:

In the current digital environment, without a unified data hub that converts customer data into insights, an organization’s capability to retain their customers will be impacted.

Such a unified data hub, enriched with AI/ML services implemented to requirements, provides improved and unique customer experiences that are aligned to their interests and preferences. Platforms that achieve this can increase return customer numbers and deliver higher customer satisfaction numbers.

Capgemini’s data and analytics practice, along with their Customer Intelligence Platforms on AWS, global experience, best-of-breed technology, process, and people can help enterprises create differentiated customer experiences.

Learn more about how Capgemini is turning customer value analytics into insights, decisions, and business value.

.

.

Capgemini is an AWS Premier Consulting Partner and MSP with a multicultural team of 220,000 people in 40+ countries. Capgemini has more than 12,000 AWS accreditations and over 4,900 active AWS Certifications.

Contact Capgemini | Partner Overview

*Already worked with Capgemini? Rate the Partner

*To review an AWS Partner, you must be a customer that has worked with them directly on a project.

=======================

By Kiran Anna, Sr. Product Manager – D2iQ
By Young Jeong, Partner Solutions Architect – AWS

D2iQ

While the global spend on artificial intelligence (AI) and machine learning (ML) was $50 billion in 2020 and is expected to increase to $110 billion by 2024 per an IDC report, AI/ML success has been hard to come by—and often slow to arrive when it does.

There are four main impediments to successful adoption of AI/ML in the cloud-native enterprise:

Kubernetes would seem to be an ideal way to address some of the obstacles to getting AI/ML workloads into production. It is inherently scalable, which suits the varying capacity requirements of training, tuning, and deploying models.

Kubernetes is also hardware agnostic and can work across a wide range of infrastructure platforms, and Kubeflow—the self-described ML toolkit for Kubernetes—provides a Kubernetes-native platform for developing and deploying ML systems.

Unfortunately, Kubernetes can introduce complexities as well, particularly for data scientists and data engineers who may not have the bandwidth or desire to learn how to manage it. Kubeflow has its own challenges, too, including difficulties with installation and with integrating its loosely-coupled components, as well as poor documentation.

In this post, we’ll discuss how D2iQ Kaptain on Amazon Web Services (AWS) directly addresses the challenges of moving machine learning workloads into production, the steep learning curve for Kubernetes, and the particular difficulties Kubeflow can introduce.

D2iQ is an AWS Containers Competency Partner, and D2iQ Kaptain enables organizations to develop and deploy machine learning workloads at scale. It satisfies the organization’s security and compliance requirements, thus minimizing operational friction and meeting the needs of all teams involved in a successful ML project.

Kaptain is an end-to-end ML platform built for security, scale, and speed. With some of its core functionalities leveraged from Kubeflow, Kaptain supports multi-tenancy with fine-grained role-based access control (RBAC), as well as authentication, authorization, and end-to-end encryption with Dex and Istio.

Figure 1 – Kaptain architecture.

Kaptain has many components that are geared towards development of ML models. The platform provides a Python-native user experience, although Spark support is available through Scala with Apache Toree.

The Jupyter Notebooks that many data scientists use for ML projects are pre-installed with the following:

While setting up a Jupyter notebook on a single laptop is easy, it becomes tricky with custom libraries, drivers while provisioning hardware, or with demands for portability, security profiles, service accounts, credentials, and so on. In short, notebooks are not that simple to manage in an enterprise environment.

On top of providing a familiar notebook environment, Kaptain provides easy steps to create a full ML workload lifecycle. The Kaptain SDK enables data scientists to perform distributed training on a cluster’s resources, conduct experiments with parallel trials for hyperparameter tuning, and deploy the models as auto-scaling web services with load balancers, canary deployments, and monitoring.

The software development kit abstracts away the production tasks and allows data scientists to focus solely on model production.

Take, as an example, a data scientist who wants to tune the hyperparameters of a model by running four experiments in parallel, with each experiment running on two machines.

The steps to approach this on Kubernetes without the Kaptain SDK are:

With the Kaptain SDK, the process is far simpler and more data scientist friendly. The Kaptain SDK allows data scientists to do all of these steps from Python without having to do more than Step 1 above.

Here is how simple that process looks as Python code:

The open-source edition of Kubeflow has well over 50 components, integrations, and related initiatives, which can make life as a data scientist confusing. How do you choose from a set of tools you’re not that familiar with?

D2iQ ensures the best cloud-native tools are included in Kaptain, and only those that provide unique functionality that makes sense for enterprise data science use cases.

Through research and review, evaluation is done on a set of core criteria, including:

D2iQ regularly runs tests with mixed workloads on large clusters to simulate realistic enterprise environments. This ensures the entire stack is guaranteed to work and scale. Each release of Kaptain is also “soaked,” or run with a heavy load for a certain duration, to validate the performance and stability of the system. Kaptain can be updated with zero downtime.

Do-it-yourself (DIY) machine learning platforms, with so many moving parts, can easily expose your business to unnecessary security risks. In June 2020, ZDNet reported a widespread attack against DIY Kubeflow clusters that failed to implement proper security protocols. Kaptain is protected from such vulnerabilities by design, and is deployed exclusively behind strict authentication and authorization mechanisms.

All of these factors combine to reduce the complexity and friction associated with running ML workloads on AWS. By simplifying the process of getting ML workloads into production, Kaptain enables organizations to make greater use of this game-changing technology.

One of D2iQ’s customers is a government scientific research organization that addresses a broad range of technical and policy questions. Many of the organization’s data scientists work on AWS, but much of the enormous volumes of data generated by the organization are on edge locations.

The costs of moving this data from the edge to the center of the network, or from one region to another, were quickly becoming prohibitive. Data scientists were frustrated by the long lag time between requesting IT resources and actually receiving them.

Data scientists also had to manage their own tech stack, resulting in a lack of repeatable standards. From the IT team’s perspective, lack of governance for deployed workloads led to concerns around security and inefficient use of resources.

The organization quickly adopted D2iQ Kaptain upon its introduction. Kaptain runs as a workload on the D2iQ Konvoy Kubernetes distribution, which was deployed on AWS and the customer’s data centers.

Figure 2 – Kaptain deployment reference architecture.

Kaptain solved three big problems for the organization:

Plus, D2iQ’s expert support and training services extend the reach of the organization’s IT team and help them maximize the satisfaction of their internal data scientist and data engineer customers.

D2iQ Kaptain enables data scientists and data engineers to harness the scalability and flexibility of Kubernetes without having to struggle with its complexity.

D2iQ Kaptain and the Kaptain SDK provide a notebooks-first approach to managing the entire machine learning lifecycle, from model development to training, deployment, tuning, and maintenance.

By reducing the complexity and friction of getting ML models from development to production, Kaptain helps organizations increase the share of models actually being implemented and providing positive returns on investment.

To learn more about D2iQ Kaptain, see a demo, or arrange a free trial, please visit the Kaptain website.

.

.

D2iQ is an AWS Containers Competency Partner that delivers a leading independent platform for enterprise-grade Kubernetes.

Contact D2iQ | Partner Overview

*Already worked with D2iQ? Rate the Partner

*To review an AWS Partner, you must be a customer that has worked with them directly on a project.