Case Study Short Description:
A company had partnered with Tekrosta to do a greenfield deployment using AWS services, they also wanted to run their pipelines for the application that they are using. The most important part of this is that they wanted everything to be automated so for this we used the existing AWS service called AWS batch.
Problem Statement/ Definition:
The customer was operating a greenfield AWS environment with a partially manual EC2-based processing architecture that was unable to efficiently handle variable workloads and increasing user concurrency. The existing model required manual EC2 scaling, lacked automated job scheduling, retry mechanisms, and workload orchestration, resulting in processing failures, operational overhead, and inefficient resource utilization. Additionally, long-running workloads could not be effectively managed through AWS Lambda due to execution time limitations, impacting automation capabilities. Despite limited production usage, the environment incurred significant monthly costs due to inefficient compute utilization. To address these challenges, the customer aimed to modernize their AWS architecture by implementing a managed AWS batch processing solution with automated orchestration, containerized workloads, elastic scaling, improved reliability, and optimized cloud costs.
Proposed Solution & Architecture:
Tekrosta Cloud designed and implemented a modern AWS batch processing architecture to replace the existing manual EC2-based processing model with an automated, scalable, and containerized solution. The solution included:
• Amazon ECR for creating container repositories, managing pipeline images, and maintaining version-controlled Docker images for processing workloads.
• AWS Batch to deploy a managed batch processing environment using m6i.4xlarge EC2 compute resources in private subnets with automated scaling capabilities.
• AWS Batch Job Queues and Job Definitions to manage workload scheduling, priorities, retries, timeouts, and execution parameters.
• Containerized Processing Workloads: deployed through ECR images to ensure consistent execution and improved workload isolation.
• Amazon S3 configured with separate input and output buckets for data ingestion, processing, and result storage.
• AWS Lambda configured as an orchestration layer to validate uploaded datasets, extract metadata, and trigger AWS Batch jobs automatically.
• Amazon RDS configured for storing job metadata, execution status, and processing information.
• VPC, Security Groups, and IAM configured to enable secure communication between AWS Batch, S3, RDS, and supporting services.
• CloudWatch, CloudTrail, Config, and GuardDuty enabled for monitoring, logging, security auditing, and compliance tracking.
• Testing and Validation performed for concurrent job execution, long-running workloads, data processing flow, and output validation.
The implemented architecture improved scalability, automated workload processing, reduced operational dependency, and optimized resource utilization for future production workloads.
Outcomes of Project & Success Metrics
Below measurements to be done to ascertain success of implementation –
• Batch environment processes representative workloads reliably and scales automatically as per need
• All 3 pipelines get triggered and validated only then AWS batch creates a batch job successfully which launches the required compute environment
• The output of the batch job is stored in the required format i.e. User ID then case ID folder and then the required output.
• The lambda function for the database then updates the required case ID in the RDS.
• Observability dashboards and alarms in place.
Lessons Learned*
· While setting up AWS batch for the compute environment please make sure that you choose Batch compatible operating system also make sure that you have Docker 25 and your image is made from Docker version 25 so that the orchestration is smooth.
· Make sure to add the necessary environment variables if required in the job definition during setting up the AWS batch under container configuration section.
· While setting up the launch template for AWS batch compute environment , make sure to use default settings only .