Image for post
Image for post


There are a bunch of articles out there that talk about serverless logging (mostly Lambda) and best practices. In this article, I will try to cover tools and practices for the AWS serverless workload logs (observability) and critical insights they provide to build, plan and scale a reliable architecture from a Well-Architected Framework (AWS-WA framework) point of view.

Before we jump into details, just a quick recap of the fundamental difference between logs, metrics, and events provided by AWS.

Image for post
Image for post

Networking itself is a big and complicated topic of AWS. In this article, I am sharing my experiences in setting up EIP + NAT + IGW + RT to establish a connection from AWS glue connections to a MySql database where my EIP is whitelisted.

Before we get into details here is a quick summary and their one-line description for the current setup.

  • Internet Gateway (IGW)— Gateway that allows internet to your VPC
  • Public subnet — A subnet to which IGW is attached
  • Private subnet — A serverless instance where your Glue connector is running.
  • Elastic IP Address (EIP)— A…

Image for post
Image for post

It is a well-known fact that s3 + Athena is a match made in heaven but since data is in S3 and Athena is serverless, we have to use GLUE crawler to store metadata about what is contained within those S3 locations.

Even small master tables, metrics tables or daily incremental transactional data with Schema changes must be crawled to create a table on Athena.

In the beginning, my team and I used to write python scripts which upload the CSV files to s3 and then trigger a Lambda function which will invoke the relevant Crawler and create/update the table…

Image for post
Image for post

It's so surprising that I didn’t find any article that helps me to install superset on an Ubuntu 16.04 AMI of AWS EC2 instance. So, I have decided to do it myself. In this approach, I tried to install a Docker container by following superset’s cloud-native installation procedure. In my future articles, I will be installing superset build node version running on WSGI HTTP Apache or Gunicorn server.

Following are the steps I followed to successfully install superset in DEBUG mode on a default port 8080.

Step 1: Start Ubuntu 16.04 AMI instance on t2.large(optional)

Step 2: SSH into the…

Dheeraj Inampudi

Transforming complex data into powerful communications.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store