Sagar's Blog

I transferred my domain to cloudflare

2025-01-20T19:30:00+09:00

The domain sagargiri.com was registered with namecheap.com and I was using namecheap.com's nameservers to manage the domain. I decided to transfer the domain to cloudflare.com for better security and performance.

I followed the steps mentioned in this YouTube Video

Following above steps to the tee, I successfully transferred the domain to cloudflare.com.

Three-step guide to set up custom domain for your github.io pages

2021-11-27T23:08:00+09:00

Assumption: For this tutorial, I am assuming you already have bought a domain name from any domain name provider and you already have a github.io pages setup like this girisagar46.github.io. Also in this post, I'm using namecheap and setting up Apex domain name.

This blog is hosted in GitHub pages. When you visit sagargiri.com, the web pages that you see are actually hosted in github.io, and you can see the source code of this blog here. But, how does my domain sagargiri.com is pointing to girisagar46.github.io ? How do we configure such settings? All of these questions are answered in this blog post.

Step-by-step guide to use custom domain for github.io pages

Step 1: Buy the domain name.

If you do not have a domain name yet, you can use any of the domain name providers. For example, you can use namecheap, godaddy or domains.google. I bought mine from namecheap during black friday sale.

Now that you have bought the domain name, you need to set up the DNS records. Which we'll do in second step.

Step 2: Set up DNS records.

In your domain name provider page you'll have an option to set up DNS records. In the case of namecheap, you'll first have to click the MANAGE button in the Domain List page as shown in screenshot below.

Then click the Advanced DNS button as shown in screenshot below.

In the HOST RECORDS section, you need to add the following DNS records as shown in screenshot below.

Here are the IP addresses list for A record for you to copy and paste.

185.199.108.153
185.199.109.153
185.199.110.153
185.199.111.153

Also DO NOT forget to add the CNAME record as shown in screenshot. You need to add your own github.io domain name.

Once you have added the DNS records, you can now go to your GitHub page settings to actually map the CNAME.

Step 3. Configure your USERNAME.github.io page settings

Go to your github.io repository settings and click on the Settings button and then click on the Pages button as shown in screenshot below.

This will lead you to the GitHub.io settings for GitHub pages.

In the Pages section, you need to add the following domain name as shown in screenshot below.

This will commit a new file called CNAME to your github.io repository. You can verify that later on. Also do not forget to check the Enforce HTTPS checkbox. This will provide you SSL certificate for your domain name. Once you see the green tick mark in your domain name in the Pages section, you're good tpo go.

But, be aware that the DNS propagation takes some time. Take rest, drink a cup of coffee, browse other articles in this website and wait for a while.

After some time, you can visit your domain name, and you will see the github.io pages are mapped to your domain name.

You can also verify all this is working using the dig command

╰─$ dig sagargiri.com +noall +answer -t A

; <<>> DiG 9.10.6 <<>> sagargiri.com +noall +answer -t A
;; global options: +cmd
sagargiri.com.      1799    IN  A   185.199.109.153
sagargiri.com.      1799    IN  A   185.199.110.153
sagargiri.com.      1799    IN  A   185.199.111.153
sagargiri.com.      1799    IN  A   185.199.108.153

Here you can see that my domain is pointing to the A records that we have set up in the Step 2.

References:

https://docs.github.com/en/pages/getting-started-with-github-pages/about-github-pages https://docs.github.com/en/pages/configuring-a-custom-domain-for-your-github-pages-site

Tools and technology I use for my day-to-day work

2021-07-17T13:17:00+09:00

I am a professional software engineer working for a Japan-based company beBit.inc building a flagship SAAS application called USERGRAM. In my day-to-day work, I use lots of tools and technologies to get my job done and to make me productive. Here's the list of it:

1. PyCharm and Intellij IDEA

I mostly code in Python and Java and the above two are my go-to IDE if I have to work on any of these languages. I use the Kubernetes plugin in my IntelliJ IDEA to validate YAML configurations for k8s and it's such a handy tool.

2. github.com

GitHub is where we host all our source code for USERGRAM, and I use it for my personal pet projects as well. GitHub action amazing CI/CD integration in GitHub and my company and personally I love it. The PR review system is fantastic. Besides, all that, I also follow many developers' and programmers' profiles to see what they are working on, technology trends and also star some cool projects on GitHub. There isn't a single day (except some weekends) when I do not visit github.com

3. 1password

I can't emphasize enough about 1password. This is my de-facto password management tool. My company uses it and I use it. I have to say, this is the best investment I made. 100% worth it.

4. Firefox and Brave

Firefox and Brave browsers are my browser choices. I use Brave for official work and use Firefox for personal stuff such as social media. This is because I do not want to mix up my personal and official work.

5. Docker

Everything is dockerized at work and having Docker is absolutely necessary. But, while running a heavy stack, it sometimes slows down my laptop.

6. CleanMyMac X

I purchased this software to make my laptop faster by cleaning up junk files and freeing my RAM. It does more than that like finding large files, malware protection, etc.

7. iTerm2

This is my default terminal emulator for my mac. I installed ZSH shell as a default shell and I have to say iTerm2 and ZSH go hand in hand, no doubt.

8. Slack, Zoom, Around

These are the communication tools we use in our company to communicate. After the pandemic hit, we've been connected more with Slack, Zoom, and Around. We recently started to use Around in our team and it's really nice.

9. Skitch

In my day-to-day work, I have to screenshot and annotate screenshots a lot. I used to use mac's default screenshot tool in combination with some other 3rd party tools to screenshot stuff. That all got replaced when I found about Skitch. Now it's been my default and go-to tool for screenshots and annotations.

Honorable mentions

WebStorm (IDE for FrontEnd)

ExpressVPN (My Favourite VPN service)

Keybase

Spectacle

Atlassian JIRA and Confluence

Archbee (The app I use to write this blog)

Dropbox and Google Drive

G-suit

AWS

k8s

Terraform

Visual Studio Code

Draw.io

PS. I'll be adding more to this list later on.

How to resolve "Unable to validate the following destination configurations" while adding event notification to your S3 bucket?

2021-02-25T22:11:00+09:00

Problem

I have an existing S3 bucket and I wanted to add an S3 event notification to invoke my lambda function's dev alias. I read the s3api put-bucket-notification-configuration documentation and prepare the notification.json file like this:

{
    "LambdaFunctionConfigurations": [
        {
            "Id": "ToInvokeMyLambdaFunction",
            "LambdaFunctionArn": "arn:aws:lambda:ap-northeast-1:123456789101:function:TestFunc:dev",
            "Events": [
                "s3:ObjectCreated:Put"
            ],
            "Filter": {
                "Key": {
                    "FilterRules": [
                        {
                            "Name": "suffix",
                            "Value": "awesome_data.csv"
                        }
                    ]
                }
            }
        }
    ]
}

Then I execute the s3api command:

$ aws s3api put-bucket-notification-configuration \
 --bucket MyAwesomeBucket \
 --notification-configuration file://notification.json

I got an error that said:

An error occurred (InvalidArgument) when calling the PutBucketNotificationConfiguration operation: Unable to validate the following destination configurations

I checked my aws-cli version, it was the recommended one:

╰─$ aws --version
aws-cli/2.0.12 Python/3.7.4 Darwin/20.3.0 botocore/2.0.0dev16

But... It works when I do it from the S3 console...

I tried executing the same aws s3api command again now with the --debug flag. And in the middle of the long debug output, I see this:

$ aws s3api put-bucket-notification-configuration \
 --bucket MyAwesomeBucket \
 --notification-configuration file://notification.json \
 --debug

...
2021-02-25 21:39:13,902 - MainThread - botocore.parsers - DEBUG - Response body:
b'<?xml version="1.0" encoding="UTF-8"?>\n<Error><Code>InvalidArgument</Code><Message>Unable to validate the following destination configurations</Message><ArgumentName1>arn:aws:lambda:ap-northeast-1:123456789101:function:TestFunc:dev, null</ArgumentName1><ArgumentValue1>Not authorized to invoke function [arn:aws:lambda:ap-northeast-1:123456789101:function:TestFunc:dev]</ArgumentValue1><RequestId>2B8705F2FD8848F2</RequestId><HostId>bj9ahrqPxN3emWnZ008dtkVmVR9VVxfFjAJAw9hKvhoa4vtdHXaGi/x4a4Okki3oJhbaeHe0Ppk=</HostId></Error>'
...

The gist of it is Not authorized to invoke function [arn:aws:lambda:ap-northeast-1:123456789101:function:TestFunc:dev]

So the problem was with the lambda permission. :thinking:

Solution

Digging around the internet I find this And the solution is to give your lambda a permission to being invoked by S3 first. Which can be done like this:

$ aws lambda add-permission \
 --function-name TestFunc:dev \
 --profile default \
 --statement-id AllowToBeInvoked \
 --action "lambda:InvokeFunction" \
 --principal s3.amazonaws.com \
 --source-arn "arn:aws:s3:::MyAwesomeBucket" \
 --source-account 123456789101

I got the output like this:

{
 "Statement": "{\"Sid\":\"AllowToBeInvoked\",\"Effect\":\"Allow\",\"Principal\":{\"Service\":\"s3.amazonaws.com\"},\"Action\":\"lambda:InvokeFunction\",\"Resource\":\"arn:aws:lambda:ap-northeast-1:123456789101:function:TestFunc:dev\",\"Condition\":{\"StringEquals\":{\"AWS:SourceAccount\":\"123456789101\"},\"ArnLike\":{\"AWS:SourceArn\":\"arn:aws:s3:::MyAwesomeBucket\"}}}"
}

Finally, executing the aws s3api command, I was able to put S3 event notification on MyAwesomeBucket

$ aws s3api put-bucket-notification-configuration \
 --bucket MyAwesomeBucket \
 --notification-configuration file://notification.json

I checked my lambda console and I can verify the S3 trigger is applied. :confetti_ball:

References:

https://forums.aws.amazon.com/thread.jspa?threadID=182758
https://docs.aws.amazon.com/lambda/latest/dg/with-s3.html
https://docs.aws.amazon.com/cli/latest/reference/s3api/put-bucket-notification-configuration.html

Terraform: Getting Started from Zero

2021-01-17T21:32:00+09:00

Terraform is an open-source infrastructure as code software tool that enables you to safely and predictably create, change, and improve infrastructure. - www.Terraform.io

Chapter 1: Introduction to IaC
Chapter 2: Terraform
Chapter 3: Learning HCL
Chapter 4: Code Re-Use for applying DRY
- 4.1 Understanding Terraform Modules
- 4.2 Distributing modules
Chapter 5: Collaboration with Terraform
- 5.1 Terraform backends
- 5.2 Terraform workspaces

Chapter 1: Introduction to IaC

1.1 Begin with a look back

[1960-1970] - Mainframe Computers
- Big machines
- Big facilities and using fork-lifts to move machines
- Automation was not that critical in this era
[1980-1990] - Client/Server World
- Things started to get little more complicated
- UNIX or PC servers running various OS
- These machines serve users using networking
[Mid 90's-2000] - Virtualization
- Software defines infrastructures
- Hardware were virtualized
- Clients do not have any hardware debt
Cloud Era
- No hardware to manage
- It's all software management problem
- All these things could be managed by API's, Scripts
- Manage all the infrastructures by the code

1.2 Why infrastructure as code?

Repeatability
- We can deploy the same thing over and over again
- We'll have a backup
- We can have multiple environments (dev, stage, prod)
Auditability
- We can see what changed, who changed and impact of change
- We can detect the problems easily
Change control
- Allows us to use VCS to control what to change
Collaboration
- Multiple people can work together on the same code base
- Example: Use GitHub to propose a change

1.3 Terminology

IaC - Infrastructure as Code
CM - Configuration Management (Example: Ansible, Puppet, etc)
IaaS - Infrastructure as a Service (Example: AWS, GoogleCloud, Azure)
PaaS - Platform as a Service (Example: Heroku, AWS EBS)
VCS - Version Control System
CI/CD - Continuous Integration/ Continuous Deployment (Delivery)
SDLC - Software Development Life Cycle
Declarative - Declare what you want
Imperative - How to get the desired state

1.4 Explore approaches to IaC

Imperative Approach
- Procedural Approach
- Answers How?
- Example Tools: AWS Cli, Python/Boto3
Declarative Approach
- Answers What?
- Example Tools: AWS CloudFormation, Terraform
- We can change things easily for example AWS instance type
- Tool does everything for us

Chapter 2: Terraform

2.1 What is Terraform?

Terraform is a cross-platform command line tool
Declarative IaC tool
- We define what we want and Terraform will create it for us
Also, an Online Service
- Provided through Terraform Cloud
- Remote execution, Secret Storage
Terraform consists
- Configuration: Configuration files to define what we want
- State: Like a Database which stores the current state and can be stored anywhere like in local PC, S3, GCS, Google Drive, etc
  - State is the Terraform view of the world
  - Terraform looks here instead of cloud provider
  - Terraform compares Config and State to create-destroy resource

2.2 Install Terraform

https://www.Terraform.io/downloads.html

╰─$ terraform -version
Terraform v0.14.4

2.3 Terraform Tour

Initialize Terraform

╭─sagar-giri@PCN-489 ~/IdeaProjects/HelloTerraform
╰─$ terraform init
Terraform initialized in an empty directory!

The directory has no Terraform configuration files. You may begin working
with Terraform immediately by creating Terraform configuration files.

Plan Terraform

╭─sagar-giri@PCN-489 ~/IdeaProjects/HelloTerraform
╰─$ terraform plan

Error: No configuration files

Plan requires configuration to be present. Planning without a configuration
would mark everything for destruction, which is normally not what is desired.
If you would like to destroy everything, run plan with the -destroy option.
Otherwise, create a Terraform configuration file (.tf file) and try again.

Apply

╭─sagar-giri@PCN-489 ~/IdeaProjects/HelloTerraform
╰─$ terraform apply                                                                                                 1 ↵

Error: No configuration files

Apply requires configuration to be present. Applying without a configuration
would mark everything for destruction, which is normally not what is desired.
If you would like to destroy everything, run 'Terraform destroy' instead.

Destroy

╭─sagar-giri@PCN-489 ~/IdeaProjects/HelloTerraform
╰─$ terraform destroy
Do you really want to destroy all resources?
  Terraform will destroy all your managed infrastructure, as shown above.
  There is no undo. Only 'yes' will be accepted to confirm.

  Enter a value: no

Destroy cancelled.
╭─sagar-giri@PCN-489 ~/IdeaProjects/HelloTerraform
╰─$ terraform destroy                                                                                               1 ↵
Do you really want to destroy all resources?
  Terraform will destroy all your managed infrastructure, as shown above.
  There is no undo. Only 'yes' will be accepted to confirm.

  Enter a value: yes

Destroy complete! Resources: 0 destroyed.

Help (Look for the Main commands section)

╰─$ terraform help
Usage: Terraform [global options] <subcommand> [args]

The available commands for execution are listed below.
The primary workflow commands are given first, followed by
less common or more advanced commands.

Main commands:
  init          Prepare your working directory for other commands
  validate      Check whether the configuration is valid
  plan          Show changes required by the current configuration
  apply         Create or update infrastructure
  destroy       Destroy previously-created infrastructure

All other commands:
  console       Try Terraform expressions at an interactive command prompt
  fmt           Reformat your configuration in the standard style
  force-unlock  Release a stuck lock on the current workspace
  get           Install or upgrade remote Terraform modules
  graph         Generate a Graphviz graph of the steps in an operation
  import        Associate existing infrastructure with a Terraform resource
  login         Obtain and save credentials for a remote host
  logout        Remove locally-stored credentials for a remote host
  output        Show output values from your root module
  providers     Show the providers required for this configuration
  refresh       Update the state to match remote systems
  show          Show the current state or a saved plan
  state         Advanced state management
  taint         Mark a resource instance as not fully functional
  untaint       Remove the 'tainted' state from a resource instance
  version       Show the current Terraform version
  workspace     Workspace management

Global options (use these before the subcommand, if any):
  -chdir=DIR    Switch to a different working directory before executing the
                given subcommand.
  -help         Show this help output, or the help for a specified subcommand.
  -version      An alias for the "version" subcommand.

Format the Terraform file

╰─$ terraform fmt

2.4 The very "Basics of Terraform"

Terraform uses HCL (HashiCorp Configuration Language)

The extension of file ends with .tf

Create main.tf inside a directory

"greeting" {
   value = "Hello Terraform."
 }

 provider "random" {}

Initialize Terraform

╰─$ terraform init

Initializing the backend...

Initializing provider plugins...
- Finding latest version of hashicorp/random...
- Installing hashicorp/random v3.0.1...
- Installed hashicorp/random v3.0.1 (signed by HashiCorp)

Terraform has created a lock file .Terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "Terraform init" in the future.

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

Plan it

╰─$ terraform plan

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:

Terraform will perform the following actions:

Plan: 0 to add, 0 to change, 0 to destroy.

Changes to Outputs:
  + greeting = "Hello Terraform."

------------------------------------------------------------------------

Note: You didn't specify an "-out" parameter to save this plan, so Terraform
can't guarantee that exactly these actions will be performed if
"Terraform apply" is subsequently run.

Apply it. You might have to type yes in the middle of it.

After applying it creates a .tfstate file, and you should not touch the .tfstate file.

╰─$ terraform apply

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:

Terraform will perform the following actions:

Plan: 0 to add, 0 to change, 0 to destroy.

Changes to Outputs:
  + greeting = "Hello Terraform."

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes


Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

Outputs:

greeting = "Hello Terraform."

See the output

╰─$ terraform output
greeting = "Hello Terraform."

Specify which output you want

╰─$ terraform output greeting
"Hello Terraform."

Chapter 3: Learning HCL

3.1 Overview of HCL structure

By conventions main.tf , outputs.tf and variables.tf are required files inside a directory

Sometimes the main.tf file gets too big. In this case we use the submodules.

Example main.tf with AWS as a provider:

# Terraform block is optional but it's a best practice to have it
terraform {
  # required_version is the Terraform version which we want to apply
  required_version = ">=0.14.0"
}

# Which cloud provider you want to use
provider "aws" {
  # region is required
  region = "ap-northeast-1"

  # Since I am putting this file in GitHub I do not EVER put my AWS credentials here.
  # Instead I use the environment variable or Terraform cloud or the ~/.aws/credentials file
  #access_keys = "my-access-key"
  #access_secret = "my-secret-key"
}

3.2 Terraform data sources and resources

Resources
- Resource is any object that you want to manage with Terraform. For example: S3, AWS EKS, GKE, VM's.
- Resources are defined in the resource block
- Declaring a resource tells Terraform that it should CREATE and manage the Resource described.
- If the resource already exists, it must be imported into Terraform state.
- Example: resource "aws_s3_bucket" "Terraform-bucket-2021" {}
- REF: https://www.Terraform.io/docs/configuration/blocks/resources/index.html
Data sources
- Data sources are resources that Terraform does not manage. For example: Availability Zones, Account ID
- We can use data source to get the availability zones to place EC2 instances in multiple AZ's
- REF: https://www.Terraform.io/docs/configuration/data-sources.html Example:

Create a S3 bucket in AWS using resource block.

terraform {
  required_version = ">=0.14.0"
}

provider "aws" {
  region = "ap-northeast-1"
}

resource "aws_s3_bucket" "Terraform-bucket-2021" {
  # bucket name should be unique.
  # If you don't provide any name, Terraform will create it for you and saves in outputs.tf file
  bucket = "Terraform-bucket-2021"
}

Plan and save the output to a file using command: terraform plan -out s3.tfplan . This will create a file named s3.tfplan with following output:

╰─$ terraform plan -out example.tfplan

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # aws_s3_bucket.Terraform-bucket-2021 will be created
  + resource "aws_s3_bucket" "Terraform-bucket-2021" {
      + acceleration_status         = (known after apply)
      + acl                         = "private"
      + arn                         = (known after apply)
      + bucket                      = "Terraform-bucket-2021"
      + bucket_domain_name          = (known after apply)
      + bucket_regional_domain_name = (known after apply)
      + force_destroy               = false
      + hosted_zone_id              = (known after apply)
      + id                          = (known after apply)
      + region                      = (known after apply)
      + request_payer               = (known after apply)
      + website_domain              = (known after apply)
      + website_endpoint            = (known after apply)

      + versioning {
          + enabled    = (known after apply)
          + mfa_delete = (known after apply)
        }
    }

Plan: 1 to add, 0 to change, 0 to destroy.

------------------------------------------------------------------------

This plan was saved to: example.tfplan

To perform exactly these actions, run the following command to apply:
    Terraform apply "example.tfplan"

Anything beginning with the + means Terraform will create this resource for us. In the output above, it's + create which means the resource doesn't exist yet and Terraform will perform create operation on the resource.

Now, apply the plan using command: terraform apply s3.tfplan

Once applied, following output will be seen:

╰─$ terraform apply example.tfplan
aws_s3_bucket.Terraform-bucket-2021: Creating...
aws_s3_bucket.Terraform-bucket-2021: Creation complete after 5s [id=Terraform-bucket-2021]

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

The state of your infrastructure has been saved to the path
below. This state is required to modify and destroy your
infrastructure, so keep it safe. To inspect the complete state
use the `terraform show` command.

State path: Terraform.tfstate

It means that the s3 bucket has been created.

Verify if the bucket exist using aws s3 ls command

╰─$ aws s3 ls
2021-01-16 19:42:52 Terraform-bucket-2021

Bucket is successfully created 🎉

Note: In real world, you ought to share the plan with your colleagues and there's an agreement between collegues before applying it.

3.3 Terraform Outputs

When we don't define some value (bucket name for example), Terraform will create the bucket name for us, and we can know the bucket details using Terraform outputs.

In above case, let's output the bucket information of terraform-bucket-2021

To do that we need to create a output block

resource "aws_s3_bucket" "Terraform-bucket-2021" {
  bucket = "Terraform-bucket-2021"
}

output "bucket_info" {
  # <syntax> value = resource_type[.]label_name
  value = aws_s3_bucket.Terraform-bucket-2021
}

Now, plan it terraform plan -out s3.tfplan . Here, the previous filename s3.tfplan will be replaced with the new fresh file.

Not apply the s3.tfplan using command terraform apply s3.tfplan

You'll see following output:

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

Outputs:

bucket_info = {
  "acceleration_status" = ""
  "acl" = "private"
  "arn" = "arn:aws:s3:::Terraform-bucket-2021"
  "bucket" = "Terraform-bucket-2021"
  "bucket_domain_name" = "Terraform-bucket-2021.s3.amazonaws.com"
  "bucket_prefix" = tostring(null)
  "bucket_regional_domain_name" = "Terraform-bucket-2021.s3.ap-northeast-1.amazonaws.com"
  "cors_rule" = tolist([])
  "force_destroy" = false
  "grant" = toset([])
  "hosted_zone_id" = "Z2M4EHUR26P7ZW"
  "id" = "Terraform-bucket-2021"
  "lifecycle_rule" = tolist([])
  "logging" = toset([])
  "object_lock_configuration" = tolist([])
  "policy" = tostring(null)
  "region" = "ap-northeast-1"
  "replication_configuration" = tolist([])
  "request_payer" = "BucketOwner"
  "server_side_encryption_configuration" = tolist([])
  "tags" = tomap({})
  "versioning" = tolist([
    {
      "enabled" = false
      "mfa_delete" = false
    },
  ])
  "website" = tolist([])
  "website_domain" = tostring(null)
  "website_endpoint" = tostring(null)
}

As you can see Apply complete! Resources: 0 added, 0 changed, 0 destroyed. because no resources were created even though our .tf file has resource block.

You can also get the output using terraform output command like this: terraform output bucket_info

Example:

List output of availability zones of region configured in your AWS configuration

First, create the data block because regions are not managed by Terraform and it's a data. And the output block. Since the output is from the data block, we define value as: data.<type>.<label>

data "aws_availability_zones" "available" {
  state = "available"
}

output "aws_availability_zones" {
  value = data.aws_availability_zones.available
}

Now, plan it terraform plan -out s3.tfplan and apply terraform apply s3.tfplan

You'll see following output:

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

Outputs:

aws_availability_zones = {
  "all_availability_zones" = tobool(null)
  "blacklisted_names" = toset(null) /* of string */
  "blacklisted_zone_ids" = toset(null) /* of string */
  "exclude_names" = toset(null) /* of string */
  "exclude_zone_ids" = toset(null) /* of string */
  "filter" = toset(null) /* of object */
  "group_names" = toset([
    "ap-northeast-1",
  ])
  "id" = "2021-01-16 10:53:54.916848 +0000 UTC"
  "names" = tolist([
    "ap-northeast-1a",
    "ap-northeast-1c",
    "ap-northeast-1d",
  ])
  "state" = "available"
  "zone_ids" = tolist([
    "apne1-az4",
    "apne1-az1",
    "apne1-az2",
  ])
}

Note: The output of AZ's can be used later on if we want to provision EC2 instances in multiple regions. We can use this output to do it so.

3.4 Interpolation in HCL

Interpolation is just the means of concatenating different values to create one value. In case of S3 bucket the value of S3 bucket has to be globally unique. Hence, inorder to create a bucket with unique name, we can use interpolation.

Example:

Create aws bucket which name has the accountID in it.

data "aws_caller_identity" "current" {}

resource "aws_s3_bucket" "bucket2" {
  bucket = "${data.aws_caller_identity.current.account_id}-bucket2"
}

Here data.aws_caller_identity.current.account_id is interpolated which is account_id + "-bucket2" got from the data block aws_caller_identity

3.5 Dependencies in Terraform

Terraform has the concept of implicit and explicit dependencies.

Whe Terraform processes the HCL configs, it evaluates the dependency graph, and it'll create dependencies before it creates the dependent resource. If there is not any dependencies, Terraform will create resource in parallel which will give performance boost.

Example:

Implicit dependency resolve when the dependent resource already exists.

resource "aws_s3_bucket" "bucket3" {
  bucket = "${data.aws_caller_identity.current.account_id}-bucket3"
  tags = {
    # I already have bucket2 available so this is implicit
    dependency = aws_s3_bucket.bucket2.arn
  }
}

Explicit dependency when the dependent resource does not exist yet, and we need to wait for it to be created first. (analogy depends

resource "aws_s3_bucket" "bucket4" {
  bucket = "${data.aws_caller_identity.current.account_id}-bucket4"
  # Explicit
  depends_on = [
    aws_s3_bucket.bucket3
  ]
}

To see the dependencies, generate the dot file using terraform graph > graph.dot

And paste the content in online tool https://dreampuf.github.io/GraphvizOnline/ to see the output.

The terraform graph command is only used for debugging purpose.

3.6 Variables and Locals

Variables are ways to get input to the Terraform config

Ways to provide variables:

Using command line while applying
Specify them in files such as .tfvars or .auto.tfvars for complex data types
Using environment variables but all the environment variable name should have prefix of TF_VAR_ In Terraform config, we use variable block to define expected variables.

Example:

Create a S3 bucket, but the bucket name should be provided by the user

variable "bucket_name" {
  type = string

  # `default` is optional. If default is omitted, then value must be supplied
  # default = "My_Bucket1234"
}

resource "aws_s3_bucket" "bucket5" {
  bucket = var.bucket_name
}

And when plan is executed, the bucket name is expected as an input value:

$ terraform plan -out var.tfplan
var.bucket_name
  Enter a value: MyPiBucket31415

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # aws_s3_bucket.bucket5 will be created
  + resource "aws_s3_bucket" "bucket5" {
      + acceleration_status         = (known after apply)
      + acl                         = "private"
      + arn                         = (known after apply)
      + bucket                      = "MyPiBucket31415"
      + bucket_domain_name          = (known after apply)
      + bucket_regional_domain_name = (known after apply)
      + force_destroy               = false
      + hosted_zone_id              = (known after apply)
      + id                          = (known after apply)
      + region                      = (known after apply)
      + request_payer               = (known after apply)
      + website_domain              = (known after apply)
      + website_endpoint            = (known after apply)

      + versioning {
          + enabled    = (known after apply)
          + mfa_delete = (known after apply)
        }
    }

Plan: 1 to add, 0 to change, 0 to destroy.

Local values

Local values make Terraform configs more readable.

If we want to use same value multiple times in the code, it's better to save it as a local value.

We can have multiple locals, but the names should be unique because names are global.

Example:

locals {
  aws_account = "${data.aws_caller_identity.current.account_id}-${lower(data.aws_caller_identity.current.user_id)}"
}

resource "aws_s3_bucket" "bucket6" {
  bucket = "${local.aws_account}-bucket6"
}

Note: Here the lower() is a Terraform function. See more here: https://www.Terraform.io/docs/configuration/functions.html

3.7 Conditionals and the Count property

Count is the meta attribute of Terraform

Count means instead of creating 1 resource, you can define n number of count to replicate the same resource.

Example:

Create 2 buckets

resource "aws_s3_bucket" "bucketX" {
  count = 2
  bucket = "${local.aws_account}-bucket${count.index+7}"
}

Q. What happens when count=0 ?

Terraform will destroy all resource if created or not create anything if it's not created yet.

We can also replicate resources using for_each** **

locals {
  buckets = {
    bucket101 = "mybucket101"
    bucket102 = "mybucket102"
  }
}

resource "aws_s3_bucket" "bucketIterator" {
  for_each = local.buckets
  bucket = "${local.aws_account}-${each.value}"
}

output "bucketIterator" {
  value = aws_s3_bucket.bucketIterator
}

We can also define list instead of map:

locals {
  buckets = [
  "mybucket101",
  "mybucket102"
  ]
}

resource "aws_s3_bucket" "bucketIterator" {
  # Since buckets is a list, we need to convert to set using toset()
  for_each = toset(local.buckets)
  # Here we can use either .key or .value
  bucket = "${local.aws_account}-${each.key}"
}

output "bucketIterator" {
  value = aws_s3_bucket.bucketIterator
}

count vs for_each

count is generally used for toggle switch when in development phase
for_each is more powerful, as we can use map in for_each hence, better to use for_each in prod

3.8 Data types and operators

REF: https://www.Terraform.io/docs/configuration/expressions/types.html

locals {
  a_string = "This is a String"
  a_number = 3.1415
  a_boolean = true
  a_list = [
    "element1",
    true,
    [1, 2, 3]
  ]
  a_map = {
    key = "value"
    nums = [10, 20, 30]
    is_active = true
    configs = {
      instance_type = "t2.micro"
      vpc_enabled  = true
    }
  }

  # Operators
  operators = (3+3)-(3*3/3)

  # Logical
  t = true && true
  f = true || false

  # Comparison
  gt = 3 > 2
  lt = 4 < 9
  eq = 4 == 4
  neq = 4 != 5
}

Example:

Terraform config to create n number of buckets where n not more than 5** **

variable "bucket_count" {
  type = number
}

data "aws_caller_identity" "current" {}

locals {
  min_bucket_count = 5
  # Conditional operators similar to ternary operator
  num_buckets = var.bucket_count > 5 ? local.min_bucket_count: var.bucket_count
  aws_account = "${data.aws_caller_identity.current.account_id}-${lower(data.aws_caller_identity.current.user_id)}"
}

resource "aws_s3_bucket" "buckets" {
  count = local.num_buckets
  bucket = "${local.aws_account}-buckets${count.index+7}"
}

3.9 Functions

REF: https[://www.Terraform.io/docs/configuration/functions.html](https://www.Terraform.io/docs/configuration/functions.html)

Example:

locals {
  ts = timestamp()
  current_month = formatdate("MMMM", local.ts)
  tomorrow = formatdate("MMMM", timeadd(local.ts, "24h"))
  upper = upper("lowercase")
  lower = lower("UPPERCASE")
}

output "func" {
  value = "${local.ts} ${local.current_month} ${local.tomorrow} ${local.upper} ${local.lower}"
}

Output:

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

Outputs:

func = "2021-01-16T15:36:41Z January January LOWERCASE uppercase"

3.10 Iterations in collections

HCL uses for syntax dfor iterating over list values

Example:

locals {
  my_list    = ["one", "two", "three"]
  upper_list = [for item in local.my_list : upper(item)]
  upper_map  = { for item in local.my_list : item => upper(item) }

  # Filtering
  n     = [1, 2, 3, 4, 5, 6, 7, 8, 9]
  evens = [for i in local.n : i if i % 2 == 0]
}

output "my_list" {
  value = local.my_list
}
output "upper_list" {
  value = local.upper_list
}

output "upper_map" {
  value = local.upper_map
}

output "evens" {
  value = local.evens
}

Output after apply:

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

Outputs:

evens = [
  2,
  4,
  6,
  8,
]
my_list = [
  "one",
  "two",
  "three",
]
upper_list = [
  "ONE",
  "TWO",
  "THREE",
]
upper_map = {
  "one" = "ONE"
  "three" = "THREE"
  "two" = "TWO"
}

3.11 Directives and heredocs

heredocs
- is a multiline string that is used to create an inline JSON documents
- They are used in outputs, variables description and inline documents Example:

locals {
  count = 0
  nums  = [1, 2, 3]
}
output "heredocs" {
  # EOT means EOF
  value = <<-EOT
    This is the `heredoc`.
    This is multiline String.
    Used for writing documentations.
  EOT
  # Dont forget EOT at the end
}

output "directive" {
  value = <<-EOT
    We can use directive in heredoc.
    %{if local.count == 0}
    The count is 0, destroying everything...
    %{else}
    The count is ${local.count}
    %{endif}
  EOT
}

output "iterated-directive" {
  value = <<-EOT
    Directive can also be iterated.
    %{for num in local.nums}
    ${num}
    %{endfor}
  EOT
}

Output after apply:

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

Outputs:

directive = <<EOT
We can use directive in heredoc.

The count is 0, destroying everything...


EOT
heredocs = <<EOT
This is the `heredoc`.
This is multiline String.
Used for writing documentations.

EOT
iterated-directive = <<EOT
Directive can also be iterated.

1

2

3


EOT

3.12 Clean up

Since we created multiple buckets and all, it's time to clean up.

Even though the buckets don't cost any, it's wise to clean up since we're just playing with Terraform in development mode.

First look for states that's tracked by Terraform using terraform state list . If it shows bunch of outputs of your bucket, execute terraform destroy which will destroy all resources that were created.

Chapter 4: Code Re-Use for applying DRY

4.1 Understanding Terraform Modules

What is a module in Terraform?

All Terraform is in a module. The top level module is called the Root module
Modules are just a directory that contains Terraform file, and they can be nested
Modules can be imported which are already available
Modules written by others can be get here https://registry.Terraform.io
Module helps in implementing DRY principle
Module naming convention (if you want to publish to Terraform registry)
- terraform-<PROVIDER>-<NAME>
- <NAME> can contain hyphens Module layout

├── Terraform-aws-s3
│   ├── README.md
│   ├── datas.tf
│   ├── locals.tf
│   ├── main.tf
│   ├── outputs.tf
│   ├── variables.tf
│   └── versions.tf

You can use https://github.com/QuiNovas/cookiecutter-Terraform-module to automate the process of creating these files

To use module

# Module located in local filesystem
module "local-module" {
    source = "/path/to/module"
}

# Module which is published
module "published-module" {
    source = "rojopolis/Terraform-aws-lambda-python-archive"
    version = "1.2.3"
}

# Module which is in GitHub
module "github-module" {
    source = "https://github.com/rojopolis/Terraform-aws-lambda-python-archive"
}

Play with https://github.com/rojopolis/Terraform-aws-lambda-python-archive/ to know about modules.

4.2 Distributing modules

Name the module properly terraform-<PROVIDER>-<NAME>
Should be in public repository in GitHub.
Should have license and examples
Repo should be released with a tag
Sign-in to https://registry.Terraform.io
Select the repo which is named correctly
Done 🎉
Profit 🏦

Chapter 5: Collaboration with Terraform

5.1 Terraform backends

Backend is a shared storage medium that stores state files
Backends officially supports blocking
Blocking is required inorder to prevent multiple user's updating resources at the same time
Most of the providers support blocking
We can use S3 as a backend provider
Backend provider is not in Terraform Example:

terraform {
  required_version = ">=0.14.0"
  backend "s3" {
    # bucket should already exist
    bucket = "my-Terraform-backend"
    key = "sagar-Terraform-resources"
    region = "ap-northeast-1"
    # dynamo db table also should already exist
    dynamodb_table = "sagar-Terraform-lock"
  }
}

You must init again inorder to change the backend.

Locks are stored in dynamodb table. Whoever acquires the lock first has the precedence of getting the resource.

We can also have remote backend

  backend "remote" {
    hostname = "app.Terraform.io"
    organization = "beBit"
    workspace = {
      name = "hello-Terraform"
    }
  }

If we change the S3 backend to remote backend, we need to do terraform init again which will ask us if we'd like to move the state file from S3 to the remote backend.

5.2 Terraform workspaces

Workspace helps us to create different environment with the same sets of Terraform configurations.
You can view the list of your Terraform workspaces using the command terraform workspace list
The default workspace name is called default
You can not delete the default workspace
Execute terraform workspace --help to view the workspaces sub-commands

Create a new workspace:

╰─$ terraform workspace new staging
Created and switched to workspace "staging"!

You're now on a new, empty workspace. Workspaces isolate their state,
so if you run "terraform plan" Terraform will not see any existing state
for this configuration.

List the workspaces:

╰─$ terraform workspace list
  default
* staging

Where the * is the current one, and we can also use terraform workspace show to get the current workspace.

Switch the workspace back to default

╰─$ terraform workspace select default                                                                                                                                                                                              1 ↵
Switched to workspace "default".

Deleting the workspace:

╰─$ terraform workspace delete staging
Deleted workspace "staging"!

Note: The workspace should be clean i.e. destroyed before deleting it. Learn more here: https://www.Terraform.io/docs/state/workspaces.html

Instrument your Java Code with Micrometer, Prometheus, and Grafana.

2020-12-31T15:33:00+09:00

Micrometer provides a simple facade over the instrumentation clients for the most popular monitoring systems, allowing you to instrument your JVM-based application code without vendor lock-in. Think SLF4J, but for metrics. - https://micrometer.io/

I'm assuming you have some Java code, and you'd like to instrument it (measure how well it's performing, what is the heap space in current state, are there any exceptions. etc)

With this 3 steps, you'll be up and running in no time.

Step 1: Setup the development environment

Here I am using Docker and Docker compose to set up the environment for simplicity purposes. If you do not have these in your machine, I highly recommend you to install it so that you can follow along.

The full source code is available here: https://github.com/girisagar46/prome-java

First and foremost, I will set Prometheus and Grafana in my docker-compose.yml file.

The docker-compose.yml file looks like this:

version: "3.8"

services:
  prometheus:
    image: prom/prometheus
    container_name: prometheus
    volumes:
      - ./monitoring/prometheus.yml:/etc/prometheus/prometheus.yml
      - ./data/prometheus:/data
    command:
      - "--config.file=/etc/prometheus/prometheus.yml"
      - "--storage.tsdb.path=/data"
    ports:
      - "9090:9090"
    restart: always

  grafana:
    image: grafana/grafana
    volumes:
      - ./grafana:/var/lib/grafana
      - ./grafana/datasources:/etc/grafana/datasources
      - ./grafana/dashboards:/etc/grafana/dashboard
    ports:
      - "3000:3000"
    restart: always
    environment:
      - GF_SECURITY_ADMIN_USER=admin
      - GF_SECURITY_ADMIN_PASSWORD=admin
      - GF_USERS_ALLOW_SIGN_UP=false

Since we're mounting /monitoring/prometheus.yml into prometheus service, let's create the directory monitoring and create a file prometheus.yml inside the monitoring directory.

The contents of the prometheus.yml looks like this:

Once the docker-compose.yml is ready, start the services with docker-compose up

And when the services are up, visit http://localhost:9090/. You'll see something like this:

And if you navigate to http://localhost:9090/targets (Status dropdown -> Targets), you'll see something like this:

As you can see there's an error message that says:

Get "http://host.docker.internal:8080/metrics": dial tcp 192.168.65.2:8080: connect: connection refused

because our Java application is not running at the moment. We'll fix this a little letter.

If you visit http://localhost:3000/, you'll see the Grafana login page:

Use the default username: admin and password admin to get inside the Grafana dashboard. When asked to change password, skip it as it's just a local development environment. Obviously in production, this has to be more sure by enabling the login by Gmail or any other OAUTH mechanism.

Now you need to add Prometheus data source so that our Grafana can get the piece of the metrics we're exposing. To add the datasource directly go to http://localhost:3000/datasources

Then click on Add data source button and fill up the form as shown in the screenshot below.

Step 2: Expose metrics from your Java application

Create MetricService class which will provide the singleton instance of PrometheusMeterRegistry

Create HTTP metric endpoint where prometheus can go to scrape the metrics. As you can see in the prometheus.yml file we've defined metrics_path: "/metrics". So, let's create HTTP endpoint. I'll add this in the Actuator.java file:

Create the Main class (the entry point to our application)

At this point, the source code directory looks like this:

Now if you run your application (Execute Main.java) and go to http://localhost:8080/metrics, you'll see following output:

...
# HELP jvm_gc_live_data_size_bytes Size of long-lived heap memory pool after reclamation
# TYPE jvm_gc_live_data_size_bytes gauge
jvm_gc_live_data_size_bytes 0.0
# HELP jvm_buffer_count_buffers An estimate of the number of buffers in the pool
# TYPE jvm_buffer_count_buffers gauge
jvm_buffer_count_buffers{id="mapped - 'non-volatile memory'",} 0.0
jvm_buffer_count_buffers{id="mapped",} 0.0
jvm_buffer_count_buffers{id="direct",} 2.0
# HELP jvm_memory_used_bytes The amount of used memory
# TYPE jvm_memory_used_bytes gauge
...

Which means your Java application is exposing metrics for the Prometheus to grab.

Step 3: Visualize the exposed metrics in Grafana

Since, we're only exposing JVM metrics for now let's see those exposed metrics on Grafana through a sweet dashboard.

Let's import the Micrometer grafana official dashboard. The ID of the grafana dashboard is 4701. So, let's import the dashboard to Grafana.

Click on the Import button which is shown in the screenshot below.

Type 4701 in the input field Import via grafana.com and click Load button.

Once, you click load, you'll see this page. Follow the 3 steps shown in the screenshot.

Now your dashboard is imported, change the date range to view recent data.

Congratulations!! your dashboard is fully functional and ready to use.

You can play with Prometheus and Grafana as much as you like because everything is running in local. Try creating some alerts, Prometheus rules, write some PROMQL and expose some other metrics with Micrometer.

If you want to take this to production, you can do it in various ways. Using K8s, using 3rd party service, hosting your own, using AWS managed service, etc. There are plethora of options, and you can choose any depending on your cost and requirements.

Going further

Now the basic setup is done, the metrics world does not stop here. It's a vast ocean. Here are some further additional resources which will help you to go forward.

https://micrometer.io/docs/concepts
https://prometheus.io/docs/prometheus/latest/getting_started/
https://prometheus.io/docs/prometheus/latest/querying/basics/
Video: PromQL for Mere Mortals
Video: The 4 Types Of Prometheus Metrics
https://www.robustperception.io/blog
https://grafana.com/tutorials/
https://github.com/prometheus/client_java
https://www.reddit.com/r/PrometheusMonitoring -> Reddit community
https://www.reddit.com/r/grafana -> Reddit community
https://slofile.com/slack/micrometer-metrics -> Official Slack channel for Micrometer (Join and ask any questions related to Prometheus, Grafana and Micrometer)

The Insides of Kubernetes and how it actually works.

2019-09-06T14:30:00+09:00

"It groups containers that make up an application into logical units for easy management and discovery. Kubernetes builds upon 15 years of experience of running production workloads at Google, combined with best-of-breed ideas and practices from the community." — Kubernetes, Production-Grade Container Orchestration

What is Kubernetes?

In a typical micro service architecture using k8s, we have these things:

In a simple term, k8s is composed of Nodes and Pods.
Combination of nodes creates a cluster aka kubernetes cluster.
You can consider nodes as a physical or virtual server.
There are two kinds of nodes:
1. Master node (Controls what's going on)
2. Node aka Minions (That does the actual work)
Each node has pods.
Each pods have one or more container running inside it.

In the birds eye view, Kubernetes architecture can be viewed as:

Nodes:

A node is a worker machine in Kubernetes. Learn more here

There are two types of nodes:

Master node
Worker node

Master Node

There can be one or more master node in a k8s cluster.
Best practice is to make master node free of workloads so that master node can only look after cluster.
You can think master node as a shepherd who who tends and rears sheep. (worker nodes)

What's inside a master node?

Remember a Node is a machine. So the base layer is the hardware layer. It can be bare metal, a virtual server, EC2 instance or even your personal laptop (if you are in local environment)
Then the layer after that is OS layer. And remember, K8s only works in Linux OS. Kubernetes is platform agnostic in terms of hardware lever, but it needs Linux OS to operate.

Components of Master node:

Scheduler: Watches for new pods and assign work to that pod.
Controller: This watches for changes in the cluster
Key Value Store: This is the persistent storage service. It uses etcd to maintain it's state. So in a master node, only this component is stateful.
API server: This exposes the REST API for developers and clients to interact with k8s cluster. Developers deal directly with api server via command line tool provided by kubernetes called kubectl. API server can also be linked with authentication service to provide extra layer of security for k8s cluster.

Worker Node

Worker node (also used to be called as minion) are also a machine that does the actual work. For example, updating a shopping cart or authenticating a user in your web application.

What's inside a worker node?

Kubelet: They are main k8s agent They instantiate pod, register node with a cluster and operate on port 10255 Kubelet just report back to master node if a container dies. The endpoint provied by kubelet are: /spec, /healtz , /pods
Container Engine: They do container management (pulling image, running container)
Kube Proxy Responsible for K8s networking This makes sure that each pod get unique IP address for network communication If more than 1 container is running inside a pod, then all pod share the same IP address but the port is different This also does load balancing across all pods in a service (#service section)

Declarative model and Desired state

Kubernetes work in declarative model. Which basically means that we provide the APIServer (which is inside the master node) some specification to describe the desired state. The specification is written in JSON or YAML format. Once we provide the specification declaration, then it's the responsibility of Kubernetes to maintain the desired state.

Declarative Model

We give manifest to do something. For example: At each time, there must be 5 containers running.

Desired State

This is done by k8s internally. For example: Since we declared to run 5 replicas, but at a particular instant of time, if only 4 are running (which don't match with declaration), then k8s kicks in and tries to maintain the desired state from the declaration manifest (written in YAML or JSON which is provided to master node)

Pods

Pods are atomic unit of deployment in k8s world and they exists inside a node
Containers always runs inside a pod
Pods can have multiple containers (only in advanced cases, but ideally one)
If two containers are running inside a same pod, then they share same memory space and volumes. This running more than one containers are useful if we want tightly coupled architecture.

Here in above figure, these two containers share the same DB and also same file system and also the same IP address.

In detail:

Containers inside the pod share the same IP address of the pod, but the port of each container is different. Intra container communication happens inside the pods by localhost.

For loosely coupled system, the containers from two different pods can talk to each other through network communication.

Pods are also the unit of scaling in kubernetes world. We scale pods up and down (add or remove pods) while scaling.
Pods are atomic. Which means we just create replicas of pods. We don't wait for the containers to be up before creating a pod. We just create create replicas of already up and running pods.
Pod lifecycle is: pending -> running -> success or failed -> die
There is no half (partially up) pod. Either it's up or not.
Ideally 1 pod gets scheduled to one node

Pod Lifecycle:

First we provide the manifest file to API server in k8s master node
API server receives manifest file and tries to run container and pod goes to pending mode
In pending mode, image is downloaded
Then container inside the pods are run
Then it goes to running mode
And if the purpose of pod is full filled, then when pod is gracefully stopped, it goes to succeed state

Note: If for some whatever reason, if some problem arises in pending mode, then the pod goes to failed mode.

Replication Controller

But, in real world, we really don't work with pods directly. We use ReplicationController to work with pods. ReplicationController sits on top of pod and it instantiating the pod(s)

Services

Pods are volatile. They are not persistent.
Pods go alive and die rapidly. When they die and generate their IP address changes rapidly. If pods die or new pods are added it's very difficult to manage the IP address and connection between these pods. To address this issue, we make use of service.
Service sits between pods layer and manages the network connection between the pods
They also do load balancing tasks
It's the job of service to send traffic to healthy pods and it uses TCP protocol by default
When our app is deployed in Kubernetes, we want it to be used by public. Hence, Service also helps to expose our application to public world (in the internet)
Service gets single DNS, static IP, and never changing port.
Services are stateful and they never change once defined.
Services and pods are tied together by label
Our client can talk to service in-order to get access to the application hosted in the pods

Deployments

These are the one which we interact the most
They sit above the replication controller. And in the world of k8s deployment, replication sets are called replica set
Deployments are:
Self documenting
Write spec once, deploy many
Versioned
Simple rolling updates and rollbacks
Proper first class REST objects in k8s api
Defined in standard manifest file
Deployed via API server (in master node)
Add features to replication controller

Build AWS CodePipeline CI/CD for your Django Application

2019-02-15T22:50:00+09:00

Assumption: For this tutorial, I am assuming you already have an EC2 instance where you've configured and deploy a Django application.

In previous tutorial, you learnt about syncing your BitBucket repository to AWS CodeCommit. Now, since your source code is already into AWS CodeCommit, you can now setup AWS CodePipeline for CI/CD.

This tutorial will be a little bit long and I will guide you to setup AWS CodePipeline for your Django application step by step with no step skipped.

Okay, Let's begin!

Step 0 (True programmer starts counting from zero ;) )

Go to AWS Console and select AWS CodePipeline which is under the Developer Tools section. Or you can also search for AWS CodePipeline by typing the name under Find Services

AWS Console

Step 1

Once you are in AWS CodePipeline click on Create pipeline button. You'll be then redirected to Choose pipeline settings

Give yor pipeline a name under the Pipeline name input field.
Select New service role under the Service role section.
Under Role name section, you'll be provided a default role name. But, you may wish to give a proper name which you can remember. In that case, provide your own Role name.
Check the checkbox that says Allow AWS CodePipeline to create a service role so it can be used with this new pipeline
Under Artifact store, select Custom location. And here's a tricky part. Artifacts are files that is generated after the CI part is completed. These artifacts need to be stored somewhere. In CodePipeline's case, it's the S3 bucket. If you choose Default location, your artifacts will be stored in the S3 location defined by the CodePipeline itself. In my case, I choose to store my artifacts in my own S3 bucket. Here's a tutorial if you don't know how to create a S3 bucket.
After filling up this form, click the Next button.

Step 2

After clicking Next at the end of Step 1, you'll see another form where you'll Add source

Select Source provider. In this case, I am selecting AWS CodeCommit because I've already synced my BitBucket repository to AWS CodeCommit.
Select your Repository name from the drop down option.
Select the Branch name from the dropdown option. This Branch name indicates triggering of CI when a commit is made to this branch.
On Change detection options select the recommended option Amazon CloudWatch Events
Click Next and start Step 3

Step 3

Select Build provider as AWS CodeBuild
Now, you need to setup a Project for AWS CodeBuild. Click on Create project button as shown in the picture.
When you click that Create project, a new window will open where you will setup a CodeBuild project. Let's call it Step 4 for now. Now, let's jump to Step 4

Step 4

I'll be breaking down this step into small chunks.

Step 4-1 (Create build project)

Give your project a name under Project name input field.
Give a project Description if you like. It's totally optional but it's a good practice to describe what it does.
You can add tags to your project but, I am leaving it blank for now.

Step 4-2 (Environment)

Just below Create build project, you'll see Environment configuration
Select Managed image under Environment image section
Select Operating system as Ubuntu
Runtime Python (Remember, we are doing CI for Django application)
aws/codebuild/python:3.6.5 as Runtime version
Select Always use the latest image for this runtime version under the Image version section
I am not selecting Privileged option. It's description is self explanatory. If you want more details, see the AWS CodeBuild official documentation
Choose New service role under the Service role section. We are selecting New service role because we haven't created any role yet for the code build from IAM. So, we'll just create a new one.
Give a proper Role name so that you can remember and if you see it in the IAM console, you'll just instantly recognize that this is a role or AWS CodeBuild
I am leaving Additional configuration as empty. This is where you set up Environment variables, system configuration and so on. You can setup some Additional configuration if you like

Step 4-3 (Buildspec)

Select Use a buildspec file under Build specifications
Under the Buildspec name section, you can add your own custom name if you have custom buildspec yml file but that's completely optional
Fill out the Logs section. Your build logs are all streamed to AWS CloudWatch if you want. You can also use S3 to store your logs. In my case, I chose AWS CloudWatch and gave a Group name and Stream name
Now click on Continue to CodePipeline button and you will be redirected to that page (Step 3)

Step 5

After Step 4-3, you'll be redirected to that Add build stage page and you'll see your newly created build project. If you don't see it there, then just refresh the page.

Now, click the Next button.

Step 6

This step is called Add deploy stage
First, choose the deploy provider. Select AWS CodeDeploy
Under the Application name section, you should choose the CodeDeploy application. Since, we haven't yet created the CodeDeploy application, let's just create one. Don't close this browser tab. Just open new browser tab and go to Step 7

Step 7

To create the AWS CodeDeploy application, first go to the CodeDeploy section from AWS Console and click on the Create application button. You'll then see Application configuration form. Give your application a name Application name and choose the Compute platform as EC2/On-premises since we are deploying our application on our EC2 server.

After you create the CodeDeploy application, we need to divert a little bit to setup a New service IAM Role and a Deployment group. We'll continue from Step 6 after setting up the deployment group. For now, let's go to step 8. We'll revisit Step 6 after this.

Step 8 (Create New service IAM Role for CodeDeploy)

Go to the AWS IAM Console
Click on Roles on the sidebar menu
Then click on Create role button.
Select CodeDeploy and also CodeDeploy under Select your use case
You'll see something like this:

Click Next: Tags
Give your role a tag with Name and Value if you like
Click Next: Review button
Give your role a name. We'll be using this role while creating deployment group.
Then click on Create role button

Step 9 (Create deployment group)

Under CodeDeploy > Application section, you'll see the CodeDeploy application name hello-world-codedeploy-application that you just created in Step 7
Click on the application link and under the Deployment groups tab, you'll see a button that says Create deployment group. Click that.
You'll see a form like this:

Under the Service role section, select the service role that you just created on Step 9
Under Environment configuration section, select Amazon EC2 instances. Then add tags. These tags are actually the EC2 instance tag that you've already setup for Django application deployment
Fill out the form and click Create deployment group button
Your deployment group will be created and you'll see a page just like this:

Step 10

Now go to Step 6 where you've left off. Since, we've just created CodeDeploy application and Deployment group. Select those values in specific fields. If they are not populated under the select dropdown, just refresh the page and you'll be fine.
Click Next button and go to Step 11

Step 11

This is probably the last step of our code pipeline setup. After clicking the Next button from Step 10, you'll see a CodePipeline Review page.

Just go through it and check if everything is setup correctly
If you need to change something, go to previous section. And if you think everything is fine, click on Create pipeline button

Step 12

You'll see a success page something like this.

Step 13 (Create buildspec.yml file)

At this point, your pipeline will obviously fail in the Build section because we haven't setup the buildspec.yml file.

Here's a buildspec.yml file that I setup for my django project pipeline. Take a reference and change accordingly.

**I'll be writing a separate post on what's going inside the buildspec.yml file. But, for now just copy-paste this in your buildspec.yml file and push it to your repository.

Step 14 (Create appspec.yml file)

Your CodePipeline will still fail on Deploy part because we haven't still setup an appspec.yml file yet. This file is used by the CodeDeploy to deploy our application to the EC2 instance.

Here's the configuration for my appspec.yml file that I setup for my django project pipeline. Take a reference and change it accordingly.

After this, your CodePipeline should be Green.

From Bitbucket Pipeline to AWS CodeCommit

2019-02-08T22:08:00+09:00

Setup codecommit IAM user

Create new user from AWS IAM console
Give two permission to that new codecommit user:
- AWSCodeCommitFullAccess and
- AmazonS3FullAccess
Open that new user from IAM console. Under user's section and go to Security Credentials tab
Under SSH keys for AWS CodeCommit upload new SSH public key.
To do this open up your terminal and add following commands:
- $ ssh-keygen -f ~/.ssh/codecommit_rsa
- And inside .ssh folder you’ll find codecommit_rsa and codecommit_rsa.pub file
- Open codecommit_rsa.pub in your editor and copy the contents
- Click Upload SSH public Key button and in the popup and paste the contents of codecommit_rsa.pub file and click Upload SSH public Key

After uploading this, you’ll get the “SSH key ID”

Open your terminal and edit this file: ~/.ssh/config
Set the values as described below and save the file.

Host git-codecommit.*.amazonaws.com
User Your-IAM-SSH-Key-ID-Here
IdentityFile ~/.ssh/codecommit_rsa

Now, create new codecommit repository:
- AWS Console > Code Commit > Getting started > Create repository
- Give your repository a name and description and hit the create button.

Setup BitbucketPipeline

Open your repository in BitBucket on which you want to setup pipeline. [Note: You must have admin access to that repository]
After opening your repository go to settings > Pipeline Settings > Enable Pipeline
Then go to Pipeline > Settings -> Repository variables and set these values:

Description of each of these variables:

CodeCommitConfig -> 64 bit encoded version of the contents inside your ~/.ssh/config file. Generate this by doing: bash $ base64 ~/.ssh/config And copy the output to the value field of CodeCommitConfig
CodeCommitHost: your AWS codecommit host. Blurred part is the AWS region
CodeCommitKey: 64 bit encoded version of the contents inside your private key at ~/.ssh/codecommit_rsa. Generate this by doing:

bash $ base64 ~/.ssh/codecommit_rsa And copy the output to the value field of CodeCommitKey
CodeCommitRepo: The link of repository that you just created
CodeCommitUser: Your-IAM-SSH-Key-ID

Create bitbucket-pipelines.yml

Create bitbucket-pipelines.yml file with following contents:

pipelines:
  default:
    - step:
       script:
         - echo $CodeCommitKey > ~/.ssh/codecommit_rsa.tmp
         - base64 -d ~/.ssh/codecommit_rsa.tmp > ~/.ssh/codecommit_rsa
         - chmod 400 ~/.ssh/codecommit_rsa
         - echo $CodeCommitConfig > ~/.ssh/config.tmp
         - base64 -d  ~/.ssh/config.tmp > ~/.ssh/config
         - set +e
         - ssh -o StrictHostKeyChecking=no $CodeCommitHost
         - set -e
         - git remote set-url origin ssh://$CodeCommitRepo
         - git push origin $BITBUCKET_BRANCH

Make sure your indention are correct. You can use yml validator (https://jsonformatter.org/yaml-validator) to do this.
Push the bitbucket-pipelines.yml to any branch, then your pipeline will run and sync the changes to AWS Codecommit repository

Docker 101 (Part 1)

2018-12-02T00:01:00+09:00

Step 1: Install Docker

Docker can be installed in pretty much any OS platform, be it Windows, Mac or Linux. Since I use a Linux machine this whole article is based on the Linux system. For installing docker in Windows and Mac, refer the official documentation.

Ok, enough talk. Let's get started.

Docker is a company as well as a software. And docker is released every month which is called edge release. And every quarter, a stable version of docker is released.

One of the ways to install docker on a Linux machine is by using the apt package manager but, the problem with this is approach is that you'll not get the current release of docker which sucks IMO because in every new release new features are baked in. So, I advise you to not install via the apt package manager (if you are in Debian ubuntu variant of Linux) or any other package manager for other Linux distributions.

There are two versions of docker available to download and install. One being free aka Community Edition and another paid version Enterprise Edition. For learning purpose, we'll install Community Edition docker. Hey! who does not like free software? ;)

Okay, to install the latest and greatest version of community edition docker, fire up your terminal and execute the following command:

$ wget -qO- https://get.docker.io/ | sh

What this command does is that it pulls the installation script from the https://get.docker.io/ and execute as a shell command. After the installation is complete, check the version of docker that is installed in your system by:

$ docker --version

You'll see that the current stable release of the docker has been installed.

Docker version 18.09.0, build 4d60db4

Step 2: Configure Docker

Couple of things you need to configure after installing Docker. At the time of installation, docker is attached to the root user and every time you need to execute a docker command, you have to pass the sudo command. This situation is not idle for many cases. To fix this, we should bind docker to another user. Execute the following command to attach docker to another user:

$ sudo usermod -aG docker ubuntu

Note: Replace ubuntu with your username

After this, install docker machine. Docker machine should also be the latest version. For Windows and Mac user, docker machine is already installed while installing Docker. To install docker machine, head over to this link https://github.com/docker/machine/release. You'll see the command which is like this...

$ curl -L https://github.com/docker/machine/releases/download/v0.16.0/docker-machine-`uname -s`-`uname -m` >/tmp/docker-machine && chmod +x /tmp/docker-machine && sudo cp /tmp/docker-machine /usr/local/bin/docker-machine

Execute the above command and check if docker machine is installed or not by executing

$ docker-machine version
docker-machine version 0.16.0, build 702c267f

Now, install docker compose.

Head over to this link https://github.com/docker/compose/release. You'll see the command which is like this...

$ curl -L https://github.com/docker/compose/releases/download/1.23.2/docker-compose-`uname -s`-`uname -m` -o /usr/local/bin/docker-compose
$ chmod +x /usr/local/bin/docker-compose

Grab that command and execute it. Once complete, check if docker machine is installed or not by executing

$ docker-compose version
docker-compose version 1.23.2, build 1110ad01
docker-py version: 3.6.0
CPython version: 3.6.7
OpenSSL version: OpenSSL 1.1.0f  25 May 2017

That's pretty much it. Now you have docker installed in your Linux machine.

PS. If you are using windows or MAC, visit the official docker website https://docs.docker.com/docker-for-windows/install/ to get more details.

In the next part, well dive deeper into the Docker world.

Installing Python3.6 in Ubuntu 16.04 LTS Instance Server

2018-05-23T23:32:00+09:00

Here is the step by step guide to install Python 3.6.3 in your AWS EC2's Ubuntu 16.04 server instance or in your local machine with Ubuntu 16.04 LTS Desktop or in Linux Mint.

Note: If you are using Ubuntu 18.04 LTS, then it ships with Python 3.6.3 by default.

Check which Python version is pre-installed in the system by typing $ python3 -V in the terminal. Generally it is Python 3.5.2 for Ubtuntu 16.04 LTS.

Pre-requisites

First you need to upgrade and update your system.

$ sudo apt update && sudo apt -y upgrade
Then you need to install zlib1g-dev package because of this issue.

$ sudo apt -y install zlib1g-dev To make sure it is installed correctly, type $ cat /usr/include/zlib.h. It'll show you the source of zlib header file.
Because of this issue, you also have to install another package called libreadline-dev

$ sudo apt install libreadline-dev
Now, install build-essential and other bunch of stuffs which is required by Python 3.6.3 afterwards.

$ sudo apt install build-essential checkinstall libreadline-gplv2-dev libncursesw5-dev libssl-dev libsqlite3-dev tk-dev libgdbm-dev libc6-dev libbz2-dev

These packages are needed for setting up SSL for downloading packages over PIP, for setting up sqlite, g++ and so on.
Export LD_LIBRARY_PATH for secure SSL connection.

$ export LD_LIBRARY_PATH=/usr/local/ssl/lib/

Begin installation of Python 3.6.3

First, download tar zipped file from Python FTP server.

wget https://www.python.org/ftp/python/3.6.3/Python-3.6.3.tgz
Extract the zipped file:

$ tar zxvf Python-3.6.3.tgz
Move Python3.6.3 directory to /opt/ directory and cd into it.

$ sudo mv Python-3.6.3 /opt/

$ cd /opt/Python-3.6.3/
Configure Python directory.

$./configure --enable-optimizations
Make it:

$ make
Test the Make if you want (this is optional)

$ make test
Install it via Make:

sudo make install
Verify that Python 3.6.3 is installed in your system by typing

$ python3

If it again shows Python 3.5.2 then type $ python3.6. To make things easier, set up an alias. To set up alias do the following:

Edit .bashrc file located in your home directory.

$ nano ~/.bashrc

Add following line at the end

alias python3=/usr/local/bin/python3.6

At this point, Python 3.6.3 is correctly installed in your system

Further steps:

Make sure yout PIP is correct. In Python2 is pip and in Python3 its pip3

$ pip3 -V shows you which Python it is pointing to.
If you are using pyodbc then you need to install unixodbc-dev

$ sudo apt -f install unixodbc-dev

Then you can insall pyodbc with:

$ pip3 install pyodbc

Be a self taught Python programmer in 2018.

2018-03-11T22:38:00+09:00

Beautiful is better than ugly. Explicit is better than implicit. Simple is better than complex. Complex is better than complicated. - The Zen of Python, by Tim Peters

I am not writing this post to tell you what is Python? You can find what it is just by doing a simple Google search on "What is Python?"

An StackOverflow post The Incredible Growth of Python quotes "Python has a solid claim to being the fastest-growing major programming language" and it is absolutely true. This blog post also claims that "Python was the most visited tag on Stack Overflow within high-income nations". And these are not just a random quotes by a random blogger. This facts is based on year long data dump in StackOverflow website.

Just look at this graph which tells the future trend of Python programming language. Isn't is beautiful?

source:https://stackoverflow.blog/2017/09/06/incredible-growth-python/

Now, let me tell something about myself. I am a self taught Python programmer. I started learning Python in 2015 when I was a third semester student. I've not taken any formal course on programming in Python except that one time when our college conducted four classes on Python programming language with one class being of approximately 2-3 hours. But, I had already completed the Python basics before so, it wasn't fruitfull for me. But, I kept on taking the class because "ATTENDANCE" was a must.

So, inorder to gain more knowledge in Python, I started learning from online resources. And over the course of time, I think I've gained enough (but, not enough) knowledge in Python. So, Here's the MOOC you can follow (which I followed) to be a self taught Python programmer.

Start with Python3. Because, Python2 is retiring and Python3 is the future. However, back in 2015 I started with Python 2.7 (Irony). Now, I code almost everything in Python 3.6.3.
Learn the basics. Some resources where you can start are:
- If you want to learn interactively and online. You may want to try Codecademy Python
- If you are a learner through video tutorials, follow these online courses:
- Youtube is also also an enormous library of video tutorials. Here are some channels and playlist where you can learn Python from the basics.
  - Python 3.4 Programming Tutorials
  - Python 3 Basics Tutorial Series
- If you are learner by reading books, then are are few books that I recommend and are also free.
  - Here is the huge collection of online of books: https://pythonbooks.revolunet.com/. However, the ones I like are:
    - Think Like a Computer Scientist
    - Automate the Boring Stuff
    Pro Tip: Follow just one book. Don't ramble.

Here is my GitHub repository which I've prepared while teaching Python programming language in Deerwalk. This covers basic Python course if you are interested. Please give a star if you like this and contributers are welcome.

Make a habit of reading and understanding the core python documentation. Python3 Docs
If you get stuck, do not hesitate to ask questions in StackOverflow with tag #Python3. But, be formal while asking questions in StackOverflow. Follow this guide How do I ask a good question?
After completing basics, try learning advance stuffs. You can do this by following some of the python blogs:
PS. This are some of the blogs that I follow. Or else, I simply do a Google search.
Don't be afraid to do Google search if you get stuck while solving a problem or while trying to learn some concepts like decorators, generators, design patterns, etc.

Pro Tip: Try solving any problem by yourself first. That's the only way to learn it. If you can't, then ask questions.

While writing code, always follow the protocols of good coding styles in Python. Learn about python coding ethics here: http://pep8.org/
Make maximum use of code editors and IDE like: PyCharm, VsCode, SublimeText
Projects, Projects, Projects (Obviously)
There's no need to do advance commercial projects, but make sure you implement the basic python core concepts.
Try to implememt core libraries that comes within the Python package like json, csv, urllib3, etc. And also try to use third package libraries.
If you are searching for project topics and ideas, here is a good resource to follow:
- https://knightlab.northwestern.edu/2014/06/05/five-mini-programming-projects-for-the-python-beginner/
- http://www.practicepython.org/
Create your GitHub account. Push the projects you've done to your GitHub repository. If you don't know how to use Git and GitHub learn it http://try.github.io/. And trust me, you won't regret of learning how to use Git and Github.

Trust me, GitHub is your new resume now a days

Follow Python groups in Facebook. I am saying the real groups with huge number of members. You can find them just by searching. Why follow these groups? because there are some good fellow members that share awesome blog posts and projects that they've done.
Practice Practice Practice
Follow, star the open source projects in Github. Here's the curated list of Open source python projects. https://awesome-python.com/
Learn about tools:
- Virtual Environment
- PIP Package Manager
Now, you are confident in basics, try learning to use libraries and web frameworks like:
- Django
- Flask
- Data Science Libraries

Links to each libraries can be found here: https://awesome-python.com/

Do small projects with the framework and libraries that you've learned and push your code to GitHub.
Learn new topic everyday. P.S. I learned about Python enums today :)
After this you can try applying for Python dev jobs. Here is an interesting Toptal article about what recruiters expect in Python dev.

Download Youtube Videos in Ubuntu for FREE!

2017-02-22T22:21:00+09:00

youtube-dl is a command-line program to download videos from YouTube.com and a few more sites like vimeo, 9gag, Dailymotion, etc.

It is purely build in Python and requires the Python interpreter (2.6, 2.7, or 3.2+) to use it and it is not platform specific. There's also a Windows executable for Windows user. And the most cool thing about youtube-dl is that it's opensource and released to the public domain.

This tutorial illustrates how to install and use it in Ubuntu. But to use this free software you need Python installed in your system.

Step 1: Install youtube-dl

$ sudo apt-get install youtube-dl

or, install via PIP

$ sudo pip install youtube-dl

Step 2 : Use it for free

Fire up your terminal. And type

$ youtube-dl [URL for video]

eg. $ youtube-dl https://www.youtube.com/watch?v=KWZGAExj-es

Choosing the quality:

youtube-dl gives option to download videos of different quality. To download videos of high quality, type the commands provided below:

$ youtube-dl -F https://www.youtube.com/watch?v=KWZGAExj-es

This command gives the information of the video.

linuxsagar@ubuntu:~ $ youtube-dl -F https://www.youtube.com/watch?v=KWZGAExj-es
[youtube] KWZGAExj-es: Downloading webpage
[youtube] KWZGAExj-es: Downloading video info webpage
[youtube] KWZGAExj-es: Extracting video information
[info] Available formats for KWZGAExj-es:
format code  extension  resolution note
249          webm       audio only DASH audio   53k , opus @ 50k, 1.81MiB
250          webm       audio only DASH audio   73k , opus @ 70k, 2.35MiB
140          m4a        audio only DASH audio  128k , m4a_dash container, mp4a.40.2@128k, 4.65MiB
171          webm       audio only DASH audio  137k , vorbis@128k, 4.44MiB
251          webm       audio only DASH audio  174k , opus @160k, 5.84MiB
278          webm       256x144    144p   92k , webm container, vp9, 12fps, video only, 2.60MiB
160          mp4        256x144    144p  111k , avc1.4d400c, 24fps, video only, 3.86MiB
242          webm       426x240    240p  223k , vp9, 24fps, video only, 6.27MiB
133          mp4        426x240    240p  243k , avc1.4d4015, 24fps, video only, 8.57MiB
243          webm       640x360    360p  420k , vp9, 24fps, video only, 11.11MiB
134          mp4        640x360    360p  517k , avc1.4d401e, 24fps, video only, 11.36MiB
244          webm       854x480    480p  715k , vp9, 24fps, video only, 17.63MiB
135          mp4        854x480    480p  970k , avc1.4d401e, 24fps, video only, 21.71MiB
247          webm       1280x720   720p 1499k , vp9, 24fps, video only, 32.43MiB
136          mp4        1280x720   720p 1666k , avc1.4d401f, 24fps, video only, 39.95MiB
248          webm       1920x1080  1080p 2744k , vp9, 24fps, video only, 55.46MiB
137          mp4        1920x1080  1080p 2752k , avc1.640028, 24fps, video only, 71.87MiB
17           3gp        176x144    small , mp4v.20.3, mp4a.40.2@ 24k
36           3gp        320x180    small , mp4v.20.3, mp4a.40.2
43           webm       640x360    medium , vp8.0, vorbis@128k
18           mp4        640x360    medium , avc1.42001E, mp4a.40.2@ 96k
22           mp4        1280x720   hd720 , avc1.64001F, mp4a.40.2@192k (best)

Notice the format column in the output above? Yes, that's what you need. Also, remember capital F in the above command (-F) gives the format information.

Now, lets download a 720p mp4 file with format code 136.

$ youtube-dl -f 136 https://www.youtube.com/watch?v=KWZGAExj-es

You only want the audio of a file? No problem, youtube-dl got your back

youtube-dl --extract-audio --audio-format mp3 https://www.youtube.com/watch?v=5dKGK81z4js

That's it. Similarly, you will be able to download videos/ audio from any site using youtube-dl tool from command line interface.

This opensource project has been hosted in GitHub. You can checkout the source code from here: https://github.com/rg3/youtube-dl

UI Change For Stackoverflow and Github

2017-02-14T19:25:00+09:00

GitHub and Stack Overflow both of these websites have updated their UI. As far as I have seen, mainly the header part or the main navigation menu has been modified.

GitHub has changed it's header color from old silverish background to full black background.

Similarly, Stack Overflow also has updated it's navbar UI. I totally liked these new navbar interface and I'm totally digging it.

Recently wired.com has also published a news about how Linkedin is changing it's interface and making it look like facebookish. More at: https://www.wired.com/2017/01/new-linkedin-looks-just-like-facebook-smart-move/

Top Websites That Computer Science Students Must Visit.

2017-02-13T21:34:00+09:00

Develop a passion for learning. If you do, you will never cease to grow. - Anthony J. D'Angelo

I was going through Quora feeds and my eye caught one particular question, "What are the top websites computer science students must visit?" This question had more than 3k followers. I immediately opened the question link and bookmarked it.

Different users had given many answers and some of the links that I really liked are as follows:

These are some of the links I really liked. But, this isn't the limit. Go through this answer in Quora by yourself. https://www.quora.com/What-are-the-top-websites-computer-science-students-must-visit

And, now some of my personal favorites are:

Data School - If you are intersted in data science.
Big Data University - If you are really into big data, machine learning, spark.
Machinelearning Mastery - Great place to start your machine learning career.
Data Science Central - Awesome blog about machine learning and data science.
Oh shit, git! - How to not screw up your git repo.
Edx - Quality education for everyone, everywhere for Free!
How to contribute to Open Source - Get started to contribute to opensource projects.
Egghead.io - Bite-size video tutorials for badass web developers
Javatpoint - Get started with any popular programming language you like.
Free Code Camp - Open source community that helps you learn to code. (Nepali version coming soon)
Medium Blog for CS
Algorithms Subreddit
Live Edu - Watch professionals code in real time. Visit website, meet coders and learn programming.
Idiot Inside - Python and others
Awesome Python - Info about different Python libraries.
javascriptissexy.com - Indeed JS is sexy. Awesome blog only and about JS.

Many of the links that are mentioned above are related to programming but, Computer Science is not only about coding/programming. A competent computer science students should be able to design new algorithms, reduce complexity of the problem set and algorithms, or design a new programming language, implement a compiler or an interpreter, etc. But, without strong knowledge of programming, nothing is possible.

If you have some cool links, do share it in the comment section below. :)

The One Stop CSIT Guide

2017-02-06T18:44:00+09:00

Warning: This is a challenging task. DMCA might take down some repo. Let's see.

The main problem I faced while studying CSIT was that I had to stumble upon networks of hyperlinks to find the exact resource or course material that I needed to do my assignments/reports. Whenever I find something interesting or something that I need, I immediately downloaded that. And that resulted a vast collection of the resource that might be helpful for the students who are studying B.Sc. CSIT.

To distribute these resources, I tried zipping the folder and uploading to GitHub, but GitHub didn't accept files more than 100MiB. Hence, I had to upload each folder separately. In Order to do that I created a GitHub organization called CSIT-GUIDE.

All these repositories inside CSIT-GUIDE has subject wise notes, books, guides, question papers, lab works and other resources. All these were collected by me on 4 years of studying B.Sc. CSIT. I'm just trying to help all other students who are studying CSIT course.

The organization CSIT-GUIDE has eight repositories which are (semester wise) .

** NOTE: These repositories has the subjects that I was taught. The elective courses I was taught might not be taught to you. **

If you find anything missing, or you have some extra notes, you can always contribute.

You can contribute by:

Forking the repo you want to contribute.
Adding your resource and pushing to your forked repo.
Sending a new pull request.

DIRECT LINKS:

If you want to be a part of CSIT-GUIDE, just email me your GitHub username at [email protected].

P.S. The DMCA laws might block these repos. This is just an experiment. If it goes haywire, I'll be sharing these via Google Drive. :-)

Installing Git Version Control System

2017-02-01T19:01:00+09:00

"Software is like sex: it's better when it's free." - Linus Torvalds

Git is available for all platforms Linux, Mac OS X and Windows.

Install Git on Mac OS X

There are different ways to install Git on a Mac. If you've installed XCode (or it's Command Line Tools), Git may already be installed. To find out, open a terminal and enter git --version. Inorder to install a newer version of Git, use one of these methods:

Git for Mac Installer

The easiest way to install Git on a Mac is via the stand-alone installer:

Download the latest Git for Mac installer.
Follow the prompts to install Git.
Open a terminal and verify the installation was successful by typing git --version
Configure your Git username and email using the following command

$ git config --global user.name "Your Name" 
$ git config --global user.email "[email protected]"

Git for Windows stand-alone installer

Download the latest Git for Windows installer.
Execute the installer, finish the Git Setup wizard screen with default options.
Open a Command Prompt or Git Bash (search git in all programs).
Run the following commands to configure your Git username and email using the following command. These details will be associated with any commits that you create:

$ git config --global user.name "Your Name"
$ git config --global user.email "[email protected]"

Git for Linux (Debian/ Ubuntu)

1 From your shell, install Git using apt-get (Ubuntu 14.04) or apt (Ubuntu 16.04 or later):

$ sudo apt-get update
$ sudo apt-get install git

Verify the installation was successful by typing git --version:

$ git --version
git version 2.9.2

Configure your Git username and email using the following command. These details will be associated with any commits that you create:

$ git config --global user.name "Your Name"
$ git config --global user.email "[email protected]"

source: https://www.atlassian.com/git/tutorials/install-git

Install Python pip package management system in Ubuntu

2017-01-31T19:21:00+09:00

pip is the replacement for the easy_install python package management tool.

If you're running Python 2.7.9+ or Python 3.4+ pip is already installed in the bundle.

If you are below Python 2.7.9 then these are the steps you need to follow to install PIP.

For Ubuntu

$ sudo apt-get update
$ sudo apt-get -y install python-pip

But, even if you are running python 3.4+ version, and pip is not available in the sytem, you can install by:

$ sudo apt-get install python3-pip

For Mac OS

$ sudo easy_install pip

$ brew install python

For Windows

You can install pip via installer.

Download, pip-1.1.win32.exe and run setup.

Verify the pip setup

Check version of the pip via from command line.

pip -V

pip usage

$ sudo pip install some-package-name

Users can also easily remove the package:

$ sudo pip uninstall some-package-name

pip has a feature to manage full lists of packages and corresponding version numbers, possible through a "requirements" file. And is mainly used for virtual enviromnent

$ pip install -r requirements.txt

Sources:

Integrating Disqus Comment Section In Pelican Blog

2017-01-30T21:51:00+09:00

After you write a blog post, you expect people to comment in your post. Commenting is necessary because it can increase reader engagement and help build relationship among author and reader.

Many Pelican themes support Disqus commenting natively. All you need to have is a Disqus account. Go to disqus.com and quickly create an account for yourself.

Now, lets integrate Disqus to our each and every blog posts.

Steps

While logged in to your Disqus account, head over to admin create panel.
You'll see a form. Fill up that form
Click Create Site button.
Now on the top left side of the page, you'll see your disqus site. Click on the link. [refer picture below]

Now headover to setting menu. [refer picture below]

In the setting page, you'll see the Shortname of youtr site as shown below.

Copy that shortname
Now open your pelicanconf.py file. On that file, add the following lines:

DISQUS_SITENAME = 'your-site-name'

Reserve your page via $ fab reserve
Browse the page and you'll see Disqus has been integrated to your blog post.

If you're confused, see how I configured Disqus comments in my blog from here

Building a static site and hosting in Github (part 2)

2017-01-28T17:00:00+09:00

In Part 1 we've setup our github.io website and kick started out [Pelican] blog. Now, in this second tutorial, we'll configure Fabric and Pelican configuration and when it's all done, we'll create a blog post. And push it to github source branch and generated HTML to master branch.

First, let's setup our blog theme.

Pelican has tons of themes. All themes can be previewed at pelicanthemes.com. You can pick any themes you like. And, if you are not satisfied with the existing themes, you can also create your own theme. For this tutorial, we'll be using a theme called Flex by Alexandre Vicenzi.

To setup a theme, create folder inside your username.github.io directory and name it themes. Now cd into that directory and clone the Flex repository.

$ git clone https://github.com/alexandrevicenzi/Flex.git

You can setup plugins in same way as the themes. The official repository for the plugins is pelican-plugins. We won't setup plugins in this tutorial. You can read the documentation by yourself to setup the plugins or I'll be posting a seperate tutorial post on how to setup plugins shortly*.

Configuration

We have our Flex theme inside our theme directory. Our theme needs a site logo. Site logo should be placed in a images directory. So, create an images folder inside the content folder. Paste an image in that folder. If you don't have any, default image will be rendered.

For the configuration, we'll be working on pelicanconf.py, publishconf.py and fabfile.py.

First let's look at pelicanconf.py. Open it in a text editor and update it as described in the snippet below. See the comments beginning with # to know what's happening.

from __future__ import unicode_literals

AUTHOR = u'Your Name'
SITENAME = u'Your Blog Name'
SITEURL = u'http://localhost:8000' #used for local build and preview
SITETITLE = AUTHOR
SITESUBTITLE = 'Your own text' #needed for theme
SITEDESCRIPTION = u'your site description' #used for meta tag for SEO purposes
PATH = 'content' # location to where your articles/posts are located

TIMEZONE = 'Asia/Kathmandu' # Your current time zone

DEFAULT_LANG = u'en'

DATE_FORMATS = {
    'en': '%B %d, %Y',
}

USE_FOLDER_AS_CATEGORY = False
COPYRIGHT_YEAR = 2017
DEFAULT_PAGINATION = 7

# Theme Settings
SITELOGO = '/images/Your picture' #your site logo
FAVICON = '/images/favicon.png' #your favicon
THEME = 'themes/Flex' #path to your theme
BROWSER_COLOR = '#333333'
PYGMENTS_STYLE = 'default' #for code highlighting

# Blogroll
# if you want to link to external page
LINKS = (('link1', 'Your link'),)

# Social widget
# You can add your own links for facebook, twitter
SOCIAL = (('Facebook', 'Your facebook link'),
         ('Twitter', '#'),)

DEFAULT_PAGINATION = 10

If you want to know more details about the configuration, go here.

Next, open publishconf.py file and modify these lines:

...
SITEURL = 'https://your_username.github.io'
RELATIVE_URLS = False
...

Our first post

cd to your content directory and create a folder called articles. Inside that articles folder create a new file called FirstPost.md. Open that FirstPost.md file in a text editor and paste the lines as shown below:

Title: My super title
Date: 2010-12-03 10:20
Modified: 2010-12-05 19:30
Category: abc
Tags: abc, xyz
Slug: my-super-post
Authors: Your name
Summary: Short description

This is the content of my super blog post.

To know more, read the official documentation.

Generate HTML

Go to the root directory of the project or simple do $ cd .. in the terminal, and type

$ fab clean; fab build; fab serve;

and browse to http://localhost:8000. You'll see new shining blog with new design.

Commit and push to Github

** First, let's push our source code.**

Type these commands one by one.

$ git checkout source
$ git add .
$ git commit -m "Configuration, themes and first post"
$ git push

** Push the HTML contents in the master branch.**

$ ghp-import -m 'First push to github.io' -b master output"
$ git push -u origin master

Now, redirect your browser to 'https://your_username.github.io' to view your online blog.

Upto this you have your blog hosted in github pages and is available to public viewing. Next, I'll be posting some small tutorials on how to add disqus comments, google analytics, automate publish using Fabric API, adding and configuring plugins and many more.

Keep visiting my website.

Building a static site and hosting in Github (part 1)

2017-01-28T13:00:00+09:00

For this tutorial series, I'm assuming that you are working in a Linux environment. If not, you can use virtualenv for Windows or Mac.

Getting started

First the big question, "What is Pelican?"

In short, Pelican is a static site generator that supports Markdown and reST syntax. Powered by Python. For more visit, getpelican.com

For this tutorial, you only need to understand basic Git push/pull and branching command and as well as some basic Markdown syntax. If you are a noob, no worries I'll cover each and every topics in detail.

Prerequisits:

Setup Github page

First you need to have a github.io page. Noew, clone your github.io repository and cd into it.

$ git clone https://github.com/username/username.github.io
$ cd username.github.io/

Create a new Git branch called source or your any custom name. This new branch only holds your source code. The HTML generated by Pelican will be pushed to master branch which are then rendered in the website. We'll do Git push shortly. But, first let's create a new branch.

$ git checkout -b source

Push the new branch on Github:

$ git push origin source

You are now working on the source branch.

Setup Pelican

Let's start with installing Pelican at the first place. Pelican, Pelican-Plugins and Pelican-Themes are dependent on different python modules and installing them one by one is a pain in the butt. Hence, for the simplicity, create a new file called requirements.txt.

Open requirements.txt file in a notepad or gedit and paste the lines given below and save it.

appdirs==1.4.0
beautifulsoup4==4.5.1
blinker==1.4
bs4==0.0.1
docutils==0.12
ecdsa==0.13
Fabric==1.12.0
feedgenerator==1.9
ghp-import==0.4.1
Jinja2==2.8
Markdown==2.6.7
MarkupSafe==0.23
packaging==16.8
paramiko==1.17.2
pelican==3.6.3
pycrypto==2.6.1
Pygments==2.1.3
pyparsing==2.1.10
python-dateutil==2.5.3
pytz==2016.7
six==1.10.0
Unidecode==0.4.19

These are the modules that we need to get started with the Pelican static website generation.

To install the modules which are inside requirements.txt, type the following command in the terminal.

$ pip install -r requirements.txt

If you don't know what is pip and how to install it, go here.

Upto this step you've setup a github.io blog and installed Pelican and it's dependencies.

Pelican Quick Start

After installing pelican, you can leverage the command $ pelican-quickstart from the command line.

Quick Setup

cd into your username.github.io repository that you cloned earlier. $ cd username.github.io
Save this .gitignore file in your directory. It's needed to make your Github repo clean.
Run $ pelican-quickstart in the terminal

The quickstart will ask you various questions, which you can answer in turn. Here are some specific answers you should give:

> Where do you want to create your new web site? [.] (hit enter)
> What will be the title of this web site? My Awesome Blog
> Who will be the author of this web site? (Your Full Name)
> What will be the default language of this web site? [en] (hit enter) 
> Do you want to specify a URL prefix? e.g., http://example.com   (Y/n) Y
> What is your URL prefix? (see above example; no trailing slash) http://your_username.github.io
> Do you want to enable article pagination? (Y/n) Y
> How many articles per page do you want? [10] 10
> What is your time zone? [Europe/Paris] Asia/Kathmandu
> Do you want to generate a Fabfile/Makefile to automate generation and publishing? (Y/n) Y
> Do you want an auto-reload & simpleHTTP script to assist with theme and site development? (Y/n) Y
> Do you want to upload your website using FTP? (y/N) N
> Do you want to upload your website using SSH? (y/N) N
> Do you want to upload your website using Dropbox? (y/N) N
> Do you want to upload your website using S3? (y/N) N
> Do you want to upload your website using Rackspace Cloud Files? (y/N) N
> Do you want to upload your website using GitHub Pages? (y/N) Y
> Is this your personal page (username.github.io)? (y/N) Y
Done. Your new project is available at /<path>/your_username.github.io

That's it. Now you have the basic structure of your Pelican blog. The tree structure of the pelican blog is:

.
├── content
├── develop_server.sh
├── fabfile.py
├── Makefile
├── output
├── pelicanconf.py
└── publishconf.py

2 directories, 5 files

Here, each files and folder have a specefic meaning. I'll discuss more on it later.

Build and serve

Generate HTML. All HTML files are generated in the output directory.

$ fab build

Here fab is a build tool that comes with Fabric module. Now, serve the HTML.

$ fab serve

And redirect your browser to http://localhost:8000. You'll see the result.

** Note: The website generated is only visible in local environment. We haven't pushed anything to Github yet. It'll be done after all configurations have been setup correctly.**

Now, just sitback, relax and observe every files inside the directory. If you understood nothing, maybe learn some Markdown.

In Part 2 we'll write our first blog post, configure and fabricate pelican, and our first commit to the github.