Tuesday, November 17, 2015

What is Docker Containers?




Docker, a new container technology, is hotter than hot because it makes it possible to get far more apps running on the same old servers 

Containers, use shared operating systems. That means they are much more efficient than hypervisors in system resource terms. Instead of virtualizing hardware, containers rest on top of a single Linux instance. This in turn means you can “leave behind the useless 99.9% VM junk, leaving you with a small, neat capsule containing your application,”
Therefore ,  with a perfectly tuned container system, you can have as many as four-to-six times the number of server application instances as you can using Xen or KVM VMs on the same hardware
Docker, however, is built on top of LXC. Like with any container technology, as far as the program is concerned, it has its own file system, storage, CPU, RAM, and so on. The key difference between containers and VMs is that while the hypervisor abstracts an entire device, containers just abstract the operating system kernel.

This, in turn, means that one thing hypervisors can do that containers can’t is to use different operating systems or kernels. So, for example, you can use Microsoft Azure to run both instances of Windows Server 2012 and SUSE Linux Enterprise Server, at the same time. With Docker, all containers must use the same operating system and kernel.
On the other hand, if all you want to do is get the most server application instances running on the least amount of hardware, you couldn't care less about running multiple operating system VMs. If multiple copies of the same application are what you want, then you'll love containers.
Docker brings several new things to the table that the earlier technologies didn't. The first is that it's made containers easier and safer to deploy and use than previous approaches. In addition, because Docker's partnering with the other container powers, including Canonical, Google, Red Hat, and Parallels, on its key open-source component libcontainer , it's brought much-needed standardization to containers.

Docker containers are easy to deploy in a cloud. Docker has been designed in a way that it can be incorporated into most DevOps applications, including Puppet, Chef, Vagrant, and Ansible, or it can be used on its own to manage development environments. The primary selling point is that it simplifies many of the tasks typically done by these other applications. 

Tuesday, November 3, 2015

50 diagrams illustrating IoT to Cloud/Big Data Analytics interfaces



My colleague Bob Marcus  collected about 50 diagrams (with reference URLs) showing alternative architectural approaches to interfacing IoT to Cloud/Big Data Analytics. See the Link below. The diagrams are generally very high level with more details available on the source Web sites.  



50 diagrams illustrating IoT to Cloud/Big Data Analytics interfaces

Monday, September 14, 2015

My 100th Blog: A milestone




To everyone who visited my blog on daily basis: A Big Thank you. Its little more than a year and i am writing today my 100th  blog post. its a milestone for me.

Below is some of my most important blogs. 



General Blogs
·         Hyper Converged System: EVO RAIL  



Cloud Computing

·               vCoud Director Installation and configuration
·               HP CSA Implementation
·               HP CSA Vs VMware VCAC
·               10 best practices for cloud design
·               vCloud Suite Architecture-1
·               vCloud Suite Architecture-2
·               Application Suitability For Cloud
·               AWSElasticity Principles of elasticity
·               VMwarevCloud Air : Architecture and Principles
·               vCloud Air: ROLE BASED ACCESSCONTROLS
·               vCloud (vCAC 6): Architecture and components
·               vCAC 6: Installation Pre-requisites 
·               vCAC 6: Installation 
·               vCAC 6: Configuring IaaS-Tenants and Roles
·               Azure AD Fundamentals
·               Changing Cloud Market

·               Building Microsoft Private Cloud Using System Center 2012: High Level design

·               vCAC Identity Service Architecture

·               Amazon Web Services- Elastic Load balancer

·               Amazon Web Services: Auto scaling and Cloudwatch

·               Amazon Web Services: Cloudformation

·               Amazon Web Services: Storage

·               Amazon Web Services : CloudFront

·               Amazon Web Services: ElastiCache

·               Amazon Web Services: Virtual Private Cloud

·               Amazon Web Services: Virtual Private Cloud Advanced

·               Amazon Web Services: Simple Notification Service

·               Amazon Web Services: Route 53






VMware Technologies/Disaster Recovery:


Storage Sizing:
·       Storage Sizing: A start

Big Data:
·               Big Data- The Rise and the Future
·               Big data: Technology Stack
·               Big Data: Hadoop Distributed Filesystem (HDFS)
·               Big Data: Map Reduce
·               Big Data- Installing Hadoop ( Single Node)
·               Big Data- Apache Hadoop Multi Node
·               Big Data: Managing HDFS
·               Big Data: Map Reduce Development
·               Big Data: Introduction to Pig
·               Big Data:Introduction to Hbase
·               Big Data: Introduction toZookeeper
·               Big Data: AMAZON ELASTICMAPREDUCE (EMR)
·               Big Data: Microsoft HDinsight
·               HP Vertica Building Blocks

·               Installing Vertica Analytics and Management Console




Datacenter, Transformation and Migration
·       Data Center Migration
·       Green Datacenter

Enterprise architecture- Solution Architecture


Citrix:

Architecting a Citrix Virtualization solution
 There are 2 parts to it:
  1)    Assessment
  2)    Design

Assessment is further divided into
·       User Community
·       Application Delivery
·       Server Virtualization
·       Infrastructure
·       Operation and Support
·       Conceptual Architecture

  Design is further divided into
·       Desktop Delivery design

·       Access Design

HP Blogs - Delivering business agility with Infrastructure as... - The HP Blog Hub

HP Blogs - Delivering business agility with Infrastructure as... - The HP Blog Hub

Sunday, September 13, 2015

Amazon Web Services: Route 53


In this blog we will discuss Route 53. It is also knowan as amazon distributed DNS. It is fully featured. It supports multiple DNS record types, multiple routing algorithm.It also has resource level IAM Control

Route 53 is distributed DNS. It is distributed geographically.


User in North America will use DNS locally located. It makes it faster

Route 53 supports all major record types. It has TTL control



Route 53 have 2 types of Algorithm

  • Weighted Round Robin (WRR)
  • Latency Based Routing (LBR)
So this is all about Route 53. Now you should have high  level understanding how amazon works

Sunday, September 6, 2015

Amazon Web Services: Simple Notification Service



In this blog, we will discuss Simple notification service

SNS sends notices to internal assets from AWS:from Application to user OR to application or user. SNS is useful for monitoring, alerts and logging.

SNS Setup

To set up a SNS,

  •  Create a Topic. Think it as a mailbox or email alias
  • Add Subscribers: Think of it like a  multiple recipients.If more than just email
  • Subscribers must confirm since this is an internal message
  • And you send messages to the Topic. All confirmed subscribers immedialy et the message
SNS Overview

  • it obviates the need to build custom messaging
  • its push based messaging
  • Deep integration with other AWS services
  • it has API and AWS console support

.Next we identify SNS subscriber types

  • Http/s
  • email/ email-JSON
  • SMS
  • SQS
SNS Service integration

  • Autoscaling: Publish instance add/remove messages
  • EC2:Application on EC2 can leverage SNS
  • S3
  • Cloudwatch



Thursday, September 3, 2015

Amazon Web Services: Virtual Private Cloud Advanced


In the last blog we  covered VPC. In this blog we will cover advance VPC
In this blog primarily the  focus will be on

  • Isolation internal tiers/Customers from each other
  • Programatically connecting/disconnecting Internet gateway


Isolation internal tiers/Customers from each other


Please look at teh diagram below: There is a web/App tier and compute Tier Both are in AZ-A and AZ-B. Both have different VPC and Routing table

So this way you can isolate different tiers and customers from each other

Programatically connecting/disconnecting Internet gateway






Monday, August 31, 2015

Amazon Web Services: Virtual Private Cloud

In this blog, we will discuss Amazon Web Services,Virtual Private Cloud (VPC)

 Please have a look at the diagram below:

Customer 1 (C1), Customer 2(C2) and Customer 3 (C3) have their EC2 instances. Without VPC, C2 can ping and try to access easily C3 s EC2 instances.

VPC Benefits:

  • Logical isolation of your resources from other customers, from internal divisions and application tiers.
  • Control Over:
          --Address Spaces
          --Subnets: Do you know minimum size of subnets should be /28 or 14 ip addresses and maximum size should be eual to size of VPC. We can have 200 Subnest per AZ
          --Route Tables and NATs
          --Network Gateway (VPN and Internet)
          --Ingress and egress security groups
          --Multiple Elastic Network Interface (ENI) on VPC EC2
          --Control over dedicated hardware


VPC Limitations are:
  • Only Select resources like ELB( w/support to balance web/app tier), EC2( and EBS, Autoscaling), RDS, Elasticache can be launched in a VPC
  • You will want to launch all of them together either inside or outside VPC
  • No multicast or broadcast support
  • 5 VPCs per account per Region
  • 200 subnets per VPC
  • 5 vpc eip per account per region
  • 1 internet gateway per VPC
  • No. Of ENIs per instance depend upon type of instance

VPC Setup:

  • Create VPC: Choose reaonable IPs
  • Creeate Subnets
  • Create route tables, NACLs
  • Launch assets into VPC
  • Create IG.VPG

So this is the basic understanding of VPC




Thursday, August 27, 2015

Amazon Web Services: ElastiCache


In This blog we will discuss Elastic Cache

ElastiCache is the in memory caching mechanism. It is the memcache under the hood. It is 100% API compliant.More appropriately it is a memcache cluster


Memcache is a popular , opensource in -memory cache. it is  very fast read and writes.It takes burden of backend resources

Memcache is a key value store.It is a NOSql DB. Its only access method is a key

ElastiCache is actually a memcache cluster. it is a distributed collection of cache nodes

Its available in only Single AZ
Cache nodes are very user specified types
  • very much like EC2 Instance type
  • can be wrapped with security groups
  • can buy capacity reservations
  • Resizeable ( remove or add nodes)


To Setup and ElastiCache


  • Choose an AZ
  • Chose node types, # of nodes
  • Launch Cluster
  • Connect application by pointing to elasticache endpoint
  • Benchmark, resize ( if necessary)


Elasticache Expiry
  • Expiring Key is controlled via the elasticache API (memcache commands)
  • Scheduled
  • on demand

ElastiCache Pricing
  • It has per hour node charge
  • Bandwidth Charges- Bandwidth Charges are free inside the AZ but Charged $0.1/GB in or out of AZs

Wednesday, August 26, 2015

Amazon Web Services : CloudFront


In this blog we will discuss amazon cloudfront. What is cloudfront?

Cloudfront is a CDN or content delivery network. It utilizes Edge Locations near to user. Since its a CDN we need to remember content needs and origin and it can be expire also.

CDN are popular static websites.Using CDN, you can either make


  • Download distributions like products, documents and data
  • or Streaming distribution like audio or video
Cloudfront has native integrations to AWS (S3 origins).It is very cost effective

To setup a cloudfront, one must create a distribution
  • Must define Http or steaming
  • Must Specify an origin: Any web accessible server will work ( EC2, S3, Private server)\
  • Can also specify TTL, logging, Path matches
  • Distribution domain created for you for example like d1234.cloudfront.net or can also easily create vanity URLS
Use Distribution domain in links

When user access the CDN URL, Cloudfront directs them to their closest Edge. Everyone loads content faster.

Cloudfront is a pull mechanism.Each edge pulls from origin if it doesnt have the requested content
       
Each Pull is charged
  • from origin
  • from each user
Invalidation request are charged.
But you can get Reservation capacity discount if you know your load in future and you reserve in advance.





Tuesday, August 25, 2015

Amazon Web Services: Storage


In this blog we focus on Storage. We will discuss many storage options available in AWS.

So we will look at S3 Storage, RDS( Managed MySQL, Oracle and SQL) and DyanamoDB( NOSQL database)

Lets start with S3 Storage.

S3 Storage ( Simple Storage Service) is an object storageand is best used for static assests. It is the cheapest option to store and serve data. There is no limits on the number of the objects that you can create. It is extremely durable.


S3 is extremely scalable too. All objects are natively web accessible.It can also host static websites.

There are certain S3 limitations:

  • Latency
  • Its a WORM ( Write Once Read Many) Storage.
  • It cannot serve Dynamic content
  • Its 100buckets and 5TB Max sizeon the objects
  • bare bones (write, read, delete)


RDS ( Relation Database service) is a hands off RDBMS (MySQL, Oracle and SQL). RDS is manually scalable like EC2. RDS has HA already baked in ( multi-AZ master/slave). RDS also has DR baked in with automated backups and real time snapshots. It is very cost effective. It is also very scalable.

RDS has some limitations too. It has upward vertical limitation. RDS has 3TB max db size.It has only SQL Interface with No SSH or hardware access.


DyanamoDB deals awesomely with 3Vs of big Data( variety, velocity and volume). It has HA baked in. It is highly scalable and cheap. It has drop dead administration



Sunday, August 23, 2015

Amazon Web Services: Cloudformation


In this blog, we will discuss Cloudformation.
cloudformation helps in codifying creation of stack of resources.

What are stack of resources? A stack could be:

  • ELB
  • Auto scaling group
  • EC2
  • RDS (Database)
  • All connections between them

There are many benefits of cloudformation, namely:

  1. Your infrastructure as a code
  2. can be version controlled
  3. Modularization ( templates can contain other templates)
  4. Enforce one way to deploy
  5. Cost nothing
Signing up with Cloudformation means signing up with AWS services that Cloudformation can create

With Cloudformation you can setup billing alarms. To learn how to setup billing alarms go to

There are 200-300 pre-packaged amazon templates available. The templates are JSON based. The templates can accespt run-time parameter [Instance type (small for test), Kep pair (prod vs Test)]
Templates can also update running stack
Templates have 7 section
  • version
  • description
  • parameters
  • mapping
  • resources
  • properties
  • output
The link for templates is

Friday, August 21, 2015

Amazon Web Services: Auto scaling and Cloudwatch


 In this blog, we will discuss Amazon Web Services: Auto scaling and Cloudwatch

The question is : What does Auto scaling provides?  The answer is:

  • Linear Horizontal Scaling 
  • HA

Second question is What Cloud watch enables? Cloud watch enables:

  • monitoring and alerting for resources
  • Alarms Can fire auto scaling policies, SNS notifications

 auto scaling, ELB and Cloud watch makes up what Amazon calls Triangle services

AUTO SCALING SET UP

To set up Auto scaling, we need to do the following things:

  • Create ELB( Its recommended but not necessary)
  • Create Instances in 2 or more AZs ( Its recommended but not necessary
              a) Create Launch Config: Launch config consists of
                          1) EC2 instance launch profile
                          2) Image ID (AMI)
                          3) Instance Type ( e.g t1.micro or m1.small)
                          4) key pair ( recommended)
                          5) Security Group
                          6) Name

  • Create Auto scaling Groups: In a auto scaling group you have AZs, Max or Min servers etc
  • Create Auto scaling Policy: The policy consists of buttons that an API/Scheduled/ cloudwatch can push
  • Create Cloud watch Alarms: Which can fire policies




CLOUD WATCH


Now we will briefly discuss cloud watch. So what does Cloud watch exactly do? Well, the answer is
  1. Monitors/Alarms on resources
  2. Basic Monitors provided free with most basic services ( EC2, ELB, SQS..etc)
  3. Alarms can fire auto scaling policies or SNS notifications
  4. cloud watch is statistics in , statistics out. below is an example














Cloud Watch alarms trigger when metric statistics goes in( or out  ) out of state for specific time

Hope it clears the functionality of cloud watch




Featured Post

Amazon Route 53

Amazon Route 53 is a highly available and scalable Domain Name System (DNS) web service.Route 53  perform three main functions in any...