Thursday, July 31, 2014

Architecting a Citrix Virtualization solution-Assessment: Conceptual Architecture

Lets refresh where we are! We are discussing  about architecting a Citrix environment from Architects perspective. There are 2 parts to it:

  1)    Assessment
  2)    Design

Assessment is further divided into
·   Conceptual Architecture

In this blog, we will discuss assessment of Conceptual Architecture. Each topic will be covered in separate blog.
After the architect completes the initial assessment of all aspects of the current environment, the design phase begins. During the design phase, the architect uses the information gathered during the assessment to design a virtualization solution that meets the needs of the organization.
This design is then transferred into a conceptual architecture that can be used to communicate the design.
Conceptual Architecture Overview
The conceptual architecture is a diagram or series of diagrams that provides an abstracted overview of a proposed design that non-technical people can understand. The diagram should not be overly detailed, but should depict all major components and the connections between the components

If necessary, additional diagrams can be included to cover additional areas of the design. For example, the XenApp and XenDesktop delivery conceptual designs might be depicted in a different diagram than the high-level overview of the datacenter conceptual design

Conceptual Architecture Documentation
The conceptual architecture diagram should be accompanied by supporting documentation.
This supporting documentation should identify:
  • The purpose of each component and connection in the architecture
  • The assumptions and decisions that were made
  • The rationale for each assumption and decision
·         Conceptual Architecture Validation
·         After the conceptual architecture has been documented, it must be validated to ensure that it is technically sound and satisfies the goals and concerns of the organization.
·         At this point, the architect should present the conceptual architecture to the stakeholders whose goals and concerns are being addressed by the design. During this presentation, the architect should point out how the architecture addresses the project objectives and requirements and keep track of any issues or changes that are suggested.
·         After the validation is complete, the conceptual architecture can be used to collect feedback from other architects and developers.
Conceptual Architecture Implementation
·         When the conceptual architecture is complete, it can be used as a starting point for creating a more elaborate document that provides detailed design and component specifications. This document will be used to implement the design

Citrix Virtualization Conceptual Architecture
In order to create the conceptual architecture for a Citrix virtualization project, the architect must understand how the Citrix products will be implemented in the environment to meet the goals and objectives of the organization.
The remainder of this course focuses on the considerations that affect the design of a Citrix virtualization solution and will influence the conceptual architecture diagram and the documentation provided for the solution

Wednesday, July 30, 2014

Big Data: Managing HDFS

So far we have covered the following topics in the big data. You can click on the hyperlink and go to a specific topic.

Technology Stack
hadoop distributed file system (HDFS)
Map Reduce
Installing Hadoop ( Single Node)
Apache Hadoop installing Multi Node
Big Data: Troubleshooting, Administering and optimizing Hadoop

In this blog, we cover the topic, Managing HDFS.
 Lets start with DATA.
Below are the URLs for getting  data on the internet of varying shapes and sizes.

When people get into the hadoop, first thing they want to do is to see the whole process. Thats why small data is really good to start with   Books (Small):  has thousand of thousand of free book , which you can download as text files. put these text files into hadoop and start mining this information.

Other data sets examples

S3 Data (Varying) :,com/datasets

Public datasets (varying) :

So, we download this data, on to our computer in the Cygwin directory,

so we have small data under the books folder and semi large data under the weather folder. Now we have get this data to HDFS.
 now we connect our Client to hadoop cluster and make a test directory

# ~S hadoop fs -mkdir test
when we make directory without specifying the path, the directory would be created in the users home directory.
We need to put our data to  place where everyone can access it in hadoop
# ~S hadoop fs -mkdir hdfs://hnname:10001/data/small
The above command is putting the data for our book data that we have
# ~S hadoop fs -mkdir hdfs://hnname:10001/data/big

also if you want to remove a directory, then we can do this

# ~S hadoop fs -rnr test

now we will move the data in to our small and big directory

# ~S hadoop fs -moveFromLocal /home/abuser/data/war_and_peace.txt hdfs://hnname:10001/data/small/war_and_peace.txt

since we have loaded the data, now we can do some admin work

# ~S hadoop fs -report
it will give cluster summary node info etc

we can also put it in safe mode
# ~S hadoop dfsadmin -safemode enter

to get out of safe mode

# ~S hadoop dfsadmin -safemode leave

We can also run the file system checker since we have the data now. we cant run this command from the client machine. you would need to ssh to the name node

# ~S hadoop fsck -blocks

we can also check file system of a specific directory

# ~S hadoop fsck /data/big


1) Shutdown the cluster: shutdown down the map reduce first, then HDFS.
2) Install new version of hadoop
3) Start Hadoop with upgrade option: -upgrade
4) Check status with dfsadmin
5) When status is complete:
   - Put in safemode
   - use fsck to check health
   - Read some files
6) Rollback if -rollback
7) Finalize if successfull: hadoop dfsadmin -finalizeUpgrade

 Next thing we need to discuss is RACK Awareness

-Name node executes script. Passes IP address as argument
-Script returns rack id
-accomplished via direct code, file lookup or dyanmic

the above process is manual and hard coded. we need to find a better way to do this. the better way is to make a file

Tuesday, July 29, 2014

Architecting a Citrix Virtualization solution-Assessment:Operation and Support

Lets refresh where we are! We are discussing  about architecting a Citrix environment from Architects perspective. There are 2 parts to it:

  1)    Assessment
  2)    Design

Assessment is further divided into
·   Operation and Support
·   Conceptual Architecture

In this blog, we will discuss assessment of Operation and Support. Each topic will be covered in separate blog.
During the assessment of the operations and support processes in an environment, the architect should gather data about the monitoring, systems management, testing, and change control practices, support environment and user issues and the backup and recovery plans.
Monitoring and Systems Management
To optimize the user experience, monitoring and management must be systematically addressed in a Citrix virtualization environment. During an assessment project, architects assess the current monitoring tools used, and the process for implementing and managing them. For example, they should find out which administrators are responsible for the tools, data, and resolving major issues.
Architects should ask if the following monitoring tools are being utilized for monitoring and systems management:
  • Health Monitoring and Recovery (Citrix XenApp feature)
  • Citrix EdgeSight
EdgeSight is highly recommended for Citrix environments.
  • Resource Manager
  • Microsoft Performance Monitor
  • Microsoft Systems Center Operations Manager (SCOM)
  • Third-party tools such as Process Monitor
Ideally, in a Citrix environment, the following systems are in place:
    • A hardware managing solution
    • A server monitoring solution
    • Health Monitoring and Recovery
    • Citrix EdgeSight
Processes for Monitoring Systems Management
Processes for monitoring systems management are relevant to any assessment. An architect checks the monitoring tools implemented to monitor the health of client devices, hardware, networking and application usage, and to track metrics.
Architects should ask the following questions:
  • If the environment contains XenApp, are Health Monitoring and Recovery tests utilized?
  • What is used to monitor servers?
  • Is a network monitoring tool used?
  • What transpires when an alert happens?
From the Architect
A virtualization architect should have a watchful eye on the user experience and demonstrate a deep level of understanding of the users and the administrators responsible for monitoring the environment.
Testing and Change Control
In Citrix virtualization environments, effective testing and change controls ensure environment stability, while adding to the flexibility inherent in the solution. Architects review testing and change control during the assessment to evaluate the infrastructure for implementing desktop changes. Items assessed may include lab environment hardware, test processes, change control tools, documentation, approval processes and general procedures for implementing changes to the environment.
For example, if a company plans to upgrade the service pack on the desktop operating system, testing and change control is critical in order to ensure a smooth process. In addition, a rollback process should be documented in the event that the planned change is not successful or cannot be completed within the required timeframe.
A best practice is to schedule maintenance windows or rollout dates where all changes will be planned, tested and released. Scheduling maintenance windows increases the consistency of the maintenance schedule and makes planning ahead easier.
Architects should ask the following questions:
  • Is there a testing environment in place to test changes before going into production? If so, is it isolated?
It is recommended to have an isolated and designated test environment that is identical to the production environment but is specifically used for testing.
  • How closely does the test environment mirror the production environment?
  • How many servers are allocated for testing?
  • Do production servers have the same configuration as test servers?
  • Are virtual machines used in the testing environment?
  • Which change process is used? For example, implement, test, rollback.
  • Is there a formal test process? Which types of tests are regularly conducted, including scalability, integration and functional unit testing?
  • Are there any negative issues, such as disruption of users, or scheduling difficulty, associated with the current change process?
  • What is the time interval between change request approval and implementation?
  • Are automated methods used to rollout changes to users?
  • Is the XenApp configuration log used?
  • Are ticketing requests in-line with the help desk?
The support environment is vital in any enterprise environment, including one involving desktop virtualization. However, if the customer has not yet implemented virtualization, an architect may examine the support trouble tickets and determine which issues can be mitigated by using virtual desktops. Based on the current environment, an architect should be able to document the potential and limitations of desktop virtualization.
When assessing the existing support environment, architects usually meet with the help desk or support manager to gain an understanding of users' issues and to review consolidated reports that can reveal trends or areas of concern.
Architects should ask the following questions:
  • What Service Level Agreements or Service Commitments exist?
  • What are the existing support tiers? (Example- Level 1: Help Desk, Level 2: Escalation, Level 3: Engineering)
  • Which support contracts exist?
If an active Citrix Technical Support agreement exists, XenServer incidents associated with XenApp farm servers are covered.
  • What process is used for incident tracking?
  • What is the escalation process?
  • What, in detail, are the common troubleshooting steps?
  • What happens when users call the help desk? Does the help desk associate help the person or record the incident for follow up by another employee?
  • Does IT maintain data on how long tickets are open until they are resolved? What is the average time to resolution?
  • If a serious problem arises and additional on-site support is required, how long does it take for someone to fix the problem, on average?
  • What percentage of calls are related to common issues? What are the largest ticket categories?
  • Which tasks are automated to minimize help desk calls?
  • Do support personnel have delegated administration rights to the Delivery Services Console, if applicable?
Delegated administration rights in the Delivery Services Console is considered a best practice.
  • Where are the call center locations, relative to the users?
  • How many employees make up the help desk staff?
Gathering specific answers to these questions about the environment provides the architect with an understanding of the support environment, which will help with making initial recommendations and with the solution design as well.
Support Recommendations
General recommendations for support in a Citrix virtualization environment include the following:
  • Document clearly defined roles and procedures to ensure continuity between support tiers.
  • Track all issues through an enforced ticketing system to prevent circumvention and improve the overall support process.
  • Determine the appropriate support model and support structure for the overall desktop and application virtualization effort after the design effort.
  • Limit the number of administrators and ensure that they are adequately trained. Curtail individual administrator rights.
  • If necessary, repurpose and retrain existing staff during an implementation of the virtualization solution.
  • Ensure that all systems engineers and system administrators are formally trained in the technologies and methodologies that support the overall desktop and application virtualization environment.
Citrix and Microsoft training are specifically recommended to ensure optimal design and rapid troubleshooting.
  • Train help desk staff to identify whether issues are occurring on the virtual desktop or the local, physical workstation. Help desk staff will also need to be trained on troubleshooting basic virtual desktop issues.
Backup management is another important area of the environment to assess. Servers and all important data need to be backed up, as a best practice.
Architects should ask the following questions:
  • What is the current backup procedure?
  • How much redundancy exists for Citrix infrastructure components?
  • What backup procedure is in place for SQL server, if applicable?
  • Is there a backup schedule for servers?
  • What is the maximum amount of time the environment can be down?
  • Have all single points of failure been mitigated?
  • In the event of a hardware failure, is there a quick way to recover the servers?
  • Are the organization's web sites backed-up?
  • Does the environment use IBM Tivoli Support Manager, or a similar tool?
  • Is a centralized backup and recovery manager system being used?
  • Is there a backup restoration testing procedure in place?
Disaster Recovery
Disaster recovery is a critical aspect of virtualization projects. If a disaster recovery plan is already implemented in an organization, the architect should assess the current plan and make recommendations to improve it.
Architects should ask the following questions:
  • Is there a full check-list of critical system components and ordered steps for recovery and rebuild? During serious system outage, having documented steps in a quick format ensures that the system is built according to pre-defined and pre-tested plans.
  • Is the plan updated each time systems are upgraded?
  • How often is the disaster recovery plan revisited and tested?
  • How long does it take to execute the disaster recovery plan?
  • What percentage of users can be facilitated at a backup site?
  • Which manual steps are required in order to facilitate a backup site?
  • Has off-site storage of the critical software and data required to rebuild a system been implemented?
A full backup of the server operating system and Citrix software should be available.

Monday, July 28, 2014

Big Data: Troubleshooting, Administering and optimizing Hadoop

In this blog, we discuss Troubleshooting, Administering and optimizing Hadoop.


Lets see hadoop base directory

-Remember Logs are your best friend
    *error messages
    *java exceptions
In the hadoop main directory, you can find all the logs in the logs folder. For some reason if there is no logs folder, then logs will be found in libexec folder. lets see where hadoop stores its files

-Logs are named by machine
    *user, daemons

-cluster start up issues are most likely due to configuration.


- Commission and decommission: Commissioning is just adding a node into the slaves file, configuring its hdfs site& mapred files and bringing it online.
To decommission we need to add an exclude file to the machine we want to decommission. Also exclude from slave files.

-Check for corruption: fsck, a great way to check file corruption.

-Default override configuration

-copy data, in, out , across clusters.:Distributed copying, which actually reduce map reduce to spread the data across the clusters.

-Tuning and Optimization

-Troubleshooting Jobs and nodes

-safe mode: its actually a read only node for HDFS.

Below are the options that we can optimize

Most of the optimization will come under map reduce.

Architecting a Citrix Virtualization solution-Assessment:Security and operation

Lets refresh where we are! We are discussing  about architecting a Citrix environment from Architects perspective. There are 2 parts to it:
  1)    Assessment
  2)    Design

Assessment is further divided into
·   Security and personalization
·   Operation and Support
·   Conceptual Architecture

In this blog, we will discuss assessment of security and operation. Each topic will be covered in separate blog.

Most organizations spend significant amounts of money for security. When assessing enterprise security, an architect should gather information about the environment in regard to physical security including restrictions, permissions, the management of systems and personalization settings implemented through profiles and policies.
Enterprise Security
Security concerns with desktops include viruses and malware, persistent cache and employees sending confidential information by way of the backend infrastructure. Assessing security in an virtualization environment is essential to ensure that the environment is as secure is possible.
During the security assessment, an architect should also inspect the following topics in an environment:
  • Administrator access
  • Application and server security
  • Network security
  • Remote access security
  • Password change policies
  • Password security issues
  • Antivirus security
  • Service pack updates
  • Server certificates
  • Event logs
A best practice is to never grant anonymous access unless absolutely necessary, require authentication to the desktop and require application-level passwords.

Security Assessment
Architects should ask the following questions:
  • For existing XenApp environments, is ICA encryption used?
  • How do external users access their desktop data?
  • Is there a dedicated security team?
  • If Web Interface is implemented, are security certificates installed on the Web Interface servers?
If not, passwords are transmitted in plain text and can be easily accessed by an internal administrator.
  • Are internal or third-party certificates being used?
  • Are endpoint analysis scans being run or is the organization performing any other type of endpoint analysis?
Endpoint analysis should be performed in most environments, even if the organization is running a non-Citrix appliance, such as Cisco or Juniper.
  • Do users have the ability to perform all of their required tasks?
  • Does any sensitive data leave the network?
  • Is accessing applications and resources safe?
  • Do any security measures negatively affect performance?
  • Is VPN access from a PC allowed?
  • Can users access mapped drives through a VPN?
  • Is Single Sign-on being used?
  • What are the audit policies?
  • Are there security considerations between internal and external networks?
  • What are the enterprise-wide password policies?
  • Are Service Pack updates performed? What is the process?
  • Are server security logs monitored by administrators?
  • How much retention exists in the security logs?
Assessing each area of security is helpful to the architect during the design phase, in order to recommend a solution that is secure for the organization and its users.
Browsers and Encryption
During the security portion of the assessment, architects also gather information related to browsers and encryption levels.
Architects should ask the following questions:
  • Which browsers are supported by the organization?
  • What are the browser security settings?
  • Are any applets, such as ActiveX or Java, blocked?
  • Which encryption level does the business require?
General recommendations include the following:
  • Standardize on a supported browser that meets the business requirements. Using multiple browser types can result in inconsistent access between devices.
  • Ensure that browser settings do not block Java applets. Strict security settings might result in launch failures.
  • Ensure that encryption standards can be met by all supported client devices. Not using encryption is a security risk.
User Authentication and Authorization
An architect should examine the user authentication process during the assessment. Authentication is usually based on one of the following:
  • Explicit
  • Pass-through
  • Smart Card
  • Pass-through with Smart Card
For example, if explicit authentication is used for accessing desktops and local applications, an architect must determine whether that process is ideal or recommend an alternate type of authentication in the design.
Explicit authentication is usually recommended in Citrix environments.
User authentication also incorporates access to subsequent resources. For example, if Smart Card authentication is used for the desktop, access to an application may or may not support that type of authentication. In addition, an authentication tool such as Citrix Single Sign-on (formerly Password Manager) may be used to address subsequent authentication requirements.
User authorization involves assessing the permission levels for the categorized user types. Architects should ask the following questions:
  • Which user types are power users?
  • Which user types are allowed to install their own applications?
  • Do users have administrator status on their local desktops?
  • Are any users using Single Sign-On?
  • What are the user permissions on the XenApp servers, if applicable?
  • Are there any applications that require less restrictive or modifying the registry?
An understanding of user authorization in the environment will help the architect determine if any special security templates or modifications will need to be made in the design phase.
External Access Scenarios
To appropriately design a secure access solution for external users, architects must identify the various external access scenarios either currently used or required. These scenarios describe which users will be connecting to the environment externally and which resources those users will be able to access. For example, employees connecting externally to the environment from managed laptops might be granted full VPN access, which provides access to all the same resources those employees get when connecting from within the office. However, contractors might be granted limited VPN access or access only to published applications available through Web Interface. When identifying an organization's access scenarios, architects should answer the following types of questions:
  • Is external access currently provided for any users?
  • Who are the external users? Are they employees, contractors or vendors?
  • Are the client devices used for external access managed or unmanaged? Are they laptops or desktops?
  • How are ICA connections from external users secured? Does the organization currently have an SSL VPN solution such as Access Gateway or Secure Gateway?
  • How are users authenticated? Is Active Directory or two-factor authentication required?
    • Is Windows single sign-on to the Access Gateway plug-in required?
    • Is automatic single sign-on to web applications required?
  • Do any users require full SSL VPN access into the environment or can secure access be limited to XenDesktop and XenApp resources?
  • Are endpoint analysis scans (EPAs) needed to verify client device requirements, such as the anti-virus version, a Registry setting or the presence of an internal certificate? Will users that fail the endpoint analysis scans be quarantined or provided with limited access?
Policy Management
Policy management is important to assess in a virtualization environment.
Architects should ask the following questions:
  • What are the organization's policy settings?
  • Which resultant policies have been implemented?
  • Which group policies exist?
  • Which Citrix-specific policies have ben implemented, if applicable?
  • How are policies generally applied in the environment?
In an environment containing Citrix XenApp or XenDesktop, there are a number of ways to apply a configuration or security setting onto a group of servers. Policies can be applied through numerous methods and impact different aspects of the environment.
For more information, see the Citrix Consulting white paper "How Policies Impact XenApp Environments" on the web site.

Featured Post

Amazon Route 53

Amazon Route 53 is a highly available and scalable Domain Name System (DNS) web service.Route 53  perform three main functions in any...