Sunday, March 23, 2014

What is mean by vagrant?

Vagrant is used to set up one or more virtual machines by:

Importing pre-made images (called "boxes")Setting VM-specific settings (IP address, hostnames, port forwarding, memory, etc.)Running provisioning software like Puppet or Chef

Note that it doesn't install software or set up the machine past loading the VM and setting VirtualBox settings. Think of it as a scripting engine for VirtualBox.

Here are some reasons I've seen for using Vagrant over just VirtualBox.

1. Set Up Multi-VM Networks with Ease

Most of the Vagrant power-user content I've read has been about setting up multiple VMs at the same time. Vagrant gives you a single config file to set these up, enabling you to launch all of them with one command.

Say you've configured three VMs to network with each other using static IPs on the 192.168.1.* subnet. You find yourself in a location that is already using that subnet to hand out IP addresses, and your VMs now conflict. With Vagrant, you can simply edit the Vagrantfile and reload the VMs, whereas with VirtualBox you'd have to open the settings for each VM, if not boot each VM and change them inside.

2. Source Control

By putting the settings in a text file, it enables the configuration to be put under source control. Made some changes last week and accidentally broke the image? Just revert the changes and reload the VM. You can accomplish this with VirtualBox snapshots, but it will take up much more space than just a Vagrantfile.

3. Various Platforms

There's a large number of boxes available at sites such as This enables you to try various OSes or distributions, applying the same provisioning to set up similar environments. This can help with testing or adding support to new platforms, and would be time-consuming using just VirtualBox.

Thursday, March 20, 2014

How Twitter Monitor Millions of Time Series

Techniques to scale your Web Site fast

Find the bottleneck

To find the bottleneck first empty the browser cache and reloaded the page with firebug running. The Net panel showed that it took 24 seconds to load the initial page. After that, the remaining files loaded quickly.

I should have realized right away that this behavior meant the server was hamstrung by a thread limit. It took me ten minutes to figure that out the bottle neck

Step 1: Cut image quality

Since the new post was my first image-heavy post, I realized I could cut bandwidth consumption in half by compressing images.
ImageMagick's convert tool can shrink images at the command line:
 $ convert image.ext -quality 0.2 image-mini.ext
I wrote a shell script to compress every image in my images directory.
Then, I did a search-and-replace in emacs to switch everything to the -miniimages. Page load time dropped from 24 to 12 seconds.
Quick and dirty, yet effective.

Step 2: Make content static

I use server-side scripts to generate what are actually static pages.
Under heavy load, dynamic content chews up time.
So, I scraped the generated HTML out of View Source and dropped it in a static index.html file.
Page load times dropped to 6 seconds. Almost bearable!

Step 3: Adding threads in the apache conf

In firebug, pages were sill bursting in after the initial page load.
On a hunch, I checked with my linode control panel. I saw that the CPU utilization graph was near 3%, and that there was plenty of bandwidth left.

Suddenly, I remembered that the default apache configuration sets a low number of processes and threads.

Requests were streaming in and getting queued, waiting for a free thread.
Meanwhile, the CPU was spending 97% of its time doing nothing.
I opened my apache configuration file, found the mpm_worker_module section, and ramped up processes and threads:
 <IfModule mpm_worker_module>
    StartServers          4  # was 2
    MaxClients          600  # was 150
    MinSpareThreads      50  # was 25 
    MaxSpareThreads     150  # was 75
    ThreadsPerChild      25
    MaxRequestsPerChild   0
Page load times fell to two seconds.