eXpand yOur cReativity by Bhushan G Ahire

17Feb/100

Setup Capistrano to deploy Rails application on Amazon EC2 with Git

1: Create a new Rails app - we'll call is 'deploytest'

$ rails deploytest
$ cd deploytest

2: Create a local Git repository for it

$ git init
$ git add *
$ git commit -a -m 'initial commit'
$ git status

3: Create a couple of Capistrano files

$ capify .

4: Edit config/deploy.rb

# The name of your app
set :application, "deploytest"
# The directory on the EC2 node that will be deployed to
set :deploy_to, "/var/www/apps/#{application}"
# The type of Source Code Management system you are using
set :scm, :git
# The location of the LOCAL repository relative to the current app
set :repository,  "."
# The way in which files will be transferred from repository to remote host
# If you were using a hosted github repository this would be slightly different
set :deploy_via, :copy

# The address of the remote host on EC2 (the Public DNS address)
set :location, "ec2-xxx-xxx-xxx-xxx.compute-1.amazonaws.com"
# setup some Capistrano roles
role :app, location
role :web, location
role :db,  location, :primary => true

# Set up SSH so it can connect to the EC2 node - assumes your SSH key is in ~/.ssh/id_rsa
set :user, "root"
ssh_options[:keys] = [File.join(ENV["HOME"], ".ssh", "id_rsa")]

The only account on a default EC2 instance is root. You probably want to create a second user that is responsible for your application.

5: Copy your SSH public key to your EC2 node

$ scp -i ~/my-ec2-keypair ~/.ssh/id_rsa.pub root@ec2-xxx-xxx-xxx-xxx.compute-1.amazonaws.com:/root/.ssh/authorized_keys2

NOTE the filename authorized_keys2 - not authorized_keys!!

6: Setup the EC2 node for Capistrano deployment.
From your LOCAL machine, not the EC2 node:

$ cap deploy:setup

7: Finally, deploy your application

$ cap deploy

You will see lots of output and with this dummy application some of those will report errors/warnings. Don't worry about that for now.

8: Check that the Deployment was successful
Connect to the EC2 node with SSH the regular way, cd to the app directory and check that everything is there. If that is all working then you are ready to deploy a real application and add custom tasks for managing the database, restarting the server etc.

Bear in mind that Capistrano add new 'releases' of your software in separate directories and symlinks the 'current' directory to the latest. So the root of your deployed application is the 'current' subdirectory.

Hope this will help you setting up your ec2 instance with capistrano.

Tagged as: , , , , , No Comments
22Jul/091

Step-By-Step setup slicehost server (fedora) for rails application.

Hi all as I worked on setting up 4 to 5 servers, I thought its better to document the stuff so that I/developers can refer it, So I am documenting the step-by-step process for setting up the slicehost server for rails application work with nginx as a web server and mongrel as a rails application server.

Update

To update your system:

yum update

MySQL

to install mysql client and server apps

yum install mysql-server

The client package “mysql” will automatically be installed as a dependency.

Apache

yum install httpd

PHP5

To install php with soap, xml and mysql plugins:

yum install php php-soap php-xml php-mysql

some other php plugin utilities that are common:

yum install php-mbstring php-gd

Ruby

yum install ruby

Utilities

other useful utilities:

yum install nano wget elinks subversion vi

Nginx

install nginx web server:

yum install nginx

Mongrel Cluster

install mongrel_cluster:

gem install mongrel_cluster –include-dependencies

Setup your nginx configuration for your rails application and mongrel_cluster (proxy).

Below you found sample config file for nginx.conf (normally located in “/etc/nginx/conf/nginx.conf”)

user  deploy;
worker_processes  1;
error_log   logs/error.log debug;
#error_log  logs/error.log  notice;
#error_log  logs/error.log  info;
#pid        logs/nginx.pid;
events {
    worker_connections  1024;
}
http {
  include        conf/mime.types;
  default_type   application/octet-stream;
  sendfile        on;
  #tcp_nopush     on;
  keepalive_timeout  65;
  tcp_nodelay        on;
  gzip  on;
  gzip_min_length  1100;
  gzip_buffers     4 8k;
  gzip_types       text/plain;
  upstream mongrel {
    server 127.0.0.1:8000;
    server 127.0.0.1:8001;
  }
  server {
    listen       80;
    server_name  example.com;
    root /var/www/apps/example/current/public;
    index  index.html index.htm;
    location / {
      proxy_set_header  X-Real-IP  $remote_addr;
      proxy_set_header  X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header Host $http_host;
      proxy_redirect false;
      if (-f $request_filename/index.html) {
        rewrite (.*) $1/index.html break;
      }
      if (-f $request_filename.html) {
        rewrite (.*) $1.html break;
      }
      if (!-f $request_filename) {
        proxy_pass http://mongrel;
        break;
      }
    }
    error_page   500 502 503 504  /50x.html;
    location = /50x.html {
        root   html;
    }
  }
}

Restart the nginx server:

/etc/init.d/nginx restart

Mongrel Configuration:

Now Go to the application directory and create a configuration for Mongrel:

mongrel_rails cluster::configure -e production -p 8000 -a 127.0.0.1 -N 2 -c /home/APP/production/APP/current

it will create the config/mongrel_cluster.yml file; basically you do not need to edit it.

You can test cluster with

To Start:

mongrel_rails cluster::start

To Stop:

mongrel_rails cluster::stop

To Restart:

mongrel_rails cluster::restart

Capistrano Configuration:

Then create a configuration for Capistrano:

cap –apply-to .

we need to modify the generated file config/deploy.rb:

require 'mongrel_cluster/recipes'

#you set the APP name with the cap command
set :application, "APP"
#a path to your repository
set :repository, "svn+ssh://USERNAME@SVN_SERVER/projects/#{application}/trunk"

role :web, "SERVER"
role :app, "SERVER"
role :db, "SERVER", :primary => true

#where to deploy (copy the files) on the server; I created a special user APP for the
application (if you do not like it, replace the /home/#{application} part with your path
set :deploy_to, "/home/#{application}/production/#{application}"
set :mongrel_conf, "#{current_path}/config/mongrel_cluster.yml"
#if the server login name is different to the development computer login name; in my case
the user name is the APP nameset :user, "APP"

Make necessary changes to the config/database.yml file.

Capistrano Deployment:

Create the basic structure on the server:

cap deploy:setup

For the first deployment you can use cold deploy:

cap cold_deploy

After that for next deployments you have to use:

cap deploy

You can also run migration from capistrano with:

cap deploy:migrate*

* Note: It wont work for me, it migrating to the previous release available in the releses folder, So I suggest you manually run migration on server.

That’s it, your server is ready to run.

Hope this documentation will be helpful to you, Enjoy ;-) .

Tagged as: , , , , 1 Comment
15Apr/09Off

Install and Configure FTP Server in Amazon EC2 instance

For many users, running FTP Sever in Amazon EC2 instance is headache at the first time. You need to experiment before being able to transfer data. The main problems are Ingress firewall in Amazon environment and NAT traversal.

Here I’m using vsftp (vsfptd) Server, which is one of the most popular and easy to configure. The instance is running from base Fedora 4 AMI but the setup should be identical to other Red Hat based distros.

Install vsftpd FTP server, if not installed earlier:

# yum install vsfptd

Its upto you which FTP method i.e. Active or Passive you want to use. The problem with active mode is that your computer is sending a request out of port 21 when all of a sudden, the server attempts to initiate a request with your computer on port 20. Since communication on port 21 does not imply communication on port 20, it appears as if some unauthorized host has attempted to initiate a new connection with your computer. Kind of sounds like a hack right? Your firewall may think so too (or your NAT router may have no idea to which computer to route the request). Active mode is not used as default method of ftp transfer in many clients these days.

On the other hand, as the Ingress firewall is running in AWS, from the firewall’s standpoint, to support passive mode FTP the following communication channels need to be opened:

FTP server’s port 21 from anywhere (Client initiates connection).
FTP server’s port 21 to ports > 1023 (Server responds to client’s control port).
FTP server’s ports > 1023 from anywhere (Client initiates data connection to random port specified by server).
FTP server’s ports > 1023 to remote ports > 1023 (Server sends ACKs (and data) to client’s data port).

That second part is the problem: FTP server listens on a random port and hands that back to the client, so the client initiates a connection to a random server port, which you must allow.

Opening up all ports > 1023 isn’t so good for security. But what you can do is allow the ports through the distributed firewall and then setup your own filtering inside your instance. Instead, you would better open a fixed number of ports (such as 1024 to 1048) and configure your FTP Server to only use that ports.

Check whether required ports are open or not in your EC2 security group. (if you are unaware about security group, it should be ‘defaul’ unless you created a new one).

# ec2-describe-group

This command will print all ports which are currently open. If you dont find port 20,21,1024-1048 then you need to open these ports but if you dont find the command itself i.e.
# ec2-describe-group
-bash: ec2-describe-group: command not found

You need to install ec2 command line tools. You can find them here and the instructions to setup/configure can be found here.

Open the ports now:

# ec2-authorize default -p 20-21
# ec2-authorize default -p 1024-1048

Here, ‘default’ is the name of security group. You can also open ports for specific IPs. For ease of use, you better install ElasticFox, a firefox extension to manage EC2 stuff. you can find more about it here.

At this moment, you can start your FTP server and if you try to connect it, the process will get failed. By checking logs, you should find something like:

Status: Connected
Status: Retrieving directory listing...
Command: PWD
Response: 257 "/" is current directory.
Command: TYPE A
Response: 200 Type set to A
Command: PASV
Response: 227 Entering Passive Mode (216,182,238,73,129,75).
Command: LIST
Error: Transfer channel can't be opened. Reason: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
Error: Could not retrieve directory listing

Time to configure vsftpd.conf file:
# vi /etc/vsftpd/vsftpd.conf
---Add following lines at the end of file---
pasv_enable=YES
pasv_min_port=1024
pasv_max_port=1048
pasv_address=Public IP of your instance

Put public IP of your EC2 instance and then Save the file. Now restart the server:

# /etc/init.d/vsftpd restart

Tagged as: , , , , , , Comments Off
19Feb/090

Setting up a new remote git repository

For the impatient

Set up the new bare repo on the server:

$ ssh myserver.com
Welcome to myserver.com!
$ mkdir /var/git/myapp.git && cd /var/git/myapp.git
$ git --bare init
Initialized empty Git repository in /var/git/myapp.git
$ exit
Bye!

Add the remote repository and push:

$ cd ~/Sites/myapp
$ git remote add origin ssh://myserver.com/var/git/myapp.git
$ git push origin master

Set the local master branch to track the remote branch.

Read further for a step-by-step explanation of what’s going on.

Pre-flight sanity check

Setting up a remote repository is fairly simple but somewhat confusing at first. Firstly, let’s check out what remote repositories are being tracked in our git repo:

$ cd ~/Sites/myapp
$ git remote

None. Looking good. Now let’s list all the branches:

$ git branch -a
* master

Just one branch, the master branch. Let’s have a look at .git/config:

$ cat .git/config
[core]
        repositoryformatversion = 0
        filemode = true
        bare = false
        logallrefupdates = true

A pretty bare-minimum config file.

Creating the bare remote repository

Before we can push our local master branch to a remote repository we need to create the remote repository. To do this we’ll ssh in and create it on the server:

$ ssh myserver.com
Welcome to myserver.com!
$ cd /var/git
$ mkdir myapp.git
$ cd myapp.git
$ git --bare init
Initialized empty Git repository in /var/git/myapp.git
$ exit
Bye!

A short aside about what git means by bare: A default git repository assumes that you’ll be using it as your working directory, so git stores the actual bare repository files in a .git directory alongside all the project files. Remote repositories don’t need copies of the files on the filesystem unlike working copies, all they need are the deltas and binary what-nots of the repository itself. This is what “bare” means to git. Just the repository itself.

Adding the remote repository to our local git repository configuration

Now that we’ve created the remote repository we’ll add it to our local repository as a remote server called “origin” using git remote add, which is just a nicer way of updating our config file for us:

$ git remote add origin ssh://myserver.com/var/git/myapp.git

Let’s see what it added to the config file:

[core]
  repositoryformatversion = 0
  filemode = true
  bare = false
  logallrefupdates = true
[remote "origin"]
  url = ssh://myserver.com/var/git/myapp.git
  fetch = +refs/heads/*:refs/remotes/origin/*

We now have a remote repository “origin” that will fetch all of it’s refs/heads/* branches and store them in our local repo in refs/remotes/origin/* when a git fetch is performed.

Pushing to the remote repository

The time has come to push our local master branch to the origin’s master branch. We do that using the git push command.

$ git push origin master
updating 'refs/heads/master'
  from 0000000000000000000000000000000000000000
  to   b379203bc187c2926f44a71eca3f901321ea42c6
 Also local refs/remotes/origin/master
Generating pack...
Done counting 1374 objects.
Deltifying 1374 objects...
 100% (1374/1374) done
Writing 1374 objects...
 100% (1374/1374) done
Total 1374 (delta 89), reused 0 (delta 0)
refs/heads/master: 0000000000000000000000000000000000000000 -> b379203bc187c2926f44a71eca3f901321ea42c6

and that’s all, folks. Further pushes can be done by repeating the git push command.

Now you can tell your co-conspirators to:

$ git clone ssh://myserver.com/var/git/myapp.git

and push and pull to your heart’s content.

Track the remote branch

You can specify the default remote repository for pushing and pulling using git-branch’s track option. You’d normally do this by specifying the --track option when creating your local master branch, but as it already exists we’ll just update the config manually like so:

[branch "master"]
  remote = origin
  merge = refs/heads/master

Now you can simply git push and git pull.

Sharing the remote repository with the world

If you want to set it up as a public repository be sure to check out the Git manual’s chapter on public git repositories.

Working with remote repository branches

git remote show is used to inspect a remote repository. It goes and checks the remote repository to see what branches have been added and removed since the last git fetch.

Doing a git remote show at the moment only shows us the remote repo’s master branch which we pushed earlier:

$ git remote show origin
* remote origin
  URL: ssh://myserver.com/var/git/myapp.git
  Tracked remote branches
    master

Let’s create a new local git repository and push to a new branch on the remote repository. We can then use git remote show to see the new remote branch, git fetch to mirror it into our local repo and git checkout --track -b to create a local branch to do some work on it.

We’ll start by creating a new local repo and pushing some code to a new branch in the remote repository.

$ mkdir /tmp/other-git
$ cd /tmp/other-git
$ git init
Initialized empty Git repository in /tmp/other-git
$ git remote add origin ssh://myserver.com/var/git/myapp.git
$ echo "Rails 2... woo" > afile
$ git add afile
$ git commit -m "Added afile" 
Created initial commit 0ac9a74: Added afile
 1 files changed, 1 insertions(+), 0 deletions(-)
 create mode 100644 something
$ git push origin master:refs/heads/dev/rails-2
updating 'refs/heads/rails-2' using 'refs/heads/master'
  from 0000000000000000000000000000000000000000
  to   0ac9a7457f4b21c9e058d4c54d262584bf35e528
 Also local refs/remotes/origin/rails-2
Generating pack...
Done counting 3 objects.
Deltifying 3 objects...
 100% (3/3) done
Writing 3 objects...
 100% (3/3) done
Total 3 (delta 0), reused 0 (delta 0)
Unpacking 3 objects...
 100% (3/3) done
refs/heads/rails-2: 0000000000000000000000000000000000000000 -> 0ac9a7457f4b21c9e058d4c54d262584bf35e528

Now let’s switch back to our old git repository and see if it detects the new branch on the remote repository:

$ git remote show origin
* remote origin
  URL: ssh://myserver.com/var/git/myapp.git
  New remote branches (next fetch will store in remotes/origin)
    dev/rails-2
  Tracked remote branches
    master

Let’s update our mirror of the remote repository by doing a git fetch:

$ git fetch
* refs/remotes/origin/master: storing branch 'dev/rails-2' of ssh://myserver.com/var/git/myapp.git
  commit: b379203
$ git remote show origin
* remote origin
  URL: ssh://myserver.com/var/git/myapp.git
  Tracked remote branches
    master
    dev/rails-2

We should now be able to see this in a our list of remote branches:

$ git branch -a
* master
origin/dev/rails-2
origin/master

If we then wanted to do some work on this remote dev/rails-2 branch we create a new local tracking branch like so:

$ git checkout --track -b dev/rails-2 origin/dev/rails-2
Branch dev/rails-2 set up to track remote branch refs/remotes/origin/dev/rails-2.
Switched to a new branch "dev/rails-2"

To keep up-to-date and push new changesets we simply use git push and git pull when working in the local dev/rails-2 branch.

Also notice, like we manually changed for master, .git/config has a new entry for this new tracking branch:

[branch "dev/rails-2"]
  remote = origin
  merge = refs/heads/dev/rails-2
Reference from http://toolmantim.com/articles/setting_up_a_new_remote_git_repository
Tagged as: , , , No Comments
18Mar/080

god Server process monitoring and notifying tool written in ruby

Progress on god is moving along as quick as ever. Most interestingly you'll find several useful new command line functions:

  • god status prints out the status of each Watch
  • god log shows realtime logs for a specific Watch (even if you don't have god logging to file)
  • god load loads or reloads a config file into a running god instance
  • god terminate stops all Watches and then stops god (useful when testing your setup)

The logging system has been beefed up with proper timestamps and criticality levels. Log messages are more complete overall. You can also get the STDOUT/STDERR of a god-daemonized process written to a log file by specify 'w.log = ' in your Watch config.

If you let god daemonize your process for you, there's no need to provide a stop command. A default killing lambda will take care of gracefully (or not so gracefully if necessary) stopping your god-daemonized process.

The validity of your config file is checked better than previous versions to point you to the problem area of your config.

The bug that prevented group control from working has been fixed so you can now start/stop/etc groups of Watches.

Updated documentation is now available on the website:

http://god.rubyforge.org/

WHAT IS GOD?

God is an easy to configure, easy to extend monitoring framework written in Ruby.

Keeping your server processes and tasks running should be a simple part of your deployment process. God aims to be the simplest, most powerful monitoring application available.

INSTALL

sudo gem install god

FEATURES

  • Config file is written in Ruby
  • Easily write your own custom conditions in Ruby
  • Supports both poll and event based conditions
  • Different poll conditions can have different intervals
  • Easily control non-daemonized processes

EXAMPLE

The easiest way to understand how god will make your life better is by looking at a sample config file. The following configuration file is to keep the mongrels running:

# file:      application_name.god
# run with:  god -c /path/to/application_name.god
# 
# This is the actual config file used to keep the mongrels of
# application.com running.
RAILS_ROOT = "/var/www/application_name/current"
%w{8001 8002 8003}.each do |port|
  God.watch do |w|
    w.name = "application_name-mongrel-#{port}"
    w.interval = 30.seconds # default
    w.start = "mongrel_rails cluster::start --only #{port} \
      -C #{RAILS_ROOT}/config/mongrel_cluster.yml"
    w.stop = "mongrel_rails cluster::stop --only #{port} \
      -C #{RAILS_ROOT}/config/mongrel_cluster.yml"
    w.grace = 10.seconds
    w.pid_file = File.join(RAILS_ROOT, "log/mongrel.#{port}.pid")    
    w.behavior(:clean_pid_file)
    w.start_if do |start|
      start.condition(:process_running) do |c|
        c.interval = 5.seconds
        c.running = false
      end
    end 
    w.restart_if do |restart|
      restart.condition(:memory_usage) do |c|
        c.above = 150.megabytes
        c.times = [3, 5] # 3 out of 5 intervals
      end    
      restart.condition(:cpu_usage) do |c|
        c.above = 50.percent
        c.times = 5
      end
    end
  end
end

Configuration for nginx and mysql
I found a gr8 post for sample configuration here

DOCS

Detailed documentation is available at http://god.rubyforge.org/

Original Source...

Tagged as: , , No Comments
22Jan/080

Shorcuts for kill and restart rails server

More lovely alias commands... this time to kill/restart Rail's script/server from any Terminal session or login on your box... (as long as your the same user).

alias dierails='ps -a|grep "/usr/local/bin/ruby script/server"|grep -v "grep /usr"|cut -d " " -f1|xargs -n 1 kill -KILL $1'
alias resetrails='ps -a|grep "/usr/local/bin/ruby script/server"|grep -v "grep /usr"|cut -d " " -f1|xargs -n 1 kill -HUP $1'
Tagged as: , No Comments
   

Bhushan G Ahire



Pages

Categories

Blogroll

Archive

Meta