VPS experiance & Domain Name Registrar

This has been a busy week. I acquired a Virtual Private Server, a Domain and I have been having my first experiences with Apache2 since I am normally an Nginx man.

Firstly my reasons for getting a VPS are a few:
I did not want to leave my computer on all the time.
If I did leave my computer on, my IP is not static.
The electric bill would be more expensive than a small VPS anyway.
My main computer is windows so I would not get any of the lovely *NIX goodies through SSH.
A VPS is on all the time if I choose.
It is more online storage.
I wanted a personal GIT server.
I wanted to learn more about servers and networking.

So with this in mind, I started my search. Because I wanted it for personal use and GIT, I was looking for hard drive space over bandwidth or transfer limits.
A friend of mine at the university owned fusevps but sadly when I looked it appeared to be nuked from orbit.
My next stop was a VPS offers list: Low End Box, I like helping the local economy so I restricted my search to UK hosting and found several great offers from EaseVPS, CastleGem, Edis and MiniVPS. MiniVPS's offering fitted my criteria best and so I chose them to be my first VPS.

They have a nice control panel to change Operating Systems, shutdown, restart, change hostname, see usage  graphs and the like which is nice, and their installs come with Apache2 running automatically which I did not expect. Overall I like the machine, it feels speedy enough and I have Gitolite set up so it really has everything I was going for.

So onto the next thing.

I managed to get my hands on bookofgreg.co.uk for a reasonable price from 123-reg (I didn't search around too much except to avoid godaddy), someone already had bookofgreg.com which is a little strange to think someone might be using my alias but seems it is not actively used at the moment. Using 123-reg's control panel I managed to set the DNS to point everything but mail to my VPS so I can happily use git@bookofgreg... now. It was at this point when checking the DNS was pointing to the right place I discovered Apache was running until it gave it's "everything is Ok" message when going to the browser.

Overall it is surprisingly quick, simple and easy to get a Domain name pointing to a VPS. Both VPS provider and Domain Name Registrar have everything automated to a T and it is such a quick and easy system. Very pleasantly surprised. Next Step, mail server setup (This is going to be HELL!)

Process Watchman

Update: If the code it is running completes before the maximum time, it will block the program from completing until the time runs out. Still needs work! (Read: It doesn't work XD)

I was looking for a process monitor, sort of like monit, or god but these two solutions are for keeping a process alive or restarting a process if it is misbehaving. What I wanted was a simple process watcher that would simply kill a process if it misbehaves or runs out of time, while returning the completed or partial output of the process it was watching.

For this I quickly wrote a small tool to do this on *nix machines.
It is quite hacked together so I must apologise for the mess, it is in its first version here and the program was written as fast as I could imagine without prior design. There will be numerable ways of improving this tool.

Currently it accepts 3 args, a command, an integer value for seconds and a float value for ram megabytes. There is probably a problem with commands that are more than one word long which is most of them, experimentation and improvements are needed.

#Author: Greg Myers
#Date: 30/08/12
@command = ARGV[0]
@time_limit = ARGV[1].to_i #seconds
@ram_limit = ARGV[2].to_f #megabytes

def watch_time(sec)
  exception = Class.new(Interrupt)
  begin
    x = Thread.current #x will contain whatever runs in yield
    y = Thread.start { #y watches and causes x to throw when timeout.
      begin
        sleep(sec)
      rescue => e #This raises any error x naturally hits
        x.raise e
      else #This executes if no exceptions happened until now
        x.raise "Process ran out of time to execute."
      end
    }
    return yield(sec)
  ensure
    if y
      y.kill
      y.join
    end
  end
end

def watch_ram(megabytes, pid)
  exception = Class.new(Interrupt)
  begin
    x = Thread.current #x will contain whatever runs in yield
    y = Thread.start { #y watches and causes x to throw when timeout.
      begin
        loop {
          rss_use = `ps -o rss= -p #{pid}`.to_i #use ps to get rss of pid
          raise "Hit Ram Limit #{megabytes*1024}kb, with #{rss_use}kb" if megabytes*1024 < rss_use
          sleep(0.5)
        }
      rescue => e #This raises any error x naturally hits
        x.raise e
      end
    }
    return yield
  ensure
    if y
      y.kill
      y.join
    end
  end
end

def get_payload_child_pid(pid)
  pipe = IO.popen("ps -ef | grep #{pid}")
  pipe.readlines[2] =~ /\w+\s+(?\d+)\s+(?\d+)/ #Always line 3, Line 1 = spawn cmd, Line 2 = IO.popen, Line 3 = ruby, Line 4 = grep
  pipe.close
  return $~[:child_pid]
end

if @command
  if @time_limit > 1
    if @ram_limit > 0
      watch_time(@time_limit){
        require 'pty'
        PTY.spawn("#{@command} 2>&1") do |r,w,p|
          child_pid = get_payload_child_pid(p)
          watch_ram(@ram_limit, child_pid){ loop { puts r.gets } }
        end
      }
    else
      puts "Invalid memory limit #{ARGV[2]}"
    end
  else
    puts "Invalid time limit #{ARGV[1]}"
  end
else
  puts "Invalid command #{ARGV[0]}"
end

I will be putting this on github publicly shortly.

Multicore programming with ruby

As I understand it multicore programming is about the ability to utilize multiple CPUs.
Also I hear a lot of people saying ruby cannot use multiple CPUs.

This is a load of rubbish if you are on a *nix system.

All you have to do if you want to use multiple CPUs is fork{} and it is running in its own process.
You can leave whatever is in the block to run and reap the process later, or detatch it if you do not need to see the results.

Fork returns the PID of the process it spawns, and used with Process.waitpid(), you can make the main process wait for the results to come back from the child. If you want the results yourself, you can just open a pipe between the child and the main process with IO.pipe

Simple.

read, write = IO.pipe
pid = fork do
  write.puts "test"
end
Process.waitpid(pid)
write.close
puts read.read
read.close


Resources:
http://www.ruby-doc.org/core-1.9.3/IO.html
http://www.ruby-doc.org/core-1.9.3/Process.html
Example based on:
http://stackoverflow.com/questions/1076257/returning-data-from-forked-processes

Rails Time

Earlier in the month I was experiencing some odd behaviour in ruby.

irb(main):002:0> DateTime.now
=> Mon, 13 Aug 2012 17:04:34 +0100
irb(main):003:0> DateTime.now + 2.days
=> Sat, 22 Sep 2485 17:04:42 +0100

DateTime is a class with a limit that goes up to year 9999 and as far as I can tell isn't stored as seconds, the more commonly used Time class is measured in seconds and goes up to 03:14:08 UTC on 19 January 2038 on 32 bit (thanks Wikipedia). All the Railsy time goodies (example, 5.weeks.from_now) are measured in seconds too, and so are only compatible with Time (or Date) class but not DateTime.

It is almost misleading that ActiveRecord says time is stored as DateTime in databases, it is actually stored as Time.


References
DateTime
http://www.ruby-doc.org/stdlib-1.9.3/libdoc/date/rdoc/DateTime.html

Date
http://www.ruby-doc.org/stdlib-1.9.3/libdoc/date/rdoc/Date.html

Time in Core
http://ruby-doc.org/core-1.9.3/Time.html
Time in Date. The difference is this has to_date, to_time, and to_datetime methods.
http://www.ruby-doc.org/stdlib-1.9.3/libdoc/date/rdoc/Time.html

Story of Compiling Kernel for Linux VServer

So I've been having a fun week.

This is my first time compiling a kernel and It should be straight forward as there are some helpful websites available on just this topic:

pre-requisites:
Patience. This will take an hour (or more) on a decent quad core machine.
gcc-<version>-plugin-dev
fakeroot
make-kpkg
libncurses5-dev

On compiling the kernel with the VServer patch
http://linux-vserver.org/Installation_on_Linux_2.6
On compiling a general kernel
http://www.howtoforge.com/kernel_compilation_ubuntu
On using your complete kernel
http://www.centos.org/docs/2/rhl-cg-en-7.2/buildkernel-bootloader.html

I was running the commands on a Ubuntu12.04 virtualbox machine, following the Vserver instructions and while running make, it kept running out of disk space.
A quick google tells me the result should be about 500MB at most, and doesn't even tell you what the result should be. My attempts kept running out of space after about 40 minutes and after hitting 8GB used space, of which almost all the room was taken up by vmlinux (2.9GB) and vmlinux.o (4.9GB)
Apparently this is not normal.
Eventually after making a virtual hd with a size of 100GB (of which the result was just short of 9GB itself) it appeared to say a build of bzimage in /arch/boot/x86 folder was completed. This I could never find anywhere in the folder.

I am currently compiling using the howtoforge.com link so we will see how much success I shall have later on.
Currently I am getting GCC error "array subscript is above array bounds" quite a lot but it appears to be ignoring it.

#Edit
Its been 2 hours. I didn't set CONCURENCY_LEVEL= because of fear of it crashing. Starting to wonder if it was a mistake given how long this is taking. It is still compiling and it has just moved from ./drivers to ./fs. My favourite file list, sudo du --max-depth=1 -h | sort -rh , reports file is at 7.9GB and growing.


7.9G .
4.8G ./drivers
689M ./net
602M ./fs
316M ./arch
259M ./sound
126M ./kernel
76M ./security
58M ./crypto
41M ./mm
34M ./lib
31M ./include
19M ./Documentation
17M ./block
13M ./.tmp_versions
6.8M ./firmware
6.2M ./ipc
5.0M ./grsecurity
4.7M ./init
4.1M ./tools
3.8M ./scripts
2.8M ./virt
2.0M ./debian
156K ./samples
92K ./usr

Foolproof FactoryGirl Sequence

Turns out Sequencing in FactoryGirl isn't as easy as it used to be.
Factory.next has been deprecated and the two obvious ways to make a sequence now throw Trait error or Attribute already defined error "FactoryGirl::AttributeDefinitionError"

The problem with this first approach is name. In a normal sequence you would just say name but I want to use it in the description too. This somehow makes FactoryGirl believe name is a trait.

FactoryGirl.define do
  
  sequence(:code) {|n| "#{n+1000}"}
  
  factory :course do |u|
    name "CS#{ generate :code }"
    description "#{name}, 1st year, 1st semester, Foundation Of #{name}"
  end
end

This next example has a problem with the description line. This causes the description to generate 2 more names and throw an AttributeDefinitionError because you now have more than 1 name generated.
For some reason you cannot get around it by making the name line name = Factory.generate :name because name becomes a Trait again.

FactoryGirl.define do
  
  sequence(:name) {|n| "CS#{n+1000}"}
  
  factory :course do |u|
    name
    description "#{name}, 1st year, 1st semester, Foundation Of #{name}"
  end
end

This is the only thing that works. Static forcing FactoryGirl to use the already defined name again using the |a| block in description. It isn't as pretty as it could be but I have to live with it unless someone can tell me a better way. There is probably a solution where you use an after_build tag but that makes it longer and more complicated unnecessarily and it would still be ugly.

FactoryGirl.define do
  
  sequence(:name) {|n| "CS#{n+1000}"}
  
  factory :course do |u|
    u.name
    u.description {|a| "#{a.name}, 1st year, 1st semester, Foundation Of #{a.name}"}
  end
end

Importing a CSV File from Upload in Rails >= 3

For some reason I found the documentation for CSV's really bad!

What I wanted to do is accept a file from a form and parse it into a list of users to be added to the database without saving the CSV file. (Note: I DO save the file via the Paperclip Gem for logging purposes but I do all the processing from the file before it is saved to the database.)

When you have uploaded the file using Form_Tag, the file is usually in params[:name] where :name is tag you gave to file_field_tag :name
If you are using a Form_For, which by the way is strange if you do not intend to save it to the model in question, the file will be in params[:model][:name] from  f.file_field :name
The magic part is to actually get to the file you need to call .read on it.

In the View remember to make it multipart to accept files.


<%= form_for @model, :html => { :multipart => true } do |f| %>
  <%= f.label :file %><br />
  <%= f.file_field :file %><br /> 
  <%= f.submit "Upload" %>
<% end %>



In the Controller require csv, you do not need to install it as a gem but you do need to require it as it is in standard library not core.

require 'csv'
def my_method
@lines = []
CSV.parse(params[:submission][:file].read) do |row|
      @lines << row
end


Replace the yellow bits with whatever you want to do with your CSV file.
Have fun!