While on Internship I've been asked not to blog anything that may appear in my report until after it has been marked. To minimize risk, this means radio silence. So far I've learned such an absolute mountain of information that it makes my earlier blog posts seem novice and I can fill in those blanks at a later date.
Thanks.
G
Notes Of Greg
Monday, 24 March 2014
Sunday, 25 August 2013
Current Windows Environment
Small intro:
I have been introduced to vagrant and rvm several years back when they were both rather unpolished and I ran into several issued with them at the time. rvm still has some obscure bugs when you are really stretching the limits of what it was built for but I now feel comfortable that both of these tools have come far enough to surpass my usual workflow and they have entered my normal use now.
This is all installed ontop of Windows 7 x64.
You may need to install VirtualBox before vagrant as it needs to hook into it.
Install cygwin for the terminal, Vagrant for the machine and Sublime for the editor.
In cygwin: using Solarized Dark set in my .minttyrc for theme.
Using Arnold Clark's polygot for aliases in cygwin and vagrant.
I picked a debian box running on virtualbox but you can set this how you like. Vagrant Boxes
In cygwin terminal, cd ~
vagrant box add deb http://puppet-vagrant-boxes.puppetlabs.com/debian-607-x64-vbox4210-nocm.box
vagrant init deb
vagrant up
vagrant ssh
That's the machine done.
Now for ruby.
curl -L https://get.rvm.io | bash -s stable --ruby=1.9.3
Done!
Sublime setup:
Download sublime text 2
Package manager:
https://sublime.wbond.net/installation#st2
Get packages:
"Arnold Clark Snippets for Ruby",
"CoffeeScript",
"ERB Snippets",
"Rails Partial",
"rails_go_to_spec",
"rspec-snippets",
"Sass",
"SideBarEnhancements",
"StringEncode",
"TrailingSpaces"
Set keybindings (I've been working on a mac lately so some of these keybindings may seem wonky.)
{ "keys": ["alt+shift+l"], "command": "reindent", "args": { "single_line": false } },
{ "keys": ["alt+shift+`"], "command": "delete_trailing_spaces" },
{ "keys": ["alt+shift+h"], "command": "html_entitize" },
{ "keys": ["alt+shift+."], "command": "rails_go_to_spec" },
{ "keys": ["alt+control+right"], "command": "indent" },
{ "keys": ["alt+control+left"], "command": "unindent" },
{ "keys": ["alt+control+up"], "command": "swap_line_up" },
{ "keys": ["alt+control+down"], "command": "swap_line_down" },
{ "keys": ["alt+left"], "command": "move_to", "args": {"to": "bol", "extend": false} },
{ "keys": ["alt+right"], "command": "move_to", "args": {"to": "eol", "extend": false} },
{ "keys": ["shift+alt+left"], "command": "move_to", "args": {"to": "bol", "extend": true} },
{ "keys": ["shift+alt+right"], "command": "move_to", "args": {"to": "eol", "extend": true} }
I have been introduced to vagrant and rvm several years back when they were both rather unpolished and I ran into several issued with them at the time. rvm still has some obscure bugs when you are really stretching the limits of what it was built for but I now feel comfortable that both of these tools have come far enough to surpass my usual workflow and they have entered my normal use now.
This is all installed ontop of Windows 7 x64.
You may need to install VirtualBox before vagrant as it needs to hook into it.
Install cygwin for the terminal, Vagrant for the machine and Sublime for the editor.
In cygwin: using Solarized Dark set in my .minttyrc for theme.
Using Arnold Clark's polygot for aliases in cygwin and vagrant.
I picked a debian box running on virtualbox but you can set this how you like. Vagrant Boxes
In cygwin terminal, cd ~
vagrant box add deb http://puppet-vagrant-boxes.puppetlabs.com/debian-607-x64-vbox4210-nocm.box
vagrant init deb
vagrant up
vagrant ssh
That's the machine done.
Now for ruby.
curl -L https://get.rvm.io | bash -s stable --ruby=1.9.3
Done!
Sublime setup:
Download sublime text 2
Package manager:
https://sublime.wbond.net/installation#st2
Get packages:
"Arnold Clark Snippets for Ruby",
"CoffeeScript",
"ERB Snippets",
"Rails Partial",
"rails_go_to_spec",
"rspec-snippets",
"Sass",
"SideBarEnhancements",
"StringEncode",
"TrailingSpaces"
Set keybindings (I've been working on a mac lately so some of these keybindings may seem wonky.)
{ "keys": ["alt+shift+l"], "command": "reindent", "args": { "single_line": false } },
{ "keys": ["alt+shift+`"], "command": "delete_trailing_spaces" },
{ "keys": ["alt+shift+h"], "command": "html_entitize" },
{ "keys": ["alt+shift+."], "command": "rails_go_to_spec" },
{ "keys": ["alt+control+right"], "command": "indent" },
{ "keys": ["alt+control+left"], "command": "unindent" },
{ "keys": ["alt+control+up"], "command": "swap_line_up" },
{ "keys": ["alt+control+down"], "command": "swap_line_down" },
{ "keys": ["alt+left"], "command": "move_to", "args": {"to": "bol", "extend": false} },
{ "keys": ["alt+right"], "command": "move_to", "args": {"to": "eol", "extend": false} },
{ "keys": ["shift+alt+left"], "command": "move_to", "args": {"to": "bol", "extend": true} },
{ "keys": ["shift+alt+right"], "command": "move_to", "args": {"to": "eol", "extend": true} }
Set preferences:
"ensure_newline_at_eof_on_save": true,
"ignored_packages":
[
"Vintage"
],
"save_on_focus_lost": true,
"tab_size": 2,
"trailing_spaces_trim_on_save": true,
"translate_tabs_to_spaces": true
Sunday, 2 December 2012
Understanding REST In Practical Terms
I have been using Rails for a little while now and REST is always floating around conversations about designing things in Rails. It seemed like a nebulous topic so I did a little research of my own.
To my understanding it is all about building resources and nouns that respond to the HTTP verbs.
Instead of sending a GET request to http://www.example.co.uk/user/create , you PUT to http://www.example.co.uk/user
To my understanding it is all about building resources and nouns that respond to the HTTP verbs.
Instead of sending a GET request to http://www.example.co.uk/user/create , you PUT to http://www.example.co.uk/user
Similarly you POST to user/151 if you wish to update him.
It really is that simple.
There is one Gotcha though. If you want to delete user 151, you send a DELETE request to /user/151, you can then perfectly validly send another DELETE request to /user/151 because it is a word new to me, Idempotent, which roughly means you can do the same request to the same noun and the system will not change.
If I were to PUT (create) the same user twice, it is also idempotent and should only have the first user assuming you have some valid way of identifying duplicates.
There is one Gotcha though. If you want to delete user 151, you send a DELETE request to /user/151, you can then perfectly validly send another DELETE request to /user/151 because it is a word new to me, Idempotent, which roughly means you can do the same request to the same noun and the system will not change.
If I were to PUT (create) the same user twice, it is also idempotent and should only have the first user assuming you have some valid way of identifying duplicates.
Tuesday, 23 October 2012
Random interesting system thing of the day
Just a short one.
When you have delayed job (Or anything with lots of children)t gets tedious writing ps aux | grep del then 100x kill -STOP 11992, kill -STOP 11993, kill -STOP 11998...
Turns out pkill takes regex, meaning you can pkill -STOP delayed which will pause everything with delayed in the process name.
Also pgrep -l delayed finds all PID's of processes with Delayed in it.
Really handy!
Reference: http://linux.about.com/library/cmd/blcmdl1_pkill.htm
Turns out pkill takes regex, meaning you can pkill -STOP delayed which will pause everything with delayed in the process name.
Also pgrep -l delayed finds all PID's of processes with Delayed in it.
Really handy!
Reference: http://linux.about.com/library/cmd/blcmdl1_pkill.htm
Sunday, 21 October 2012
VPS experiance & Domain Name Registrar
This has been a busy week. I acquired a Virtual Private Server, a Domain and I have been having my first experiences with Apache2 since I am normally an Nginx man.
Firstly my reasons for getting a VPS are a few:
I did not want to leave my computer on all the time.
If I did leave my computer on, my IP is not static.
The electric bill would be more expensive than a small VPS anyway.
My main computer is windows so I would not get any of the lovely *NIX goodies through SSH.
A VPS is on all the time if I choose.
It is more online storage.
I wanted a personal GIT server.
I wanted to learn more about servers and networking.
So with this in mind, I started my search. Because I wanted it for personal use and GIT, I was looking for hard drive space over bandwidth or transfer limits.
A friend of mine at the university owned fusevps but sadly when I looked it appeared to be nuked from orbit.
My next stop was a VPS offers list: Low End Box, I like helping the local economy so I restricted my search to UK hosting and found several great offers from EaseVPS, CastleGem, Edis and MiniVPS. MiniVPS's offering fitted my criteria best and so I chose them to be my first VPS.
They have a nice control panel to change Operating Systems, shutdown, restart, change hostname, see usage graphs and the like which is nice, and their installs come with Apache2 running automatically which I did not expect. Overall I like the machine, it feels speedy enough and I have Gitolite set up so it really has everything I was going for.
So onto the next thing.
I managed to get my hands on bookofgreg.co.uk for a reasonable price from 123-reg (I didn't search around too much except to avoid godaddy), someone already had bookofgreg.com which is a little strange to think someone might be using my alias but seems it is not actively used at the moment. Using 123-reg's control panel I managed to set the DNS to point everything but mail to my VPS so I can happily use git@bookofgreg... now. It was at this point when checking the DNS was pointing to the right place I discovered Apache was running until it gave it's "everything is Ok" message when going to the browser.
Overall it is surprisingly quick, simple and easy to get a Domain name pointing to a VPS. Both VPS provider and Domain Name Registrar have everything automated to a T and it is such a quick and easy system. Very pleasantly surprised. Next Step, mail server setup (This is going to be HELL!)
Firstly my reasons for getting a VPS are a few:
I did not want to leave my computer on all the time.
If I did leave my computer on, my IP is not static.
The electric bill would be more expensive than a small VPS anyway.
My main computer is windows so I would not get any of the lovely *NIX goodies through SSH.
A VPS is on all the time if I choose.
It is more online storage.
I wanted a personal GIT server.
I wanted to learn more about servers and networking.
So with this in mind, I started my search. Because I wanted it for personal use and GIT, I was looking for hard drive space over bandwidth or transfer limits.
A friend of mine at the university owned fusevps but sadly when I looked it appeared to be nuked from orbit.
My next stop was a VPS offers list: Low End Box, I like helping the local economy so I restricted my search to UK hosting and found several great offers from EaseVPS, CastleGem, Edis and MiniVPS. MiniVPS's offering fitted my criteria best and so I chose them to be my first VPS.
They have a nice control panel to change Operating Systems, shutdown, restart, change hostname, see usage graphs and the like which is nice, and their installs come with Apache2 running automatically which I did not expect. Overall I like the machine, it feels speedy enough and I have Gitolite set up so it really has everything I was going for.
So onto the next thing.
I managed to get my hands on bookofgreg.co.uk for a reasonable price from 123-reg (I didn't search around too much except to avoid godaddy), someone already had bookofgreg.com which is a little strange to think someone might be using my alias but seems it is not actively used at the moment. Using 123-reg's control panel I managed to set the DNS to point everything but mail to my VPS so I can happily use git@bookofgreg... now. It was at this point when checking the DNS was pointing to the right place I discovered Apache was running until it gave it's "everything is Ok" message when going to the browser.
Overall it is surprisingly quick, simple and easy to get a Domain name pointing to a VPS. Both VPS provider and Domain Name Registrar have everything automated to a T and it is such a quick and easy system. Very pleasantly surprised. Next Step, mail server setup (This is going to be HELL!)
Friday, 21 September 2012
Process Watchman
Update: If the code it is running completes before the maximum time, it will block the program from completing until the time runs out. Still needs work! (Read: It doesn't work XD)
I was looking for a process monitor, sort of like monit, or god but these two solutions are for keeping a process alive or restarting a process if it is misbehaving. What I wanted was a simple process watcher that would simply kill a process if it misbehaves or runs out of time, while returning the completed or partial output of the process it was watching.
For this I quickly wrote a small tool to do this on *nix machines.
It is quite hacked together so I must apologise for the mess, it is in its first version here and the program was written as fast as I could imagine without prior design. There will be numerable ways of improving this tool.
Currently it accepts 3 args, a command, an integer value for seconds and a float value for ram megabytes. There is probably a problem with commands that are more than one word long which is most of them, experimentation and improvements are needed.
I was looking for a process monitor, sort of like monit, or god but these two solutions are for keeping a process alive or restarting a process if it is misbehaving. What I wanted was a simple process watcher that would simply kill a process if it misbehaves or runs out of time, while returning the completed or partial output of the process it was watching.
For this I quickly wrote a small tool to do this on *nix machines.
It is quite hacked together so I must apologise for the mess, it is in its first version here and the program was written as fast as I could imagine without prior design. There will be numerable ways of improving this tool.
Currently it accepts 3 args, a command, an integer value for seconds and a float value for ram megabytes. There is probably a problem with commands that are more than one word long which is most of them, experimentation and improvements are needed.
#Author: Greg Myers #Date: 30/08/12 @command = ARGV[0] @time_limit = ARGV[1].to_i #seconds @ram_limit = ARGV[2].to_f #megabytes def watch_time(sec) exception = Class.new(Interrupt) begin x = Thread.current #x will contain whatever runs in yield y = Thread.start { #y watches and causes x to throw when timeout. begin sleep(sec) rescue => e #This raises any error x naturally hits x.raise e else #This executes if no exceptions happened until now x.raise "Process ran out of time to execute." end } return yield(sec) ensure if y y.kill y.join end end end def watch_ram(megabytes, pid) exception = Class.new(Interrupt) begin x = Thread.current #x will contain whatever runs in yield y = Thread.start { #y watches and causes x to throw when timeout. begin loop { rss_use = `ps -o rss= -p #{pid}`.to_i #use ps to get rss of pid raise "Hit Ram Limit #{megabytes*1024}kb, with #{rss_use}kb" if megabytes*1024 < rss_use sleep(0.5) } rescue => e #This raises any error x naturally hits x.raise e end } return yield ensure if y y.kill y.join end end end def get_payload_child_pid(pid) pipe = IO.popen("ps -ef | grep #{pid}") pipe.readlines[2] =~ /\w+\s+(?I will be putting this on github publicly shortly.\d+)\s+(? \d+)/ #Always line 3, Line 1 = spawn cmd, Line 2 = IO.popen, Line 3 = ruby, Line 4 = grep pipe.close return $~[:child_pid] end if @command if @time_limit > 1 if @ram_limit > 0 watch_time(@time_limit){ require 'pty' PTY.spawn("#{@command} 2>&1") do |r,w,p| child_pid = get_payload_child_pid(p) watch_ram(@ram_limit, child_pid){ loop { puts r.gets } } end } else puts "Invalid memory limit #{ARGV[2]}" end else puts "Invalid time limit #{ARGV[1]}" end else puts "Invalid command #{ARGV[0]}" end
Multicore programming with ruby
As I understand it multicore programming is about the ability to utilize multiple CPUs.
Also I hear a lot of people saying ruby cannot use multiple CPUs.
This is a load of rubbish if you are on a *nix system.
All you have to do if you want to use multiple CPUs is fork{} and it is running in its own process.
You can leave whatever is in the block to run and reap the process later, or detatch it if you do not need to see the results.
Fork returns the PID of the process it spawns, and used with Process.waitpid(), you can make the main process wait for the results to come back from the child. If you want the results yourself, you can just open a pipe between the child and the main process with IO.pipe
Simple.
read, write = IO.pipe
pid = fork do
write.puts "test"
end
Process.waitpid(pid)
write.close
puts read.read
read.close
Resources:
http://www.ruby-doc.org/core-1.9.3/IO.html
http://www.ruby-doc.org/core-1.9.3/Process.html
Example based on:
http://stackoverflow.com/questions/1076257/returning-data-from-forked-processes
Also I hear a lot of people saying ruby cannot use multiple CPUs.
This is a load of rubbish if you are on a *nix system.
All you have to do if you want to use multiple CPUs is fork{} and it is running in its own process.
You can leave whatever is in the block to run and reap the process later, or detatch it if you do not need to see the results.
Fork returns the PID of the process it spawns, and used with Process.waitpid(), you can make the main process wait for the results to come back from the child. If you want the results yourself, you can just open a pipe between the child and the main process with IO.pipe
Simple.
read, write = IO.pipe
pid = fork do
write.puts "test"
end
Process.waitpid(pid)
write.close
puts read.read
read.close
Resources:
http://www.ruby-doc.org/core-1.9.3/IO.html
http://www.ruby-doc.org/core-1.9.3/Process.html
Example based on:
http://stackoverflow.com/questions/1076257/returning-data-from-forked-processes
Subscribe to:
Posts (Atom)