Whether to store data in SQL as key value pairs
Posted: August 27, 2012 Filed under: Database, Ruby, Ruby on Rails | Tags: hstore, index, key, lucene, pairs, postgres, query, Rails, redis, schema, solr, sql, storage, sunspot, unstructured, value Leave a comment »It can be tempting to store unstructured data as key value pairs to avoid lots of schema changes and capture dynamic data. But it can lead to problems.
- Overly complex queries/reporting
- No type support
This article explains some of the issues.
My conclusion is that it is fine if you will only be accessing the data rather than querying it, or as a temporary capture means until you’ve identified where the data really belongs.
If you have another way to query, like a Lucene index it may also be perfectly fine.
Update:
A key value store NoSQL solution may be appropriate, but that brings with it other baggage. Depending on the NoSQL solution you choose, you may have to do some extra work to get the type of search you need or use a Lucene style indexing solution anyway.
If you are using PostgreSQL, you can get much of the best of both worlds using hstore.
And you can easily use it with rails!
This will have the drawback of not being portable to other SQL backends, but that may be a tradeoff worth making.
With the PostgreSQL solution, you can retain one datastore and still allow some ability to store unstructured key values. Then as needed, you can integrate these back into the schema to permanent normalized columns.
Ruby Time to Integer does give you seconds since Epoch
Posted: August 26, 2011 Filed under: Ruby, Scripting | Tags: 1970, epoch, int, Ruby, time Leave a comment »“Time is stored internally as the number of seconds with fraction since the Epoch, January 1, 1970 00:00 UTC.”
>>require 'time'
>>Time.parse("January 1, 1970 00:00 UTC").to_i
=> 0
As expected, but nice to verify.
For the reverse, be sure to use Time.at rather than Time.new
>> Time.at(0).to_i => 0 >> Time.new(0).to_i => -62167201200
Tips for writing command line tools in ruby
Posted: August 2, 2011 Filed under: Mac OS X, Ruby, Scripting, Uncategorized, Unix | Tags: command line, development, exit status, file extension, irb, load, option parsing, require, require ruby file without extension, Ruby, tool Leave a comment »- Option parsing: Read this article by Allen Wei on RubyLearning Blog for a great overview. I recommend sticking with the built-in OptionParser if you want to reduce dependencies.
- If you want your code to be loadable so you can access functions and classes in the irb console for testing, use the following pattern:
def main#option parsing and execution code hereendif __FILE__ == $0main()endThis way the main function will only be automatically called if the script is being executed on the command line.
And in irb, you can call your functions and classes as you see fit for testing without triggering your whole script to run.Note that this is similar to the python __main__ test if you are coming from a python background.
- Naming without ruby’s.rb extension: You can name your executable without the rb extension if you wish, just be sure to include your shebang (#!)
To use the user’s default ruby, use:
#!/usr/bin/env ruby
But in some cases you may want the specify the path to ruby so you can use macruby or rubycocoa if you are need those frameworks to be available
#!/usr/bin/ruby
When testing in irb,
require 'script-name'
won’t work without the rb extension, but
load 'script-name'
does work. - Use exit codes. When your script fails or needs to communicate status at exit, use standard exit status codes.
Exit zero for default success status
exit 0
Exit any other number for a failure or warning status. You choose the exit codes for your tool, but be sure to document them if they require more explanation than simple success or failure.
exit 27 - Output to stderr using:
$stderr.puts "error: problem ...."
Read and Write to a Process using Ruby IO popen without blocking
Posted: February 19, 2010 Filed under: Ruby, Unix | Tags: blocking, command, IO, popen, Ruby, stdin, stdout 2 Comments »Once in a while you need to control an interactive command line tool. I kept getting a block when trying to read from a ruby IO popen pipe that was waiting for input. Here is my simple solution/workaround:
sh_process = IO.popen('sh > out.log', 'w')
f = File.open("out.log", "r")
sh_process.puts("ls")
f.read
sh_process.puts("uptime")
f.read
...
f.close
Basically, just create a temp file and read from that. A little hackish, but it works.
Disable protect_from_forgery when load testing rails
Posted: June 3, 2009 Filed under: Deployment, Ruby, Ruby on Rails | Tags: CSRF, httperf, load testing, Rails Leave a comment »Rails turns on protection from CSRF Cross-Site Request Forgery by default. It can make load testing more challenging since you need to get an authenticity_token for posting form data.
More information here: http://api.rubyonrails.org/classes/ActionController/RequestForgeryProtection/ClassMethods.html
Ruby Telnet For Logging Into RAIDs and Devices
Posted: March 19, 2009 Filed under: Ruby, Scripting | Tags: device, lun mask, raid, router, Ruby, telnet Leave a comment »Ruby’s built-in telnet capability has been extremely useful for scripting automated RAID changes like LUN masking, etc.
It is super simple. Connect, login, issue commands, read and parse results.
The example code I used is here.
This is telnet, so don’t do this if eavesdropping is a concern.