Viking's Notes

Screen

To change the escape sequence from C-a to `, add this to your .screenrc:
escape ``

Kubuntu on MacBook

Helpful Tutorial

https://help.ubuntu.com/community/MacBook

Dual Display

To get dual display working, it took a lot of time. After much cursing, I finally ended up with this xorg.conf file, and it actually works! In KDE 4 no less! I didn't think it was possible. ALERT! Turn off your desktop effects or you'll get nothing but black/white screens on both monitors.

Git

Ignoring changes in tracked files

To ignore changes in already tracked files, run this:

git update-index --assume-unchanged

Submodules

After cloning a repository with submodules, you still need to clone the submodule respositories:
git submodule init
git submodule update

Importing from Subversion

Here's the command to import from SVN. The --no-metadata option makes this more viable for a one-shot import.

git-svn clone --stdlayout --no-metadata -Aauthors.txt svn+ssh://biostat.mc.vanderbilt.edu/usr/local/subversion/Ryaml

File Permissions

Git doesn't track file permissions for most files (except for the executable bit). The reason for that is permissions aren't content. If you want the files in your git repository to be world-unreadable, set the core.sharedRepository option like so: =git-config --add core.sharedRepository group=

Clipboard manipulation

To output to the X clipboard on the command-line, use xclip (apt-get install xclip). Sweet.

Awk output

To change how awk prints things (with print), change the OFS variable like so:

awk 'BEGIN { OFS="," } {print $1, $2}'

Subversion Diff Wrapper

Modified from the SVN book:

#!/bin/sh

# Configure your favorite diff program here.
DIFF="/usr/bin/vim-diff"

# Subversion provides the paths we need as the sixth and seventh 
# parameters.
LEFT=${6}
RIGHT=${7}

# Call the diff command (change the following line to make sense for
# your merge program).
$DIFF $LEFT $RIGHT

# Return an errorcode of 0 if no differences were detected, 1 if some were.
# Any other errorcode will be treated as fatal.

» Download

Password-less SSH

Shortest passwordless ssh tutorial, ever

Here's a script I wrote based on this tutorial to copy keys to remote hosts: add-ssh-key.sh

MySQL

Adding primary key field to existing table

ALTER TABLE foo ADD id INT NOT NULL AUTO_INCREMENT FIRST, ADD PRIMARY KEY(id);

Credits to Daniel Schneller.

Paging

To turn on paging, do: pager less -S
To turn it off, do: \n

Fixing other people's code

http://drnicwilliams.com/2007/06/01/8-steps-for-fixing-other-peoples-code/

Exim4

Exim4 configuration sucks. It took me 3 or 4 hours to figure this out.

Exim4 and TLS don't work together well, apparently. To make exim work correctly, edit /etc/exim4/exim4.conf.template and add this line just under the driver line in remote_smtp_smarthost section, so that it looks something like this:
remote_smtp_smarthost:
  debug_print = "T: remote_smtp_smarthost for $local_part@$domain"
  driver = smtp
  hosts_avoid_tls = *
  hosts_try_auth = ${if exists {CONFDIR/passwd.client}{DCsmarthost}{}}
  tls_tempfail_tryclear = false
  DEBCONFheaders_rewriteDEBCONF
  DEBCONFreturn_pathDEBCONF

Re-run update-exim4.conf and then restart exim4.

Internet Explorer Quirks

IE doesn't support transparent PNG. Stupid. Check out the following page for a helpful script to fix it:
http://homepage.ntlworld.com/bobosola/

ALERT! If you use this script with Rails, make sure you do something about the asset ID that Rails puts onto the end of image files. The script file expects the last 3 characters of a PNG filename to match /png/i, so either take off the asset ID or modify the script.

To use in Rails, drop the script into public/javascripts and put this into your application layout:
  <!--[if lt IE 7]>
  <script defer type="text/javascript" src="<%= javascript_path "pngfix" %>"></script>
  <![endif]-->

.NET

Links

Page Life Cycle Overview

Deleting Users

To delete users using stored procedures, run this:

DECLARE @RC int
DECLARE @ApplicationName nvarchar(256)
DECLARE @UserName nvarchar(256)
DECLARE @TablesToDeleteFrom int
DECLARE @NumTablesDeletedFrom int

SET @ApplicationName = 'foo';
SET @UserName = 'some guy';
SET @TablesToDeleteFrom = 15;

EXECUTE @RC = [foo].[dbo].[aspnet_Users_DeleteUser] 
   @ApplicationName
  ,@UserName
  ,@TablesToDeleteFrom
  ,@NumTablesDeletedFrom OUTPUT

SELECT @NumTablesDeletedFrom;

The 15 in TablesToDeleteFrom is a flag that means to delete from all user-related tables (4 total).

Web Application Design Principles

REST: Representational State Transfer
CRUD: Create/Retrieve/Update/Delete

Firefox and KDE

If Firefox is open when KDE shuts down, Firefox does not exit gracefully. Therefore, every single time you log back in, you'll get a session restore dialog box. AFAIK, there's no way to turn this off. KDE says it's a Mozilla bug (https://bugs.kde.org/show_bug.cgi?id=136350 link), and vice versa (https://bugzilla.mozilla.org/show_bug.cgi?id=354686 link). So, whatever.

HTML redirection

I always forget how to do this, so here it is:

<html>
  <head>
    <title>Redirecting...</title>
    <meta http-equiv="refresh" content="1;/rhubarb">
  </head>
  <body>
    Please click <a href="/rhubarb">here</a> if you are not redirected.
  </body>
</html>

PostgreSQL

Authentication

To set up postgresql so that non-ident users can connect, change the pg_hba.conf file:
# TYPE  DATABASE    USER        CIDR-ADDRESS          METHOD
local   all         all                               md5

Logging

To turn off annoying and hateful notices to stdout, play around with /etc/postgresql/8.1/main/postgresql.conf.

Passwordless Connection (psql)

Create the ~/.pgpass file, chmod it to 600, and add entries that follow this format:
hostname:port:database:username:password

ALERT! localhost doesn't seem to work for hostname, so use * unless you want to manually specify localhost on the psql command line via --host.

Debian

X Display Manager

To set the default X display manager, change /etc/X11/default-display-manager.

Building a Custom Kernel (brief)

Packages needed:
apt-get install gcc kernel-package libc6-dev libncurses5-dev modutils module-init-tools

Steps:
  1. Get the source
  2. Unpack it
  3. Make a symlink
  4. Configure
  5. Build it by running:
    make-kpkg --append-to-version=-2019-10-20 --revision=1 kernel_image
  6. Install the package
  7. Run update-grub
  8. Reboot and do a jig

This is handy for oldconfig's:
yes $'\n' | make oldconfig

ALERT! Double check the permissions on the source tree

Ruby

Snippets

See VikingRuby.

fastri

Here's an init.d script to get the fastri server running on boot. You need to create the fastri user, make the /etc/fastri directory, and change ownership of /etc/fastri to be owned by fastri. Whammo.

» Download

set_trace_func

Here's the set_trace_func example from Pickaxe: set_trace_func proc {|event, file, line, id, binding, classname| printf("%8s %s:%-2d %10s %8s\n", event, file, line, id, classname) }

Postgresql

To get the postgres gem to compile on debian, do this:
sudo gem install postgres -- --with-pgsql-include-dir=/usr/include/postgresql

Mysql

To compile the mysql gem on Mac (with the macports version of mysql installed), run:
sudo gem install mysql -- --with-mysql-include=/opt/local/include/mysql5 --with-mysql-lib=/opt/local/lib/mysql5 --with-mysql-config=/opt/local/lib/mysql5/bin/mysql_config

(credit)

1.9 on Mac

First, you need an updated readline (probably). Get it from macports. Then, download ruby 1.9 and run:
./configure --prefix=/usr/local --program-suffix=1.9 --with-readline-dir=/opt/local
make
sudo make install

Rails

Cloning Regexp's

Ruby doesn't allow the cloning of literal regular expressions.
$ irb
>> /blah/.clone
SecurityError: can't modify literal regexp
        from (irb):1:in `initialize_copy'
        from (irb):1:in `clone'
        from (irb):1
>>

This is a problem if you serialize a literal regular expression in a model. ActiveRecord clones its attributes before saving, and since you can't clone a literal regexp, you best be changin' that. Use Regexp.new("blah") or Regexp.new(/blah/) instead.

Selenium File Uploads

Firefox doesn't allow Javascript to type things into file upload fields. Here's a fix: http://cakebaker.wordpress.com/2006/03/29/file-upload-with-selenium/

Freezing Third Party Gems

I wanted to do this without having to generate a template, like gemsonrails wants, so I wrote a script based on the package's rake task. Credits to Dr Nic Williams for the gemsonrails package.

  1. mkdir vendor/gems
  2. Run this script. The syntax is:
    ruby freeze_gem.rb [ -o ENV ] [ -v VERSION ]
  3. Add this to your config/environment.rb:
# yanked from gemsonrails gems = Dir[File.join(RAILS_ROOT, "vendor/gems/*")] if gems.any? gems.sort.each do |dir| lib = File.join(dir, 'lib') $LOAD_PATH.unshift(lib) if File.directory?(lib) init_rb = File.join(dir, 'init.rb') require init_rb if File.file?(init_rb) end end

The -o (--only) option lets you specify which environments you want the gem to be loaded in. If you only want to load the gem in development and test environments, do -o development,test.

The -v (--version) options lets you specify which version of the gem you want to freeze. Hip, hip, hoorah.

Redefining Rake Tasks

This should probably be under the Ruby sub-heading, but I'm using this for Rails, so there. If you want to re-define an existing rake task, you have to first remove the old one, or both will run. Here's my Rakefile that adds this capability:

# Add your own tasks in files placed in lib/tasks ending in .rake, # for example lib/tasks/capistrano.rake, and they will automatically be available to Rake.

require(File.join(File.dirname(FILE), 'config', 'boot'))

require 'pp' require 'rake' require 'rake/testtask' require 'rake/rdoctask'

# credits to Blair Zajac (blair@orcaware.com) for this idea: module Rake class Application def remove_task(name) @tasks.delete(name) end end end

require 'tasks/rails'

Then in your task file, add Rake.application.remove_task 'test:functionals', or whatever the relevant rake task is.

Customizing RSpec Behavior Type

If you want to run <insert some other behavior type> tests in a spec file that isn't in the <that other behavior type you inserted before> spec folder, you can specify the behavior manually by using :behaviour_type => :<yes, that same behavior type again>.

For example, if I want to run a controller test in my spec/libs directory, I do: describe PockyTestController, :behaviour_type => :controller do # do stuff here end

Look at Spec::DSL::BehaviourFactory::BEHAVIOUR_CLASSES to see what the behavior types are. Actually, the BEHAVIOUR_CLASSES constant is in the metaclass of Spec::DSL::BehaviourFactory, which makes it a little tricky to get at if you're not in the right context. First install the metaid gem, then run irb (not script/console) from your Rails root directory, and paste this in:

$LOAD_PATH.unshift(File.expand_path(File.dirname(FILE) + "/vendor/plugins/rspec/lib")) require 'spec'

require 'rubygems' require 'metaid' require 'spec/spec_helper'

p Spec::DSL::BehaviourFactory.metaclass.const_get("BEHAVIOUR_CLASSES")

Or you could just take my word that BEHAVIOUR_CLASSES looks like this:

{:default=>Spec::DSL::Behaviour, :helper=>Spec::Rails::DSL::HelperBehaviour, :controller=>Spec::Rails::DSL::ControllerBehaviour, :view=>Spec::Rails::DSL::ViewBehaviour, :model=>Spec::Rails::DSL::ModelBehaviour}

smile

Asset ID

Rails likes to put a timestamp on the end of filenames (i.e. /images/edit.png?1159723626). To change this, set the RAILS_ASSET_ID environment variable. You can do this in config/environment.rb like so: ENV['RAILS_ASSET_ID'] = ""

Converting Specs to Test::Unit

Here's a script I wrote to convert RSpecs back to Test::Unit::TestCase.

Connecting to MS SQL

See instructions on the Rails Wiki. Ubuntu packages needed:
  • odbcinst1debian1
  • unixodbc
  • unixodbc-dev
  • tdsodbc
  • freetds-dev
  • libodbc-ruby1.8
  • libdbd-odbc-ruby
  • sqsh

apt-get install odbcinst1debian1 unixodbc unixodbc-dev tdsodbc freetds-dev libodbc-ruby1.8 libdbd-odbc-ruby sqsh

The instructions actually work!! Wow.

Database connections

The magic happens when something calls one of the <adapter>_connection methods from ActiveRecord::Base. Most (if not all) of the time these methods are called as a result of a call to the connection() method, which tries to retrieve a connection. The establish_connection() method doesn't actually connect to a database, it just records the information needed to connect to a database.

ActiveRecord::Associations magic

Let's say we have a model Foo, which has_many :bars. The has_many class method defines the instance method bars, which returns a HasManyAssociation instance object. So what the hell is going on with the code below? f = Foo.find(1) f.bars.class #=> Array; wth?

Why isn't f.bars.class HasManyAssociation?? The reason why this happens is because of some tomfoolery by AssociationProxy, the superclass of the association classes. AssociationProxy does this number: instance_methods.each { |m| undef_method m unless m =~ /(^__|^nil\?$|^send$|proxy_)/ }

This deletes most methods (including class, inspect, to_s, etc.). So when I do: f.bars.class, method_missing is called. Here's what method_missing essentially does: if load_target @target.send(method, *args, &block) end

It sends the method right along to @target, which is the Array. Jeez. So, the class of f.bars really is HasManyAssociation.

RSpec

How to make deprecations flunk a spec

Put this in spec/spec_helper.rb: ActiveSupport::Deprecation.behavior = Proc.new { |message, callstack| raise message }

script/spec customizations

Here's a custom script/spec that automatically uses the options in spec/spec.opts. Also, if called without arguments, it runs the spec file that was most recently changed.

#!/usr/bin/env ruby $LOAD_PATH.unshift(File.expand_path(File.dirname(FILE) + "/../vendor/plugins/rspec/lib")) require 'spec'

args = ["-O", File.expand_path(File.dirname(FILE) + "/../spec/spec.opts")] + ARGV

# run the last touched file if no arguments if ARGV.empty? spec_files = Dir[File.expand_path(File.dirname(FILE) + "/../spec") + "/**/*_spec.rb"] spec_files.sort! {|a, b| File.mtime(b) <=> File.mtime(a) } args << spec_files[0]

puts "====== running #{spec_files[0]} ======" end

#exit ::Spec::Runner::CommandLine.run(::Spec::Runner::OptionParser.parse(ARGV, STDERR, STDOUT)) exit ::Spec::Runner::CommandLine.run(::Spec::Runner::OptionParser.parse(args, STDERR, STDOUT))

Rails Logger Custom Formatter

Here's a custom rspec formatter that puts example names in the Rails logger: class CustomFormatter < Spec::Runner::Formatter::SpecdocFormatter def add_behaviour(name) super(name) @behaviour_name = name end

def example_started(name) RAILS_DEFAULT_LOGGER.info RAILS_DEFAULT_LOGGER.info "=============== #{@behaviour_name} #{name} ===============" RAILS_DEFAULT_LOGGER.info end end

Windows text file conversions

Sometimes Windows likes to save text files in the UCS-2 encoding. You can convert this back to latin1 in vim by setting fenc to be 'latin1' (instead of 'ucs-2le'). Or, do:
iconv -f UCS-2LE -t LATIN1 somefile.txt

File creation permissions (umask)

To get X and KDE (and everything else that uses PAM for that matter) to use a default umask, install the libpam-umask package (in Debian). Then put this in /etc/pam.d/common-session:
session optional    pam_umask.so    umask=027
This doesn't really seem to work.

EncFS

To encrypt folders, install the encfs and fuse-utils packages. Then, run encfs ~/.crypt ~/crypt or something similar to create a virtual filesystem that's encrypted!

Regular Expressions

RFC2822-compliant E-mail Validator

Here's a regular expression to validate an e-mail address, mostly compliant with RFC 2822 (see also Wikipedia:E-mail_address):
/^[\w!#\$%&'*+\-\/=?^`{|}~][\w!#\$%&'*+\-\/=?^`{|}~.]{0,62}[\w!#\$%&'*+\-\/=?^`{|}~]@([\w\d\-]+\.)+[\w\d\-]+$/

Bash Scripting

To change the internal field seperator ($IFS) in Bash to just a newline, do this:
export IFS=$'\n'

R

Last.fm analysis

Use this bit of code to download the last.fm weekly artist chart and convert it into a data frame. Note: you need the XML package.

library('XML')
user <- 'kindlyviking'
url <- paste("http://ws.audioscrobbler.com/1.0/user/", user, "/weeklyartistchart.xml", sep="")
xml <- xmlTreeParse(url)

num <- length(xml$doc$children$weeklyartistchart$children)
artists <- data.frame(name=I(character(num)), count=integer(num))

compileInfo <- function(i, parent) {
  artists[i,'name'] <<- xmlValue(parent$children[[i]]$children$name$children$text)
  artists[i,'count'] <<- as.integer(xmlValue(parent$children[[i]]$children$playcount$children$text))
}
lapply(1:num, compileInfo, parent = xml$doc$children$weeklyartistchart)

Local install

To install a local copy of R without optimization, lots of debug information, and as a shared library, run this:
    CFLAGS="-g3" FFLAGS="-g3" CXXFLAGS="-g3" FCFLAGS="-g3" ./configure --enable-R-shlib
    make

My script to run my local copy of R (~/bin/myR):

#!/bin/sh

R_VERSION=2.3.0
R_EXEC=~/src/build/R-${R_VERSION}/bin/R
R_ARGS=
if [ -z $1 ]; then
    R_ARGS="--no-save $*"
else 
    if [ $1 = "CMD" ]; then
        R_ARGS="$*"
    else
        R_ARGS="--no-save $*"
    fi
fi

R_LIBS=~/src/build/R-${R_VERSION}/library
LD_LIBRARY_PATH=~/src/build/R-${R_VERSION}/lib
R_HOME=~/src/build/R-${R_VERSION}

eval "$R_EXEC $R_ARGS"

VIM

Paste mode

To paste in VIM (console) without getting a mess, put VIM in paste mode like this:
:set paste

For a toggle, put this in your ~/.vimrc:
set pastetoggle=<F9>

I got this tip from here.

Bootstrapping

(information taken from Frank's book, chapter 5)

The idea behind bootstrapping is taking several samples (with replacement) of your original data set and performing calculations on those samples (such as mean, median, etc). You can then figure out the behavior of those calculations. Use this method when you don't want to make any assumptions about what distribution your data conforms to.

Validating the accuracy of models when predicting stuff about future data is possible with bootstrapping. Efron's "enhanced bootstrap" technique involves estimating the "bias due to overfitting". Overfitting occurs when a model takes too many variables into account. The bias estimated by bootstrapping validation can be subtracted from the original accuracy index (like R-squared) in order to produce the "expected bias-corrected index".

Receiver Operating Characteristic (ROC) Curves

An extremely helpful webpage about this can be found here.

ROC curves can be used to determine how good a scoring algorithm is. If you have some data you want to test, you first need some "gold standard" to separate your data into true positives and true negatives. First you run your test on the data and get scores. Then, you need to put each element in your data into a positive or negative subset depending on what the gold standard says. Next, you plot those two subsets on the same graph. The ROC curve is the plot of true positives against false positives based on the threshold you set for your test. If the area under the ROC curve is close to 1.0, your test is close to the "gold standard".

MAST's validation technique

(information taken from the MAST paper)

To test their algorithm, the MAST folks did the following (almost verbatim from page 6 of their paper):
  • select a large number of protein sequence families
  • construct a set of characteristic motifs for each family
  • create a database of pseudo-random sequences
  • calculate the combined p-value of each pseudo-random sequence and a family of motifs
  • measure the classification accuracy of combined p-values when searching a database of real proteins

They took 75 families of proteins from Prosite and scored each pseudo-random sequence (about 1000 total). They plotted the computed scores against the expected scores using negative logarithms. They also tested real sequences (from SWISS-PROT) and computed the ROC50 score based on the Prosite standard.

From the footnote about the ROC50 metric:
The ROC50 metric considers only the top of the sort down to the fiftieth non-family member. The metric has a value of one if all the true family members come before any non-family members in a sort of the sequences in the database. It has the value zero if fifty non-family members appear before the first family member.

Other

When installing MEME, make sure the MEME path line goes at the end, since it has it's own install program. They really should have called it meme.install instead, since if you run ./configure with the MEME path line before the path to the actual install program, ./configure will make its Makefiles to call that install instead of the real one. Bad idea.

If gdb stops working, make sure the SHELL environment variable is set! Yarrrr.

Data Capturing

  • Unique field names across tables (avoids problems when merging)
  • Camel case field names (avoids periods and underscores, which can conflict with some languages)
  • Include backup keys
  • Use strings for categorical values (easier to read, harder to change)

Running Latex through RApache

LaTeX might crash when running through RApache because it can't create the temporary files it needs. Check out a solution here: http://stackoverflow.com/questions/11240725/how-can-i-get-php-to-compile-a-latex-document-if-it-www-data-cant-get-access

Scrypt Encryption

This section describes how the scrypt utility encrypts files.

Setup Phase

  1. Calculate parameters from the following options: maxmem (maximum memory to use), maxmemfrac (maximum fraction of memory to use), maxtime (maximum time to spend)
  2. Get random 32-byte salt from /dev/urandom
  3. Generate a 64-byte derived key from the password, salt, and parameters
  4. Prepare a header to write to the front of the output file, which contains the scrypt magic word, values of parameters, and the salt
  5. Create a checksum for the header using SHA256
  6. Create a header signature using HMAC with the header data, the header checksum, and the last 32 bytes of the derived key (the MAC is used to verify the password)

After the setup phase, we have the derived key and a header that is 96 bytes long.

Encrypt Phase

  1. Start a new HMAC and add the header data to it
  2. Write the header to the output file
  3. Initialize AES encryption algorithm with the derived key
  4. Read in unencrypted data, encrypt, and write encrypted data a block at a time. The encrypted data is also added to the HMAC.
  5. Finalize the HMAC and write it to the output file
  6. Zero out the memory containing the derived key and AES key

Topic attachments
I Attachment Action Size Date Who Comment
add-ssh-key.shsh add-ssh-key.sh manage 0.2 K 23 Jan 2008 - 17:46 JeremyStephens  
diffwrap.shsh diffwrap.sh manage 0.4 K 01 Feb 2008 - 12:52 JeremyStephens  
fastriEXT fastri manage 1.6 K 19 Nov 2008 - 10:37 JeremyStephens fastri init.d script
freeze_gem.rbrb freeze_gem.rb manage 2.5 K 25 Jul 2007 - 17:46 JeremyStephens script to freeze gems into a rails directory
rails-dev.shsh rails-dev.sh manage 3.6 K 26 Feb 2007 - 11:01 JeremyStephens my rails development konsole script [OLD]
rails.pngpng rails.png manage 1.7 K 02 Apr 2007 - 11:22 JeremyStephens rails icon
rails_dev.rbrb rails_dev.rb manage 7.5 K 19 Dec 2007 - 10:23 JeremyStephens KDE-integrated rails development script
spec_conv.rbrb spec_conv.rb manage 2.9 K 08 Jun 2007 - 17:11 JeremyStephens script to convert a spec into a test::unit
xorg.confconf xorg.conf manage 2.0 K 03 Jul 2008 - 01:20 JeremyStephens dual-display configuration for a macbook running kubuntu
Topic revision: r76 - 30 Oct 2013, JeremyStephens
 

This site is powered by FoswikiCopyright © 2013-2017 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Vanderbilt Biostatistics Wiki? Send feedback