Showing posts with label learning. Show all posts
Showing posts with label learning. Show all posts

Tuesday, February 24, 2009

Using sort and uniq to find duplicate lines in a file

I love my Mac, mostly because I can run all those great Unix utilities that make tasks easy. Today I want to point out a pair of tools that come in handy often. They are sort and uniq.

sort can be used to sort the lines in a given file. A simple use would be

echo -e "c\na\nb" | sort

which produces:

a
b
c

Sweet! Now lets look at the sort program:

echo -e "a\na\nb\nc\nb" | uniq

gives:

a
b
c
b

Removing the duplicate a, but not the b! The tool only removes any consecutive duplicate lines. So what if you want to remove all duplicates? Easy:

echo -e "a\na\nb\nc\nb" | sort | uniq

gives:

a
b
c

Taa Dah! Thats easy!

Now this duo can be used in very many useful ways. Just the other day I needed to find two XML elements that had the same value. I used 'sed' to pull out all the values in the given element, sort to put these values in lexographically sorted order, and uniq to tell me the duplicates found:

sed -n 's/.*<tag>\(.*\)<\/tag>/\1/p' | sort | uniq -d

For the 'sed' part you could put any tag name in there you need to find duplicate value for. should be the name of the file you are using. Notice that each XML element was on it's own line, so this wouldn't work for documents that are not formatted nicely (for that see xmllint :).

Tuesday, January 13, 2009

Potion: Installing and Running

New to the computer language scene is Potion. This language is the work of Why (or Why the luck stiff if you like typing) a guy who created a wonderful guide to learning the Ruby programming language.

Potion is a very 'small' language in both terms of lines of code to implement and the tools and API of the language itself. Batteries are not included as they are in well established languages such as Python. NO SWEAT, this of course is why you are interested.

First things first you'll need to have Git installed. Git is source control software. I am on Ubuntu and so I just typed in the command 'sudo apt-get install git-core'. That should do it... oh not on Ubuntu well then go here and figure it out.

Second you are going to need Ragel. Ragel is a state machine compiler. Potion uses it to create the source code tokenizer. Again I used apt-get: 'sudo apt-get install ragel' and voila I had myself some Ragel. You might need to go here and download it or something.

Lastly you need the Potion source. Using Git you will download this. You should be able to type
git clone git://github.com/why/potion.git potion
and you'll now have the folder 'potion' containing the sources.

Now you'll want to change to the 'potion' folder. Type 'make' to start the process of building Potion.

Once finished copy the potion executable create in the make step into your local bin folder. For me that is '/home/rob/bin'. Make sure this folder is on the path. Now you can run potion anywhere. test it out using the command 'potion examples/fib.pn'. This will run the simple fibonacci app in the example folder.

Now you are ready to build a great program!

Tuesday, October 28, 2008

Openssl, when checking a digest...

I've been working on my own implementation of SHA-1 hashing function. In the bigger picture I've put together a mostly working OAuth client using Groovy, and part of that is using HMAC-SHA1 to compute a secret value. Well I started by using the built in Java version but decided i would create my own... this is good exercise.

So first thing I did was write some unit tests. I needed SHA1 hashes to start with to make sure that my algorithm implementation works. The obvious choices are either sha1sum or openssl. I'm on a iMac running Leopard, and by default openssl is installed. I used that to compute some hash values. Unfortunately for me I spent a lot of time trying to get my algorithm working when it was already working. The problem was I computed the values wrong with openssl. I used the following:

echo "the quick fox jumps the brown dog" | openssl sha1

Which gave me a incorrect value. Why you ask? Well I didn't remember the -n flag and a newline character was being included in the string passed to openssl. Ooops! Well once I added the -n flag everything was fine and I now have a number of hash values to use in my tests.

Tuesday, September 2, 2008

Languages

This is a quick post about some great learning options for a pair of functional languages. You may be interested in seeing a functional approach to computer languages. I recommend Haskell or Erlang. For Haskell I recommend the book "Real World Haskell". You can read it online; it will also be published by O'Reilly. For Erlang check out "Getting Started with Erlang", also free. Once you've wet your feet with Erlang you can check read the manual whenever you need a refresher.