jarv.org http://jarv.org blog and projects Tue, 18 Dec 2012 03:26:42 +0000 en-US hourly 1 http://wordpress.org/?v=3.5.1 Simple highscore tracking with python / Flask http://jarv.org/2012/12/simple-highscore-tracking-with-python-flask/?utm_source=rss&utm_medium=rss&utm_campaign=simple-highscore-tracking-with-python-flask http://jarv.org/2012/12/simple-highscore-tracking-with-python-flask/#comments Tue, 18 Dec 2012 03:26:42 +0000 jarv http://jarv.org/?p=1254 This is a followup on the last post which was a fun little game to teach myself canvas and create a little physics simulation with javascript.

All of the source code is now on github.

Everything covered below plus much more is already covered in the official Flask tutorial.
This code will accept an AJAX GET or POST at the end of each game so that scores can be tracked and read.

Since the use-cases are simple it’s easy to write the tests first:

class StateTestCase(unittest.TestCase): def setUp(self): self.db_fd, state.app.config['DATABASE'] = tempfile.mkstemp() state.app.config['TESTING'] = True self.app = state.app.test_client() state.init_db() def tearDown(self): os.close(self.db_fd) os.unlink(state.app.config['DATABASE'])

This above code will create an sqlite db before each test-case and remove it when completed.

def test_empty_db(self): rv = self.app.get('/d') assert json.loads(rv.data) == [] def test_single_entry(self): test_data = { 'name': 'john doe', 'score': 1234, 'time': 1234 }rv = self.app.post('/d', data=test_data) assert 'ok' in json.loads(rv.data) rv = self.app.get('/d') assert json.loads(rv.data) == [test_data] def test_multiple_entries(self): for score in range(0, 25): data = {'name': 'john doe', 'score': score, 'time': 1234} rv = self.app.post('/d', data=data) assert 'ok' in json.loads(rv.data) rv = self.app.get('/d') assert len(json.loads(rv.data)) == 10

These three test-cases cover basic functionality of the high-score tracker:

  • Ensure that if a GET request is made before adding entries an empty JSON array is returned.
  • If a single entry is added with a POST ensure that the same results are returned with a GET.
  • If there are more than 10 entries added ensure that we only get 10 back.

For the actual app there isn’t much more code than what was written for the tests.

@app.route('/d', methods=['GET']) def get_scores(): results = g.db.execute( 'select name, score, time from hs order by score limit 10') q = [dict(zip(['name', 'score', 'time'], [item for item in row])) for row in results] return json.dumps(q)

Retrieve the top 10 scores, return the results as JSON.

@app.route('/d', methods=['POST']) def post_score(): data = request.form if (not re.match('[sw]{1,20}$', data['name'])): return json.dumps({'error': 'invalid name'}) if (not re.match('[0-9]+.?[0-9]*$', data['time'])): return json.dumps({'error': 'invalid time'}) if (not re.match('[0-9]{1,15}$', data['score'])): return json.dumps({'error': 'invalid score'}) g.db.execute('insert into hs (name, score, time) values (?, ?, ?)', [data['name'], data['score'], data['time']]) g.db.commit() return json.dumps({'ok': 'success'})

Receive a new score, do some basic input validation and insert the new high score entry.

Of course it’s easy to inject fake high scores with something like this, abuse it, cheat, etc. But in this case it’s just for fun.

]]>
http://jarv.org/2012/12/simple-highscore-tracking-with-python-flask/feed/ 0
The State Game http://jarv.org/2012/07/the-state-game/?utm_source=rss&utm_medium=rss&utm_campaign=the-state-game http://jarv.org/2012/07/the-state-game/#comments Sun, 22 Jul 2012 17:16:07 +0000 jarv http://jarv.org/?p=1239 Below is a recent weekend programming project that I did to play with canvas. Who doesn’t like a geography game?

It uses a simple svg image for the map with a canvas overlay to detect state clicks. On the back-end it uses a very short python script with web.py. Most people who tried it out found the rotation to be annoying and difficult; for me it makes it more interesting :)

]]>
http://jarv.org/2012/07/the-state-game/feed/ 0
Automating Drupal 7 Installs Using Drush and Install Profiles http://jarv.org/2012/02/automating-drupal-7-installs-using-drush-and-install-profiles/?utm_source=rss&utm_medium=rss&utm_campaign=automating-drupal-7-installs-using-drush-and-install-profiles http://jarv.org/2012/02/automating-drupal-7-installs-using-drush-and-install-profiles/#comments Sat, 18 Feb 2012 20:47:42 +0000 jarv http://jarv.org/?p=1187 Recently I needed to install Drupal 7 non-interactively on a Linux Server running a typical LAMP software stack. There is a lot of information out there already for this type of installation but nothing concise for what I needed to do.  Hopefully by adding this one to the pile it will help someone else who went through a similar exercise, if you found it useful please let me know!

What you get after installation with this custom install profile:

  • Drupal 7 core
  • Core + selected module(s) installed and enabled
  • A custom user role and user name passed in on the command-line

Server Setup:

For a testing sandbox it will be easiest to use a Linux box or VM that has the following installed:

  • Apache or lighttp w/ php and sqlite
  • php w/ sqlite
  • Drush v5.x

Use your OS’s packaging system to install the webserver with php and database support.

Make sure you drush is version 5.x, to install it using php pear:

pear channel-discover pear.drush.org pear install drush/drush

Or see http://drupal.org/project/drush

Drush Make

The command “drush make” will read a makefile that has some directives to tell drush where to fetch the core, modules and themes. Here is the drush makefile that we will use that installs the core plus the Role Delegation module.

; Core Drupal ; ------------- core = 7.x api = 2 projects[drupal][version] = 7 ; Modules ; ------------- projects[role_delegation][subdir] = contrib

You can tell drush where to fetch the drupal core and the module but in this case it will get it from drupal.org. There are more powerful things you can do with the makefile, see the docs for more info.

To use this makefile and Drush to install Drupal into a web directory:

drush make /path/to/makefile /path/to/install/dir

This will download and explode the Drupal core and the module into a directory which will then allow you to run the install wizard by pointing a browser to it. What you most likely want however is to automatically install Drupal and get everything set up in one shot. If all you want to do is install the core and an admin user then you can stop reading here; use the drupal site-install command which has command-line arguments for the basic options.

Read on if you want to learn how to make additional customizations.

Drupal Install Profiles

The standard install profile is the default one that is used for a vanilla Drupal core installation. Using that profile as a starting point you can start making customizations like adding modules to the default list and you can add your own install steps. This example will create an additional user and permission role called “Content Creator.” This user will have the abilities to create new article content and some admin abilities like changing themes.

Copy the standard profile into a new one called my_profile

cd /path/to/install/dir cp -r profiles/standard profiles/my_profile

In the “my_profile” directory, rename all references to the name “standard” to “my_profile.”

my_profile.info

The info file has key=value pairs that informs the profile which modules will be activated, add the “role_delegation” module to the list. This is necessary to activate the module on install.

... dependencies[] = role_delegation

my_profile.profile

This file can be customized to add additional install steps. We are going to add a step to ask for a username and password for an additional user who will be given the custom “content creater” user role. Add the following php code to the end of my_profile.profile.

... /** * Implements hook_install_tasks(). */ function my_profile_install_tasks() {$tasks = array(); // Add a page allowing the user to specify a "content creator" user $tasks['my_profile_cc_form'] = array( 'display_name' => st('Content creator username'), 'type' => 'form', ); return $tasks; }/** * Task callback: returns the form allowing the user to add * a "content creator" user */ function my_profile_cc_form() {drupal_set_title(st('Content Creator Username')); $form['cc_uid'] = array( '#type' => 'textfield', '#title' => st('Username for Content Creator:'), '#description' => st('Enter the content creator userid'), ); $form['cc_email'] = array( '#type' => 'textfield', '#title' => st('Email for Content Creator:'), '#description' => st('Enter the content creator email'), ); $form['cc_pass'] = array( '#type' => 'textfield', '#title' => st('Password for Content Creator:'), '#description' => st('Enter the content creator password in both fields'), ); $form['actions'] = array('#type' => 'actions'); $form['actions']['submit'] = array( '#type' => 'submit', '#value' => st('Create content creator role and user'), '#weight' => 15, ); return $form; }/** * Submit callback: creates the "content creator" role and user */ function my_profile_cc_form_submit(&$form, &$form_state) {$uid = $form_state['values']['cc_uid']; $email = $form_state['values']['cc_email']; $pass = $form_state['values']['cc_pass']; // Create a role for "content managers" $c_role = new stdClass(); $c_role->name = 'content manager'; user_role_save($c_role); // additional permissions beyond what the authenticated // user receives user_role_grant_permissions($c_role->rid, array( 'assign content manager role', 'create article content', 'edit own article content', 'delete own article content', 'create page content', 'edit own page content', 'delete own page content', 'administer themes', )); $cc_user = array ('name' => $pass, 'pass' => $pass, 'roles' => array($c_role->rid => $c_role->rid), 'mail' => $email, 'status' => 1, # status: active ); $user = user_save(NULL, $cc_user); }
  • my_profile_install_tasks() – adds the additional install task
  • my_profile_cc_form() – specifies the custom form to get user input
  • my_profile_cc_form_submit() – runs when the form is submitted, creates a custom role with limited permissions and creates the user with this role assigned.

my_profile.install

This file has php code that will run on installation, it sets up the basic views, sets the theme, etc. This can be changed or extended in whatever way you want but for this example we will stick with the standard setup.

Once this profile is created it will be available as a new option on the install wizard:

And the new custom screen to create the “content creator” user:

There are many different form elements and attributes that you can read about in the Drupal form api documentation. For this example it would probably be better to use a password_confirm text input for the password but since the goal is to automate it from the commandline it doesn’t really matter.

Automating everything from the command-line

Now that the profile is customized instead of running through the wizard it would be nice to input everything from the command-line. To do this the drush site-install gives you the option of passing form parameters into the command line.

Here is what you would do to automate everything from the command-line where shell variables correspond to the “CC” (content creator) user information and admin account info.

(create the makefile) drush make /path/to/makefile /path/to/install/dir (create the new profile) cd /path/to/install/dir drush -y site-install --clean-url=0 --db-url=sqlite:sites/default/files/db.sqlite --account-name=$ADMIN_USER --account-pass=$ADMIN_PASS --account-mail=$ADMIN_MAIL --site-mail=$SITE_MAIL my_profile my_profile_cc_form.cc_uid=$CC_USER my_profile_cc_form.cc_email=$CC_MAIL my_profile_cc_form.cc_pass=$CC_PASS

“my_profile_cc_form” is the name of the form for the custom “Content Creator” install step and cc_uid, cc_email, and cc_pass are the parameters that entered. When that command completes you will have a fully functional site with a custom user and role.

If you something isn’t working for you I have put all of the above into a standalone shell script, simply install drush and and change the variable assignments at the top to suite your needs.

]]>
http://jarv.org/2012/02/automating-drupal-7-installs-using-drush-and-install-profiles/feed/ 1
A Text Analysis of Supreme Court Oral Arguments http://jarv.org/2012/02/a-text-analysis-of-supreme-court-oral-arguments/?utm_source=rss&utm_medium=rss&utm_campaign=a-text-analysis-of-supreme-court-oral-arguments http://jarv.org/2012/02/a-text-analysis-of-supreme-court-oral-arguments/#comments Sun, 05 Feb 2012 21:56:28 +0000 jarv http://jarv.org/?p=1004 Transcripts of the Sitting Justices

This post takes a look at supreme court transcripts as corpus for natural language processing. Lately I’ve been playing around with the nltk python module and I thought this might be an interesting data set (given that I’m also an avid follower of SCOTUS).

Supreme Court transcripts are made available by supremecourt.gov and can be downloaded in PDF form. Extracting data from the PDFs is not an exact science since the format varies a bit. I can get a majority of the cases broken up by speaker for what is available for download. The script does its best to compensate for transcription errors, typos, etc. Nothing is perfect though so it’s only useful to look at this data in the aggregate.

Click here for transcripts by person organized by name, or here for organized by case.

Stats from all available transcript data

statementssentenceswordsstopwordsunique words
JUSTICE_ROBERTS10338219251275001671538683
JUSTICE_ALITO3042670651472704385897
JUSTICE_SCALIA11870273471491022207589239
JUSTICE_THOMAS81911416986
JUSTICE_KENNEDY608112686763221130827125
JUSTICE_BREYER9105317391812602767869202
JUSTICE_GINSBURG7336178981187251691268740
JUSTICE_KAGAN912222217048224743026
JUSTICE_SOTOMAYOR3263714543937632135277
other speakers595311858481552665195229621500

“Other speakers” are petitioners and respondents (not justices)
“stopwords” are high frequency words like the, to, also, etc.
“words”, “unique words” do not include “stopwords”

Stats from 500 randomly selected statements

sentenceswordsstopwordsunique words
JUSTICE_ROBERTS1061697179001952
JUSTICE_ALITO10869563116182443
JUSTICE_SCALIA1146743593541989
JUSTICE_KENNEDY1045750892392121
JUSTICE_BREYER172911315145822329
JUSTICE_GINSBURG12069298114932483
JUSTICE_KAGAN124510867127272250
JUSTICE_SOTOMAYOR11508189100512170
other speakers148413256150473178

Justice Thomas is not included in the above data set because there are only 8 statements from him in the generated corpus
“Other speakers” are petitioners and respondents (not justices)
“stopwords” are high frequency words like the, to, also, etc.
“words”, “unique words” do not include “stopwords”

dyerware.com


Linguistic diversity is a coarse measure of a varied vocabulary. The chart below displays the total number of unique words divided by the total number of words.

dyerware.com


Nothing too interesting or surprising here. Justices may use less words in their sentences for a variety of reasons. This data isn’t normalized to factor out introductions or interruptions. Nevertheless the trend appears to be that Justice Kagan and Alito’s sentence lengths are longer than the others and about equal to the petitioners and respondents.

Long words in the oral transcripts

18 or 19 letters is the typical length for the longest words (that are in the dictionary) used by the various speakers. This includes non-sitting justices where the data was available.

  • Justice Alito – (18) misrepresentation
  • Justice Thomas – (16) unconstitutional
  • Justice Kagan – (18) misrepresentations
  • Justice Rehnquist – (18) telecommunications
  • Justice Sotomayor – (18) unconstitutionally
  • Justice Ginsberg – (18) misrepresentations / telecommunications / disproportionately
  • Justice Scalia – (19) unconstitutionality
  • Justice Breyer – (18) representativeness / telecommunications / disproportionately / unconstitutionally
  • Other speakers – (19) counterintelligence / unconstitutionality / extraterritoriality

Sentiment Analysis

Sentiment analysis can yield interesting results for corpus data though in this case there is not very good training material. One of the standard data-sets used for this are movie reviews, widely available and with clear negative and positive denotations. For more information about sentiment analysis there is some good information here and in these two articles. Applying this to oral arguments? Well let’s leave it as just one way to look at this data..

dyerware.com



Justice Thomas is not included in the above data set because there are only 8 statements from him in the corpus
“Other speakers” are petitioners and respondents (not justices)

Once the sentiment engine is trained, each statement registers as either positive or negative based on the words that match closest to the language in positive and negative movie reviews.

I’m sure the error of margin is high and without training data for similar text I would hesitate to draw any conclusions. One that I might make from this is that petitioner and respondents tend to use more positive language than the judges on the bench. If that is a valid hypothesis this data certainly seems to validate it.

Laugh lines in the oral transcripts

Note: there are venturesome academic papers like this one from the Communication Law Review that address laughter in the SCOTUS courtroom. I make no attempt to go into that depth here though my results more or less agree with previous studies on this topic.

It’s not uncommon to get laughter after a statement from the bench, this is denoated in the transcripts as either [laughter] or (laughter). This chart displays the total number of laughter lines by sitting Judge

dyerware.com


As usual Thomas is excluded for generally not speaking when he sits on the bench.
Here is the same data but instead of the total number of [Laughter] lines it divides it by the number of statements for each justice.

dyerware.com


Justice Breyer appears to be funnier by this measure.

Additional Notes

Unfortunately it’s impossible to cleanly extract the argument data from PDFs. Older transcripts have the Justice’s remarks labeled as “QUESTION”; without specific name references the data had to be discarded. Some transcripts have the Justice’s name spelled incorrectly, for example: JUST SCALIA or JUDGE SCALIA

Here is an example of where there is wrong attribution:

MR. LANDAU: JUSTICE O'CONNOR: Your Honor, that is not -And I think it is conceivable that the Florida court was correct that you could draw the line some way and say contracts that are void should be handled differently.

For this reason don’t take these results too seriously though hopefully the errors are down in the noise (I have no desire to go through and correct them).

The pdfs were parsed and the data was generated with two crufty python script, they are on my github if you want to look at this data yourself. If you make any improvements please let me know!

]]>
http://jarv.org/2012/02/a-text-analysis-of-supreme-court-oral-arguments/feed/ 2
Popular names from ssh break in attempts http://jarv.org/2012/02/popular-names-from-ssh-break-in-attempts/?utm_source=rss&utm_medium=rss&utm_campaign=popular-names-from-ssh-break-in-attempts http://jarv.org/2012/02/popular-names-from-ssh-break-in-attempts/#comments Wed, 01 Feb 2012 22:40:15 +0000 jarv http://jarv.org/?p=1050 Lately I noticed that there were more than the usual amount of ssh invalid logins on a machine I manage. For kicks here is data from a script that extracts some name statistics from auth logs. These stats are compared to the male and female names contained in one of the nltk corpora.
Below is data collected over the month of January, 2012 using authlog as my input. Over the course of a month there were over 125,000 invalid ssh attempts.

  • Total number of male name login attempts: 17,804
  • Unique male names: 1,565
  • Logest male names: christopher, bartholomew, constantine
dyerware.com


  • Total number of female name login attempts: 22,483
  • Unique female names: 2,217
  • Logest female names: alexandrina, constantine
dyerware.com


“billy” wins the prize; the corpus for “names” has it as both male and female. Also the corpus has “temp” as a male name which appears in the top 1% for obvious reasons.

What about all logins? For these we will just look at the top 0.1%.

  • Total number of login attempts: 125,588
  • Unique logins: 35,627
  • Longest login name: fidelu142muiesteaua8642jet184
dyerware.com


 

Conclusion: if you name your kid billy or test make sure he uses certificate authentication.

]]>
http://jarv.org/2012/02/popular-names-from-ssh-break-in-attempts/feed/ 0
Amazing Musical Giftbox http://jarv.org/2012/01/amazing-musical-giftbox/?utm_source=rss&utm_medium=rss&utm_campaign=amazing-musical-giftbox http://jarv.org/2012/01/amazing-musical-giftbox/#comments Tue, 24 Jan 2012 18:25:20 +0000 jarv http://jarv.org/?p=1037 I received mail from Piotr Zimnowlodzki last week who showed me this cool hack using the PlayTune library and my midi converter script to play “Can you feel the love tonight” an attiny. Much better than the cheap greeting card hack, amazing craftmanship!

]]>
http://jarv.org/2012/01/amazing-musical-giftbox/feed/ 0
Musical Ms. Pacman Candy Tin Hack http://jarv.org/2011/12/musical-ms-pacman-candy-tin-hack/?utm_source=rss&utm_medium=rss&utm_campaign=musical-ms-pacman-candy-tin-hack http://jarv.org/2011/12/musical-ms-pacman-candy-tin-hack/#comments Sun, 04 Dec 2011 19:53:25 +0000 jarv http://jarv.org/?p=954 Here is another weekend hack that plays around with my midi to AVR conversion script and library. With xmas fast approaching I thought it would be fun to convert a pacman candy tin to an xmas ornament and have it play music. Below is the result, pressing a button on the tin will cycle through three Ms. Pacman songs converted from midi files found online.

Construction

Once I had the circuit working on a breadboard it was just a matter of finding a prototype board that fit and some ugly soldering to glue it all together. In my case I had these parts lying around (including the tin) but if you wanted to buy everything it would cost between $5-$10.

Other materials..

The circuit is only slightly more complicated than the musical greeting card. Two 1K potentiometers are used to mix the two square waves into one speaker. The push switch is connected to the external interrupt pin which is set low when pressed. On the prototype board the switch is wired on the opposite side of the circuit so that the speaker faces down when placed in the candy tin. This makes it louder by drilling holes in the back piece (see below). Standoffs are used to prop up the non-speaker side of the circuit.

Nothing is needed to hold the the circuit board in the tin since the button keeps it in place and there isn’t a lot of extra room when it is put together.

Software

(for more on the PlayTune library see my earlier post on using the PlayTune library with an Arduino)

Like the musical greeting card we will use the PlayTune library to play the melody and the xml2h.py to handle the musical conversion. The conversion takes a single track and converts it into two byte arrays of pitches and delays. The pitch values are a function of clock frequency and the prescaler.

The total size of the program ends up being around 2k so there is plenty of room to add more songs if you are inclined.

I used three midi files for the songs and loaded them into musescore. In the application it was necessary to transpose it an octave, other than that there wasn’t much else to do since these songs are already two-tracks which is exactly what we want for the attiny.

After saving the midi file as a MusicXML file a header file is created for each song. These header files are what the PlayTune library uses for tone and delay values.

The prescale values scale the frequency of the clock by a power of 2. Ideally you want the lowest value given in the list though TIMER0 only supports values of 1024, 256, 64 and 8 so for the first part “64″ is chosen.
For TIMER1 (part2) the lowest number can be selected to give the highest timer resolution. The reason this is important is because the frequency of the square waves generated on the two microcontroller pins are only an approximation of the note frequency. Higher frequency == greater timer resolution == better pitch accuracy.

The AVR is immediatly put into power-down sleep mode. When an external level change on the INT0 pin is detected (button press) the ISR routine will run which will play one of the three tunes.

#include <avr/io.h> #include <avr/pgmspace.h> #include <avr/interrupt.h> #include <avr/sleep.h> #include <util/delay.h> #include <avr/interrupt.h> #include "playtune.h" #include "songs/mspacman-acti-they-meet-attiny.h" #include "songs/mspacman-game-start-attiny.h" #include "songs/mspacman-actii-the-chase-attiny.h" int main(void) {// setup interrupt GIMSK |= (1<<INT0); // INT0 enabled for interrupts while(1) {set_sleep_mode(SLEEP_MODE_PWR_DOWN); sleep_mode(); }return(0); }volatile uint8_t tune = 0; ISR (INT0_vect) {PlayTune theymeet0(0,MSPACMAN_ACTI_THEY_MEET0); PlayTune theymeet1(1,MSPACMAN_ACTI_THEY_MEET1); PlayTune gamestart0(0,MSPACMAN_GAME_START0); PlayTune gamestart1(1,MSPACMAN_GAME_START1); PlayTune thechase0(0,MSPACMAN_ACTII_THE_CHASE0); PlayTune thechase1(1,MSPACMAN_ACTII_THE_CHASE1); switch(tune) {case 1: while ( theymeet0.isPlaying() || theymeet1.isPlaying() ) { theymeet0.playNote(); theymeet1.playNote(); _delay_ms(65); }break; case 2: while ( gamestart0.isPlaying() || gamestart1.isPlaying() ) { gamestart0.playNote(); gamestart1.playNote(); _delay_ms(15); }break; case 3: while ( thechase0.isPlaying() || thechase1.isPlaying() ) { thechase0.playNote(); thechase1.playNote(); _delay_ms(65); }break; }if (tune == 3) {tune = 1; } else {tune++; }}
]]>
http://jarv.org/2011/12/musical-ms-pacman-candy-tin-hack/feed/ 3
Picked up on lifehacker/hackaday http://jarv.org/2011/11/picked-up-on-lifehackerhackaday/?utm_source=rss&utm_medium=rss&utm_campaign=picked-up-on-lifehackerhackaday http://jarv.org/2011/11/picked-up-on-lifehackerhackaday/#comments Tue, 22 Nov 2011 22:13:27 +0000 jarv http://jarv.org/?p=944

Usually that number is closer to zero :)

]]>
http://jarv.org/2011/11/picked-up-on-lifehackerhackaday/feed/ 0
Custom musical greeting card for less than $5 http://jarv.org/2011/11/custom-musical-greeting-card-for-less-than-5/?utm_source=rss&utm_medium=rss&utm_campaign=custom-musical-greeting-card-for-less-than-5 http://jarv.org/2011/11/custom-musical-greeting-card-for-less-than-5/#comments Sun, 20 Nov 2011 23:03:26 +0000 jarv http://jarv.org/?p=676 After playing around this week translating midi tracks into songs that the Arduino can play I thought that would it be interesting to see how cheap I could make a musical greeting card.  These days you can pick up one at the store for about $5 that plays sampled music. If one or two part square wave tones are good enough it’s doable to make one for about the same price.

Other applications? Maybe a “musical throwy”, It’s almost cheap enough :)

Here is “You’ve got a friend in me” played on an attiny85 and two speakers converted from a midi version of the song:


Materials

Total: $4.80 (cheaper than Hallmark)

(I’m going to use two speakers for mine bringing the cost up to $6.42)

Other misc materials

Construction


There are only 6 connections to solder, 5 if you tie black speaker wires together.  I set all pins that are not being used to low to be used as ground connections.  I would avoid connecting anything to the pin right beneath your +3V connection to avoid shorts.

Using a poor-man’s paper switch to complete the connection for the battery the tune only plays once and stops (the AVR is put into power-down mode after the song finishes).  There is plenty more that could be done here including using more pins for LEDs, etc.

It would also be possible to mix the two outputs into one speaker with some resistors, this was easier though.

Creating the music

There are many places to find midi tracks, the challenge is finding one that will work for one or two parts.  With Musescore you can merge multiple parts together if they don’t overlap.  This example I’ll use “You’ve got a friend in me”, the Disney ToyStory theme-song.

First thing to do is to clean it up in musescore, copy the “zap chords” plugin into your Musescore plugin directory.

Loading the midi file in Musescore

After removing the chords, transposing it, increasing some note lengths in the second part and cleaning it up:

After running chord-zap and some transposing

Looks pretty, right? Now it needs to converted into something that the Attiny85 AVR chip can read.
To do that we will use the xml2h.py script to create a header file for the PlayTune library.

Unlike running this for the Arduino in the last post we select a clock frequency of 8MHz.

Selecting Prescaler Values

Prescaler values of 2,4,16,32,128,512 are ONLY valid for the attiny85 TIMER1

The attiny85 has two timers each with their own prescaler value.  The prescaler value is an integer clock modifier that will divide the clockspeed (in our case 8MHz) so the timer runs slower.  On one hand this allows us to play lower frequencies, on the other it does not give us as much resolution resulting in less precise frequencies for the tones.  In general you want to select the lowest pre-scaler value that is shown in the list that the timer supports.

For the attiny85 TIMER0 supports 1,8,64,256,1024.  TIMER1 supports 1,8,16,32,64,128,256,512,1024,2048,4096

For the arduino atmega chips you should only use prescale values of 1,8,64,256,1024.

Software

Here is the code that plays the song using the PlayTune library, you can see more details about it here.

#include <avr/io.h> #include <avr/pgmspace.h> #include <avr/interrupt.h> #include <avr/sleep.h> #include <util/delay.h> #include "playtune.h" #include "songs/gotafriend.h" int main(void) {// set everything low that we aren't using DDRB |= (1<<PB2) | (1<<PB4) | (1<<PB3) | (1<<PB5); PORTB &= ~(1<<PB2) | ~(1<<PB4) | ~(1<<PB3) | ~(1<<PB5); PlayTune p0(0,PART0); PlayTune p1(1,PART1); while ( p0.isPlaying() || p1.isPlaying() ) { p0.playNote(); p1.playNote(); _delay_ms(70); }set_sleep_mode(SLEEP_MODE_PWR_DOWN); sleep_mode(); return(0); }

After the song is finished the Attiny85 will be put to sleep in low-power mode.

]]>
http://jarv.org/2011/11/custom-musical-greeting-card-for-less-than-5/feed/ 26
Arduino music from a midi file http://jarv.org/2011/11/arduino-music-from-a-midi-file/?utm_source=rss&utm_medium=rss&utm_campaign=arduino-music-from-a-midi-file http://jarv.org/2011/11/arduino-music-from-a-midi-file/#comments Sat, 19 Nov 2011 16:32:13 +0000 jarv http://jarv.org/?p=674 Recently I wrote an Arduino / AVR lib and python script so that I could take three parts from a midi file and playing them simultaneously using the three Arduino timers.

There are already plenty of other ways to generating tones using an Arduino.   You can play music from a midi input there is a library for playing PCM audio and similar projects for playing music using all three Arduino timers simultaneously. I’ve also played around with AVR sound generation before in previous posts.

Note: if you just want to use the pre-generated header files in the songs/ directory you can download the lib and hook up your Arduino to a speaker to play tunes..

What you need to get started

  • Arduino (or an atmega328, atmega168)
  • A speaker and some potentiometers to mix the inputs.
  • A midi file you want to convert
  • Python and the beautifulsoup library for parsing xml
  • Sequencing software like the Musescore (free)  that will also convert the midi file to musicxml
  • “xml2h.py” which parses a musicxml file and generates a header file with notes and delays for the arduino
  • PlayTune library that will play the song on the atmega for any tune for up to three tracks simultaneously.

The result..

How music is generated using an Arduino

On the Arduino and in general on the atmega168 and 328 there are 3 timers that operate independently of each other in hardware.   The timers count up (starting at 0) to either 255 or 65535 and reset to zero.  When these timers are put into “CTC” (clear on timer compare) a pin is toggled every time the counter matches a specific value called the “match value”. The speed of the timer depends on two things, one is the speed of the clock ( for the Arduino this is 16MHz ) and a clock prescaler. The “prescaler” is an integer that can be set that will slow down the timer by a factor of 1024, 256, 64, or 8. Once the timer is setup in CTC mode and a match value is specified there will be square wave generated on a pin at a frequency proportional to how frequently the timer hits the match value.

Here are the match values that correspond to different pre-scalers and their corresponding piano notes at 16MHz. (This is from a spreadsheet I made which comes in handy especially when dealing with different clock frequencies)

Let’s start making music..


Using the PlayTune library

Copy playtune.cpp and playtune.h to a “PlayTune” directory in the libraries folder of your Arduino install.  Once it’s copied there you should see it in the “import library” list under “Sketch”.

To initialize PlayTune pass into the constructor what timer you want to use, what prescaler, a list of match values and a list of corresponding delays.

  • “isPlaying()” lets us know if there are still notes left to play for the part. When it’s finished it will reset back to the beginning.
  • “playNote()” will iterate over the note list, playing the tone that corresponds to the match value or a rest if zero.

By default the PlayTune lib will play a scale from middle C to tenor C if you just pass in the timer you want to use.

In this example we will play the scale on all three timers simultaneously.
(connect your speaker to pins 6,9 and 11)

#include #include void setup(void) {PlayTune p0(0); PlayTune p1(1); PlayTune p2(2); while (p0.isPlaying() || p1.isPlaying() || p2.isPlaying() ) { p0.playNote(); p1.playNote(); p2.playNote(); _delay_ms(500); }} void loop(void) {}

Note that the _delay_ms(500) function requires the additional include of util/delay.h.

Do not use the Arduino “delay()” function! It will cause problems because it uses TIMER0 which is being used by the lib for generating the tone.

So that was fun (but boring).. Now on to creating some more interesting music…

Midi, musescore and xml2h.py

Musescore is a wonderful free sequencing tool. Using that you can open midi files, adjust them, and convert them to musicxml.

“xml2h.py” is a script I wrote that takes a musicxml file and generates an header file that tracks two arrays for up to 3 parts, one for note values and another for delays (rests are note values equal to zero)

The conversion from midi to arduino tones is a bit of a hack :/
xml2h.py will do its best to:

  • pick the best prescaler
  • ignore chords
  • combine voices if it can
  • handle tied notes

I’ll walk through a video game theme as an example (it works well since the songs are simple). There are a number of sites out there with video game midi files most of them look like they were designed in 1997. Below I’ll use the Super Mario Brothers theme. In this score there are 4 parts, we will convert the first three to play on the Arduino.

SMB theme in Musescore

There are some limitations to keep in mind for the midi conversion:

  • If notes span multiple clefs then you there will probably be too large a range for the resolution of the timer, not many (simple) songs have this problem though.
  • When you convert you will need to select between 1-3 parts to be played simultaneously.
  • Chords will be ignored by the xml2h.py script.
  • If there are more than voice on a part the script will try to merge them together.

I wrote a plugin called “chord zapper” for musescore that will (attempt to) remove all the chords. Copy this to the musescore plugin folder, select all, run the plugin to get rid of all the chords if you want to get a better sense of how the song will sound when converted. Whether or not you remove chords the “xml2h.py” script will ignore them, taking the top note in the chord if there are any.

zapping all the chords (optional)


When you are satisfied save the midi file save it as a “MusicXML” file and run it through xml2h.py.
The script takes the xml plus the name of the header file you want to write out.

For example:

Pick the clock frequency. This defaults to 16MHz which is what you want for the Arduino
Pick the part you wish to process (part0 corresponds to the top clef in the midi file)
Pick the prescaler, the script will give you a subset of prescalers, you should select the lowest one that is supported. (see note on prescalers below)

If it finishes without error you will end up with a header file that contains some arrays and macros for passing them into the PlayTune constructor.

Here is what the resulting header looks like, the macros for PART1, PART2, PART3 are used in the sketch below:

#include #include #include void setup(void) {PlayTune p0(0,PART0); PlayTune p1(1,PART1); PlayTune p2(2,PART2); while (p0.isPlaying() || p1.isPlaying() || p2.isPlaying() ) { p0.playNote(); p1.playNote(); p2.playNote(); _delay_ms(50); }} void loop(void) {}

Assuming you are converting two parts and the first part will be using Timer0 and the second part Timer1 pick the lowest value from the two sets that show up in the prescaler selection.

The only difference between this sketch and the last one is the additional include for the “smb.h” header file, the addition of the PART# macros used in the PlayTune constructors and a smaller delay.

The value of the delay will determine the tempo of the song and corresponds to the delay between each tick in the MusicXML file or the shortest note.  Usually this is either 1/32 or 1/64.

If you have questions about the lib or if you create more songs let me knowand I will add them to collection.

Selecting Prescalers

In general you want to select a low value for a prescaler to give you the highest resolution (to match the pitch of the song as close as possible).
The two timers on the attiny support different prescalers though so you need to be aware of what timer you are using (0 or 1) and what prescalers it supports.

Timer0 supports:  1024, 256, 64, 8
Timer1 supports: 1024,512,256,128,64,32,16,8

]]>
http://jarv.org/2011/11/arduino-music-from-a-midi-file/feed/ 18