Music Coach

Requirements

Submit (via email) a tentative project proposal, including: team member names, brief description of project (a paragraph or so), the main technical challenges, milestones/tasks to accomplish this, the primary obstacles/concerns, how you propose to evaluate the outcome.

Idea description

Use the phone as a feedback device which will tell you if the notes you are playing on an instrument are on pitch. Visual or haptic feedback could get more or less intense depending how far you are off the correct note. Could also be used to check if you are playing in time with a metronome. Might be best to load music you will play into phone as midi files.

Extra Ideas

If we loaded a pre-defined music score using midi - and displayed this on the screen which scrolls as you play. You would need to tell it how fast you are going to play the music - usually expressed as beats per minute. It then would measure the pitch correctness as well as rhythmic accuracy using visual feedback - but often music needs to speed up and slow down in sections to make it more expressive. If we strapped the phone onto a piece of headgear which positions it about 10 inches in front of your eyes. you could use head movements to either speed up or slow down the music - which is quite natural - many musicians move their heads or their feet in time to music when performing anyway.

Main challenges

1. Real-time processing of the audio input to recognize the frequency of a note.
2. Different instruments have a range of timbre with transient harmonics which can dominate the fundamental frequency momentarily - the challenge will be to decide on an optimal time window to use for the pitch recognition.
3. Real-time feedback display
4. If time permits, inferring rhythm from a trace of accelerometer inputs so that the user can control performance speed.

Milestones

1. Test some FFT code on a PC which can take as input - sample rate, number of bins and display the frequency of a single note on the screen
2. Build some software which can read in a midi file and use this to compare notes performed in real time against the pitch of notes in the midi file.
3. Evaluate the accuracy of the attack and release times of the live performance with the start and stop times of the notes in the midi file
4. Write a graphical interface on the Maemo simulator which creates feedback on the pitch and timing accuracy of the performance. This could be done with vertical bars for pitch and horizontal bars for timing but a more creative graphical output will probably be developed. Pitch and timing accuracy can be displayed as a percentage in real time and the accuracy of the performance can finally be displayed at the end.
5. Test the FFT code and graphical display on the N900
6. Test the midi file input on the N900
7. Time permitting add the option to change the speed of the performance using the accelerometer

Primary obstacles

• Processing capability to carry out a real-time FFT, we might need to use the DSP which adds an extra level of complexity to the problem
• Allowing for a certain level of musical expressiveness in a performance and not seeing this as pitch or rhythm inaccuracy. For example vibrato playing oscillates above and below the note - so an average over a time window would be more appropriate.
• Inferring rhythm for small subtle changes of phone height when attached to the body - depends on accuracy of accelerometer

Evaluation

The real-time pitch recognition software can be compared against some more accurate non-real time pitch recognition on a standard PC. The non-real time recognition software will use a recording of the musical performance which was being used by the N900 to evaluate the pitch in real-time.

Useful links

Pitch recognition

• http://www.nicholson.com/rhn/dsp.html
• http://www.hotpaw.com/intuna/singintuna.readme.html