Playing Musical Notes by Image Processing

An integration activity that will use all the image processing techniques to be able to play musical notes from a scanned music sheet.

I selected a simple music sheet to show the proof of principle in doing this.

simple version of old mcdonald

This is the code I used to be able to play the note.

** Code**
stacksize(100000000);
Score = gray_imread('ScoreSheet2.png'); // must be columnar
Score1 = im2bw(Score, 0.1);
SE1 = gray_imread('SEnote12.png');
SE1 = im2bw(SE1, 0.1);

//correlate
Score = conj(fft2(Score1));
SE = fft2(SE1);
Corr = Score.*SE;
A = abs(fftshift(fft2(Corr)));
//imshow(Img, []);
imwrite((A-min(A))/(max(A)-min(A)), 'test1.png');

function n = note(f, t)
n = sin (2*%pi*f*t);
endfunction;
C = 261.63*2;
D = 293.66*2;
E = 329.63*2;

C5 = 523.25*2;
G4 = 392.00*2;
A4 = 440.00*2;
E5 = 659.26*2;
D5 = 587.33*2;

t=soundsec(0.25);
//s = [note(E,t), note(D,t),note(C,t), note(D,t), note(E,t) note(E,t) note(E,t)];
//sound(s);

//image thing
Img = gray_imread('test1.png');
Img2 = im2bw(Img, 0.9);
imwrite(Img2, 'test2.png');
[L,R] = bwlabel(Img2);
ycoor = [];
poresize = [];
s = [];
for j = 1:max(R),
[r,c] = find(L==j);
rc = [r' c'];
x = min(rc(:,2));
a = [size(rc(:,2))];
poresize = [poresize; a];
// xcoor = [xcoor; r'];
ycoor = [ycoor; x];
end

for k = 1:length(ycoor);
if ycoor(k) == 35,
s = [s, note(C5,t)];
elseif ycoor(k) == 20,
s = [s, note(G4,t)];
elseif ycoor(k) == 25,
s = [s, note(A4,t)];
elseif ycoor(k) == 46,
s = [s, note(E5,t)];
elseif ycoor(k) == 41,
s = [s, note(D5,t)];
end
end
sound(s);
wavwrite(s, 'mcdonald.wav');

**END of CODE**

Using this code, it yielded this:

This code uses correlation, Fourier transformation and bwlabel. There are several limitation to this code. First off, since there are only quarter notes, timing is constant. I have an idea on how to modify it to be able to vary the timing. It also involves correlation with other types of notes. :)
Another one is that notes are to be binarized, and rotated with the first note on top. because I used the ever powerful bwlabel. the blob indicators of the bwlabel is used for chronology purposes. Meaning, the first note is the top most with blob value 1 and so on.

I was also thinking of a possibility to use Fourier filtering to filter out the staff so that only notes will be left. Use bwlabel function to label the notes from 1 to last note. After that, use a for loop, then every loop, correlate every note. This correlation will identify the type of note and the timing. Store every information and then the information can be played. Voila! I haven't tried that method though.

I think I can have aroung10/10 for this activity. I haven't tested the prcedures on my mind regarding this activity but there are a lot of possibilities. It is exciting.

Thanks Maam Jing for the note function. And my roomates to fix my little coding mistakes.