July 2013 Archives

I am working on incorporating the MPTK library (0.7.0) into a JUCE application. Everything is fine, until I put only the line "#include " in one of my files. (I am including "-l mptk", and adding to the search paths "/usr/local/includes".) Then, in XCode (4.6.3) I get these errors:

In file included from /Users/bobs/Aalborg/research/201307/mpdgui/mpdgui/Builds/MacOSX/../../Source/MainComponent.cpp:9:
/usr/local/include/mptk.h:1332:4: error: "No FFT implementation was found !"
# error "No FFT implementation was found !"
^
/usr/local/include/mptk.h:3217:78: error: unknown type name 'GP_Pos_Book_c'
MPTK_LIB_EXPORT virtual MP_Support_t update_ip( const MP_Support_t *touch, GP_Pos_Book_c* );
^
/usr/local/include/mptk.h:3225:128: error: unknown type name 'GP_Param_Book_c'
MPTK_LIB_EXPORT virtual void update_frame( unsigned long int frameIdx, MP_Real_t *maxCorr, unsigned long int *maxFilterIdx, GP_Param_Book_c* );

and so on. The first error I can get rid of by "#define HAVE_FFTW3 1" before the include of mptk.h. The second error and others I can get rid of by including in the header search paths, "/Users/bobs/Aalborg/research/201307/MPTK-Source-0.7.0/src/libmptk"

This leads me to my first question: why during the installation of mptk, it creates the libmptk.dylib, and creates and copies the mptk.h, but it does not copy all the other headers (which should already be in mptk.h) into the includes path? Is it expected to have to include the above source?

This doesn't solve all errors. Now, I get the following:

/Users/bobs/Aalborg/research/201307/MPTK-Source-0.7.0/src/libmptk/block.h:67:3: error: unknown type name 'MP_Signal_c'
MP_Signal_c *s;
^
/Users/bobs/Aalborg/research/201307/MPTK-Source-0.7.0/src/libmptk/block.h:82:3: error: unknown type name 'MP_Real_t'
MP_Real_t* atomBufferTemp;
^
/Users/bobs/Aalborg/research/201307/MPTK-Source-0.7.0/src/libmptk/block.h:85:3: error: unknown type name 'MP_Real_t'
MP_Real_t* frameBufferTemp;

and so on. This is strange because "MP_Signal_c" is defined in "mp_signal.h", which is in "/Users/bobs/Aalborg/research/201307/MPTK-Source-0.7.0/src/libmptk". Some of the errors go away if I put at the top of block.h the forward declaration, "class MP_Signal_c;" as well as "#include "mp_types.h". But, then I get new errors:

/Users/bobs/Aalborg/research/201307/MPTK-Source-0.7.0/src/libmptk/block.h:126:3: error: unknown type name 'MPTK_LIB_EXPORT'
MPTK_LIB_EXPORT virtual int plug_signal( MP_Signal_c *setSignal );
^
/Users/bobs/Aalborg/research/201307/MPTK-Source-0.7.0/src/libmptk/block.h:126:19: error: expected member name or ';' after declaration specifiers
MPTK_LIB_EXPORT virtual int plug_signal( MP_Signal_c *setSignal );
~~~~~~~~~~~~~~~ ^
/Users/bobs/Aalborg/research/201307/MPTK-Source-0.7.0/src/libmptk/block.h:128:3: error: unknown type name 'MPTK_LIB_EXPORT'
MPTK_LIB_EXPORT virtual ~MP_Block_c();

and so on. I can get rid of these errors by "#include "dsp_windows.h" in block.h. But now I get these errors:

/Users/bobs/Aalborg/research/201307/MPTK-Source-0.7.0/src/libmptk/block.h:163:45: error: unknown type name 'TiXmlElement'
MPTK_LIB_EXPORT bool write_to_xml_element(TiXmlElement * blockElement);
^
/Users/bobs/Aalborg/research/201307/MPTK-Source-0.7.0/src/libmptk/block.h:171:78: error: unknown type name 'GP_Pos_Book_c'
MPTK_LIB_EXPORT virtual MP_Support_t update_ip( const MP_Support_t *touch, GP_Pos_Book_c* );
^
/Users/bobs/Aalborg/research/201307/MPTK-Source-0.7.0/src/libmptk/block.h:179:128: error: unknown type name 'GP_Param_Book_c'
MPTK_LIB_EXPORT virtual void update_frame( unsigned long int frameIdx, MP_Real_t *maxCorr, unsigned long int *maxFilterIdx, GP_Param_Book_c* );

and so on. I get rid of these errors when I include in block.h "#include "tinyxml.h", the forward declarations "class GP_Pos_Book_c; class GP_Param_Book_c; class GP_Book_c; class MP_Atom_c; class MP_Dict_c;". Then, I get new errors:

/usr/local/include/mptk.h:4305:39: error: unknown type name 'GP_Block_Book_c'; did you mean 'GP_Pos_Book_c'?
MPTK_LIB_EXPORT MP_Real_t update (GP_Block_Book_c*);
^
/Users/bobs/Aalborg/research/201307/MPTK-Source-0.7.0/src/libmptk/block.h:57:7: note: 'GP_Pos_Book_c' declared here
class GP_Pos_Book_c;
^
In file included from /Users/bobs/Aalborg/research/201307/mpdgui/mpdgui/Builds/MacOSX/../../Source/MainComponent.cpp:12:
/usr/local/include/mptk.h:7790:26: error: unknown type name 'GP_Pos_Range_Sub_Book_c'
MPTK_LIB_EXPORT virtual GP_Pos_Range_Sub_Book_c* get_range_book(unsigned long int minPos,
^
where the problem is now in the installed mptk.h file. I delete the mptk.h in /usr/local/includes, and reinstall MPTK with the changes to the source above. This does not help. So, I see in mptk.h that the header having problems is dict.h. So, to dict.h I add the forward declarations "class GP_Block_Book_c; class GP_Pos_Range_Sub_Book_c;". Then I remove mptk.h from /usr/local/includes, and reinstall it. This takes care of those errors, but then I get four more errors:

/usr/local/include/mptk.h:7803:51: error: return type of virtual function 'get_range_book' is not covariant with the return type of the function it overrides ('GP_Pos_Range_Sub_Book_c' is incomplete)
MPTK_LIB_EXPORT virtual GP_Pos_Range_Sub_Book_c* get_range_book(unsigned long int minPos,
^
/usr/local/include/mptk.h:8062:52: error: return type of virtual function 'get_pos_book' is not covariant with the return type of the function it overrides ('GP_Pos_Range_Sub_Book_c' is incomplete)
MPTK_LIB_EXPORT virtual GP_Pos_Range_Sub_Book_c* get_pos_book(unsigned long int pos);
^
/usr/local/include/mptk.h:8069:52: error: return type of virtual function 'insert_pos_book' is not covariant with the return type of the function it overrides ('GP_Pos_Range_Sub_Book_c' is incomplete)
MPTK_LIB_EXPORT virtual GP_Pos_Range_Sub_Book_c* insert_pos_book(unsigned long int pos);
^
/usr/local/include/mptk.h:8077:52: error: return type of virtual function 'get_range_book' is not covariant with the return type of the function it overrides ('GP_Pos_Range_Sub_Book_c' is incomplete)
MPTK_LIB_EXPORT virtual GP_Pos_Range_Sub_Book_c* get_range_book(unsigned long int minPos,unsigned long int maxPos);

Now, how do I resolve these errors?

I am hoping that these changes will make it more clear how to use the MPTK library in building a new application.

UPDATE: After ripping out all of gp (gradient pursuit), and modifying the relevant CMakeLists.txt, and reinstalling MPTK, I can now "#include " without errors. MPTK-Source-0.7.0.zip

I have, for the past 6 hours, been trying to figure out why I could no longer build an XCode project of MPTK, like I could in March. Now I know why. Apparently, XCode and cmake begin playing together differently after XCode 4.3 or so. I upgraded my XCode installation after my visit to Rennes in March, and this made some non-trivial changes.

Now, before running "cmake -G Xcode" in a terminal, one must run, "sudo xcode-select -switch /Applications/Xcode.app/Contents/Developer". At least, on my computer. (Thanks to this post.) Now, before running the Xcode project, we must turn off the matlab support. So, run "ccmake .", turn off the MATLAB flag, and generate the output. Now, MPTK and its utils can be compiled in Xcode.

Evaluation slides

| No Comments
It is now five times that I have delivered my seminar (sermon?) on evaluation. My slides are available here (but without the sound files). There are several important take-home messages applying far beyond the problem of music genre recognition:

  • REQUIRED READING: Clever Hans (the horse of Mr. von Osten) a contribution to experimental animal and human psychology (1911)
  • Know your data. Know real data has faults. Know faults have real impacts.
  • The principal goal of a system means, at the very least, decisions by that system should be based upon criteria relevant to the object of the goal (e.g., genre), not irrelevant aspects confounded with labels in a dataset.
  • If there are uncontrolled variables in a classification experiment, that "a system reproduces the labels of a dataset" says nothing about whether the system is considering characteristics relevant to the meaning of those labels. Classification accuracy is not enough.
  • "My artificial system can recognize music genre/emotion" is as remarkable a claim as "my horse Hans can do arithmetic." An experiment that validly addresses such a claim is necessary.
  • The design, implementation, and analysis of a valid evaluation requires creativity and effort. There are no shortcuts!
  • A poor evaluation cannot be rescued by statistics. (Garbage in, garbage out.)
  • Just because something can be precisely measured does not mean it is relevant. (See, "You Get What You Measure")
  • If you can elicit any response from a system by changing irrelevant factors, then you have a horse.
  • However, horses are not necessarily bad: depending on the use case and its success criteria, a horse may be entirely sufficient!
Finally published is my article, Classification accuracy is not enough: On the evaluation of music genre recognition systems. I made it completely open access and free for anyone.

Some background: In my paper Two Systems for Automatic Music Genre Recognition: What Are They Really Recognizing?, I perform three different experiments to determine how well two state-of-the-art systems for music genre recognition are recognizing genre. In the first experiment, I find the two systems are consistently making extremely bad misclassifications. In the second experiment, I find the two systems can be fooled by such simple transformations that they cannot possibly be listening to the music. In the third experiment, I find their internal models of the genres do not match how humans think the genres sound. Hence, it appears that the systems are not recognizing genre in the least. However, this seems to contradict the fact that they achieve extremely good classification accuracies, and have been touted as superior solutions in the literature. Turns out, Classification accuracy is not enough!

In this article, I show why this is the case, and attempt to torpedo "business as usual", i.e., using evaluation approaches that are standard in machine leaning. I step through the evaluation of three different music genre recognition systems, pointing out published works making arguments at each level of evaluation. I start with classification accuracy, and move deeper into precision, recall, F-scores, and confusion tables. Most published work stops with those, invalidly concluding that some amount of genre recognition is evident, or that an improvement is clear, and so on. However, "business as usual" provides no evidence for these claims. So, I move deeper to evaluate the behaviors of the three systems, and to actually consider the content of the excerpts, i.e., the music. I look closely at what kinds of mistakes the systems make, and find they all make very poor yet "confident" mistakes. I demonstrate the latter by looking at the decision statistics of the systems. There is little difference for a system between making a correct classification, and an incorrect one. To judge how poor the mistakes are, I test with humans whether the labels selected by the classifiers describe the music. Test subjects listen to a music excerpt and select between two labels which they think was given by a human. Not one of the systems fooled anyone. Hence, while all the systems had good classification accuracies, good precisions, recalls, and F-scores, and confusion matrices that appeared to make sense, a deeper evaluation shows that none of them are recognizing genre, and thus that none of them are even addressing the problem. (They are all horses, making decisions based on irrelevant but confounded factors.)

The evaluation approaches of "business as usual" are entirely inadequate to address not only music genre recognition, but also music emotion recognition (see my ICME paper), and music autotagging (forthcoming submission). With all the work I have done in the time since submitting this work, I now immediately reject any submitted work proposing a system in these application areas that has an evaluation going no deeper than classification accuracies and confusion matrices. "Business as usual" is a waste of time, and a considerable amount of such business has been performed.

Will "business as usual" continue?
Hello, and welcome to the Paper of the Day (Po'D): SOM Based Artistic Styles Visualization edition. Today's paper comes straight from ICME2013: Y. Wang and M. Takatsuka, "SOM Based Artistic Styles Visualization", in Proc. ICME, July 2013. My one-line description of this work is:

Extract features from digital images of paintings, arrange them by self-organized map, show that paintings with similar artistic styles cluster together.
The authors take several images of paintings by six artists (Gris, Braque, Raphael, Titian, Van Gogh, Gauguin, all available here. Looking through the data, I find at least three replicas), extract from them 37 numerical features related to color (temperature, weight, expressiveness), composition (rule of thirds, golden section) and lines ("different types of straight lines"), and then cluster them using the self-organizing map. They analyze the results and make several conclusions. For the paintings of the six artists, some results are shown below.

SOM.jpg One conclusion is:

We can see that painting collections of the same art movements are much more similar to painting collections of different art movements.
I think this is supposed to be: "a painting in an art movement is more similar to other paintings in the same art movement, than paintings in other art movements." This is a good sanity check, but the observation does not prove anything. What is needed is to answer why the feature vectors of the paintings have arranged themselves so. Is it due to the artistic style of the paintings, or something irrelevant?

Among three art movements, postimpressionism is closer to renaissance than cubism in feature space. This could be explained by different time periods of three art movements (renaissance being the earliest, cubism being the latest). Also cubism, different from postimpressionism and renaissance, is a form of abstract art which does not directly depict reality.
None of these conclusions (and many others in the paper) follow from the evaluation because the experiment provides no proof that artistic style is the cause of the results. Particularly troubling to me are the following. First, a centered, cropped, and right-sided image of the entire painting is necessary to compute the compositional features used in the paper. Does this mean that a detail of a painting using the impressionist style is not impressionist? Does this mean that a sideways image of a cubist painting is not cubist? Shouldn't stylistic features be invariant to irrelevant transformations? Is style a property only of an entire painting? Second, since several features rely on color, the clustering results could be quite sensitive to changes in the illumination of the paintings, digital processing of the colors of the images, and so on. Is a black and white image of a cubist painting less identifiable as cubist? Is style something that ceases to exist then? Third, since all the images are downloaded from museum websites, it could be that all digital images of paintings by Van Gogh and Gauguin are taken by the same photographer, or processed by the same software, etc. Hence, the clustering results in the image above could be museum/software-related, and not style-related.

In summary, I am highly suspicious that the results in this paper are not at all related to style. A reproduction of the evaluation, and a deeper analysis of the results, are necessary. For instance, how do things change when images become gray-scale? Does an artificially-generated image smack in the middle of the cubist cluster look cubist? Where does an image like this lie, and why? If we blur the image of a cubist painting, does it move closer to impressionism?

If the problem is to be addressed, much better evaluation is needed.

Blog Roll

About this Archive

This page is an archive of entries from July 2013 listed from newest to oldest.

June 2013 is the previous archive.

August 2013 is the next archive.

Find recent content on the main index or look in the archives to find all content.