A Max/MSP newbie muddling through the of subjective testing for audio! This is the third of three posts which cover the test design for my research project. Click for one and two.
Dare I say it, everything looks to be very close to finished in terms of the testing interface and crucially, the method of collecting and sifting through the results! This post is going to take you through the testing procedure and processing of results. Hopefully everything will make sense, if not then thankfully I can catch it before the actual testing starts!
The first thing participants will see after the section to input details such as age, will be the test section. Here users can familiarise themselves with the interface. A totally separate surround sound recording will be playable from this section. Once they are comfortable, they can move on to the test proper.
The picture above is what participants will see at the end of the test. There are 12 sections so there is no need to post all that. The important thing is the test reference number, which is the main point behind this blog post. Previously, I talked about my “Randomisatron”. This messy, complex and head melting monster of a subpatch allows me to shuffle the standard order of the test. The randomisation allows my test to meet the ITU BS1116 standard for the playback of test material where test samples must be presented in random order for each test participant.
Since I know the default playback order for each section (A and B, C and D, E and F etc.) I can shuffle the playback order with a code using the Randomisatron (this name may stick, but maybe not for the write up) With the participants answers filled out, I should be able to (un)shuffle the answers back into the original order with reference to that randomising code. This calls for a trip to Excel!
Processing the Results
My friends have always slagged me off because of how much I like using Excel. That is not to say that I spend my evenings pondering some Excel magic. The reason I like Excel is because at the end of this test, I am going to have 5 answers for each of the 12 sections from at least 20 participants. This is not something I want to do on paper. Excel appeals to my lazy side and if I can get rid of a lot of leg work with one swift click of the mouse, I will gladly take it!
Bear in mind that the Excel file is made so I understand it where the test interface has to be much more sleek. Hopefully I can make this make sense for you! The two columns on the left are the default order which the whole test interface has been built around. This info also pops up on the far right as a reference to make things a bit easier. What I have highlighted in yellow is the randomising part of it.
When I get a set of results from a test participant, I will fill in all the answers for questions 1 to 5 all the way down the 12 sections. These answers can only be A or B/1 or 2, the numbers I have in at the moment were just for when I was testing the thing out. With the answers filled in, I will then take a look at the test reference number and type it into the left hand yellow column. The right hand yellow column is a copy of what the default is, 1 through to 12.
Here is that first Excel image again. Imagine there are 12 CDs with the recordings on them. Take a look at the first CD which is “SF1, T1”. The randomising code is telling the Max patch to play CD1 2nd in the queue. It is telling the second CD to play 12th in the queue and the third CD to play 1st.
The first step is to sort the New Question Order column by the Rand. column. This turns the Rand. into 1 to 12 while also moving the New Question Order numbers with it. What this does is match up the numbers in the New Question Order with the answers I filled in at the start. Remember earlier I said that the third CD was being told to play 1st? Take a look at the first number in the orange column. Remember that the first CD is to be played 2nd in the queue? You can see that from the second spot in the orange column too. Finally, you can also see that the 2nd CD was played in the 12th position. Notice how the CD1 and CD2 answers are A, B, C and D in the image. We know that these below to the first two sets, recordings 1 to 4, so they ideally should be at the very top and lined up with their “SF1, T1” counterparts at the far left of the spread sheet.
I do these steps for each participant and when I get all 20+ finished I can then sort all the Question/Answer columns by the Original Order column and now all the answers for each set of recordings are grouped together, ready for me to do some whole other amount of work so I can get statistics and results processed, something I best read up on soon.
Thanks again for reading. It looks like I have made a small trilogy of posts about how to administer a subjective test in terms of playing sounds to meet the ITU standard, collect the answers to the questions and then process them for ease of use later on. Take a look at the top of this post for links to the other posts.
Thanks very much for reading, I hope someone finds it helpful!
3 thoughts on “Surround Sound Subjective Test Design 3 – Collecting the Data”