Tag: testing

Surround Sound Subjective Test Design 3 – Collecting the Data

Posted on by robkellysound in Audio Production Projects, Surround Sound Recording Research

A Max/MSP newbie muddling through the of subjective testing for audio! This is the third of three posts which cover the test design for my research project. Click for one and two.

Hello all,

Dare I say it, everything looks to be very close to finished in terms of the testing interface and crucially, the method of collecting and sifting through the results! This post is going to take you through the testing procedure and processing of results. Hopefully everything will make sense, if not then thankfully I can catch it before the actual testing starts!

Collecting Results

This test section allows participants to play around and learn the interface they will be using for the test proper.
This test section allows participants to play around and learn the interface they will be using for the test proper.

The first thing participants will see after the section to input details such as age, will be the test section. Here users can familiarise themselves with the interface. A totally separate surround sound recording will be playable from this section. Once they are comfortable, they can move on to the test proper.

test 3The picture above is what participants will see at the end of the test. There are 12 sections so there is no need to post all that. The important thing is the test reference number, which is the main point behind this blog post. Previously, I talked about my “Randomisatron”. This messy, complex and head melting monster of a subpatch allows me to shuffle the standard order of the test. The randomisation allows my test to meet the ITU BS1116 standard  for the playback of test material where test samples must be presented in random order for each test participant.

Since I know the default playback order for each section (A and B, C and D, E and F etc.) I can shuffle the playback order with a code using the Randomisatron (this name may stick, but maybe not for the write up) With the participants answers filled out, I should be able to (un)shuffle the answers back into the original order with reference to that randomising code. This calls for a trip to Excel!

Processing the Results

My friends have always slagged me off because of how much I like using Excel. That is not to say that I spend my evenings pondering some Excel magic. The reason I like Excel is because at the end of this test, I am going to have 5 answers for each of the 12 sections from at least 20 participants. This is not something I want to do on paper. Excel appeals to my lazy side and if I can get rid of a lot of leg work with one swift click of the mouse, I will gladly take it!

Excel 1

Test Ref
The test reference code contains the randomising information.

Bear in mind that the Excel file is made so I understand it where the test interface has to be much more sleek. Hopefully I can make this make sense for you! The two columns on the left are the default order which the whole test interface has been built around. This info also pops up on the far right as a reference to make things a bit easier. What I have highlighted in yellow is the randomising part of it.

When I get a set of results from a test participant, I will fill in all the answers for questions 1 to 5 all the way down the 12 sections. These answers can only be A or B/1 or 2, the numbers I have in at the moment were just for when I was testing the thing out. With the answers filled in, I will then take a look at the test reference number and type it into the left hand yellow column. The right hand yellow column is a copy of what the default is, 1 through to 12.

Excel 1Here is that first Excel image again. Imagine there are 12 CDs with the recordings on them. Take a look at the first CD which is “SF1, T1”. The randomising code is telling the Max patch to play CD1 2nd in the queue. It is telling the second CD to play 12th in the queue and the third CD to play 1st.

Excel 2The first step is to sort the New Question Order column by the Rand. column. This turns the Rand. into 1 to 12 while also moving the New Question Order numbers with it. What this does is match up the numbers in the New Question Order with the answers I filled in at the start. Remember earlier I said that the third CD was being told to play 1st? Take a look at the first number in the orange column. Remember that the first CD is to be played 2nd in the queue? You can see that from the second spot in the orange column too. Finally, you can also see that the 2nd CD was played in the 12th position. Notice how the CD1 and CD2 answers are A, B, C and D in the image. We know that these below to the first two sets, recordings 1 to 4, so they ideally should be at the very top and lined up with their “SF1, T1” counterparts at the far left of the spread sheet.

Excel 3BOOM!

I do these steps for each participant and when I get all 20+ finished I can then sort all the Question/Answer columns by the Original Order column and now all the answers for each set of recordings are grouped together, ready for me to do some whole other amount of work so I can get statistics and results processed, something I best read up on soon.

Thanks!

Thanks again for reading. It looks like I have made a small trilogy of posts about how to administer a subjective test in terms of playing sounds to meet the ITU standard, collect the answers to the questions and then process them for ease of use later on. Take a look at the top of this post for links to the other posts.

Thanks very much for reading, I hope someone finds it helpful!

Surround Sound Subjective Test Design 1

Posted on by robkellysound in Audio Production Projects, Surround Sound Recording Research

A Max/MSP newbie muddling through the of subjective testing for audio! This is the first of three posts which cover the test design for my research project. Click for two and three.

Hello all,

With the project moving at a nice pace I wanted to share some information with you about the test design.

Playback Requirements

Iin the subjective testing stage of my project, I am asking listening test participants to compare two recording extracts and answer some questions on them. I am testing three arrays, a traditional and two variants of a Soundfield recording. All recordings were done simultaneously. The ITU BS1116 standard for “the subjective assessment of small impairments in audio systems including multichannel systems” applies to my project in the most suitable way. There are certain key requirements which are set out by the standard which include:

 – Test participants must be able to switch between recordings as they wish with no loss of place or jump in the sound (eg. Skipping a few sections)

 – To avoid bias, test material should be randomised (see post 2)

 – The transition between each sound extract must be 80ms in length.

 – other things (too much to list!)

My problem was, I did not know of any way to do this without me being in the test room with the participant and even then, human error would be inevitable with timing differences between each switch being different, that and a whole host of problems to say the least! The bottom line is that this could be off putting to participants. Added to that, I needed a way to collect information and answers from the participants. So having one thing for that and another to get them to play sound tracks and switch between them would probably be messy.  I needed something so instead of blindly researching a way to do it, I made my own!

Super Max! (Supermacs if you’re Irish)

Max/MSP  was the program what I used to solve my problem. (now known as Max 6, despite my constant use of Max/MSP in these blogs) It is a program I was introduced to back at Queen’s University. Coding and programming is not my thing but I always found this program fun and logical. When I encountered Reaktor at Salford Uni I was less than impressed after coming from Max/MSP so when I had the idea of using object based programming to sort my problem out, Max was my first port of call. I have a feeling Reaktor could have done it, but not as well as I knew Max could. I have used both programs to complete the same objective in the past so that is what I am basing those remarks on.

What I have designed here is probably very likely an inefficient way of doing things, so if any Max savvy people are reading then don’t say I didn’t warn you! While I get more research done into what is required in subjective testing for audio, the layout and things like will probably change but under the hood I have the exact program I need to meet the fundamental aims of BS1116.

I wanted to play two surround sound files and allow the user to switch between them instantly without losing place, like I mentioned earlier. I also wanted it to record the answers to the test questions. By combining all these things into one interface, it means that I can save the participants answers using a unique file name and load up a new blank patch for the next person.

1The GUI features of Max are fantastic. Maybe my colour scheme needs a look at (probably spelling too, but it’s a late night draft!). What the patch allows me to do is play two surround sound files at the same time, select between them in realtime and select my answer.

2Here I selected the recording I preferred, hit confirm and the patch them told me what to do. But what if one of my participants makes a mistake like selecting both the options?

3BOOM! The patch recognises that both preference buttons have been selected and gives a warning message. The user can then reset, reselect and confirm again.

4This is the magic behind it.There are three main sections.The upper right quarter of the image is the playback system which is routed into the audio outputs in the lower middle. The upper left quarter to middle is where users can select what recording they are listening to and also the display which tells each participant this for their own reference. The bottom right quarter is where the participants answers are filled in and tested to make sure that two options or no options were selected by accident.

What the participant will see is what was in the previous pictures. This under the hood view is how they are all tied together.  Max is an object based programming language. Each visual object you see has some traditional coding behind it, but the user does not need to worry about it. It allows the less code savvy like myself the ability to get some seriously powerful custom made projects done without worrying about syntax errors and languages. In fact, the only bit of traditional code that I have come across that you could need are if statements. That said, the help files are brilliantly done so help is always at hand.

5

Take the play and stop buttons for example. The button has some code under the hood that says when pressed, it sends a signal or a bang as it is called in Max. The output of that play button eventually goes into the object called “sfplay” by following the blue path. Sfplay plays soundfiles and there are two of them because I have two surround recordings that I need played at the exact same time. Note that the output (bottom of the boxes) have 6 green/grey lines. Each of those corresponds to a audio channel on my soundcard. Since 5.1 needs 6 channels I asked for an sfplay with 6 outputs.

Once sfplay gets the message “1” it will start playing, and when it gets a “0” it stops! You can see how the blue line from play goes through a grey box with “1” in it. So, what happens here is that the bang comes out of the button, bangs into a 1 message which is then sent into the audio player which starts playing the music! All this took was to create a few objects and tie them together. If I was to do this in a traditional coding language (I very much couldn’t) there would be a lot of code to mess around with. Some people love it, but I am not one of them so I love this method.

Anyway, I just wanted to share this short post inspired by my newly rekindled love affair with Max/MSP. A severe amount of fun was had designing this, thanks to my friend Michael McLoughlin for some assistance. What you saw here was very much a draft, and I am definitely not a master of the program and I am sure there is a better way to do what I want done but it does the job which is the main thing. If all goes well and it is accepted by my project supervisor, I will release the source code for anyone else doing surround sound subjective work. Hopefully, someone reading this may realise that Max/MSP may help them in some way so I hope this post gets you into it!

Keep on rocking!!