Surround Sound Subjective Test Design 3 – Collecting the Data

Posted on January 26, 2014 by robkellysound in Audio Production Projects, Surround Sound Recording Research

A Max/MSP newbie muddling through the of subjective testing for audio! This is the third of three posts which cover the test design for my research project. Click for one and two.

Hello all,

Dare I say it, everything looks to be very close to finished in terms of the testing interface and crucially, the method of collecting and sifting through the results! This post is going to take you through the testing procedure and processing of results. Hopefully everything will make sense, if not then thankfully I can catch it before the actual testing starts!

Collecting Results

This test section allows participants to play around and learn the interface they will be using for the test proper.

The first thing participants will see after the section to input details such as age, will be the test section. Here users can familiarise themselves with the interface. A totally separate surround sound recording will be playable from this section. Once they are comfortable, they can move on to the test proper.

The picture above is what participants will see at the end of the test. There are 12 sections so there is no need to post all that. The important thing is the test reference number, which is the main point behind this blog post. Previously, I talked about my “Randomisatron”. This messy, complex and head melting monster of a subpatch allows me to shuffle the standard order of the test. The randomisation allows my test to meet the ITU BS1116 standard for the playback of test material where test samples must be presented in random order for each test participant.

Since I know the default playback order for each section (A and B, C and D, E and F etc.) I can shuffle the playback order with a code using the Randomisatron (this name may stick, but maybe not for the write up) With the participants answers filled out, I should be able to (un)shuffle the answers back into the original order with reference to that randomising code. This calls for a trip to Excel!

Processing the Results

My friends have always slagged me off because of how much I like using Excel. That is not to say that I spend my evenings pondering some Excel magic. The reason I like Excel is because at the end of this test, I am going to have 5 answers for each of the 12 sections from at least 20 participants. This is not something I want to do on paper. Excel appeals to my lazy side and if I can get rid of a lot of leg work with one swift click of the mouse, I will gladly take it!

Test Ref — The test reference code contains the randomising information.

Bear in mind that the Excel file is made so I understand it where the test interface has to be much more sleek. Hopefully I can make this make sense for you! The two columns on the left are the default order which the whole test interface has been built around. This info also pops up on the far right as a reference to make things a bit easier. What I have highlighted in yellow is the randomising part of it.

When I get a set of results from a test participant, I will fill in all the answers for questions 1 to 5 all the way down the 12 sections. These answers can only be A or B/1 or 2, the numbers I have in at the moment were just for when I was testing the thing out. With the answers filled in, I will then take a look at the test reference number and type it into the left hand yellow column. The right hand yellow column is a copy of what the default is, 1 through to 12.

Here is that first Excel image again. Imagine there are 12 CDs with the recordings on them. Take a look at the first CD which is “SF1, T1”. The randomising code is telling the Max patch to play CD1 2nd in the queue. It is telling the second CD to play 12th in the queue and the third CD to play 1st.

The first step is to sort the New Question Order column by the Rand. column. This turns the Rand. into 1 to 12 while also moving the New Question Order numbers with it. What this does is match up the numbers in the New Question Order with the answers I filled in at the start. Remember earlier I said that the third CD was being told to play 1st? Take a look at the first number in the orange column. Remember that the first CD is to be played 2nd in the queue? You can see that from the second spot in the orange column too. Finally, you can also see that the 2nd CD was played in the 12th position. Notice how the CD1 and CD2 answers are A, B, C and D in the image. We know that these below to the first two sets, recordings 1 to 4, so they ideally should be at the very top and lined up with their “SF1, T1” counterparts at the far left of the spread sheet.

BOOM!

I do these steps for each participant and when I get all 20+ finished I can then sort all the Question/Answer columns by the Original Order column and now all the answers for each set of recordings are grouped together, ready for me to do some whole other amount of work so I can get statistics and results processed, something I best read up on soon.

Thanks!

Thanks again for reading. It looks like I have made a small trilogy of posts about how to administer a subjective test in terms of playing sounds to meet the ITU standard, collect the answers to the questions and then process them for ease of use later on. Take a look at the top of this post for links to the other posts.

Thanks very much for reading, I hope someone finds it helpful!

Surround Sound Subjective Test Design 1

Posted on January 20, 2014 by robkellysound in Audio Production Projects, Surround Sound Recording Research

A Max/MSP newbie muddling through the of subjective testing for audio! This is the first of three posts which cover the test design for my research project. Click for two and three.

Hello all,

With the project moving at a nice pace I wanted to share some information with you about the test design.

Playback Requirements

Iin the subjective testing stage of my project, I am asking listening test participants to compare two recording extracts and answer some questions on them. I am testing three arrays, a traditional and two variants of a Soundfield recording. All recordings were done simultaneously. The ITU BS1116 standard for “the subjective assessment of small impairments in audio systems including multichannel systems” applies to my project in the most suitable way. There are certain key requirements which are set out by the standard which include:

– Test participants must be able to switch between recordings as they wish with no loss of place or jump in the sound (eg. Skipping a few sections)

– To avoid bias, test material should be randomised (see post 2)

– The transition between each sound extract must be 80ms in length.

– other things (too much to list!)

My problem was, I did not know of any way to do this without me being in the test room with the participant and even then, human error would be inevitable with timing differences between each switch being different, that and a whole host of problems to say the least! The bottom line is that this could be off putting to participants. Added to that, I needed a way to collect information and answers from the participants. So having one thing for that and another to get them to play sound tracks and switch between them would probably be messy. I needed something so instead of blindly researching a way to do it, I made my own!

Super Max! (Supermacs if you’re Irish)

Max/MSP was the program what I used to solve my problem. (now known as Max 6, despite my constant use of Max/MSP in these blogs) It is a program I was introduced to back at Queen’s University. Coding and programming is not my thing but I always found this program fun and logical. When I encountered Reaktor at Salford Uni I was less than impressed after coming from Max/MSP so when I had the idea of using object based programming to sort my problem out, Max was my first port of call. I have a feeling Reaktor could have done it, but not as well as I knew Max could. I have used both programs to complete the same objective in the past so that is what I am basing those remarks on.

What I have designed here is probably very likely an inefficient way of doing things, so if any Max savvy people are reading then don’t say I didn’t warn you! While I get more research done into what is required in subjective testing for audio, the layout and things like will probably change but under the hood I have the exact program I need to meet the fundamental aims of BS1116.

I wanted to play two surround sound files and allow the user to switch between them instantly without losing place, like I mentioned earlier. I also wanted it to record the answers to the test questions. By combining all these things into one interface, it means that I can save the participants answers using a unique file name and load up a new blank patch for the next person.

The GUI features of Max are fantastic. Maybe my colour scheme needs a look at (probably spelling too, but it’s a late night draft!). What the patch allows me to do is play two surround sound files at the same time, select between them in realtime and select my answer.

Here I selected the recording I preferred, hit confirm and the patch them told me what to do. But what if one of my participants makes a mistake like selecting both the options?

BOOM! The patch recognises that both preference buttons have been selected and gives a warning message. The user can then reset, reselect and confirm again.

This is the magic behind it.There are three main sections.The upper right quarter of the image is the playback system which is routed into the audio outputs in the lower middle. The upper left quarter to middle is where users can select what recording they are listening to and also the display which tells each participant this for their own reference. The bottom right quarter is where the participants answers are filled in and tested to make sure that two options or no options were selected by accident.

What the participant will see is what was in the previous pictures. This under the hood view is how they are all tied together. Max is an object based programming language. Each visual object you see has some traditional coding behind it, but the user does not need to worry about it. It allows the less code savvy like myself the ability to get some seriously powerful custom made projects done without worrying about syntax errors and languages. In fact, the only bit of traditional code that I have come across that you could need are if statements. That said, the help files are brilliantly done so help is always at hand.

Take the play and stop buttons for example. The button has some code under the hood that says when pressed, it sends a signal or a bang as it is called in Max. The output of that play button eventually goes into the object called “sfplay” by following the blue path. Sfplay plays soundfiles and there are two of them because I have two surround recordings that I need played at the exact same time. Note that the output (bottom of the boxes) have 6 green/grey lines. Each of those corresponds to a audio channel on my soundcard. Since 5.1 needs 6 channels I asked for an sfplay with 6 outputs.

Once sfplay gets the message “1” it will start playing, and when it gets a “0” it stops! You can see how the blue line from play goes through a grey box with “1” in it. So, what happens here is that the bang comes out of the button, bangs into a 1 message which is then sent into the audio player which starts playing the music! All this took was to create a few objects and tie them together. If I was to do this in a traditional coding language (I very much couldn’t) there would be a lot of code to mess around with. Some people love it, but I am not one of them so I love this method.

Anyway, I just wanted to share this short post inspired by my newly rekindled love affair with Max/MSP. A severe amount of fun was had designing this, thanks to my friend Michael McLoughlin for some assistance. What you saw here was very much a draft, and I am definitely not a master of the program and I am sure there is a better way to do what I want done but it does the job which is the main thing. If all goes well and it is accepted by my project supervisor, I will release the source code for anyone else doing surround sound subjective work. Hopefully, someone reading this may realise that Max/MSP may help them in some way so I hope this post gets you into it!

Keep on rocking!!

My Project Description Take 2 – With a Meme!

Posted on January 19, 2014 by robkellysound in Audio Production Projects, Surround Sound Recording Research

Hi all,

I was chatting with a friend recently and the meme below came up. A few minutes ago, it hit me that the meme could be a good way to structure a post or set of posts about my MSc project, that and it means that I don’t have to do this through the medium of interpretive dance!

It has been pointed out to me that my blogs are usually quite long. I agree, I do love the look of my text! Anyway, anyone who could have a passing interest may get pushed away by my blabbing so like a meme, this post is going to describe the project I am doing in a short and sweet way. It is important that I use the blog to keep track of the project in all its technicalities, but I also want to keep an informality to posts at the same time. What better way to do that then through a meme!

Observation

The Soundfield system allows an engineer to replace the 5 microphones normally used for surround sound recording of classical music with a single microphone. The Soundfield microphone allows an engineer to adjust what has been recorded after the recording takes place to create the best sounding production that is possible from the microphone being placed where it is. It is quite easy to setup and use. This is a huge selling point. Imagine taking a photo and wishing you could change the lighting days, months or years after the photo is taken!

A “traditional” surround recording system uses 5 separate microphones, also known as a multi-microphone array. Each microphone has its own stand with each having a specific angle and distance relationship with each other. The rear microphones are generally spaced a fair bit away, sometimes meters away. They can be difficult to set up and if a mistake is made, it can not be fixed after recording, a bit of a hassle and you can’t adjust things after the recording like you can with the Soundfield.

Questions

Well, there are significant physical differences between the Soundfield and traditional arrays. Does the Soundfield sound better or worse to a set of listeners in a specific recording session? Can they both achieve excellent results? If one does and the other doesn’t, why is that? Would the Soundfield rate with listeners given the significant differences between it and traditional arrays? Is the ease of use and after recording touch ups worth any quality issues that the Soundfield may have?

Hypothesis

Well, this is a tough one. A lot of those questions can be answered by saying each recording method has its own characteristics. Certainly, there wouldn’t be one better or one worse across music recording as a whole. But, if you record a musical ensemble and play the results to some listeners and ask them what they prefer, maybe one method would stand out. At this early stage of the project, I would wonder what people think. I think that the two methods are both as good as each other and are viable methods of recording music, but when recording a musical ensemble I would question how listeners would react to how differently the Soundfield deals with the rear microphones.

Prediction

Well, I am not too sure about this one yet. Since my last post, I have been doing a lot of reading and my literature review so far do not show much research directly comparing the traditional arrays vs. the Soundfield for the type of music I want to use (small classical group). So I am on the fence. As an engineer, it would be great to know that the easier to use Soundfield is as preferred if not more preferred than traditional arrays. So, lets find out.

Experiment

Through research of the various Audio Engineering Society papers, I have found that the Decca Tree and its derivative called the Fukada tree surround arrays are two of the most preferred traditional arrays. So, to ensure that the Soundifeld is getting a fair fight as such, I want to record a small classical ensemble with the arrays. To make sure everything is equal, the arrays will be setup simultaneously and recorded at the same time too. Then I want to play extracts to listeners to find out which is the most preferred.

I ruddy hope so!

My next project post will be about what I have found during my literature reviews!

Thanks for reading!

My Masters Project – Surround Sound Recording – An Informal Run Through

Posted on January 14, 2014 by robkellysound in Audio Production Projects, Surround Sound Recording Research

Hello, long time no blog!

Today, I want to talk in brief about my final project for my masters which is designed to be a piece of research in the general area of audio production. This is an informal post, meant to be gloss over certain points in an effort to keep things short and to the point in an effort to make the topic as accessible as possible. Once the project is finished, I will be able to make new posts about each part of the project in more detail! If you have any questions, just get in touch through here. The final dissertation can be read here.

Recently, I recorded a choir for surround sound playback. What this means is that when listening back to the recording in a room properly equipped, you will feel inside the room where the music was being performed. The sound will envelop you from all around you, simulating what it would have been like to be in the concert hall which is a very cool experience. In surround sound you have five speakers. Three at the front for the left, centre and right with two are the rear. Check out the below image from Sound On Sound Magazine about the placement of these speakers.

http://media.soundonsound.com/sos/jan01/images/surround2.gif

Recording Arrays
Many ways of recording surround sound have been developed over the years. The most common are what I call traditional arrays which is a catch all term for multi-microphone recording arrays. If you take note of the image above, there would be one microphone for each speaker. In general, the left, centre and right point at the respective areas of the stage and the rears point into the rear corners of the space. What the microphones “listen” to then gets recorded and played through their respective speakers outlined in the picture.

The collection of these microphones is called an array and then can usually subdivided into the front array and rear array as there can be a larger spacing between the front and rear. If you are at a concert and stand at the very front, you will get a great clean sound. If you stand at the back you get a more reverberant sound so an aim of these types of arrays is to capture both as best possible for use in the production. A pleasing recording for the listener can be achieved by placing the front array fairly close to the performers to get a clear sound and then by placing the rear array into something known as the reverberant field.

Here is a photo of a traditional array called INA5 from www.sanken-mic.com. You can see the distances and angles involved, especially between the front and rear.

The Soundfield
For the recording engineer, traditional arrays have around five microphones which means there are a lot of cables which are sometimes quite long, a lot of stands which are usually heavy and/or wobbly, a lot of measurements and angles which can be cumbersome to get correct and then possible headaches to worry about when setting everything up, for example, someone walking into or moving the stands.

There is a relatively new microphone called The Soundfield microphone. Roughly speaking, this is 4 microphones in one. This can be placed in a recording environment just like the front section of a traditional array can. The Soundfield microphones pickup can be seen as mainly based on figure of 8 microphone pattern, to keep things simple. A microphones capsule is in the shape of a big coin and listens in certain ways around it. Some microphones listen to just what is in front and on either side of it. Others listen all around it. The figure of 8 listens to the front and back while ignoring the sides. Imagine two tennis balls placed on either side of those big chocolate coins, this gives an impression of what directions the microphone is picking up from. Here is a photo of a microphone capsule from recordinghacks.com

Here is a diagram of a figure of 8 microphone from the Recording Review Forum, think of those tennis balls.

Basically, these types of microphones are listening in a shape on front and behind the microphone while ignoring what is going on at the sides (includes top and bottom). For the sake of example, imagine there are two of you and if you say something and your duplicate says the exact same thing in the exact same way. If the two of you are placed either side of the microphone and you both say something, anyone that is listening to what the microphone is listening to will hear nothing. That is because one side the microphone listens in a positive way and the other negative, which doesn’t mean one side is happy and the other angry, what it means is that everything can be boiled down to numbers.

What this means means is that what one of you saying could gets turned into the number 1 and the other gets turned into a -1. When the microphone combines these things you get 0, or nothing being heard. With that admittedly odd example out of the way, the Soundfield works in a similar enough way. Depending on what way you add and subtract the signal that the microphone creates, you can hear what is happening at any direction. Think of it as a 360 degree security camera. If you are watching something from the left and then move to the right, you use a control to point the camera in that direction. The Soundfield is similar, but for sound and instead of the camera moving around, the mathematics are being changed to adjust the direction of what the microphone is listening to.

Why mention all this?
Well, it is easier to setup that a traditional array. One stand, one main cable and less headaches. More importantly, after the recording is done and you are mixing the recording you can change where the microphone is pointing as you wish. This is because you are combining what was recorded from the microphone, not having to adjust the direction its pointing in on the day of the recording itself as you would with a traditional array. On top of that you can derive the five different directions at the same time which are the signals you need for the 5 speakers in a surround sound setup. With modern technology and the ease of having a powerful computer for audio production, this can be automated which means no more messing with angles and protractors at 6 feet in the air! With a traditional array things need to be set up exactly and if you make a mistake or something happens to the array which you didn’t know about, you can not fix the problems in the mix!

The Soundfield is also fairly expensive compared to the five standard microphones you need for a traditional array. Expensive not only in terms of money but also in versatility as if you are an engineer who does recording work with bands and close micing, the five standard mics can be much more useful to you than the single Soundfield mic.

Here is a photo of one of my recordings with the traditional array of five microhones in red and the single Soundfield in blue. (very well drawn eh?)

What my project is about.
The Soundfield may be much easier to set up and it may be versatile after the recording in terms of changing its settings to fix issue but does it sound as good as the traditional setup? Remember when I said that the rear microphones of a traditional array are placed further back into the reverberant field? You can’t do that with the Soundfield. The Soundfield is one microphone in the sense it is a single enclosed unit. If you move it further back to get more reverb, you make the front more reverberant too and lose the clarity. Compromise! Additionally, the two concepts simply sound different to each other, not necessarily worse than the other but finding out what a sample of listeners think could help in making the decision between what to use or what to buy.

What I want to do is find out what is “better” when asking a sample of listeners. I intend to record a choir and set of classical musicians with a Soundfield microphone and a traditional array simultaneously. Then, I want to play sections of the songs in a subjective listening test where expert and not so expert listeners can sit in a surround sound listening room and decide what is their favourite, without knowing which is which. The result of this, paired comparison test, will hopefully highlight which recording method is the most preferred. That said, they could be equally preferred, showing parity, and that would not be a bad result. All that means is that the engineer can be faced with a choice of what type of sound they want rather than facing one which could have an impact on listener enjoyment.

Thanks
Thanks for reading, this was intended to be an informal and accessible look into the background of the project. More detailed and precise information can be found in the final dissertation here. If you are new to the concepts I outlined here and are interested, do let me know and I can guide you to more formal information. =)