Appendix A. Max Patch
The test interface, shown in Figure A.1, was built in a Max 6.1 (Cycling 74, 2014). A number of elements had to be designed in order to meet the aims of the test which can be summarised as to:
- Provide a test interface which will allow for pairwise comparison of two surround sound recording extracts,
- Provide a means to randomise playback of test material,
- Provide looped playback of test material,
- Provide real time switching of test material,
- Allow for participants to submit their answers to test questions,
Once a system which meets the above criteria is established, it can then be copied and pasted as required. For example, by building and troubleshooting one test section, the final result can be copied and patched in with ease. This Appendix outlines the design of the major components the patch.
A.1 Randomisation
To meet the randomisation section of the ITU-R BS.1116-1 recommendation, a way of randomising the playback of test stimuli had to be incorporated into the test interface.
Figure A.2 shows one of the twelve components of the randomising system of the patch. Each of the audio extracts for each pairwise comparison was fed into one of these randomising sections. The objects ‘open sf1.aif’ and ‘open t1.aif’ correspond to the Soundfield ITU and MMA recordings of the first section of musical piece 1. This file information is then routed to the sound file players via a randomising system.
By entering a randomisation code, where each number is applicable to each randomising section, the destination of each array pair could be randomised. For instance, if Pair A is played back first by default and if the first number of the randomising code was eight then Pair A would be the eighth pairwise comparison presented in the test. Table A.1 shows how a full randomising code changes the default playback order of the test with arrows highlighting two of the playback order changes.
Table A.1 – Randomising Operation
This system allows the test supervisor to input a code from a pool of randomising codes before participants begin the test. The randomising code is presented with the participant’s answers so they can be integrated into the master spread sheet where the reverse of the randomisation process is applied to allow for the analysis of the results. To avoid using the default playback order by accident, the patch can only be initialised after the randomisation code is confirmed as being entered correctly.
With reference to Figure A.2, the twelve number buttons which are routed inputs one to twelve inclusive and inputs thirteen to twenty-four inclusive of a sub-patch denoted by the name ‘p 14’. The sub-patch, shown in Figure A.3, routes the signals from its inputs into two gate objects. There is one gate object per recording extract. Sound file information from the array pair enter the sub-patch through inputs twenty-five and twenty-six.
With the randomising number given, the number acts on both the gates and sends the sound file information for the recording extracts through the appropriate gate output. For example, if number one is pressed, the signal of input 25 and 26 will be sent out the second output of gates one and two respectively. The addition of the one which can be seen at the top of Figure A.3 is to eliminate errors further down the playback chain. The gate outputs are routed to the sub patch outputs and routes signals to the ‘send’ sub patch shown in Figure A.4
The outputs of the gates are then routed to the appropriate send objects. The send object allows a signal to be sent around a patch without the use of patch cords. If the number one is pressed, the gates will send the sound file information for the array pair to ‘send 1a’ and ‘send 1b’ respectively.
A.2 Playback
Figure A.5 shows how the sound file information is processed. The objects called ‘receive 1a’ and ‘receive 1b’ receive the signals sent by the objects ‘send 1a’ and ‘send 1b’ respectively. The receive objects are then routed to two separate objects called ‘sfplay~’ which is a sound file player. The signals which have been sent as outlined in Section A.1 correspond to the file locations of the tests recording extracts. The ‘sfplay~’ object will then use this information to open the appropriate files for playback. The first argument for the ‘sfplay~’ object defines how many channels are required for playback. A 5.1 surround sound channel requires 6 channels, hence ‘sfplay~ 6’.
Figure A.6 outlines the playback stage. The ‘Play’ and ‘Stop’ buttons are visible to the participant as controls of the audio playback. By pressing play, the playback loop feature of the ‘sfplay~’ object is activated.
The sets of outputs from the two ‘sfplay~’ objects are routed to inputs two and three of the six ‘selector~’ objects respectively. The ‘Recording 1’ and ‘Recording 2’ objects are visible to the test participants. These are seen by participants as sound selection controls which allow for the real time switching of audio file playback. The output of these buttons is routed into a ‘1’ and ‘2’ message objects respectively.
The message boxes are in turn fed into all of the first inputs of the ‘selector~’ objects. If the selector objects receive a ‘1’, they will send the signal from the second input through to its output. If they receive the number ‘2’, they will send the signal from their third input to their output. This group action switches between the output of either ‘sfplay~’ object.
The ‘delay’ sub patch sends a ramped signal between one and zero out to the six multiplication objects which are placed before the audio output, called the ‘dac~’ object. When the participant selects a recording, the outputs are muted by the delay sub patch. The sub patch then sends out the switching signal before unmuting the audio outputs.
Figure A.7 shows a subsection of Figure A.6. The outputs of the ‘Recording 1’ and ‘Recording 2’ objects send a one or zero depending on their on/off state. They are routed to ‘if’ statements. The ‘if’ statements are then routed to a display object. The display object will display a separate message for each of the input signals ‘0’ and ‘1’. The display object is visible to the participant and provides them with the information of what recording extract they are listening to.
A.3 Answer Submission and Collection
Figure A.8 shows an example set of answers. The objects ‘Recording 1’ and ‘Recording 2’ are visible to the participants in their respective sections of the visual test interface. The two sets of answers displayed in Figure A.8 correspond to the preference and spaciousness questions of a test section. If one answer is given by a participant, the output of each answer section will be one. If no or two answers are given, the output will be zero or ten respectively.
Figure A.8 – Answer Collection
In the case of Figure A.8, the desired output of one and undesired output of ten have been sent to the answer verification section of the patch which is shown in Figure A.9. From the five test questions, the section of the patch is reacting to whether it receives a number five or not. In this example, question one was answered acceptably where question two was not with the rest of the questions unfilled.
Figure A.9 – Answer Verification 1
The ‘expr’ object adds up the output of each answer section and sends it to an if statement. As the output does not equal five, a warning message is sent to the participant instructing them to amend their answers before continuing. The desired situation of all questions being answered correctly is shown in Figure A.10 where all five questions have been properly filled out. The result of five from the ‘expr’ calculation gives an instruction to the participant to move to the next test section or contact the test supervisor where appropriate.
Figure A.10 – Answer Verification 2
Appendix B. Result Data
Appendix section B.1 details the two formulae used to calculate the Z-score and p-value of the significance tests. The remainder of the Appendix provides the results of the pairwise comparisons which were graphically presented in Section 6.
B.1 Formulae
B.1.1 One Proportion Binomial
As the data from the pairwise comparison in this project is categorical, the one proportion binomial test can be used to calculate significance between the proportions of the comparisons sample (Bower, 2014)(Elder Laboratory, 2014).
To use an example, a set forty participants were presented with a pairwise comparison between options A and B and asked which one they preferred. The null hypothesis states that the options are equally preferred, thus H0 equals 0.5. If thirty selected option A, then the number of ‘events’ would be thirty which equals x.
Equation B.1 – One Proportion Binomial Components and Example and Result
As the null hypothesis states that there is equal probability of A and B being preferred by participants, the two tailed test allows for the determination of the preference between A or B. This results in a critical region for a 95% significance level from -1.96 to 1.96. The binomial test revealed that there is a significant preference for option A at a 95% confidence level, z = 3.16, p < 0.05.
B.1.2 Bonferroni Correction
The hypothesis testing formulae used were integrated into the master Excel spread sheet which contained the test result data. The statistical software Minitab was used to confirm the correct operation of the spread sheet calculations (Minitab, 2014). Due to the number of comparisons being made as described in Section 4.1, a Bonferroni correction of three was applied (Goldman, 2014).
Equation B.2 – Bonferroni Correction 1
Equation B.2 outlines the Bonferroni correction where o is the original confidence level and b is the Bonferroni correction value. This calculation produces an adjusted significance level to take into account the number of comparisons being made per set of test stimuli. The significance levels are used to assess the results of certain hypothesis testing.
For example, if the result is below the significance level of 0.05 then the result is significant to a 95% confidence level. Similarly, if the result is below 0.01 then the result is significant to a 99% confidence level.
The binomial test does not produce p-values; however, it does operate with respect to the critical region described in Section 5.2 as the results of the test will either be inside or outside the region. The NORM.S.INV function of Excel, which calculates the critical region, requires a significance level to operate. The result of the Bonferroni correction can then be implemented into the spread sheet where the NORM.S.INV adjusts the critical regions which in turn adjust the results of the binomial test.
With a Bonferroni correction of three applied, Equation B.3 and Equation B.4 show the adjusted 95% and 99% significance levels respectively. These values have been used in the calculation of results of the project listening tests which are presented in Appendix Sections B.2, B.3 and B.4.
Equation B.3 – Bonferroni Correction 2
Equation B.4 – Bonferroni Correction 3
The original significance levels for 95% and 99% are 0.05 and 0.01 respectively which have now been adjusted to 0.017 and 0.003 respectively. These values are then used to calculate the critical regions for use in the binomial tests. For example, the original critical region for a 95% confidence level is -1.96 to 1.96. The Bonferroni adjusted critical region for the 95% confidence level is -2.39 to 2.39.
B.1.3 Chi-Square
To find the exact probability of a result in a pairwise comparison, known as the p-value, the Chi-Square test was used. In the example presented in Appendix B.1.1, the expected preference or probability that a population will select option A is equal. With a sample size of 40, the means the expected result for each category is 20. The observed result for category A is 30 and 10 for category B. For the result to meet the 95% confidence level, the result of the chi-square test must be less than the significance level of 0.05. The Chi-Square test was used, in addition to the Minitab software, to confirm the results of the binomial test.
Equation B.5 – Chi-square Example and Result
B.2 Overall Preference Results
B.2.1 MMA vs. SFI
65 out of 80 votes preferred MMA. A binomial test revealed that there is a significant preference for MMA at a 99% confidence level, z = 5.59, p < 0.003.
B.2.2 MMA vs. SFA
50 out of 80 votes preferred MMA. A binomial test revealed that there is not a significant preference for either array at a 95% confidence level, z = 2.24, p > 0.017.
B.2.3 SFI vs. SFA
29 out of 80 votes preferred SFI. A binomial test revealed that there is a significant preference for SFA at a 95% confidence level, z = -2.46, p < 0.017
B.3 Preference Results by Piece
B.3.1 MMA vs. SFI
In piece 1, 34 out of 40 votes preferred MMA. A binomial test revealed that there is a significant preference for MMA at a 99% confidence level, z = 4.43, p < 0.003.
In piece 2, 31 out of 40 votes preferred MMA. A binomial test revealed that there is a significant preference for MMA at a 99% confidence level, z = 3.48, p < 0.003.
B.3.2 MMA vs. SFA
In piece 1, 22 out of 40 votes preferred MMA. A binomial test revealed that there is not a significant preference for either array at a 95% confidence level, z = 0.63, p > 0.017.
In piece 2, 28 out of 40 votes preferred MMA. A binomial test revealed that there is a significant preference for MMA at a 95% confidence level, z = 2.53, p < 0.017.
B.3.3 SFI vs. SFA
In piece 1, 14 out of 40 participants preferred SFI. A binomial test revealed that there is not a significant preference for either array at a 95% confidence level, z = -1.90, p > 0.017.
In piece 2, 15 out of 40 participants preferred SFI. A binomial test revealed that there is not a significant preference for either array at a 95% confidence level, z = -1.58, p > 0.017.
B.4 Attributes
B.4.1 MMA vs. SFI Piece 1 and Piece 2
B.4.1.1 Spaciousness
In piece 1, 28 out of 40 participants chose MMA. A binomial test revealed that there is a significant preference for MMA at a 95% confidence level, z = 2.53, p < 0.017.
In piece 2, 26 out of 40 participants chose MMA. A binomial test revealed that there is not a significant preference for either array at a 95% confidence level, z = 1.90, p > 0.017.
B.4.1.2 Envelopment
In piece 1, 31 out of 40 participants chose MMA. A binomial test revealed that there is a significant preference for MMA at a 99% confidence level, z = 3.48, p < 0.003.
In piece 2, 28 out of 40 participants chose MMA. A binomial test revealed that there is a significant preference for MMA at a 95% confidence level, z = 2.53, p < 0.017.
B.4.1.3 Clarity
In piece 1, 12 out of 40 participants chose MMA. A binomial test revealed that there is a significant preference for SFI at a 95% confidence level, z = -2.53, p < 0.017.
In piece 2, 15 out of 40 participants chose MMA. A binomial test revealed that there is not a significant preference for MMA at a 95% confidence level, z = -1.58, p > 0.017.
B.4.1.4 Naturalness
In piece 1, 23 out of 40 participants chose MMA. A binomial test revealed that there is not a significant preference for either array at a 95 confidence level, z = 0.95, p > 0.017.
In piece 2, 21 out of 40 participants chose MMA. A binomial test revealed that there is not a significant preference for either array at a 95 confidence level, z = 0.32, p > 0.017.
B.4.2 MMA vs. SFA Piece 1 and 2
B.4.2.1 Spaciousness
In piece 1, 22 out of 40 participants chose MMA. A binomial test revealed that there is not a significant result for either array at a 95% confidence level, z = 0.63, p > 0.017.
In piece 2, 20 out of 40 participants chose MMA. A binomial test revealed that there is not a significant result for either array at a 95% confidence level, z = 0.00, p > 0.017.
B.4.2.2 Envelopment
In piece 1, 27 out of 40 participants chose MMA. A binomial test revealed that there is not a significant result for either array at a 95% confidence level, z = 2.21, p > 0.017.
In piece 2, 26 out of 40 participants chose MMA. A binomial test revealed that there is not a significant result for either array at a 95% confidence level, z = 1.90, p > 0.017.
B.4.2.3 Clarity
In piece 1, 16 out of 40 participants chose MMA. A binomial test revealed that there is not a significant result for either array at a 95% confidence level, z = -1.26, p > 0.017.
In piece 2, 9 out of 40 participants chose MMA. A binomial test revealed that there is a significant result for SFA at a 99% confidence level, z = -3.48, p < 0.003.
B.4.2.4 Naturalness
In piece 1, 16 out of 40 participants chose MMA. A binomial test revealed that there is not a significant result for either array at a 95% confidence level, z = -1.26, p > 0.017.
In piece 2, 16 out of 40 participants chose MMA. A binomial test revealed that there is not a significant result for either array at a 95% confidence level, z = -1.26, p > 0.017.
B.4.3 SFI vs. SFA Piece 1 and Piece 2
B.4.3.1 Spaciousness
In piece 1, 13 out of 40 participants chose SFI. A binomial test revealed that there is not a significant result for either array at a 95% confidence level, z = -2.21, p > 0.017.
In piece 2, 12 out of 40 participants chose SFI. A binomial test revealed that there is a significant result for SFA at a 95% confidence level, z = -2.53, p < 0.017.
B.4.3.2 Envelopment
In piece 1, 16 out of 40 participants chose SFI. A binomial test revealed that there is not a significant result for either array at a 95% confidence level, z = -1.26, p > 0.017.
In piece 2, 13 out of 40 participants chose SFI. A binomial test revealed that there is not a significant result for SFA at a 95% confidence level, z = -2.21, p > 0.017.
B.4.3.3 Clarity
In piece 1, 16 out of 40 participants chose SFI. A binomial test revealed that there is not a significant result for either array at a 95% confidence level, z = -1.26, p > 0.017.
In piece 2, 19 out of 40 participants chose SFI. A binomial test revealed that there is not a significant result for either array at a 95% confidence level, z = -0.32, p > 0.017.
B.4.3.4 Naturalness
In piece 1, 19 out of 40 participants chose SFI. A binomial test revealed that there is not a significant result for either array at a 95% confidence level, z = -0.32, p > 0.017.
In piece 2, 16 out of 40 participants chose SFI. A binomial test revealed that there is not a significant result for either array at a 95% confidence level, z = -1.26, p > 0.017.
B.5 Attribute Web Test
This section outlines the binomial test results from the attribute web test described in Section 2.3.1.
For the spacious attribute, 17 out of 20 participants chose the more spacious recording extract. A binomial test revealed that this is a significant result at a 99% confidence level, z = 3.13, p < 0.01.
For the envelopment attribute, 18 out of 20 participants chose the more enveloping recording extract. A binomial test revealed that this is a significant result at a 99% confidence level, z = 3.58, p < 0.01.
For the clarity attribute, 19 out of 20 participants chose the more enveloping recording extract. A binomial test revealed that this is a significant result at a 99% confidence level, z = 4.02, p < 0.01.
For the natural attribute, 16 out of 20 participants chose the more natural recording extract. A binomial test revealed that this is a significant result at a 99% confidence level, z = 2.68, p < 0.01.
Appendix C. Saffire Pro 24
This appendix outlines the key specifications of the audio interface used for the playback of the test material. Information is sourced from Focusrite (2014).
C.1 Line level Outputs
- Dynamic Range (A Weighted): 105dB
- SNR (A weighted): 104.5dB
- THD+N: < 0.001% (measured with 0dBFS input and 22Hz/22kHz bandpass filter)
- Maximum level (A weighted): 16.13dBu at 0.885%
C.2 Additional Conversion Performance
- Clock jitter < 250 picoseconds
- THD+N AMPL (A weighted)= 107dBFS
