On blind A/B testing

[frankclarke]

Wikepedia has this:
http://en.wikipedia.org/wiki/Randomized_controlled_trial
"Permuted block randomization" might be Gellerman. The block of 4 patients could be 4 LEDs. You sequence through the 4 trials, as you note the results, then press the switch to reveal the LED states. Would Clever Hans hear the relays?
I always liked "Complete randomization", so that would simplify the hardware (mechanical).

[brett]

Maybe this is getting too off-topic, but in stats it is important not to underestimate the importance of stratification or blocking.  In this case it could be used to "control" effects of person-to-person variation. e.g.
Person (block) 1 gets AABABAA
Person (block) 2 gets BAABABB
etc...
So it's random and replicated within blocks.  You could use elapsed time, or number in sequence, as a co-variate to pick up the effects of decreasing or increasing accuity.

However, having this debate is alreay strong evidence that no subtle differences or mojo will be detected in most tests.  You might ask: How could I know that?  It's because people believe that "the grass is always greener...".  Generally, we like it if mojo is real (e.g. NOS tubes sound better).  We like a good story, even if stretches the truth (a lot).  This makes it easy for stories about "special" or "secret" mojo to accumulate and spread.  Scepticism and testing that might eradicate mojo aren't common, especially in something as subjective as music.  Therefore we can conclude, before any testing starts, that mojo will usually fail to pass any test that separates fact from fiction, reality from fantasy.
cheers

[dirk]

Differences on the threshold of hearing need lots and lots of tests with lots and lots of people, simply because ABX testing is a statistics game.

But at how many tests can you say: Difference not audible or not big enough.

[Mark Hammer]

As much as I respect these very diligent efforts to rule out all possible confounds, the goal here is not to set standards for publishing fuzzbox findings in JAES or the Journal of Audiology or Psychology of Music with alpha values that blow reviewers out of the water.  The goal is simply to answer the very cogent question in the mind of any home-experimenter "Did I really hear what I think I heard?".  I'm satisfied that some method of stepping lightly around the most common forms of expectation-influenced perceptions will let folks answer that question for themselves reasonably well.  And, just like "real" science, we'll let the doubters carry out their own experiments at home, with the same methods, and either replicate...or not.

It occurred to me this morning that one might be able to use randomly presented soundfiles.  But then I realized that most soundfiles tend to be nonrepresentative of the full range of operating conditions.  I suppose one could record a bank of soundfiles and randomly select from Bank A or Bank B, but then you could probably eventually recognize which bank was which just by what you played.  So, practically speaking, it's probably better to just use live sound and not know in advance what path your signal is taking.

[MR COFFEE]

Hi Mark,

Just a suggestion to stick in your pipe.

Whatever you finally get going, see if you can tell the difference between A and B with the same source going to both. With high, medium and low level signals.

I frustrated and embarassed myself years ago when I realized that one of my subjects (a musician, of course) could hear the difference between the way two relays bounced. I was feeling very triumphant that he could detect a waveform difference the big guys said people couldn't hear until I asked him to describe how the two waveforms sounded different to him subjectively. When he referred to the transient (my word), I got suspicious and tried running the same signal to both. He could STILL tell them apart (ugh, deflate ego, slink away....) I couldn't believe it. Even with electronic switching without a sharp transient, unmatched rise times might still be perceptible. I'd check it to be sure if I were going to publish something.

The human ear is pretty amazing. Some people can allegedly hear phase with certain waveforms (saxaphone and other positive pressure instruments). Hard for me to know what to believe ob that one. Should I buy it's the phase of the recording or some difference in the phase response of the speakers they are hearing though. 

What are you going to test anyway?

mr coffee

[brett]

Hi
I don't find it too surprising that people can pick phase.  Only one side of the eardrum is covered in a large skull and brain, so some hysteresis seems likely.
cheers

[Mark Hammer]

Hi Mark,

Just a suggestion to stick in your pipe.

Whatever you finally get going, see if you can tell the difference between A and B with the same source going to both. With high, medium and low level signals.

Yep.  That would be the ideal. To have a variety of signals/samples/riffs/etc passing through each of the alternative paths.  So "Do tantalum caps 'sound' different in that position than electrolytics?"  Good question.  Why don't I breadboard something up and select either tantalum OR electrolytic in random fashion, not letting myself know which until later.  I can noodle around, step on a switch for the next "trial", noodle around some more, and go back and forth.  I could do the same with op-amp A vs B, cable A vs B, and so on.

I frustrated and embarassed myself years ago when I realized that one of my subjects (a musician, of course) could hear the difference between the way two relays bounced. I was feeling very triumphant that he could detect a waveform difference the big guys said people couldn't hear until I asked him to describe how the two waveforms sounded different to him subjectively. When he referred to the transient (my word), I got suspicious and tried running the same signal to both. He could STILL tell them apart (ugh, deflate ego, slink away....) I couldn't believe it. Even with electronic switching without a sharp transient, unmatched rise times might still be perceptible. I'd check it to be sure if I were going to publish something.

The human ear is pretty amazing. Some people can allegedly hear phase with certain waveforms (saxaphone and other positive pressure instruments). Hard for me to know what to believe ob that one. Should I buy it's the phase of the recording or some difference in the phase response of the speakers they are hearing though. 

Considerable research has demonstrated that infants under 8 months are capable of making very subtle differentiations between speech sounds made in virtually any of the world's many languages.  For a while, it was thought that the human brain was hard wired to process speech sounds, until other researchers started observing that animals like chinchillas and racoons could make the same discriminations; animals that lack and have no need for language.  At that point scientific reasoning flipped around, and theorists started proposing that rather than being wired for speech, instead we invented speech to exploit the sorts of auditory discriminations we're naturally capable of.  In other words, making the most out of what we could do.

[AM]

I read somewhere long ago that an A/B test can be influenced to some degree by the set of individuals selected. A test with a set of weak character people may differ from a set of strong personality people.
mac

Yeah, that makes perfect sense to me. I've also read in a scientific article recently that the person who is in charge of operating the switching between the equipment under test can sometimes unintentionally influence the person who is listening just by transfering his or her biased opinion through body language or even through his or her energy. Maybe it sounds a bit crazy but I wouldn't dismiss it.

In my opinion one should always trust his/her insticts and ears but there are also some truths which our ears can sometimes hide from us.
I always use my ears as the first step of gear evaluation. If I like what I hear then I move to the next step which is to use some more solid method to confirm what my ears hear.