Blind Pedal Shootout on YouTube

[R.G.]

It took a while to get this up, but I think the results are worth it. You can see it at http://www.youtube.com/watch?v=RrufhPHi6m8

You all know I'm a fan of blind tests to eliminate the biases we all have. This is the first blind tasting test of effects that I know of. The audience was friendly , but blind; that means that they were willing to come to a blind test as a favor, but that they did not know which pedal they were listening to during the test. They were not paid, except for the snacks and sodas we provided, plus any entertainment they got from watching and voting.

We set up a number of categories of pedals and did a blind test of the sound using a live guitarist playing the same riffs for each pedal; same guitar, same amp, same player for each pedal, and multiple live samples of each pedal to listen to.

The Visual Sound pedals did well, as we had hoped they would. There are some surprises in there as well. Your ears may hear things differently, of course, because everyone's taste is potentially different. But I think this is a good (and fair - we worked hard at that!) illustration of a number of things about judging pedal sounds. I think you may be interested in this set of videos for a number of reasons. See what you think.

We are having DVDs made.

[Ben N]

Did the players know which pedal they were playing? I think that there are subtle effects that can occur to the way a player plays depending on his/her expectations. Also, who handled settings on the pedals?

[Mark Hammer]

Ahhhhh....that DVD.

That's a whole helluva lot of work, both for the player and the rater.

You should get Bob to talk to Gordon Logan at Vanderbilt (if he hasn't already).  Gord and I go wayyyyyy back to when he and his wife first arrived at McGill from University of Alberta.  Besides being an acknowledged world-class cognitive scientist with a distinguished career and very active laboratory, Gord is also a pretty decent guitar player, avid collector of vintage gear (I'm pretty sure that George Gruhn has bought some of his many pets from the proceeds of sales to Gordon), and from what he tells me still gigging intermittently.  Gord would hip Bob to the "perfect" blind test....not that these aren't well done already.

[R.G.]

Did the players know which pedal they were playing? I think that there are subtle effects that can occur to the way a player plays depending on his/her expectations. Also, who handled settings on the pedals?

Good questions, and really pertinent ones for a fair test.

There was only one player, primarily to keep player variation down. While it is conceivable that Zac could have ferreted out what pedal was what, we tried to eliminate that, and for just the reasons you mention. From where Zac stood it was difficult to see what pedal was being used, and there were so many that locating the active one would have been difficult. We also did a lot of back and forth between pedals to ensure that there were enough samples that everyone listening could get a firm idea whether they liked Pedal #1 or Pedal #2 better. I believe a lot of that has been edited out, just because it's incredibly tedious to watch all that if you're not participating. Zac was also cautioned to do his best job of making the riffs the same. In addition, we ran Zac through a prelim session where we watched for Zac acting differently on some pedals versus others. We did catch him leaning back slightly on some pedals versus others as he played, and coached him to stand the same, play the same, keep facial expressions the same, and not to try to see what was being tested. The pedal order being tested got fairly random too, as the method was A-B comparison, where the audience voted whether Pedal #1 or Pedal #2 was better. The (to them, at the time) less-good pedal was eliminated, and another pedal in the same group (like overdrive, or distortion)  was subbed in to compare to the winner of the first comparison. Which pedal was #1 and #2 got changed a lot, and it was difficult to follow if you were not personally the one pushing the buttons. I was in the audience and knew what was happening, and it was hard for me to follow in my head. So, yes, it is conceivable that the player's response to the sound changed the way he played. But we tried really hard to wash that out.

As for knob settings, yes, that has a huge effect on the sound. Not only do the settings have an effect, the fact is that pots typically have a +/-20% tolerance on their base value. A "100K" pot may be 80K to 120K and the maker would ship it out just fine. In that kind of setup, to get rid of that variation, we had a pre-shootout shootout where we and the studio crew sat in the audience position and voted on knob settings ahead of time. We did the A-B comparisons to deliberately make all of the pedals in a class (e.g. distortion or compression) sound the same. We reasoned that it's easy to make them sound different, just diddle the knobs. And if you want to "prove" that one is better than another, you can diddle the knobs to make it come out that way for some settings rather than others. But if you deliberately try to make them the same, it reduces the variations. One subtlety is that sheer output level was tested for the same. In two setups where even identical pedals are tested blind, people will tend to think of the slightly louder one as "clearer" or "cleaner" or "better". So we and the studio crew tried to wash out the level variations ahead of time as well.  Pedal settings was a group issue, where we tried (- blind!) to get them to same level, same sound as much as possible.

[R.G.]

Ahhhhh....that DVD.

That's a whole helluva lot of work, both for the player and the rater.

You should get Bob to talk to Gordon Logan at Vanderbilt (if he hasn't already).  Gord and I go wayyyyyy back to when he and his wife first arrived at McGill from University of Alberta.  Besides being an acknowledged world-class cognitive scientist with a distinguished career and very active laboratory, Gord is also a pretty decent guitar player, avid collector of vintage gear (I'm pretty sure that George Gruhn has bought some of his many pets from the proceeds of sales to Gordon), and from what he tells me still gigging intermittently.  Gord would hip Bob to the "perfect" blind test....not that these aren't well done already.

Yeah, that DVD.    I didn't realize that they would put it on YouTube. I'll get you a hard copy anyway.

I suspect that Bob might be interested in talking to Gordon. We do know that these are not theoretically perfect blind tests. The bottom line is that they were as close to a theoretically perfect blind test as we had time and money to do and still have some results that would be interesting to see, which was also important to us. Not much use in having a great set of results that no one will see because it's so boring to look at. As you're well away, controlling extraneous variation to isolate the thing you're testing is a huge effort in tests like this. Or it should be, if you're after an honest test.

[kurtlives]

Were the pedals hooked up in series or were they each totally isolated with a TB looper or something?

Great video though, loving it!


Hahahahahah the drummer in the Op-Amp vid! 

[Mark Hammer]

The tricky part of that comparison is that the pedals are deliberately set up to sound as identical as possible.  While that accomplishes the goal of assessing whether it is possible for 4-6 pedals to achieve the same single end-point (and obviously it is), the fact remains that any of those pedals is not necessarily going to consistently provide the same range of settings as any other. So, A and B might be identical twins for tone when controls x, y, and z are set for their mid-points, but B has more gain at max, or less treble push than A at max tone, and so on.

I'm certainly not trying to paint the shootout as biased in any manner.  It isn't.  Not in the playing, and not in the blind presentation.  But when Bob holds up the various pedals and says "This is the Fnortenizer, that costs around $250, and this is the Magic Hunga-dunga that sells for around $450", the listener/viewer/rater is then supposed to ask themselves "Well if they sound the same, why the hell am I spending all that money on pedal X?".  The answer to the question may well be because X has more range on all of its control parameters, and is capable of copping not only what pedal Y can do, but an awful lot more.  It's a credit to Bob's integrity that he doesn't judge or milk that, but at the same time, the naive viewer has to remember that they are listening to an assortment of pedals trying to achieve a singular sonic objective and not any broader range than that.  If the goal they aim for is yours too, then clearly if you can nail it in amore convenient and dependable package for $100 less, great.  As Prince says, "There's joy in repetition", but judgments about a pedal ought to be based on what it can do for you, in total.  Otherwise, pedals start to become the equivalent of cheap keyboards that have a few canned sounds and insist that you leave your creativity and sense of adventure in the closet at home.

Again, this is not a harsh criticism.  Rather, it is a challenge that pretty much any blind test faces: "How do I fairly demonstrate, in blind fashion, the range of things these two or more devices can do?  How do I demonstrate both areas of overlap AND nonoverlap without the individual rater monkeying around in hands-on fashion and identifying the device (and consequently being influenced by brand knowledge)?".  Certainly this shootout comes as close to ideal as any I've seen, and comes a LOT closer to the ideal than simply comparing youtube or tonefrenzy soundclips.  But there is still a lot missing.  I think it is useful to the whole industry to figure out ways to get around that conundrum.

Still kudos for even attempting, and for going public with it.

[ianmgull]

Great work. I especially like the Op Amp comparison. 

[R.G.]

I do agree - range matters. Also, a big part of the Mekkano-Anime DethBlaster may be how it sounds when all the knobs are full up/down/12/9/etc. and if a user is looking for that given sound, knowing that it can also be adjusted to some other tone setting may be completely extraneous. I didn't fully appreciate how hard controlling the variables is and I knew a smattering about testing going into it. I'm hoping that the response we get to this will let us do some more advanced testing to make public.

[analogmike]

Very cool, thanks for doing that!!

But you should have used a true bypass loop for each pedal, as buffers before or after a pedal definately change the tone of the adjacent pedals. For example, running two TS9s in a row, the 1st one will sound different than the 2nd one. (kurtlives was hinting at this). I saw him pressing the pedals' buttons directly so it seems that all were connected.

Also,  as Mark mentioned, the Klon is no better than a TS at those settings but the Klon at the clean boost setting does things no TS clone match.

have fun!

[earthtonesaudio]

Perhaps the sequel will put the various effects in "blank" boxes and leave the player to it.  The boxes could be labeled "volume, tone," etc, but otherwise no distinguishing markings.

[Mark Hammer]

It is an industry joke that when psychologists who specialize in perception (the study of psychophysics) do their doctoral research work, the "sample" usually consists of the Ph.D. student, their advisor and the student's girlfriend or boyfriend.  Why?  Because to do solid blind assessment of "Is example 1 more X than example 2?" over the full range of possible values and combinations requires just hundreds and hundreds and hundreds of training/testing trials, and those 3 people are the only ones that would put up with that crap without being paid.    This includes presenting the same thing as both example 1 and 2, in order to calculate false alarm rates.  So, setting a Route 808 as both pedal A and B and asking people "which one do you like better?".  So, like I say, even though this is an admirable attempt, compared to what those who study human perception as science (as opposed to market research) normally do, there's a big gap still to be bridged.

It is also worth noting that such blind testing really only starts to become pertinent as the market gets cluttered with both medium-to-large-scale production pedals of every sort of national provenance (all of which can be purchased on-line from the same we've-got-everything-you-could-ever-want retailers) that all really DO make a point of trying to farm the same little strip of land.  While it makes perfect sense now, this sort of exercise would have been rather silly to contemplate in 1979.

P.S.: I used to have a Mekkano-Anime DethBlaster  I changed the input cap to plastic film and the first few resistors to metal oxide.  Sounds MUCH better.  More Deth, more Blasting.  But that titanium chassis?.....ugh!!  Couldn't drill holes for extra toggles to save my life.  Just HAD to rehouse it in a nice black plastic Hammond box from Radio Shack.

[DougH]

I started falling asleep in the overdrive vid- they all sounded so similar. I'm not an overdrive guy I guess...

The distortion vid was fun. You could really hear the differences. Box of Rock as well as the Hyde sounded great.

My faves so far are the op amp and true bypass "mythbusters" though. They are both very effective. Hopefully a few more light bulbs will go on over people's heads when they see these.

Good stuff R.G.! Tell Bob to keep it up!

[Mark Hammer]

As RG points out earlier in this thread, there are component tolerances to deal with.  The mistake that anyone doing pedal-to-pedal comparisons makes far too much of the time is that, whether blind-rated or not, there can often be unit-to-unit variations that impact on the final tone.  So, if you like pedal A more than pedal B, is that because pedal A will always sound that way, relative to pedal B, or is this just a "lucky unit"?  Translating this into statistical terms, how much variance in rater evaluations is attributable to the model/brand, and how much is attributable to the individual unit?

Really, that is simply another version of what happens with on-line soundclips, or with tryouts at a friend's house or music store.  Occasionally one hears something under circumstances which favour A over B, or which misrepresent the overall average value/quality/tone of A or B.  The mistake is to confuse what is true of an exemplar with what is true of the signal/stimulus under ANY circumstance.  That is also one of the reasons why one normally controls for false-alarm/hit rates.  If I'm tired or distracted or whatever, I could easily say that A is less whatever than B, even though A and B are actually identical.  Similarly, if pedal A has some unique constellation of component values, relative to some other exemplar of pedal A, I might prefer A to B because A gets shown off in some manner which, though not always true of A, still makes it seems "better" than the exemplar of B.

Again, these are not criticisms, but simple realities in the measurement of human perception and peferences as a function of the objective physical realities being experienced.  It's a tough gig, whether you're a pedal manufacturer or a doctoral student.

[Ronsonic]

Interesting vid and yet more pedals presented for our examination.

Any shortcomings in the comparisons I can think of have already been addressed, buffers interacting, and some pedals perhaps not having their best attributes demonstrated and the inevitable trade off between rigorous A/B procedure and having the auditors gnaw their legs off to escape. But the comparison is what it is and it was produced by a manufacturer with products in play.

Thanks.

[Zen]

Very cool, indeed.  I too liked the op-amp vid. 

Kinda like being at the optometrists 

[Paul Marossy]

Cool set of videos. Can you set up something like this where people are playing thru different length guitar cables or a stompbox with one type of capacitor vs a different type of capacitor in a second otherwise identical circuit? I'd like to see how many people can really hear the differences between these things like they think they can. 

[Mark Hammer]

Some of that stuff is already done in the demo CD that comes with Dave Hunter's recent book on guitar effects pedals.

[Ice-9]

I totally enjoyed this shootout, what i noticed in both the overdrive and dist vids was that most people were more impressed with the more bassy sounds (including me). The one point  i would make about this is that in a full band situation the results could be more positive towards the trebly sounds.

I am massively impressed with the noise floor in the hyde pedal when it is in series with another pedal.

[Paul Marossy]

I totally enjoyed this shootout, what i noticed in both the overdrive and dist vids was that most people were more impressed with the more bassy sounds (including me). The one point  i would make about this is that in a full band situation the results could be more positive towards the trebly sounds.

I am massively impressed with the noise floor in the hyde pedal when it is in series with another pedal.

I regularly take the bassier, ballsier distortions over the thinner sounding ones every time.