Our editorial review teams assign formal review ratings to all products reviewed in our product review program. In addition, we honor exceptional products with two review awards: Highly Recommended and Recommended. In addition, we offer a non-review award called the Guide's Gear Award.
Rating products that you've reviewed is an interesting exercise.
It's also an inexact science, subject to bias, prone to controversy, an equivalent to an invitation to put your head on a chopping block.
It's this last notion - that of sticking your neck out - that motivates us to not just rate the gear we review, but to explain why we rate it as such.
We see it as an opportunity to keep us accountable to our readership. That's why our ratings, starting July 2007, will be peer reviewed both internally by at least two members of our staff (not including the author) and one or more outside peer reviewers. Our outside peer reviews are passionate members who speak on behalf of the membership to make sure we're held accountable. Our peer reviewers span the breadth of wilderness experience. We're confident they can set us straight.
When we rate gear, we rate it in context of what is currently available on the commercial market. Sure, gear can stand on its own merits, but if it isn't unique, innovative, or otherwise able to distinguish itself from the vast cesspool of "average" products that are currently available, then so what? We know, this sounds a little harsh. But it's our friendly way of encouraging outdoor gear manufacturers to "put up..." - for the sake of our readers, and their limited budgets.
The highest possible rating for a reviewed product. The minimum requirements to qualify for this rating are: (1) exceptional quality in materials and workmanship, (2) outstanding field performance based on sound design principles, (3) a performance-to-weight ratio reflects the ideals of carrying out the Backpacking Light Mission: "to promote multi-day, self-sufficient ("backpackable"), backcountry travel in lightweight style," and (4) the product offer significant advantages over the next best competitive products. To earn a Highly Recommend Rating: (a) the review author(s) must recommend the rating, (b) the independent peer review panel assigned to review the article must recommend the rating as well as consider their own experience with the product, (c) the Editorial Board must review the article and must recommend the rating as well as consider their own experience with the product, and (e) the Editor-in-Chief must recommend the rating.
Solid performing products that provide good value for the money, quality materials and workmanship, and acceptable performance that should meet or exceed consumer expectations. Minor improvements in design or materials may be required to allow this product to be considered optimal for lightweight backpacking and thus warrant a rating of Highly Recommended.
Products that should meet consumer expectations, but against which other products of similar performance should also be considered.
Products in categories where better performance:weight and performance:value can be realized in several other competing products.
Products with performance:weight and/or performance:value ratios that would not meet the expectations of most consumers and are neither worth the weight nor the money for the vast majority of users.
Backpacking Light makes no effort to remove "bias" in reviews. Indeed, we hope not only that the reviewer's personality and experience come through as part of the review, but also that the reviewer communicates positive bias for products that offer unique capabilities, durability, quality manufacturing, and high performance - at a light weight, of course. Likewise, we expect products that do not meet acceptable standards in any one of these areas will be reviewed with some degree of negative bias.
Conflicts of interest, on the other hand, are something we take very seriously. Backpacking Light has no financial interests with any manufacturer beyond what is required to purchase a product that we might review*). Here is our conflict of interest policy in the context of product reviews:
We are ultralight backpackers who are part of a small community of like minded individuals that includes users, media, and manufacturers, and we have relationships (both professional and personal) with the companies that make products we review. Therefore, we adhere to strict guidelines to minimize conflicts of interest, fully realizing that they are not 100% avoidable when personal relationships are involved. We have strict policy of not granting favors ("positive" reviews) to friends, and not reviewing for companies with whom we have any financial relationship beyond that which is normally required to acquire a review product*.
* Backpacking Light acquires gear for review in one of four ways:
The four options outlined above are ordered in the order of increasing conflict of interest probability (lowest to highest) and the method we used to acquire the gear will always be disclosed.