Tight budgets, technological advances, and the impulse to experiment are leading some producers to record "not by the book." Does it work?