One of the good things about a controller aside from the smooth, smooth goodness, is it protects the motor, wiring & battery from excessively big amps.
Of course if you instrumented the thing with an ammeter and did a little bit of testing to learn what gear/speed/grade combos work without heating up the components too much, AND you keep an eye on the meter, a simple switch would be OK.
Then again, you could also just put a properly sized fuse or circuit breaker in there with the switch. That'd protect the motor without a controller. (Should be a fuse in there anyway so if the controller fails, you still have motor/circuit protection).