Could be a number of things. Do you have power brakes? If so, have someone in the car try to duplicate the problem while you observe the engine compartment. My bet is that it's a vacuum leak created either by the booster itself or by the check valve attached to the booster. Make sure there is some play in the vacuum line to that valve so that as the engine torques it doesn't pull the on the valve and create a vacuum leak.