The problem is that a drum hit is essentially white noise added to the signal. If it is of high enough volume, then it really is impossible to pick out what is "signal" and what is "noise". Your separate track of drums is not going to help that much, because percussion is chaotic, that is, the signal will look different every time you play it, and thus techniques which involve canceling out the signal in the time domain aren't going to work, especially since those techniques are sensitive to exact timing. As mentioned in another post, spectral based subtraction is also not going to help either, because percussion is generally spread spectrum: It covers all frequencies, so you can't simply remove the frequencies it's located in and be left with something decent.
My best guess would be to remove the transient sections by hand, and then simply interpolate the removed regions in the spectral domain, using a spectral editor. The assumption that this technique works under is that the everything else in the recording besides the drums is varying slowly, i.e. lacking any other transient element at the time of the drum hits. Of course, this is probably not true. But my guess is you can't get any better.
Oh, also one more possibility, is you can try using the transient-steady state separation in aubio. Then you can spectrally subtract the second recording from just the transient part. That probably should give you similar results to the above technique though.