That would be the implication only if Linus was the sole copyright owner.

As it is, no one is asked to assign their copyright when contributing code to the Linux kernel, and the amount of code still remaining in the kernel that Linus has personally written himself is a rather small portion.

So if Linus were to "take his ball and go home" as it were, the affected code could be replaced.

That said, the multiplicity of contributors that retain their copyright to their contributions is also why relicensing the kernel to GPL 3 is largely a non-starter, even if Linus weren't opposed to it (and he is).