Disappointing to see Hipp make that argument. It's trivially refuted.
Yes, all programming languages allow the programmer to write bugs. But languages very much vary in how many, and what kinds of bugs programmers write in practice. Saying "well, Rust doesn't eliminate all bugs" is attacking a straw man. If you want to argue that Rust isn't worth it, you need to convince me that C plus gcov results in fewer bugs in the important areas in practice than Rust (plus kcov [1] if you like) does. I think that's going to be pretty hard. (Especially if memory safety issues are the most important bug class you're concerned about: I think it's completely impossible for any C-based solution to compete with Rust here, regardless of how much tooling you add.)
Drawing an equivalence between undefined behavior and compiler bugs also doesn't make sense. Compiler bugs are way way less commonly encountered than undefined behavior in C. Also, they're qualitatively different: compiler bugs get fixed in new compiler versions, while UB is by design and doesn't get fixed.
> If you want to argue that Rust isn't worth it, you need to convince me that C plus gcov results in fewer bugs in the important areas in practice than Rust (plus kcov [1] if you like) does.
I don't have a dog in this fight, but I don't see how the burden of proof is on Hipp rather than the folks proposing the change. In other words, shouldn't the "rewrite it in Rust" folks have to prove that the cost of their proposed rewrite will be justified?
> In other words, shouldn't the "rewrite it in Rust" folks have to prove that the cost of their proposed rewrite will be justified?
Agreed, they do. But that it hasn't been proven to make things better doesn't mean it will make things worse. It just means that we don't know enough to say. The right way to answer the "would this software have fewer bugs if it were rewritten in Rust?" question requires a detailed look at what bugs the software has empirically encountered.
A rewrite automatically makes things worse because you start with no code.
I figure it's one of the signs of programmer maturity, that you start to look askance at rewrites. So tempting, yet so rarely even finished let alone better.
In the case of Rust I don't believe this is 100% true given C ABI compatibility. You could start rewriting in such a way that it is integrated with the existing code and slowly, but surely tease the C out of the system.
It would for the longest time be a C program with a metastasizing wart of Rust hung off the side, impossible to get into, impossible to work with, debugging hell, compilation hell. The distros would weep.
Sure, the primary burden of proof is on those proposing a change. However, any time you stand up and make an argument, the burden is on you to make sure it actually makes sense, and that goes for both sides.
This is getting a bit meta, but I disagree. It would be trivial to abuse in discussions.
A: Bash would be way better for SQLite, really!
B: But Bash is a terrible choice because X, Y, Z, ...
A: If you make those arguments, you have to prove them.
I'm not sure I take your point. If X, Y, and Z are cogent enough to be worth a rational response, B has done their job under my principle above. A is precisely the one I would want to yell at for using specious arguments.
Anything that is stated without proof can be refuted without proof. Why you think your opinion should be regarded at a higher standard than anybody else's?
...I don't. That's sort of my point. Both sides of a debate have the same responsibility to be reasonable, whatever that level of responsibility may be, whether it's a formal debate or just tossing ideas around.
I like to swing it the other way, you have to prove me that the newly proposed solution is not worth it. You can check out Amazon's policy about driving change. The "will not work" camp has to do the work to prove that something is not going to work. I think this approach yields to better results in practice.
'Saying "well, Rust doesn't eliminate all bugs" is attacking a straw man.'
This is itself a straw man.
Follow the link to Mr. Hipp's comments and read them. He did not say this.
That a programmer who has produced such high-quality and rigorously tested software as sqlite should be portrayed as either cavalier or naive about software quality is something I find profoundly mis-guided.
> This is itself a straw man. Follow the link to Mr. Hipp's comments and read them. He did not say this.
"Rust doesn't eliminate all bugs" is a rephrased version of "Some well-formed rust programs will generate machine code that behaves differently from what the programmer expected."
> That a programmer who has produced such high-quality and rigorously tested software as sqlite should be portrayed as either cavalier or naive about software quality is something I find profoundly mis-guided.
I don't think he's cavalier or naive about software quality! SQLite's quality speaks for itself, and he is completely correct about what you need to do to achieve the level of correctness that SQLite has. He knows a lot more than I do about the rigor needed to ensure that amount of software quality. I do think, though:
1. The amount of testing that SQLite undergoes is not economically feasible for most software.
2. Rust's testing tools would allow for the same level of code quality in the absence of evidence otherwise. (The example he cited, code coverage, is wrong, as kcov is available for Rust.)
3. There is value in static analysis above and beyond the value of testing (also vice versa), because testing only reveals bugs that manifest themselves in inputs available in the test suite. This is true regardless of the code coverage of the tests.
In other words, I think Hipp's argument would make sense if it were something along the lines of: "I couldn't test a Rust SQLite as well as I can test the existing SQLite because kcov is missing features X, Y, and Z, and the value I get from the static analysis is less than the value I get from those code coverage tools, since we've found lots of bugs from those code coverage features and haven't found as many memory safety/data race bugs". That'd be totally valid. But as a blanket statement that Rust would make things worse, it doesn't make sense to me.
I was not aware of kcov or its basis bcov. Thanks for pointing those out.
However, a quick glance at the bcov source code leads me to believe that it only does source-line coverage, not branch coverage. So, unless my quick reading of bcov sources is mistaken, I couldn't test a Rust SQLite as well as I can test the existing SQLite because kcov/bcov is missing the ability to measure coverage of individual machine-code branch instructions, and the value I get from static analysis is much less than the value I get from branch-coverage testing tools.
That's not being pedantic, btw. The difference between source-line coverage and machine-code branch coverage is huge. The latter really is necessary.
Thanks, that's really great feedback. We should invest in making use of llvm-cov. It should dovetail nicely with the MIR work (which is now to the point where it can build the Rust compiler), since the language-specific IR it exposes is especially suited for quick development of instrumentation passes.
I don't think he's making the argument you think he's making. This is clearer in his later comments in the thread, in which he says that UB is much scarier in systems like Fossil.
His argument is that achieving the level of quality that SQLite has requires verification (in a broad sense) after the compiler, and that's what he does. If you consider the goal to be producing quality-assured binaries, then you can treat UB, compiler bugs, and many other things as falling in a similar category, which are almost certainly eliminated by an MC/DC test suite.
As you say, this isn't feasible for almost any software (as John said in the blog post, SQLite is the only program he knows of that has MC/DC testing when not required by law). But it does mean that rewriting SQLite in Rust wouldn't provide as much value as rewriting many other things where the binaries do not have such guarantees.
> His argument is that achieving the level of quality that SQLite has requires verification (in a broad sense) after the compiler, and that's what he does.
> If you consider the goal to be producing quality-assured binaries, then you can treat UB, compiler bugs, and many other things as falling in a similar category, which are almost certainly eliminated by an MC/DC test suite.
I don't think that they're eliminated because dynamic testing can't eliminate everything—it only finds bugs given its test inputs.
Moreover, though, I'm also skeptical of the claim that binaries are all that matter. Lots of software projects (for example, Firefox) import SQLite as source into the project instead of using the binaries. They upgrade their compilers without running the SQLite test suite to catch regressions. (Firefox might, but I'm sure lots of other projects using SQLite from source don't.)
> But it does mean that rewriting SQLite in Rust wouldn't provide as much value as rewriting many other things where the binaries do not have such guarantees.
Sure; static analysis is more useful in systems that aren't as well dynamically tested. But static analysis still has value.
> I don't think that they're eliminated because dynamic testing can't eliminate everything—it only finds bugs given its test inputs.
I can't think of an example of UB-exploit in the compiler that wouldn't result in different branch behavior, and thus, I think, in failure of MC/DC. But I may simply be insufficiently imaginative.
John makes the point about source code in a follow-up comment, and I agree, but I also see Hipps' perspective.
> I can't think of an example of UB-exploit in the compiler that wouldn't result in different branch behavior,
What about Implementation-defined Behavior? This is (I think!) technically a subset of Undefined Behavior and it permits such things as setting values[1] to arbitrary (but well-defined!) values on various operations, such as "excessive" left shifts. What I'm saying is that a compiler is permitted to substitute IB for UB and still be conforming. So it could start to do strange things to arithmetic, etc. Does that make sense as an example of what you're thinking of?
EDIT: [1] I obviously meant memory locations... as referred to by "variables" which aren't really variables, but are really binders/aliases. But here we are.
I think you're confused about implementation-defined and undefined behavior. The former is not a subset of the latter, but excessive left shifts are UB.
> "Rust doesn't eliminate all bugs" is a rephrased version of "Some well-formed rust programs will generate machine code that behaves differently from what the programmer expected."
I think that sounded more like "Many of the programs which are UB in C are also UB in Rust, even though not (yet) specified as such."
> I think that sounded more like "Many of the programs which are UB in C are also UB in Rust, even though not (yet) specified as such."
Well, this seems either (a) false or (b) uninteresting to me. It's false because C and Rust have different semantics, and Rust rules out lots of programs that C doesn't. It's uninteresting because if the point is that Rust has accidental undefined behavior due to compiler bugs (and it does), then that UB is so rarely hit that it doesn't matter nearly as much in practice as the UB in C.
CPUs have bugs, too. Do we consider Java an unsafe language because of rowhammer?
Help me out here, I don't know much about Rust. Which bad programs does Rust rule out?
I know about the borrow checker, but I don't think ownership bugs is a type of bug that Mr. Hipp frequently produces. It's a program design issue -- not something you think about at every single line you write. A well-designed program does not do many ownership transfers.
As someone else noted, out-of-bounds errors are sadly a pain to debug in C. For testing code, C has at least valgrind (and certainly many less well-known tools). For production code, you might not want dynamic index checking. It would invalidate performance arguments for Rust.
I think the real nasty cases of UB in C, which frequently occur as not statically refutable, are signed arithmetic overflow and shifts. As with out-of-bounds errors, I don't think Rust has a better story here -- only better built-in tooling.
> I know about the borrow checker, but I don't think ownership bugs is a type of bug that Mr. Hipp frequently produces.
The borrow checker eliminates use-after-free. And use-after-free is one of the most common types of vulnerability exploited in practice today, if not the single most common. (For evidence, look at reports about Pwn2Own.)
Index checking is quite cheap for most programs, and LLVM is good at eliminating redundant checks. As a useful comparison in another languages, Chromium compiles with bounds checks on std::vector by default (the [] operator, not the at() method which is required to be checked.) Actually, there are a lot of issues with Rust's current machine code output that I think slow it down, but index checking isn't one of them.
I don't think you can count use-after-free as exploitable. Sure, if you have them (and I think those cases can largely be ruled out by good design), it leads to crashes. But for memory type exploits you need control over the value in it. Not a security specialist but I'm not aware of common attacks besides overflowing buffers.
In hot paths (like codecs, compression algorithms...) I'm sure you never want bounds checking. It can be optimized away for sequential loops of course, but not so easily in the case of data lookups.
"While it is technically feasible for the freed memory to be re-allocated and for an attacker to use this reallocation to launch a buffer overflow attack, we are unaware of any exploits based on this type of attack."
I have no idea what that article is talking about. Use-after-free exploits have been extremely common for years. Like I said, look at Pwn2Own (which requires submitting an actual working exploit): https://www.google.com/search?q=pwn2own+use-after-free
Gah, it seems this one isn't good. I trust OWASP and skimmed, it seems that I agree with sibling commentors that this is more dangerous than is presented here.
> If the requested behaviour is unspecified (just `a+b`), overflows cause runtime panic.
As far as I know most machines can't trap on signed integer overflow. Which means having runtime panic is not practical. The page you linked says "no checking by default for optimized builds".
Ok, I simplified this too much. For full details of what happens you'll have to read the docs.
The non-release mode panics. Release mode with debug asserts turned on panics. Release mode with forced overflow checks panics. Release mode with no special options results in the same result as a wrapping operation.
1. The amount of testing that SQLite undergoes is not economically feasible for most software.
But the argument isn't about "most software" - it's about SQLite in specific. And that's the crux of the issue. Saying that he believes it would be counterproductive to rewrite SQLite in Rust at this time is not saying that that would be true of all or most or some other programs.
> Saying "well, Rust doesn't eliminate all bugs" is attacking a straw man.
Yes, it's very much akin to saying "well, a seat belt won't eliminate all deaths". I would understand an argument of "I'm not prepared to rewrite the software at this time" or "I'm not familiar enough with replacement X to assess whether it's a good choice", but at some point more people will have to acknowledge C's failings with more than lip-service. That doesn't mean Rust has to be the solution/replacement, but something does.
Nor does Rust let you trigger "dangerous" UB in non-unsafe code. This isn't going to change, ever, so the argument that "give it time, Rust will soon be chock-full of UB" is moot too.
Yeah, it's one of Rust's strongest features, as it allows progressive migration of native code, and also allows Rust to serve as a fast/low-overhead extension language for Ruby and Python (etc.).
Yes, all programming languages allow the programmer to write bugs. But languages very much vary in how many, and what kinds of bugs programmers write in practice. Saying "well, Rust doesn't eliminate all bugs" is attacking a straw man. If you want to argue that Rust isn't worth it, you need to convince me that C plus gcov results in fewer bugs in the important areas in practice than Rust (plus kcov [1] if you like) does. I think that's going to be pretty hard. (Especially if memory safety issues are the most important bug class you're concerned about: I think it's completely impossible for any C-based solution to compete with Rust here, regardless of how much tooling you add.)
Drawing an equivalence between undefined behavior and compiler bugs also doesn't make sense. Compiler bugs are way way less commonly encountered than undefined behavior in C. Also, they're qualitatively different: compiler bugs get fixed in new compiler versions, while UB is by design and doesn't get fixed.
[1]: https://users.rust-lang.org/t/tutorial-how-to-collect-test-c...