This article is spot on. Verifying hardware is a much more complex task than designing it. I've spent ~1.5 years on verifying custom cpu core re-implementation (next gen) that was almost binary compatible with previous gen, we had own gcc backend and LOTS of legacy programs written in assembly. And even then it was a nightmare. Even after tons of time spent on covering most obscure cases, our FPGA platform, running our multi-task FW, would often crash: cue in hours of back-stepping FPGA trace dumps, where you have a total recording of program counter, load-store IF, internal registers for aroung 100k clocks and you have to trace back the execution and see where the fault occurs. Usually it would turn out to be some weirdly obscure combination of recently-fired interrupt, some pipeline stall with execution of some weird instruction. fixed in 3 seconds when found. goddamn that was exciting.
I wonder if there are any people here on HN that work on advanced HW verification techniques that are currently ongoing in the industy?
Yep — and I’ve already said too much. I get upset how secretive this industry is; I truly believe it’s at least 30 years behind the curve (and falling further) compared to SW. I also firmly believe it is the EDA vendors who sell this mindset because $$$.
As a hardware designer (well, mostly a hardware verification engineer), I feel that this kind of comment is a pure misconception from software people that gets echoed time after time in HN.
I generally see a lot of more effort and good quality tools in hardware verification than in software validation. The hardware design and verification industry is not really 30 years behind the curve compared to software development.
Hardware and software are two very different problems with their own set of constraints. It's actually more often that you see software barely tested being deployed in the wild to millions of users rather than hardware. Hardware bugs tend to get more attention because you can't really deploy a patch to fix the issue.
Having worked with both HDLs and software tooling, I don't think it's too far off the mark. HDL tooling is terrible software: it's hard to use, buggy, expensive, opaque, and the support is terrible unless you're a huge customer. Hardware verification and development works still through sheer effort, and because there's a strong business incentive to get things right the first time due to the huge costs of a respin. With better tools this could be done faster and cheaper with similar reliability, but perhaps there isn't enough to be gained to make it worth the effort (and either way I don't think the companies currently working on the tooling are capable of making anything else: this would need disruption to really make a difference, and the barriers to entry are huge).
Hardware design flows are complex and do have bugs, but not in the way it sounds. In general HDL synthesis produces a design that matches what the user wrote, so a logic bug in a synthesis engine is rare. The bugs exist in the 'other stuff', like trying to convince the tool to be efficient about how it created the design, or smart about how it's trying to close timing. Getting a working though the tools design is easy, getting an optimal design is hard. HDLs themselves have issues, but there is a wealth of quality checking tools built around them that mitigate them, and indeed mitigate/flag poor coding from inexperienced designers (to a degree). What's interesting is FPGA tools, as much as they are derieded, are much better than their Asic counterparts in terms of functionality and user friendliness.
We get decent support from our main tooling providers. We do spend a chunk on licenses but I don't think we'd be considered a huge customer. And yes, the software has bugs (with expensive consequences) so good support is very important.
> I feel that this kind of comment is a pure misconception from software people that gets echoed time after time in HN.
but this is NOT a misconception. Show them actual tools that constantly crash and lag on you, that demand checking out licence for a runtime which costs 5-100k/Year, show them environment in which we have to write our code - they will laugh at you. This was exactly the state of the software industry 30 years ago and this industry is long past that stage.
Those are some very empty counter arguments but I will try to answer them anyway.
EDA tools are expensive, yes. But you are not just paying for the tools, you are paying for all the engineering support being those tools. Bugs and problems happen in any development tool and those companies are ready to pick up the phone at any time of the day to help you with anything you might face, whether it is a problem with their tool or a problem with their setup, to the point that they can deploy a new version of their software on the spot just for you.
And they have actual engineers on the other end of the line who know the product like the palm of their hand.
While this might sound superficial, it is extremely important when you are racing towards a $100M dollar deadline for your project.
The level of quirkiness and bugginess in the tools is on another level to most software tools. You shouldn't need access to the engineers who built it in order to make it work, this basically increases costs and barrier to entry massively.
Many of the tools are bleeding edge, developing new features to support a new process node at the same time as the new process node is being developed.
Certainly on the simulator/synthesis side you can still encounter straight-forward bugs in SystemVerilog features that were standardized over a decade ago (or find they're not implemented at all) or even simple parsing bugs (the kinds of bugs that would be trivial to find with some decent randomised testing of the parser).
Things being bleeding edge and fast evolving in certainly true for some EDA tools or parts of them but there's lots of more bread and butter stuff that never feels quite right.
> The level of quirkiness and bugginess in the tools is on another level to most software tools. You shouldn't need access to the engineers who built it in order to make it work, this basically increases costs and barrier to entry massively.
I do not know what SW tools are you using but i'm still looking forward for a SW tool that doesn"t suck.
I believe software developers are split between IntelliJ, Eclipse, Visual Studio and VS Code. They do work very well compared to what I've seen in the embedded world.
Ignoring text editors here, because text editors are not programming environments (no debugger, no compiler, no source control, etc...)
> EDA tools are expensive, yes. But you are not just paying for the tools, you are paying for all the engineering support being those tools.
So I can buy the tool without the support contract?
> Bugs and problems happen in any development tool [...]
So it's automatically a wash? Why bother trying to improve things if there will always be bugs, right?
> [...] whether it is a problem with their tool or a problem with their setup, to the point that they can deploy a new version of their software on the spot just for you.
I'm not impressed. This is a kludge.
> And they have actual engineers on the other end of the line who know the product like the palm of their hand.
> So I can buy the tool without the support contract?
Probably not but it's not something you would like to. Cutting corners is a really bad idea when it comes to hardware development because once it is out there, there is not going back.
Having access to a professional engineering work force that knows how the tool works and know it will ready and interpret the Verilog and process it can make a difference between a 5 minutes delay and 2 week delay in the project.
> So it's automatically a wash? Why bother trying to improve things if there will always be bugs, right?
You are distorting my words. I'm not saying that we should be complacent and just accept incompetence.
>> [...] whether it is a problem with their tool or a problem with their setup, to the point that they can deploy a new version of their software on the spot just for you.
> I'm not impressed. This is a kludge.
So how would you handle it? Tell the customer you'd love to fix their problem but they have to wait for the next formal release?
I mean, this sounds a lot like software 20 years ago.
You can still get proper support for, say, IntelliJ if you want it, but tools and ecosystem have improved so much that it’s just not as important as it used to be.
To be clear, I’m not claiming that hardware development is the same as software development. It’s not. It has different problems. But the tooling _does_ seem well behind.
> You can still get proper support for, say, IntelliJ if you want it, but tools and ecosystem have improved so much that it’s just not as important as it used to be.
How many people are doing hardware verification work vs. cranking out webapps with IntelliJ?
JetBrains can invest a lot in software ergonomics because they're basically selling mass-market software. If you have sophisticated but niche software, your probably going to invest your engineering effort into the it's capability rather than its ergonomics.
I mean, to take one example of a company in the space: Xilinx has operating income of about 750mn per year. Jetbrains has operating income of about 80mn per year on revenues of 250mn.
Given that I’d suggest that the reason that Xilinx’s development products have such a poor user experience vs IntelliJ is NOT that Xilinx is impoverished, but rather that the market has extremely low expectations so they don’t feel the need to spend on fixing them.
This isn’t a problem unique to hardware-land; lots of verticals have incumbents who produce low quality products on large profits because, frankly, the competition is just as bad and there’s a high barrier to entry. I’m not sure what would fix this for hardware-land, though. Possibly more pressure from open source; that’s largely what did it for software development (though even before that, companies like Borland put more than the minimum effort in).
Xilinx are not just writing software, they also must use R&D to develop the FPGA hardware, dev kits, IP, etc, so it's a very different scale to JetBrains' focus. Also in terms of Sofware, the really challenging aspects are under the hood, the place and route algorithms, the static timing engine, the system verilog simulator (which is itself a huge undertaking), the hardware debug cores, the HLS SW that compiles C to accelerated HW, then SDK for embedded processors, oh.. And of course... Finally the GUI/IDE. With all that's going on underneath, Vivado does a decent job of presenting it all in a logical way. I'm blown away that I can see a timing failure on any of a million nets and in a click cros probe it to a line of verilog.
As for open source, SystemVerilog is an open standard, yet can you show me an open source simulator that competes with the free one that comes with Vivado?
On the one hand Vivado is not entirely terrible considering that it can get some work done, but on the other hand it's beyond me why it's lacking the most basic needs of software development.
Zero support for source control? SoC have no examples, why can't they systematically provide an hello world example (blink a LED) with their development boards? The IDE itself has close to zero doc and there is no help on the internet, it takes 3 freaking days for an experienced developer to figure out how to create a project.
Have you used software verification tools? They're almost universally terrible. In many cases incomprehensibly so. Many are free up front, but you pay for it one way or another. The level of sophistication and workflow integration of software verification tools, compared to hardware verification tools, also seems to lag significantly.
I don't doubt that the hardware verification tools are expensive, buggy, and probably have terrible UX. However, all those things are also true of software verification tools, but where license cost is replaced by esoteria and incompleteness cost.
Valgrind is not in the class of verification tools I was referencing. How does it correlate to VHDL/Verilog in this context? I would expect things like MBD tools, model checkers, and proof assistants, and program extractors to be the software pipeline siblings to the kinds of tools that VHDL/Verilog integrate with.
It's a tool to verify the software runs correctly and doesn't double free, write over the heap and a while bunch of failure. Unless you never run simulation and always trust the router?
Not everything gets verified 100% in HW, otherwise we wouldn't have silicon errata. It's all a question of how deep you want to go in verification and what your costs are for screwing it up.
I don't agree. I agree that if compare random software project x to a random hardware project y that the hardware project will do a lot more verification.
But that doesn't really compare to state of the art of what's possible in software. Property based testing, symbolic execution etc all have incredible power that afaik is not available in the hardware world. I'm just a hobbyist who uses FPGA's but it's way way easier for me to ensure a function has some properties compared to a VHDL module or whatever.
Yes, hardware must be right first time. Proving that is difficult, so logic equivalence, formal proof, code coverage are all standard. What's optional in many software QA flows would be absolutely required in HW flows. Proof of functionality is paramount
this industry is a Borland's wet dream - everyone uses crappy expensive software that reeks of 90s, the libraries are closed source with barely exposed interfaces, the language is a mess (SV), etc. Yes, you are right, as long as its making them huge money nothing will change. And nothing can change, since the field is a frickin mine field of patents, you cannot innovate. Also, why you've said too much? What have you said? :)
Please do not take HN threads on generic ideological tangents. It leads to generic flamewar hell. Regardless of ideological preference, none of us want to end up there.
If the low quality of the software impedes productivity generally (and, being honest, it MUST) then that’s not well-functioning. It’s more a common depressing failure of a capitalist market (and a lot of software verticals suffer from it).
It’s a space with a high barrier to entry, which is probably the biggest problem. Software engineering tools got good, and affordable, due to a combination of lowish barrier to entry and open source.
capitalism is at its best when optimizing industrial production (manufacturing materials)
knowledge creation is not quite an industrial process.
I would highlight the importance of open source as a key component for better tooling.
But knowledge does not quite play well with capitalist markets. there must be a better way, but finding it requires revising several 'fundamental' assumptions about how this (market) society works.
Not only that, but the sheer complacency of people within the industry when it comes to better tools blows my mind!
Our VDI development environment runs an OS from 2 decades ago. It's ugly and the latency is mildly infuriating. And yet, my co-workers are totally OK with it all.
I've been working for the last four years as a consultant for a chip design company on one of their internal design tools. The complexity is absolutely mind-boggling. And seeing how the sausage is made doesn't make me any less queasy about it.
I have been there, it's horrible. Often traces are in binary, and I spent time making them more readable, to take off the intellectual load of just decoding what you see from the actual debugging.
These days I design Verilog for FPGAs without a verification department. I've learned to code defensively, to use what I know is robust without creating complex corner cases. This often involves clear handshaking and dataflow. Anyway, it's more fun than fixing bugs deep in the weeds of a complex system.
I work in processor verification and my employer spends a great deal of money on it. Like "thechao" I am not allowed to talk about this, alas. A good place to look at some of what ARM do is:
CPUs are arguably the most crucial mechanical invention in history and yet the practical art of making them actually work right is shrouded in secrecy in service of greed.
On Windows/MacOS, the communication between the operating system and external devices (drivers) are completely shrouded in mystery. Theoretically the Nvidia graphics driver could be intercepting all your ethernet packets and sending them to the US government. (I have no evidence of such a attack, but it is theoretically possible) Drivers are extremely privileged, and are also a major gateway towards compromising computer systems)
If you don't think the GPU drivers are complex, then look at the perf difference between nouveau and the nvidia proprietary driver. Those guys are also working on the bleeding edge of optimization. Sure you could write a working driver in short order, but its not going to perform the same.
I'm not saying that they are not complex, that's why I chose them. But you have the source code for them, you don't have to write them from scratch. Some of the optimization techniques, even if not documented, can probably be studied from the change history. Of course, it will take a few years to grow new experts without knowledge passing from the old ones, but by and large the logic is there in the code.
From what I understand, the same can't be said of real microprocessor design - the complexities of aligning logical requirements with EE complexities, analog responses, and manufacturing process limitations are not at all captured in the 'code', not at the same level. We can hope that they are documented, at least to some extent, but we all know the general priority of internal documentation against other concerns, and the difficulty of documenting design processes.
This may be true in some cases, but from my experience (nearly 20 years in custom IC design) good design is still more difficult than verification. Verification seems to have an element of over complication. I've seen simple, totally independent sub components take many man weeks of verification, not because it needed to, but because the verification approach, even for simple things can be overly complex. So sometimes the reason verification can take more resources is not because its harder, its because the approach for that particular problem was not efficient. Of course like anything there are extremes on bot sides - but good design must be both correct, efficient, and meet timing/power goals, while verification must just be correct.
Brings me back to 1995 when I was working on the C-Cube CL4010 MPEG-2 encoder processor. A fully custom 32-bit (with some 36-bit registers) RISC architecture.
> This article is spot on. Verifying hardware is a much more complex task than designing it.
Are we sure about that? They start out talking about how hard it is to write correct state machines in Verilog, and they follow that with a lot of skepticism about RISC-V processor designs. But most RISC-V designs use Chisel, not Verilog. Chisel is much higher level than Verilog and using it should also make it quite a bit easier to prevent many design-level errors.
It's not the same thing. Chisel will help you do things like write more reusable RTL, because the language itself is more expressive -- so the potential level of abstraction is much higher than what you can achieve in Verilog. You can do more flexible kinds of parameterization, for example. And doing that can help avoid certain kinds of bugs, for sure. (And it also means you simply need less RTL to describe the same thing, because re-use is so much higher.)
But Chisel will not help you when you have some errata like "the BRAMs on these devices take 67 cycles to initialize after coming out of reset" tucked away in a manual somewhere that you forgot to read. That's just an FPGA example -- you can run a hundred simulations and your design will still fail immediately in real hardware because of such things. (Even "cycle accurate simulations" from your vendor may not account for these things.) You're going to encounter a lot of problems like this, before you even move to ASIC level flows.
Ultimately the hardware industry does need much better RTLs, because the current ones mostly suck from a language POV in a huge number of ways. But verification is still a much bigger problem in general no matter what RTL you've chosen.
With a better eco-system, the (temporal) specification (such as "take 67 cycles to initialize after coming out of reset") such be given formally, so a tool can ensure that it's not forgotten. In principle this is possible.
much better RTLs
Have you got any concrete ideas and proposals? I'm asking because (A) I agree with you, and (B) I'm very much in a position where I can influence research in this direction.
> (B) I'm very much in a position where I can influence research in this direction.
Could you elaborate on what you do?
> Have you got any concrete ideas and proposals?
My recommendation is: try to build something complex, or take something from open-source that is very complex and extend it (test it, verify it, improve performance, etc.). Notice the problems you encounter, and then try to solve those problems.
I could give you a laundry list of pain points myself (why do I have to pay money for lint, and why does lint still suck???), but I think the best research is done by people trying to solve problem X, but along the way ended up having to solve Y and Z just to get to X.
I've spoken to a lot of processor designers, usually from an EE background, and they prefer Verilog: "we tried more high-level languages in the 1980s ... it was a failure" is something I've heard more often that than I wish I had.
I think there is an interesting C. P. Snow-like Two Cultures thing at play between SW and HW people. I think the latter don't quite get the abstraction power of modern programming languages, they don't understand that syntax, types, modularity matter. They think PLs are something like C and C++: a mess that we simply accept and get on with life. The former really underestimate the extreme scale and complexities involved in processor design and verification. Never the twain shall meet?
laundry list of pain points
Think bigger: what would you do if you have a team of 3 engineers, or 30 engineers to work on a better tooling ecosystem?
I don't think that is the point of the article. Even a state machine that is very well written can be wrong if it's fed with the wrong input. And this is the core of the problem with hardware development.
You want to make sure that the hardware you are designing never ends up in a situation that you didn't foresee. So you want to make sure that during hardware verification you go through every single possible combination of inputs and you want to make sure you exercise every single possible outcome.
The quality of the hardware is not about the language you write your hardware in. It's about the tools, the methodology and the infrastructure you use to exhaustively test your hardware.
And this is the point of the article, current processors are so complex and do so many things that the hardware verification suffers from an explosion of states to exercise.
Chisel is not a magic bullet and does not make you suddenly write bug-free hardware. Its the same logic applied when people compare coding in x vs Haskell. The latter does not suddenly grant you superpowers to write bug-free software.
>Chisel is much higher level than Verilog
have to disagree here. the level of abstraction is exactly the same in chisel as in verilog, its still RTL level. High-level abstraction in hardware is a long-chased holy grail that is revived in the industry every now-and-then and then immdediately dismissed and forgotten, when faced with real-world industry challenges.
No is claiming that Chisel code is always bug free. Just like no is claiming Haskell code is bug free. However the point they are making is that these languages are way better in statically catching a whole host of bugs. They also allow certain properties to be expressed in the type system which is essentially having a proof that that invariant holds now matter what.
It sounds like hubris, y'all are trying to bat above your league.
Metaphorically, it's like the rush to build auto-autos (self-driving cars): the problem is too hard. If they had started by trying to make a self-driving golf cart Elaine Herzberg might still be alive.
If "verifying hardware is a much more complex task than designing it" (and I don't doubt it) then that is the limiting factor. (Or should be IMO. The liberties taken by software cowboys are bad enough w/o the hardware getting all squirrelly too.)
Yeah I agree. In my opinion the consequences are showing now that all processor designs are suspect and likely incompatible with the very real threat models we face day to day when we browse the untrusted web.
Its probably better stated, as its not hard to create a new processor, anymore than it is to create a toy OS.
The hard part is creating something that is competitive with top of the line commercial processors that have thousands of man years of R&D poured into them. Its not just verification, but the huge effort that goes into eaking out another couple percent on something like a branch predictor, or optimizing some "edge case" that turns out to be a significant portion of a benchmark if its not done correctly. Then there are all the general optimizations that give you a 10% uplift here and there. Worse, yet if you go with something that doesn't have a large installed software base (x86/arm/power?) because your going to be spending crazy amounts of effort doing compiler+application optimizations as well.
If anyone is interested in this sort of thing, I would highly recommend checking out the documentary Rise of the Centaur. It’s about a company that I previously hadn’t heard of who was making an x86 compatible CPU based in Austin TX.
They show a lot of the verification process throughout the film, including an exciting moment when the chip boots Windows for the first time.
For our software friends wondering what’s so special about processors: they are the most parallel state machines designs out there. Most other state machines have fairly well defined and narrow inputs and outputs. For performance reasons, a cpu pipeline is the biggest collection of state machines interacting with each other directly.
Therefore most formal methods blow up on cpu designs and random coverage is really hard to define and even harder to reach.
Anyone working with this stuff: what are some textbooks on verification etc. Preferably a bit of theorem proving too but I'm not sure if that's in the same area of study?
I'm coming from a physics background so I never really know where to start when I inevitably start looking this stuff up at 4am.
I’ve read the Universal Verification Methodology Primer by Ray Salemi.
The goal of UVM is to try to catch all the edge cases in simulation before you fab. The basic idea is to make a model of your design and compare it to your implementation by making a set of random inputs and checking the outputs.
You'll first have to become 'fluent' in EE, but for a physicist, it's just spending the time and getting used to things. Not terrible, long, but straightforward.
As towards what the article is talking about, you need to be trained in it. Honestly, you have to apprentice with the Greybeards (they are mostly men, but not always). There are other ways, like reading through Intel docs or the manuals for ICs or digging through forum posts from 2003. But those guys in the basement with funny newspaper clippings from the 80s or old xkcd printouts are a much better return on your time. They have tons of knowledge about specific chips and machines, stuff that is nearly impossible to recite unless prompted. You just got to spend long lunches blabbering with them, despite their strange political and societal views. Just listen to them, then write down every little thing they said. They are gold in terms of hardware.
It's not just HW, it's all big, niche industry, today.
Networking for example is the same. If you want to test high scale network equipment OR virtualized network functions, you will buy hundreds of thousands of dollars worth of closed-source testing hardware, software and/or professional services from one of a few big vendors. You will not let anything about your algorithms and designs slip to the outside world, and neither will your test vendors.
Edit: the same is true of most of the software world in general. Sure, you have Microsoft and Google and many others collaborating on Linux, or releasing Kubernetes, VS Code, Go and so on. But the core IP that is key to their business? That is staying in-house, fiercely guarded, developped and tested by an army of engineers.
The main difference is that there are far fewer well-defined software classes that can be tested generally, so it doesn't make too much sense to look for a 'software testing' industry, like you can for hardware. There are some tool vendors, but they offer far fewer guarantees, since it's hard to imagine a product that could find a large proportion of the bugs in both the Haskell compiler and World of Warcraft.
"Verification of a processor is different from the verification of other pieces of IP, or even an SoC."
Not sure I understand this. An SoC is a processor, plus more stuff (memory, I/O), right? Is the idea that it might be easier because the "more stuff" abstracts away some inner details?
The processor is usually already verified on its own before it is put into a SoC.
Using the raspberry pi example from another thread: broadcom designed the BCM2835 SoC, which included an ARM1176 core. Broadcom probably didn't do a ton of verification for the ARM1176 core itself, since ARM already verified it.
Have there been any attempts to use generative testing (e.g. quickcheck) or dependent types to verify processors? I am not sure how quite how this would be integrated into the synthesis of the processor, but it seems to be in line with the general "declarative" approach to building RTL through VHDL I remember from undergrad.
There is a lot of random testing in processor design. To what extent you'd call it property-based testing can be argued. Intersting factoid:
Koen Claessen, one of QuickCheck's inventors, also co-designed
Lava, a circuit designer DSL in the Haskell eco-system.
If by dependent types you mean theorem provers, then that is used, but rarely -- hand-verification doesn't scale to modern processors, usually you model check against some temporal logic formulas that the processor meets its specification. If OTOH you mean using HDLs (= hardware description languages) that use dependent types, then mostly not. Arm's ASL (= Architecture Specification Language) has a tiny bit of dependency build in to reason about length of bit vectors.
Genuine question - other than increasing the number of players, what is the horizon of value for new processors to begin with ? Is there anything more impactful than squeezing a bit more performance/watt ?
cost. it is stupidly expensive to include any sort of commercial MCU core(s) in your chip. Ancedata: ARM won't even talk with small ASIC fabless companies, even if they are willing to shell out big upfront costs and royalties per chip that ARM demands. Having free or affordable alternaties is a great driving force for the industry.
> ARM won't even talk with small ASIC fabless companies, even if they are willing to shell out big upfront costs and royalties per chip that ARM demands
[Citation needed] There's literally 0 upfront costs for a Cortex-M3 [0]
If Moore’s law is indeed over and quantum computing isn’t a silver bullet (even if they get it to work), then the future will be clever design rather than raw horsepower in order to increase the speed of computation.
It has the potential to spawn a new eco-system. Like if you look at what happened with the raspberry pis. Other players could have pulled that off technically but didn't attempt it.
raspberry pi didn't really have a new processor though. The original had an ARM11 core (designed and verified by ARM) that was almost a decade old when the raspberry pi launched.
They also leveraged existing Linux drivers and other hardware that was available to integrate without big problems.
Spawning new ecosystem is not about making something completely from scratch like processor. One have to align a lot of stars in the sky to make that happen. They had a specific goal and niche where they planted the seed for RPi.
It's more like you have a massively parallel program that you compile ten million times, but only five million complete the process usable afterwards and of those five million, they may or may not have all the parts of the program you intended in the binary. And you have very little insight into it while the compilation takes place.
And you have to check/debug those 10 million compiler passes at various stages, and each design change may require developing a new debugger or disassembler from scratch to plug into the compiler at each stage of compilation.
What I'm saying is that CPU designs aren't programs, because you can generally trust the compiler to be infallible (and compiler bugs are there, but they're rare). In a CPU process you have to consider the physical impact of the design on manufacturing, what yields you get, how the product is binned, and so on. There are feedback loops between the packaging, testing, and design teams to alter the silicon before production ramps up to go to market. There are tons of moving parts to the actual design process itself, let alone what is being designed.
I don't think this is quite right. Based on my (admittedly limited) experience, it takes a lot of work to design and verify a fast processor, but unless your processor is very similar to existing ones (in which case why bother?) it takes way way more work to write all the software needed to support it.
I guess everyone underestimates how long it takes to write software - even hardware designers.
Interesting when coupled with how many we are losing. It wasn't that long ago that PA-RISC, Sparc, Alpha, Power, MIPS, etc all credibly competed with one another and Intel. Now it's almost all x86-64 and ARM.
ISAs are consolidating sure, but the interesting parts of the chips are also consolidating. A few years ago several companies were designing new Arm cores; now it's pretty much just Arm and Apple.
There are still other companies doing new core designs: Huawei/HiSilicon (Taishan v110 in 2019), Marvell/Cavium (Thunder X2 in 2018), Samsung (M4 in 2019), Fujitsu (A64FX is 2019), Nvidia (carmel in 2018).
If you count semi-custom cores derived from ARM designs, then add Ampere Computing and Qualcomm as well.
Hardware verification engineers call it "constrained random verification", but it's basically fuzzing. This has been the backbone of most commercial hardware verification flows for a long time.
Why is it supposed to be so complicated to do "processor verification"?
Why can't you simply upload the design to an FPGA, and then check that it can:
1. Boot all available operating systems (Linux, *BSD, Windows, etc.)
2. Successfully compile and run the testsuites for a bunch of open-source software (several languages like Rust have a standardized repository and method to build and run tests, so this is very easy)
3. Correctly run stress testing software (Prime95, etc.)
4. Correctly run several software unit tests that you write to exercise instructions that may not be produced by LLVM/GCC
5. Correctly run tests you write to exercise specific processor/cache states
6. Properly handling fuzzed code without freezing the whole CPU (using afl-fuzz)
Start with the simplest possible in-order core so that you get it working very easily, and then evolve to your desired end-state with a series of small commits, and if the verification fails use `git bisect` if needed to find the offending commit, insert any instrumentation you might need to detect the issue and fix it.
I don't see why you would need a specialized tool for that, or even what a specialized tool could possibly do.
Have a look at the extensive errata Intel publish for their CPUs. There are hundreds of mistakes in the chips’ behaviour, and yet each buggy CPU would pass your set of tests with flying colours.
While you could never release a CPU that didn’t pass the tests you describe, they don’t even begin to exercise all the corner cases for a chip. Multiplying two specific numbers together, while the instruction crosses two memory pages, when an interrupt arrives? How do you even test for that kind of thing?
I feel the SW world is affected by some analogous bugs though. Any sort of race condition between two different threads accessing the same resource, maybe throw in some other piece like having the data always be valid unless a third thread happens to free some downstream resource at the same time...
We know about techniques to reduce large classes of errors. Data races in particular can be prevented by some languages statically. Other types of “once in a blue-moon” errors that happen as a result of two coupled systems doing something in tandem can be reduced by introducing stronger boundaries between the systems, and then you can test each system independently and make sure it works regardless of what the other system does (I.e. dependency testing, or maybe even fuzzing).
These approaches aren’t bulletproof, but I think they do illustrate a point: that there are techniques to reduce the likelihood of the errors you highlight. Whether they do it at a competitive cost to existing industry practices or not, I have no idea.
Hardware is usually orders of magnitude more reliable than software, so what makes you think that they aren't already using those techniques or something better?
Hmm? Maybe they do, I hope they do. I was replying more specifically to this part:
> Multiplying two specific numbers together, while the instruction crosses two memory pages, when an interrupt arrives? How do you even test for that kind of thing?
I.e. trying to despell the idea that large systems are intrinsically difficult to test.
First problem: "an FPGA". Unless it is far behind the state of the art, your processor design won't fit on a single FPGA chip. You need to partition the design, and run it on many FPGAs, and do the partition in a way that is correct and doesn't drop the performance to almost nothing. This is what the EDA emulation vendors will sell you: the systems with FPGAs organized into boards and racks of boards, and the software to target those systems, because to fit a cutting edge processor into the system you're going to need hundreds or thousands of FPGA chips.
Once you do all that, you can try carrying out your program as described above. But it guaranteed that the first time you try it, it won't work, because your design will have bugs. Then what?
You need the ability to debug. This means you need to have probes, you need to be able to extract the data, and you need very high bandwidth. You need to have testbenches that are partly in software and partly on the FPGA hardware. Again, that's what the EDA industry will sell you: the hardware and software to do it, as well as the expert consultants to walk you through the process.
And your device needs to interact with the environment. Some of the hardest verification problems have to do with the timing of interrupts; if one comes when the processor is just at the right point, and that case isn't handled in the design, it could lock up. THose cases have to be covered.
Now, for your example of a small core, perhaps it's small enough that you could get it to synthesize and fit into one large FPGA chip and avoid some of these issues. Good luck doing something that can boot Android in that size.
I thought the big cadence emulators weren't made of mainly FPGA chips, but instead arrays of absolutely tiny processor cores that only know logic and branch ops.
Not that this makes a difference for the core of the point you're making, it's more an aside.
Only speaking for myself, I think of FPGA and hw emulation interchangeably.
At the end of the day, I don't care (or know) how it's implemented. What I can say is a) they aren't super fast, only ~1Mhz b) they take forever to compile down to, c) they have some, but not great visibility to what went wrong, and d) they cost a stupid amount of money.
Well if you've just run your verification on FPGA like that chances are your silicon will fall over on first boot because you've totally missed a whole bunch of corners cases that only occur on the real memory system. Yes you can attempt to emulate these in FPGA but that's one of the reasons verification is not as easy as it seems.
> Start with the simplest possible in-order core so that you get it working very easily, and then evolve to your desired end-state with a series of small commits
You can't just trivially evolve a simple design into something more complex, much in the same way when Linux does a new major release they haven't started with some stripped down basic *nix and worked their way up from there.
Great question. For starts though, you've just given me a list worth hundreds of trillions of instructions of tests.
If you can actually test on an FPGA (which is usually not possible, and if it is, it's not representative of the actual silicon/analog system you're building anyways), you can get 1T instructions in roughly ~6 hours at 50 MHz. But most hardware emulations are ~1 MHz, so now you're looking at weeks to hit 1T instructions (SPECint alone is 20T).
And what happens when you hit a bug, 2 weeks in? It may not be because you actually wrote new, buggy code, but because a new, higher performance branch predictor uncovered existing bugs. But you'll never know, because the FPGA historically gives you terrible visibility.
But in simulation (where testing is actually done), you're looking at ~1 Hz for a cpu core. Ouch. Obviously a better approach is required (unit-tests against models, formal, etc.), since at the level of detail you can't test much of anything.
This is a very well formulated question, well asked.
People are already doing every step you mentioned but there are three problems:
- Processors nowadays are so advanced and complex that you can't simply approach them as if it was one single block. You need to divide the processor into smaller blocks, develop those smaller blocks and put them together by the end of the development. Like in any engineering problem. The main problem is that it takes a lot of time for all sub blocks to be mature and stable enough for top level integration.
- Once you have all your blocks ready, you can start integration and bring up the system with FPGAs. But now you face the problem that FPGAs are really slow and are not really usable as a normal system. You can run some preliminary tests, the short ones but it would take months to properly execute a normal benchmark.
- You could create and tape out test chips but then you would also need to create all the infrastructure needed for the processor to work like the memory system, memory RAM, communication buses, firmware and etc.
Just a random FPGA won't cut it, if you want to simulate a reasonably large processor design. Also, you would only be able to simulate the basic logic for its validity. But that is only the very first and easiest step in the verification chain. Things start to get more interesting, when you look into the real-world analog properties of you chip. You want to check the correct timing so that the logic still works correctly at high clock speeds - depends very much on the acutal placing of the components, e.g. wire length. Then there is the question of thermal behavior. Long term stability (years of operation). And then we enter the space of manufacturing. You need to optimize your design so that it can be manufactured reliably. There are some systematic variations across the wafer and your design should account for that. There are also a lot of constraints given by the production process, how you have to distribute components on your chip.
Getting from an initial logic design to a manufactured chip is a big adventure. This requires a lot of layers of software, a lot of it highly specified. And yes, you can buy very good simulators for chips, just search for Cadence Palladium for example. They are huge monsters.
Imagine if your test suite with 400 tests simply returned true or false instead of telling you which tests have failed. It's going to take forever to find the failing tests.
