If you're a target of a major intelligence agency, I think that you have to assume that all of your computers are irretrievably compromised. From Vault 7, we know that the CIA has long developed implants to infect both the EFI and hard drive firmware that load before any potential code that could detect them. These could be made arbitrarily hard to detect without physically opening the computer and dumping these flash devices and comparing them against a known good image. Who knows what other embedded processors with a little bit of flash lurk in various peripherals in your laptop that they've figured out how to wheedle their way into... If the flash is integrated into the microcontroller itself, there may not even be an easy way of reliably dumping its contents.
I think you are absolutely correct with your assessment. I recall Alan Cox (welsh bloke, big beard, Linux kernel hacker (well: simply hacker in general will do)) posting on G+ about someone booting enough Linux on a hard disc to get a prompt. No not the disc itself, off the firmware on the controller.
You may also like to consider that nearly all modern server systems have an iLO/iDRAC or whatever that can do all sorts of things, and at least one internal USB interface. PCs can have the Intel ME and other horrors. The best you can hope for is that it is only your local intel. agency that potentially have routine access to your system.
Could all firmware be on WORM chips? Which can't be rewritten, no matter what an adversary does. Updates would require switching chips. But at least driveby implants would be impossible.
Most computers have an Embedded Controller (with integrated flash) that does a lot of motherboard/system specific stuff like power management, flashing leds and even scanning the keyboard matrix on laptops.
