> “The next day [after reports about the Russian jamming effort hit the media], Starlink had slung a line of code and fixed it,”
The notion that SpaceX is basically using a Full CI/CD pipeline on a massive satellite constellation is kind of amazing. Meanwhile most large government software requires months of contract negotiation before anyone is even hired to fix the problem.
I want the government to have the in-house expertise to just do things, especially things that can't be specified in a contract by a KO making $53k per year with no real-life experience.
I'm responsible for hiring exactly the kind of engineers we're talking about. We can't pay as much on gov dime, and equity is nonexistent, but we have some of the best, domestic engineering teams around doing amazing things.
Our people go on to work at (eg) SpaceX and to found exactly the kind of startups that we're all dog whistling about. (Lars Blackmore?). Lean, focused, fast and providing valuable services to gov and private industries.
This discussion about how 'the government' cannot 'do this' is a vague rorschach blot that reveals people's own motivations only, and ignores the completely different priorities gov has. Do you believe the deep technical expertise that gov-funded experts are surely providing to Ukraine would be posted on Twitter?
For essential services, the government should be paying competitive salaries with the private industry. Contracting rarely makes things cheaper due to all the grift and lack of persistent oversight.
The government probably needs to build a whole new pay schedule for tech workers because they just don’t fit into the General Schedule. This would require an act of congress and I think they are going to have a hard time making the decision to pay government tech workers more than people like the director of the FBI, secretary of defense, senators, etc.
The fact that we don't has a lot more to do with government culture and a post-Reagan idea that the government should be a thin and highly reactive operation than the money not being there.
Mathematically, sure. But good luck explaining to a farmer in Iowa why you're paying a 28-year old $250k/yr to "do computers" for the VA. And that's ignoring the legal hurdles to paying some government employees $250k/yr and continuing to pay everyone else around them the same garbage GS wages.
Aren't there already people not on GS anyway? From researching, it seems like other areas have already figured out how to do this: NSA and "some parts of the Navy" are both called out as using non-GS pay scales.
I don't see why they can't have like "Advanced Science" wage table, to compete for the best engineers and scientists rather than hiring either bottom of barrel or end of career people.
I make much more right now with 8 YoE than my boss in the gov't did and he had a PhD with 20 YoE.
I know several people working as engineers for government contractors and they all make well over the top of the GS Pay Scale, and that's before other benefits like profit sharing.
Even if Congress corrected for that, they'd still be locked into slow moving pay raises, and they'd probably become a political target at some point, when some anti-spending congressperson begins harping on how there are so many federal employees make a quarter million dollars a year.
Being a military contractor, for better or worse, inoculates you from a lot of politics. So, yes, the government could pay competitively, but I would be leery about accepting such a position.
The way to make money working for the gov't is to get an established company to price gouge them through contracts so you can make a living wage.
The gov't is wasting a huge amount of money paying a 100% premium for engineers because they aren't allowed to pay what the market demands.
A "data entry specialist" and PhD scientist are paid off the same pay scale at different levels, which is great for equality but kind of difficult when you are trying to pay $31k (which is what someone with a BS in CompSci would get, GS-5) for someone who is going for $150k.
If you have experience or a master's, they'll pay you $51k.
It's absurdist. The GS scale caps out at $143k at GS-15 and requires a PhD and probably decades of work.
exactly, who the hell would want to work for the US government when they can work for spacex? i worked as a contractor to multiple DOD projects in the past, it was incredibly bureaucratic and the quality of engineers that were involved was very low compared to the private sector.
I get that space and defense are apples to oranges, but there is no reason to think it is significantly different.
I think you read the opposite of my intended message. I work with the best, and we take from, found, and give to all the best companies as well. It's a point of pride that we compete with them ( as a workplace) with much less pay. Fostering a highly technical workforce for the nation and helping commercial space is part of our mission.
Ha there's so many assumptions (driven by bad experience -- I'm sorry about that). But I don't know where to begin.
One indication that I work with the best is where they go when their work here is done. In some cases, the finest work e.g. SpaceX has done is done with our alumni in leadership and contributor positions.
Another indication is top tier publications. Or the fact that our tech does what has never been done before by definition.
Another indicator is how many people are willing to work for a pay cut to be here and work on what we do. Speaking of coming from SpaceX and every major aerospace company.
Another indication comes from the partnerships we have where we collaborate as equals with any of those mythical "better" places.
It may not fit your world view or experience, but it is possible for good people with top notch skills to choose a career that does not focus solely on pay. We just work in an unprofitable domain. We don't work with incompetent people.
> who the hell would want to work for the US government when they can work for spacex?
Can a SpaceX employee comment here? Work at SpaceX looks stressful. Elon Musk looks like a workaholic asshole expecting all their employees to do the same, but without getting rich. Maybe government work is boring by comparison, but you get a personal life.
My good friend worked on the landing faring (those strut things that flare out when it gets near the ground) for the Falcon 9. He said it almost killed him. Now he builds decouplers and other mechanisms for planetary rovers and enjoys it just fine.
One of the members of my group just left for SpaceX last year, because he just wanted to work somewhere else for a change. He was one of the better FSW engineers we had seen, and I'm sure he's helping SpaceX do great things!
Another friend of mine went off to Amazon to oversee their warehouse automation.
And we just hired a bunch of really stellar folks from NG who were fed up with their workplace. They seem to love it here.
I meant that there isn't the non-contracted manpower, because in a lot of places 90% of the staffing is contracted to SIs with a skeleton crew of govies for legalities sake. There are a LOT of programs whose project plan is set in stone by the contract, and are simply unable to change course (especially once the SI puts their lobbyists on it).
All to avoid pension liabilities, allow downsizing that can't be blocked by a union, and provide a layer of indirection to coders being paid anything close to market rates. We need a way to hire more full-time govies, but the culture has swung so hard to contracting that there are only small pockets left of in-house talent.
Obviously this used to the the case in most developed countries, but then more and more stuff started getting outsourced and pushed into the private sector to drive shareholder value. Suddenly, space, research, military, healthcare, education are all getting outsourced, so we can make more of a profit off it.
While it's true that the government (obviously) cannot supply equity and is capped at 174k (by law capped at a senators salary), they can try to make it up other ways. Providing free room and board (or funds for the same), paying higher education costs and providing free healthcare. Also, allowing those benefits (and a healthy percentage of your income salary) to continue into retirement.
There is also the fact that they have more take home pay then you might expect, because government salaries are exempt from FICA taxes
Even ignoring those searching through different departments shows plenty earning over 174k -- Alcohol and Tobacco Tax and Trade Bureau (6), Immediate Office of the Secretary of Education (8), Maritime Administration (34), etc.
(Of course then there's state employees where PE Teachers get multi-million salaries)
Lots to unpack here. These are all medical officers. I hope I don't need to justify a physician making more than a developer. And if you look at the averages,[0] these folks are still making less than they would in the private sector.
What state PE teachers are making multi-million dollar salaries? I hope you're not conflating huge D1 coaching programs with "PE teacher."
A physician only impacts the limited patients they see where a developer can impact millions and a team leader can impact many developers who impact millions.
Using common logic we see in the field on why ic should be paid lower than team leaders.
First, "physician" is not equivalent to "practicing doctor in a clinical setting".
Second, a good physician - even a run-of-the-mill family doctor - should be collaborating with their peers. They're keeping up to date on new practices and informing the next generation of those practices by reporting observations of the results of today's practices and any anomalies that arise.
I don't think it's unreasonable for some developers to make more than most physicians or for most physicians to make more than most developers. Neither field is narrowly defined.
A typical licensed physician is a run of the mill family doctor or walk-in clinic doctor. I can assure you they are spending however many hours required to keep their license but are not keeping upto date on the latest practices. They are sending people to labs where data is generated that can be used by the next generation but we are not paying lab workers 1/2 a million.
That list of 78 (not 100) making over 400k are administrator/doctors who work for the VA (except Fauci). The VA does have it's own pay scale for medical professionals - it's a legal limit not a constitutional one and the VA has it's own rules.
Administrators who are Senate approved can also get exemptions to the salary cap.
Taking for granted that this would be possible, from an economic and political standpoint, I would consider equity in the US Treasury to be about as potential valuable as equity in a randomly selected tech startup - which is to say, almost certainly worthless.
I'm trying hard to come up with a response that's not just an insult and dismissal.
I have no idea what you're talking about or thinking about. A percentage of tax receipts? Why on earth would that make more sense than just money up front?
I don’t know why you feel an urge to be dismissive or insulting about the government issuing securities. Or why some money up front is necessarily better than more money later, which surely is the whole point of equity in the private sector.
I’ll grant you that dusty old war bonds are unlikely to get strapping young devs very excited, but I’m sure that they can come up with a higher risk, higher reward financial instrument with some imagination.
Government issued bonds are the antithesis of high-risk, high-reward. And I don't see any reason why you wouldn't just use whatever salaries make sense. Google et al use RSUs for vesting purposes, but aside from tradition (and tax benefits) they could just issue a cash amount.
In a smaller company, the stock may appreciate due to your actions, but there's absolutely no reason to think most employees would do anything that moves Google's stock price. And even less to think anyone would do anything that would impact US government tax receipts.
Working for Google and getting RSUs isn't really higher-risk, higher reward compared to a salary.
> Government issued bonds are the antithesis of high-risk, high-reward.
That’s exactly what I said, that they need to come up with a higher-risk, higher-reward instrument because bonds aren’t high risk, high reward enough to act as an incentive. Your problem clearly isn’t that the subject matter calls for insulting or dismissing, it’s that you decided in advance to act insulting and dismissive regardless of what the response may be.
The government makes the rules in the first place, I’m sure they can figure something out. At least I object to the claim that it’s obvious that they can’t.
Agreed, but this is largely an American problem. In Europe and where I am in the UK, salary isn't the problem, but there is still a drive to outsource.
When I've been a contract developer, I've probably been making more than board members in some countries, but that seemed to be accepted over the short term. There's also an issue where enterprise companies wouldn't trust their IT departments, for historical reasons so the good people left or they got consulting businesses in and that made everything worse...
They can recruit very good software engineers out of the Southeast & Midwest for under 170k just fine. It's still going to cost them 6 figures, but they are going to be able to hire good talent.
I will admit though, they're going to have trouble retaining the tippy top of the seniority ladder at 170k.
You say that, but some senator will make his career talking about the 300k salaries paid to these teams and their managers during times when their expertise is not saving lives in a war zone.
Now I agree the airforce or some branch could do this, but that would be a military intervention.
I've read (need to dig up the source again) that high infrastructure costs in the US are in part due to the same problem. The government doesn't even have experts anymore who could write the request to take offers on. Maybe in some cases a slightly larger government can be a leaner government.
When Alon Levy was comparing transit construction costs in various places around the world they found that having in house experts was one of the biggest causes of low costs. Though see also this critique[1] of Boeing's massive outsourcing and compare to SpaceX.
Raise taxes, close loopholes, fund and modernize the IRS, incentivize R&D and entrepreneurship, increase government spending and expertise, improve government auditing, cut defense spending graft, refill positions left vacant during the 2016-2020 state department staffing fiasco, fund childcare, expand Pre-K schooling, subsidize critical infrastructure build out, and nationalize some right of way land for infrastructure to start with. Sounds easy.
Nah, it's just wishful thinking. I've knocked on doors, done the calling campaigns, ran the emails and texts. I don't think it'll happen, and I don't try so much anymore. I just vote nowadays.
I think the issue with a lot of governments is: expertise and competence is expensive.
If you don't want to pay for that because the government is then expensive, fine, but eventually you'll have an incompetent government that is slow and still expensive due to the mishaps it leads to, for itself and it's constituents.
The idea that the federal government cant hire "competent engineers" is kind of odd to me.
Federal government can pay enough to get software engineers. FAANG salaries are an outlier. They could easily target some midwestern state, put the headquarters for federal software agency there, find a feeder school, and get employees fed to an agency making a comfortable 6 figures in a low cost of living area. Will there be some attrition? Sure. But plenty will stay cause thats where family is, and where they have their house, and thats where there kids go to school and etc.
I can guarantee you that the contractors they hire from Accenture and IBM are using devs that arent making FAANG salaries. Its kind of a meme how bad some of the government contractors are, its pretty well known you can just be a body that sits at a desk if you can get a security clearance and make easy money just manually running an SQL script once a week on some of these government contracts.
The real reason nothing gets fixed is because the federal workers get free salaries and benefits packages that far outstrip their value that oversee these projects, and the contracting firms get easy money.
> They could easily target some midwestern state, put the headquarters for federal software agency there, find a feeder school, and get employees fed to an agency making a comfortable 6 figures in a low cost of living area.
They have this already: midwestern schools which have affiliations to the military and act as feeder schools for technical people into the military industrial complex.
These people still make more than the GS schedule cap by mid career with TS clearance. It surely doesn't pay faang salaries, but $150-200k with 25% bonus/profit sharing, and totally paid family health care, and <asks someone> 320 hours of pto. Plus, these places often have nice company events, like corporate retreats to nice locales (including family), etc.
Well, option 2 is cheaper at first. Then everything breaks down eventually and that is costly, but later.
Kind of like technical debt, but for a whole government.
Also, there will be hell from the public when the govt. makes a mistake. So, there's all sorts of bureaucracy, checklists etc, to keep that from happening or to at least have people accountable. Which also slows stuff down.
The government fields a very well trained, powerful military. This is arguably within the mandate of government. Provide for the people's defense.
However, the same military has been unable to complete its stated objectives in Iraq, Syria, Afghanistan, etc. 10 times out of 10, the situation has remained exactly as it was before, or worse.
If the military fails at a core mandate that we mostly agree to be a reasonable one... what makes you think they could maintain complex code?
That is a false argument. Usually US military solves their military targets quite effectively. The failing parts are usually tied to the flawed logic in the political goals they are activated to accomplish. I say this as a foreigner observing them for 50 years. The US military cannot compensate for flawed US political goals, eg bombing some democracy into appearing. And they dont set those goals.
E.g. the balkan conflict in the 90s was not handled until the US bombed the serbs, and it worked.
I would argue that for the military to complete the stated objectives (which shift almost as much as a software project often does), is far more complex than maintaining complex code.
> However, the same military has been unable to complete its stated objectives in Iraq, Syria, Afghanistan
the military is under civilian control. The US military is very very good at killing people and breaking things. I have no doubt, given the orders, the US military could kill everyone and break everything they've been ordered to kill and break. Politicians direct the military, if objectives were failed to be met then politicians determined those objectives were no longer politically viable.
Tech's churn and rapid rate of change prevent a slow government from making solid long-term choices in technologies that are both reliable and cutting edge. You used to hear a lot about how you'd become un-hireable if you didn't keep learning new things (not so much these days), so there's not a great incentive for experts to stick with older technologies and build years-to-decades long solutions.
There's a lot more to it than just contracts and contractors. There's a good book that talks about improving the military's ability to respond (The Kill Chain). The general point is that the face of war has evolved but the U.S. military's primary method of changing direction and decentralizing decisions (The Kill Chain) has not.
in my experience dealing with governments (and i have a lot since the pandemic started) there's months of negotiation to make a decision that there _may_ be a problem in the first place. The next months are spent on identifying the person who will make the decision that there is an actual problem. Then they'll start negotiation on how to define the process that will pick the team responsible for qualifying the team responsible for fixing the problem. Then your "months of negotiation" start.
edit: i'm being a bit dramatic but it's pretty bad. no one will make a decision because a culture of CYA means whoever dared raised their hand is the one on the cross if things go wrong.
I think it's interesting to summarize this thread as:
- Government can't offer competitive pay due to politics.
- Government contracts big projects to 3rd parties who pay competitively, but lack the transparency that government is required to have.
In a way it would seem to be that government contractor organizations are acting as an abstraction, and are in essence a way of government allocating funds without needing to provide the transparency and accountability of doing things in house.
In essence, if you follow this line --> contractors are basically government. (i.e. Working for SpaceX is basically working for government without the page/scale political challenges of getting stuff done, SpaceX is more or less funded by Gov)
You can celebrate SpaceX correcting their issue quickly, without also inventing a reality where the same issue wasn't fixable in a timely manner due to some worst-case-scenario government contract hell.
The director of electronic warfare for the Office of the Secretary of Defense said as much
> The government, on the other hand, has a “significant timeline to make those types of corrections” as it muddles through analyses of what happened, decides how to fix it and gets a contract in place for the fix.
> “We need to be able to have that agility,” Tremper said.
The article literally has the government officials saying this would take months and contracts. That's the whole point of the article.
> The government, on the other hand, has a “significant timeline to make those types of corrections” as it muddles through analyses of what happened, decides how to fix it and gets a contract in place for the fix.
Or did you not even read it, but decided to come comment on what you presumed it said?
But this also cuts both ways - really nice if it works but you will also look pretty stupid if you accidentally brick an entire satellite constellation or some malicious actor uses this capability against you. But I obviously have absolutely no clue what they are actually doing, so I will abstain from any judgement.
Generally there are contractors employed in some fashion at all times on a broad enough contract who would be available to jump on something like this. Whether pen or they’d get the approval to ship it quickly is another story.
> "Jamming attacks are common threats to wireless communications and happen when a hostile device creates radio signals that disrupt communications by decreasing the Signal-to-Inference-plus-Noise ratio (SINR)."
> "Possible ways that a software update can help bypass a jamming attack is if the jamming device is targeting a particular frequency. The software can implement FHSS or DSSS, Frequency Hopping / Spreading Sequences of frequencies for anti-jamming purposes. However, FHSS or DSSS, frequency distribution type of protection against jamming is flawed. It is ‘limited by a common assumption that the jammer can jam only part of the communication channels or has a limited transmit power.'"
That article points to this report, which discusses all kinds of novel ways to defeat such attacks, like using counter-jamming as a means of delivering bits between sender and receiver, bit synchronization schemes, and others:
How did it shut it down? What setbacks did it induce to Russians? The headline reads like sensationalized clickbait for satelites changing service frequencies.
I had never considered this before - but it seems to me that lean development methodologies and CI/CD (and other DevOps practices) will become a defense priority, at least a priority of the nations that effectively wage war.
I think it extends beyond "do development of software in a lean way", and should also include "make more things configurable with software/firmware instead of hardcoded in hardware". In the starlink case, I imagine they could somehow reconfigure the SDR on the device to get around jamming.
If instead the entire chain were done in hardware, then making these kinds of changes would be rather difficult.
At the same time, if more things are software more things become targets, so you need to care much more about the actual security of these kinds of update systems.
It needs to be, at least for some projects. Unfortunately, it's the organization, internal bureaucracy, and controls that fight it the most. Someone on hacker news said that governance is a glue you pour over an organization to keep it in the current, working form. Changing those mindsets, and avoiding a compliance group from squashing progress, is in my experience nearly impossible.
The ability to deliver a critical fix on a timeline of minutes to hours is contingent on having well exercised infrastructure and processes in place to do so.
If you don’t do CD, it will either be more expensive because you have to maintain two processes or there is a good chance it will fail when you need it most because the critical delivery path wasn’t well exercised.
> "If you don’t do CD, it will either be more expensive because you have to maintain two processes"
Why two? Release process is always the same, only decision to go live is manual and can be based on different criteria in different circumstances.
CD is enabled by certain level of automation, which is nice to have, but if you do not aim to have CD, a good manual process can still work and may be actually less expensive in certain scenarios if you consider delivery pipeline TCO.
> decision to go live is manual and can be based on different criteria in different circumstances
..this conflicts with previously stated requirement:
> a proper support process where critical fixes can be delivered within hours or minutes.
That the decision to go live is based on criteria and circumstances implies that there are criteria and circumstances that make a release ineligible to go live at those times.
If those criteria and/or circumstances are frequently blocking, then the manual pipeline is not prepared to deliver critical fixes on short notice, because the release pipeline is often blocked (whether CD or not)
If those criteria and/or circumstances are infrequently blocking, and the manual release process is also exercised infrequently, then it increases the risk that the manual release process may not work on a critical timeline, because something may be broken but unnoticed from lack of use.
If those criteria and/or circumstances are infrequently blocking, and the manual release process is also exercised frequently without problems, then the critical release should be fine. (Although this case seems suspiciously like CD and I’d be a little surprised to hear that the cost of automating such a finely tuned pipeline is significantly greater than the cost of the manpower to run it manually).
Well, the thing is there are many organizations that live without CD/D (delivery/deployment) and still can release critical fixes quickly. CD does make things easier, but it is not a necessary condition. Let's go through your line of reasoning:
>> a proper support process where critical fixes can be delivered within hours or minutes.
>That the decision to go live is based on criteria and circumstances implies that there are criteria and circumstances that make a release ineligible to go live at those times.
Indeed: you normally do not release the code that is not ready and you do not release the code that will alter the business process before business users are ready to accept the change. It does not mean that it is impossible to deliver a critical fix in such circumstances. Let's look at the delivery process first to understand, if there's really any contradiction.
In a well organized process you have branches and you probably use pull requests to deliver the code to the main branch. This branch does not have to be the same as the branch of the last released code, meaning that you can actually work in both branches. The release cycle on the main branch can be long if your company can manage the complexity of bigger projects. However, if there's a critical bug found, you can develop the fix on the production code branch, then release it quickly and merge the changes to main. The fact that you normally do not push changes as fast as possible to production means that you do not have continuous delivery, but it does not mean that you cannot have automated deployment or even continuous deployment (which would be in this case triggered by a merge to designated branch after automated tests on certain branches are passed).
Now, how about continuous deployment? It appears, you can actually have continuous delivery or automated deployment, but not continuous deployment. For continuous delivery it is sufficient to produce quickly build artifacts via CI, but the way they are released can be really different and may not involve deployment by merge. I worked in an organization where production system was physically separated from delivery pipeline and artifacts had to be copied to a flash drive. We could still produce a change and get it live within 10-15 minutes, because the manual process was actually well-thought and optimized to reduce human errors. We had releases almost on daily basis and we had few times when we actually delivered changes very fast (once it involved a rollback for a complicated $100M transaction through 20 systems that went through because of a missing validation requirement). You may even have a fully automated deployment triggered by one click: the only difference with continuous deployment will be that it's not done once the build is green.
To sum up, CD (continuous delivery) is not the same as CD (continuous deployment) and you can have just one of them or none, still having the automated deployment process. Or not having it, but still being able to deliver critical fixes quickly and with minimal risks.
I do not know, what is the process at SpaceX, but the fact that they did fast rollout does not mean they have CD in either sense or it even makes sense for them. It's better to ask their team. :)
CI and CD are different things: you can use CI without CD, e.g. when you have manual process of deployment (if CD is continuous deployment) or long waterfall-like development cycles (if CD is continuous delivery).
In highly secure environments CD may not be possible at all, because of formal processes of verification and manual processes of deployment (e.g. I once worked in a project, where deployment process involved copying build artifacts from CI server to an authorized flash drive and papers signed by customer).
What is the evidence of this "electromagnetic warfare attack"? What is claimed to be attacked and how? How was it affecting SpaceX? Is SpaceX/Starlink really crucial to anything in Ukraine? Etc, etc.
The only "explanation" this article offers is a link to a meandering video that doesn't answer any of these questions.
After "establishing" the premise, the article immediately launches into demands for more resources:
"We need to be able to have that agility"
"We need to be able to change"
"The U.S. needs to think a lot more innovatively"
"This includes incorporating artificial intelligence and machine learning into next-generation systems to be able to respond faster"
Over time articles like this one make people accumulate an extensive set of assumptions that basically aren't supported by anything except endless repetition.
agreed that the article is trash, but Starlink is important to Ukraine - the Ukrainian drones use Starlink for their uplinks (note that terminal roaming is enabled in Ukraine, you couldn't do this in the US!)
This article uses incorrect terminology. What the Russians did was electronic warfare, not electromagnetic warfare. Electronic warfare jams, blocks, or otherwise disrupts electronic communication. Electromagnetic warfare damages or destroys communication equipment.
> Electromagnetic Warfare, more commonly known as Electronic Warfare (EW), is defined by the U.S. Department of Defense (DoD) as, “any military action involving the use of electromagnetic and directed energy to control the electromagnetic spectrum or to attack the enemy.”
Doesn't seem like your distinction is commonly used.
I find this article dubuous (or lets say "thinly sourced"). I.e. this phase
"its Starlink satellite broadband service, which was keeping Ukraine connected to the Internet" is just blatantly wrong. Yes they've send hundreds or maybe a few thousands of terminals ot a country with tens of millions of people. I have not seen any evidence that this plays any role for Ukraine.
It's good that Spacex did send the equipment, but it is still more of advertisement as far as I can see, rather than something keeps the connectivity of Ukraine to the internet. (I'm happy to see reports that show otherwise).
It was apparently used to reconnect local cellular networks [1] to the Internet. In such a case, a single antenna scales better than if it is used by only one person. Still, the overall benefit was probably limited, otherwise we would have seen more reports about it.
> Is Starlink that important? It wouldn't be possible to restore 10 km of cable connection between villages in Chernigiv region after serious battles so quick. Normally it takes few months. Another case: provider Baryshivka-Net resumed the network for 5 villages by using only 1 .
I feel like this article is over-representing how much SpaceX kept the country online. How much of Ukraine's traffic went over the SpaceX satellites? Even in the east where most of the destruction and fighting happened, people have cell reception. Article made it sound like without SpaceX, Ukraine would not have internet. But doesn't Ukraine have connections to Europe through Poland and it's other neighbors? Russia would need to literally cut the cables or destroy every cellphone tower and ISP and backbone providers in the country (Russia didn't do any of these things). This also kind of reveals that SpaceX and other radio based technology is more vulnerable to attacks than wired connections. Also shows how resilient the design of the internet is! Routing algorithms FTW!
I call BS that it was SpaceX that kept the country online.
- In Ukraine (similar to other ex-USSR countries) govt laws require from phone companies, to provide service "at least 8 hours after power off". To be exact, this is requirement of all phone licenses, land and cell.
So in real life, on most cell towers accumulators withstand much more - I have worked in telecoms, 24h of work without power is normal for them.
For smartphone power, in Ukraine most people have powerbanks, so also could tweet for 24h and more.
But when without power for few days or more, sure business activity fall and less people will be active on internet.
Also huge factor in this war, huge number of refugees. Even considering not all people flee abroad, but about 12 millions (from approx 40 millions) moved from east/north/south regions to west regions.
How did the update propagate to the ground terminals if the connection was jammed? I assume they switched frequencies, but it would still be interesting how it went exactly.
That's why I disappointed in high uptimes, which most tech persons love.
What really need in modern environment, high availability, so it is not problem when one server rebooting, while others could continue deliver service.
March 25, 2022 "Starlink, at least so far, has resisted all hacking & jamming attempts"
It even more impressive that SpaceX was able to update it that quickly as well as a huge selling point for the military markets they are targeting so I can't quite understand why Musk has to make such a false statement.
Probably much more banal than the article leads us to believe. Russia probably began jamming on <frequency x>, Starlink deployed a change to fail over the terminals to broadcast on <frequency y>. Would be my guess.
My naive assumption was that since the Starlink terminals use phased arrays for beamforming, they were much less susceptible to brute force frequency jamming to begin with. Is there some truth to that?
As I understand, it is prohibitively hard to make constant signal with high enough power, so SL will not work.
So usually made trick - EW find when sync should be, and emit short high energy impulses, so snr become too low to receive sync part, and transmission effectively blocked.
Spacex implemented new encoding, where sync is in random place in time-frequency, so now EW could emit, and this will lead to speed degradation, but not to full blocking.
This is the kind of stuff we're building at https://www.cognitivespace.com/. If you're an engineer or developer and want to build this stuff. Come join us.
Although his was a defensive move by SpaceX we have to be careful private industry doesn't pull us into WWIII. We are not sending heavy armament to Ukraine for a reason and the government needs to make sure that their policies also apply to companies like SpaceX.
By the way how will we "feel" when the Chinese or some Russian billionaire launches their own "Starlink" which will also be operating over the US? From what I have read there are several such systems in the pipeline in China.
If a private corporation releasing an update to its code base to allow its own service to keep working brings us into WWIII... the root cause of the WWIII is definitely not the code update.
Exactly, this bizarre blaming the victim for provoking more violence from the aggressor is pernicious and ill conceived imo. But they might go nuclear or attack the NATO/US! Maniacs will do what maniacs do, that genie is out of the bottle. Countries have a right to control their airspace, countries have the right to acquire military equipment as they please. Invaders do not get to dictate that.
We should be careful of outright demonizing Putin. He's not just a monster and he can be reasoned with. Maybe he even has some legitimate grievances (or at least things we might choose to accomodate to avoid ...)
Perhaps we can to have systems, like the "we're testing an ICBM" system we have in place, to avoid escalations that will draw us into WW3. There must a way to get a reasoned, thoughtful and cooperative answer to this question. For example, let's say we place 5 test satellites in orbit and if you shoot one (?) of those down, you get to request blocking internet service in a geographical region, without escalating things into a space war.
That may have been the case in the past, but today he's just a monster. He had his chance at statesmanship and royally messed it up and the sooner the mad dog is put down the better and then we have to hope that the next dog won't be madder. I wonder if we'll ever see a world where Russia plays nice and stays nice.
> I wonder if we'll ever see a world where Russia plays nice and stays nice.
Unfortunately any real hope for that ship has sailed down the toilet at hypersonic speeds. Its not about the amount of good, smart, moral etc. folks that are still in Russia, in relatively educated population of 160 million there will always be quite a few of those.
But this layer is, and will be marginal and powerless in Russia unless its already part of government and thus on side of corruption, theft and organized murder. Mainstream applauds Putin genocide or not, or in best possible scenario indulges in complacency, preferring seemingly strong dictator over any uncertainty that more freedom can bring. Weakening any russian influence and strength is by far the best course of action, short term and especially long term.
Only a fool comes again and again with open hands for an embrace to a sociopath wielding blood stained knife.
Its a mistake when trying to apply western thinking and actions to Russia, they are very far from it and getting further away as we speak.
All this, and much much more. Plus yeah, Putin is a mass murderer, allies didn't negotiate much with Hitler neither. He also said and put in contracts whatever gave advantage at that moment, only to break it at first occasion.
Your recent comments have broken the site guidelines so badly that I've banned the account.
If you don't want to be banned, you're welcome to email hn@ycombinator.com and give us reason to believe that you'll follow the rules in the future. They're here: https://news.ycombinator.com/newsguidelines.html.
They had been trying to launch Starlink service there since before the war and the government was dragging their feet on regulatory approval. The war caused the government to rethink that.
"@elonmusk
, while you try to colonize Mars — Russia try to occupy Ukraine! While your rockets successfully land from space — Russian rockets attack Ukrainian civil people! We ask you to provide Ukraine with Starlink stations and to address sane Russians to stand."
Root cause analysis in software postmortems and in business is important, but knowing the root cause that initiated WWIII is only really useful to understand in hopes of preventing WWIV.
It's irrelevant whether a private corporation software update or the assassination of Archduke Ferdinand causes a world war to break out, it must be avoided.
Yes, god forbid private companies protect their assets. All this 'oh how will the dictator respond to perceived sleight 'x'' stuff really has to stop. It's the aggressor that is to blame not the (intended) victims.
if the Taliban militants were using it then it's a legitimate target and I would take out the antennas. I assume the same holds for Ukraine. If Starlink is being used by Ukraine military then I think Russia is perfectly within the rules of war to target the antennas. If a software update makes them harder to target then does that make SpaceX HQ a military target? idk, probably, but in the same way FedEx is a military target. I don't see it very feasible nor advisable to launch a cruise missile targeting Hawthorne CA...
I'm no fan of the Taliban, but I'm not a big fan of the US trying to control other countries either. But I am a fan of regular Afghani people being able to communicate.
He's the thing with these services: if they're controlled by a single government with the intend of using them as leverage over others, that means that at some point they're going to be denied to the people. I would strongly prefer redundancy in such systems to increase their reliability.
Take GPS, for example. The US controls it, and at least initially, their goal was to be able to reduce the accuracy of GPS for other parties. Europe has its own competing Galileo satellites to increase reliability for ordinary citizens. I'm strongly in favour of that. I don't see military applications as a higher purpose than civilian use.
They are operating over Ukraine, not Russia. Very different circumstances. Much like supplying missiles for Ukraine to use is very different to supplying men.
Every major conflict since WW2 has some degree of ‘proxy’ to it - with the US and Russia arming or supporting one or both sides to some degree. Sometimes even providing direct troop support.
It’s been accepted that as long as troops from one of the side are not directly shooting at uniformed troops from the other side (sides here being US/Russia!), it’s a proxy conflict, not a direct one.
Afghanistan, Vietnam, Korean War being some of the larger versions of this.
With Russian troops on the ground right now, adding any NATO or US troops would mean it isn’t a proxy conflict anymore once someone from the US or NATO side shoots at someone in a Russian uniform. But providing weapons to Ukrainians has a lot of precedent as being ‘not a war’ between them. Even if the Ukranians use them to shoot Russians.
> It’s been accepted that as long as troops from one of the side are not directly shooting at uniformed troops from the other side
During the Korean and Vietnam Wars, Soviet pilots engaged in dogfights with Americans, which makes the amount of handwringing over just sending weaponry being a escalation laughable.
Also MacArthur’s pursuit of the North Koreans across the 38th parallel brought China directly into the war and precipitated a much, much larger conflict, and cost a great many American and UN Allie’s lives. It
got him (rightly, IMO) fired. But that was too late.
If everyone agrees that shooting at each other’s pilots isn’t a ‘direct conflict’, then it isn’t.
The US didn’t want to officially care to that level that Chinese jets (or troops) were killing US troops, because the US could waltz back home and Korea would be the one to take the hit. At least one of the parties could accept a failure, somewhat.
The stakes for Russia here (and especially for Putin), like when MacArthur crossed the 38th, are much, much higher for them. A well armed, actively hostile (well earned hostility or not), and aligned with the west Ukraine is right on their doorstep, with no meaningful geographic barriers. Historically a scary situation for Russia.
And Europe, which has a lot (depending on if we’re counting countries or $$) of influence over NATO and is nuclear armed themselves, is also existentially threatened by Russia doing what they are doing. Russia has a recent and very painful history with many of those countries. And even if the WW2 push west from Russia was precipitated by Nazi Germany, nothing they did afterwards left a warm and fuzzy feeling.
The hand wringing is because after a long period of peace that people became accustomed too, this is a real and present danger to a lot of folks livelihoods and safety even if nothing goes nuclear.
And this turns up the dial on stress in a way that no one can look at it and really be sure the other side wouldn’t go nuclear over the next step or two.
Would he be super stupid and self destructive? Yes.
But when in an information bubble and when threatened by existential threats is exactly when folks are most likely to do stupid and self destructive things.
Would you be willing to bet there is no (crazier) MacArthur, potentially one that has access to a nuke or dirty bomb or the like, in the power structure somewhere there, or even in Ukraine if you really looked?
Madmen need to be checked. Getting nuked is also terrible. Starting WW3? Just yikes.
MAD still applies, disincentivizing a direct war between the US and Russia. US nuclear policy is unspecified, Russian policy explicitly calls for the use of nuclear weapons if the survival of the Russian state is on the line. Direct war with the US could very easily lead to the Russian state fearing for its future.
Thanks for your answer which is insightful.
As you explain there is asymmetry in the case of countries that are close to a border of one of the actors (either US or Russia).
So can we expect other similar moves from Russia into countries that are not part of NATO and which have no common frontiers with US?
They’ve been threatening countries like Finland, Sweden, etc. every time they consider it too.
When it was non-western aligned Soviet breakaway republics, it was easier to ignore.
The invasion of Ukraine was too obvious and ham handed. Moscow has doubled down on the threats to those on the fence since then, enough to push folks back over to the other side.
Which likely just pushes Russia more solidly into doubling down/existential threat mode.
According to Russia Russians protesting the war aren't really Russians either. But what Russia thinks is utterly irrelevant, it is no longer connected to objective reality.
Neither of the electronic warfare officials referenced in the article - Brig. Gen. Tad Clark, Dave Tremper in the office of SecDef — raised any hint of concern.
The US is sending artillery, those are heavy weapons by any reasonable definition. The main "red line" that has been repeated over and over is that NATO will not intervene with its own troops.
There is no objective standard and no court that decides what action will 'pull is into WWIII'. Maybe German tank shipments will be too much, maybe Starlink terminals, maybe a bad Putin meme. What crosses the line is solely decided by the Russian government, and they do not care at all whether coming generations will deem their actions legitimate or not.
In this case it is extremely dangerous to err on the side of caution, or Putin will be able to apply salami tactics and get whatever he wants.
If a Chinese / Russian company wants to sell internet access to US customers, why would I want the government to be able to tell me I don't have the right to connect to it.
If winning means more tens of thousands civilians dead then the logical thing is to not to want Ukraine to win. But, then again, most of the users from here are from half a world away, so those are only numbers for them, all that counts is the W.
So apparently you claim to be Romanian. Just one question: what is wrong with you? It seems like you don't understand that the orcs will attack us next, or you're an orc yourself pretending to be Romanian.
The what now? They didn’t invade us in August ‘68, they definetly won’t do it now, with the Americans on our side. And I’d ease it down with the warmongering rhetoric (again, “orcs”, really?) if I were you, because last time we thought we were militarily better than them, the Russians, we had our grand-fathers and grand-grand-fathers die horrible deaths from Odessa all the way to Kazan.
This is a former-USSR internal kerfuffle, it’s the “Yugoslav” civil war translated to the former Soviet space. Thankfully we’re a not a former Soviet republic. I wouldn’t want to live in Estonia or Lithuania right now, though.
Your advise came late... WWIII is already started few years ago, only most still haven't noticed. And yes large part of it was born out of corporate interest and their pernicious mix with the public.
You fail to spot the real enemy though: it's not a WWIII between Russia, China and USA/UK. It's a war between a little oligarchy and the rest of the humanity. War between USA/UK +submissive allies and Russia is just a war between two oligarch cohort, war for Taiwan and south Pacific in general (perhaps Solomon and Chagos Islands (see Diego Garcia) just toward the Philippines will probably follow again for more practical purposes like the China desperate need for food vs the USA/UK desperate struggle to keep dominance of the seas. Those wars though are not much WW wars, they are just some big&powerful confronts to remain in power/increase it. The real global war is the classic élites vs the people. Time since first nazi-fascism series against socialism is enough to restart a new nazi-fascist era and that's precisely what happen now.
I think you overlooked that Putin's Business Empire is itself private industry, funding itself via the work of the Russian people and land, and is trying to do just that.
Edit: your track record is already beyond the line at which we'd normally ban an account. I'm not going to ban you this time because you've also posted good comments, but it's past time to correct this, so please do.
You can't seriously be questioning how snarkily attacking someone for the ethnicity of their name is against the HN guidelines? That, plus all your other comments attacking other users, are way outside what's acceptable here. If that's not obvious to you after reading https://news.ycombinator.com/newsguidelines.html, I'm not sure what else to say.
If we didn't ban such accounts, we might as well not moderate this place at all, so please fix this.
I’m just pointing out the unsurprising coincidence that he is a German speaker repeating falsehoods German leadership is peddling to Germans exclusively in the German language.
Nowhere else in the world will you see anyone pushing the lie that Ukraine isn’t being supplied with heavy weapons.
I really don’t know how you can spin this into an attack. sschueller is simply a victim of propaganda, that’s not an unreasonable thing to point out. I just added useful context to the outrageous lie he is spreading.
If you can't or won't do that, we're going to have to ban you, even if you're right that your comments are perfectly innocent and everyone else is just reading them wrong. That's because we have to go by how these things affect the threads. If you post things that look like trollish swipes, and feel like trollish swipes, they're guaranteed to have the effect of trollish swipes. There isn't much difference between arson and negligence if the goal is not to have the house burn down.
What tanks specifically are they sending and how are they training the Ukrainians to operate them?
The Ukrainians do not own any US tanks and do not have any infrastructure to operate or maintain such tanks. If the US were to send such equipment it requires a lot more than just the tank. This is also why the US and others do not want to do this as it involves sending US personnel to train/coordinate etc.
US has not sent over any tanks to Ukraine. It is, however, sending tanks to Poland to replace their tanks sent to Ukraine, and has sent artillery to Ukraine.
Yes, exactly, this is very different than directly sending such weapons to Ukraine. It also has to be seen from the Polish view, free upgrade to US tanks. They also tried it with their aircraft hoping for US fighter jets in return.
> We are not sending heavy armament to Ukraine for a reason and the government needs to make sure that their policies also apply to companies like SpaceX.
First, that's wrong (apart from Germany), second, how does this reason (that Western tanks require training and logistics Ukrainians don't have) square with 'mak[ing] sure that their policies also apply to companies like SpaceX'? It's a complete nonsequitur if that's the case.
The notion that SpaceX is basically using a Full CI/CD pipeline on a massive satellite constellation is kind of amazing. Meanwhile most large government software requires months of contract negotiation before anyone is even hired to fix the problem.