• DigitalDilemma
    link
    fedilink
    English
    665 months ago

    I lost a day’s holiday, and our team spent 8 man days on this entirely preventable mistake.

    $10? Try extending our licence by another year for free, that might start going towards it.

    • @MrMcGasion@lemmy.world
      link
      fedilink
      125 months ago

      Why would you want another year of their software for free? This is their second screw up (apparently they sent out a bad update that affected some Debian and RHEL machines a couple years ago). I’d be transitioning to a competitor at the first opportunity. It seems they aren’t testing releases before pushing them out to customers, which is about as crazy to me as running alpha software on a production system.

      I’m sure you have reasons, and this isn’t really meant to be directed at you personally, it’s just boggling to me that the IT sector as a whole hasn’t looked at this situation and collectively said “fuck that.”

      • DigitalDilemma
        link
        fedilink
        English
        55 months ago

        Why would you want another year of their software for free?

        Because AV, like everything else, costs a fortune at enterprise scale.

        And yeah, I do understand your real point, but it’s really hard to choose good software. Every purchasing decision is a gamble and pretty much every time you choose something it’ll go bad sooner or later. (We didn’t imagine Vmware would turn into an extortion racket, for example. And we were only saying a few months ago how good value and reliable PRTG was, and they’ve just quadrupled their costs)

        It doesn’t matter how much due diligence and testing you put into software, it’s really hard to choose good stuff. Crowdstrike was the choice a year ago (the Linux thing was more recent than that), and its detection methods remain world class. Do we trust it? Hell no, but if we change to something else, there are risks and costs to that too.

        • @xavier666@lemm.ee
          link
          fedilink
          English
          35 months ago

          Do we trust it? Hell no, but if we change to something else, there are risks and costs to that too.

          Unfortunate reality for lot for medium to big size businesses.

        • @ayyy@sh.itjust.works
          link
          fedilink
          15 months ago

          Maybe AV, at an enterprise scale, is actually a horrible idea that reduces security, availability, and reliability and should be abolished through policy.

          • DigitalDilemma
            link
            fedilink
            English
            15 months ago

            Maybe, but it’s not going to happen soon. Any malware type insurance requires effective AV on all devices, and C-levels do love their insurance.

        • Scrubbles
          link
          fedilink
          English
          15 months ago

          Nah, I don’t buy that. When you’re in critical infrastructure like that it’s your job to anticipate things like people being above or below versions. This isn’t the latest version of flappy bird, this is kernel level code that needs to be space station level accurate, that they’re pushing remotely to massive amounts of critical infrastructure.

          I won’t say this was one guy, and I definitely don’t think it was malicious. This is just standard corporate software engineering, where deadlines are pushed to the max and QA is seen as an expense, not an investment. They’re learning the harsh realities of cutting QA processes right now, and I say good. There is zero reason a bit of this magnitude should have gone out. I mean, it was an empty file of zeroes. How did they not have any pipelines to check that file, code in the kernel itself to validate the file, or anyone put eyes on the file before pushing it.

          This is a massive company wide fuckup they had, and it’s going to end up with them reporting to Congress and many, many courts on what happened.

          • suoko
            link
            fedilink
            15 months ago

            Even an AI is good enough to avoid (or let someone avoid) pushing a similar bug 🫣

        • DigitalDilemma
          link
          fedilink
          English
          15 months ago

          Not just Crowdstrike - any vendor that does automatic updates, which is more and more each day. Microsoft too big for a bad actor to do as you describe? Nope. Anything relying on free software? Supply chain vulnerabilities are huge and well documented - its only a matter of time.