• RussianBot8453@lemmy.world
    link
    fedilink
    arrow-up
    14
    ·
    32 minutes ago

    I’m a data engineer that processes 2 billion row 3000 column datasets every day, and I open shit in Excel with more than 60k rows. What the hell is this chick talking about?

  • zalgotext@sh.itjust.works
    link
    fedilink
    arrow-up
    71
    arrow-down
    1
    ·
    3 hours ago

    my hard drive overheated

    So, this means they either have a local copy on disk of whatever database they’re querying, or they’re dumping a remote db to disk at some point before/during/after their query, right?

    Either way, I have just one question - why?

    • Bosht@lemmy.world
      link
      fedilink
      English
      arrow-up
      14
      ·
      46 minutes ago

      I’d much sooner assume that they’re just fucking stupid and talking out of their ass tbh.

    • GoodEye8@lemm.ee
      link
      fedilink
      English
      arrow-up
      4
      ·
      19 minutes ago

      My one question would be “How?”

      What the hell are you doing that your hard drives are overheating? How do you even know it’s overheating as I’m like 90% certain hard drives (except NVMe if we’re being liberal with the meaning of hard drive) don’t even have temperature sensors?

      The only conclusion I can come to is that everything he’s saying is just bullshit.

      • spooky2092@lemmy.blahaj.zone
        link
        fedilink
        English
        arrow-up
        40
        ·
        2 hours ago

        Plus, 60k is nothing. One of our customers had a database that was over 3M records before it got some maintenance. No issue with overheating lol

        • surph_ninja@lemmy.world
          link
          fedilink
          arrow-up
          20
          ·
          edit-2
          2 hours ago

          I run queries throughout the day that can return 8 million+ rows easily. Granted, it takes few minutes to run, but it has never caused a single issue with overheating even on slim pc’s.

          This makes no fucking sense. 60k rows would return in a flash even on shitty hardware. And if it taxes anything, it’s gonna be the ram or cpu- not the hard drive.

        • AThing4String@sh.itjust.works
          link
          fedilink
          arrow-up
          7
          ·
          1 hour ago

          I literally work with ~750,000 line exports on the daily on my little Lenovo workbook. It gets a little cranky, especially if I have a few of those big ones open, but I have yet to witness my hard drive melting down over it. I’m not doing anything special, and I have the exact same business-economy tier setup 95% of our business uses. While I’m doing this, that little champion is also driving 4 large monitors because I’m actual scum like that. Still no hardware meltdowns after 3 years, but I’ll admit the cat likes how warm it gets.

          750k lines is just for the branch specific item preferences table for one of our smaller business streams, too - FORGET what our sales record tables would look like, let alone the whole database! And when we’re talking about the entirety of the social security database, which should contain at least one line each in a table somewhere for most of the hundreds of millions of people currently living in the US, PLUS any historical records for dead people??

          Your hard drive melting after 60k lines, plus the attitude that 60k lines is a lot for a major database, speaks to GLARING IT incompetence.

        • Korhaka@sopuli.xyz
          link
          fedilink
          English
          arrow-up
          1
          ·
          edit-2
          37 minutes ago

          Pretty sure I run updates or inserts that count over 60k fairly often. No overheats. Select queries sometimes way higher.

      • baldingpudenda@lemmy.world
        link
        fedilink
        arrow-up
        1
        ·
        8 minutes ago

        Reminds me of those 90s ads about hackers making your pc explode.

        Musk gonna roll up in a wheelchair, “the attempt on my life has left me ketamine addicted and all knowing and powerful.”

      • Dave.@aussie.zone
        link
        fedilink
        arrow-up
        11
        arrow-down
        1
        ·
        2 hours ago

        You’ve got it all wrong, in traditional computer terminology the “hard drive” is the box that sits under the desk that collects cat fluff and cigarette tar.

        /s …?

    • Adalast@lemmy.world
      link
      fedilink
      arrow-up
      4
      arrow-down
      1
      ·
      2 hours ago

      Why? Because they feel the need to have local copies of sensitive financial information because… You know… They are computer security experts.

  • madeinthebackseat@lemmy.world
    link
    fedilink
    arrow-up
    35
    ·
    3 hours ago

    As a reasonably experienced “data guy,” this seems obviously laughable, but the discussion on X is scary. This guy is a savior in the MAGA world.

    We can criticize and poke fun all day, but it doesn’t matter much if our message isn’t challenging the mindset of those with other opinions.

    How do we make better use of our time to impact outside opinion?

    • jjjalljs@ttrpg.network
      link
      fedilink
      arrow-up
      1
      ·
      18 minutes ago

      I’ve been told violence isn’t the answer and we shouldn’t just shoot nazis and nazi enablers dead.

      The way most people change their mind isn’t based on facts or figures, but emotions. Specifically, in-group belonging. For most people, and this certainly includes me and you some of the time, what our in-group believes is more compelling than an out-groups supposed facts.

      They see that guy as someone in their group so they believe him. They see you as a bad outside bad bad bad liar, so nothing you say is likely to get through. (This comic is worth reading on this topic: https://theoatmeal.com/comics/believe )

      If you want to change someone’s mind, they have to see you as in-group. Not necessarily the same group as what you’re arguing with. We all belong to many groups. American, new yorker, white guy, middle aged, yankees fan, etc etc there are many such slices. Like how you can’t get a republican to recycle by appealing to environmental concerns (because environmentalists are out-group, so fuck them), but you might be able to get them to recycle via something like “only american ingenuity can turn trash into bridges and tanks!”

      This takes a lot of time and effort, and if you don’t get them to stop hanging out with the other group, you won’t make any lasting changes.

      So I think you’d need a multi prong approach:

      • Get them off bad media. Facebook, fox news, etc. This is reinforcing their bad beliefs. Because they see this stuff as trustworthy in-group, it goes right into the worldview.
      • Get them to stop hanging out with their shitty maga-hat friends. This is the social in-group that’s reinforcing bad beliefs.
      • Get them to trust you.
      • Gently introduce the idea that maybe the extreme right doesn’t have their interests at heart, etc

      All of which takes a lot of time and effort, and your opposite number is basically trying to do the same thing. Except they have fox news, trump, and such in their corner.

      And, again, I’m told we definitely shouldn’t just shoot extreme right wingers and other nazi sympathizers dead. Nor should we burn their houses down. If we’re an emergency responder, we definitely shouldn’t let them die while thinking to ourselves “they would let so many die. without a thought, their passing deserves no mourning” or similar.

      You should definitely nullify if you’re on a jury and someone allegedly did violence to a shitty ceo or red-hat, though, bu that’s getting off topic.

    • lucster@lemm.ee
      link
      fedilink
      English
      arrow-up
      3
      ·
      1 hour ago

      Compelling point. I just found that arguing with „these kind of people“ (livibg in europe, so no MAGA‘s here but like-minded, conservative fundamentalists etc.) leads to nowhere. It‘s kind of like the covid-conversations. And often I heard „you can‘t make them change their minds, so just let them be“. Still, I think this behaviour leads to isolation and separates us as a people even more.

      Long story short: good question. If you found the answer, let me know.

    • _stranger_@lemmy.world
      link
      fedilink
      arrow-up
      21
      ·
      edit-2
      3 hours ago

      They make nothing. They’re compensated for destroying things, and considering it’s musk, they’re likely given relatively little money in return for their time.

      Even if the only thing you do all day is sit on the toilet and yell at the Internet, you’re already a bigger net positive on society.

        • jjjalljs@ttrpg.network
          link
          fedilink
          arrow-up
          2
          ·
          17 minutes ago

          I would be absolutely shocked if we had anything approaching justice for what this administration is doing.

          We barely got anything for that whole ass insurrection attempt.

        • trafficnab@lemmy.ca
          link
          fedilink
          arrow-up
          5
          ·
          1 hour ago

          The reason he recruited a bunch of incompetent college kids is because nobody with any experience or wisdom would touch this rolling crime wave with a 10 foot pole

      • ironcrotch@aussie.zone
        link
        fedilink
        English
        arrow-up
        3
        ·
        2 hours ago

        Most likely compensation is the promise of being part of privatization of whatever the fuck they’re destroying.

        • zarkanian@sh.itjust.works
          link
          fedilink
          arrow-up
          6
          ·
          1 hour ago

          They’re dumb kids. They probably see Musk as some kind of god. He gave them some line like “Hey, you guys wanna save the United States?” and they jumped at the chance.

          You get to be a piece of history! Whheeeeeee

  • vga@sopuli.xyz
    link
    fedilink
    arrow-up
    15
    ·
    3 hours ago

    I’ve used local hard drives from like 1992 and I have never ever gotten them to overheat.

  • Psaldorn@lemmy.world
    link
    fedilink
    arrow-up
    123
    arrow-down
    1
    ·
    5 hours ago

    From the same group that doesn’t understand joins and thinks nobody uses SQL this is hardly surprising .

    Probably got an LLM running locally and asking it to get data which is then running 10 level deep sub queries to achieve what 2 inner joins would in a fraction of the time.

    • _stranger_@lemmy.world
      link
      fedilink
      arrow-up
      39
      ·
      edit-2
      3 hours ago

      You’re giving this person a lot of credit. It’s probably all in the same table and this idiot is probably doing something like a for-loop over an integer range (the length of the table) where it pulls the entire table down every iteration of the loop, dumps it to a local file, and then uses plain text search or some really bad regex’s to find the data they’re looking for.

      • indepndnt@lemmy.world
        link
        fedilink
        arrow-up
        3
        ·
        54 minutes ago

        I think you’re still giving them too much credit with the for loop and regex and everything. I’m thinking they exported something to Excel, got 60k rows, then tried to add a lookup formula to them. Since you know, they don’t use SQL. I’ve done ridiculous things like that in Excel, and it can get so busy that it slows down your whole computer, which I can imagine someone could interpret as their “hard drive overheating”.

      • morbidcactus@lemmy.ca
        link
        fedilink
        arrow-up
        17
        ·
        2 hours ago

        Considering that is nearly exactly some of the answers I’ve received during the technical part of interviews for jr data eng, you’re probably not far off.

        Shit I’ve seen solutions done up that look like that, fighting the optimiser every step (amongst other things)

      • makingStuffForFun@lemmy.ml
        link
        fedilink
        arrow-up
        3
        ·
        edit-2
        2 hours ago

        I have to admit I still have some legacy code that does that.

        Then I found pandas. Life changed for the better.

        Now I have lots if old code that I’ll update, “one day”.

        However, even my old code, terrible as it is, does not overheat anything, and can process massively larger sets of data than 60,000 rows without any issue except poor efficiency.

  • Tiefling IRL@lemmy.blahaj.zone
    link
    fedilink
    arrow-up
    111
    ·
    5 hours ago

    60k isn’t that much, I frequently run scripts against multiple hundreds of thousands at work. Wtf is he doing? Did he duplicate the government database onto his 2015 MacBook Air?

    • socsa@piefed.social
      link
      fedilink
      English
      arrow-up
      6
      ·
      2 hours ago

      I’ve run searches over 60k lines of raw JSON on a 2015 MacBook air without any problems.

        • Korhaka@sopuli.xyz
          link
          fedilink
          English
          arrow-up
          1
          ·
          29 minutes ago

          I have an sqlite db that is a few GB in size, game saves using the format. Sadly almost all blob data, would love to play with it if it was a bit more readable

    • 4am@lemm.ee
      link
      fedilink
      arrow-up
      61
      ·
      5 hours ago

      A TI-86 can query 60k rows without breaking a sweat.

      If his hard drive overheated from that, he is doing something very wrong, very unhygienic, or both.

    • arotrios@lemmy.world
      link
      fedilink
      English
      arrow-up
      6
      ·
      3 hours ago

      Seriously - I can parse multiple tables of 5+ million row each… in EXCEL… on a 10 year old desktop and not have the fan even speed up. Even the legacy Access database I work with handles multiple million+ row tables better than that.

      Sounds like the kid was running his AI hamsters too hard and they died of exhaustion.

        • arotrios@lemmy.world
          link
          fedilink
          English
          arrow-up
          1
          ·
          22 minutes ago

          You’re correct - the standard tabs can only hold roughly 1.2 million rows.

          The way to get around that limitation is to use the Data Model within Power Pivot:

          It can accept all of the data connections a standard Power Query can (ODBC, Sharepoint, Access, etc):

          You build the connection in Power Pivot to your big tables and it will pull in a preview. If needed, you can build relationship between tables with the Relationship Manager. You can also use DAX to build formulas just like in a regular Excel tab (very similar to Visual Basic). You can then run Pivot Tables and charts against the Data Model to pull out the subsets of data you want to look at.

          The load times are pretty decent - usually it takes 2-3 minutes to pull a table of 4 million rows from an SQL database over ODBC, but your results may vary depending on datasource. It can get memory intensive, so I recommend a machine with a decent amount of RAM if you’re going to build anything for professional use.

          The nice thing about building it out this way (as opposed to using independent Power Queries to bring out your data subsets) is that it’s a one-button refresh, with most of the logic and formulas hidden back within the Data Model, so it’s a nice way to build reports for end-users that’s harder for them to fuck up by deleting a formula or hiding a column.

    • IsoKiero@sopuli.xyz
      link
      fedilink
      English
      arrow-up
      9
      arrow-down
      1
      ·
      edit-2
      4 hours ago

      Don’t know what Elmos minions are doing, but I’ve written code at least equally unefficient. It was quite a few years ago (the code was in written in perl) and I at least want to think that I’m better now (but I’m not paid to code anymore). The task was to pull in data from a CSV (or something like that, as I mentioned, it’s been a while) and it needed conversion to XML (or something similar).

      The idea behind my code was that you could just configure which fields you want from arbitary source data and on where to place them on the whatever supported destination format. I still think that the basic idea behind that project is pretty neat, just throw in whatever you happen to have and have something completely else out of the other end. And it worked as it should. It was just stupidly hungry for memory. 20k entries would eat up several gigabytes of memory from a workstation (and back then it was premium to have even 16G around) and it was also freaking slow to run (like 0.2 - 0.5 seconds per entry).

      But even then I didn’t need to tweet that my hard drive is overheating. I well understood that my code is just bad and I even improved it a bit here and there, but it was still so very slow and used ridiculous amounts of RAM. The project was pretty neat and when you had few hundred items to process at a time it was even pretty good, there was companies who relied on that code and paid for support. It just totally broke down with even a slightly bigger datasets.

      But, as I already mentioned, my hard drive didn’t overheat on that load.

    • vga@sopuli.xyz
      link
      fedilink
      arrow-up
      1
      ·
      edit-2
      3 hours ago

      I mean if we were to sort of steelman this thing, there sure can be database relations and queries that hit only 60k rows but are still hteavy as fuck.

  • golden_zealot@lemmy.ml
    link
    fedilink
    English
    arrow-up
    31
    ·
    edit-2
    5 hours ago

    I used to perform data analysis of robotics firmware logs which would generate several million log lines per hour and that was my second job out of college.

    I don’t know how you fuck up 60k lines that bad. Is he nesting 150 for loops and loading a copy of the data set in each one while mining crypto??

  • kryptonianCodeMonkey@lemmy.world
    link
    fedilink
    arrow-up
    64
    ·
    edit-2
    5 hours ago

    This shit sounds like when you’re mom tells you that the Facebook printed out her bank statement on the tax machine. I’m not smart enough to even guess how you did something dumb enough to make that happen.

    How bad are you at writing queries? How does your hard drive overheat even under 100% load? Do you have it smothered under a blanket? Did you crack it up and expose it to cheeto dust? What does running a query on your, presumably, remote database even have to do with your harddrive in the first place? Are you trying to copy the entire database locally to a laptop? Do you know how to tie your shoes yet, or are you still on the velcro?

    • The Bard in Green@lemmy.starlightkel.xyz
      link
      fedilink
      arrow-up
      3
      ·
      edit-2
      33 minutes ago

      This shit sounds like when you’re mom tells you that the Facebook printed out her bank statement on the tax machine.

      My dear sweet mother asked me somewhere around 2005-06 “If we can fax paper, why not groceries, or pizza delivery?”

      Apparently she had believed, for decades, that fax machines literally transported physical paper over phone lines. She has a college degree, and my family is wealthy.

      Do not underestimate the mind boggling technical and scientific ignorance of old people who should know better.

    • floofloof@lemmy.ca
      link
      fedilink
      arrow-up
      14
      ·
      4 hours ago

      A laptop should easily handle a database of 60,000 rows. I run much bigger databases on my own laptop for development purposes.