• 6 Posts
  • 21 Comments
Joined 1 year ago
cake
Cake day: June 27th, 2023

help-circle






  • I haven’t used them in Spark directly but here’s how they are used for computing sparse joins in a similar data processing framework:

    Let’s say you want to join some data “tables” A and B. When B has many more unique keys than are present in A, computing “A inner join B” would require lots of shuffling if B, including those extra keys.

    Knowing this, you can add a step before the join to compute a bloom filter of the keys in A, then apply the filter to B. Now the join from A to B-filtered only considers relevant keys from B, hopefully now with much less total computation than the original join.















  • Focusing on code coverage (which doesn’t distinguish between more and less important parts of the code) seems like the opposite of your very good (IMO) recommendation in another comment to focus on specific high-value use-cases.

    From my experience it’s far easier to sell the need for specific tests if they are framed as “we need assurances that this component does not fail under conceivable usecases” and specially as “we were screwed by this bug and we need to be absolutely sure we don’t experience it ever again.”

    Code coverage is an OK metric and I agree with tracking it, but I wouldn’t recommend making it a target. It might force developers to write tests, but it probably won’t convince them. And as a developer I hate feeling “forced” and prefer if at all possible to use consensus to decide on team practices.


  • One aspect that does work is framing the need for tests as assurance that specific invariants are verified and preserved

    Agreed - this is the specific aspect which I hoped would be communicated by studying TDD a bit!

    The team is afraid that making changes will be more difficult when tests exist, but TDD (or maybe a more specific concept like you mentioned) demonstrates that tests make future changes easier.

    And I specifically advocated not to follow “write tests first”.

    Name-dropping concepts actually contributes to loose credibility of any code quality effort, and works against you.

    OK. If I were having an in-depth discussion with my team of fellow developers to convince them to start writing tests, I don’t think that’s name-dropping.


  • We can’t test yet, we’re going to make changes soon

    This could be a good opportunity to introduce the concept of test-driven development (TDD) without the necessity to “write tests first”. But I think it can help illustrate why having tests is better when you are expecting to make changes because of the safety they provide.

    “When we make those changes, wouldn’t it be great to have more confidence that the business logic didn’t break when adding a new technical capability?”

    You shouldn’t have to refactor to test something

    This seems like a reasonable statement and I sort of agree, in the sense that for existing production code, making a code change which only adds new tests yet also requires refactoring of existing functionality might feel a bit risky. As other commenters mentioned, starting with writing tests for new features or fixes might help prevent folks feeling like they are refactoring to test. Instead they’re refactoring and developing for the feature and the tests feel like they contribute to that feature as well.



  • I agree with how you characterized it and the term “ai engineer” didn’t resonate with me as defined by the author. If such an engineer doesn’t need to know about the data involved (“nor do they know the difference between a Data Lake or Data Warehouse”) then I don’t think they will be able to ship an AI/ML product based on data.

    New titles can be helpful for sorting out different roles with some shared skillsets such as the distinction which emerged between Data Scientist and ML Engineer at some companies to focus the latter on shipping production software using ML.