Aggregates

Evidence: High: 0 Medium: 1 Low: 0 No Data: 30

Theme Radar

bluejekyll 2017-01-18 15:45 UTC link

No. You are correct. Honestly I think you can solve a lot of that by following on from one of Deijkstra's core priniciples: Seperation of Concerns.

When you practice good seperationof concerns, specific choice in different areas can be more easily fixed later. It requires having decent APIs and being thoughtful on the interaction of different components, but it helps immensely in the long run.

Microservices are one way to practice seperation of concerns, but it can also be practiced in monolithic software as well, by having strong modular systems (different languages are stronger at this than others).

PaulHoule 2017-01-18 15:49 UTC link

Continuous integration is a good thing. Back in the bad old days you'd have three people working on parts of the system for 6 months and plan to snap them together in 2 weeks and it would take more like another 6 months.

Agile methods are also useful. If you can't plan 2 weeks of work you can probably not plan 6 months.

When agile methods harden into branded processes and where there is no consensus on the ground rules by the team it gets painful. The underlying problem is often a lack of trust and respect. In an agile situation people will stick to rigid rules (never extend the sprint, we do all our planning in 4 hours, etc.) because they feel they'll lose what little control they have otherwise. In a non-agile situation people can often avoid each other for months and have the situation go south suddenly. In agile you wind up with lots of painful meetings instead.

Also I think it is rare for one language to really be "best for a job". If you want to write the back end of a run of the mill webapp, you can do a great job of that in any mainstream language you are comfortable in.

jMyles 2017-01-18 15:50 UTC link

My first reaction to your (very thoughtful) review is that #4 seems out of place.

CI can be a way of enforcing the simplicity of the others - it can be a way of tunneling the build process into assuredly straightforward steps and preventing individual team members from arbitrarily (or even accidentally) adding their own complications into build requirements.

Other than that, I think you are definitely on to something here.

dwc 2017-01-18 15:50 UTC link

Much of the problem in the things you mention is that those things are specific solutions that have been confused with goals. I.e., "we're supposed to build microservices" is a horrible idea, as opposed to "given this particular situation a microservice is a great fit".

Understanding the possible benefits and drawbacks of any solution is important. It's important in whether or not that solution is selected, but also to make sure that the implementation actually delivers those benefits.

It's very common in our industry to use "best practices" without understanding them, and therefore misapplying the solutions.

rolodato 2017-01-18 16:00 UTC link

I think the "learn to code" movement as well as overly-technical interviews for developers are partly to blame for this. It's well-known that developers are tested on how to do something that's considered technically difficult, such as abstract CS problems or a complicated architecture, but they are rarely asked why certain tools, practices or architectures should or should not be used. Comparative analyses to make objective recommendations between different solution alternatives are also rare in my interviewing experience, but they are one of the most valuable skill a competent software engineer should have.

I don't agree on point 4 though - CI can be something as basic as running a monolith's tests on each commit, which makes sure that builds are reproducible (no more "works on my machine").

majewsky 2017-01-18 16:03 UTC link

1) False dichotomy. Developer familiarity is one of the most important metrics for choosing "the best tool for the job".

2) Conway's Law applies in reverse here: If your organization consists of a lot of rather disjoint teams, then microservices can be quite beneficial because each team can deploy independently. If you're one cohesive team, there is not much benefit, only cost.

3) Depends. If you have a well-designed distributed system, it can be amazingly resilient and reliable without introducing much administrative overhead. (From my experience, OpenStack Swift is such a system. Parts may fail, but the system never fails.) There are two main problems with distributed systems: a) Designing and implementing them correctly is really hard. b) Many people use distributed systems when a single VM would do just fine, and get all the pain without cashing out on the benefits. See also http://idlewords.com/talks/website_obesity.htm#heavyclouds

4) Continuous integration was not meant to help with complexity. Its purpose is to reduce turn-around time for bugfixes and new features. If your release process is long and complicated, the increased number of releases will indeed be painful for you. Our team sees value in "bringing the pain forward" in this way. Your team obviously puts emphasis on different issues, and that's okay.

corysama 2017-01-18 16:04 UTC link

One of my fav tech talks ever (and I watch a lot of tech talks) is Alan Kay's "Is it really 'complex'? Or, did we just make it 'complicated'?" It addresses your question directly, but at a very, very high level.

https://m.youtube.com/watch?v=ubaX1Smg6pY

Note that the laptop he is presenting on is not running Linux/Windows/OSX and that the presentation software he is using is not OoO/PowerPoint/Keynote. Instead, it is a custom productivity suite called "Frank" developed entirely by his team, running on a custom OS, all compiled using custom languages and compilers. And, the total lines of code for everything, including the OS and compilers, is under 100k LOC.

BjoernKW 2017-01-18 16:04 UTC link

No, it's not just you and yes, we often do overcomplicate software development.

It's been that way long before agile methodology or microservices though. Complexity-for-the-sake-of-complexity EverthingHasToBeAnAbstractClass frameworks have been plaguing the software development business since at least the 1990s and I'm sure there are similar stories from the 80s and 70s.

It's hard to find a one-size-fits-all easy method for not falling into that over-engineering / over-management trap. I try to focus on simple principles to identify needless complexity:

- There is no silver bullet (see "microservices"): If the same design pattern is used to solve each and every problem there probably is something amiss.

- Less code is better.

- Favour disposable code over reusable code: Avoid the trap of premature optimisation, both in terms of performance and in terms of software architecture. Also known as "You aren't gonna need it".

- Code means communication: By writing code you’re entering a conversation with other developers, including your future self. If code isn't easily comprehensible again there's likely something wrong.

mhotchen 2017-01-18 16:07 UTC link

Many of the programmers I have worked with actually love complexity, despite trying to convince others (and most likely themselves) that they hate it.

Advice tends to be cherrypicked to suit an agenda they already have (with your example on microservices, the vast amount of resources saying they're very difficult, should be driven by a monolith first approach, and solve a specific set of problems is largely brushed under the rug).

I think because our industry moves so fast there's a fear of becoming irrelevant. Ironically companies are so scared of not being able to employ developers that they're also onboard with complicating their platform in the name of hiring and retention. I think this is down to the sad truth that most developer roles offer very little challenge outside of learning a new stack.

rb808 2017-01-18 16:08 UTC link

I've seen the addition of unit testing is a big cause of complexity. Previously simple classes now have to be more abstracted in order to unit test. Add mocks, testing classes & test frameworks. Some unit tests are handy, but I dont think it justifies the additional complexity. For the apps I write I'd like to see more emphasis on automated integration testing and fewer unit tests - so we can write simple classes again.

emeraldd 2017-01-18 16:18 UTC link

- 1) Choose languages that developers are familiar with, not the best tool for the job

95% of the time, a language that your developers are familiar with is the correct tool for the job simply for that reason! There are cases where it is not the case but those involve special case languages and special case systems. If you don't know what special case means then you're situation is almost certainly on that list.

- 2) Avoid microservices where possible, the operational cost considering devops is just immense

"If your data fits on one machine then you don't need hadoop ..." Same thing applies here. Microservices have place and putting them in the wrong one will bite you bad.

- 3) Advanced reliability / redundancy even in critical systems ironically seems to causes more downtime than it prevents due to the introduction of complexity to dev & devops.

Then there's probably something wrong or limited with the deployment that needs to be reviewed (2 node when you need a 3 node cluster, bad networking environments, etc.) If you have a reasonable setup with solid tech under it, deployed per specs then this should not be true. If, on the other hand, something is out of whack (say running a 2 node cluster with Linux HA and only a single communication path between them) you're going to have problems and the only way to fix them is to get it done right.

- 4) Continuous integration seems to be a plaster on the problem of complex devops introduced by microservices.

I'm not sure about this but, if your deployment system requires CI you have a problem. An individual, given hardware and assets/code, should be able to spin up a complete system on a fresh box cleanly and in a reasonable timeframe. (Fresh data restores can take longer of course but the system should be runnable barring that.) If this requires (i.e. it can't reasonably be done manually) something like an CI script or ansible/chef/etc. script then you're deployment process is probably too complex and needs to be re-evaluated.

- 5) Agile "methodology" when used as anything but a tool to solve specific, discrete, communications issues is really problematic

Agile is commonly used to gloss over a complete lack of structured process or a broken. Even with Agile there should be some clean process and design work that goes into things or you're hosed.

kekub 2017-01-18 16:20 UTC link

I am working in huge non-IT company as a software developer. I guess that is what gives me a totally different point of view on your lessons:

1) Without a unified technology stack and a common framework we would not be able to build and maintain our applications. We decided on C# as it works best for us. Currently we are 5 developers. Not a single one of us has ever written a line of C# code before entering the company - learning the language from ground up enables us to pick up patterns that our colleagues who joined the company earlier found to be best practices.

2) If you are not introducing a whole new stack with every micro service that you develop the devops costs are quite low.

3) I agree with you on that - I think redundancy always introduces more complexity. However there are systems that handle that job quite well (e.g. SQL Server). For application servers we use hot-spares and a load balancer that only routes traffic to them, when the main servers are not reachable. This works for us, as all our applications are low traffic applications.

4) Continuous integration works brilliant for our unified stack. In the last two years we went down from 1d setup + 20min deploy to 10min setup + 20s deploy.

5) We use agile methodology whenever possible and it works like a charm. However we had a lot of learnings. Most recent example: Always have at least one person from all your target groups in any meeting where you try to create user-stories.

Planning our software architecture has been a key element in my teams success and I do not see a point where we are going to cut it.

sebringj 2017-01-18 16:21 UTC link

It never seems complicated when I am doing my own side work for some reason. There are no design meetings, no hours tracking, no arguments on best practices, no scrum, no testing frameworks, dev ops, etc. I do use git and minimally create bash scripts to simplify repetitive tasks for deployment but its just a huge contrast to working in teams where something simple takes about 50 times longer.

I think keeping things as simple as possible and always going for that goal will increase velocity overall. Everything should be subject to scrutiny for promoting productivity and open to modification or removal. I know there is a balance where you have to increase complexity in a team environment but keeping friction as low as possible in terms of process and intellectual weight couldn't hurt.

The most productive place I've seen so far is a huge athletic brand I worked for where they kept teams at max 5 people in mini projects. This forced the idea of low overhead and kept the scale of management needed small. The worst place I worked for in terms of unnecessary complexity is a well known host, although it is the best place to work in terms of people, hired offshore that has a one-size fits all mentality and layered in as much shit as possible to slow down development to a mud crawl. I don't buy into process over productivity.

SatvikBeri 2017-01-18 16:37 UTC link

Many of these practices are popularized by Google/Facebook/Amazon but don't make sense for a company with 100 or even 1,000 people. I try to focus on whether a practice will solve a concrete problem we're facing.

Switching from Hadoop to Spark was clearly a good idea for our team, even though it required learning a new stack, but there isn't a strong reason to switch to Flink or start using Haskell.

Agile makes sense when your main risk is fine-grained details of user requirements, but not when you have other substantial risks, such as making sure a statistical algorithm is accurate enough.

Microservices probably reduces the asymptotic cost of scaling but add a huge constant factor.

Relational databases are the right choice 95% of the time, non-relational stores require a really specific use case.

TDD is good for fast feedback in some domains, but for others, manually investigating the output or putting your logic into types is better. E.g. a lot of my time comes from scaling jobs that work on 10gb of data but crash on 1tb, TDD is not that helpful here.

Continuous integration mostly makes sense when you're making a lot of small changes and can reliably expect a test suite to catch issues.

In short, ask the question "when is practice X useful?" instead of "is practice X a good idea?"

hd4 2017-01-18 16:46 UTC link

For me, the trinity of development as a solo developer seems to be:

1. Writing code while using as many useful libraries and tools as possible to avoid recreating wheels

2. Continuous integration set up early on to handle the menial work and to let me concentrate on 1.

3. Constantly evaluating and researching what technology is available and newly appearing to give me an edge, because having an edge is never a bad thing in this field.

Agree with some of what OP said, especially with methodologies become hindrances and HA tools becoming points of failure.

mschaef 2017-01-18 16:54 UTC link

Continuous Integration is (with a reasonable test suite) one of few elements of software development that I would consider almost essential for any long running project. It's just too useful to have continual feedback on the quality of the system under construction. (And this is before bringing in micro-services or any other complicating architectural pattern.)

Where I might agree with you more are on points 3 and 4: 'Advanced reliability' and 'Microservices'. While I have no doubt that these are useful to solve specific problems, I think as a profession we tend to over-estimate the need for these things and under-estimate the costs for having them. To me this implies that there needs to be a very clear empirical case that they support a requirement that actually exists. I'd also make the argument that the drive for microservices within an organization has to come from a person or team that has the wherewithal to commit resources over the long-term to actually make it happen and keep it maintained. (ie: probably not an individual development team.)

marcosdumay 2017-01-18 16:54 UTC link

Well, yes, we are overcomplicating it. Except on the parts we are undercomplicating... And I still couldn't find anybody that can reliably tell those apart, but the first set is indeed much larger.

1 - Do not pick a new language for an urgent project. Do look at them when you have some leeway.

2 - Yep.

3 - There's something wrong with your ops. That happens often, and it is a bug, fix it.

4 - If CI is making your ops more complex, ditch it. If less complex, keep it. In doubt, choose the safest possible way to try the other approach, and look at the results.

5 - Do not listen to consulting experts, only to technical experts. The agile manifesto is a nice reading, read it, think about it, try to follow, but don't try too hard. Ignore any of the more detailed methodologies.

ChuckMcM 2017-01-18 17:27 UTC link

Yes we are over complicating it, but that it primarily about trying to take what is essentially an artistic process and turning it into a regimented process (a known hard problem).

Rob Gingell at Sun stated it as a form of uncertainty principal. He said, "You can know what features are in a release or when the release will ship, but not both." It captured the challenge of aspirational feature development where someone says "we have to have feature X" and so you send a bunch of smart engineers off to build it but there is no process by which you can start with an empty main function and build it step by step into feature X.

That said, it got worse when we separated the user interface from the product (browser / webserver). And you're rants about microservices and continuous integration are really about releases, delivery, and QA. (the 'delivery time' of Gingell's law above).

These are complexities introduced by delivery capabilities that enable different constructions. The story on HN a few days about about the JS graphics library is a good example of that. Instead of linking against a library on your computer to deliver your application with graphics, we have the capability of attaching to a web service with a browser and assembling on demand the set of APIs and functions needed for that combination of client browser / OS. Its a great capability but to pull it off requires more moving parts.

beat 2017-01-18 17:37 UTC link

1. What problem are you optimizing for? "The job" encompasses code, but it also encompasses staffing. It's a lot easier to hire Java developers than Scala developers. In a leadership role, your responsibility isn't just the day-to-day code - it's the whole project.

2. Microservices vs monoliths is a see-saw. You build a monolith, find it's a brittle, incomprehensible hairball, and you break out microservices. You build microservices, find that operational headaches are killing you, and start consolidating them into monoliths. Which kneecap do you want the bullet in?

3. Fix what breaks.

4. Continuous integration is vital. But it needs to be evolved along with the system. There's this thing I say... "Have computers do what computers do well, have humans do what humans do well". Handling complex and repeatable behavior (i.e. builds and test suites) should absolutely be automated as much as possible. Think continuous integration sucks? Try handing it off to humans for a while! You'll learn whole new levels of pain.

5. All process is about (or should be about) specific, discrete communications issues.

DanielBMarkham 2017-01-18 20:36 UTC link

You've thrown together a bunch of buzzwords and asked if we are over complicating things.

Buzzwords can mean freaking anything. I've seen great Agile teams that don't look anything like textbook Agile teams. Microservices can be a total clusterfuck unless you know what the hell you're doing -- and manage complexity. (Sound familiar?) CI/CD/DevOps can be anything from a lifesaver to the end of all life in the known universe.

So yes, we are over complicating software development, but the way we do it isn't through slapping around a few marketing terms. The way we do it is not understanding what our jobs are. Instead, we pick up some term that somebody, somewhere used and run with it.

Then we confuse effort with value. Hey, if DevOps is good, the more we do DevOps, the better we'll be, right? Well -- no. If Agile is good, the more Agile stuff we do the better we'll be, right? Hell no. We love to deep dive in the technical details. If there aren't any technical details, we'll add some!

Software development is too complicated because individual developers veer off the rails and make it too complicated. That's it. That's all there is to it. Throw a complex library at a good dev and they'll ask if we need the entire thing to only use 2 methods. Throw a complex library at a mediocre Dev and they'll spend the next three weeks writing 15 KLOC creating the ultimate system for X, which we don't need right now and may never need.

It has nothing to do with the buzzwords, the tech, or software development in general. It's us.

reacharavindh 2017-01-18 16:10 UTC link

> Agile methods are also useful. If you can't plan 2 weeks of work you can probably not plan 6 months.

Hmmm. I was just thinking the opposite yesterday. I'm a performance engineer working closely with two teams. One doing Agile and the other basing on wikis and Adhoc in-person whiteboard discussions. I find the non agile team more productive, efficient and dare I say happy. The Agile based team makes me sit in on their daily scrum meetings. Although every one uses it to sync up on their dependancies, it just drags for an hour almost every day. I can visibly tell the devs walking out of the room spend more time worrying about "velocity" and "organisation of work" than the money making work that needs to be done. It almost feels like the agile process gives them "one more job" of picking the doable things from the list of stuff that needs to be done so they look better than their peers with better velocity.

Simply put, I was thinking if Agile is just not a good method when you can strive for good leadership and a healthy collaboration among individuals of the team?

UK-AL 2017-01-18 16:23 UTC link

I find microservices can help in just keep everything small and focused. I know you can do this with a monolith. But having a process boundary really enforces it.

julsonl 2017-01-18 17:13 UTC link

In my experience, it's usually not the existence of unit tests themselves that's causing an issue, but that most of them are badly written. One telltale sign is when writing the unit test becomes overly painful (like too much code setting up mocks), it usually means that your class is not simple enough or has too many dependencies.

Proper unit testing also complements integration testing in that corner cases can be handled at the unit test level, therefore reducing the amount of integration test code which arguably is much more brittle, runs slower and more complicated to write.

throwawayish 2017-01-18 17:36 UTC link

It's interesting to work alone on a big-ish project with no one telling you what to do and not having to explain anything to anyone. It easily feels ten-times more productive (in terms of accomplishment), but then again it won't have a business case and one doesn't get paid, either.

I think I'm least productive in open source (again, in terms of felt accomplishment), because if one isn't the sole maintainer (like above), then it's a pretty safe bet that few changes take less than a certain baseline (eg. 1 hour) -- someone always has a nitpick, CI always takes it's sweet time, oh, did we discuss yet in which branches we wanna merge this? Ah, please avoid puns in documentation and comments. Do we want this? Can you write this differently, like ...? Did you manually test this or that scenario ...?

(Now this also has advantages in terms of stability, quality and consistency -- but it's also obviously far, far less efficient)

On the clock it's more like "Meh, change that and that, otherwise it's good, so merge it after these changes and tell ops to put it in prod"

cookiecaper 2017-01-18 17:36 UTC link

"Agile methods" as a term has zero meaning at this point. Talk about the specific things you do to make things work well instead of lumping them under the heading "agile".

The issue of people refusing to coordinate and working independently on two useless things for months at a time is an issue that transcends any popular software terminology. The role of a good manager is to see that units A and B are not in sync and resolve that. That's true in all disciplines. You can't attribute this to "agile" just because you meet every 2 weeks.

cookiecaper 2017-01-18 17:46 UTC link

I think the tendency to over-engineer is a symptom of retrofitting an assembly-line 9-5 shift onto the creative process of writing code.

You sit a guy there 5 days a week for many years. He has to look busy, he has to do something with all of that time. He's not going to get paid if he writes the code in the most simple, concise, and straightforward way possible and then goes home until they're ready to make a new feature two weeks later. He has to sit around and make up something for himself to do.

Contrast with side projects. I have many simple weekend projects that continue to work well and provide their promised utility years later. Because you just write what you need and stop, you don't get sucked into the disastrous complexity spiral that every company-internal software project ends up as.

The other factor here is that people need some signal to say "I'm good at my job" (because no one can actually tell). That signal has to go to colleagues, superiors, and peers outside the workplace. People therefore invent artificial complexity or take intentionally convoluted approaches so they can sound fancy. In the most extreme cases, this is a conscious decision designed to block out "competitors" (colleagues). In many cases, it's a subconscious way to ego-stroke (and to mix in a little bit of variety per point one above).

This is especially true when a household brand like Google or Facebook pushes out some new esoteric thing; everyone wants to see themselves as a Google-or-Facebook-in-waiting and it makes it easy to pitch these things to the bosses, when the fact is that the kinds of things that work at large public companies like Google are probably not going to work in small companies.

OJFord 2017-01-18 17:48 UTC link

> Microservices probably reduces the asymptotic cost of scaling but add a huge constant factor.

If this were Medium, I'd highlight the hell out of that.

That's so true, and so nicely, succinctly put - it ought to be the reply to end every argument about whether microservices are good or bad.

scandox 2017-01-18 17:56 UTC link

> Favour disposable code over reusable code

I prefer this way of putting it over YAGNI. This makes it sound more like the trade-off it is.

fsloth 2017-01-18 17:58 UTC link

I can't understand why people don't refer more often on Mr. Kays message. To be bluntly uncharitable and only half kidding, I do understand why consultants don't buy into it. Simpler systems that are less fragile mean less work.

GnarfGnarf 2017-01-18 18:00 UTC link

That is absurd. You can't write an OS in 100KLOC.

apercu 2017-01-18 18:01 UTC link

I think it's the nature of how a company implements agile, rather than agile itself. I have a client that subcontracts a lot of work to me (my work is mostly gathering business requirements and BPI).

They use "agile" but I have budget and timeline constraints on every project.

petra 2017-01-18 18:09 UTC link

I wonder why did he chose to built demo applications, instead of a powerful and useful development tool that has strong value somewhere ?

krptos 2017-01-18 18:18 UTC link

"putting your logic into types is better"

Can you elaborate on that one? Sounds interesting to me.

crdoconnor 2017-01-18 18:22 UTC link

I find agile works and works well when you limit the definition to "do what you can to avoid waterfall".

Where it starts to be about how you conduct meetings then it tends to fall to pieces.

ashark 2017-01-18 18:39 UTC link

One of the things that helps when you're developing your own projects is that you can single-handedly decide to ruthlessly cull parts of the project that take lots of time but provide little value, and you (probably) have decent insight into what those are. You're also probably not at a scale where doing certain really crappy, slow parts of the job can pay off, so you can skip those.

Default form elements with some basic, nice styling to fit your theme? Form done in one hour. Special snowflake version of the same thing from the design team, which has no idea what the platform can and cannot easily be made to do, but the client is absolutely in love with? Two days, a third party dependency or two, some extra environment-specific bugs to track down later, and generally increased fragility (so more time lost in the future). This has slowed you down now and increased the resources required for the project indefinitely. But the client looooooves it.

Support Android pre-5.0, at the cost of 20% more development time, a pile of extra bug reports, an uglier, harder-to-maintain codebase, and a much much longer testing cycle, for a side project? Hell no. Client says that will cost them $4 million/yr not to support those? Ugh. FINE.

And so on, and so on.

rbalicki 2017-01-18 18:52 UTC link

The "threat" of having to add unit tests should force developers to write their classes and components in a way that is easy to reason about. In particular: * put as much functionality into pure functions * depend less on statically-linked globals * import all significant collaborators across seams that can be mocked. * keep state in a small atom, rather than strewn about

If you write code like that, you get many of the benefits of unit tests, whether or not they are actually written.

Perhaps it's a good idea to write a test harness (e.g. larger integration tests) for old so that you have a reasonable chance of catching it if it becomes broken, and focus on writing new code in a testable fashion.

juancampa 2017-01-18 19:01 UTC link

> Favour disposable code over reusable code: Avoid the trap of premature optimisation, both in terms of performance and in terms of software architecture.

Some people call it "premature generalization". Relevant C2 page: http://wiki.c2.com/?PrematureGeneralization

cookiecaper 2017-01-18 19:02 UTC link

As you've intimated, most people have a very superficial mental model.

Facebook == respected tech brand == someone I should copy. The end.

Guy I know uses Cassandra == developed by hot tech brand Facebook == cool by mental association with Facebook.

Guy I know uses MSSQL or Oracle == developed by crusty old Evil Empire Company that cool people don't want to work for == bad.

Conclusion: We must use "big data" so we can be like the cool people -- err, because we really have some big data.

This doesn't sound like the outcome we'd expect from technical people making these decisions, but we can obviously see that it's what we're getting.

algesten 2017-01-18 19:26 UTC link

> Relational databases are the right choice 95% of the time, non-relational stores require a really specific use case.

Relational databases are great, but I spent large parts of my life as a developer writing layers converting to/from SQL and later ORMs. There's a huge gain in just not translating data. I know Postgres (and others) deal with JSON, but I can't escape the feeling it's a bit shoe horned in there – basic SQL statements have strange new operators like ->> -> #>>.

Relational databases are great for, well, relational data with strong consistency requirements. The popularity of the original MyISAM tables without integrity checks baffled me at the time. Why spend time marshalling data in/out of table form when you don't gain the benefits of a RDBM?

Not doing data translation saves _a lot_ of time. Plain key-value stores are amazing, document stores like Elasticsearch are great, ultimately the choice comes down to requirements and time saving is often a very heavy argument, especially for small companies/startups.

eikenberry 2017-01-18 19:33 UTC link

TDD is not about the tests, it is about teaching yourself to code better. About forcing yourself to break up your code into small components with well defined interfaces. Something we all want to do but which is hard in practice without a tool to guide you. TDD is that tool.

brwr 2017-01-18 20:11 UTC link

> Less code is better.

I'd change this to "Write as little code as necessary, but no less." The problem with "Less code is better" is that some folks use that as justification to write clever one-liners that are difficult for other developers to read. That is not better.

That aside, I agree with everything else you said!

jamesmccann 2017-01-18 20:17 UTC link

> I think this is down to the sad truth that most developer roles offer very little challenge outside of learning a new stack.

This is a gem observation from this thread. In my own tech sphere the first thing developers are talking about with each other is the new x,y,z lib or framework they're using to accomplish something relatively banal. There's still a lot of work out there that really boils down to basic CRUD and reporting at the end of the day, and developers naturally begin to invent complexities on top of that CRUD to make the work interesting and challenging. I'm absolutely guilty of this first hand.

I've found personally it also doesn't help that past work on projects e.g. large Rails apps that were never architected well turn out to be such nightmares to work on. The memory of the end state of these projects lingers with developers as they move onto the next piece of work, and they're inclined to say "no that doesn't work" and pick up shiny new-tech to do the old job instead.

As a side analogy: most small business construction jobs, e.g. building a timber frame house, don't involve the builders arriving on site and are stumped by the challenge of how to put up the framing for the bedroom walls - there's also very little challenge in these projects, yet the reward is in the completion.

user5994461 2017-01-18 21:51 UTC link

> 3) Depends. If you have a well-designed distributed system, it can be amazingly resilient and reliable without introducing much administrative overhead.

About that points. Scaling developments across many devs is a very difficult problems. It just doesn't scale.

A lot of organizations recruit/grow a lot of people and they try to get away from the human scaling problem by having them work independently/in their corner.

This allows people execute a lot of stuff... usually the same stuff 10 times, with little coordination or collaboration between them to the point it can be felt it in the resulting system(s).

gregmac 2017-01-18 22:06 UTC link

> Continuous integration is a good thing. Back in the bad old days you'd have three people working on parts of the system for 6 months and plan to snap them together in 2 weeks and it would take more like another 6 months.

Also extremely importantly is it brings you:

* Working tests. If you make changes and forget to or don't run all tests, the CI server will catch it and make you aware. You still have to write (useful) tests, of course, but that's a discrete problem.

* Entirely kills the excuse "Well, it builds on my machine". This means no undocumented dependencies, and the entire build is scripted.

* "Release builds" are a non-event. They just happen all the time, and your "release" is just the latest build that includes the changes you want and passes testing. This removes situations where there's only a limited set of people (eg: one) that can do a full release build.

Doing CI early on is much simpler. Aside from being beneficial from day 1, it is much easier to incrementally add to your build script/environment as needed than to try to create a script later based on a complicated manual process.

Not having CI is not nearly as bad as not using source control, but it's in the same ballpark.

devaroop 2017-01-19 05:50 UTC link

Alan just complicated his own laptop by not using which is proven to work.

misha67 2017-01-19 08:22 UTC link

> Which kneecap do you want the bullet in?

Funniest thing I've heard all day!

misha67 2017-01-19 08:37 UTC link

Done well, unit tests are invaluable. I'm a relative late-comer to unit testing, but can attest to its value.

The discipline of unit testing forces me to think about what risks I'm introducing with my new code.

Unit tests drive better design - smaller classes, looser coupling, better separation of concerns, functions that don't have side effects.

Best of all, unit tests reduce regressions. I can't count the times my test suite has prevented me from introducing a bug in my app.

I can refactor code with much more confidence than if I did not have 400 unit tests checking my work.

Most recently, these tests proved their worth when upgrading my app to Swift 3.0.

akamaozu 2017-01-19 11:35 UTC link

Link to the post about the graphics library please?

jondumbau 2017-01-21 00:21 UTC link

4: if my stack requires a message broker to run, how is setting one up manually supposed to be better to using the ansible scripts.

jon49 2017-01-22 01:21 UTC link

Something missing from this entire discussion is that developers have a hard time understanding what are truly best practices for their product they are creating/maintaining. It is a bold assertion to say that everyone understands all the different nuances in creating software.

Epistemic Quality

Propaganda Flags

2 techniques detected

Solution Orientation

Emotional Tone

Stakeholder Voice

Temporal Framing

Geographic Scope

Complexity

Transparency