Essential Pillars of Developer Infrastructure for 10X Tech Companies
13 min read
Hello my friends
Its been a while since I last posted. Even though we have quite a few drafts ready, but since each article is 100% written by me only, and I value quality more than quantity, it takes its own sweet time. And being a passionate techie, a founder, content creator & social entrepreneur cum coach has my calendar quite booked these days. If you want a glimpse of where I live, and what else do I do or like outside of tech, you may check this Linkedin post to get an idea (and maybe visit us someday too! :)
About this blog
In this blog we will discuss the role of engineering infrastructure for a company to be 10X from now to long term. This blog is for CTOs, founders, tech leaders but as well aspiring engineers and future leaders to know how awesome tech orgs are made :)
As always, let me add the disclaimer that here I am presenting only my personal views as proposals. You are urged to think upon and verify yourself as discussed in my notes on scientific enquiry in the primer blog for the 10X Engineering series.
Let's take today's dynamic technological landscape, where software development stands as the backbone of industries. Every Chief Technology Officer (CTO), tech leader or founder grapples with a set of KPIs and critical questions. These queries serve as the compass, guiding the creation of highly scalable, high quality engineering machinery, which can sustain 10X velocity, maintainability & agility over the course of time.
A company's KPIs in tech ops
Overarching goal
Shipping products with highest possible quality, long term maintainability and agility, in least possible time and cost, with least experienced engineers.
Pure Tech KPIs
Some can be directly measured through automation and processes. Some can be indirectly measured.
Story Throughput Per Developer Per Sprint - the higher the better
Bugs and performance issues encountered every month (Bug Introduction Rate aka B.I.R.) - the lower the better
Average turn around time to debug and resolve an issue - (Mean time to resolution M.T.T.R.) - the lower the better
Average time, cost and resource (developer) per story point shipped - the lower the better
Average experience of team members - lower the better
Average time to onboard a new team member on existing projects - lower the better
Average lines of code per story point (shipped till date) - lower the better
Maintainability, Agility, Reliability and Performance - higher the better
Technical debt - lower the better
Cost of infra and entire engineering ops - lower the better
Compliance and security - lesser gaps are better
Timely releases - Higher the better
And this has to be shown consistently over years, not just first six months (or first lap) of preparing a demo, launching a new product or a new startup where one may even cut corners to get the MVP rolled out fast.
A CTO (or tech org's) need is of striking intricate balance between business needs & constraints with the KPIs.
Questions of every Tech Leader
As the goals and situations of your business evolve constantly and as members come and go, each with their own experience and style of development, how can you ensure perpetual productivity, effectiveness and reliability of your software delivery?
Is your approach cost-effective, making optimal use of resources, while simultaneously removing obstacles for the rising stars – the younger developers who represent the future of your team?
How can your infrastructure adapt and scale six months from now and remain so for long term future? Can the foundations and buildings you lay today withstand the test of time, remaining robust, adaptable and effective two, five, ten years into the future?
As I discussed in one of my previous blogs on 10X Engineering Mindset, the code we authored for iXigo 16 years ago is still running its core, as the 4 engineer startup from then has scaled to being a highly profitable OTA in India, about to launch its initial public offering in June' 24. Here is a snippet from the same.
Can you imagine the meaning and implication of "code once shipped remaining valid forever", from the growth standpoint of a tech centred business?
These questions are not mere reflections; they are the cornerstone of a thriving software development journey. By the end of this blog we will mention the pillars of developer infrastructure, each representing a crucial aspect of the software development lifecycle. These pillars are not just standalone elements but interconnected components that shape the efficiency, collaboration, and adaptability of the entire development & delivery process.
Now where is your company?
In all typical API and event driven systems, the business logic, schemas and configurations are less than 10% (tip of the iceberg) of what it otherwise takes to run the software development and delivery operations from end to end. But till date, most of the global teams whether startup or enterprise, own also the rest of the iceberg (the 90% infra part).
A common engineer doing things which are more than business logic, without guardrails, makes companies sluggish even though they may say - it works! And it is the definition of "what works", which makes the difference between the average and the great (or 10X) businesses.
The more more work your teams owns, and the less infra and guardrails you have, more is the
Complexity: Which the team is expected to handle
Time to market: Time taken to ship features and products
Design mistakes and chaos: More complexity and more heads to think means increased probability of design mistakes & diversified implementations across projects
Bugs and performance issues: More work and less guardrails means more chances of errors and technical debt.
Incident resolution time: Finding the root cause of error could turn out to be finding a needle in a haystack. And fixing it may be a nightmare depending on your technical debt and design choices in the given project.
Skill level and learning curve: More expertise needed to learn, develop and debug the more complex stuff
Technical debt and chaos: When there are more code and design decisions to do, and given each team's unique design sense, preferences and perspectives, each project gets implemented differently. This means its scaffolding & solutioning, technical debt and challenges are also unique to it. So basically the organisation's code bases diversity, complexity & problems grow proportionately with the number of projects, features and members it adds over time.
Tribal Knowledge: Only the developers who develop a particular project know how it is implemented.
Higher switching cost: Varying implementations, designs, problems and tribal knowledge in every project increase higher switching costs
Lack of manintainability and agility: it becomes more and more difficult to refactor or enhance your codebase or replace integrations or cloud. This also means more work to do for compliance and security checks. This reduces your agility as a company.
Bottlenecks: All of this may lead to various bottlenecks for growth, in form of software's lacklustre performance, increased technical debt, lack of maintainability, talent bottleneck, tribal knowledge bottleneck etc.
Cost to org: More time taken and higher level of expertise required means shooting the development and delivery cost by many times.
Competitive risks to org: You may soon have competitor(s) who are faster, leaner and better than you in shipping products and features.
Loss to org: More costs, more time to market, more bugs, more downtimes, more migration efforts, more risks - all of these also mean - lost customers and revenue opportunities, making you less competitive.
The Two Paths to 10X
In this post shared on Linkedin recently I talked about which strategy has better chances to make a company 10X tech company.
A. Striving to maintain a 100% 10X engineer team?
B. Reduce the complexity and scope of work by 10X (and use guardrails)?
Let's visualise the iceberg of software development and delivery
Should your application teams work on
A. The actual business logic (tip of the iceberg) or
B. Business logic + rest (the whole iceberg) - rest includes everything which is not business logic but is the infra or boilerplate required to develop and deliver your software at scale.
Now visualise the guardrails too!
Guardrails protect us from going in the wrong direction (astray).
Should various teams be allowed to take their own path in every project, or follow an org wide standard with best practices baked in?
Absolutely Essential Guardrails for 10X
Here are some guardrails that I really believe are absolutely essential for any software delivery well done. You can not remove even one of them and expect to be 10X.
Schema Driven Development
Modular Architecture
Security & Compliance
Configure Over code
Scaffolding, IDE and Developer environment setup
Automation - in infra setup, development, delivery and reporting
Guardrails can be baked in at these four stages
Infra setup (First time setup of the infra and system layer shown below)
Develop and commit code (scaffolding & framework used + pre-commit hooks)
Code merging and integration (CI pipeline with automated scans for testing, security and compliance) - can be configured using Github, Gitlabs, Jenkins etc.
Deployment (Deployment, migration and rollback process) - Part of your CD.
Post deployment - Continuous security and compliance scans, monitoring, auto-scalability and self-healing)
The layers of typical software delivery
And how a rightly baked infra can help manage them. The business dimension is the coding (business logic and data) layer - requires fourth generation frameworks. The System and Infra dimension requires a platform driven automation thinking.
System and Infra layer solutioning includes
A. Infra: Setting up hardware & network infra plus common system services on cloud of choice. System services cover CD service, monitoring and other shared services like messaging, IAM, policy engine etc
B. Application deployments through CI/CD/CT with testing, compliance/security checks, monitoring and auto-scaling - Automated setup pipeline and checks for delivering domain services over K8s.
Goal
Imagine you had everything available to get started with the choice of your microservices in 30 minutes, for any new product or business line, with costs, data, no lockin & maintainability!
The 7 Pillars of 10X Developer Infrastructure
When designing infra for 10X, here are the 7 pillars which I believe are absolutely essential and can be brought in at different layers and steps of your software delivery. Without even one pillar working well, your temple of 10X engineering work will falter (or is already faltering).
In my view, every CTO, engineering lead or founder needs to ensure these pillars are robust and working fine in their organisation
Standardization and Maintainability (published)
Productivity and Democratization
Collaborative Excellence
Testing and Quality Assurance
Deployment, Monitoring and Scalability
Development Environment
Security and Compliance checks
In future entries, we will deep dive further into these pillars of the organisation's infra.
The Dilemma of a Tech Leader
It is not uncommon to see tech leaders grapple with the needs of the hour, and never getting time to solve the needs of all the future hours of the company. This means they are constantly reacting to situations with short term measures addressing immediate needs. As for the future, they make do with faith - "that things will somehow work well".
There are two main types of situations with leaders here
A. The current state of affairs is acceptable to them, even with the multitude of challenges. I believe this is because
Ignorance - They do not have information or can not visualise their gaps and how much better their company could be tech wise
Lack of motivation for bold steps - This could be for N number of reasons, but is truly detrimental for such an organisation.
B. The current state is not acceptable to them but,
They don't have resources to modernise and enhance their tech ops
They have the resources, but their developers are resisting bringing anything new because they are comfortable with whatever is existing today. The case where you are comfortable with "a known devil" and don't wish to step out of the "comfort zone" to try something new. This is detrimental not only for the engineers career, but as well the tech leader and organisation, if they let it be.
Its critically important for you as a tech leader to see your company as an integral being. This being exists, will exist for many years and probably outlast you and all the teams and developers working for it today. When you see this integral being's needs and challenges from the birds eye view, you will gain the right perspective to solve problems from the being's standpoint. If you do not have the being's view, you will be dealing from situation to situation, project to project and team to team - never working with a cohesive and deterministic company strategy, never achieving your fullest potential, compromising and saying to yourself for the next few years "It works for now!"
Is "it works" acceptable to a 10X mindset, when they know things can work better?
Every CTO and tech founder should contemplate building a company whose projects are scalable and maintainable from day one - She should fight if needed, to prevent technical debt. The questions of scalability, ease of adding features, and making developers productive over time should be at the forefront.
Being 10X is not a choice. Its a need. You better realise it.
Question Yourself
So, is your company and project teams working on the tip of iceberg today or the whole iceberg today?
Lets set aside the notion of 10X, even if you achieve 2X, you will not only save half resources, half time to market, half costs, but you will increase your chance of business success by manifold.
Would you like to settle for less when you can have much more by a little effort and investment?
As well feel free to reach out to me or us in case you wish to have a friendly discussion on any of the aspects mentioned here.
The Conclusion
Taking the analogy of iceberg. In order to be 10X, tech companies need to reduce their scope of work by 10X and introduce guardrails based infra. This means you need to ensure your teams work only on the pure business logic, configurations and schemas. Everything else below your business logic should be left to a pre-tested, reliable infra and process automation setup.
You can adopt proper infra, processes and practices which handle or abstract 70-90% work of the software delivery lifecycle. You can have a strategy to get high quality work done by fresh engineering graduates from a tier B town, or even aspiring school dropouts with 1-3-6 month training program. Even if people switch between projects, cloud or language ecosystems, they can expect to see a standardised implementation, making the switch fast and easy. As a CTO or tech leader I will take special efforts to establish infrastructure & processes with guardrails and automation, to ensure we constantly move with 10X velocity from the short to the long term.
Our Mission at Godspeed
At Godspeed we envision to assemble together a stack once and for all for tech startups and enterprises and make it available in an equitable fashion so as to help them solve great engineering problems with ease, with tech as their enabler.
The Meta Framework already covers few of these guardrails when developing apps currently over Nodejs. Its polyglot implementation is expected to release this year with couple of more languages. We are also adding stack for automating infra and application deployment with guardrails.
In case you do not use Godspeed's stack for whatever reason, you will need to think and develop your infra on the same lines.
Reference
Curious about Godspeed's Meta Framework?
Check out
A. Getting started section and guide in documentation
B. Video playlist introducing the concepts of Godspeed along with these guardrails