For engineers and engineering leaders who have been in the software industry for some time, you’ve likely encountered or collaborated with a productivity engineering team. However, for many who primarily focus on developing core products and services, the concept of productivity engineering may seem elusive. While test teams prioritize testing and quality, SRE teams maintain production environment health, OPS teams manage operations, and devops teams handle environment setup and changes, the role of a productivity engineering team remains unclear.
After dedicating 18 years to engineering core products, platforms, cloud, and distributed systems, I shared the same curiosity before assuming the role of head of productivity engineering three years ago. Now, having gained experience in this domain, I believe I can offer insights that may clarify its purpose for the wider community.
Henceforth, I’ll use ‘PE’ as an abbreviation for productivity engineering in the remainder of this article.
What is PE?
As engineering organizations expand, numerous tasks become integral to the daily interactions of every engineer. These tasks range from development environment setup, coding experience, building, testing, continuous integration (CI), and continuous deployment (CD), to adhering to security best practices, utilizing tooling such as feature flags, and managing releases, among others. It’s neither efficient nor practical for each engineering team to tackle these tasks individually due to the potential for redundant efforts. Moreover, expecting every engineer to possess the expertise and enthusiasm required to implement best practices across these diverse areas is unrealistic. Additionally, the adoption of divergent approaches by different teams can lead to inefficiencies in maintaining and evolving the engineering organization as a cohesive whole.
This is where productivity engineering specializes:
- Addressing common challenges encountered by engineers, allowing them to concentrate on core product development.
- Serving as an amplifier for the engineering organization by pinpointing bottlenecks and eliminating them, thereby enhancing engineers’ productivity.
- Advocating for the adoption of best engineering practices.
What are the common elements required to build a robust PE team compared to other engineering teams?
All the essential components you would anticipate: recruiting talented engineers and inspiring leaders, cultivating a passion for the team’s mission, promoting transparency, collaboration, innovation, and ownership, attentively listening to customers, consistently delivering high-quality results, establishing a clear vision, and aligning priorities.
What sets productivity engineering apart from other engineering disciplines?
No product team to work with
PE teams primarily focus on projects that directly impact engineers, rather than customers. Consequently, they often operate without a dedicated product manager. In essence, PE teams must fulfill the product management role internally: researching the needs of their ‘customers’ (in this case, other engineering teams), justifying business value and return on investment (ROI), crafting project proposals, designing solutions, determining adoption and rollout strategies, measuring impact, and deciding on subsequent actions.
While there may be similarities with infrastructure teams, productivity engineering (PE) remains distinct from traditional product development teams. In common product development scenarios, the product team typically dictates product direction and engages directly with customers. Requirements for a product or feature are established by the product team, and the engineering team executes on them. In contrast, PE teams focus on enhancing the engineering process itself, working without the direct involvement of a product team and instead taking ownership of setting direction and addressing the needs of other engineering teams.
Diverse skill set
Merely possessing coding, design, and service development skills isn’t sufficient for a productivity engineering (PE) team. Additionally, expertise in devops, networking, and library development is essential.
- Devops: The PE team frequently integrates purchased tools and products rather than building everything in-house. This integration spans various environments, including private data centers, Kubernetes (k8s), or public clouds, making strong devops skills indispensable.
- Networking: Given the diverse array of tools and products that must interact, troubleshooting becomes inevitable. A solid understanding of networking greatly facilitates this process.
- Library development: Many tools developed by the PE team take the form of libraries or command-line interfaces (CLI), as opposed to the service development focus on service architecture, APIs, data storage, query optimization, and traffic management. Library development involves less boilerplate code and necessitates specialized skills.
Customers who are close but sometimes are too close
The engineering teams, as customers, are part of the same engineering organization within the company, making them naturally close to the PE team. This relationship is internal, as opposed to interactions with external customers.
On one hand, this proximity offers significant advantages for the PE team, as working directly with engineering teams fosters honest, friendly, and collaborative communication. Booking meetings with stakeholders, conducting surveys, and engaging in conversations are relatively straightforward.
On the other hand, due to this internal relationship, members of engineering teams may escalate issues easily and reach out to EP engineers directly via slack/email, potentially causing unnecessary stress if not managed effectively. My recommendation is to establish a clear engagement protocol and encourage all incoming requests to adhere to it. This approach helps bring order to the process. Additionally, publishing FAQs and service level agreements (SLAs) while enabling self-service options for various tasks can help minimize the volume of requests.
In a large company, there will always be individuals who may be demanding or occasionally rude when making requests. It’s crucial to handle such situations professionally. I suggest ensuring that managers have channels to swiftly address these issues and provide effective coverage for engineers. This aspect is vital for fostering a healthy PE team environment.
Important to engage the community
Managing a productivity engineering (PE) initiative resembles overseeing numerous open-source projects. It’s crucial to have a clear vision and direction, alongside intentionally cultivating communities through scheduled meetups, attentively listening to feedback, understanding concerns, and offering timely support as reliable partners.
A mandatory approach seldom yields favorable outcomes. PE cannot introduce process changes or new tools in isolation and abruptly impose them on everyone. Failing to engage properly misses the opportunity to enlist engineering teams as collaborators. Additionally, it’s essential to recognize that teams often have varying priorities and timelines. Consequently, enforcing unified adoption without adequate engagement is likely to present challenges for some teams and result in pushback.
Frequent decisions of build vs purchase
Compared to traditional engineering teams, productivity engineering (PE) places a greater emphasis on evaluating purchase options versus building solutions in-house. Key considerations typically include:.
- Cost Analysis: Assessing the total cost of building versus purchasing, encompassing setup, hosting, maintenance, and support expenses..
- Time to Market: Evaluating the speed at which a solution can be implemented and made available.
- Future Predictions: Anticipating potential changes in scale, from initial adoption to global rollout, as well as anticipating evolving requirements over time.
- Vendor Maturity: Considering the stability and viability of the vendor over the coming years, ensuring they can meet long-term needs and remain a reliable partner.
Vendor’s solution often requires further work
It’s important to dispel the misconception that all vendor products will function flawlessly out of the box. In reality, this is often not the case.
For example, let’s consider our use of an industry-leading product like Artifactory. Despite its reputation, we found ourselves needing to develop the following features internally, as they were not available as out-of-the-box solutions but were crucial for our requirements.
- Codifying configuration changes in GitHub.
- Implementing retention management.
- Facilitating artifact promotion.
- Conducting migrations from the VM version to the Kubernetes version without downtime.
- Mapping issues to Jira, managing deduplication, and automating closure processes.
Hide complexity of many tools/services
When productivity engineering (PE) exposes numerous tools and services to engineering teams, it can quickly become overwhelming. Even if each tool has excellent documentation and is relatively straightforward to use, users may find it time-consuming to fully comprehend and effectively utilize all the tools, especially when they are subject to frequent changes.
To alleviate this challenge, it’s advantageous to implement an orchestration layer of services that abstracts the details of individual tools and their integrations. For example, providing a portal as the central entry point can offer:
- View: Access to the status and information of all tools provided by the PE team.
- Insights: Alerts and trends for enhanced visibility.
- Management: A centralized platform for executing changes such as creation, updates, and deletions.
ROI
The PE team is typically a small entity, comprising only 2–5% of all engineers within the organization. This translates to limited resources. However, the scope of responsibility and ownership of various areas and tools can be extensive. This dynamic presents a common challenge reminiscent of startups: how to effectively focus limited resources to achieve the most impactful goals.
Consequently, ensuring a favorable return on investment (ROI) becomes paramount for the success of a PE team. Prioritizing initiatives and allocating resources judiciously are critical strategies in navigating this challenge.
