Building a Technical Organization Around Research

Overview

In 2021 I took over the technology organization of an acquired startup that was coming apart. The CEO and my predecessor were leaving. Only three productive contributors remained, only one of them trained as a software engineer. We had valuable finished research that needed engineering and customer discovery to ship. Corporate policy on pay, remote work, and cloud compliance blocked obvious fixes. Over the next three years we grew to twenty-four contributors across research and engineering, stood up a product organization alongside us, and shipped an energy management platform that cut fleet energy costs by roughly 25%. The platform passed deployment testing, produced a NeurIPS paper, and was sold into a $675MM partnership where it runs on multiple sites today, but the division was wound down when interest rates rose. Here is how we got there, and what I got wrong.

Startups organized around a research breakthrough begin with the blessing and the curse of a technical organization staffed with outstanding researchers. That population includes some of the most talented and hard-working people I have met. They have a complicated relationship with software engineering. Making that relationship productive underlay solving all the other issues.

A Tricky Pitch

A former colleague of mine recruited me one spring with this appeal:

1. I want you to take over my role running our technology organization
2. We’re building an energy management platform based on our research into electric vehicles and power control systems
3. We have to turn the current proof of concept into a production system by next March
4. The CEO and I are quitting because the acquiring executives, your future bosses, won’t listen to us, making our jobs impossible
5. Several of our best workers are also quitting, leaving you with two researchers and three engineers, two of them unproductive
6. You’ll also need to perform the duties of the Head of Product, a second 70-hour-a-week job for which you are underqualified
7. We don’t have the people we need on staff to deliver the system, and H.R. policies (below-market pay, in-office requirement at the height of the pandemic) make it impossible to add them
8. Corporate I.T. wants to absorb us into their compliance-centric cloud management system incompatible with shipping in less than a year

Why would anyone take such a flawed opportunity?

1. The business model: with previous experience in electricity markets and enough education to follow the math, I could see that the optimization engine inside the platform was good enough to extract real value from electricity pricing mechanisms. Savings would vary, but typically reach 25% while simultaneously increasing operational reliability by monitoring battery level against projected use until next charge. The researchers talked about what they wanted to do next, but what they had finished could provide a step-change reduction in the cost of electrifying road transportation, presenting an excellent business model that also reduced pollution significantly
2. The acquirer was the country’s largest utility: if you want to have an impact in energy, working with the large old companies that run the country’s energy systems is essential
3. The tractability of most of the obstacles:

• lack of understanding among executives about how solid the current research was, which we could fix with demonstrations
• lack of software engineering expertise, which I had
• hiring and talent in general, which I could supply
• specific hiring and I.T. policies, which I could negotiate
• a very aggressive deadline, which I didn’t have a plan for

4. When my old colleague elaborated on the executives’ “not listening to me”, I heard a low context communicator trying to overcome a close-to-the-vest hierarchy by insisting upon his facts, which suggested that a change in tone might prove fruitful

Buying Room to Work

I began in early June with the usual listening tour, but compressed it into two weeks and focused on my senior VP. He owned our effort and had the clout to fight for it. Once he was comfortable with me, I walked through the opportunity cost of the H.R. and I.T. policies: we would not hit our deadline. I asked him what we should do. He secured an indefinite exemption from the I.T. policies and the H.R. requirement for on-site hiring. With the possibility of success in hand, it was time to rebuild the technical organization. Who did I have to work with?

The staff consisted of four researchers (two holdovers from the startup, two who joined soon after I did, all excellent) and three junior engineers, only one of them productive. The productive engineer had to work full-time on operations to keep the lights on, leaving the researchers to engineer the new system. The researchers had an admirable zeal to see their valuable insights help real people. They accepted the need to serve as software engineers for months.

Learning the Research and Teaching the Researchers

For researchers, building a production-quality platform for managing gigabytes a day of data from thousands of HTTP and Internet-of-Things clients can be a daunting prospect. In order to survive their academic context, they become effective programmers but with skillsets determined by the problems they stumble across rather than a desire to master the discipline. They learn just enough of the domain to survive whatever their current assignment is before returning to the science they love. Such a startup’s models have typically far outstripped the supporting software infrastructure, leaving the organization with better science on its laptops than it can package for customers and a staff of high-energy contributors complaining in their 1:1s that they’ve spent the quarter “writing utilities” (doing software engineering) and can they please get back to research?

We began with an onsite. I emphasized my commitment to the team’s vision. Then we switched to learning from each other. The researchers taught me how our core technology, model predictive control, worked. It represents real-world phenomena mathematically, and chooses a desired state of the controlled entities, in our case the energy needs of charging electric vehicles, as a target to optimize for. Some phenomena (prices, power and energy levels, ambient temperature) come in as numbers and can be put directly into the model; others require research and judgment. I needed to understand MPC, and the researchers needed to see their technical leader do his homework, demonstrating respect for their achievements and cognitive humility. Then it was my turn to teach. Solutions to our data platform needs were well-known in the larger software industry, and I had helped build and maintain comparable platforms several times. I produced a straightforward architecture for a production distributed system that could handle our planned load. As we scaled towards production it would surely need revision, but that was next year’s concern. I laid out a concrete plan for building it:

1. a clear design based on focused components interacting via simple interfaces
2. a sequence of crawl-walk-run targets:

• generate a power schedule for a single charger
• generate the schedule for one depot’s chargers for one day
• simulate one charger going offline and show how fault is detected and communicated to the scheduler, which rebalances the load
• …

Those targets gave the researchers months of actionable steps to take. Mentoring I restricted to the daily status meetings and Sunday afternoons, when I reviewed the week’s new code (often after it had landed) while switching my focus to staffing. My priority was to build engineering capacity, so our software could catch up to our research. Product direction I simply put off. We would be much better off a few months later with an engineering staff and a product discovery gap than with no staff and an extensive roadmap. I put the two unproductive software engineers, call them “Alice” and “Bob”, on performance improvement plans. Everyone “knew” that Alice needed to go but that Bob was just a little disorganized and would be fine. I wrote them the same fair PIP: land useful code every sprint (which is in fact the core of their job description). Alice worked nights and weekends and developed into a productive contributor, while Bob offered only excuses. I fired him.

Our First Accomplishment and First Break

The researchers reached the first, “crawl” milestone a few weeks ahead of schedule in September. This accomplishment, due significantly to Jamie, one of the new researchers, strengthened morale on the team and trust with our senior VP. By showing him rapid progress, we justified the clout he had spent to exempt us from I.T.’s heavyweight procedures. The bad news was that we were not on track to generate optimal charging schedules by the March commercial operating date. There was too much to build and not enough capacity on the team to build it. The out was to decouple the platform’s full optimization functionality from the immediate need for operational readiness. The contract had been written long enough ago that it didn’t have a specific energy savings target. We felt out the customer to verify that they didn’t care if we were temporarily paying for a marginal increase in energy costs, and got our executive sponsors to accept the modest increase in short-term expenses. One researcher rolled their eyes at yet another delay in seeing their work in production, but it kept us out of crisis mode. We could work calmly to complete the MPC integration.

Growing the Organization

For the rest of the quarter my time went to the difficult and time-consuming effort of implementing a hiring process that could land excellent senior and staff engineers who would function as a team and not a collection of strangers. Meanwhile, one of the researchers whose tenure predated mine left for a startup. The other would leave the following year for a different startup. They never felt at home working inside the energy industry and missed doing science. I don’t blame them. They had signed up for different jobs at a different company than they found themselves in. Sourcing excellent researchers was an embarrassment of riches. I was able to fill one of their spots with Michelle, an outstanding scientist who was also an experienced manager of researchers willing to work as an individual contributor.

A Partner

All this time I had served as interim Head of Product by largely putting the responsibilities off. We needed to build out a product organization, starting with someone to run it, to tackle the backlog of customer discovery I had neglected. When we interviewed Olya in February, not 15 minutes went by before the two of us were debating how to conceive of product in an energy management context. Of course I advocated for hiring her, and we did. She knew much more than I about the energy industry and large organization dynamics, caught various mistakes I made, and coached me towards her level.

We ran the organization as a team, working through all major decisions together. This practice sometimes added hours of effort and a day or three of delay up front. It also meant that the team wasted no time on the all-too-common organizational conflict between technology and product — especially important for us given the latent tension between research and engineering. And because we collaborated daily, when we challenged each other it was from a shared set of facts.

Growth Pains

As Olya built out her team and they studied the market, the technical organization hit a series of inflection points:

• one team became two, then three, then a different three, and eventually four
• product wanted user-facing applications sooner rather than later so we could demo and get feedback
• velocity slowed

We weren’t in trouble exactly, but I struggled with a mismatch between how quickly needs were shifting and a healthy cadence for organizational change. There were multiple interacting causes:

1. The researchers had built the initial architecture past the point where it needed revising, but they didn’t know how to reduce technical debt
2. The new senior engineers knew how to clean up the platform, but couldn’t overcome the heavily consensus-based process to replace familiar code with intimidating, unfamiliar approaches
3. I took too long to add a management layer between myself and the team, rejecting candidate after candidate for failing to meet all three of excellent technically, excellent managerially, and a devotee of the style I had established

We leveraged the architectural decision record process as a structured way for senior engineers to show how a change would be an improvement, and for the team that built the original to name specific objections or accept it. Olya persuaded me to hire engineering leaders who weren’t precisely in my own image. Of course, this meant they brought different strengths to the team, a second big win in addition to supplying the coordination needed to realize the ADRs. The researchers stayed under me for the time being; dedicated leadership for them was a need but not yet a top priority.

Two Lines of Business

Around the time we hired Olya, negotiations began on an ambitious project. One of the world’s major truck manufacturers and a global investment bank wanted to build public charging infrastructure for along interstate freight trucking corridors across the United States. Our energy management platform was essential to making the $675MM financial model pencil out. My boss asked me to negotiate the technical terms.

I sat down with the counterparty’s software group, who would build the fleet operations stack. They understood that energy management meant years of work for a substantial technical organization, and they wanted to be that organization. Their strategy was to propose a division of responsibilities that quietly left energy management on their side — despite their own business people having already agreed it was ours. When I pointed this out, their representative explained that energy management was an operational concern, and they owned operations. He would not move. I escalated, got confirmation, and scheduled a session to finish the contract. They sent a different person, who took the same line. This happened twice more.

So I drafted the technical exhibit myself, specifying our ownership of energy management and theirs of operations software in precise detail. Both sides’ business and legal teams accepted it without edits. It secured $54MM in funding for my organization.

New opportunities kept up the pressure on us to adapt. Our initial business targets had been fleets of road vehicles with existing private parking and fueling areas: school buses, drayage, and so forth. We would swap in electrical “fueling” depots to complement the existing diesel depots, with essential energy management services provided by our technology. As companies looked at electrifying their urban fleets, a new opportunity arose. Urban commercial vehicles refuel at regular gas stations. Electrified, they would need dedicated new depots with hundreds of chargers and many megawatts of capacity. Because their vehicles couldn’t run without access to these depots, we could guarantee revenue over years. Because utilization was essential for reliability and profitability, a new set of optimization problems confronted our researchers (to their delight). We soon had a FAANG company bidding against a tech unicorn for access to one of our leased depots. Ironically, this success would prove the final nail in our coffin with the C-suite, as I explain below.

Shipping

As a successful startup, we were being pulled in multiple directions. The struggle to prioritize was vexing when viewed through the lens of user requests: a legacy application we had to keep running, leased depot design, new conventional dedicated depots, better UIs for more impressive demos to all new customers, a huge defense industry opportunity that tied up a chunk of the team for nine months before vanishing into the federal budget process … and that first deployment. Its commercial operating date had moved from March to July, then to no date at all, then eventually to the following November. There were delays with plumbing, permitting, vehicle availability, and on and on: none of it under our control. The November date was the first one in a year that looked real, and we restructured the prioritization effort around it. Fortunately, everybody wanted to ship the first deployment at high quality. Working backwards from that shared goal gave us the leverage we needed to get product, engineering, and research on a common sequence of priorities.

Taking model predictive control into production for managing vehicle power requires converting difficult business evaluations (cost of lateness, cost of risk, …) into non-obvious quantitative representations so they can be weighed against each other, ratcheting up the evaluative challenge. The power and complexity of the model derives from its integration of everything it sees. Researchers and product owners must understand each other’s domains well enough to reach consensus on what should happen, then researchers must implement it mathematically. It may not occur to either group to model some aspect of the real world that humans know tacitly. The initial algorithm tended to prefer charging just before departure. That eliminated a small amount of parasitic drain on the battery (optimal!) while creating significant operational fragility in the face of late-developing needs such as another vehicle’s needing more power than anticipated in that window (oops). The researchers implemented a crude weighting towards earlier charging before digging into the subtleties — once they had noticed the gap.

It was past time to give the researchers dedicated leadership. I promoted Michelle. She worked with product and engineering on the crucial cross-technology effort to define hundreds of hardware integration tests that would demonstrate correct behavior from the platform in the face of myriad hardware, network, and software failures. Product owned conditions of satisfaction. Then all of technology was involved in determining when issues lay with research, with engineering, or across domains. By November, we had a rock-solid software platform covering more than the essential feature set and an exhaustive hardware integration test plan ready to run. The team had done it.

What We Delivered

The commercial operation date slipped again in rolling two- to four-week increments until the following February. We used the time to deepen the product rather than hold it frozen — a reversal of the discipline we’d just imposed on ourselves, and the right call with capacity idle and no date to protect. When we finally got site access for our first deployment the 300 hardware integration tests took days to run, as many of them required human actions such as plugging a vehicle into a charger. We found many minor issues but no release stoppers. Unfortunately, further chaos from our partners prevented the platform from entering commercial operation at that site. The platform runs at multiple Greenlane sites today.

Winding Down

The researchers pivoted to fleshing out requirements for dedicated depots. Some of them wrote up our technology and got the paper accepted at NeurIPS, a leading AI conference. We had grown from three beleaguered contributors to a technical organization of twenty-four researchers, engineers, and managers.

Our business was conceived during the era of 0% real interest rates, when our corporate parent was flush with capital and eager to find productive uses for it. Unfortunately, the post-pandemic interest rate surge led to the wind-down of our division as the capital-intensive business plan became the opposite of what our parent needed. The dedicated depots in particular were a fatal success: they needed large up-front investments that would pay off handsomely for a decade of operation that wouldn’t start for months. We sold the energy management platform to the partnership whose technical terms I had negotiated, which continues to add routes that it serves. The researchers, engineers, and product owners who built the platform know it continues to save energy, emissions (>1MM tons CO2e annually if commercial goals are reached), and even lives via PM2.5 abatement.

What I Got Wrong

I came into this job believing that technical leadership is founded on a business model and left it with more evidence for that perspective. The core challenge for my managing this group of researchers was that we were selling what they had done, not what they wanted to do next, but that proved a tractable problem. Startups succeed by outrunning their mistakes. I made several:

1. Accepting a serious responsibility without understanding how I was going to deliver on it, gambling on my ability to work it out as I went
2. Not doing more engineering management by diktat in the early months
3. Not anointing a staff engineer to set technical direction in the middle months
4. Delaying the hiring and empowerment of engineering managers
5. Not understanding how low cost of capital drove the business model from the perspective of the C-suite

I was saved from the first mistake partly by decoupling operations from optimization and partly by a schedule that slipped for reasons that had nothing to do with technology. The middle three were the price of empowering an organization that needed to believe in itself again. On the last, I came in knowing that I had to manage key relationships: making an ally of my senior VP, partnering with H.R. on what they could do without asking them to do what they couldn’t, and fending off I.T. None of that affected how a utility would reallocate capital when the interest rate regime changed. I couldn’t have controlled that, but I could have seen it coming. Like the first mistake, it was a failure to read what my organization depended on. If you want to work at continent scale, you need to grapple with finance.