The security industry has come a long way from traditional username/password authentication. Password managers such as LastPass and 1Password prevent attacks such as password guessing/brute force while decreasing the blast radius in the event of credential compromise. We also saw the introduction of Multi-Factor Authentication (MFA) such as Timed One-Time Password (TOTP), app-based push notifications, and email magic links.

Unfortunately, all these MFA mechanisms share the same vulnerability; they can be stolen just like traditional passwords. If an attacker can trick a user into sharing their password, they can undoubtedly trick them into sharing their MFA code. This is precisely how 

"0ktapus" targeted 130 organizations and 10,000+ individuals in a massive phishing campaign

MFA vulnerabilities are why industry leaders like Microsoft, Google, and Apple joined forces to create a standardized authentication solution called Fast Identity Online (FIDO). FIDO utilizes hardware keys to authenticate the user instead of passwords. The key never leaves the physical device, so an attacker can't steal it. The best part; FIDO keys provide a better authentication experience because a user no longer has to care or worry about passwords, timed codes, etc. All they need to do is tap a button.

Enabling FIDO key authentication is one of the best things an organization could do to protect its users and improve its security posture.

Contrary to popular belief, organizations can achieve passwordless authentication 

 purchasing hardware keys such as Yubikey.

In fact, most devices these days have FIDO keys built right into them! Apple's FaceID/TouchID, Windows Hello, and Android Fingerprint are all FIDO keys that organizations can use to protect their employees.

If a device has any of these biometric authentication methods, then it likely uses FIDO key technology.

At Pipe, we use Okta as our identity provider. Our internal sites and services use Okta to manage accounts and permissions. Okta includes functionality to register and enforce FIDO authentication. When a user is prompted to enroll their FIDO key, they can enroll their laptop and smartphone in a few steps.

Once entered the FIDO key enrollment process, simply follow the steps and authenticate yourself with TouchID/Windows Hello when prompted.

After registering a laptop FIDO key, the user can enroll another key. Chrome will display a QR code which can be scanned by a compatible iPhone or Android device. Now, the user can use their smartphone to authenticate themselves.

At Pipe, we successfully onboarded every employee to FIDO authentication with minimal disruption. We did this by first making FIDO enrollment optional and providing employees with enrollment instructions, and then gradually enforced the policy onto employees until we hit 100%. The period for optional registration meant employees could enroll on their own time, and the IT team could give general support and document FAQs without being overwhelmed. We used the Okta API to track enrollment and was pleased to see that 

~40% of employees self-enrolled without any enforcement.

There were a few cases where an employee's workstation was not FIDO key enabled. In this scenario, we ship them a Yubikey.

Firefox has yet to support device-enabled FIDO key authentication. If your team or users rely on Firefox, they must be shipped a Yubikey.

Unlike Yubikeys, device-enabled FIDO keys such as TouchID or Windows Hello have a unique registration per browser. Users must enroll their FIDO key for each browser if they use multiple browsers for work, such as Chrome and Safari.

Some in-app browsers, such as Microsoft Office on Windows Devices, do not support FIDO key authentication. In our experience, using in-app browsers is only used once and does not happen often. In this scenario, the IT team verifies the employee via Zoom and grants them the temporary ability to authenticate via another MFA method.

Overall, our migration caused minimal disruption or overhead and required no additional hardware or costs. 

It took the IT and Security team 90 days to onboard 100% of the company from start to finish

. More importantly, employee credentials are as protected as ever, and we provide our employees with the best authentication experience possible.

We are proud to give our customers and employees the best-in-class security. If you want to join us on our mission to transform the future of finance and become an industry leader in cybersecurity, 

Contrary to popular belief, organizations can achieve FIDO authentication without purchasing hardware keys such as Yubikey.

Passwordless Authentication — Moving away from Yubikeys 

Achieving Passwordless Enterprise Authentication — Moving away from Yubikeys 

, our goal is to share the lessons we learn while building our product, our team, our culture, and Pipe. No two companies are alike – and over time, no one company remains the same. We believe that the path of any successful company is carved by its unique combination of customer needs, team, and environment. As we build Pipe, we hope to share stories that will be of use and interest to anyone taking a similar journey.

Since Pipe started at the end of 2019, we've gone through many evolutions. We've grown our team from a handful to just over 100 people, working remotely 

. We've experimented with LightMode, sprints, offsites, design systems, Kubernetes, and more. And we've helped some wonderful customers scale by giving them access to capital without requiring them to give up any equity. The market has gone up; the market has gone down. The tide has come in and gone back out*. We've grown through it all by viewing everything we do as a work in progress. And as we've experimented, we've learned a lot about what works and what doesn't. 

Although Pipe has grown faster than any of us could have hoped, it's not because we explicitly planned it that way. It happened because we planned to build a team of incredibly talented and dedicated people who think critically about our company's goals and adjust as needed to make sure that we're always improving our product experience. As a fully distributed and remote product and engineering team, we've naturally aligned around a culture of resilience, collaboration, and compassionate communication.

It's as common for team members to delegate responsibility as it is for us to step up and make a decision to try something so we can move forward even without 100% agreement. Disagree and champion - we like to say. We plan ahead; we learn and always reflect afterward. The details of these decisions, and the lessons that come with them, are what we'll be talking about in Deploy.

Deploy will be home to anecdotes, stories, how-tos, conversations, and more—written by and with our team of Plumbers*, for prod-eng practitioners everywhere. Our goal is to publish anything that we think would have helped us had we been able to read it while building Pipe, and we hope you'll follow along.

* (yes, we know – but consider the alternative!)

What would you like to hear from the Pipe Prod-Eng team about?

A blog about how Pipe’s technology teams build, operate, and think about scaling in a remote-first world.

Check out the lessons learned by our product, engineering, and tech teams as they've built the world's first revenue trading platform

Deploy: A product, engineering, and tech blog from the team at Pipe

Introducing Deploy

The last few years have been marked by era-defining challenges of public health, market instability, and war. These have completely altered how we work, how we communicate, and what we value. Perhaps more than ever before, it’s up to leaders at organizations to create a culture where people know they will be supported in their work and career, regardless of the challenges that they may face.

An empathy-led organization is one which takes its employees’ macro and micro circumstances into account and cultivates a work environment where they can thrive professionally without sacrificing their authentic selves and personal lives.

At Pipe, we continuously work to ensure our values of empathy, safety, and sustainability percolate from leadership to our product and engineering teams, visibly impacting the way we work. We’ve found that while there is no single hotfix that alone builds an empathy-led organization, there are certain steps leaders can take to get started. 

Building empathy requires a deliberate process to be put in place, which is open to iteration and improvement and tailored to the company’s culture and the people it hires.

Assess for culture fit at the hiring stage

Culture usually trickles down in the early stages of startups. The values that leadership brings to the table end up reverberating through the teams. Before hiring, leaders need to vibe-check themselves. To build an empathy-centric culture, leaders must first learn what that means. Only then can they identify candidates for empathy and culture fit.

Assess your current company culture for empathy. 

Empathy is not a project you can ship—it’s a horizontal track that requires continuous improvement; therefore, it’s difficult to answer the question, “Does your organization lead with empathy?” The answer will never be a yes or no, but somewhere along a scale.

Involve your team in the interviewing process. 

Great leaders will involve the rest of the team in the hiring process. At the end of the day, it’s the team that will interact with the new hire on a daily basis and will have the most insight into whether the candidate is a good fit for the team’s dynamic and specific needs.

Help your team define what they are looking for, and teach them how to find it. 

If the team is involved in the hiring process, it’s helpful for them to have a reference guide they can rely on to help them assess a potential candidate. The same guide may not work for every team, but it should at least cover a few basic questions and follow-ups that help them conduct interviews with potential hires.

Ask your team to choose an onboarding buddy. 

An onboarding buddy is someone a new hire can go to with any question they may have about their role, the onboarding process, the product, or anything else without feeling like they are being a burden. The onboarding buddy is usually a member of the team the new hire is joining, and supports them for anywhere between 2 to 4 weeks.

Make sure that the onboarding buddy is not always the same person, and that this responsibility is evenly distributed across the team. The buddy’s workload must be adjusted to account for this new responsibility they will be taking on.

For fully distributed companies, it helps if the onboarding buddy and new hire share similar backgrounds and time zones. It is much easier for a EU-based person to onboard another EU-based teammate—they will most probably be going through a very different HR process than a teammate from the US, and can be available to answer questions about this and more throughout the working day.

Every person has a different communication style—some people thrive in large meetings and video calls, others prefer async Slack messages and deep focus work. Some people may be open to very candid and specific feedback, while others may prefer to be gently nudged in the right direction. As a leader, you first need to understand what your team member’s style is, then adapt your interactions based on that. This usually makes conversations more productive—people will focus more on the topic of discussion, rather than its delivery method.

Ask your teammates to write a guidebook for themselves.

This is a document that captures the basics of a person’s work and communication style. New hires are asked to create this document for themselves, and then share it with the rest of their team. They can also see the guidebooks for the members of the team they are joining.

These documents can be as unique as people want, but there are a few key questions, shared in a guide, that your teammates should answer in order to cover the basics:

While meetings are essential to working together, they need not follow an expected structure—synchronous, weekly, hands-on—to be effective. The types of meetings that a leader or team holds can be thoughtfully tailored to the agenda, team members’ communication styles, time zones, and more. 

Hold project check-ins in the format that works best for your team. 

Project check-ins and status updates are necessary to keep teammates on the correct course, provide guidance where needed, and identify potential issues early. A lot of managers tend to fall back to a one-on-one meeting structure for these, however, there are plenty of other mediums that are effective.

Noticing how a team member is doing at work and, if they bring this up themselves, in their personal life, can help you ensure you’re supporting them in the best way possible. The kinds of things to notice can include a variety of things:

One-on-ones should be structured in a way that enables your teammates to not only share what they’re working on and any blockers they’re facing, but also to grow personally and professionally.

Structure your one-on-one meetings in a way that makes it very easy to check in on all of the above—your mental notes will help you cover what seem to be the most important aspects of their personal and professional life. Filter the meeting topics based on your insight into their work and life. Focus on motivating them and enabling them to grow, rather than micromanaging their work.

Create a safe space for people to work in

Feeling safe and working in a positive environment is a must. Particularly for fully distributed companies where people rarely meet and do not have cues like body language to rely on—furrowed brows to indicate they’re having a tough time or conversations over lunch to share personal news. Remote communication has less fidelity and people sometimes fill in the gaps with negative information, so it’s important to provide less opportunities for miscommunication and misunderstanding.

Encourage “watercooler talk” in a way that it’s not forced.

 Creating spaces to encourage people to chat about non-work related topics is a great way to build camaraderie and team cohesion. People can share what they did over the weekend, their hobbies and interests, their grandma’s best recipes, and more—the possibilities are endless.

 “Unlimited time off” seems to be a common selling point on many job listings these days, but people almost always take less time off than they need. A vague leave policy can lead to insecurity and anxiety when it comes to requesting PTO. Ultimately, it’s up to managers to keep an eye on whether employees are taking sufficient time off, and proactively suggest they do so in order to avoid burnout from creeping in.

Creating an empathy-driven culture is hard work—it requires as much thought and effort as building a product. And it’s never done in one shot, with reflections and iterations being necessary to the process. Leaders not only have to choose empathy as a core value, but thoughtfully work to refine practices that help it seep through every team and aspect of their company.

Taking your employees’ macro and micro circumstances into account to cultivate a work environment where they can thrive.

How product and engineering managers can lead with empathy  

At Pipe, we’re building the world’s first trading platform for revenue. Asynchronous background processes are critical to this, enabling us to download data from various integrations, like billing, banking, accounting; process trades; automatically rate companies; and much more. Almost two years ago we built a no-frills task queueing system to simply and reliably manage these processes. This system is still serving us well, handling up to 700,000 jobs a day. In this blog post, I’ll cover the thought process behind it and its technical implementation. 

Pipe’s backend is built using Go, so for the initial version of the system we implemented the obvious solution—we would launch a new goroutine, Go’s lightweight concurrency primitive, for each job that needed to run asynchronously. The job would then run in the goroutine until completion. For example, we would launch a new goroutine each time we received a webhook from one of the many integrations (such as Stripe, Chargebee, or Plaid) that provide us with secure access to data about the businesses using Pipe’s trading platform. The original process that received the webhook would complete successfully and, in the background, the newly launched goroutine would do the work of parsing the webhook’s payload and updating our database state.

Unfortunately, this approach had significant drawbacks:

We quickly realized that this approach was not appropriate. When we receive a webhook containing information about our customers’ business, we have to process that data reliably and consistently for Pipe to work. All of the problems listed above are usually solved by using a task queueing system. There are many existing systems available off the shelf, each with its own set of advantages and drawbacks. We evaluated them against the following criteria:

We didn’t find any existing solution that fit all our criteria so we decided to explore implementing a barebones task queue ourselves. If we weren’t able to create something that met our requirements in a day or two, we would reconsider the off-the-shelf systems. We ended up with an implementation that has successfully scaled with us as we’ve grown, that’s easily observable using our existing tools, and that’s easy to use from our existing codebase.

 table in our PostgreSQL database. If you want to run a job, you simply insert a row, and the job will be processed automatically. If you want to know, for example, the status of a job or how big the queue is, you can also query the table. The core part of the schema looks something like:

CREATE TABLE backgroundjobs (
    ...
    job_type TEXT NOT NULL,
    arguments JSONB NOT NULL,
    status TEXT NOT NULL, -- (pending|running|completed|failed)
    shard_key TEXT NOT NULL
    ...
)

The columns are mostly self-explanatory—there are multiple job types, each job gets supplied arguments as JSON, and each job has a status. Perhaps the least obvious column is the 

. We use this to control which jobs are allowed to run at the same time. Two jobs with the same 

, say two trades executed by the same company, are not allowed to run concurrently. More on this later.

 etc.) has a job controller that runs in a separate goroutine and is responsible for (1) keeping track of currently running jobs, (2) polling for pending jobs and starting them when there is free capacity, and (3) marking completed jobs.

The logic for doing all of this ends up being rather intuitive—the controller spawns new jobs as goroutines and provides each job a channel through which the job communicates when it’s complete.

type controller struct {
	db          *sql.DB
	runningJobs map[string]struct {
		job    *models.BackgroundJob
		comm   chan (jobOutput)
	}
    ...
}

This is then tied together into a main polling loop where we, in turn, spawn new jobs when we have spare capacity, and check to see if any of the currently running jobs have completed. Polling for new jobs is accomplished using a simple SQL query:

SELECT DISTINCT ON (shard_key) *
FROM background_jobs
WHERE name = $1
  AND status IN ('running', 'pending')
  AND shard_key NOT IN (
      select unnest(${running_shard_keys}::text[]))
ORDER BY created_at ASC;

 jobs that may have been interrupted due to the controller getting stopped during a deploy. By also filtering out any 

) we ensure that there’s at most one instance of each job running at a time.

The second part of the main loop, checking the status of the currently running jobs, works as follows:

for _, jobCtx := range c.runningJobs {
    select {
        case output := <-jobCtx.comm:
            delete(c.runningJobs, shardKey)
            err := c.processCompletedJob(ctx, c.db, output)
            // handle error
        default:
    }
}

Here, we check if any of the jobs have signaled through their channels that they’ve finished. Note the 

The main loop ends up looking something like below. After adding error handling, logging, metrics, logic for updating job statuses, basic dependencies between jobs, and dynamic settings, the complexity of the code grows slightly, but the system is still easy to understand and internalize.

for {
    err = c.spawnNewJobs(c.numWorkers - len(c.runningJobs))
    // handle error
    err = c.checkRunningJobs()
    // handle error
    time.After(sleepTime)
}

To add a controller for a new job type, we implement a function with a standardized signature (

) and register that function along with the name of the job. A nice side-effect of this design is that the code for creating jobs (and marshaling the arguments as JSON) can live next to the code for running the job (and unmarshaling the same arguments).

When we want to run jobs on a schedule, we utilize Kubernetes cron jobs. The cron jobs themselves can be lightweight; each one only lives long enough to insert the desired tasks as new rows with 

The design presented above assumes that only a single process is polling for jobs of each type. This means that each job type can only ever run on a single server, which might sound like a major blocker—that's not webscale!—but the system has held up well for us. This is largely because our workloads are generally not CPU- or memory-bound but rather, are limited by the database or an external API. It’s also possible, to a limited extent, to scale this system horizontally by manually sharding job types between different servers. We haven't had to do this yet, but it wouldn't be hard.

Keeping the polling logic single-threaded simplifies the implementation a lot. Implementing a distributed task queue using PostgreSQL is rather tricky (see, for example, 

 on the pitfalls that exist, as well as how to do it correctly) so it’s generally better to pick an existing solution.

 package has processed all of our asynchronous tasks without any major issues or downtime and without us spending significant resources on maintenance or upgrades. We've done some obvious things like improving logging, adding the ability to automatically retry certain jobs with a specific backoff schedule, and even schedule jobs to run only after certain dates. I’m sure we’ll eventually have to upgrade to a more complex off-the-shelf system (like 

) when we reach a scale that our current system can’t handle. But until that day we’re confident that 

Software engineering is often about tradeoffs—writing your own simple system, adapted to your needs, can be worth the investment, especially if you approach the problem with the attitude that you can adopt a more complex system later on if need be.

A simple custom solution has been managing critical asynchronous tasks for our trading platform for two years. This is how we implemented it.

Deploy

Passwordless Authentication — Moving away from Yubikeys

Introducing Deploy

How product and engineering managers can lead with empathy

How Pipe manages asynchronous tasks using PostgreSQL