Starting Over With Purpose - My Serverless Adventure

Serverless has been quite the adventure for me. It started in early 2019 when I was given the task to "go figure out this whole cloud thing."

As I mentioned in my last post, my team and I had our work cut out for us. In addition to learning about serverless, we also had to learn about multi-tenancy, NoSQL data modeling, CI/CD, pretty much every buzzword related to the cloud was on our radar.

We spent about a year learning and building proof of concepts until we finally began building "production-ready" software. The first day we started building for real, the whole team showed up to work all gung ho and ready to code. It took until around noon before we regrouped and all mutually felt like we weren't ready.

There was so much to consider that proof of concepts hadn't prepared us for. But we were an R&D team. We were pioneering this space for our company. We didn't have a team of experts to turn to when we had questions.

So we just....did it.

We went with what we knew, or thought we knew at the time, and tried to solve the business problem. We'd learn as we go and promise ourselves to come back around and improve our solutions as we learned.

You'll never know less than what you know know right now.

I couldn't be more proud of my team. While many engineering teams might be stopped dead in their tracks with uncertainty, mine charged right ahead. We built not one, but two serverless apps and got them live in production in record time.

Of course we made some mistakes along the way, but we all know there is no such thing as perfect software. Especially when you're learning.

I can confidently say we were able to accomplish what we did because we followed the do it, do it right, do it better mindset.

The "Do It" Phase

The initial development phase of any project is often considered an upgraded proof of concept. Meaning you need to do what you need to do in order to get it out the door, but you also need to follow security guidelines and any best practices you might already have.

Your objective with the "do it" phase is to do just that.

Do it.

Solve the business problem. Do whatever you have to do to scrape by with the tools at your disposal. You know that you don't know much at this phase.

When working with serverless, that means use the services that most easily do what you need. Don't worry too much about optimizing performance costs or scraping milliseconds off an API response. Focus on getting the job done and getting your hands dirty. Tie API Gateway to Lambda to DynamoDB as often as you need. It's the swiss army knife of the serverless world.

The experience you build while writing production software is going to be crucial when you come back around the second time. You don't need to worry (too much) about doing it right. This phase is about proving that you can do it in the first place.

On our first serverless app we started small. Our application was designed to send documents to someone to sign then return it to the owner. We took the simplest use case from the application domain and built it.

The "do it" phase for us was to allow users to upload a document to our website then pass it along to a preconfigured user. The user would get an email notifying them of something to sign. When they login, they are presented with a hosted version of the document. After the document was signed, the uploader received an email saying it was done. Simple.

We hardcoded values instead of writing configuration pages. We built targeted API endpoints that did.....more than they should. But we solved the business problem.

The fact that we took an unknown tech stack (to us) and built something with it meant the world to my team. We saw the value in serverless and proved to ourselves and to the company that it could be used for more than just fun side projects.

And most importantly, we learned about the business problem and the tech stack.

The "Do It Right" Phase

The next phase is where you start to get serious. You've seen what serverless can do and gotten exposed to how to build. Now it's time to build out more use cases, but with a different approach.

Now that your comfort level is higher at building a production-ready app, it's time to start thinking ahead. About building it right.

After your application goes live, how are you going to monitor usage and track issues? What is a sustainable way to deploy to production via CI/CD? Have you recognized any patterns in what you built so far and are ready to establish development governance rules?

These are questions that must be answered early on in your development cycles, but not so early that it's in the "do it" phase. That phase is for learning.

The "do it right" phase is about establishing norms and setting yourself up for success.

With my first project, this was the phase where we started tightening the bolts. Configuration pages were created. Dead letter queues were configured. Deployment pipelines were hardened and templated.

We also took the skills we had developed from the earlier phase and applied them to more difficult business problems. We added features to build dynamic data entry pages that populated documents before sending them to configurable destinations.

We began building direct integrations from API Gateway to DynamoDB and incorporating Step Functions. Features for post-signature document management were added as well. We built our application to be truly production ready.

Build on your knowledge from the first phase. Identify repeatable patterns and normalize them. Make your team more efficient and your code more maintainable.

The "Do It Better" Phase

Toward the tail end of the project, we had a shift in focus. Once we had a go-live date set and the deadline was approaching, we had a renewed sense of urgency to optimize the application.

Early on in development, I had done some initial serverless cost modeling for our app. But as our application grew, so did our confidence in serverless. Which ultimately meant the app increased in complexity and the cost model needed to be revisited.

By now, we were about 2 years into development. We considered ourselves to be advanced serverless developers and knew how to quickly identify and resolve issues when they popped up.

Now we were onto optimization. Not just cost and performance optimization, but complete project optimization. Refining processes, fine-tuning app features, and building for our support organization were at the top of mind.

Optimization is the key focus of the "do it better" phase.

We performed load tests at 5x expected traffic to identify bottlenecks when we reached scale. We began refactoring the application to ensure all microservices, Lambda functions, and DynamoDB tables were following our best practices. We developed our disaster recovery strategy.

These are the hard pieces of software development that we needed to wait on. Expertise needed to be built in order to make informed decisions on these mission critical components. If we had done them initially in the "do it" phase, we inevitably would have setup guard rails that held us back the whole way.

Iteration is crucial in building strong, production-ready software.

You won't build it right the first time.

Ever.

Learn from your mistakes, enhance your skills, and build informed decisions on them. Use your experience to guide you to into making better software.

Conclusion

Each phase is not meant to be a complete refactor. Rather, you want each phase to teach you something you can build on for next time. With the "do it" phase, you're understanding the broad strokes about how your tech stack works. Can you build production software with the skills you have on your team?

The "do it right" phase is meant to build on your comfort and lessons learned and turn your groundwork into a production application. Get comfortable with the nuance of serverless. Get to know stateless architectures and how they differ from a stateful app.

Identify and establish development patterns as best practices. Start spreading the word about how to build with serverless.

The "do it better" phase focuses on optimation. How do you take your application from "adheres to best practices" all the way to "singing like an angel and costing next to nothing"? Try to identify areas that will help others support your application.

Optimization does not always mean performance enhancements.

It's about building better processes, updating existing best practices, reducing costs, and so much more.

In my journey, we spent about 25% of our time in the "do it" phase, 65% in the "do it right" phase, and 10% in the "do it better" phase. But that isn't necessarily the best breakdown for you.

Time in each phase will vary based on the comfort level of your team and the amount of experience they have going into it. My team was brand new to serverless, so we spent some extra time discovering the tools at our disposal. By the time we reached the "do it better" phase, the team was giving presentations to the rest of the company about serverless.

I talked a lot about my journey building an application from scratch, but that's not the only use case for the "do it, do it right, do it better" mantra. You can do this on a much smaller scale, even down to a feature level.

If you want to implement new tech or something totally unknown to you into your application, follow the process above. Take a first pass while you learn and make it work. Then revisit with the knowledge you gained from the first time.

Finally, try to poke holes in it and fill them immediately. Figure out what you need to do to make it sustainable for everyone.

This has been my mantra for years. I don't imagine I will ever build software another way again. Continuing to make improvements as you learn and staying away from premature optimizations will help keep you agile and in a mindset where change is good.

Happy coding!