jonoropeza.com

Making software and software development teams. Mostly the people parts.


Adding CI & CD is a zero hour task

I've written previously about the importance of continuous deployments in software projects and how, in addition to enabling rapid delivery of value, they play a part in enabling high availability / high uptime systems. Call this a doubling down then.

Adding CI & CD has become a zero hour task for anything I'm building. Meaning that during the first hour of coding on any new deployable, I add continuous integration and continuous deployments.

Why? My experience is that the longer you wait to add CI and CD to a project, the more complexities you've built and the more decisions you've made that might conflict with how you'd like to do CI and/or CD.

Not to mention the habits that have been built by everyone working on the project, and the culture that's been built around how testing and deployments are done around here.

In other words, it just gets harder, and sometimes it gets impossible intractable - technically, culturally or both - to do CI or CD the way you want to do it and should do it.

Whereas if you build CI and CD in from the start, then as long as you're keeping your commits tidy (you are keeping your commits small and digestible, right?) any decisions you make or paths you take that might stray you from CI/CD will be immediately obvious - you'll have a broken build.

This works for all code: data pipeline code, terraform, a web service or a web experience (React with or without Next.js, Vue, Angular, etc). If it's code and you're going to run it somewhere (and if not, why write it?), then test it every time you go to change and, if the build is green, run it automagically when you commit it.

For web services or experiences it's so easy. Init the repo. Install the boilerplate. Then write a test that exits 0. Tell Github to run it automatically on PRs and commits to main. Add a Dockerfile, docker-compose, yaml for Cloud Run or AWS or Azure, done.

If you or your shop write more than 5-6 projects a year, all of this should be in your grab & go toolbox when you do File -> New Project.

It might be a bridge too far, but I'd hazard to say that simple observability should be an hour zero task as well. Or maybe hour one. Nothing elaborate: give me an alert, preferably in Slack or Teams or whatever I'm using, that lets me know if my deployment is red. A simple ping will do.

posted in Software Development


Art of Travel

1997. I was 20 going on 14. Aimless, listless, nihilist, angsty, milquetoast, nervous, still afraid of my own shadow, afraid of doing anything in or to the world. And then luck, or serendipity: I found this proto-blog / unpublished book / howto guide website called How To See The World; Art of Travel; European and World Backpacking; On $25 a Day or Less.

I was hooked. In the evenings after work and community college classes and more work (barista in the mornings, busboy / kitchen jack-of-all-trades by night) I'd read Art of Travel. Chapter by chapter, and then rereading favorite chapters. Absorbing what backpacking was, how to do it, and why.

Ultimately this reading catalyzed a vision: First, I would save every spare dime for a year. Second, I would use my savings to spend a summer - the summer of 1998 - backpacking across Europe.

As rights of passage go, that's fairly banal. A modern version of the hero's journey where the dragon was mostly in the actually doing and the actually going. Still, I believe banalities have the potential for strong personal significance. This one certainly did. I flew from SFO to CDG on May 28th, 1998 still a little boy. I came home on August 4th a human, a man. All thanks in large part to this guy John Gregory who wrote and published this book on How To See The World. (If anyone knows John, or John reads: I'd really like to buy that guy a beer!)

If you haven't clicked on the links above, here's another chance: Art of Travel. Yes, it's still online! Coming up on 30 years old at this point, and parts of it do feel like they're from another time in terms of the authority in the author's voice. Maybe this is a middle aged dude talking, but I find this sort of writing refreshing. From Part 1, Chapter 1 on People and Diplomacy:

THE GREAT REWARD of backpacking independently on a low budget is the people you meet. Because all roads have not been smoothed before you, because your feathers are likely to be ruffled when things don't turn out exactly as expected, and because you are likely to be left in somewhat of a lurch now and then, you will have far greater opportunity to mix with local people, as well as backpackers from all over the world, than any tour group or first-class traveler.
Those spending big bucks for guided travel get peace-of-mind in return. They are guaranteed no worries, no hassles, an experience as close as possible to being home, without being home. They get an hour and fifteen minutes for the guaranteed-open museum, then a two-hour sightseeing ride that catches all the picture-postcard highlights. They break for lunch at a "recommended" restaurant, where the food is reasonable and ordering is easy. And as the next bus pulls in they re-board theirs to repeat the routine, ending with an easy check-in at a reasonable hotel, populated with plenty of other tourists, pretty much like themselves.
While all travel is good for the human spirit, budget backpacking is unparalleled for meeting people and experiencing worlds on their own intimate terms. There are many travelers who have the resources for pampered-class but choose to strap on a backpack and see the world via the seat-of-their-pants, because they know it's the best way to experience cultures and interact with local people.
The best travel is not about a list of monuments, museums, and landscapes. The best travel is about people, and if you travel well it is people that you are going to remember most. People that are strange, unique, foreign, similar, friendly, nice, hospitable, loving, kind, rude, outrageous, and normal. These will be the experiences that stay with you forever, that no postcard can ever reproduce.

And from the section in the same chapter called Tourists, Travelers, and Local Culture:

Travel theorist Stanley Plogg places the personalities of tourists and travelers along a broad scale. On one end are people who want their travel experience to be as "like home" as possible. They want to take it easy and not be faced with stressful situations and decision-making. They want everything to "go right." These are frequently (but not always) the people found at posh resorts, or on group tours.
On the other end are those travelers who enjoy new situations, dig deeply into local culture, and travel as if they were natives of the land. They find lodging where the locals sleep, eat where the locals dine, and use their transportation. They may hitch rides to get from place to place, not only as a means of saving money, but as a way to meet local people.

At the moment, here in early 2023, my wife and I are planning our next trip to Europe. This will be my 10th trip to Europe and 17th 'big trip' since the Art of Travel read including Asia and extended trips in the US, Canada and Mexico. Not bad for a formerly ambitionless 20 year old!

My hot take on reading that passage above from Art of Travel is a bit of embarrassment. I'm remembering that 20 year old sitting in his room reading the above, welling with righteous enthusiasm, thinking "This is how I'm going to travel, I'll never be one of those guy who spends big bucks for peace of mind travel".

Seeing as how I've just booked a West London Airbnb that's costing me as much per night as I would spend on my entire travel budget for a WEEK in 1998, I'm suddenly aware that I've lost it: I am now the traveller spending big bucks for peace-of-mind, an experience that largely feels like being at home during the downtimes, and most of all for everything to "go right".

Combined with an impressive shift towards efficiency in travel markets in the last 25 years - the almost complete loss of the hidden little travelers restaurant or hotel - that makes traveling as a tourist a turnkey pipeline that flows money in significant denominations into the local tourist economy, this is getting really expensive, and often simply feels wasteful.

I'm wondering: Could I somehow shift back on the scale, revert towards if not to budget travel, be ok with things not always just going right?

A reasonable question to ponder on the eve of another trip. On this one we'll spend close to a month in the UK and Iberia. On the way, I think I'll write here and there about the places I'll visit, but mostly about travel, how travel works, why even still travel in 2023 when we've condensed towards a monoculture, and what I think might happen going forward.

posted in Travel


Travel: How Many Nights Should You Stay?

TLDR: Daytrips, two night stays and five night stays are my preferred travel durations.

A problem I often wrestle with when planning trips is how to long to stay at a destination. Should I make the drive to the beach a daytrip, or an overnight stay? A weekend in San Diego: Go on Friday and do two nights, or just Saturday? And then on multi-week international trips, the question is usually about how many nights in each place: 4 nights in Paris, 1 in Lyon and 2 in Burgundy? Or 5 nights in Paris, 2 in Lyon and Burgundy as a daytrip?

What am I trying to do when traveling? I think the answer is something like this: Maximize the experience of being in a different place, hit minimum thresholds of pleasure and comfort, weighed against the investment of time and money.

The part about pleasure and comfort being requirements to be satisfied rather than variables to be maximized is key, and it took me a long time and a lot of suboptimal trips to learn that.

Pleasure. There's no point flying halfway around the world to maximize pleasure. Unless you have a particular fascination for the chef, the ingredients or the cuisine? I think you do the blowout Michelin dinner at home. Massages, laying on the beach, the blowout bottle of wine - best done as close to home as possible. A dessert on top of an already big meal is great on a mundane weekend, but on a trip if it means a bad nights sleep with indigestion then it's best skipped. Etc.

And comfort: Comfort is incredibly valuable up to a mostly subjective point, and beyond that it's all diminishing returns. A comfy bed in a fairly quiet, safe hotel is fantastic. Spending extra for the best of the best hotels is pointless when you'll be jetlagged and can't sleep at 3AM anyway. The second class section of the train gets there at the same time as the first class. Planning and booking 80% of the trip ahead of time is great, but going to 99% and having every last detail plotted is pointlessly restricting.

All of this as we're currently planning a trip to Europe. 3 1/2 weeks in the UK, Spain and Portugal. We're probably going to stay in quite a few different places, possibly up to a dozen. One of the questions I've been wrestling with is how long should I stay in each place? After we get a car, should I book overnights? For the cities, should I book 3 nights or 4?

I wanted to get a formula down that I could just plug in to a complex trip like this and go.

It probably won't surprise those of you who know what I nerd I am that my approach was to put all my recent trips into a spreadsheet with one row for each destination and the number of days as a column.

(Those of you who really know me will know that I already had all my trips in a spreadsheet like that)

Here's what I found, and how I'm going to plan all my travels from now on:

Daytrips. I underrate daytrips. Especially during the long summer days. The beauty of the daytrip is threefold: They're light on the logistics, especially if you have a car. They're also budget friendly, especially if you've got a taste for spendy Airbnbs and hotels. And they require the least commitment, either the (a) front or backend of a weekend with another weekend day to relax or (b) a mid-week day off.

A daytrip is the MVP of trips: Out of the house, into a new space, and if that space is within a couple hours of home then even though the driving (or the train or bus travel) can be hectic, there's also time to explore and experience a different space.

I need to take more daytrips.

Overnights. If daytrips are great because they're lightweight, overnights suffer from the opposite problem. They're the worst from a logistics-to-quality-time ratio perspective, since everything static in your pack the bags game is done whether it's for a one night stay or a ten night stay. The same applies to costs: cleaning fees, service fees, anything static is going to be at its maximum percentage of total spend on an overnight.

The worst aspect of overnights though, and what really makes them my least favorite duration, is wrapped up in this: Any day I spend flying, driving or taking a train or bus long distance, checking in to a place, checking out of a place or otherwise lugging luggage and settings things up is a measurably worse day than a day where I do none of those things.

I don't know exactly what's going on here, if it's the stress of movement or the uncertainty of setting up somewhere new, but I've talked to a lot of people about this and nobody has disagreed that they feel an elevated amount of stress and discomfort around these movement days.

And that stress or discomfort or whatever it is, it gets in the way of experiencing the place I want to experience. Which, as mentioned, is the primary motivation for traveling in the first place.

So if days of movement are to be avoided, and days with none are to be maximized, what's so bad about overnight stays becomes evident: Overnights are two days of doing the things I don't want to do and zero days of doing the things I do.

The worst worst might be an overnight on a Saturday. It's usually the most expensive night to stay anywhere, and it takes two days that are usually relaxing and makes them stressful. Blech!

Two nights. Now we're talking. I love two night trips for one important reason: That day in between the day getting there and the day getting away is almost always a great day.

I've thought about this, and tried to break down why that is, and I suspect it comes down to having two travel days bookend the one true vacation day. There's a focus that happens. All the best activities, sites, conversations, whatever get concentrated on that middle day. In the memories that follow, the middle day is the trip, the bits of the experience that live on in my mind.

Three or four nights. I won't turn down three or four nights, but when I look at the column where I wrote What Would I Do Differently, half of the three or four night stays say "stay shorter" and another significant chunk, maybe 20%, say "stay longer".

Here's what I think happens: When we stay three nights there's two of those inner days. The first one is usually good and then the second one gets contrasted with the first one. Why? I don't know, that's just what we do, or at least what I do.

I don't know why four nights doesn't work. But it doesn't. I almost always either wish I'd stayed an extra day or I'd stayed a day or two less.

Five nights. YES. Over winter break we took some time between Christmas and New Years and drove to Joshua Tree. Staying five nights in Yucca Valley allowed us to have an intense hiking day, an intense cocktails + vintage shopping day in Palm Springs, and several more relaxing days with easy hikes or sightseeing. In short, five nights in a destination gives us the opportunity to get a little bored. And getting a little bored is a great way to notice things, do something seemingly banal, and really experience the place you're in.

Another thing that happens with five nights is that we almost certainly get so see weekdays turn into a weekend or vice versa. That's an interesting thing to do anywhere, and especially in a big world city like San Francisco, New York, London, Paris.

On our upcoming trip to Europe, we're starting in London and staying five nights. Arriving on a Wednesday, we'll get to settle into the city and then see a weekend arrive, feeling the anxiousness of a Friday lunch, the excitement building in and around the pubs as five o'clock arrives, and finally the rush of a buzzing happy hour with, if April graces us with sunny weather, pubs pouring out into the street as they do in London. I've seen that the summer I worked in London 23 years ago, and I'm excited to see it again.

Six or seven nights are great, too. I think in terms of memories, there's not a lot of difference between them and five nights, so while I wouldn't hesitate to spend one more night in a great destination, I do think there's an element of diminishing returns there.

Beyond that, well, I don't have a lot of experience being in one destination longer than seven days. I suspect that there's a serious gap between a week and about six weeks, which is about the minimum amount of time you need to be in a place to feel a seasonal change, which is magical and probably pretty rare for most of us - it certainly is for me. Six weeks to three months probably does that really well.

And then, just to really get into the silly, I think the next step up is at 3-5 years which is the amount of time I was living in San Diego and then Portland before I really felt like I knew the cities, their rhythms, their mindsets and how they worked at a deep level.

Hope this helps someone out there think about how to max their days off and travel spend. It's helped me plan my current trip. Big asterisk: All this is mostly subjective, so I'm probably overrating the advantages of the timeframes I like, and conversely overrating the disadvantages of the ones I don't.

posted in Travel


Ernest Hemingway's Writing Technique (That Works For Writing Code Too)

About writing, Ernest Hemingway had this to say:

The best way is always to stop when you are going good and when you know what will happen next. If you do that every day when you are writing a novel you will never be stuck. That is the most valuable thing I can tell you so try to remember it.

You should always stop when you know what happens next.

This has nothing to do with stopping, by the way. It has everything to do with the difficulty of starting again. The unspoken premise of the quote - the part of the iceberg that's below the water, to just keeping rolling with Papa - is that inertia is powerful, and getting started is the hardest part.

If you know what happens next, then starting again is easy.

And once you're rolling, you've got momentum.

I've written a novel, and I've written software big and small. These are completely different skills but a lot of the frameworks, processes and pitfalls are similar. Just like a novel, a software project develops threads and themes and patterns and a voice. You want to produce in a sprint, a month, a quarter a thing that is logically consistent and emerges at a consistent pace so you and your teammates (editors) can shape it, critique it and hone it.

For me, when I apply this to coding, I try to end each session when I know exactly what work needs to be done next. For example, towards the end of the day on a project to build a new product page, instead of trying to squeeze a few more lines of code out or fixing a bug, you might stop and write the following note to Future You:

Next step: Add inventory to the product route.

  • fetch inventory
  • render it on screen
  • interstitial (spinner) and error handling
  • think about retries and caching, probably not but at least explore it and make notes
  • observability and monitoring, how will we know if it's down, add a test or two
  • come back to that bug in the product description

And it's there for you at the start of the next day.

Contrast this to how many of us work, which is trying our hardest to completely finish something every session. Often ending up either with a mad dash to the finish that burns you out for the next day, or failing and falling short which bums you out for the next day.

Here's what the two ways of working look like visually:

That's your work until it's done pattern in Red, and then Hemingway's system in Green.

The important two things to note are that Red had a cold start at the beginning of a task four out of the five days (compared to one on Monday for Green), and that Red had to push extra hard (maybe getting to work early or staying late) on Friday while Green had a more evenly paced week.

A trick to this system is to make the first task an easy one to get you going with a win. So in the above example, only start by fetching the inventory if the endpoint is known, you already have the creds, you already have patterns in your app for fetching etc. If not, I might start by mocking it and putting it on the screen. Piece of cake.

A potential benefit to this system in software - and I don't have any data to back this up, but I have a strong gut feeling that it's true - is that a significant number of bugs are written and patterns violated when we're stretched and stressed. The traditional way of working produces two stressful states that the Hemingway method avoids: The cold start at the beginning of a session, and the rush or grind to finish a deliverable at the end of a session.

Finally, it should be noted that Hemingway's technique can be applied at multiple levels of regress. Hemingway wrote daily, so the advice came from the perspective of stopping one day and picking it up the next. But you can apply it to your morning session and your afternoon session. Or to pomodoros, or maybe most realistically to those days when you have 8-10 to code, then you have meetings, lunch, more coding 1-2, another meeting, then finish out the day with a third coding session from 3-5. Finish each coding session at a good stopping point, and then spend the rest of the time writing out exactly what comes next.

posted in Software Engineering


Models of Automation

With Bitcoin price weakness and the novelty and/or wonder of the latest ChatGPT iteration pulling our collective heads out of the Web 3 space, I've been thinking about the models of automation, the variables used and how they're chosen.

To illustrate the models, let's use an example of making a service that provides answers to questions such as "what are the approved treatments available for this or that disease" based on the latest biomedical literature.

There are at least five models we might use to enable this service, they are:

  • No automation at the core
  • Automation as Sentry
  • Automation as Helper
  • Autonomation
  • Full automation

No automation at the core - The human does the thing. You're dealing with a small local company, you call a phone number or send an email, a human answers the phone or replies. I used "at the core" deliberately because a modern phone call or email relies on a lot of automation to move information around. Here I'm referring specifically to how the service creates and ships the answer to its queries.

If we don't use automation at the core of our service for looking up treatments, human operators are answering each message without anything automatic happening to either help them, guide them or guard them against bad or wrong answers. This is the default state of any operations-driven company that hasn't set up their service tech-first. There's nothing wrong with it. And nothing particularly exciting, either.

The sentry - The human does the thing, with warnings provided by machinery. Think collision detection in a car. Or a data entry form where if you put in an outlier, it warns "hey, that number you just entered is 10x the other thousand and one numbers in that column, you might want to check it."

With this model applied to our service, human operators would be responsible for answering questions. The interface they type into might provide some guardrails around length of reply, and might prevent them from including offensive language or committing a HIPAA violation.

The helper - The human does the thing, aided by the machinery. A classic example is narrowing down 100 choices to the likeliest 10. That would be helpful convergence. Or, a canonical example in healthcare being a radiologist assisted by AI to say "hey, I think you should look again at that scan, it looks like these other tumors I've been trained on" - in this case, helpful re-divergence after a human might have converged too quickly.

Here, in our service, human operators are still answering questions. Their interface shows them likely answers, previously accepted answers, or otherwise points them towards papers where they might find the answers. Ideally they would usually use the automated help but in rare cases would use their judgment to go outside the suggested responses.

Autonomation - https://en.wikipedia.org/wiki/Autonomation - Here the machine does the thing, with a human there to detect edge cases, take over from the machine when needed, and ultimately improve the system.

In this case, our service operators would watch questions come in, and see machine proposed answers on their screen. There would be a human-friendly-timescale pause, let's say twenty seconds, during which time the operator would have an opportunity to "stop the line" and prevent an incorrect, offensive or otherwise undesirable answer from being sent.

Full automation - Here the machinery does the thing unsupervised, usually (hopefully!) with heavy post-hoc observability. This sounds frightening but it's actually extremely common. Go to Google and type a search. Your results are returned fully automated. Google has observability over failures, but there's no humans involved in each resultset.

This last model would look a lot like Google, or frankly most popular sites where you're searching for a thing (Amazon, Airbnb, Door Dash, etc). Answers are shipped automatedly to the user, with operators reviewing logs, aggregates, complaint reports, etc. This is really hard. This is why search is so hard. All the above models take longer than our expectations of a web application. Autonomation might be the most frustrating one because it's possibly the best, but it can't operate at the speed we expect and it's impossible to scale wide.

There are probably a lot of markets where the answers are valuable enough and the application narrow enough where autonomation is not just possible, but the preferred model.

posted in Artificial Intelligence