
In the realm of system design and architecture, how backend services communicate with each other hold the key to delivering seamless functionality to users. Internally within a company, this can be done through shared databases, remote procedure calls, or queue messaging systems.

However, providing functionality often involves tapping into external services provided by other companies. A prime example is the use of payment acceptance endpoints, typically APIs accessible over HTTPS, offered by payment service providers (PSPs). PayPal, Adyen, and Stripe are some examples of PSPs which require API consumers to talk to their services through the APIs they provide.

But how do we ensure this communication remains robust, even in the face of faults, timeouts or downtime? Designing APIs and communication streams effectively becomes crucial, especially in cases where speed is of utmost importance because it affects the experience of the users of the system. For this discussion, let's consider several things:

- endpoint: a specific place where a web service can both receive requests and respond
- resource: data which we want to perform actions upon, for example a hotel booking
- state: there may be a number of state which the resource is currently in. The actions and their statuses govern the possible transitions between states.
- action: the act of doing something on a resource, for example creating a hotel booking
- status: the current result of the action on a resource - typically there are 3 statuses for each action - pending, success and failed.

In the internet experience of today, we expect webpages to load within the order of 500ms, and websites to be stay responsive to human interactions. However, not all actions can complete within seconds, and sometimes it may even take longer than stipulated connection time limits set along the communication chain. We wouldn't want to hit a timeout error and be left hanging wondering whether the action was completed successfully or not. As service providers, setting up endpoints requires thoughtful planning.

## Restaurant Booking Service Example

Let's take the scenario of designing a restaurant booking service as an example with the following requirements:

- resource: the resource here is a single restaurant booking
- action: actions on the resource can include: create, cancel, update

The states of the booking resource can be something like this:

<center>
<img src="/assets/fsm.drawio.svg" style="width: 50%; min-width: 500px" />
</center>

Time to time, service consumers may send in requests via the `PUT /booking/{bookingCode}` endpoint to create a booking:

<center>
<img src="/assets/seq01.drawio.svg" style="width: 50%; min-width: 500px" />
</center>

Let's say the service level agreement was for the create booking endpoint to return a status within 5 seconds. If a final status was reached within 5 seconds, the response of the create action from the booking service would tell whether the creation was a success or whether it failed.

If the booking creation takes longer than 5 seconds, then the booking service can respond with a "pending" status of the "create" action. Subsequently, the upstream services can retry sending the request again. The `bookingCode` would then serve as an idempotency key, and would ensure that no duplicates of the booking resources would be created from multiple instances of the same request identified by the `bookingCode`.

## Getting Details about the Resource

Another way that upstream services can learn about the current state of the resource is to ask booking service at the `GET /booking/{bookingCode}` endpoint about the current state about the resource.

<center>
<img src="/assets/seq02.drawio.svg" style="width: 50%; min-width: 500px" />
</center>

When and how frequent to poll the endpoint is a key design consideration when building the services.

## Providing Latest Knowledge About Resource to Upstream

To also ensure timely updates on status changes, a callback or webhook design can be factored into the design. This setup facilitates communication on state changes on the resource back to the service consumer, essentially providing a downstream to upstream synchronization of data. This allows information exchange between your service and the service consumer, even if either one party faces a temporary downtime.

In the FSM above, you probably noticed that there's a transition defined from the `ready` state to `archived` when the date of the booking is one day before the current date/time. Perhaps there is a background task that would clean up all bookings in the previous day and move them from `ready` state to `archived`.

In such scenario, the upstream services wouldn't know about the state change, unless the booking service provides an update in the form of a callback.

<center>
<img src="/assets/seq03.drawio.svg" style="width: 50%; min-width: 500px" />
</center>

This robust setup ensures that regardless of any temporary hiccups in communication, crucial information about actions like creating a booking can still be transferred between service provider and consumer when services are back online. The design isn't just about making things work — it's about making sure they work reliably.



Starting a new software engineering role can get a little overwhelming at a new place. As I navigated through my new role in the past months, I was reminded that as much as I was observing the existing work culture, processes and team norms in my new position, from the moment I joined I'm now a participant rather than an observer.

These are 3 things that can be of helpful reminder for anyone starting out.

## 1. Ask questions, often

Whether it's your first day or 5 years in, communication is key to alignment and understanding.

It's often that we don't ask questions, for fear of asking a "wrong" or "stupid" question. That shouldn't be the way. I think it's alright to just ask, clarify and understand regardless of what others think of your question.

That way as you go along, you learn how to ask better questions that can (1) put you and your team on track, (2) navigate you and your team around potential roadblocks, and (3) help you deepen your understanding of the value and meaning of your contribution. 

It's way better to ask a "wrong" or "stupid" question than to be a muted team member, because being muted no one else will know what you know. Silence doesn't always mean alignment. 

> If you find yourself in a place where you're often condemned for asking a "wrong" or "stupid" questions, you might want to consider if the environment you're in is helping you grow. 

You might want to find out what's a good way to ask your question, whether synchronously in a Zoom call, asynchronously on the meeting chat, in a 1:1 direct message, or asking it in an open space like wiki comments. 

## 2. Get involved, early

There are opportunities for growth for you when you get involved. You get to learn and hone both technical skills and soft skills when you do this.

For example, as a junior software engineer, if you're assigned a project to build a backend software system, there are several benefits of getting involved early with:

- you can shadow your lead / principal software engineer on the system design and pick up after them on the good stuff
- you can learn about the problem domain better, what are the business / product requirements, how are they derived etc. in turn this helps provide meaning and context to your contributions

## 3. Close the feedback loop

Words matter, a lot. 

As much as you want to feel appreciated for your work from peers and leads, you can take the first step to thank others every step of the way. For a peer who has gone out of their way to help or guide you, you can:

- Leave a thank you message to appreciate them
- Treat them to coffee

It cost nothing to say thanks genuinely and to play a part in building a positive work environment.

<div style="width:100%;height:0;padding-bottom:56%;position:relative;"><iframe src="https://giphy.com/embed/xZxtYEV4GHQtgccxIl" width="100%" height="100%" style="position:absolute" frameBorder="0" class="giphy-embed" allowFullScreen></iframe></div><p><a href="https://giphy.com/gifs/TheSwoon-netflix-swoon-theswoon-xZxtYEV4GHQtgccxIl">via GIPHY</a></p>

All the best in your journey too. Check out the additional resources:

- [Top 10 pieces of advice for rookie software engineers](https://medium.com/singapore-gds/top-10-pieces-of-advice-for-rookie-software-engineer-f1a2b38f1d56)


Backend services are the procesing blocks that empower the logic of an application served well. They do all sorts of things: talk to databases, keep things in caches, process data from message queues, serve webhooks for external systems, process images, or even send push notifications to users' mobile devices - indispensible part to a well designed software architecture, whether for a web or mobile app.

To make services more reusable and being able to change behaviour without changing code, configuration parameters are typically provided to tell the services what kind of environment it is operating in. For example:

- `NODE_ENV=production`: to inform that a service is running in production environment
- `DB_HOST, DB_PORT`: which database server to connect to
- `DB_USERNAME, DB_PASSWORD`: what user account to use when talking to database
- `JWT_SECRET`: what JWT shared secret to use
- `REQ_TIMEOUT_LIMIT=5000`: what timeout in milliseconds to use when sending requests to other services

Sometimes backend services deployment fail way past deployment because of misconfiguration. It is possible to enter the wrong `DB_HOST` hostname, missed out on setting the value for `JWT_SECRET` or even have a typo in the key or value of any configuration parameters. As services grow larger, we tend not to encounter these errors early on until the particular section of code gets executed under some specific conditions.

For example, if your database connection uses lazy loading and/or connection pooling, chances are that your backend service doesn't establish a connection to the database at start up (or shortly after deployment) until it is needed while serving some request much later.

> We end up celebrating too early at a successful deployment only to find the services failing while handling workload.

## Enter Power-on Self-test

In the world of hardware and embedded systems, it is typical for engineers to embed a set of self-testing routines called [Power-on Self-test](https://en.wikipedia.org/wiki/Power-on_self-test) (POST). Simply put, when the system starts up

Power-on Self-test is much more common than we know it. They are commonly found in

- airplanes
- medical equipment
- computers
- phones
- gaming consoles
- TVs

> Can you imagine yourself in a plane up in the air, only to find the plane stalling mid-air because of fuel amount was in reserves during take-off?

Could the plane's software system checked on the amount of fuel remaining before allowing the pilot to start the engine?

Likewise, to build resilient backend services, it's important to include some form of POST routine at service pre-initialization. By allowing backend services to fail early on shortly after deployment, services wouldn't fail while serving requests or processing important data.

In an Express.js HTTP API service, I'd write a POST routine like this:

```typescript
// post.ts

/// .... imports goes here

const { JWT_SECRET } = process.env;

export async function powerOnSelfTest() {
  // check if JWT secret was configured and can be used to sign and verify payload
  assert(JWT_SECRET);
  const token = jwt.sign("testData", JWT_SECRET);
  const payload = jwt.verify(token, JWT_SECRET);
  assert(payload === "testData");

  // check if database connection and credentials are OK
  const dbConn = createDbConnection();
  try {
    await dbConn.authenticate();
  } catch (e) {
    console.error("Error connecting to DB", e);
    throw e;
  }
}
```

Before starting up the HTTP listener, I'd run the POST routine:

```typescript
// index.ts

/// .... other imports
import { powerOnSelfTest } from "./post";

async function startApp() {
  const app = express();

  const db = await initDbModels(createDbConnection());
  app.locals.db = db;

  /// ... setup routes

  const port = process.env.PORT || 3333;
  const server = app.listen(port, () => {
    console.log(`Listening at port ${port}`);
  });
}

// start up the application
powerOnSelfTest()
  .then(startApp)
  // catch POST and startup errors here
  .catch((err) => console.error(err));
```

Even if all the environment variables provided to the application were correct, the POST routine would at least check for connectivity between the service and dependent servers like the database. Problems like version incompatibility, network connectivity or incorrect infrastructure setup, can also be caught in the POST - _at very little cost_.

Having POST routines in place, I can be _slightly more_ confident that my backend services would work beyond post-deployment.

POST routines work as well as they are written. As [Edsger W. Dijkstra](https://en.wikipedia.org/wiki/Edsger_W._Dijkstra) puts it, POST routines - like any other tests - can only prove the presence, **not** the absence of bugs. You can instead embrace POST routines as another hammer in your toolbox in catching catch potential failures early on.



I've found [react-query](https://react-query.tanstack.com/) to be pleasantly straightforward and easy to use in managing server-side state on React and React Native apps, with plenty of support and resources from the likes of [TkDodo's blog](https://tkdodo.eu/blog/practical-react-query) and [online courses](https://ui.dev/react-query?coupon_code=aQcnd).

React Query deals with, you guessed it, [queries](https://react-query.tanstack.com/guides/queries). As you build up your React application, you will find yourself facing some of these scenarios:

- you have queries that execute depending on the data returned from an earlier query: you can run conditional queries like in the guide on [Dependent Queries](https://react-query.tanstack.com/guides/dependent-queries).
- you have queries that can run in parallel: [`useQueries()`](https://react-query.tanstack.com/guides/parallel-queries) is there to solve parallel queries
- you have the same query that will be requested by multiple components in the same render cycle: It's more favourable to request for the same query multiple times than to perform [props drilling](https://blogs.perficient.com/2021/12/03/understanding-react-context-and-property-prop-drilling/), as you can configure [`staleTime`](https://react-query.tanstack.com/guides/important-defaults) to your needs

TkDodo has written about [creating custom hooks](https://tkdodo.eu/blog/practical-react-query#create-custom-hooks). Using custom hooks makes reading a component's code a breeze, because `useQuery` and how data is fetched gets colocated together in a custom hook. For example, a custom hook to fetch the list of todo items:

```typescript
export const useTodoListQuery = (filter: Filter = {}) => useQuery(['todos', filter], () => fetchTodoList(filter));
```

As I encountered more of those scenarios I described earlier, I found the need to reuse queries written in `useQuery` to also be used in `useQueries` for parallel execution. To make things more manageable, I decided to write utility function for constructing query options:

```typescript
export const buildTodoListQueryOptions = (filter: Filter) => ({
  queryKey: ['todos', filter],
  queryFn: () => fetchTodoList(filter),
});

export const useTodoListQuery = (filter: Filter = {}) => useQuery(buildTodoListQueryOptions(filter));
```

In the event that I need to build a portion of User Interface that needed to run the `useTodoListQuery` in parallel with something else, say loading calendar items, I can do this:

```typescript
export const buildTodoListQueryOptions = (filter: Filter) => ({
  // ...
});

export const useTodoListQuery = // ...

export const buildCalendarItemsQueryOptions = (dateRange: DateRange) => ({
  queryKey: ["calendar", dateRange],
  queryFn: () => fetchCalendarItems(dateRange),
});

export const useCalendarItemsQuery = // ...

export const useLandingInfoQuery = (
  dateRange: DateRange,
  filter: Filter = {}
) =>
  useQueries([
    buildCalendarItemsQueryOptions(dateRange),
    buildTodoListQueryOptions(filter),
  ]);
```

This makes it easier to build composable queries. That said, I don't think everyone should start going around and pick out their query options from whereever `useQuery` appeared. The abstraction should only occur when there is a good enough reason to do so, like when a particular query needs to be used in multiple instances of `useQuery` and `useQueries` to enhance user experience and loading time.

What do you think?



Last week, I integrated [Utterances](https://utteranc.es) to my blog over a couple of hours and shortly after integration, I found out about [Giscus](https://giscus.app/) and [laymonage](https://github.com/laymonage) came over to tell me that Giscus comes "with a ready-to-use [component library](https://github.com/giscus/giscus-component)".

Being "heavily inspired by utterances", Giscus is similar in many aspects (client-side integration, custom themes, reactions etc.) leveraged on [GitHub Discussions](https://docs.github.com/en/discussions) to host comments rather than GitHub Issues. Since I was familiar with how Utterances worked by then, I was able to quickly switch over to Giscus in an hour or so. Additionally, Giscus allows you to:

- add reactions to the page itself - these are stored on the discussion original post and can be disabled by option
- replies to comments, nesting - this is a feature I understand to be not supported by GitHub Issues
- sorting of comments by oldest/newest
- configuring if the comment box should be placed above all comments - I preferred not to scroll through the long list of comments before adding my comment, though this is purely the user's preference.

One thing I was facinated about, was that the comments widget did not reload fully when themes changed. I dug into the component library's source code and found that Giscus's component library leverages on [message passing to the iframe](https://github.com/giscus/giscus-component/blob/main/web/src/giscus.ts#L202) to update changes occurred in the component props. When using Utterances, any change in the props would result in a full reload of the script and iframe.

<img src="/assets/d3c0caadbf.gif" />

Giscus worked nicely, and I guess it's here to stay on my blog. Kudos to the Giscus team for building on the goodness that Utterances brought about.



Awhile back when I was reading up on [TkDodo's commentaries of react-query](https://tkdodo.eu/blog/practical-react-query), I noticed that TkDodo's blog has a nice comments section that looked very familiar. Upon closer inspection, I realised it was a widget called [Utterances](https://utteranc.es/), which allowed comments to be stored on a GitHub repository's issues.

This was neat for a fully static blog like mine, where I just don't want to host a database somewhere just to store comments. So I decided to try adding it to my built-time generated blog.

The setup was straightforward, I simply fill up the form in the `configuration` section, and it populates my input into the `<script>` tag below. I thought it would be a straightforward copy-paste, but TypeScript came complaining about `repo` being an invalid attribute for the `<script>` tag.

I looked around GitHub for React implementation or integration of Utterances, and found these two:

- <https://github.com/wicksome/utterances-react/blob/master/packages/component/src/Utterances.js>
- <https://github.com/b6pzeusbc54tvhw5jgpyw8pwz2x6gs/react-utterances/blob/master/packages/component/src/ReactUtterances.js>

So the trick was to dynamically assign the special attributes (like `repo`) to get by TypeScript complains for `<script>`. Along the way, I encountered the issue of the injected `<script>` tag being removed while the `<iframe>` tag as being injected asynchronously via `insertAdjacentHTML` (See [line 77](https://github.com/utterance/utterances/blob/36695bf89ffb2ea10eaa26f1b5cbc90477704b9f/src/client.ts) of the utterances source code).

My solution? To enable proper re-render of the Utterances widget, [I wrote a copy of the component](https://github.com/mauris/mauris.github.io/blob/main/apps/main/components/Utterances.tsx) taking references from the two other implementations mentioned above. My implementation takes care to track the number of script instances being loaded, and once the last instance has loaded, it will remove all duplicated instances, leaving only the last "up-to-date" instance in the DOM. This works neatly with changes in the site theme and current post.

I was quite pleased at how easy and straightforward Utterances can be integrated to provide comments and discussions into static sites. I hope to hear from people who's come by to read what I've written and give their thoughts too.

Thank you to the folks behind [utterances](https://github.com/utterance) for making the comments widget that made sense.



It's been many years since I last blogged anything, especially on programming, software engineering and tech.

Watching [Yang Shun](https://github.com/yangshun)'s GitHub activities, I couldn't help but notice that he's been dabbling with _Next.js_ - a name that I came across occasionally but never really looked into seriously. Since I had some time on hand this week, I took the opportunity to read up and understand what Next.js was trying to achieve.

# Solving Common React Pain Points

As I dug, it turned out, Next.js was already solving common problems that I often encounter when building React apps:

- Deciding how files, pages, components are structured in the project folder
- Deciding how routing is done (despite everyone's using `react-router`, there's variance in flavour and how routers are structured)
- Deciding how code is transpiled (e.g. configuring Webpack, TypeScript etc.)
- Figuring how to do Server-Side Rendering (SSR) right
- Figuring how to get asset optimization done right

Next.js, as a framework, has structured and simplified the development workflow to get web developers straight into building the app.

# Learning Next.js

For me, the [tutorial](https://nextjs.org/learn/foundations/from-react-to-nextjs) was easy to follow as I had some background in React. If you only have experience in vanilla JavaScript, the [tutorial comes with steps](https://nextjs.org/learn/foundations/from-javascript-to-react) to help you get up to speed.

Once I was through with the tutorial, I figured that I'll try out Next.js by creating a blog that's similar to the tutorial, yet incorporating some of the niceties I'm familiar with, like:

- [Yarn Monorepo](https://semaphoreci.com/blog/javascript-monorepos-yarn-workspaces) - breaking down apps into reusable and composable packages sparks joy. In my blog's monorepo, I have two types of workspaces: `apps` and `libs`
- [TypeScript](https://nextjs.org/learn/excel/typescript) - I find typings help with santiy checks and sometimes make code easier to read
- [Tailwind CSS](https://tailwindcss.com/docs/guides/nextjs) - I could have gone with [MUI](https://mui.com/), but at this point I find it simpler to just have utility CSS classes rather than importing a full blown component library. I also like how Tailwind CSS makes it easy to incorporate dark/light theming
- [Sass](https://nextjs.org/docs/basic-features/built-in-css-support#sass-support) - expressive CSS rules, which goes hand in hand with CSS Modules in Next.js

# Additional Resources

After spending about 3 days on it, I was able to piece together [the blog](https://github.com/mauris/mauris.github.io) you see here, with much thanks to the following resources:

- The full tutorial to create a Next.js app: <https://nextjs.org/learn/basics/create-nextjs-app>
- An example of how to setup a monorepo with Next.js: <https://github.com/belgattitude/nextjs-monorepo-example>
- [Julia](https://bionicjulia.com/) gave some guidelines on setup (picked up `react-markdown` and `react-syntax-highlighter` from here, without having to explicitly write code to parse Markdown using `remark`): <https://bionicjulia.com/blog/setting-up-nextjs-markdown-blog-with-typescript>
- Guide on using GitHub Pages to host the exported Next.js app as a static webapp: <https://wallis.dev/blog/deploying-a-next-js-app-to-github-pages>
- Using `next-transpile-modules` to ensure workspace dependency packages/libraries get transpiled properly: <https://www.npmjs.com/package/next-transpile-modules>
- Using GitHub Actions to magically update GitHub Pages `gh-pages` branch whenever there are changes made on the `main` branch: <https://gregrickaby.blog/article/nextjs-github-pages>

# Conclusion

I must say it was quite a joy to work with Next.js and it took away some cognitive load I mentioned earlier. With that, I might continue working on more features for this blog (e.g. pagination, search) in my free time. Hopefully this will encourage me to write and reflect more upon my software engineering career and journey.


Sam's Blog

Architecting Asynchronous State Synchronization between Backend Services

3 reminders when starting a Software Engineering journey

Backend Services: Power-on Self-test

React-Query: abstracting query options

Giscus: Blog Comments with GitHub Discussions

Utterances: Blog Comments with GitHub Issues

Next.js + TypeScript in a Monorepo