jarv.org

My RSS client has an MCP server

Tue, 05 May 2026 00:00:00 +0000

Although the days of every site having an RSS feed have long past, I still find that it remains one of the few ways left to curate your own garden of content on the internet. There’s no algorithm deciding what you see or optimized feed pushing rage bait to the top (like the terrible trend of short-form video content).

In a previous post, I mentioned how I am using RSS to monitor individual files or directories in GitHub and GitLab repositories, like tracking changes to a CHANGELOG, a package.json. My RSS client is configured with tech blogs belonging to people I know, some interesting ones I find randomly, YouTube channel feeds, GitLab activity feeds, and some traditional news sources.

Because for me, all of this content sits in a local database, already fetched and cached it makes it a perfect candidate for an MCP server.

Why MCP + RSS

MCP (Model Context Protocol) lets AI assistants call tools to retrieve information. An RSS reader that exposes its cached articles via MCP essentially gives an AI assistant access to your curated view of the world.

Some examples of what this enables and how I am using it:

“Summarize all GitLab activity from the last week”
“Were there any changes to the Go release notes this month?”
“What’s been happening in my news feeds today?”

The responses end up being grounded in content that I’ve already chosen to follow instead of a global web search.

Implementation

I don’t see terminal based readers implementing this yet, but I assume it is going to catch on given how useful it is. If you know of ones that do have this already please let me know! In the meantime, I added this to my customized news reader newsgoat. This reader of is highly tailored to myself, and might not be for everyone. With that said, if you have used or are using newsbeuter/newsboat you should give it a test drive!

The MCP implementation exposes four tools:

search_articles — full-text search across titles, descriptions, and content with a time window
list_recent_articles — list recent articles, filterable by folder or feed name
get_article — get the full content of a specific article
list_feeds — list all feeds with their folders and unread counts

The configuration for an MCP client:

{
 "newsgoat": {
 "command": ["newsgoat", "mcp-server"]
 }
}

The MCP server opens the same SQLite database as the TUI, runs read-only queries, and returns results as text.

So I think every RSS client should do this and hope it catches on. If you are not using RSS or maybe if you used to use RSS but gave up on it, I would encourage you to give it another look!

Filtering YouTube Shorts from RSS Feeds

Mon, 27 Apr 2026 00:00:00 +0000

I subscribe to a lot of YouTube channels via RSS in NewsGoat, and the one thing that I can’t stand are Shorts. Every channel seems to produce them now, and they clutter up the feed with content I never want to watch.

To help, I just added regex-based filters to NewsGoat to deal with this.

Setting Up a Filter

In newsgoat, press f on a feed or folder to open a filter in $EDITOR. The filter is a simple YAML file with regex patterns for URL, Title, and Description fields:

# Filter for folder: YouTube
# Regexes are ANDed together. Prefix with ! to negate.
# Examples:
# - "golang" # items must match "golang"
# - "!/shorts/" # items must NOT match "/shorts/"
#
URL:
 - "!/shorts/"
Title:
Description:

The ! prefix negates the pattern — items matching !/shorts/ are excluded. Setting this on a YouTube folder applies it to every feed inside that folder, so I only had to configure it once.

How It Works

Filters can be scoped at three levels: global (F), per-folder (f on a folder), and per-feed (f on a feed). They compose with AND logic, for example if you have a global filter and a folder filter, both must pass for an item to show.

The regex patterns within each field are also ANDed together, so you can stack multiple conditions:

URL:
 - "!/shorts/"
 - "!/playlist"
Title:
 - "!LIVE"

This would exclude Shorts, playlists, and anything with “LIVE” in the title.

Feeds and folders with a configured filter show a ✦ indicator in the feed list, so it’s easy to see where filters are active. Items that don’t match the filter are hidden from the item list — they’re still in the database, just not displayed.

Neovim Configuration for 2026

Fri, 13 Feb 2026 00:00:00 +0000

A year after my last Neovim configuration post, I’ve made another significant change to how I manage my setup. This time, I switched from kickstart-modular.nvim to MiniMax, a minimal configuration built on top of the mini.nvim ecosystem.

Why I Switched

While I was happy with kickstart-modular.nvim, the upstream project (kickstart.nvim) appears to be no longer actively maintained. There is an effort to maintain a fork, but I decided to explore alternatives rather than rely on a community fork.

MiniMax embraces the mini.nvim philosophy of having small, focused, well-integrated plugins. It’s actively maintained by the author of mini.nvim itself, and provides a solid foundation that’s both minimal and fully-featured.

My Configuration and Workflow

I maintain my own fork of MiniMax.
I apply my customizations in a single commit.
I periodically pull in upstream changes, staying up-to-date with improvements while keeping my modifications intact.

This workflow continues to work well for me, it’s the same approach I used with kickstart-modular.nvim.

What Changed in the Migration

The transition from kickstart-modular.nvim to MiniMax was surprisingly smooth because both configurations share similar philosophies. Here are the key differences I encountered:

Feature	Before (kickstart-modular)	After (MiniMax)
Plugin Management	`lazy.nvim`	`mini.deps` (simpler, more integrated)
File Explorer	`telescope.nvim` for file browsing	`mini.files` (lightweight, tree-style interface)
Auto-completion	`nvim-cmp` with multiple completion sources	`mini.completion` (simpler, built-in)
Configuration Structure	Multiple Lua files in `lua/custom/` directory	Single customization file at `configs/nvim-0.11/plugin/50_custom_plugins.lua`

Customizations

I continue to keep my customizations minimal. MiniMax already includes most of what I need through its carefully selected mini.nvim modules and a few additional plugins like LSP support, Tree-sitter, and formatters.

Here are the plugins and customizations I added on top of MiniMax:

Plugin/Feature	Description
mason.nvim	LSP server package manager. Makes it easy to install and manage language servers.
clipboard-image.nvim	Paste images directly from clipboard into Markdown posts—essential for my blog workflow.
opencode.nvim	AI-powered code assistant integrated with my terminal. Mapped to `<Leader>c` prefix.
Spell checking	Custom commands (`:Spell`, `:NoSpell`, `:SpellToggle`) with auto-enable for Markdown files.
Custom formatters	Added formatters for Caddy, Markdown, JSON, and shell scripts to `conform.nvim`.

Additional Customizations

Beyond plugins, I made several configuration tweaks:

Buffer Navigation: Mapped <Tab> and <Shift-Tab> to cycle through buffers.
Mouse Behavior: Changed from mouse='a' to mouse='nci' to disable visual selection mode while keeping other mouse features.
Diagnostic Display: Enhanced inline diagnostics with custom icons (✘ for errors, ▲ for warnings) and a <Leader>od toggle.
Relative Line Numbers: Enabled with vim.o.relativenumber = true (commented out in base config).
Shell Integration: Set default shell to /usr/bin/zsh for :term commands.
Hidden Characters: Commented out vim.o.list = true to hide tab characters.

Plugins I No Longer Need

Some plugins from my kickstart-modular.nvim setup became unnecessary with MiniMax:

bufferline.nvim: MiniMax’s mini.tabline provides similar functionality
karen-yank.nvim: I have a love/hate relationship with this plugin. For now I’m going to go back to the default cut/paste behavior.
rose-pine.nvim: Using the default MiniMax colorscheme (though I may add this back)
vim-fugitive: mini.git covers most of my Git workflow needs

Thoughts on the Switch

I’m impressed with how cohesive MiniMax feels. The mini.nvim ecosystem provides a surprisingly complete feature set with minimal overhead. The configuration is easier to understand because most functionality comes from a single, well-documented plugin family.

The main trade-off is that some mini.nvim modules are less feature-rich than their standalone counterparts (e.g., mini.completion vs nvim-cmp). However, I’ve found the simpler implementations to be sufficient for my needs, and the performance benefits are noticeable.

If you’re looking for a minimal, maintainable Neovim configuration in 2026, I highly recommend giving MiniMax a try. You can see my complete configuration with customizations here.

Monitor a single file on GitHub/GitLab with RSS

Mon, 20 Oct 2025 00:00:00 +0000

I’ve been using RSS to monitor single files or directories on GitHub and GitLab for a long time, and it still surprises me when people don’t know this is possible. Granted, not everyone uses RSS so that part isn’t surprising, but it is quite a convenient way to keep track of files in repositories without having to manage email notifications for the entire project.

Why would you do this?

For one, I find it nice to track updates to README.md files in a repository, a CHANGELOG, or a file that tracks dependencies like package.json or Gemfile.*

Both GitHub and GitLab give you the ability to do this easily, and it works on both individual files and directories containing them.

For GitHub adding .atom to the end of the file will give you the atom feed (example).
For GitLab you can create a feed by adding ?format=atom to the end of the file (example).

This works for both public and private repositories, for private repositories it requires a ?feed_token=xxxxx query parameter. However, you shouldn’t expose your feed token to a third party website, so if it is necessary I would only use a local feed application.

* It is possible to do this with a site like github-file-watcher.com if you really want to do this with email notifications.

The wild vibe

Wed, 08 Oct 2025 00:00:00 +0000

It’s wild that these days you can vibe code an entire app in the same time it would previously take to contribute a single feature. Take the RSS reader I use regularly, Newsboat. It always had a few stability issues and was only about 80% there as far as the features I wanted in a terminal-based reader. There were other alternatives that came along, but nothing matched what I was used to layout-wise. While there were a few things I would have liked to improve, I never had the will or time to contribute any improvements myself.

That all changed recently with Claude, as it suddenly became very easy to bootstrap a new project and create a very polished-looking TUI application in Go with barely any effort. In about a single day, I created a TUI RSS reader alternative NewsGoat and couldn’t be happier with the results. Not only was it extremely satisfying to have full control over every single feature that I wanted in the TUI, but it was actually less effort and hassle than contributing new features to the existing project.

Consider the two options:

Create a fork of an open source project, refactor and remove the features you don’t want, add the new features you do.
Create a brand new project from scratch and add only the features you want.

Now what is wild is that with vibe coding, (2) is not only possible to do but is often more straightforward and easier than (1). I can also choose the language I want to use, the build process, toolchain, distribution, and the boring parts of bootstrapping the project, creating build scripts, and workflows are all done by AI without even opening a text editor.

Here has been my experience with this and other projects to efficiently create new projects from scratch:

Pick an implementation that I am already familiar with. For me, this is usually Go and/or JavaScript.
Create a project README that outlines exactly the features you want and be as specific as possible on implementation constraints. For example, for Go projects that require state, I will be specific about the database driver and specify to use sqlc.
Create the barebones scaffolding of the project in addition to the README. This includes bringing in all the common dependencies for lint checking and mise for task management.

I’ve been doing this so often recently that I even created a vibe-kickstart template for bootstrapping new (web-based) projects that use Go and JavaScript.

Using Mise and 1Password for secrets

Mon, 11 Aug 2025 00:00:00 +0000

2025-08-17 UPDATE: I’ve modified this post to use hooks.enter and hooks.leave after seeing how my previous solution was not working in all cases.

Since my last post on using mise I’ve updated how I use Mise with 1Password.

Before, I was only using hooks.enter to set env variables from 1password. This had some disadvantages:

Variables would be left set when leaving the project directory
It made assumptions that you would cd into the root of the project directory first

This modification addresses those two issues by sourcing a script that unsets variables when leave the project directory. This still has a major shortcoming which is that it doesn’t restore variables that have the same name when leaving, so that is something to be mindful of.

Here is the approach using hooks.{enter,leave} in mise.toml:

[hooks.enter]
shell = "bash"
script = """
source "{{ config_root }}/.1pass.sh"
"""

[hooks.leave]
shell = "bash"
script = """
source "{{ config_root }}/.1pass.sh" unset
"""

Add secrets by adding to the ENV_MAPPINGS array.

1pass.sh:

ENV_MAPPINGS=(
 "GOOGLE_KEY:op://personal/direnv/GOOGLE_KEY"
 "GOOGLE_SECRET:op://personal/direnv/GOOGLE_SECRET"
 # etc.
)

unset_env_vars() {
 echo '⏳Unsetting environment variables ..' >&2
 for mapping in "${ENV_MAPPINGS[@]}"; do
 env_var="${mapping%%:*}"
 unset "$env_var"
 done
 echo '☑️ Environment variables unset!' >&2
}

set_env_vars() {
 if ! command -v op >/dev/null 2>&1; then
 echo "error: 1Password CLI (op) not found in PATH" >&2
 return 1
 fi

 echo '⏳Setting secrets in environment ..' >&2

 # Build the template for op inject
 template=""
 for mapping in "${ENV_MAPPINGS[@]}"; do
 env_var="${mapping%%:*}"
 op_path="${mapping#*:}"
 template="${template}export ${env_var}={{ ${op_path} }}
"
 done
 if eval "$(echo "$template" | op --account my.1password.com inject)"; then
 echo '☑️ Done!' >&2
 else
 echo "error: failed to inject/source secrets" >&2
 return 1
 fi
}

if [ "$1" = "unset" ]; then
 unset_env_vars
else
 set_env_vars
fi

See also this GitHub discussion for others who are trying similar things with mise to use 1password to populate the env.

Thinking about Trade Wars trading mechanics

Tue, 05 Aug 2025 00:00:00 +0000

It’s been a while (3 months!) since my previous post on this little game I’ve been writing in my free time, so I thought it was probably a good time to give an update.

The game is called STAR DEWAR which is a space trading game that is inspired by the early BBS game Trade Wars. Three months ago, I had created the basic hexagon map generation, multiplayer, and movement UI, but that was about it. This update includes a few more features:

Trading at ports!
Updated UI
Navigation computer messages

tldr; you can play a (pre-alpha) demo here that illustrates the new trading mechanics.

There isn’t much to discuss regarding (2) or (3), you can see these changes yourself by playing the demo. The trading mechanics seemed interesting enough to warrant a post explaining how they’re implemented.

In the game, you have three commodities, Fuel Ore, Organics and Equipment. I found myself going back to the original game (yes, there are still a small number of people who play) to remind myself how trading worked, how negotiations functioned, and how prices were set.

In general, I wanted to implement something that captures the original feeling of the BBS game and trading experience while giving it a modern interface.

To start, every port is designated with B or S for each commodity and shown on the mini-map like this:

In the original game, when you traded at a port, a commodity table would be presented like this:

 Docking...
 One turn deducted, 14 turns left.

 Commerce Report for Tarsus: 08/19/05 02:01:48 PM

 -=-=- Docking Log -=-=-
 USS Enterprise NCC-1701-D docked 1 day(s) ago.

 Items Status Trading % of max OnBoard
 ----- ------ ------- -------- -------
 Fuel Ore Buying 23 4% 0
 Organics Buying 193 17% 0
 Equipment Selling 741 42% 0

The way I understand it worked in the original game, there was a finite limit of resources for each type, and they would slowly get replenished on a daily interval. There was also a negotiation step where you could offer more or fewer credits than the original offer.

The counter-offer’s success was based on various factors including the port’s characteristics, your experience, and probably other things I don’t remember.

The trading screen in STAR DEWAR has a similar look with some notable differences:

There’s no text input for trades, so negotiations are fixed at “full price,” “10%,” or “20%” more or less than the offer. Accepting or rejecting a trade is where things get interesting, there’s a deterministic score given to every trade that sets bounds for the minimum and maximum accepted values. This determination uses the following factors:

How experienced the player is (experience points)
The port’s willingness to negotiate (set per port)
How much of the commodity is present at the port
How reasonable the price is

In STAR DEWAR, each of these factors is assigned a weight, which results in a minimum and maximum trade value for the commodity. Like the original game, a successful negotiation will give you some experience. Conversely, a trade rejection will result in losing experience. As the port reaches its maximum capacity, it will be less inclined to negotiate. If the port is rejecting many trades, it will reduce its willingness to haggle. Lastly, successful trades will self-reinforce future successful trades as players increase their experience.

Counter-offers are where some randomness comes into play. If an offer comes in that’s outside the port’s acceptable range, there’s a 50% chance of a counter-offer, which will be a random value within the acceptable range. Upon receiving the offer, the player can either accept it for the full amount or take an additional chance and try to get a better deal.

I’ve been thinking a little bit about how to make trading less of a grind and more interesting, which was a problem that was definitely present in the original game. Maybe automating trade routes that consume unused turns?

You can check out how trading works by playing the pre-alpha demo, which is a universe loaded up with a large number of ports to illustrate how it works. If you have any feedback or suggestions, please don’t hesitate to reach out or submit your email for these infrequent updates.

Accounts and Centralized Survey View for FlyEmoji

Sat, 12 Jul 2025 00:00:00 +0000

A quick update about some new features recently added to FlyEmoji. These are a couple of changes that should make the site easier to use, especially for anyone running multiple surveys or wanting to track what they’ve participated in.

User accounts and login

With the new account system you can sign up with an email and password (or log in if you already have an account). Once logged in, any survey you create will be linked to your account automatically.

This also lays the groundwork for better access control in the future (like making surveys private, sharing them with specific users, etc.).

Central survey dashboard

Another piece that’s now live is a consolidated view of your surveys. Once logged in, you’ll land on a new dashboard page that lists:

All surveys you’ve created
Surveys you’re participating in (for example if someone invited you to respond)

If you want to try it out, head over to https://flyemoji.com and register for an account. If you have any feedback please let me know!

A Trade Wars 2002 inspired pocket game

Mon, 26 May 2025 00:00:00 +0000

As I mentioned in my last post in my free-time I’ve been tinkering around with multi-player space themed game. The game is called STAR DEWAR, and will be a tribute to an old BBS game called Trade Wars 2002 that I played as a kid.

Trade Wars was a classic multi-player BBS game that had its peak popularity in the 1990s. It was one of the insiprations for Eve Online (early 2000s) and plenty of other games that focus on space exploration and trading.

I’ve always thought about extracting some of the elements I loved the most in the game and turning it into a “pocket game”, or something that is approachable by the more casual gamer. This is the first (hopefully not only) journal entry, to help keep myself motivated and to chart progress as the game develops.

The first version is quite minimal; it includes map generation, player tracking, ship navigation and support for multiple simultaneous clients. One of the main goals was to write something that works well on mobile devices which is a challenge when your main inspiration is game played on a text based interface.

If any of this sounds interesting, please read on, or if you would like to see what is working so far check out https://stardewar.com/v01.

When the game server starts, a random hex based sector map is generated.
When you connect your browser, I assign you a player name (stored in a session cookie) and place your ship on the map.
All other players are automatically get “limpet mines” that allow you to see where they are.
You can move around the map and see other player movement in real time.

That’s it for now! Notthing terribly exciting but getting this far did take a fair bit of time thinking about design and writing code.

The universe

The classic TradeWars 2002 BBS game had a map divided up into sectors. Each sector might contain a planet, a trading port, as well as other ships, mines, fighters, etc. The typical TW2002 map had around 1000 sectors that were represented as a directed graph. Adjacent sectors were both bidirectional or one-way.

My first obstacle was to think about how to replicate the experience of exploration, finding optimal trade routes, and looking for places that aren’t easily discovered by other players. For this inspired version, I went with a tiled map with hexigon tiles. For now, all movement is bidirectional but that could change if I can manage to come up with a nice way to show that visually.

It is quite different than the original, where there wasn’t a 2D representation of space. What I think I would like to add later is the ability to connect distant sectors using wormhole paths.

The universe fully zoomed out with all sectors discovered

After five minutes, a “Big Bang” button will appear allowing anyone to restart the game server and start from scratch.

Development

I’ve never written code for a game like this and definitely never have written a 2D game UI for the browser. Here were some of the more challenging parts:

Sequencing movement on a 2d board for multiple players
Zoom, drag, and select with touch events
Centrally managing multiplayer real-time map updates
Tracking per player visited sectors and fog-of-war
Dealing with a hexagon tile map on both the front-end and back-end

Front-end

For developing the front-end, I decided to use phaser.io. I probably could have gotten away without using a game engine at all, but Phaser has been a great tool so for managing sprites and animating them.

Funny one of the more difficult parts of the front-end (something I take for granted), was coding a natural way to distinguishing a drag, select and a pinch zoom. Modern browsers allow you use multiple pointers for pinch detection. That, along with manual event tracking of how long the pointer is down made it possible to handle all three on a 2D map.

The other part was queuing movement and animations. Because the backend is authoritative for player state, all movement needs to be sent over a websocket. The “shortest path” calculation is done on the backend as well. When you click to a cell that is on a different spot in the map (even if you haven’t explored tiles in-between), the backend generates the calculated route. If multiple movements commands are issued before the movement animation completes they need to be queued up. The same applies to other player movements that are broadcast to players that have tracking mines attached to the ship.

Movement updates from the server provide only the new movement path (so we don’t redraw the entire map on every move), however, any time there is a reconnection is lost or if the browser loses focus the map is redrawn from scratch.

Back-end

On the back-end I decided to use Go as I’m already familiar with writing Go for non-game software. Most of the work was coming up with a schema and queries and then generating code with sqlc. All player state is tracked in sqlite

The map generation is done on the back-end and for more details see my earlier post on my variation of Drunkard’s Walk. After playing around with it a bit I went with spherical map that sets a boundary using a radius from the center.

For package dependencies I used the following:

github.com/coder/websocket
github.com/dustinkirkland/golang-petname (for generating player names)
github.com/gorilla/sessions v1.4.0 (for tracking players in cookie based sessions)
github.com/ncruces/go-sqlite3

Using embed, all assets are compiled into the binary making deployments a breeze, especially with a in-memory db to start.

What next?

The most challenging part of modern adaptations of games like Trade Wars, is coming up with the mechanics around trading. This was a central part of the original game and I would like to keep it that way with this “pocket size version”. Doing that, but also not making it too much of a grinding game going back-and-forth between ports is a challenge. Right now, I’m thinking that I want to keep the trading routes concept of Fuel Ore, Organics and Equipment but also have a way to set up trading routes to automatically use-up unused turns to trade.

For the next addition, what I have in mind is to create artwork and framework for placing ports. Who knows how long that will take or whether I can keep up, but my love and nostalgia for this game is keeping me motivated.

If you would like to receive updates when I make future posts like these, submit your email on https://stardewar.com.

A Drunkard's Walk with hiding spots

Wed, 26 Mar 2025 00:00:00 +0000

I’ve been playing around lately with procedural map generation and different ways to generate contiguous areas on a hexagonal grid. To keep things a simple as possible I started with a Random Walk (also known as “Drunkard’s Walk”) but found it a bit lacking. Part of my map generation needs to include dead-ends (or hiding spots) that will be difficult to discover when traversing distance from one hex to another hex.

A simple “Drunkard’s Walk” is to start at one cell and pick a random neighbor that hasn’t been visited yet. If there are no neighbors that haven’t already been visited, take all remaining visited cells that have unvisited neighbors and pick one at random. This can be repeated until a percentage of cells have been picked and will result in a contiguous area that can be used for a map.

Here is how it looks, taking a random contiguous path for 70% of the cells on the grid:

Note that in the above hex map, every cell has at least 2 entry points. What I would like to change is an increased likelyhood of “hiding spots” or cells with only one entry point so that when traversed, players in these dead-end cells are less likely to be discovered.

Here is the same walk, filling 70% of the cells but with more dead-ends:

This is very similar to the other logic, except with a set probability, if the current cell has 5 neighbors that haven’t been visited, it is added to a list of dead-ends. Then, instead of picking one of those neighbors to visit next, we pick a random hex in the list of visited cells. If there are no cells that meet that criteria, then we start at a random cell in the dead-end list.

As the map density is increased, there are fewer dead-ends as they will be re-purposed to fill the map, resulting in mostly “horseshoe” hexes. Below you can play with the cell density and the probability of marking a dead-end. For my own map, a density of 60% and a dead-end probability of 40% is about right.

Dead end count: 0

Density: 0.5

Dead end probability: 0

If you read this far, I’m hoping to write a few posts on writing a multi-player 2D game and hopefully this is the first post of many (we will see). The game is called Star Dewar which is an anagram for an old BBS game I liked as a kid.

Neovim Configuration for 2025

Sun, 23 Feb 2025 00:00:00 +0000

Here are some notes on how my Neovim configuration management evolved in the last year. For longtime Vim → Neovim users like me who prefer a simple and sane way to manage their configuration, I highly recommend kickstart.nvim as a starting point. It provides a minimal yet functional configuration that is easy to understand, making it ideal for those who don’t want to commit to a pre-built template without knowing what it does.

My Configuration and Workflow

I maintain my own fork of kickstart-modular.nvim. This is essentially kickstart.nvim, but structured into directories rather than a single init.lua, making it easier to manage.
I apply my customizations in a single commit.
I periodically pull in upstream changes, allowing me to stay up-to-date with core plugins while maintaining my modifications.

Customizations

I try to keep my customizations on top of kickstart.nvim minimal and avoid having too many plugins in general. Even though I customize it a bit, the base plugins might be all you need as it already includes the essentials like plugin management, LSP, and general quality-of-life improvements for writing code.

Here are seven plugins that I add on top of ones included in kickstart.nvim:

Plugin	Description
bufferline.nvim	Displays open buffers as tabs at the top of the session. I find this helpful for keeping track of my open buffers but there are a lot of different methods to do that.
clipboard-image.nvim	Useful for quickly inserting images into Markdown blog posts via copy/paste.
cmp-spell	Adds spell-checking suggestions to `nvim-cmp` auto-completion.
karen-yank	Prevents deletions (`D`, `dd`) from copying content into the paste register, making yanking more explicit.
key-analyzer.nvim	Provides a quick overview of key mappings.
rose-pine.nvim	My current colorscheme.
vim-fugitive	Git integration for Neovim.

In addition to these plugins, I have several customizations to the base config that enhance my workflow:

Backups: I enable backups using set backup, which has saved me a few times.
Buffer Navigation: I map <Tab> and <Shift-Tab> to switch between buffers, which pairs well with bufferline.nvim.
Auto-complete: I add hrsh7th/cmp-buffer to cmp.lua to enable auto-completion of words from open buffers.
Mini.nvim Enhancements: While kickstart.nvim includes some mini.nvim plugins, I also add mini.indentscope, mini.icons, and mini.files.
Telescope Optimization: I modify telescope.lua so that when inside a Git repository, searches start from the repository root.
Relative Line Numbers: I set vim.opt.relativenumber = true for relative line numbering.
Mouse Behavior: I disable visual selection with the mouse, as I find it more disruptive than helpful.

Not many changes to be honest, and I’ve been really happy with this setup so far. If you would like to see the configuration in its entirety, it can be viewed here.

Notes on Cookie partitioning

Sun, 05 Jan 2025 00:00:00 +0000

The Partitioned attribute on browser Cookies is supported on Firefox and Chrome and these are some notes on changes I observed recently with it, and how these browsers handle Cookies. This post assumes you have a basic understanding of how HTTP cookies work.

This feature targets analytics providers, specifically those that drop the same cookie in your browser when visiting multiple domains.

Let’s assume you are loading two webpages, one on siteA.com and another on siteB.com.
Both sites make requests to an analytics provider on analytics.com.

Without partitioning, when you visit siteA.com and get a cookie from analytics.com, and then later visit siteB.com, which also makes a request to analytics.com, analytics.com will receive the same cookie that was received from siteA.com. This is one of the primary reasons why browsers often block third-party cookies.

With partitioning, the browser maintains partitioned cookie stores so that analytics.com can’t access the same cookie for both domains.

How are Browsers Using the `Partitioned` Attribute?

Here, we can see that how the browser treats Secure and Paritioned is quite different depending on the browser.

This table shows what cookies are stored with different values for SameSite in the browser after a web request.

(These screenshots are taken from one of the SameSite scenarios on this cookie playground)

(S)	(P)	Chrome (131.0.6778.205)	Firefox (133.0.3)*
t	t
t	f
f	t
f	f

Browser Cookies: request sent to the same Domain
IFrame Cookies: request sent to a different Domain
🍪: Cookie received in the browser
🍪: Cookie not received in the browser
(S): Secure attribute
(P): Partitioned attribute

*Firefox is running in Strict mode

Some surprising differences seen here:

With Secure=true and Partitioned=true, if the SameSite attribute is not set, third party cookies will not be stored. This is due to Chrome treating the default Samesite value as Lax.
We see the descrepency for Secure=true and Partitioned=false for the same reason.
With Secure=false and Partitioned=true, Chrome does not store cookies, regardless of the SameSite value.
With Secure=false and Partitioned=false, Chrome does store cookies, but none from a different Domain.

Why is the `partitioned` Attribute Necessary?

What I found a bit confusing is why we need a partitioned attribute on cookies if this is a browser feature? The purpose of the attribute is to signal to the browser that the cookie works with browser partitioning. Without it, the browser might reject the cookie outright by default.

cookie has been rejected because it is foreign and does not have the “Partitioned“ attribute.

Before, I believe this was just a warning. I’ve updated the samesite testing sandbox with the option to enable and disable partitioned cookies to play around with this.

Using Mise for All the Things

Mon, 30 Dec 2024 00:00:00 +0000

I’ve been using Mise for some time now for both personal projects and work to manage versions of the tools used in my projects. Recently, Mise has added functionality for tasks and environment handling, leading me to consolidate tools like direnv, Make, and entr.

When I create a new project, I often set up the following:

Mise for versioning binaries like node, go, etc.
A bin/ directory with scripts for executing tasks using entr, and sometimes a Makefile for calling scripts.
direnv for setting up the environment.

With the recent addition of Mise hooks and tasks, all three of these can now be handled effectively with Mise itself.

Environment and Secrets

Note: See my updated post on using mise and 1Password.

In this example, I have an environment variable named CLOUDFLARE_EMAIL and a secret CLOUDFLARE_API_KEY.
Both are now set in my Mise configuration and loaded when I enter the project directory.

The .mise.toml:

[env]
CLOUDFLARE_EMAIL = "john@jarv.org"

[hooks.enter]
shell = "bash"
script = """
echo -e '⏳ Setting secrets in environment...'
source <(cat .1pass | op --account my.1password.com inject)
echo -e '☑️ Done!'
"""

Every project has a file .1pass that uses 1Password template syntax to inject secrets as environment variables:

export CLOUDFLARE_API_KEY={{ op://personal/direnv/CF_GLOBAL_API_KEY }}

Tasks and Watching Files on Changes

Now that the environment is sorted out, we can replace utility scripts and a Makefile for running them with Mise tasks.
These are also configured in the project’s .mise.toml file.
For example, for this blog I have:

[tasks.serve]
run = "hugo server -D -F"

With this, I can run mise run serve to start the hugo server.

Watching files and restarting a process on changes can be configured in .mise.toml using watchexec.

Here is a .mise.toml config for a task named run that builds assets and recompiles my Go app on changes:

[tasks.run]
depends = ["build-assets"]
sources = [
 "**/*.{tmpl,go}",
 "src/**/*",
 "package.json",
]
run = "go run . -dbFile /tmp/db.sqlite3"

mise watch -r run will start watching files and rebuild everything whenever one of the files in sources is modified.
The -r is passed to watchexec to force a process restart.

Or if that is too hard to remember, you can add it as a task:

[tasks.watch]
run = "mise watch -r run"

Then, all you need to do is type mise run watch.

Centering an emoji character on a button

Thu, 15 Aug 2024 00:00:00 +0000

One of my emoji-based side-projects has button elements and centering emoji characters on them is a key part of the design.

My initial thought was that this would be simple for a single emoji character by setting font-family, font-size and line-height for consistent centering. I mean, how complicated can it be, right?

Once I had things looking decent on chrome/firefox/safari it seemed as though my work was done, but then I pulled up the page on Linux/Firefox and my nice centering shown here:

Firefox/Chrome/Safari on OSX

looked something like this :(

Firefox on Linux

Around this time I read Hardest Problem in Computer Science: Centering Things and realized that maybe I shouldn’t spend too much time on it.

As an alternative, I thought to bundle a font like Noto Emoji but seeing how many MB it would add to page I decided not to go that route.

In the end I only needed around 30 emojis, so the simplest workaround was to take my emojis used for the layout and convert them to images. Following that all was well again and I was even able to let people select multiple emoji types (google, apple, facebook, openemoji).

If you want to see the new image based emoji layout in action check it out here.

Using Strict for the SameSite attribute

Mon, 27 May 2024 00:00:00 +0000

I was building a site that uses QR codes that when scanned take the user back to the same site. For example, if the site was https://example.com the QR code would bring them to https://example.com/qr like this:

The site uses a session cookie that is necessary to track user settings for anonymous users. Initially, I set the SameSite attribute to Strict on the cookie. What was not obvious to me then was how SameSite treats QR code navigation. For the browser, this is no different than a clicking link from a different site, even though you are scanning a QR code.

Here is a high level description of the different values for SameSite on a browser cookie:


Strict	Cookies are only sent in a first-party context or navigating directly to a site.
Lax	Cookies are sent in some cross-site contexts, but not with embedded content.
None	Cookies are sent in all contexts, including cross-origin requests
(not set)	If the SameSite attribute is not specified, the behavior is browser dependent.

Scanning a QR code does not count “navigating directly to a site”, instead it is a navigation event from a different site. The problem with using Strict is that after page load following the QR code scan, the session cookie was not sent to the server. This meant I was unable to load the user’s default settings on the first page load.

The solution was to set the SameSite attribute to Lax. This covers this QR code navigation scenario. After fixing the issue, I was curious about what other things don’t count as direct navigation in the context of SameSite. This led me create a sandbox for testing SameSite behavior. Check it out here.

Creating a ring buffer using Go Channels for Server-Sent Events

Fri, 05 Apr 2024 00:00:00 +0000

Server-Sent Events (SSE) are a simple way to send real-time updates from the server to the browser. Go channels pair nicely with SSE as they allow easy communication between Go routines. This post will examine using SSE in a recent side project and the implementation of a ring buffer in Go.

If you aren’t already familiar with SSE, like WebSockets, they enable updates on the web browser without having to poll or refresh the browser. However, unlike WebSockets, they don’t support bidirectional communication. This makes them more suitable for sending updates in one direction. The main advantage of using SSE is that it builds on top of HTTP (normal HTTP request with additional headers). This allows requests to pass through firewalls and proxies.

flyemoji.com sets up a reaction poll that users access with a QR code. The idea is that during a presentation or a video call, you can collect feedback from an audience. From the poll, users select an emoji reaction to a question. The reactions get tallied and sent to one or more connected clients listening for results.

It looks something like this:

How are you feeling today?

Press one of the reactions, see tallies in real-time.

For simplicity, we will assume a large number of clients are sending reactions, and there is a single client receiving the tallies.

The browser makes an SSE request to get real-time updates for the emoji tallies and blocks on a Go channel.
Emoji presses send a PUT request; the server takes that request and updates emoji tallies in an SQLite database and sends it to the channel.
The browser displays the tally result after receiving the tallies via the SSE request.

On the server, sending the tallies in real-time is the easy part since all it requires is a HTTP handler and headers set on the response:

mux.HandleFunc("GET /events", func(w http.ResponseWriter, r *http.Request) {
 w.Header().Set("Content-Type", "text/event-stream")
 w.Header().Set("Cache-Control", "no-cache")
 w.Header().Set("Connection", "keep-alive")

 // Setup a channel for receiving data
 // ...

 for tally := range ch {
 // Send the data
 fmt.Fprintf(w, "data: %s\n\n", tally)
 w.(http.Flusher).Flush()
 }
})

The trickier part is how to send the SSE data through a Go channel. By default a Go channel blocks on send until the other side receives data. Obviously for web clients, we cannot assume that the receiver will be continuously available for receiving tallies.

For this reason a buffered channel is necessary. It takes a buffer length parameter on initialization and when data is received the channel won’t block until it’s full. Because we only care about the most recent tally data, it’s fine to discard old tallies as newer tallies come in. This is essentially a ring buffer with a buffer length of one, that discards the oldest value when new data comes.

A channel based ring buffer in Go

Below we will discuss two implementations of a ring buffer that pass messages using channels for SSE notifications. Here is the first implementation that has a couple issues, can you spot them?

 1package main
 2
 3import "fmt"
 4
 5type RingBuffer struct {
 6 ch chan string
 7}
 8
 9func NewRingBuffer() *RingBuffer {
10 return &RingBuffer{
11 ch: make(chan string, 1),
12 }
13
14}
15
16func (r *RingBuffer) Send(item string) {
17 select {
18 case r.ch <- item:
19 default:
20 <-r.ch
21 r.ch <- item
22 }
23}
24
25func (r *RingBuffer) C() <-chan string {
26 return r.ch
27}
28
29func main() {
30 rb := NewRingBuffer()
31
32 rb.Send("some value")
33 rb.Send("some other value")
34
35 complete := false
36 for !complete {
37 select {
38 case v := <-rb.C():
39 fmt.Println(v)
40 default:
41 complete = true
42 }
43 }
44}

run on the go playground

When an item is sent, we call select { ... } on the channel.
If the channel is empty, we add the item
If the channel is full, we remove an item and add a new one

Two race conditions exist in this code, one is fairly obvious and other one might be so at first glance.

First issue: Concurrent calls to Send()

If we have a large number of clients, calling Send() concurrently then we may end up in a race condition causing the channel to block. In the above code if one sender adds an item to the channel on line 18 as another sender adds an item on line 21 the channel will block because it’s trying to add to a full channel.

Here are two possible options to resolve it:

Wrap Send() in a locking Mutex
Use a single Go routine that moves data in and out of the channel

Option (2) creates an input and output channel and moves data between them.

Here is an implementation using two channels:

 1package main
 2
 3import "fmt"
 4
 5type RingBuffer struct {
 6 inputChannel chan string
 7 outputChannel chan string
 8}
 9
10func NewRingBuffer() *RingBuffer {
11 return &RingBuffer{
12 inputChannel: make(chan string),
13 outputChannel: make(chan string, 1),
14 }
15}
16
17// https://tanzu.vmware.com/content/blog/a-channel-based-ring-buffer-in-go
18func (r *RingBuffer) Run() {
19 for v := range r.inputChannel {
20 select {
21 case r.outputChannel <- v:
22 default:
23 <-r.outputChannel
24 r.outputChannel <- v
25 }
26 }
27 close(r.outputChannel)
28}
29
30func (r *RingBuffer) C() <-chan string {
31 return r.outputChannel
32}
33
34func (r *RingBuffer) Send(item string) {
35 r.inputChannel <- item
36}
37
38func main() {
39 rb := NewRingBuffer()
40 go rb.Run()
41
42 rb.Send("some value")
43 rb.Send("some other value")
44
45 complete := false
46 for !complete {
47 select {
48 case v := <-rb.C():
49 fmt.Println(v)
50 default:
51 complete = true
52 }
53 }
54}

run on the go playground

Now the Send() function does not have the same issue. To setup the ring buffer we call go rb.Run() before sending data that in the background will move data from the input channel to the output channel.

Note that in this version you might see both values printed or just the first one printed instead. This is because we are reading from the output channel while a background Go routine moves the messages to it.

Second issue: Concurrent reads and calls to Send()

As I mentioned earlier, there is another race condition that is also present in this version. Not only do we have to account for clients sending concurrently, but also sending and receiving concurrently! In the above code, if we read all the data from the channel (line 48) while trying to Send simultaneously on line 23, then it will block forever since the channel will be empty.

Here is a modification to Run() that resolves it:

 1func (r *RingBuffer) Run() {
 2 for v := range r.inputChannel {
 3 select {
 4 case r.outputChannel <- v:
 5 default:
 6 select {
 7 case <-r.outputChannel:
 8 r.outputChannel <- v
 9 default:
10 // channel is empty
11 r.outputChannel <- v
12 }
13 }
14 }
15 close(r.outputChannel)
16}

This adds an additional select {...} to catch the scenario where the channel is empty immediately after we detect it’s full.

For the full implementations of both the good and bad ring buffers see https://github.com/jarv/ringbuffer/tree/master .

Benchmarks

After writing these impelementations, I was curious which one can move data faster. Not surprisignly, a background Go routine that moves data between channels is much slower than calling moving data in and out of a channel in the call to Send().

Here are benchmarks of both ring buffer implementations with a buffer length of 1:

$ go test -bench=. ./...
goos: darwin
goarch: arm64

## 1 channel where data is moved in the call to Send() with a locking Mutex
pkg: github.com/jarv/ringbuffer/good/singlechan
BenchmarkParallelSendReceive/BufSize:1-10 6284988 191.2 ns/op
BenchmarkSendReceive/BufSize:1-10 21763442 54.01 ns/op

## 2 channels with a Go routine moving data between them
pkg: github.com/jarv/ringbuffer/good/dualchan
BenchmarkParallelSendReceive/BufSize:1-10 4342477 288.8 ns/op
BenchmarkSendReceive/BufSize:1-10 5103054 229.2 ns/op

https://github.com/jarv/ringbuffer/tree/master/good

The two channel implementation is slower because when the buffer is full, sends will block until the Go routine moves the data to the output channel. It is exacerbated by a short buffer because it’s necessary for the background Go routine to move data between channels frequently. While performance improves when the buffer length increases, it’s generally slower than discarding data in the call to Send() even though it requires a locking Mutex.

Notice that in the first benchmark test, it slows down considerably when we add parallelism, which is likely due to contention because of the locking Mutex.

Conclusion

Figuring out how to use channels for SSE was a lot of fun! It gave me the opportunity to learn a bit more about how to troubleshoot and think better about concurrency in Go. One valuable takeaway for me is that it’s important to validate background Go routines in unit tests. This was essential to uncover bugs that were not immediately apparent.

If you like this post, check out my other post on debugging a deadlock in Go or subscribe to the RSS feed. If you would like a fun way to send out a reaction poll during your next video or conference presentation, check out flyemoji.com.

The joys of self-hosting and tiny side-projects

Fri, 01 Dec 2023 00:00:00 +0000

Tiny side-projects

Lately there hasn’t been much time in my life for recreational programming but I still find time to create tiny services and put them on the public internet. I’m sure this stretches the definition of “side-projects”, but one advantage of having a VM somewhere in the cloud and a registered domain is you can create your own little kingdom of subdomains. Below are some examples of tiny side-projects, some no more than webserver configuration that either serves a small practical need or written for fun.


🌐 ip.jarv.org	Responds with the IP address of the request.
🤯 *.resp.jarv.org	Generates an http response based on the subdomain. E.g., 200, 404, 418, etc.
🔓 nossl.jarv.org	Forces an http connection, sometimes useful for getting to the wifi login page, same as `neverssl.com` or `example.com`.
💤 sleep.jarv.org	Sleeps for an arbitrary number of seconds or milliseconds. E.g., 1s, 100ms.

I was thinking recently about doing little projects like this, and I think that not only does it make learning something new fun, but it is also nice to simply build something that is very small and self-contained. More than anything else it ends up being a forcing-function to learn new things that I wouldn’t learn otherwise. For example, when writing front-end code there is always a bit more I learn about css media queries since I don’t spend much time in the front-end and the landscape is constantly changing. Or another is that I just learned that there is no browser enforced CORS with websockets, and I also figured out how to properly respond to a pre-flight cors request.

Read more about some of these on the post about cool caddy config tricks.

Self-hosting

At one point I would have never considered hosting a simple static site like this blog on a VM but now that I host this static blog on a VM I would recommend it over hosting a static site on object storage or using service like GitHub Pages. For one, you have more control over access, headers, responses, caching, etc. Another reason to consider self-hosting is that it used to be a pain to deal with SSL certificates but this has gotten so much easier with webservers like Caddy that come with out-of-the-box support for LetsEncrypt.

As far as spending money, self-hosting a static site, running a webserver, and running services in Docker containers can be done a small VM for around 5eu/month (or less!). Once you have your own server you can self-host applications you write yourself or select from a large selection of open-source services. I run all of the following on the same host:

Prometheus and node exporter
Loki for logs
Grafana
GoatCounter

Self-hosting analytics (e.g., GoatCounter) means you know you will be seeing an accurate view of traffic since browser ad-blockers won’t block requests. Collecting metrics with Prometheus, dashboarding with Grafana , and log management with Loki gives a nice consolidated view into all the services running on the host. There is nothing is quite as as simple as running everything on a single VM with systemd managing monitoring, logging and application services with docker.

Debugging a deadlock in Go

Wed, 11 Oct 2023 00:00:00 +0000

Occasionally, cmdchallenge.com would get into a state where requests to list user-submitted solutions to challenges would hang. The first few times it happened, I restarted the service which was enough to get it working again. Only later did I try to figure out what is going on, and thought I would do a small write-up as it ended up being a very common problem with deadlocks in Go.

Background

When I rewrote this application in Go, rate limiting was added using a popular package called tollbooth.

The reasons for rate limiting were:

Every solution submission starts a new Docker container so I didn’t want that happening too frequently per IP.
Every solution request does a lookup on a sqlite3 DB so I wanted to limit that as well.

Adding rate limits for (2) was hardly necessary since it only does a simple sqlite lookup, but this was where I was seeing the issue and also the place where I was mostly likely to receive concurrent requests.

The first time I investigated this problem I created a test deployment and ran a bunch of concurrent requests. I didn’t see the issue and was a bit perplexed, the problem was that my testing environment had rate-limiting disabled!. A good lesson here is that when you are trying to reproduce a bug, keep your testing environment as similar as possible to where it is happening.

The bug itself was pretty easy to extract, here is a small program that reproduces it (also on GitHub):

package main

import (
 "fmt"
 "log"
 "net/http"
 _ "net/http/pprof"

 "github.com/didip/tollbooth/v7"
)

func testHandler(w http.ResponseWriter, r *http.Request) {
 fmt.Fprintf(w, "OK\n")
}

func main() {
 listenStr := fmt.Sprintf(":%d", 6060)

 lmt := tollbooth.NewLimiter(float64(1), nil) // 1 req/s
 lmt.SetOnLimitReached(func(w http.ResponseWriter, r *http.Request) {
 log.Printf("Rate limit reached StatusCode: %d\n", lmt.GetStatusCode())
 })

 http.Handle("/", tollbooth.LimitFuncHandler(lmt, testHandler))

 log.Printf("Server started %s\n", listenStr)
 log.Fatal(http.ListenAndServe(listenStr, nil))
}

If you would like to see the bug yourself:

git clone https://github.com/jarv/godeadlock
go run app.go

And then send concurrent requests using ab or bombardier

# ab https://httpd.apache.org/docs/2.4/programs/ab.html
ab -n 100 -c 50 'http://localhost:6060/'

# bombardier https://github.com/codesenberg/bombardier
bombardier -n 100 -c 50 -l http://localhost:6060

Once connections start timing out you will have a deadlock and nothing will get through. You can try to curl http://localhost:6060 to be sure. Once it is in this state, try to figure out what caused the deadlock yourself, or read on if you would like to learn how to find the cause.

View the stack traces of all Go routines

Once the deadlock occurs, the best way to figure out where your program is stuck is by viewing the stack trace of all Go routines. There are two options, one is using pprof which is configured in the example code, another is to issue a SIGQUIT (CTRL-/ in your terminal) which will dump the Go routines to STDERR.

The pprof method is the easiest since we have it setup in the code to reproduce the issue. Connect your browser to http://localhost:6060/debug/pprof/goroutine?debug=1 to see the traces for all Go routines. The ones to focus on are the ones that call sync. From these we can see that there are multiple go routines blocked on RLock() and Lock().

Lock() allows only one Go routine to read/write at a time, RLock() allows multiple Go routines to read, but not write.

The traces reveal that there are multiple Go routines are waiting for a Rlock(), and another Go routine is waiting for a Lock() (abbreviated output below):

197 @ 0x1027596c8 0x10276b6f8 0x10276b6d5 0x102788358 0x102954a3c 0x1029549e9 0x102956e34 0x102958afc 0x102958f80 0x1029049b8 0x102905ff4 0x102906ccc 0x102903a48 0x10278c8f4
# 0x102788357 sync.runtime_SemacquireRWMutexR+0x27 /Users/jarv/.local/share/rtx/installs/go/1.21.1/go/src/runtime/sema.go:82
# 0x102954a3b sync.(*RWMutex).RLock+0x7b /Users/jarv/.local/share/rtx/installs/go/1.21.1/go/src/sync/rwmutex.go:71
# 0x1029549e8 github.com/didip/tollbooth/v7/limiter.(*Limiter).GetMax+0x28 /Users/jarv/go/pkg/mod/github.com/didip/tollbooth/v7@v7.0.1/limiter/limiter.go:182
...
1 @ 0x1027596c8 0x10276b6f8 0x10276b6d5 0x102788358 0x102955238 0x1029551e5 0x1029590d0 0x10295549c 0x102958f9c 0x1029049b8 0x102905ff4 0x102906ccc 0x102903a48 0x10278c8f4
# 0x102788357 sync.runtime_SemacquireRWMutexR+0x27 /Users/jarv/.local/share/rtx/installs/go/1.21.1/go/src/runtime/sema.go:82
# 0x102955237 sync.(*RWMutex).RLock+0x77 /Users/jarv/.local/share/rtx/installs/go/1.21.1/go/src/sync/rwmutex.go:71
# 0x1029551e4 github.com/didip/tollbooth/v7/limiter.(*Limiter).GetStatusCode+0x24 /Users/jarv/go/pkg/mod/github.com/didip/tollbooth/v7@v7.0.1/limiter/limiter.go:247
# 0x1029590cf main.main.func1+0x1f /Users/jarv/src/jarv/godeadlock/app.go:21
# 0x10295549b github.com/didip/tollbooth/v7/limiter.(*Limiter).ExecOnLimitReached+0xdb /Users/jarv/go/pkg/mod/github.com/didip/tollbooth/v7@v7.0.1/limiter/limiter.go:268
...
1 @ 0x1027596c8 0x10276b6f8 0x10276b6d5 0x1027883b8 0x1027a47e8 0x102956ac4 0x102956cf8 0x102957724 0x102958b74 0x102958f80 0x1029049b8 0x102905ff4 0x102906ccc 0x102903a48 0x10278c8f4
# 0x1027883b7 sync.runtime_SemacquireRWMutex+0x27 /Users/jarv/.local/share/rtx/installs/go/1.21.1/go/src/runtime/sema.go:87
# 0x1027a47e7 sync.(*RWMutex).Lock+0xf7 /Users/jarv/.local/share/rtx/installs/go/1.21.1/go/src/sync/rwmutex.go:152
# 0x102956ac3 github.com/didip/tollbooth/v7/limiter.(*Limiter).limitReachedWithTokenBucketTTL+0x63 /Users/jarv/go/pkg/mod/github.com/didip/tollbooth/v7@v7.0.1/limiter/limiter.go:572
# 0x102956cf7 github.com/didip/tollbooth/v7/limiter.(*Limiter).LimitReached+0x57 /Users/jarv/go/pkg/mod/github.com/didip/tollbooth/v7@v7.0.1/limiter/limiter.go:599
...

limiter.go:182:

// GetMax is thread-safe way of getting maximum number of requests to limit per second.
func (l *Limiter) GetMax() float64 {
 l.RLock()
 defer l.RUnlock()
 return l.max
}

limiter.go:247

func (l *Limiter) GetStatusCode() int {
 l.RLock()
 defer l.RUnlock()
 return l.statusCode
}

limiter.go:572

func (l *Limiter) limitReachedWithTokenBucketTTL(key string, tokenBucketTTL time.Duration) bool {
 lmtMax := l.GetMax()
 lmtBurst := l.GetBurst()
 l.Lock()
 defer l.Unlock()

...

GetStatusCode() stands out here as it is a function we call in the example. Also, if we delete lmt.GetStatusCode() the deadlock is resolved, but why?

Fixing the deadlock

To figure this out, we follow the trace up to see how function we pass to SetOnLimitReached function is called:

// ExecOnLimitReached is thread-safe way of executing after-rejection function when limit is reached.
func (l *Limiter) ExecOnLimitReached(w http.ResponseWriter, r *http.Request) {
 l.RLock()
 defer l.RUnlock()

 fn := l.onLimitReached
 if fn != nil {
 fn(w, r)
 }
}

Right there is the issue, there is an RLock() acquired right before the function call. In that function we are calling GetStatusCode() which calls RLock() again. This is a very common source of deadlocks in Go and is even called out in the documentation:

If a goroutine holds a RWMutex for reading and another goroutine might call Lock, no goroutine should expect to be able to acquire a read lock until the initial read lock is released. In particular, this prohibits recursive read locking. This is to ensure that the lock eventually becomes available; a blocked Lock call excludes new readers from acquiring the lock.

This illustrates why recursive read locking is problematic, if a concurrent request calls lock between the two read locks, the second read lock and the write lock will wait forever for the first read lock to release.

This can be resolved by releasing the read lock immediately after assigning the function from l.onLimitReached, which is exactly what I did in my local fork of this package to resolve the issue and submitted an upstream issue.

 func (l *Limiter) ExecOnLimitReached(w http.ResponseWriter, r *http.Request) {
 l.RLock()
- defer l.RUnlock()
-
 fn := l.onLimitReached
+ l.RUnlock()

 if fn != nil {
 fn(w, r)
 }

After this there are no more deadlocks and we can send a large number of concurrent requests that are rate limited. I hope you found this explanation clear and maybe learned something new about deadlocks in Go. If you have any comments or questions please don’t hesitate to reach out!.

One huge shell script

Sun, 01 Oct 2023 00:00:00 +0000

There are a lot of DevOps tools for managing both configuration and infrastructure as code. While some can be extremely helpful for managing configuring for a large distributed service, not all are as helpful for deploying something simple in the cloud, or a personal project that runs on a single virtual machine.

IaC or a configuration thingy	$job	personal stuff
Kubernetes	✅	❌
Configuration management (Ansible, Chef, etc.)	✅	❌
CI pipelines for code	✅	❌
CI pipelines for infra	✅	❌
Code reviews	✅	❌
Changes without downtime	✅	❌
Docker registry	✅	❌
Prebuilt machine images	✅	❌
Cloud “appliances”, load balancers, hosted DB, etc.	✅	❌
CDN (CloudFlare/CloudFront)	✅	❌
Hosted Git	✅	✅
Terraform	✅	✅
One friggin huge shell script	❌	✅

Over time, my approach to using tooling for my personal projects has changed. From a high level, my own lessons have been:

Avoid complexity for troubleshooting, monitoring and logging by leaning too much on “severless” components.
For the same reason, putting everything on a single VM is “good enough” for most applications.
CDN is probably not necessary unless you are very worried about downtime, self-hosting static files is just as easy as hosted options.
Put all of the infrastructure configuration (scripts, IaC, etc) in a single repository.
Avoid cloud dependencies in general, make it work locally and then replicate that on a virtual machine as close as possible.
Use a minimal base image and avoid putting complexity or configuration in user-data scripts.

This resulted in what I have now, which is a single VM provisioned in Hetzner Cloud where everything is configured in a single shell script and most services run in Docker containers.

Before converting to a single VM, I would do something like this any time I had an experiment or idea to play around with:

Create a new AWS account.
Create a bucket for Terraform state
Terraform configuration for CloudFront, Lambda, EC2, Route53, etc.
In the user-data script configuration the image from scratch to support running docker and create systemd unit files.
In the CI pipeline copy a binary, or tag a new docker image for every push to master.
Deploy the container using a public or private registry, pull the new image on the instance with a systemd unit configuration to manage the service.

For this to work, A lot of complexity was baked into the user-data script associated with the instance. It also meant keeping track of multiple AWS accounts (usually using the free-tier resources) and then possibly paying money if an EC2 instance was required after the free-tier ended. Often, there would be a separate testing environment which was ephemeral, that spun up an identical instance and cloud configuration.

My current approach of a single VM eliminates the need for AWS and a lot of complexity that comes along with it. For the configuration management part of it, I was confronted with the following questions:

How do I deal with configuration drift if manual changes are made on the instance while it is running?
Will it be possible to rebuild a single VM from scratch easily, and without much downtime?
If there will be multiple containers running on the instance, does it make sense to any container orchestration?
How much should be automated in CI if I am the only person deploying changes?
Should I depend on a container registry?

My approach to managing the single VM deployment

The questions above lead me to the following approach for configuration management when deploying to a single VM:

Caddy is the main webserver. It has a lot of great features including automatic HTTPs and some other great features that I talked about in a recent post.
For packages, add them to the VM and simply update the packages in cloud-init as I go, there is no extra boot logic other than installing packages.
For other manual changes, maintain a single bash script that is idempotent, so all changes are made in a single script and the script can be re-run when it is updated.
Run almost everything in Docker and use Systemd for service management including stopping and starting docker containers.
Use (self-hosted) Prometheus, Grafana for monitoring.
Don’t lean too much (or at all) on CI pipelines, run deploys locally.
To deploy and update individual services that are built using Docker, dump and copy the image directly to the VM instead of pushing it to a public registry.

The big shell script

I keep one repository named config-mgmt that has the following content:

Terraform for provisioning an instance, and configuring DNS in CloudFlare
A directory named files/, this has all of my OS configuration files like Systemd unit files, Caddy config files, and a single big shell script called bootstrap.
A shell script named configure that runs rsync to sync everything under files/ to the single VM and then after the sync runs the script named bootstrap.

The bootstrap script along with all files in files/ are rsync’d to the VM first. Then bootstrap is run which does all the little shell maintenance stuff like ensuring services are enabled, installing and configuring things that run on the VM, reloads SystemD, etc. The script ends up being around 500 lines of bash, a dozen or so simple functions that are called in sequence at the end of the script.

If I have a new project to deploy to my single VM I simply create a new config_ function, add it to the script and in most cases create a small bin/deploy shell script in the project’s repository that dumps and imports the docker container to the host.

Using Docker and a single VM for side-projects

There is no Docker container orchestration in this setup, just a bunch of Systemd unit files for starting Docker containers with Caddy configurations to route traffic to them. I do however not run everything in Docker, on the VM I run the following services:

Prometheus for collecting node metrics and for scraping application metrics
Grafana for dashboarding
Self-hosted GoatCounter for web stats

Rebuilding my single VM from scratch

This is something that is important as you don’t want to create a “snowflake” that will be impossible to recreate. To make this work in Terraform I maintain a list of servers and an “active server”.

locals {
 servers = ["lisa", "bart"]
 active_server = "lisa"
}

I can add a new server to the list to create a new one, and then switch the active to the new one I created which points all of the service DNS entries to it.

Cool Caddy config tricks for your self-hosted domain

Wed, 23 Aug 2023 00:00:00 +0000

Very recently I switched from running a couple side-projects that were hosted on AWS EC2 to a single Hetzner Cloud VM. During that transition, this page (jarv.org) was moved to it from GitHub pages. Static generation using Hugo stayed the same, but in the process of switching VMs I checked out Caddy as a new webserver. After making the switch, this ended up being a great NGINX replacement for all the sites hosted on what is now a single VM. The ergonomics of the config language was particularly impressive so I wanted share how simple and flexible it can be!

HTTPs made easy

First to state the obvious, Caddy is incredible easy to configure when it comes to HTTPs. It supports automatic HTTPs out of the box with Let’s Encrypt or Zero SSL. With only a few lines of config you can setup a large number of sites without having to go through the certificate issuer dance, or use a CDN like CloudFlare or CloudFront to handle certificates for you.

For example, here is how simple the configuration is for this site:

jarv.org {
 @cache path /font/*
 header @cache Cache-Control max-age=604800

 handle {
 root * /var/opt/www/jarv.org
 file_server {
 precompressed gzip
 }
 }
}

This does the following:

Sets a cache-control header for /font/ requests with a long expiry
Tells Caddy that files under /var/opt/www/jarv.org serve the site, and to also to expect .gz files in the same directory so we can serve assets pre-compressed

This will automatically redirect HTTP requests to HTTPs, which is likely what you want most of the time. If you want to have a plain HTTP version and HTTPs version that is possible too, for that see the next example.

Creating a site that echos an IP address

There are some sites that I sometimes use to echo my public IP like curl ifconfig.io. With Caddy, it’s very simple to replicate this with a Caddy configuration, so I replicated it with curl ip.jarv.org.

Caddy has a simple respond directive, here is how ip.jarv.org echos your IP address:

Note: This uses the new heredoc syntax in version 2.7.0


(echoIP) {
 templates
 header Content-Type text/plain

 respond <<EOF
{{.RemoteIP}}

EOF 200

 encode zstd gzip
 import logging ip.jarv.org
}

ip.jarv.org {
 import echoIP
}
http://ip.jarv.org {
 import echoIP
}

Instead of only ip.jarv.org there are two servers, ip.jarv.org and http://ip.jarv.org for HTTP and HTTPs requests. This is so we don’t automatically redirect HTTP to HTTPs which is nice for using curl on the command line.
echoIP is a template, which includes the content in each block. The template is used for both HTTP and HTTPs to respond with the requester’s IP address.
Again, using the same logging snippet from above, a dedicated request log is created on disk

Creating a subdomain that echos a HTTP status code

Finally, I was thinking that it would be handy to have a way to echo back an HTTP status code, using curl <http status code>.resp.jarv.org. This requires a wildcard certificate, which is a bit more involved since it requires a DNS provider module to be compiled into Caddy (only required here for wildcard certs).

I run Caddy on a Debian server in Hetzner Cloud, with Cloudflare provisioning DNS. Caddy is installed using the Caddy debian package, and xcaddy to install the Cloudflare DNS provider module. The module is compiled using xcaddy and there is great documentation for how to properly incorporate your custom build using the Debian package.

Here is the configuration for responding with any specified HTTP status code:

(echoResp) {
 templates
 header Content-Type text/plain

 respond <<EOF
{http.request.host.labels.3}

EOF {http.request.host.labels.3}
 encode zstd gzip
 import logging resp.jarv.org
}

*.resp.jarv.org {
 import echoResp
 tls {
 dns cloudflare REDACTED
 }
}
http://*.resp.jarv.org {
 import echoResp
}

REDACTED is a CloudFlare API token with Zone Read and DNS Write permissions.
http.request.host.labels.3 returns whatever is used as the wildcard for *.resp.jarv.org.
In addition to returning the passed in status code, it will respond in plain text whatever status code was sent.

Now, if I have want to generate a 404 I can request 404.resp.jarv.org, or for a 500 500.resp.jarv.org or any other status code as a subdomain!

Update: Below is the more complicated configuration running jarv.org that responds with emojis, short description, and also catches strings that are not valid HTTP response codes. Here are some example responses:

$ curl 200.resp.jarv.org
200 😃 OK
$ curl 201.resp.jarv.org
201 🎉 Created
$ curl 499.resp.jarv.org
499 http status code
$ curl 418.resp.jarv.org
418 🍵 I'm a teapot
$ curl herpderp.resp.jarv.org
💥 herpderp doesn't look like a valid HTTP status code!

And this is the Caddy configuration, that utilizes the map directive to look up the short descriptions for status codes:

(echoResp) {
 templates
 header Content-Type "text/html; charset=utf-8"

 @valid header_regexp host Host ^([1-5]\d{2}|599)\..*

 map {http.request.host.labels.3} {status_desc} {
 100 "Continue"
 101 "Switching Protocols"
 102 "Processing"
 103 "Early Hints"
 200 "😃 OK"
 201 "🎉 Created"
 202 "Accepted"
 203 "Non-Authoritative Information"
 204 "🙅 No Content"
 205 "Reset Content"
 206 "Partial Content"
 207 "Multi-Status"
 208 "Already Reported"
 226 "IM Used"
 300 "Multiple Choices"
 301 "Moved Permanently"
 302 "Found"
 303 "See Other"
 304 "Not Modified"
 305 "Use Proxy"
 306 "Switch Proxy"
 307 "Temporary Redirect"
 308 "Permanent Redirect"
 400 "❌ Bad Request"
 401 "🔒 Unauthorized"
 402 "Payment Required"
 403 "🚫 Forbidden"
 404 "🕳️ Not Found"
 405 "Method Not Allowed"
 406 "Not Acceptable"
 407 "Proxy Authentication Required"
 408 "Request Timeout"
 409 "Conflict"
 410 "Gone"
 411 "Length Required"
 412 "Precondition Failed"
 413 "Payload Too Large"
 414 "URI Too Long"
 415 "Unsupported Media Type"
 416 "Range Not Satisfiable"
 417 "Expectation Failed"
 418 "🍵 I'm a teapot"
 421 "Misdirected Request"
 422 "Unprocessable Entity"
 423 "Locked"
 424 "Failed Dependency"
 425 "Too Early"
 426 "Upgrade Required"
 428 "Precondition Required"
 429 "Too Many Requests"
 431 "Request Header Fields Too Large"
 451 "Unavailable For Legal Reasons"
 500 "🤯 Internal Server Error"
 501 "Not Implemented"
 502 "Bad Gateway"
 503 "🚧 Service Unavailable"
 504 "Gateway Timeout"
 505 "HTTP Version Not Supported"
 506 "Variant Also Negotiates"
 507 "Insufficient Storage"
 508 "Loop Detected"
 510 "Not Extended"
 511 "Network Authentication Required"
 default "http status code"
 }

 handle @valid {
 respond <<EOF
{http.request.host.labels.3} {status_desc}

EOF {http.request.host.labels.3}
 }


 handle {
 respond <<EOF
💥 {http.request.host.labels.3} doesn't look like a valid HTTP status code!

EOF 400
 }
 import logging resp.jarv.org
}

*.resp.jarv.org {
 import echoResp
 tls {
 dns cloudflare REDACTED
 }
}

http://*.resp.jarv.org {
 import echoResp
}

Forcing plain HTTP

This is something similar to neverssl.com, or example.com, I think both of these are still used frequently when connecting to wifi when you want to get to the wifi login which required a plain http connection. This couldn’t be simpler with Caddy, below is the configuration I use for nossl.jarv.org.

nossl.jarv.org {
 redir http://nossl.jarv.org permanent
}

http://nossl.jarv.org {
 header Content-Type "text/html; charset=utf-8"
 header Cache-Control "no-cache, no-store, must-revalidate"
 respond <<NOSSL
<!DOCTYPE html>
<html lang="en">
<head>
<title>nossl</title>
<link rel="icon" href="data:image/svg+xml,<svg xmlns=%22http://www.w3.org/2000/svg%22 viewBox=%220 0 100 100%22><text y=%22.9em%22 font-size=%2290%22>🔓</text></svg>">
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
</head>
<style>
pre {
 font-family: "Courier New", Courier, monospace;
}

div {
 display: flex;
 align-items: center;
 justify-content: center;
 height: 100vh;
}
</style>
<body>
<div>
<pre>
⠀⠀⠀⠀⠀⠀⠀⢀⣀⣤⣤⣤⣄⣀⡀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⢀⣴⣾⣿⣿⣿⣿⣿⣿⣿⣿⣷⣄⠀⠀⠀⠀⠀
⠀⠀⠀⣰⣿⣿⣿⠟⠉⠀⠀⠀⠈⠙⠿⣿⣿⣷⡄⠀⠀⠀
⠀⠀⢰⣿⣿⡿⠁⠀⠀⠀⠀⠀⠀⠀⠀⠙⣿⣿⣿⡀⠀⠀
⠀⠀⣸⣿⣿⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢹⣿⣿⡇⠀⠀
⠀⠀⣿⣿⣿⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢸⣿⣿⡇⠀⠀
⠀⠀⢿⣿⣿⠇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢸⣿⣿⡇⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠸⣿⣿⡇⠀⠀
⢠⣤⣤⣤⣤⣤⣤⣤⣤⣤⣤⣤⣤⣤⣤⣤⣤⣤⣤⣤⣤⠀
⣿⣿⣿⣿⣿⣿⣿⣿⡿⠋⠉⠉⠛⣿⣿⣿⣿⣿⣿⣿⣿⣷
⣿⣿⣿⣿⣿⣿⣿⣿⡇⠀⠀⠀⠀⣸⣿⣿⣿⣿⣿⣿⣿⣿
⣿⣿⣿⣿⣿⣿⣿⣿⣿⡶⠀⠀⣾⣿⣿⣿⣿⣿⣿⣿⣿⣿
⢻⣿⣿⣿⣿⣿⣿⣿⣿⠃⠀⠀⠸⣿⣿⣿⣿⣿⣿⣿⣿⠏
⠀⠙⢿⣿⣿⣿⣿⣿⡏⠀⠀⠀⠀⢻⣿⣿⣿⣿⣿⡿⠃⠀
⠀⠀⠀⠈⠛⢿⣿⣿⣶⣶⣶⣶⣶⣾⣿⣿⠿⠛⠁⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠉⠉⠙⠛⠛⠉⠉⠉⠀⠀⠀⠀⠀⠀⠀
</pre>
</div>
</body>
</html>
NOSSL 200
}

Running a command with a timeout in Go

Tue, 27 Sep 2022 00:00:00 +0000

I’ve been slowly re-writing most of cmdchallenge in Go. It started with porting all of the Python code running in AWS Lambda, and now I am in the process of re-writing the command runner. This is the component that executes user provided commands, originally written in Nim.

The command runner takes user submitted shell commands and executes them in a Docker container. In the Go program, I initially used exec.CommandContext() and .CombinedOutput(). This met the following requirements:

A command to be executed but the execution needs to be time constrained
Fetching the output of the command with combined STDOUT and STDERR
Getting the command’s exit code

Initially using exec.CommandContext() and .CombinedOutput() seemed like a good fit since I could use context to send a kill signal to constrain how long the command would run. However, I noticed some odd behavior as it related to setting timeouts.

Take the following two examples:

log.Println("starting")
ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
defer cancel()

shArgs := []string{"-c", "sleep 10"}
_, err := exec.CommandContext(ctx, "sh", shArgs...).CombinedOutput(); err != nil {
 log.Fatalf(err.Error())
}
log.Println("finished")

// Output:
//
// 2022/09/27 19:08:04 starting
// 2022/09/27 19:08:14 signal: killed

log.Println("starting")
ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
defer cancel()

shArgs := []string{"-c", "sleep 10"}
if err := exec.CommandContext(ctx, "sh", shArgs...).Run(); err != nil {
 log.Fatalf(err.Error())
}
log.Println("finished")

// Output:
//
// 2022/09/27 19:10:13 starting
// 2022/09/27 19:10:15 signal: killed

Note how the first example reports that the program is killed, but not until a full 10 seconds despite setting a 2 second timout on the context. Passing the context doesn’t seem to do anything at all! In the second example, the program resumes after the 2 second timeout as we would expect. The key difference is that in the first example we are using CombinedOutput().

Let’s take a closer look at the process tree and what happens to the sleep 10. Assuming our program is called timeouttest, the process tree starts like this:

bash,1
`-timeouttest,8019
|-sh,8024 -c sleep 10
| `-sleep,7694 10

After 2 seconds, sh is sent a SIGKILL and we are left with:

bash,1
|-sleep,7694 10
`-timeouttest,8019

As seen above, sleep 10 becomes an orphan that is adopted by PID 1. sleep 10 is not killed, because the kill was sent to sh, and the signal is not propagated to its child. The main difference between calling .Run() and .CombinedOutput() is that the latter creates a buffer for Stdout for the process and its children. The progrem will will wait for that descriptor to close. This causes Go to hang, while it waits to copy sleep’s standard output to the buffer.

To ensure we can timeout properly, and that the sleep 10 is also killed, we will need to send a SIGKILL to the process group. This solves the problem because children created via fork will inherit the parent’s process group ID, and a kill sent to the group ID will kill the process and all of its descendants.

Here is code to kill the group ID after a 2 second timeout. The implementation below uses a channel with select:

log.Println("starting")
shArgs := []string{"-c", "sleep 10"}
cmd := exec.Command("sh", shArgs...)
cmd.SysProcAttr = &syscall.SysProcAttr{Setpgid: true}

type cmdResult struct {
 outb []byte
 err error
}
cmdDone := make(chan cmdResult, 1)
go func() {
 outb, err := cmd.CombinedOutput()
 cmdDone <- cmdResult{outb, err}
}()

select {
case <-time.After(2 * time.Second):
 syscall.Kill(-cmd.Process.Pid, syscall.SIGKILL)
 log.Fatal("signal: killed")
case <-cmdDone:
 log.Println("finished")
}

This sends a SIGKILL to the process ID (negated) which is how you send a signal to the process group, which is documented on the kill man page.

If pid is less than -1, then sig is sent to every process in the
process group whose ID is -pid.

This ensures that we will both timeout after the given time, and that all processes including children are killed.

From serverless to server

Wed, 29 Sep 2021 00:00:00 +0000

I’m pleased to announce that the transition from a distributed serverless architecture in Python to a single VM with a service written in Go is done for CMDChallenge. The final patch was merged and there is nothing sweeter than removing close to 100K lines of vendored Python.

It was around 4 months ago that I started on this adventure but honestly a lot of life got in the way and I probably spent a total of around 4 full days of work for the transition. Overall, I’m quite satisfied with the result, as you can see here many of the “cloud native” services went poof

What enabled this was moving all of the dispatching to the same server that was running containers for commands. I realize now in hindsight that other than wanting to play around with API gateway when I first built this, there was really no good reason why I shouldn’t have done this from the beginning. Like many side-projects, I guess this turned into a yak-shave where you start thinking about how you can use one cloud service, which eventually turns into ten.

A quick summary of the “new” architecture for those who are curious:

There is now a single Go service that is responsible for both request handling and launching Docker containers. Previously I was using AWS Lambda and issuing commands to a remote Docker server.
I still front this with Cloudflare but I removed all command caching at the edge. Every single request goes down to the VM where a Go service is running and returns a cached result (using sqlite) if it was already sent.
Using Cloudflare means I can use AWS certificates for HTTPs, the origin for Cloudflare is an EC2 instance
Instead of DynamoDB, I keep the command cache locally on the instance in an SQLite db which is much better and faster for lookups. A backup is periodically kept and restored on instance creation.

Before, querying solutions from Lambda out of Dynamodb was too slow so I needed to cache solutions with a Lambda cronjob. Now, with a local SQLite db, they are generated in the context of a request. For example, this request is generated with:

SELECT
cmd
FROM challenges
WHERE slug=$1 and correct=1
ORDER BY LENGTH(cmd) LIMIT 50;

I can’t overstate how nice this is compared to do doing very slow lookups over the network to Dynamo.

Previously I was using CloudWatch alerts and logs, now I have logs going to journald and using Prometheus for metrics. I also got rid of GoatCounter for anlaytics, because most people have ad blockers anyway, not sure how useful it was. I quickly burned through Sentry’s free useage cap so I went ahead and removed that as well.

With Prometheus running the VM for metrics, it’s now more convenient to add metrics for everything imaginable for executing commands:

Another benefit that came out of this is that now it is much easier to validate and run locally, if you want to run the entire site yourself check out https://gitlab.com/jarv/cmdchallenge.

Update on the Go rewrite

Thu, 02 Sep 2021 00:00:00 +0000

I’ve been stealing small chunks of time in the evening here and there and finally have a functioning Go rewrite of cmdchallenge which will replace all of the Python code that is running in AWS Lambda.

The first diff for this is massive, I am really looking forward to removing all of this vendored Python!

And we will say goodbye to some AWS services:

Cloudflare
Route53
~~DynamoDB~~ Replaced with an sqlite DB
~~Lambda~~ Now written as a Go service running on a VM
~~CloudWatch~~ Not needed
~~API Gateway~~ Not needed

What I didn’t expect during this rewrite was the joy of having everything running locally, including end-to-end tests in Go. It’s quite nice to be able to validate requests and being able to test using a browser locally without having to go through Lambda in the cloud.

Since a VM for the runner was already a requirement, I will likely stick the new web service on the same VM and keep a local SQLite DB. What I haven’t figured out yet, is how to manage persistence so that I can rebuild the single VM without losing any (or too much) data. Right now, I’m thinking something as simple as a shutdown script that syncs to object storage, and reads on provision.

Go rewrite of cmdchallenge

Mon, 05 Jul 2021 00:00:00 +0000

I’ve decided to take what was written years ago for cmdchallenge.com and port it to a single Go app. This is a pretty big change because I will be removing most of the AWS serverless components including API Gateway, Dynamo and possibly CloudFront. In the process, I would also like to simplify the architecture a bit. It seems like a single process with Docker running locally, and a db like sqlite should be sufficient. Here are some notes for the transition:

This will be a small program and I’m pretty sure I won’t need any frameworks for the REST server
I’ll need to do some lex parsing of the submission, https://pkg.go.dev/github.com/google/shlex should work.
1 years worth of data is about 300K unique entries in the submission table, in dyamoDB this is about 300MB of data.
Currently I’m keeping track of every single request in a larger submissions table that is used for rate limits, I think this won’t be necessary anymore since I can implement in-memory rate limiting without a tracking database.
I recently moved the runner from Go to Nim, but I think I’ll stick with Nim for now for the runner.
I would like to look into exporting some metrics to prometheus as well via a /metrics endpoint

So first thing to do is create a schema for commands:

 CREATE TABLE `commands` (
`fingerprint` TEXT,
`challenge_slug` TEXT,
`cmd` TEXT,
`cmd_length` INT,
`correct` BOOL,
`create_time` INT,
`output` TEXT,
`source_ip` TEXT,
`resp` TEXT
);

We will use the fingerprint of the command as the primary key for fast cache lookups
Scans on the sqlite db will be much cheaper than against dynamodb (in both time and money) so I don’t think we need a sort key like we did for DyamoDB

On removing Cloudflare caching, I think it will be interesting to remove it altogether, I assume that cache lookups in sqlite will be performant enought to bypass the extra caching layer.

Does Serverless make sense for a side project?

Sat, 28 Nov 2020 00:00:00 +0000

Command Challenge has been running for over 3 years now and I would consider it one of those “serverless” architectures. The front-end services requests using API Gateway and Lambda functions and commands are forwarded to a Docker executor running on a VM.

Overall, the costs have been minimal thanks to mostly low-ish traffic and keeping it inside the AWS free-tier limits. By “free-tier” I mean the “always-free” limits of an AWS account, since the 12 month trial period for new accounts expired quite some time ago.

Here is a 2020 update for the services that are used to run the site and a short explanation of what they do.

Serverless Architecture

Where we could probably move all of this to a single VM and reduce some of the configuration complexity, I don’t think it would be any cheaper except on a shared hosting provider.

As you can see in the diagram, this project isn’t completely “serverless” because a server is required for executing user-submitted commands in Docker. For this, it’s a matter of finding the cheapest VM available for this type of workload.

Free Cloud Services

GoatCounter: An analytics service that respects your privacy.
Slack: Used to receive notifications for unique submissions and errors, useful to keep an eye on what is going on.
Grafana Cloud: Grafana offers a free starter plan. This connects to Prometheus which is running on the GCP VM for monitoring Docker and node level metrics.
Sentry: Sentry.io is very nice for side-projects, offering a free tier that I found indispensable for tracking down front-end JS errors in different browser configurations.
GitLab: GitLab.com drives the CI pipeline, every new commit on master updates the testing environment, with a manual promotion to prod.

AWS - $2/month

CloudFront: Used as the CDN for all requests to the site. HTTP GET requests to /r/* are for command submissions and are forwarded to the API Gateway. These requests are cached here, so if multiple submissions are sent for the same challenge CloudFront will return a cached response. Other requests that are not command, assets and the static page are forwarded to S3.
S3 bucket: Hosts all static content and serves as the origin for CloudFlare. Also receives periodic updates for user solutions, stored as JSON files.
API Gateway: Previously REST API, this was recently switched to the HTTP API Gateway. Accepts an HTTP request and forwards it to Lambda function for submissions.
Lambda:
- Submission Handling: Responsible for handling submission and rate-limiting logic. It takes a submission and forwards it to the Docker executor. If it receives the same input for a challenge that has already been evaluated, it will return a cached response
- Solutions Updater: This is split into multiple Lambda functions that look up solutions and writes them to S3 as json files. It runs in multiple jobs because the queries to DynamoDB span many records and take awhile to run.
- Slack Notfications: On every unique successful submission, or error a notification is sent to Slack. This is helpful to keep an eye on what is going on and aids troubleshooting when there are problems.
DynamoDB: Every submission is stored in the database which is used for both rate limiting, and tracking correct/incorrect submissions.
CloudWatch Logs: Collects logs from Lambda for debugging.
Event Bridge: Previously called CloudWatch Rules, this triggers periodic events that are used to trigger a Lambda function that generates a list of user-submitted solutions.
Route53: The one service that doesn’t have a free tier, for .50 a month it manages the cmdchallenge.com DNS zone.
Amazon Certificate Manager: Free HTTPs certificates for all of cmdchallenge.com

Breakdown of a typical monthly bill

GCP - $6/month

The production site uses an e2-micro instance costing around $6/month, this is a bit more than we need typically but it gives us some headroom when the site gets busy. There is only a single VM which is a single-point-of-failure for the site. This can cause a bit of disruption if there is a spontaneous reboot or if an upgrade is required. Thankfully, in GCP this is very fast, and thanks to the multiple-layers of caching (CloudFlare, DynamoDB) in the serverless stack, it only causes a problem for unique submissions that haven’t been seen before.

Memory utilization hovers around 300MB when there isn’t a lot of activity, this means that on smaller shared-core VMS like the f1-micro some swap is necessary.
CPU Utilization is generally low, though it varies because traffic due to site usage tends to be bursty.

VM Comparison for shared-core or burstable instances

This architecture spans cloud providers to keep costs low as I found GCP gives a better value for the money on VMs compared to AWS at this lower tier. Here is a current-as-of-now overview of cheap VMs under $15 from the main cloud providers.

All of these are non-preemptable shared core instances or burst-rate limited in the case of AWS. This means that you cannot count on using the full CPU all of the time.

Instance	vCPUs	Memory	Price $/month
GCP f1-micro	.2	.6GB	$5
GCP e2-micro	2	1GB	$6
GCP e2-small	2	2GB	$12
GCP g1-small	.5	1.7GB	$12
AWS t2-nano	1	.5GB	$4
AWS t2-micro	1	1GB	$8
AWS LightSail	1	0.5GB	$3.50
AWS LightSail	1	1GB	$5
AWS LightSail	1	2GB	$10
Digital Ocean	1	1GB	$5
Digital Ocean	1	2GB	$10
Digital Ocean	2	2GB	$15

Notes

For GCP, preemptable instances would be cheaper but with only one it doesn’t really work since it would mean a service interruption.
The f1-micro is a bit too small for the main site, I use it for the testing environment.
AWS LightSail / Digital Ocean have basically the same offering and might also be a good fit.

For now, the main site is powered by an e2-micro instance for $6/month. It offers 2 cores, which is more than the other options in the same price-range though it’s not uncommon to see stalls due to CPU steal:

It runs Container Optimized OS and only runs Docker and a Prometheus exporter for node metrics.

So, is it worth it?

From a cost perspective for now I still think so. Of course there is always a risk that something happens where my cloud spend will spiral out of control but if this were to happen I would just shut it down. There is the meme “Wouldn’t it be easier to run this on a $5 DO droplet”? I think it might apply here, though the type of workload it runs is ideal for serverless because it is a single-page app where API Gateway requests are only required for submissions. This keeps the number of Lambda executions low.

Ignoring spend I think there are two big disadvantages to running servless. One is the complexity of gluing the cloud-native components together and keeping a configuration that is easy to manage and update. This is solved by keeping everything in a single terraform script, and though updates are automatically applied in CI I think there is a rather big initial investment in getting it to work properly.

The other, bigger reason I found is that it’s difficult, if not impossible to test everything locally, without a lot more work which would be my number one complaint about this setup. The way I have worked around this is to have an identical testing environment running in parallel, this actually doesn’t cost anything extra because it doesn’t get any usage beyond a small amount of testing.

To sum up, while it’s been fun for someone who enjoys doing infrastructure, I have been thinking about re-writing it all as a single server that run on a single VM, with sqlite as the db. I’m pretty sure it would perform better :)

Moving to HTTP from REST for the AWS API Gateway

Sun, 25 Oct 2020 00:00:00 +0000

HTTP API Gateway

When HTTP API Gateway was announced in 2019 the Amazon said:

Our goal is to make it as easy as possible for developers to build and manage APIs with API Gateway. We encourage you to try the new HTTP APIs and let us know what you think.

Today I decided to switch from using the REST API for cmdchallenge.com to the HTTP API and I must say it is a lot easier to build and setup for simple HTTP APIS. In Terraform, it now only requires two resources:

resource "aws_apigatewayv2_api" "default" {
 ...
}

resource "aws_lambda_permission" "default" {
 ...
}

Compare this to the previous configuration using REST API:


resource "aws_api_gateway_rest_api" "default" {
 ...
}

resource "aws_api_gateway_method" "default" {
 ...
}

resource "aws_api_gateway_integration" "default" {
 ...
}

resource "aws_lambda_permission" "default" {
 ...
}

resource "aws_api_gateway_deployment" "default" {
 ...
}

For a full working example, see the API Gateway Terraform configuration for cmdchallenge.com. In addition to making the configuration simpler, it’s supposed to be optimized for performance and this doesn’t disappoint either. Testing against an API endpoint that invokes a Lambda function and makes some DynamoDB calls there is a nice saving of around 100ms for the 50th percentile and 200ms for the 90th.

REST API Gateway:

 $ bombardier --latencies --rate 50 -d 30s 'https://g4jrkpyb3d.execute-api.us-east-1.amazonaws.com/r/?cmd=echo+hello+world&challenge_slug=hello_world'
Bombarding https://g4jrkpyb3d.execute-api.us-east-1.amazonaws.com:443/r/?cmd=echo+hello+world&challenge_slug=hello_world for 30s using 125 connection(s)
[============================================================================================================================================================================] 30s
Done!
Statistics Avg Stdev Max
 Reqs/sec 48.45 33.81 211.47
 Latency 1.13s 845.04ms 6.77s
 Latency Distribution
 50% 0.88s
 75% 0.99s
 90% 2.71s
 95% 2.92s
 99% 4.78s
 HTTP codes:
 1xx - 0, 2xx - 1501, 3xx - 0, 4xx - 0, 5xx - 0
 others - 0
 Throughput: 86.94KB/s

HTTP API Gateway:

$ bombardier --latencies --rate 50 -d 30s 'https://ddzt9hixi4.execute-api.us-east-1.amazonaws.com/?cmd=echo+hello+world&challenge_slug=hello_world'
Bombarding https://ddzt9hixi4.execute-api.us-east-1.amazonaws.com:443/?cmd=echo+hello+world&challenge_slug=hello_world for 30s using 125 connection(s)
[============================================================================================================================================================================] 30s
Done!
Statistics Avg Stdev Max
 Reqs/sec 48.44 30.62 235.42
 Latency 0.93s 679.98ms 6.70s
 Latency Distribution
 50% 703.40ms
 75% 731.99ms
 90% 1.66s
 95% 2.70s
 99% 3.98s
 HTTP codes:
 1xx - 0, 2xx - 1501, 3xx - 0, 4xx - 0, 5xx - 0
 others - 0
 Throughput: 33.19KB/s

Caching Responses with the HTTP API Gateway

Command Challenge has multiple layers of caching for anonymous usage. If there wasn’t any caching, every single submission would execute in a Docker container, which would not only be atrocious from a performance standpoint but also expensive as it would require multiple/large VMs executing commands in containers.

Every command that is submitted is hashed, and written to a DynamoDB table. If a subsequent command is identical for the same challenge, we return a cached response. Putting CloudFront in front of API Gateway is a nice way to get additional caching for free, which works for the HTTP Gateway which doesn’t have a caching feature. To cache HTTP API Gateway responses, create a CloudFront distribution with the API endpoint as the Origin Domain Name.

Displaying the percentage of hosts completed during a rolling Ansible Deploy using serial

Fri, 02 Oct 2020 00:00:00 +0000

Recently we were looking at enhancing our deployment pipeline so that we run validation checks between every batch that is upgraded. For our VM deployments in Ansible we normally have

serial: 10%

so that 10% of the fleet is operated on at once, this includes draining from a load balancer before we upgrade the node, and adding it back to the load balancer after.

With Ansible, it doesn’t seem like there is any built-in way to get the host percentage, so here is a hacky solution for others who want to do something similar:

Initialize a fact to an empty array:

If we put this in pre_tasks: you will need to keep in mind it is called on every batch, so you don’t want to override it:

 - name: Initialize completed_hosts
 set_fact:
 completed_hosts: "{{ hostvars['localhost']['completed_hosts'] | default('[]') }}"
 delegate_to: localhost
 delegate_facts: true
 run_once: true

Another option is initialize it in a play that runs before the play that is operating on hosts, in that case you can just initialize it to an empty array:

 - name: Initialize completed_hosts
 set_fact:
 completed_hosts: []

In post_tasks, append the current batch to completed_hosts:


 - name: Appends to completed hosts
 set_fact:
 completed_hosts: "{{ hostvars['localhost']['completed_hosts'] + ansible_play_batch }}"
 delegate_to: localhost
 delegate_facts: true

Calculate a percentage:


 - name: Percentage complete
 debug:
 msg: "Percentage complete: {{ ((hostvars['localhost']['completed_hosts'] | length) / (ansible_play_hosts | length) * 100) | int }}%"
 run_once: true
 delegate_to: localhost

And here is an example test play:

- name: test
 gather_facts: false
 hosts: some_host_group
 serial: '10%'
 pre_tasks:
 - debug:
 msg: drain
 - name: Initialize completed_hosts
 set_fact:
 completed_hosts: "{{ hostvars['localhost']['completed_hosts'] | default('[]') }}"
 delegate_to: localhost
 delegate_facts: true
 run_once: true
 tasks:
 - debug:
 msg: deploy
 post_tasks:
 - debug:
 msg: enable
 - name: Appends to completed hosts
 set_fact:
 completed_hosts: "{{ hostvars['localhost']['completed_hosts'] + ansible_play_batch }}"
 delegate_to: localhost
 delegate_facts: true
 - name: Display current batch
 debug:
 var: ansible_play_batch
 delegate_to: localhost
 run_once: true
 - name: Display completed hosts
 debug:
 msg: "Percentage complete: {{ ((hostvars['localhost']['completed_hosts'] | length) / (ansible_play_hosts | length) * 100) | int}}%"
 run_once: true
 delegate_to: localhost

Ensuring a consistent PID in a container

Fri, 31 Jul 2020 00:00:00 +0000

Recently a challenge was added that asks you to identify a process and kill it. This was a new type of exercise that requires a running process in the docker container where all commands are run. This itself wasn’t really an issue except that with caching, it requires that the state of the container to be identical from one run to the next. PIDs however, are not so predictable, even when you only have a very small wrapper.

PID 1: wrapper that run the command
PID 6 or 7: forked command passed to the wrapper

I’m not sure why sometimes the PID would be 6 and other times 7, but this causes bit of a problem since if you executed a command like:

ls -d /proc/[0-9]*

You might get a cached result, which doesn’t represent the PIDs that are running in the context of your next command.

So like most, I went to stackoverflow and found this stackoverflow post it can be done by setting ns_last_pid

Open /proc/sys/kernel/ns_last_pid and get fd

flock it with LOCK_EX

write PID-1

fork

But you can’t set this in a docker container unless you run it privileged, which is not a great idea for running arbitrary commands passed in by the Internet.

Instead, this ended up getting solved with a very boring solution. For challenges where we need a running process, the wrapper cycles through all the PIDs up to PID number 41, and then forks the process that needed to be killed. The logic is quite simple, and looks something like this:

proc start*(oopsProc: var OopsProc, prog: string = OOPS_PROG, targetPid: int = 42) =
 if not oopsProc.slug.startsWith("oops"):
 return

 while true:
 let p = startProcess(
 command=OOPS_PROG, args=["-t", "0"], options={poUsePath}
 )
 let _ = p.waitForExit()
 p.close

 if p.processId == targetPid - 1:
 break

 oopsProc.p = startProcess(
 command=OOPS_PROG, options={poUsePath}
 )
 oopsProc.pid = oopsProc.p.processId

This is how for challenge “kill a process”, the answer is always kill -9 42!

Cat without cat on the commandline

Thu, 30 Jul 2020 00:00:00 +0000

Say you want to display the contents of a file on the command line. The first tool we most of us reach for is cat, which does a fine job at just this. But what happens when you are on a Linux machine and when you try to cat a file this happens:

$ cat file.txt
-bash: cat: command not found

or even:

$ cat file.txt
bash: fork: retry: No child processes

This post explores this idea and was a feature of this challenge where you needed to display a file’s contents without using any utility outside of the shell.

Using shell built-ins, redirection and subshell

Using the shell builtin read you can display the contents of a file, without a forking a new process:

while read line; do
 echo $line;
done <file.txt

From the help page for read:

One line is read from the standard input, or from file descriptor FD if the -u option is supplied, and the first word is assigned to the first NAME,

In this example, the contents of file.txt are redirected to the STDIN of read, which processes the input line by line, until it reaches the end of the file. read also can take a file descriptor as its input instead of STDIN, so this will also work:

exec 3<file.txt # Assign file descriptor 3 for reading
while read -u 3 line; do
 echo $line
done

This ends up being a lot more typing than just cat file.txt. With the bash or zsh there is a another way to display a file’s contents without using cat:

echo "$(<file.txt)"

this method uses redirection and command substitution, and is mentioned in the bash man page:

The command substitution $(cat file) can be replaced by the equivalent but faster $(< file).

It’s faster, because you are not forking a cat, but does it matter? Probably not, and may not be clear to everyone what you are doing, but you can see a difference with a quick test on your shell:

$ time for n in {1..1000}; do \
 echo $(</etc/resolv.conf) >/dev/null; \
 done

real 0m0.977s
user 0m0.380s
sys 0m0.604s

$ time for n in {1..1000}; do \
 cat /etc/resolv.conf >/dev/null;
 done

real 0m1.980s
user 0m0.626s
sys 0m1.224s

This syntax $(<file.text) may look a bit strange, what you are doing is command substitution, where the contents of file.txt are sent to STDIN which is then echo’d as STDOUT. If you want to learn a bit more about redirection using > and < see my earlier post about shell redirection.

Using other utilities

How about other options? Without using shell built-ins but instead using other standard utilities you can also cat without cat:

ul < /file

ul might inadvertently underline some words in your file but I think it might be the only way to cat a file with only two characters.

tac /file | tac

If you didn’t already guess, tac is GNU core util that is the reverse of cat so if you want to be clever you can pipe the output of tac to tac which is just a cat.

Of course using tools like sed, perl, python, etc. will allow you to cat files as well, happy cat’ing!

Interested in other ways to cat without cat? Try the oops challenge!

Switching to Hugo

Sun, 26 Jul 2020 00:00:00 +0000

Hello world from the new blog using Hugo as the static site generator. After cutting away the cruft from the old one, which used Pelican as a static site generator, I decided to make a switch on a lazy Sunday. I definitely think the design is much nicer and site generation is much faster! There is a great selection of themes, this one uses the kiera which is exactly what I wanted for a layout that doesn’t have too many bells and whistles.

Understanding shell redirection

Sat, 25 Jul 2020 00:00:00 +0000

When I was learning to use the shell for the first time I remember teaching myself to do things as I ran into problems, and the first problem I ran into was how to take the output of one command and write it to a file or pipe it a second command.

So like most people, I learned:

cmd1 | cmd2

to take the output of cmd1 and send it to cmd2, and

cmd1 > path/to/file

to take the output of cmd1 and write it to a file.

This got me pretty far for what I needed to do without fully understanding how it worked or why. Eventually, I learned as well that there were multiple types of output, STDOUT and STDERR, and you could redirect one to the other by adding 2>&1 to the end of commands. Early on, I wish I took the time to understand how redirection works and how it relates to file descriptors.

To understand this a bit better, first it’s helpful to understand file descriptors. At a very high level, file descriptors are numbers assigned by the operating system that reference open files. The shell will assign the following:

0: STDIN - program input
1: STDOUT - program output
2: STDERR - program error

When using a pipe | or redirection </> it’s helpful to think abut these 3 file descriptors and their corresponding number assignments.

Going back to the example where we are redirecting STDOUT of a program to a file:

ls > path/to/file

This is is the same as:

1> /path/to/file ls

Where 1 refers to STDOUT of ls

Redirecting Output

Let’s break down exactly what > is doing, from the bash man page:

The general format for redirecting output is:
 [n]>word

Where [n] is a file descriptor, but in the example > /path/to/file cmd there isn’t a file descriptor on left side! But there is actually, if you don’t specify a specific file descriptor, for output redirection, STDOUT is the default. Which means all of these redirections are exactly the same:

$ ls > path/to/file
$ > /path/to/file ls
$ 1> /path/to/file ls

Note where it is more intuitive to put the > after the command, it’s not necessary. This is because the position doesn’t matter because the descriptor to the left of > is implicit. Pretty neat!

Redirecting Input

Redirecting input is almost exactly the same, from the bash man page:

The general format for redirecting output is:
 [n]<word

Note that again there is a file descriptor on the left, and a file on the right. The only difference is that when using a < the default file descriptor is 0 (STDIN) if one isn’t specified. Therefore, all of these commands are equivalent:

$ cat < path/to/file
$ < path/to/file cat
$ 0< path/to/file cat

Again, I think the first one is a bit more intuitive but it doesn’t matter whether the redirection is placed before or after the command.

Redirecting stdout to stderr

So back to 2>&1, why is the & necessary and what does it mean? Remember that for > there needs to be a file descriptor on the left and a file on the right? If we used 2>1 this would simply redirect STDOUT to a file named 1. By adding the & it tells the shell that 1 is referring not to a file, but a file descriptor!

So the main things to remember when it comes to shell redirection:

For redirecting input and redirecting output, there should always be a file descriptor on the left, and a file (for reading, or writing) on the right.
If a file descriptor is not specified, the default for output > is 1 (STDOUT) and the default for input < is 0 (STDIN).
For redirecting STDOUT to STDERR, the & is necessary on the right side, because you are redirecting to another file descriptor, instead of a file
The best reference for all of this is the bash man page, which contains a lot more in-depth information on ways to redirect!

A new new new blog

Wed, 17 Jun 2020 00:00:00 +0000

Yeah it’s another blog update, just really some style updates and looking back at something I did a long time ago and say what the heck was I thinking?. I don’t know, I think things that I don’t have time to spend up usually end up being a craptastic design by copy-and-paste. Anyway, still using Pelican but upgraded to Python3 and fixed the CI configuration a bit. Got rid of the bourbon/{sass,neat} mixins because that was so 2013. Still using sass though.

pelican with a custom theme
~~bourbon/sass - CSS compilation and general typography~~
~~bourbon/neat - For layout and responsive design~~

Oh and remove as much as possible like javascript easter-eggs and social links, because who needs that stuff :)

Never use git submodules

Wed, 17 Jun 2020 00:00:00 +0000

I’m writing this as a reminder to my future self never to use git submodules. For most things I would say never say never but let’s make an exception for this one.

I’ve used submodules for the following reasons:

There is some logical separation or a clear defined interfaces between two things, and repos are cheap
There is a 1 to many (usually 2 or 3) relationship between this thing (that has stable interface) and something else

The problem is that this 1-to-many thing usually starts as 1-to-2 and usually doesn’t go much further than that. It’s always this could eventually increase in scale but usually those hunches are wrong.

So where does it fall down? It seems like it rarely works well:

Cognitive overhead for multiple git commands to keep things updated
Doing normal things with git (conflict resolution, reviews, etc) feels super clumsy
With reviews, it often means reviewing multiple times and messes up approval workflows
It’s 2020 and this process still looks unnecessarily complex https://stackoverflow.com/questions/1260748/how-do-i-remove-a-submodule

My alternative for now for better gitops workflows is mono-repo everything, this seems to work better with CI anyway

The stages of a side project

Wed, 17 Jun 2020 00:00:00 +0000

One experience of mine, though it has roughly mapped to previous side projects:

Wow this thing that has no business model is a cool idea, I wonder if anyone has done it before?
This is fun, let’s spend an evening hacking on it
Wow this is taking way more time than I thought it would …
Can I do this without spending any money? Let’s use AWS free tier to get this going
OK let’s ship this thing, wow it’s getting a lot of attention! Front-page on hackernews, subreddit, etc.
I’m totally ready for this, I have a scaling plan
Oops my scaling plan is completely useless because the bottlnecks are not where I expected they would be
Wow traffic really drops off when it isn’t on hackernews 🤔
Oh wait someone tweeted it, there goes traffic again..
I should probably add CI to make this a bit easier to maintain
OK things are stable again
Wait my free tier is about to expire, let’s figure out how to host this for free by using multi-cloud
Whoa this is taking longer than I thought
.. years pass
OK something is broken and getting some email complaints, I really have lost interest in this thing
Have a free weekend, let’s look at the code from years ago
Good god, it’s terrible
Well, a bunch of things are now deprecated, let’s at least fix that
OK so if I am going through this trouble I should probably re-write some of it, especially the last parts that were just thrown together…
Whoa this is taking longer than I thought…

Introducing a newer version of cmdchallenge .. A lone survivor in a sea of broken things I have had an itch to build.

Using GitLab CICD for your static blog

Tue, 05 Jun 2018 00:00:00 +0000

In the process of making all of my private repositories public on Github and moving off of GitHub pages I have also decided to move most of my repos over to GitLab. One reason is that setting up a CICD pipeline makes it extremely easy to publish these posts automatically. This in combination with GitLab’s web IDE makes minor changes a little bit easier to make.

CICD piplines in GitLab are controlled with a single file, .gitlab-ci.yml that is placed at the root of the repository. Wit this file, on every commit, the following pipeline runs that deploys to draft.jarv.org and on the master branch for jarv.org.

GitLab has integrated CI/CD runners that allow you to execute whatever you want in a docker image of your choice for generating a deployment pipeline.

Here is what the gitlab-ci.yml configuration looks like for the jarv.org repository:

Some notes about the setup:

There are two stages, build and deploy. Build generates the html for the blog and deploy deploys it to both draft and the main site.
Deploying to the main site only happens with a manual job
I am using a custom image that has some of the blog dependencies pre-installed, it is created with this docker file.
Every time the deploy happens, an invalidate is sent to the cloudfront distribution in front of it.
Since I am no longer using GitHub pages, the blog is hosted in an S3 bucket so the CICD pipeline does an aws s3 sync ... to get the static files on the bucket.

There is a special key named environment that hints to GitLab that the stage is a deploy step. With this the URL provided will show up under operations->deployments like this:

That’s it! It couldn’t be simpler and I think one of the nicest things about this setup is the ability to use the GitLab web ide to make quick changes.

Disclaimer: I work for GitLab though the opinions here are my own.

Removing all of my private repos from GitHub

Thu, 31 May 2018 00:00:00 +0000

Today I decided to remove all private repositories from GitHub. Why? Interesting that having private repositories generally meant that I was not being as careful about managing secrets properly. Checking in API tokens, keys and especially cloud tokens into git never a good idea and looking through some of my old private repos I was doing exactly that. So for others who are interested in going from a developer GitHub account back to free I highly recommend doing it! Granted it’s only $5 a month savings but I definitely feel a bit more transparent than I did 15 minutes ago. :) For finding secrets you can use something like trufflehog to ensure that there no sensitive bits in your git history.

Building cmdchallenge using Lambda and API Gateway in the AWS free-tier with Docker and Go

Mon, 24 Apr 2017 00:00:00 +0000

Have you ever thought about building a side-project for fun without spending a lot on hosting? This post might be for you. With the most tech-buzz-wordy title I could conjure up here is a quick overview of how cmdchallenge.com is built. The site is a simple web application side-project that executes shell commands remotely in a docker container in AWS. The front-end gives the feeling of a normal terminal but underneath it is sending whatever commands you give it remotely on an EC2 instance inside a Docker container.

The source code for most of it is located on github including a tiny command executer written in Go, the challenge definitions, and a test harness.

The following AWS services are used for the site:

Cloudfront
API Gateway
S3 bucket
Lambda function
DynamoDB
t2.micro EC2 Instance running coreos
CloudWatch logs

In addition to this Amazon Certificate Manager and Route53 was used but for everything above you can keep costs close to zero in AWS. There is no free tier for Route53 (sad panda) but it’s like 50 cents a month for a single zone.

Features

Submit commands, execute them in a bash sub-shell.
Check the output of the command for different challenges.
Run tests for challenges that need them in addition or in place of checking output.

Deployment tools (simple and boring)

Makefiles.
Python fabric for running commands and copying files over ssh.
Kappa, zips up code, sends it to lambda, also manages Lambda permissions.

With these tools the following automated steps are taken to deploy the site:

Create a Docker image that holds the challenges.
Launch a new coreos EC2 instance.
Run a fabric script that does the following on the instance over SSH:
- Configures TLS so that a Lambda function can communicate to Docker on an EC2 instance.
- Executes some periodic commands to ensure that the host cleans up old containers.
- Downloads the docker image that has the challenges.
- Copies the read-only volume that is used on the container for the tests and command runner.
Update Lambda with new code.
Sync the static assets to S3.
Invalidates CF cache for the main site.

Architecture diagram:

There are two public entry points for the site, one is the main web-site which is static and served S3. The other is the API gateway at api.cmdchallenge.com which is also fronted by CloudFront so that it can use a certificate from ACM and cache requests.


api.cmdchallenge.com cmdchallenge.com
******************** ****************
+---------------------+ +---------------------+
| Cloudfront | | Cloudfront |
+---------------------+ +---------------------+
| |
+---------------------+ +-----------+
| API Gateway | | s3 bucket |
+---------------------+ +-----------+
|
+-----------------+
| |
| Lambda Function | +----------+
| |--- | |
+-----------------+ \| DynamoDB |
| | |
+--------------+ +----------+
| EC2 t2.micro |
| (coreos) |
+--------------+

One nice thing about using AWS server-less components was that a single t2.micro instance ended up being fine for handling all of the load, even at peak.
See section on caching/performance below.

Here is what happens when a command is submitted in the cmdchallenge.com terminal:

Javascript code sends an HTTP GET to https://api.cmdchallenge.com
If it is cached it returns a response immediately. If not, it forwards the request to the API gateway which in turn sends it to a Lambda function.
The Lambda function looks up the challenge and the command in DynamoDb and if it already has an answer it returns that. If the challenge doesn’t exist in DyamoDB it is forwarded to the EC2 instance as a command using the docker API.
The command that the user provides is passed to a Go command runner that executes the command in a bash sub-shell inside a docker container, checks the output and runs the tests.
Results are returned to the Lambda function, it writes them to DynamoDb and returns the response.

The challenges are expressed in a single YAML, here an example of one challenge:


 - slug: hello_world
 version: 4
 author: cmdchallenge
 description: |
 Print "hello world".
 Hint: There are many ways to print text on
 the command line, one way is with the 'echo'
 command.

 Try it below and good luck!
 example: echo 'hello world'
 expected_output:
 lines:
 - 'hello world'

Interested in coming up with your own? You can submit your own challenge with a pull request. Your challenge will be added to the user-contributed section of the site.

Caching:

You may notice that when you do echo hello world on the hello world challenge it returns almost immediately. As it is shown above there are two layers of cache, one at CloudFront and one at DynamoDb to reduce the number of command executions on the Docker container. API Gateway can provide caching but it costs money, I worked around this by sticking CloudFront in front of it but this is only possible with HTTP GETs. With Cloudfront in front the cache-control header in the response from Lambda is set to a very long cache lifetime with every request. The version of the challenge as well as a global cache buster param is passed in so we never have to worry about returning a response from a stale challenge.

Performance:

If you are wondering how well this would scale for a lot of traffic, the Lambda function currently dispatches commands to a random host in a statically configured list of EC2 instances making it pretty easy to add more capacity. So far it seems to be operating fine with a single t2.micro EC2 instance handling all command requests that are not cached.

Time to get a echo hello world response from a cached cloudfront command - ~50ms
Time to get a echo hello world response from a cached command in dynamoDB - ~2.5s
Time to get a echo hello world response, executed in a container - ~4s

Without caching this wouldn’t be possible and also the caching at CloudFront enables most commands to return fairly quickly.

Wrapping up

If you like the site please follow @thecmdchallenge on twitter or if you have suggestions drop me a mail at info@cmdchallenge.com.

Thanks!

User Submitted Solutions

Sat, 04 Mar 2017 00:00:00 +0000

Adding to the interesting 191 ways to echo hello world I’ve now added the ability to see user-submitted solutions to cmdchallenge.

There are some gems if you dig through them including maybe the longest regex I’ve ever seen for pulling an IP address out of a file:

(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)

Also scrolling down the page of solutions to the corrupted text problem is glorious.

~~The solutions are not updated regularly right now but would be easy enough to do in the future if people want to see more, let me know on twitter and also~~ Update: Solutions are now generated every five minutes. Feel free to submit suggestions for new challenges on github.

figlet breakout

Fri, 24 Feb 2017 00:00:00 +0000

I was looking for a cool ending for cmdchallenge and decided to dust off a 2 year old javascript project which created a breakout game from figlet fonts. Not quite a full re-write but fixed a lot of bugs and did away completely with coffee-script. More info on the github page.

Or you can click here to play.

191 ways to echo hello world on the command line

Wed, 08 Feb 2017 00:00:00 +0000

It’s been about 12 days since the launch of cmdchallenge, a weekend project to create some common command-line tasks that can be done in a single line of bash. One common request has been to share user-submitted solutions. Or to put it another way, you may be wondering what do random people on the internet and hackernews type if you give them some basic command-line tasks and a shell prompt? Well lucky for me this is no longer a mystery!

Starting off with Challenge #1:

CMD Challenge #1: print “hello world” at the bash prompt

There has been a lot of diverse input for such a simple task. I really love how people do weird stuff even when it is totally unnecessary.

Here are some of my favorites:

( for i in h e l l o \  w o r l d ; do echo "$i" |awk -F, ' {print $NR}'; done ) |tr -d \\n; echo
echo ifmmp xpsme |tr bcdefghijklmnopqrstuvwxyza abcdefghijklmnopqrstuvwxyz
touch helloworld && echo "hello world" > helloworld && cat helloworld
sed 's.\.\...;yhHh\hh;ywWw\ww;2,$d' README

A few things to note that may clarify some of the more interesting submissions:

There is a README in each challenge directory, in this case it contains the string “hello world” so some people took advantage of that.
The directory itself was named hello world.

Here are all of the correct submissions for the first challenge as of yesterday:

a=(d e h l o r w X \");s=(2 1 3 3 4 7 6 4 5 3 0);for i in ${s[@]} ; do echo -n ${a[$i]}|tr X\n ' ' ; done ; echo ""
( for i in h e l l o \  w o r l d ; do echo "$i" |awk -F, ' {print $NR}'; done ) |tr -d \\n; echo
for i in h e l l o; do echo -n $i; done; echo -n " "; for j in w o r l d; do echo -n $j; done
cat README | head -n 4 | tail -n 1 | awk '{ print $3 " " $4 }' | sed -e 's/\"//g;s/\.//g'
echo ifmmp xpsme |tr bcdefghijklmnopqrstuvwxyza abcdefghijklmnopqrstuvwxyz
< README grep hello | tr -d '#' | cut -f3- -d' ' | tr -d '"' | tr -d '.'
touch helloworld && echo "hello world" > helloworld && cat helloworld
letters=(h e l l o \  w o r l d); printf '%s' "${letters[@]}"
touch test.txt ; echo "hello world" > test.txt ; cat test.txt
grep "hello world" README | sed -E 's/.*(hello world).*/\1/'
cat README | grep Print | sed -E 's/^.+"([^"]+)".+$/\1/'
echo "hello world" > ./hiworld.txt && cat ./hiworld.txt
head -n 1 < README | sed 's/# //' | tr '[A-Z]' '[a-z]'
cat $0|cut -d\; -f4;echo ";hello world;" > /dev/null
touch file && echo "hello world" > file && cat file
cat README | sed -e 's/.*/hello world/' | head -1
touch "hello world"; ls *hell*; rm "hello world"
head -1 README | cut -c3- | tr '[A-Z]' '[a-z]'
echo "hello world" > hello.txt | cat hello.txt
awk 'BEGIN {print "hello world"}' < /dev/null
echo "hello world" > hello.txt; cat hello.txt
echo "hello world" > testfile; cat testfile
echo "hello world" > test.txt; cat test.txt
grep hello README | awk -F\" '{ print $2 }'
for n in 'hello world' ; do echo ${n}; done
sed 's.\.\...;yhHh\hh;ywWw\ww;2,$d' README
echo "hello world" > foo.txt; cat foo.txt
echo -n "h""e""l""l""o"" ""w""o""r""l""d"
touch 'hello world' && ls h*| xargs echo
msg="hello world"; printf '%s\n' "$msg"
cat README| grep hello | cut -d '"' -f2
echo " " | awk '{print "hello world"}'
echo "hello world">hello | tail hello
awk ' BEGIN { print "hello world" } '
export HAHA="hello world";echo $HAHA
echo 'hello world' > henk | cat henk
echo "hello world" > test & cat test
awk 'BEGIN { print "hello world"; }'
echo "$(echo hello)" "$(echo world)"
perl -e 'printf "%s", "hello world"'
mkdir "hello world" && ls | tail -1
awk 'BEGIN { print "hello world" }'
pwd | cut -d'/' -f4 | sed 's/_/ /g'
touch "hello world"; ls | grep ello
printf '%s%s%s%s\n' hel lo\  wor ld
cat README | grep -o "hello world"
echo "hello world" | sed 's/b/h/g'
echo | awk '{print "hello world"}'
echo "hello world" > tmp; cat tmp
echo hello world > henk; cat henk
cat >/dev/stdout <<<"hello world"
awk 'BEGIN{print "hello world";}'
awk 'BEGIN{print("hello world")}'
cat >/dev/stderr <<<"hello world"
awk 'BEGIN {print "hello world"}'
echo "hello world">algo;cat algo
hello="hello world"; echo $hello
perl -e 'print "hello world\n";'
GGG="hello world" ; cat <<< $GGG
echo|awk '{print "hello world"}'
echo 'hello world' > /dev/stdout
awk 'BEGIN{print "hello world"}'
ls .. | grep hel | sed 's/_/ /g'
printf "%s %s\n" "hello" "world"
perl -e 'print "hello world\n"'
echo s | sed 's/s/hello world/'
echo -n "hello"; echo " world";
#!/bin/bash\necho "hello world"
printf -- '%s\n' 'hello world'
printf '%s %s' 'hello' 'world'
printf "%s %s" "hello" "world"
perl -le 'print "hello world"'
text='hello world'; echo $text
perl -e 'print "hello world";'
perl -e "print 'hello world';"
echo "hello world" > a ; cat a
echo -n hello ; echo " world"
perl -e 'print "hello world"'
perl -e "print 'hello world'"
STR="hello world"; echo $STR
perl -E 'say "hello world";'
grep hello <<< "hello world"
perl -e'print "hello world"'
printf "%s\\n" "hello world"
perl -E 'say q[hello world]'
touch "hello world"; ls *\ *
printf '%s %s\n' hello world
printf "hello world" | cat -
VAR="hello world";echo $VAR
cat < <(echo "hello world")
perl -E 'say "hello world"'
date|sed s/.*/hello\ world/
touch 'hello world'; ls he*
echo "hello world" | cat -
printf '%s\n' 'hello world'
perl -e'print"hello world"'
printf '%s\n' "hello world"
printf "%s\n" "hello world"
printf "hello %s\n" "world"
printf "%b\n" "hello world"
echo|sed -n "i hello world"
touch 'hello world'; ls h*
echo "hello world" | cat -
printf %s\\n "hello world"
printf 'dlrow olleh' | rev
printf '%s %s' hello world
true && echo "hello world"
printf "%s %s" hello world
i='hello world'; echo $i;
printf '%s' "hello world"
cat <<EOF\nhello world\nEOF
printf -- 'hello world\n'
cat <(echo "hello world")
echo `echo "hello world"`
printf 'hello %s' "world"
printf "hello %s" "world"
printf "%s" "hello world"
printf '%s' 'hello world'
printf -- "hello world\n"
echo "dlrow olleh" | rev
echo 'hello world' | cat
echo ${PWD##*/}|tr _ ' '
printf "hello world\013"
printf '%s ' hello world
echo "hello world" | cat
/bin/echo 'hello world'
/bin/echo "hello world"
printf %s 'hello world'
cat - <<< 'hello world'
echo -e "hello world\n"
a='hello world';echo $a
printf -- "hello world"
printf "hello world" \n
printf %s "hello world"
cat - <<< "hello world"
cat - <<<"hello world"
printf "hello world\n"
echo -ne "hello world"
echo 'hello world'|cat
printf 'hello world\n'
echo hello world | cat
echo """hello world"""
echo -n "hello world"
cat <<< "hello world"
/bin/echo hello world
echo -e "hello world"
printf "hello world";
echo -n 'hello world'
echo -e 'hello world'
echo -n "hello world"
cat <<< 'hello world'
echo dlrow olleh|rev
(echo "hello world")
cat <<<"hello world"
echo "hello world" ;
'echo' 'hello world'
echo "hello world"
printf "hello world"
echo ''hello world''
printf 'hello world'
cat <<<'hello world'
echo ""hello world""
echo -n hello world
echo "hello world"
printf hello\ world
echo 'hello world'\
echo $"hello world"
echo 'hello world';
echo 'hello world '
echo "hello world "
echo $'hello world'
echo 'hello world'
echo -e hello world
echo "hello world";
echo "hello world"\
echo hello world""
echo ""hello world
echo "hello world"
echo hello\ world;
echo 'hello world'
echo hello world
'echo' hello world
echo hello world \
echo hello' 'world
echo hello world\
echo hello world;
echo hello \world
echo hello world
echo hello\ world
echo hello world
echo hello world

Of course there are a lot of quoting and space variations here as well. I will make a data-dump available soon with the responses for all challenges.

On the building and hosting side of things this is getting more than the usual tiny trickle of side-project internet traffic. So far I have tried to contain the entire thing in an AWS free tier account, it has worked out OK so far with a few hiccups here and there. Since several people have asked, I will share more details about how the site is put together in a future post, follow cmdchallenge on twitter or use rss.

Reddit front-page visualization

Wed, 21 Jan 2015 00:00:00 +0000

Using data from page-watch.com this shows a time-lapse of reddit overnight on January 20th taken at 5minute intervals. This excludes images which is why there are occasional jumps where advertisements are.

A new new blog

Thu, 18 Dec 2014 00:00:00 +0000

Before this year is over it’s time for a small redesign. I’ve replaced the old custom theme, water-iris with a new theme jarvican. Most of my updates these days are when I redesign the blog and I think I’m fine with that.

A new blog

Sun, 24 Nov 2013 00:00:00 +0000

Hello world! It’s been awhile and what better way to re-visit a blog then to do a re-design.

pelican with a custom theme
bourbon/sass - CSS compilation and general typography
bourbon/neat - For layout and responsive design

I’ve cleaned house and removed some old articles, kept some of the AVR hacks and hopefully will post updates more than once a year :)

Simple highscore tracking with Python / Flask

Mon, 17 Dec 2012 00:00:00 +0000

Below is a quick tutorial on how flask web framework might be used for simple highscore tracking. All of the source code the state game is now on github.

This code will accept an AJAX GET or POST at the end of each game so that scores can be tracked and read. Since the use-cases are simple it’s easy to write the tests first:

 class StateTestCase(unittest.TestCase):

 def setUp(self):
 self.db_fd, state.app.config['DATABASE'] =
 tempfile.mkstemp()
 state.app.config['TESTING'] = True
 self.app = state.app.test_client()
 state.init_db()

 def tearDown(self):
 os.close(self.db_fd)
 os.unlink(state.app.config['DATABASE'])

This above code will create an sqlite db before each test-case and remove it when completed.

 def test_empty_db(self):
 rv = self.app.get('/d')
 assert json.loads(rv.data) == []

 def test_single_entry(self):
 test_data = {
 'name': 'john doe',
 'score': 1234,
 'time': 1234
 }
 rv = self.app.post('/d', data=test_data)
 assert 'ok' in json.loads(rv.data)
 rv = self.app.get('/d')
 assert json.loads(rv.data) == [test_data]


 def test_multiple_entries(self):
 for score in range(0, 25):
 data = {'name': 'john doe',
 'score': score,
 'time': 1234}
 rv = self.app.post('/d', data=data)
 assert 'ok' in json.loads(rv.data)
 rv = self.app.get('/d')
 assert len(json.loads(rv.data)) == 10

These three test-cases cover basic functionality of the high-score tracker:

Ensure that if a GET request is made before adding entries an empty JSON array is returned.
If a single entry is added with a POST ensure that the same results are returned with a GET.
If there are more than 10 entries added ensure that we only get 10 back.

For the actual app there isn’t much more code than what was written for the tests.

 @app.route('/d', methods=['GET'])
 def get_scores():
 results = g.db.execute(
 'select name, score, time from hs order by score limit 10')

 q = [dict(zip(['name', 'score', 'time'], [item for item in row]))
 for row in results]

 return json.dumps(q)

Retrieve the top 10 scores, return the results as JSON.

 @app.route('/d', methods=['POST'])
 def post_score():
 data = request.form
 if (not re.match('[\s\w]{1,20}$', data['name'])):
 return json.dumps({'error': 'invalid name'})
 if (not re.match('[0-9]+\.?[0-9]*$', data['time'])):
 return json.dumps({'error': 'invalid time'})
 if (not re.match('[0-9]{1,15}$', data['score'])):
 return json.dumps({'error': 'invalid score'})

 g.db.execute('insert into hs (name, score, time) values (?, ?, ?)',
 [data['name'], data['score'], data['time']])
 g.db.commit()
 return json.dumps({'ok': 'success'})

Receive a new score, do some basic input validation and insert the new high score entry.

Of course it’s easy to inject fake high scores with something like this, abuse it, cheat, etc. But in this case it’s just for fun.

The State Game

Sun, 22 Jul 2012 00:00:00 +0000

Below is a recent weekend programming project that I did to play with canvas. Who doesn’t like a geography game?

It uses a simple svg image for the map with a canvas overlay to detect state clicks. On the back-end it uses a very short python script with web.py. Most people who tried it out found the rotation to be annoying and difficult; for me it makes it more interesting :)

Musical Ms. Pacman Candy Tin Hack

Sun, 04 Dec 2011 00:00:00 +0000

Here is a weekend hack that uses the midi to AVR conversion script and library. With xmas fast approaching I thought it would be fun to convert a pacman candy tin to an xmas ornament and have it play music. Below is the result, pressing a button on the tin will cycle through three Ms. Pacman songs converted from midi files found online.

Custom musical greeting card for less than $5

Sun, 20 Nov 2011 00:00:00 +0000

“You’ve got a friend in me” played on an attiny85 and two speakers converted from a midi version of the song:

Blinking motion sensor / Mario Brothers Candy Tin

Tue, 15 Sep 2009 00:00:00 +0000

With an attiny13A, an IR motion sensor, and a candy Mario Brothers mushroom tin, this project turns them into a USB powered IR sensor that also blinks and sings.

Singing Marioman

Sun, 24 May 2009 00:00:00 +0000

Using an attiny13A and some leds and a speaker create a musical marioman.

jarv.org

My RSS client has an MCP server

Why MCP + RSS

Implementation

Filtering YouTube Shorts from RSS Feeds

Setting Up a Filter

How It Works

Neovim Configuration for 2026

Why I Switched

My Configuration and Workflow

What Changed in the Migration

Customizations

Additional Customizations

Plugins I No Longer Need

Thoughts on the Switch

Monitor a single file on GitHub/GitLab with RSS

The wild vibe

Using Mise and 1Password for secrets

Thinking about Trade Wars trading mechanics

Accounts and Centralized Survey View for FlyEmoji

A Trade Wars 2002 inspired pocket game

The universe

Development

Front-end

Back-end

What next?

A Drunkard's Walk with hiding spots

Neovim Configuration for 2025

My Configuration and Workflow

Customizations

Notes on Cookie partitioning

What is Cookie Partitioning?

How are Browsers Using the Partitioned Attribute?

Why is the partitioned Attribute Necessary?

Using Mise for All the Things

Environment and Secrets

Tasks and Watching Files on Changes

Centering an emoji character on a button

Using Strict for the SameSite attribute

Creating a ring buffer using Go Channels for Server-Sent Events

How are you feeling today?

A channel based ring buffer in Go

First issue: Concurrent calls to Send()

Second issue: Concurrent reads and calls to Send()

Benchmarks

Conclusion

The joys of self-hosting and tiny side-projects

Tiny side-projects

Self-hosting

Debugging a deadlock in Go

Background

View the stack traces of all Go routines

Fixing the deadlock

One huge shell script

My approach to managing the single VM deployment

The big shell script

Using Docker and a single VM for side-projects

Rebuilding my single VM from scratch

Cool Caddy config tricks for your self-hosted domain

HTTPs made easy

Creating a site that echos an IP address

Creating a subdomain that echos a HTTP status code

Forcing plain HTTP

Running a command with a timeout in Go

From serverless to server

Update on the Go rewrite

Go rewrite of cmdchallenge

Does Serverless make sense for a side project?

Serverless Architecture

Free Cloud Services

AWS - $2/month

Breakdown of a typical monthly bill

GCP - $6/month

VM Comparison for shared-core or burstable instances

Notes

So, is it worth it?

Moving to HTTP from REST for the AWS API Gateway

HTTP API Gateway

Caching Responses with the HTTP API Gateway

Displaying the percentage of hosts completed during a rolling Ansible Deploy using serial

How are Browsers Using the `Partitioned` Attribute?

Why is the `partitioned` Attribute Necessary?