Yay I fixed a bug, that I made!

Thanks Luke!

_layouts/default.html

@@ -18,10 +18,14 @@
     <link rel="apple-touch-icon" href="/img/theme-colors/orange.png">
 <meta property="og:locale" content="en" />
 <meta property="og:type" content="website" />
-<meta property="og:title" content="Terminal">
-<meta property="og:description" content="A simple, retro theme for Hugo" />
+<meta property="og:title" content="{{page.title}}">
+<meta property="og:description"
+                            content="
+     {% if page.description %}{{ page.description }}
+     {% else %}{{ site.description }}
+    {% endif %}"/>
 <meta property="og:url" content="/" />
-<meta property="og:site_name" content="Terminal" />
+<meta property="og:site_name" content="brokenco.de" />
 <meta property="og:image" content="/img/favicon/orange.png">
 <meta property="og:image:width" content="1200">
 <meta property="og:image:height" content="627">

_posts/2023-07-10-dynamically-forwarding-ports-again.md

@@ -0,0 +1,33 @@
+---
+layout: post
+title: Dynamically forwarding SSH ports with "commandline disabled"
+tags:
+- ssh
+- bsd
+- linux
+---
+
+
+I frequently use SSH for accessing one of the many development workstations I
+use for work, which includes developing network services among other things. A
+couple of years ago I wrote about this hidden gem in `ssh` which allows
+[dynamocaily forwarding ports](/2021/05/16/dynamically-forward-ssh-ports.html). 
+This handy little feature allows dynamocailly adding local port forwards from within an already running SSH session. Recently however this feature has stopped working properly, emitting `commandline disabled`.
+
+It turns out that this is due to a backwards incompatible change which [OpenSSH released in 9.2](https://www.openssh.com/txt/release-9.2) earlier this year:
+
+> ssh(1): add a new EnableEscapeCommandline ssh_config(5) option that controls
+> whether the client-side ~C escape sequence that provides a command-line is
+> available. Among other things, the ~C command-line could be used to add
+> additional port-forwards at runtime.
+
+The reason for this change is to support some sandboxing use-case which I don't entirely understand but also don't need, so I needed to add the following option to my host entries in `~/.ssh/config`:
+
+```
+Host foobar
+    Hostname 172.16.1.1
+    EnableEscapeCommandLine yes
+```
+
+
+This can also be configured on the command line with `-o EnableEscapeCommandline=yes`. Happy port forwarding!

_posts/2023-07-26-rust-re-export.md

@@ -0,0 +1,71 @@
+---
+layout: post
+title: "Why we re-export symbols from other libraries in Rust"
+tags:
+- rust
+- opinion
+---
+
+Dependency management in the Rust ecosystem is _fairly_ mature from my perspective, with [crates.io](https://crates.io), Cargo, and some cultural norms around semantic versions, I feel safer with dependencies in Rust than I have in previous toolchains. It's far from perfect however, and [this question](https://mastodon.social/@davidpdrsn/110780897434598935) helps highlight one of the quirks of how Rust dependency management does or does not work, depending on your perspective:
+
+> What is it that makes Rust users want libraries to re-export stuff from other
+libraries?
+>
+> I often get requests for axum to re-export stuff from hyper, time, or other
+common crates. Why? Just “cargo add hyper” and you’re good to go. Hyper is in
+your crate graph regardless.
+>
+> I also often get feature requests for the few types axum does re-export so it
+does confuse some. That’s why I’m reluctant to just re-export everything.
+
+I started writing up a reply in Mastodon but then I noticed that my words were
+approaching the 500 character limit and perhaps this topic wasn't
+microbloggable! I help maintain the [deltalake](https://crates.io/deltalake)
+package for Rust and we **do** re-export a number of libraries, such as
+[arrow](https://crates.io/arrow) of which I am a strong supporter.
+
+The biggest motivation for re-exporting is to preserve ABI compatibility in our
+interfaces. For some crates your transitive dependencies may be masked entirely
+from the end-user, for example if I pull in the `regex` crate I'm typically
+just using it for regular expressions inside my crate and not exposing an
+interface which takes a `regex::Regex`. The ABI is safe from transitive version
+changes of that crate. If however my crate exposes an API which is dependent on
+a transitive dependency then I can have problems with version mismatches. Such
+is the case with `arrow` in [delta-rs](https://github.com/delta-io/delta-rs),
+which exposes `arrow_array::RecordBatch`. There is a much larger chance of ABI
+incompatibilties between a transitive version of arrow needed by the
+`deltalake` crate and what the consuming project may specify. This is
+exacerbated in our case because _another_ transitive dependency of `deltalake`
+specifies a dependency on `arrow`: [datafusion](https://crates.io/datafusion).
+
+That means that the user, `deltalake`, and `datafusion` all have to agree on
+the same version of `arrow` for types to properly interoperate between API
+calls.
+
+But it gets worse!
+
+The Rust community generally seems to follow semantic versioning, but that
+doesn't mean anything about the releases, just the version numbers used for
+them. I can make major breaking API changes every one of my 0.x.x releases, or
+in the case of `arrow` and `datafusion` I can just increment the major version
+every release.
+
+By re-exporting symbols from those two crates, downstream users of the
+`deltalake` package will have a stable `RecordBatch` type ABI to work with for
+every release, and can _largely_ ignore non-API breaking changes such as
+struct layout changes, etc.
+
+I am still mixed on whether _all_ types from other crates exposed in my APIs
+should be exported. I think there is benefit to doing so for faster moving
+dependencies. In essence:
+
+* `arrow`, moving fast, better user experience to re-export
+* `url`, moves slow, very mature, not really needed to re-export.
+
+
+The judgement call I am typically making is whether this would make my life
+easier as a downstream consumer of the crate. It's not that much effort of
+maintenance burden to `pub use` something in a crate if that's convenient.
+
+
+

_posts/2023-10-20-frustrations-with-nomad.md

@@ -0,0 +1,70 @@
+---
+layout: post
+title: "Hashicorp Nomad, almost but not quite good"
+tags:
+- opinion
+- nomad
+---
+
+My home office has grown in size and for the first time in decades I believe I
+have a _surplus_ of compute power at my disposal. These computational resources
+are not in the form of some big beefy machine but a number of smaller machines
+all tied together by a gigabit network hiding away in a server cabinet. The big
+problem has become how to effectively utilize all that computational power, I
+turned to [Nomad](https://developer.hashicorp.com/nomad) to orchestrate
+arbitrary workloads on static and ephemeral (netboot) machines. As the title
+would suggest, it's almost good but it still falls frustratingly short for my
+use-cases.
+
+I started investigating Nomad because Hashicorp pulled out a big licensing
+foot-gun and pulled the trigger, changing to a non-open source license for all
+of their projects, Nomad included. Unlike it's friend Terraform, whose
+community rightfully revolted and created [OpenTofu](https://opentofu.org/), no
+such community seems to exist for Nomad. Extension and integration points are
+the raw materials necessary to build a blossoming third-party community, and
+without something akin to Terraform's providers and modules, there simply isn't
+a common way for Nomad users to share patterns. Nomad has equivalent to [Helm
+charts](https://helm.sh), and the user community is worse off for it.
+
+While Nomad does technically have a plugin architecture, it is poorly
+documented and seems to only exist for task drivers (e.g. `docker`, `exec`,
+`pot`). The vast majority of users are not going to need to write new task
+drivers, but I can imagine a ripe opportunity for something akin to Terraform
+modules for shared workload definitions in Nomad. It just doesn't seem to have
+ever materialized.
+
+The roughness around the edges are many, but some of the ones bugging me this week are:
+
+* A glitchy web UI that rivals old [Jenkins](https://jenkins.io) in its ability
+  to hide the common user flows behind a too many clicks.
+* A description language that doesn't "cascade" properly. Some blocks can be
+  configured at the `job`, `group`, and `task` level. Others can be configured
+  at the task level, like `env`, leading to redundant definitions across every
+  `task` in a job.
+* Secrets integration is through Hashicorp Vault or ... nothing. Which means I
+  guess I'll just shove things into environment variables and hope nobody notices.
+
+I do _kind of_ like Nomad though, which makes this all the more frustrating.
+Most of what I need to do are ad-hoc on-premise compute workloads, some of
+those workloads fit "cleanly" into Docker containers, others do not. Nomad does
+meet that lovely middle ground of allowing me to orchestrate both. The support
+for `service` (run a web server), `batch` (run a nightly job), and `sysbatch` (run a
+management task on a slice of nodes) task types also covers a very useful
+spectrum of my needs.
+
+Despite all the really interesting qualities of Nomad it is a perhaps overly
+complex piece of software which never lent itself to strong open source
+contributions or community engagement. With the change in its
+[license](https://helm.sh/docs/topics/charts/) I fear it's going to fall
+further behind and ultimately be forgotten in a sea of ambitious but ultimately
+mismanaged software projects.
+
+
+Returning to the needs that led me to adopt Nomad in the first place, they're
+still not entirely met but I'm a bit lost on options to orchestrate workloads that _could_ fit in Nomad really well.
+
+Yes, the rough edges of Nomad are frustrating. What is much more frustrating is
+that I can see how Nomad could be a _great_ piece of software, but because of
+social factors rather than technical ones, will never actually get there.
+
+

_posts/2023-11-12-interface-already-exists.md

@@ -0,0 +1,77 @@
+---
+layout: post
+title: "Solving a FreeBSD Jails issue: interface already exists"
+tags:
+- freebsd
+- jails
+---
+
+For a long time after I rebuilt my jails host, I could not restart a certain
+number of jails due to an "interface already exists" error. For the life of me
+I could not make sense of it, The services running in the jails were useful but
+not _required_ so I put off tinkering with it. I thought that I would magically
+stumble into the solution in my sleep or something equally silly.
+
+```
+watermelon# service jail start gitea
+Starting jails: cannot start jail  "gitea":
+ifconfig: interface epair14 already exists
+jail: gitea: ifconfig epair14 create up: failed
+.
+watermelon# service jail stop gitea
+Stopping jails:.
+```
+
+What perplexed me about this issue is that I would run `ifconfig epair14a`
+after the failure to start the jail, and the interface would be there. "Surely
+this must be a FreeBSD bug!"
+
+The "eureka!" happened earlier today, not while I was sleeping, but rather
+while I was solving other problems. "I bet there's something fishy in the
+configuration, I should just rewrite it" I thought to myself. Most esoteric
+bugs are not bugs with the compiler, libraries, or operating systems. Usually
+they're the user doing something slightly stupid and not realizing it.
+
+
+My jail configuration (`/etc/jail.conf`) resembled the following:
+
+```conf
+gitea {
+        $id = "14";
+        $ip_addr = "10.10.10.${id}";
+
+        vnet.interface = "epair${id}b";
+
+        exec.prestart = "ifconfig epair${id} create up";
+        exec.prestart += "ifconfig epair${id}a up descr vnet-${name}";
+        exec.prestart += "ifconfig $public_bridge addm epair${id}a up";
+
+        exec.start = "/sbin/ifconfig epair${id}b ${ip_addr}";
+        exec.start += "/sbin/route add default ${public_gw}";
+        exec.start += "/bin/sh /etc/rc";
+
+        exec.prestop = "ifconfig epair${id}b -vnet ${name}";
+        exec.poststop = "ifconfig ${public_bridge} deletem epair${id}a";
+        exec.poststop += "ifconfig epair${id}a destroy";
+}
+```
+
+Looking at the block and comparing it to other _functional_ jails, I saw something missing: a `vnet;` declaration:
+
+```diff
+--- jail.conf   2023-11-12 20:09:03.028010000 -0800
++++ /etc/jail.conf      2023-11-12 19:59:02.867271000 -0800
+@@ -230,6 +230,7 @@
+        $id = "14";
+        $ip_addr = "10.10.10.${id}";
+
++       vnet;
+        vnet.interface = "epair${id}b";
+
+        exec.prestart = "ifconfig epair${id} create up";
+```
+
+
+Sometimes you have to just walk away from a problem for a bit, but yeesh was that a silly one!
+
+