Nat Torkington

Simon Willison's Weblog

2026-03-16T00:00:00Z

The inspiration for this site’s redesign. Simon builds his blog with a custom Django app and publishes a mix of entries, links, and notes.

]]>

Nat's 2022 Technical Link Pile: C# and .NET

2022-12-31T00:00:00Z

See the Intro for context. (We’re hiring WinForms seniors! Mail me: nat@ontempo.co.nz)

[20221224] How to Structure Your .NET Solutions: Architecture and Trade-Offs – what it says on the box.

[20221224] DocFx – generate static documentation site from embedded XML documentation.

[20221224] .NET Advent Calendar – most are interesting, though they’re so diverse it’s the rare person who’d be interested in all.

[20221210] ComputeSharp – run C# on the GPU.

[20221210] NLua – .NET Lua interpreter.

[20220719] Lambdas in C# – quick overview of syntax.

[20220525] Winforms UI Refactoring: A Case Study – brief, just about using their dependency mapping tool to figure out coupling.

[20220525] Moving Your Xamarin Library to MAUI – rewrite/refactor/bifurcate; understand your dependencies; upgrade CSPROJ; implement single-project multitargeting; fix problems caused by dependencies or .NET version shifts; keep same filenames to ease merging.

[20220519] Life Cycle Events in MAUI – Running, Not Running, Deactivated, Stopped are the states, and there are events you can receive when you move between the states.

[20220330] Parameter Null Checking in C# 11 – !! on a parameter name in the definition automatically performs null checking.

[20220330] Polly – a .NET resilience and transient-fault-handling library that allows developers to express policies such as Retry, Circuit Breaker, Timeout, Bulkhead Isolation, Rate-limiting and Fallback in a fluent and thread-safe manner.

[20220324] Raw String Literals in C# – stolen from Perl which stole it from the Bourne shell, I think.

[20220323] Populating an Array with the Same Value – for() loop is slow but does deep copy; Enumerable.Repeat is faster and lets us say how many copies of a value we get; Array.Fill is fastest but has no option for partial fill.

[20220323] Params in C# – get function parameters in an array instead of naming them in the function definition.

[20220323] Accessible MAUI Apps - Screen reader assistance; Colors and contrast; Size; Keyboard navigation; Writing or icon representation.

[20220323] RestSharp – HTTP client library with automatic serialization and deserialization, request and response type detection, variety of authentications.

[20220323] DI in AWS Lambda functions in C# – building an API with Lambda functions in C#, using dependency injection which apparently required some research.

[20220317] Telerik Reporting – huh, they have their own SSRS type thing? Now it’s in React as well.

[20220317] C# DateTime Formats – far more than you ever wanted to know about formatting datetimes.

[20220317] Inequality Operators – “is not” requires constant expression on RHS, but does boxing and understands inheritance. “!=” doesn’t need constant expr RHS, but will fail if types are different and doesn’t understand inheritance. Consequently errors are different (cannot be applied vs cannot convert). “!=” can be overloaded, “is not” can’t be.

[20220314] 10 .NET Packages Every Developer Needs to Know – they are:

Quartz.Net - scheduling
Serilog - logging library, lots of pre-written integrations
Automapper - copy data from one object structure to another, groks conventions like property names
Polly – allows developers to express policies when handling API calls, like retry, fallback, rate-limiting etc.
FakeItEasy – creates fake objects, mocks, stubs, and more.
XUnit – unit testing framework by authors of NUnitv2
FluentValidation – strongly-typed validation rules.
FluentAssertions – extensive set of extension methods to specify the expected outcome of a unit test.
Autofac – IoC container for .NET applications, with API and module system.
Noda Time – date and time API. E.g., date and time in a specific timezone.
SharpZipLib – compression library.
HtmlAgilityPack – HTML parser in C#.
MailKit – SMTP, IMAP, POP3.

[20220225] Const vs ReadOnly vs Static ReadOnly – const = compile time; readonly = can only set value in constructor; static readonly = only static methods can change the variable.

[20220217] WASM WinForms – nuget package that you can install and use your codes unchanged and run it on your browser!

[20220213] Bionic Framework – using the Ionic Capacitor framework for xplat apps (which comes highly recommended) but with C# and Blazor.

[20220203] Guard class in Diagnostics toolkit – simplify parameter checking

[20220203] int.Parse(), Convert.ToInt32(), and int.TryParse() – behaviour on non-int strings. Parse() throws exceptions, ToInt32() returns 0. TryParse() returns two values, success and parsed value.

[20220128] .NET Analyzer rules in Visual Studio

[20220201] .NET testing with xUnit

[20220201] Developing and shipping a working .NET 5 application including deployments, devops, migrations, and Linux

[20220201] Platform-specific code in MAUI (video)

[20220201] Building an event-driven .NET application

[20220201] DateTime to ISO8601 in C#

[20220201] Upgrading Xamarin Forms to .NET MAUI

[20220201] C# “is” and “as” operators

“is” compares types
- foo.GetType() == typeof(SomeClass) // exact only
- foo is SomeClass // understands inheritance
“as” converts types
- var x = foo as SomeClass; // x is of type SomeClass, or is null if invalid conversion

[20220201] Records in C#

[20220201] C# structs vs records

[20220201] .NET Architecture Guides

[20220201] Installing .NET Maui on macOS

[20220201] Advanced Data Binding

[20220201] Deep C# book

]]>

Nat's 2022 Technical Link Pile: Dev, Architecture, APIs

2022-12-31T00:00:00Z

See the Intro for context.

[20221231] Temporal.io – With Temporal, you write your various complex, potentially long-running actions as normal, synchronous code, which Temporal “magically” turns into distributed, asynchronous, automatically retried, idempotent jobs. The design is really elegant, and removes all of the hard work from writing code that must keep things in sync and handle failures through queueing and retrying. (as described in this Hacker News comment) Also: Temporal jobs/tasks are called workflows because the code is effectively translated into workflow steps—i.e. code progress is made by workers, and each step is persisted so that if the worker dies, you don’t lose anything—another worker picks up in the exact same place in the code, with all local variables and threads intact. It also provides helpful built-in features like retries and timeouts of anything that might fail, like network requests, and the ability to “sleep” for arbitrary periods without consuming threads/resources (other than a timer in a database that is automatically set up for you when you call `sleep(‘1 month’)`).

[20221231] ULIDs and Primary Keys – omg there’s an RFC (4122) for UUIDs. There are variants. Author of this blog post likes ULIDs. They contain more randomness and a straightforward structure at the cost of not explicitly exposing the version, variant, or monotonic counter. UUIDs are also the perfect choice if you’re writing some apocalypse-scenario software (assuming you have a working computer) as they can continue to be generated until 10,889AD compared to UUIDv7s measly 4,147AD death date.

[20221223] How Pitfall Builds Its World – Crane used a single byte to represent the layout of the current room.

[20221223] Docker Without Containers – wasm managed with the same tools as Docker containers.

[20221221] Taming Names in Software Development – balancing short v long, conventional v distinctive; make your own pattern for names (eg adjective_noun_unabbreviatedUnits giving minimum_messageLength_bytes). A good read.

[20221221] You Might Not Need a CRDT – browsers are inherently not peer-to-peer. To run an application from the web, you connect to a server. Since the server is centralized anyway, we can have it enforce a global ordering over the events. That is, every replica receives events in the same orer. With this, we can sidestep the need to pay the CRDT complexity tax.

[20221221] Abstractions are Expensive – At companies with huge engineering forces, abstraction management is what a lot of them spend their time on. Often, these engineers are actually the most “productive” in terms of money saved - infrastructure projects tend to result in 8-9 figure savings or unique capabilities, and performance engineering (another form of abstraction alignment) frequently has 8 figure returns per engineer. Another large group of engineers is in charge of making sure that the old abstractions don’t break and crash the entire system.

[20221221] Webhook Architecture Design – awful verbatim transcription of someone with a lot of repetition and “mm hmm”, but the content is good.

[20221221] Event-Driven APIs with Webhook and API Gateway – part II of Building event-driven API services using CQRS, API Gateway and Serverless.

[20221211] GitPod – Spin up fresh, automated dev environments for each task, in the cloud, in seconds.

[20221210] Algorithms I Prototyped – CRDT Fractional Indexing; CRDT Tree-Based Indexing; CRDT: Mutable Tree Hierarchy: Sync a tree hierarchy between peers. Nodes in the tree retain their identity even if they are reparented somewhere else in the tree; Logarithmically-Spaced Snapshots: Keep periodic snapshots of something, but keep more snapshots of recent events and fewer snapshots of long-ago events.

[20221210] Server Sent Events for Shopify Dashboard – Spotify’s Black Friday dashboard built with SSE.

[20221126] Domain Driven Design Intro – very gentle high-level introduction that makes sense to me. Common vocab used in requirements, implementation, tests, bug reports, etc. And an explication of entities (which have a lifetime) vs values (which just exist as a number or string), and aggregate entities (e.g., an order + orderlines, where orderlines have no meaning outside an order). DDD appears to full of guidelines like: consistency rules apply inside an aggregate, an aggregate has a single external identifier, etc.

[20221126] StackStorm – event-driven automation for auto-remediation, incident responses, troubleshooting, deployments, and more for DevOps and SREs. Includes rules engine, workflow, 160 integration packs with 6000+ actions and ChatOps. If This Then That for Ops. (I’m interested in every workflow engine)

[20221121] You Will Never Fix It Later – outlines the reasons why the probability a hacky solution will be fixed decreases with its age. For example: The further removed you are from the point where the code was written, the less you understand it. You remember less about what it does, what it’s supposed to do, how it does it, where it’s used, etc. If you don’t understand all its intended use cases, then you’re not confident that you can test all of them. Which means you’re worried that any change you make might break some use case you were unaware of. (yes, good tests help, but how many of us trust their test suites even when we’re not very familiar with the code?)

[20221121] The Modern Observability Problem – pithy summary. To resolve problems in a large microservice system: We must be able to find out why they have occurred; For this, our system must be observable; To be observable, a system needs to be instrumented such that the code emits telemetry, which is typically logs, traces and metrics; This telemetry must be sent to a backend that supports joining up that telemetry data and answering questions about the system’s behaviour.

[20221119] The Perfect Commit – a single commit that contains all of the following: The implementation: a single, focused change; Tests that demonstrate the implementation works; Updated documentation reflecting the change; A link to an issue thread providing further context.

[20221119] Build a Modular Monolith First – what it says on the box. [I]t is all about breaking up the system into modules, and then combining those modules into a monolith for deployment.

[20221107] Functional Testing – exercise public endpoints without mocking.

[20221107] Structured Errors in HTTP APIs – RFC-7807 provides a standardised error structure.

[20221107] Recovering From Crashes with Safe Mode – when a feature flag causes your program to crash, add a safe-mode to detect the crashes and rollback to a safe feature-flag state.

[20221107] Code Like an SRE – it’s like getting an SRE drunk and saying “tell me all the stuff you learned the hard way”.

[20221107] Content-Defined Chunking – very readable explanation of how to incrementally backup or synchronise files without transferring data you don’t have to.

[20221007] Mobile Developer Experience at Slack – putting dollar values on lousy build experiences is clever.

[20221107] Demystifying Software Architecture Patterns – commonalities across Clean, Hexagonal, and Onion architectures: centralized business rules; application specific rules (operation-specific orchestration and related logic); source code dependencies should only point inward; isolation of layers. Core business rules should be independent of how you persist it, how you expose it, and which framework you’re using.

[20221102] Git Bisect with Automated Tests – use a new test to zero in on the commit that failed.

[20221029] Local First – developing with CRDTs and the expectation that the local machine is always true, and the truth will be reconciled with others when connected again.

[20221008] Fundamentals of Control Theory – free ebook. See also Brian Douglas’s videos on YouTube, Brian Douglas’s MatLab videos on Control Theory, Steve Brunton’s videos, and Christopher Lum’s videos, and Kat Kim’s videos.

[20221008] GruCloud – Generate Javascript code from live infrastructures; Deploy, destroy and list resources on various clouds; Share and compose infrastructure; Automatic resource dependencies management.

[20221003] Lessons from Deploying Microservices for a Large Retailer – no containers, NoSQL, or even the cloud.

[20221003] How we Store and Process Millions of Orders a Day – Grab’s architecture.

[20220922] How to Deal with Money in Software – overview of possible representations, better recommendations, and an addendum of tests.

[20220922] Webhooks.fyi – a directory of webhook providers and a collection of best practices for providing and consuming webhooks.

[20220922] Reducing Duplicate Code in our Applications Using HATEOAS – includes a nice explanation of Hypertext As The Engine of Application State.

[20220805] Bad Microservices – modularise your monolith.

[20220804] Why You Can’t Guarantee Webhook Ordering – the discussion was interesting too. Polling, queueing, events, and call-for-response all have their advocates.

[20220803] Use One Big Server – frequent topic of discussion. One database server (or cluster that lets you separate reads from writes) can get you a long way, and the hardware cost of vertical scaling beats the complexity price of distributed data.

[20220803] The Five Ways to Deploy Microservices – the five ways of running microservices: (1) Single machine, multiple processes: buy or rent a server and run the microservices as processes. (2) Multiple machines, multiple processes: the obvious next step is adding more servers and distributing the load, offering more scalability and availability. (3) Containers: packaging the microservices inside a container makes it easier to deploy and run along with other services. It’s also the first step towards Kubernetes. (4) Orchestrator: orchestrators such as Kubernetes or Nomad are complete platforms designed to run thousands of containers simultaneously. (5) Serverless: serverless allows us to forget about processes, containers, and servers, and run code directly in the cloud.

[20220728] You Don’t Need a Monolith – lots of arguments against, mostly “you get all this power and complexity but you probably don’t need or want it.”

[20220719] Improve Your Application Testing – Use template system allocation scripts; Automate resource allocation and deallocation; Perform regular refreshes of test data.

[20220718] 14 Software Architecture Patterns – circuit breaker, client-server, command query responsibility segregation, controller-responder, event sourcing, layered, microservices, model-view-controller, pub-sub, saga, sharding, static content hosting, strangler, throttling.

[20220718] 12 Ways to Prepare Your Monolith for Microservices – Ensure you know what you’re getting into; Make a plan;Put everything in a monorepo; Use a shared CI pipeline; Ensure you have enough testing; Install an API Gateway or HTTP Reverse Proxy; Consider the monolith-in-a-box pattern; Warm up to changes; Use feature flags; Modularize the monolith; Decouple the data; Add observability.

[20220624] Serverless Reference Architectures – for AWS but will work in general.

[20220624] Serverless Microservices Patterns for AWS – The following 19 patterns represent several common microservice designs that are being used by developers on AWS.

[20220624] 7 Ways to Fail at Microservices – distributed != decoupled. (A conference talk, on YouTube)

[20220624] Infrastructure Efficiency – what to worry about at each stage of startup life.

[20220529] Learnings from 5 Years of Tech Startup Code Audits – simple outlasted smart; never deserialise untrusted data; flaws in business logic are guaranteed to affect the business; Almost no one got JWT tokens and webhooks right on the first try; Our highest impact findings would always come within the first and last few hours of the audit. (discussion)

[20220527] Refactoring and Optimizing a High-Traffic API at PayPal – identifying bottlenecks, segmenting, etc.

[20220526] Advice from 10 Years of Software Engineering – good advice. I particularly liked: Be a caretaker, rather than an owner; Write code specifically for the problem at hand, but try to spot places where you can afford to make it a little generic; Defining what is “done” is time-saving because it helps you estimate the effort required, plan for development, and avoid unnecessary revisions later; A single large release may be divided into a series of lower-risk well-understood rollouts; Coordinate reviews for the design doc and compare the design as it evolves with the original doc to verify that all the relevant constraints are being addressed.

[20220526] How We Deploy to Production Over 100 Times/Day – optimise the developer workflow for rapid delivery, and this leads to a reduction in risk too; Small revertable changes; change management process is surprisingly light on human touch points; invest heavily in out-of-the-box guardrails, monitoring, and auditing; We associate “tiers” with each service (how critical they are) and warn when PRs make changes to critical ones. This helps engineers get a sense of the relative risk of a change; We bias towards a simple and opinionated user experience, over configurability; The workflow is consistent and familiar; An unintended side effect of reducing friction from the deployment process is we see too many changes being deployed to staging for testing. This increases the number of manual steps engineers take in order to test out changes.

[20220524] Real World Event-Driven Architecture – examples of solving problems with events.

[20220524] The Balance has Shifted Away from SPAs – for simple things. Some scenarios still need MPAs.

[20220524] Application Holotypes – short on enterprise code, but still interesting.

[20220524] Quick Fixes to your Code Review Workflow – Simplify the decision of when to review; Distinguish comments from requests; Use pair reviews; Use more and more junior reviewers; Teach people how to do code review; Designated second reviewers; Decide what review is actually for; Adopt a collaborative attitude; Actually figure out what’s wrong with your code review.

[20220524] Simple Software Things That Are Actually Very Complicated – nothing is easy! If the browser - or in other environments, the operating system - can do something for you, chances are you definitely want to let it handle that for you. Otherwise you’ll learn the hard way just how much work goes in to it.

[20220524] The Tyranny of ‘What if it Changes?’ – it’s speculation, divorced from reality, and used as a conversation ender.

[20220519] Stop Aggregating Away the Signal in Your Data – discusses three alternatives to aggregation: Rearranging the data to compare “like to like”; Augmenting the data with concepts that matter, like “summer” vs. “winter” or data-defined categories like “high” or “normal” energy usage; Using the data itself as context by splitting the data into “foreground” and “background,” so the full dataset provides the context necessary to make sense of the specific subset of the data we’re interested in.

[20220504] Philosophy of Software Design – summary of Ousterhout’s book. VERY good.

[20220504] Web Release at Massive Scale – Facebook employees dogfood Facebook changes, which functions as QA.

[20220504] Spotify System Architecture – once over lightly, but it’s got all the rectangles in the architecture diagram.

[20220504] Guide to Protecting Your APIs with OAuth2 – that’s part 1, see also Part 2 The Authorization Code grant (in excruciating detail).

[20220504] Max Connections – For a server listening on a port, each incoming connection DOES NOT consume a port on the server. The server only consumes the one port that it is listening on.

[20220504] Design for Scale and High Availability – a look at Google’s infrastructure, with insights into team composition and how they view incidents.

[20220502] 12 Factor Apps in 2022 – We end up with an “app” level and a “service” level and we have to be careful when considering the original text, since it talks only about the app level, but some of its concerns now reside at the service level.

[20220502] Distributed Systems Shibboleths – nice explanation of the building blocks of distributed systems, with a smidge more meat than most overviews.

[20220421] The Big TDD Misunderstanding – Originally the unit in “unit test” did not refer to the system under test but the test itself. Meaning the test can be executed as one unit and does not depend on other tests to run upfront. Huh.

[20220421] Microservices Observability Design Patterns – Health check API; Log aggregation; Distributed tracing; Exception tracking; Application metrics; Audit logging.

[20220421] 10 REST Commandments – (1) use JSON; (2) use the right HTTP methods; (3) use nouns instead of verbs, plural instead of singular; (4) use SSL, use authorisation, don’t send data you don’t need, use an API gateway to block DDoS etc; (5) version your API; (6) be consistent; (7) use HTTP status codes, and a universal error message structure; (8) handle complex processes inside the API, not requiring the cooperation of the client; (9) optimise database, cache, send only data that’s needed, enable compression; (10) document and otherwise help new users of the API.

[20220421] When Feature Flags Do and Do Not Make Sense – Feature flags are a poor man’s alternative to binary rollbacks, and they definitely aren’t a substitute for having a great automated test suite and a robust QA process. And each feature flag immediately doubles the universe of corner cases that your programmers have to understand, and your code is required to handle.

[20220421] Narrow Waists – narrow waists are interop systems that let N systems talk to M systems with NxM translation systems being necessary. In distributed systems, types are local illusions. Big systems can’t be upgraded atomically. They’re often running a mix of inconsistent schema versions. When models and reality collide, reality wins.

[20220421] Simple Architectures – The cost of our engineering team completely dominates the cost of the systems we operate. […] By keeping our application architecture as simple as possible, we can spend our complexity (and headcount) budget in places where there’s complexity that it benefits our business to take on.

[20220406] Cohesion and Coupling with Examples – very good.

[20220404] Authorisation in a Microservices World – Fine-grained authorization in microservices is hard. Definitely not impossible, but hard. You would expect that a more standardized, all-around, full-proof solution is out there, but I am afraid there isn’t. It’s a complex matter and depending on what you are building, implementation varies.

[20220404] BBC’s Year of Serverless – no outages caused by the serverless platform. They did notice noisy neighbours (other customers on the VM) reducing BBC’s performance on hour and quarter hour.

[20220404] Networking at Slack Retrospective – again I say: networking is HARD. To make matters worse, applications interpret these variables differently too. For example, Ruby and Golang support CIDR blocks in the `no_proxy` variable while Python does not.

[20220404] On Scalable Software – a mini-bootcamp on scaling with distributed systems. Also: If we know that the workload is going to increase, we should go for a scalable design up-front. But if we are not sure of what the future looks like, a performant but non-scalable solution is a good enough starting point.

[20220404] Distributed Ingestion at Sentry – they build a CDN but to accept uploads, not cache downloads. This caught my eye: Have you ever seen unexpected 50X responses from your reverse proxy servers with no obvious explanation? You might be a victim of inconsistent idle timeouts between your upstream and downstream applications.

[20220330] Enterprise Platform API Governance – common sense behind the big words:

Info: Ensure that there are title, description, and other essential information properties.
Versioning: Require a standard semantic or date-based versioning applied to each API.
Operations: Make sure each individual operation has a summary, description, and id.
Parameters: Standardize the format of parameter names, and all have descriptions.
Responses: Push for a common set of status codes, media types, and schema responses.
Schema: Standardize all request and response schema using JSON Schema components.

[20220330] Using HTTPS in your Development Environment – what it says on the box. More nodey than applies to us.

[20220329] Penny Wise and Cloud Foolish – deep dive into Google Cloud pricing.

[20220324] How To for the Microsoft Authentication Platform – including delegated and app permissions.

[20220323] Guidance for Architecting Mission Critical Apps on Azure – on the one hand this is good architecture, on the other hand it’s All The Expensive Toys.

[20220323] Bash Best Practices – quoting, readability, tests, don’t do this, how to debug.

[20220323] Bash Pitfalls – common mistakes from the real world.

[20220323] Defaults Matter – “Even the most well-intentioned people have a finite amount of energy to fight against a system that pushes toward bad outcomes.”

[20220322] Continuous Architecture – (1) Architect products; evolve from projects to products; (2) Focus on quality attributes, not on functional requirements; (3) Delay design decisions until they are absolutely necessary; (4) Architect for change—leverage the “power of small.” Big, monolithic, tightly coupled components are hard to change; (5) Architect for build, test, deploy, and operate; (6) Model the organization of your teams after the design of the system you are working on.

[20220321] Detecting Silent Errors – a mix of opportunistic and ripple testing. Opportunistic = take a long time while the server is out of operation for other reasons. Ripple = inject smaller known workload with known outputs regularly, rotating around the fleet.

[20220315] API Debugging Tips – isolate the API issue, examine the HTTP response code, then dig into the data causing it.

[20220311] Google’s API Design Guide – a general design guide for networked APIs. It has been used inside Google since 2014 and is the guide that Google follows when designing Cloud APIs and other Google APIs. This design guide is shared here to inform outside developers and to make it easier for us all to work together.

[20220311] API Improvement Proposals – Focused design documents for flexible API development.

[20220310] Eventual Consistency – how it happens with clusters, event sourcing, and async processing. And how to handle it: server wait, client polling, push to client, read from primary after you write.

[20220309] Reproducible Builds – Repeatable Builds + Immutable Environments + Source Availability

[20220309] One Way Smart Developers Make Bad Strategic Decisions –

[20220308] Evolve APIs – use versioned endpoints, and something like APISIX as a router to make old endpoints work.

[20220308] The 10 REST Commandments – some good advice.

[20220308] Payment Orchestration at AirBnB – how they rearchitected payments as microservices. Every major workflow is divided into a directed acyclic graph (DAG) of retryable idempotent steps, each with well-defined behavior. This allows the payment orchestration layer to maintain eventual consistency with other key services (such as the payment gateway layer and product fulfillment services). This approach has led to five 9s (99.999%) of consistency for payments.

[20220308] Pair Programming Antipatterns – not “you understand this, right?” but “which part of this is hardest to follow?”.

[20220306] Making CRDTs Byzantine Fault Tolerant – CRDTs let many nodes get the same data in distributed systems. But what if a number of adversarial nodes join the network and flood it with bad data? This paper offers a solution, making them Byzantine-fault-tolerant.

[20220306] Resumption Strategies After Interrupted Programming Tasks – return to the last method edited; navigate to remember; task tracking; review source code history. Some facts about interruptions: 57% of tasks are interrupted, 40% of interrupted tasks are not resumed; developers typically require about 15m to recover from an interruption.

[20220303] WASI – WASM at the edges (of the middle), from Fastly.

[20220224] Server Sent Events, WebSockets, and HTTP – a look at all three ways of doing pub/sub over the web.

[20220220] Open Telemetry – use it, it’s an upfront cost in complexity that pays off by giving you lots of tools to choose from.

[20220219] Pushing the 100k Button – AWS S3 storage balloons, and using policy to expire content isn’t easy. HN comments were interesting: GDPR gives a hammer against PMs who want to indefinitely retain data.

[20220219] Comparing Git Workflows – centralized workflow, feature branch, gitflow, forking.

[20220219] Open Policy Agent – OPA is an open-source project that implements a single policy language and policy engine that can be applied to solve policy and authorization problems at every layer of the cloud-native stack. It uses a purpose-built declarative language, Rego, to express sophisticated logic over complex hierarchical data structures.

[20220219] Deterministic Lockstep and Secret State – game networking might have something to teach us about our distributed app state problem.

[20220218] My Favourite Debugging Hack – printing a variable with line number and filename.

[20220217] Building for 99% Developers – accept slow migrations; accept legacy subsystems. Most companies don’t have a FAANG’s technical needs nor number of developers. And most places that talk up their shiny new shit have big mounds of rusty old shit too.

[20220216] The Absolute Minimum That Every Developer Should Know About Unicode and Character Sets – what it says on the tin.

[20220216] Test Data Management – interesting description of the problem of managing data in end-to-end tests, and the open source system they built to manage it.

[20220213] Server-Sent Events instead of Websockets – and from a discussion: it’s a beautifully simple & elegant lightweight push events option that works over standard HTTP, the main gotcha for maintaining long-lived connections is that server/clients should implement their own heartbeat to be able to detect & auto reconnect failed connections which was the only reliable way we’ve found to detect & resolve broken connections.

[20220213] The Kobiyashi Maru of Comparing Dates with Times – if you’re ever saying “is 2022-02-12 < 2022-02-12 03:00” then you are doing it wrong. But you will probably have to at some point. And there’s no right answer.

[20220213] A Decade of Major Cache Incidents at Twitter – amazingly transparent incident information. Painful how many of the problems happen below the application level.

[20220209] Simplifying Your Codebase When Using Feature Flags – Remove old feature flags as soon as possible; Your flag’s “off” position should be stable; Feature flags that aren’t temporary aren’t feature flags; Test one thing at a time; Create duplicates of components for easier removal.

[20220209] Tough Questions to Ask When Making Decisions – is it reversible, does it paint us into a corner?

[20220206] Five Design Patterns for Building Observable Services – Outside-In Health Check, Inside-Out Health Check, Real time Alerting, Troubleshooting.

[20220205] Don’t replace X with Y unless you expect 10x improvement – “so cvs to git, but not MySQL to PostgreSQL” (Charity Majors rule)

[20220205] Applying Event-Driven Architecture in Digital Transformation Projects – more detail than is usually shown.

[20220203] Testing Distributed Systems – papers, books, apps, resources. (tl;dr – is hard)

[20220203] Creating the Conditions for Developer Happiness – avoid handoffs of work between teams, minimise “time to login screen” ie startup time for new developer. Most development tasks are small enough that developers can quickly and continuously flow from small unit tests to completed code and on to the next task. Feedback cycles on the code are quick. Unit tests can cover small areas of the code while still providing value such that it’s rare that a developer needs to use a debugger to understand and solve problems.

[20220203] Event Sourcing, and Event Driven Architecture – two different things. Event Sourcing = saving your data as events not current state; Event Driven Architecture = building system around messages.

[20220203] Modern Identity Onboarding Essentials – tradeoffs of where personal info is kept, account linking, etc.

[20220203] Domain Driven Refactoring – walks us through writing simple procedural code and then pulling out services, collections, etc.

[20220202] Guide to Building Context Menus – interesting look at UI design.

[20220201] Boring Technology Checklist – has more detailed checkboxes under familiarity, stability, reliability, well-understood limits & trade-offs.

[20220201] Tips on Prioritising Technical Debt – “boy scout rule” = leave codebases and systems in better shape than you found them, so refactor when you touch a part of the codebase.

[20220131] Technical Debt Gets Worse Before It Gets Better – The siren song of the ground-up rewrite calls to us, trying to make us forget the fact that it’s much harder to replace an existing system than it is to create one from scratch. You have to keep supporting old APIs until every client has moved to the new APIs. Existing data has to be migrated to new models. You have to research (and often replicate) the quirks and undocumented features of the existing system.

[20220131] What’s in a Good Error Message? – context, the error itself, and mitigation.

[20220131] Feature Flags – develop in main, deployment and release are different.

[20220131] The Legacy Mimic pattern – assumes you know what your interface IS, of course …

[20220128] The baseline for web development in 2022 – low-spec Android devices in terms of performance, Safari from two years before in terms of Web Standards, and 4G in terms of networks. The web in general is not answering those needs properly, especially in terms of performance where factors such as an over-dependence on JavaScript are hindering our sites’ performance.

[20220128] Automerge – conflict-free replicated data types in Javascript for building distributed apps, which lets you have local-first software. Not suitable for banks.

[20220128] Foundations of Computer Science – book has been taken out of print. Nothing much has changed since it was published in 1992.

[20220128] My Goal is to Ship - 25 years of software engineering. “Even with years of newfound tact, what frustrates me profoundly are projects that sit on shelves. I’ve learned to push them to a conclusion, negotiating their way out the door. If I sensed a project was permanently stuck, I’d leave it behind, despite others who chip away at it.”

[20220128] Boki - new serverless runtime that exports a shared log API to serverless functions. “The key enabler is the metalog, a novel mechanism that allows Boki to address ordering, consistency, and fault-tolerance independently.”

[20220201] Tech debt at the scale of a super app

[20220201] Next-auth.js – built in support for google, etc., BYO backend, built-in email/magic link etc, works with cookies

[20220201] Extend-Only Design

breaking changes have one thing in common: something that existed in the past was irrecoverably altered in a future release, and thus broke the downstream applications and libraries that depended upon that prior functionality.
The pillars of extend-only design apply to any and all software systems, and they are thus:
- Previous functionality, schema, or behavior is immutable and not open for modification. Anything you made available as a public release lives on with its current behavior, API, and definitions and isn’t able to be changed.
- New functionality, schema, or behavior can be introduced through new constructs only and ideally those should be opt-in.
- Old functionality can only be removed after a long period of time and that’s measured in years.
Extend-only design requires some more planning and perhaps some more flexibility on the application-programming side of things

[20220201] Why our team cancelled our move to microservices

Were heavily reliant on a third party, which limited their ability to divide into microservices.
Couldn’t sufficiently isolate the microservices. “We couldn’t identify any obvious candidates in our monolith to be broken out into a microservice. So instead, we started drawing arbitrary lines between our domain models, and from this, we had the list of microservices we were to create. However, once we started investigating, we found a lot of shared business logic and implicit coupling between the soon to be separate microservice domains. Some further attempts were made to subdivide these microservices into smaller and smaller pieces, but that left us with even more coupling, messages buses everywhere, and a potential big bang of immediately going from one service to ten or more microservices.”
Couldn’t assign ownership of any potential microservice to any team, because all teams touched all microservices for their features.
Platform wasn’t ready.
It wasn’t uncommon for requirements to change mid feature. This uncertainty made creating microservices more fraught, as we couldn’t predict what new links would pop up, even in the short term.
We had a tiny window, just large enough split our monolith into the list of microservices we had been given. What we didn’t have was any extra time to allow us to reflect on what we had created or alter course if required.
None of the people responsible for architecting or implementing the microservices architecture had any specific prior experience
We didn’t have a list of our pain points, and we had no clear understanding of how this would help solve any pain points we do have.
Adopting microservices isn’t free. There is a vast list of additional concerns that you need to address. We would need to revisit many concerns that we had previously addressed in our monolith. For example, we would need to address or revisit: logging, monitoring, exception handling, fault tolerance, fallbacks, microservice to microservice communication, message formats, containerization, service discovery, backups, telemetry, alerts, tracing, build pipelines, release pipelines, tooling, sharing infrastructure code, documentation, scaling, timezone support, staged rollouts, API versioning, network latency, health checks, load balancing, CDC testing, fault tolerance, debugging and developing multiple microservices in our local development environment.
We were using monolith like a loaded term. As if saying “monolith” implies something terrible, and “microservices” implies something good. Once we looked past the stereotypes and branding, the development team had very few issues with our “monolith.”
The more we looked into microservices, the more it seemed that it was less about technology and more about structuring teams and the work that came into them. Had we made a mistake approaching microservices as purely a technology problem?

[20220201] The Art of Immutable Architecture: Theory and Practice of Data Management in Distributed Systems – “Most software components focus on the state of objects. They store the current state of a row in a relational database. They track changes to state over time, making several basic assumptions: there is a single latest version of each object, the state of an object changes sequentially, and a system of record exists. This is a challenge when it comes to building distributed systems. Whether dealing with autonomous microservices or disconnected mobile apps, many of the problems we try to solve come down to synchronizing an ever-changing state between isolated components. Distributed systems would be a lot easier to build if objects could not change.” … $108 to buy!!!

[20220201] Store SQLite in Cloudflare Durable Objects – and HN discussion

[20220201] Textbook for system design (WIP)

[20220201] System Design Primer (for web-scale systems)

[20220201] Self-hosted architecture

[20220201] Give me events not webhooks

[20220201] Server Sent Events

[20220201] Learn how to document APIs

[20220201] UUIDs bad for MySQL performance

[20220201] API Design Patterns book

[20220201] Hoppscotch API development

[20220201] The API Book (github)

[20220201] Flutter’s short-coming as xplatform

[20220201] Common Infrastructure Errors

[20220201] Best Practices can slow your application down

[20220201] Things your CS degree didn’t prepare you for but which you must do as a developer (Twitter)

[20220201] Data Grid Performance Comparison

[20220201] Model View Update

[20220201] single-tenancy vs multi-tenancy discussion on HN

[20220201] Snowflakes Servers and Snowflakes as Code

[20220201] Microservices lessons

[20220201] Shopify’s Monolith

[20220201] Interface Segregation Principle

[20220201] 5 Tips for API Design Reviews

[20220201] API Lifecycle Blueprint

[20220201] Service Locator is not an Antipattern

[20220201] REST API Design

[20220201] 8 Simple Tips for using an API in a mobile application – from SyncFusion

Check internet connectivity before an API call
API versioning
Show loading/progress until API responds
Use the status code to display a meaningful error message on failures.
Paginate and lazy load
Code for case where a logged-in user expires.
Plan for forced updates.
Send network type in API request.

[20220201] Consider SQLite - why and how to use SQLite in real situations.

It’s only when you start needing significantly more than 99.999% uptime that you start to need truly distributed systems
Litestream and Verneuil replicate your SQLite database to other machines.
there’s no per-request client/server roundtrip latency
But …
- the type system is bad
- Support for migrations is worse
- Significantly harder to geoshard your app
- Not well supported by some web frameworks
- Heavy computations or slow language might make a single server hard
- Failure modes of a single thread are different than connection pools, so architect for it.

[20220201] Abstractions in Software Development

Good explanation of the SOLID principles
Stable Dependencies Principle = a package should only depend on packages that are more stable than it is.

[20220201] Liskov Substitution Principle - subtypes must be substitutable for their base types.

[20220201] Challenging the Pull Request orthodoxy – Ship vs Show vs Ask

[20220201] Microservice Strangler Pattern

[20220201] Batching and Caching with Dataloader - GraphQL way to avoid the N+1 problem (you make one query to determine which things to show and then, for each result in that query, make another query to fetch the information about that thing)

[20220201] Why do you need to move from CRUD to Event Sourcing? – ES seems optimised for “write a lot, read rarely” because you need to replay events to get to current state. Is there a hybrid? Ah yes, there is. It’s CQRS pattern – maintain current state in a reporting database and preserve events in the writing database. I wonder how multistep transactions work – are they events handled atomically by the processing code?

]]>

Nat's 2022 Technical Link Pile: Front End/Flutter/Dart/UI

2022-12-31T00:00:00Z

See the Intro for context.

[20221231] React Admin – claims of a full-featured data grid, which is 50-80% of many of our apps. (As well as roles & permissions and validations)

[20221224] Death to the Line of Death – the idea that an application should separate trustworthy UI from untrusted content. The typical example is in a web browser, where untrustworthy web content appears below the browser toolbar UI. Trustworthy content provided by the web browser must appear either in the browser toolbar, or anchored to it or overlapping it. If this separation is maintained, then untrusted content can’t spoof the trustworthy browser UI to trick or attack the user.

[20221224] FlexGrid – cross-platform grid for MAUI apps.

[20221210] PeriText – a CRDT implementation that supports primarily offline use – ometimes people prefer a more asynchronous collaboration style, where they can work on a private copy of a document for a while and share their updates later. The algorithms underpinning services like Google Docs are not designed to support this use case.

[20221126] Building a BZT JSON CRDT – fault-tolerant conflict-free replicated data types. The HN discussion has interesting notes about the shortcomings of CRDTs (eg, merging two changes might violate the document schema).

[20221126] Never Use a Warning When You Mean Undo – “do you really want to delete this?” is much worse than just having an Undo button to let you recover if you say “Yes” and then regret it. From HN discussion, “hold to activate” is a useful way to force consideration of truly destructive actions; “never disable a menu item or widget without providing feedback (like a tooltip) about why it’s disabled and what to do to enable it”; GNOME Human Interface Guidelines discourage Yes/No and OK/Cancel in favour of meaningful button names.

[20221119] USWDS – the US Government’s Web Design Standards. Lots of good info and recommendations here.

[20221008] Junior Designers vs Senior Designers – good list of the different behaviours between juniors and seniors. Much of it is owning success and having initiative, vs being told what to do.

[20220922] DataDog’s UI Library – code is internal use only but the documentation is public so you get to see their design vocabulary and opinions.

[20220213] Native to React Native to Flutter – one project’s experiences. They like Flutter.

[20220213] Awesome Flutter – collection of Flutter resources. Beware, these are usually up to date when the compiler got enthusiastic and compiled them, but most aren’t maintained.

[20220201] How to make your Flutter app offline-first with CouchBase Lite

]]>

Nat's 2022 Technical Link Pile: GPT-3

2022-12-31T00:00:00Z

See the Intro for context. (Also, this was a late addition and I’m focused in my interest, so this is thin)

[20221231] Reverse Engineering Notion’s Prompts – in Hacker News discussion there’s skepticism you’re getting the actual prompts, vs GPT-3 being its usual stylistically faithful yet factually inaccurate creative self. Someone from Notion confirmed this.

[20221231] Some Short-term Predictions – the age of AI, if you don’t own the query interface, you’re just assembling training data for those who do. (The behaviour of the social media tech powers mean I do not trust successful AI companies to use their powers wisely)

[20221221] On Meaning, Form, and Understanding in the Age of Data – In this position paper, we argue that a system trained only on form has a priori no way to learn meaning.

[20221221] On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? – In this paper, we take a step back and ask: How big is too big? What are the possible risks associated with this technology and what paths are available for mitigating those risks? We provide recommendations including weighing the environmental and financial costs first, investing resources into curating and carefully documenting datasets rather than ingesting everything on the web, carrying out pre-development exercises evaluating how the planned approach fits into research and development goals and supports stakeholder values, and encouraging research directions beyond ever larger language models.

[20221221] ChatGPT Playing a Complex Game – basically a German board game with different resources and rates of conversion.

[20221221] Prompt Engineering Guide – pointers to papers and tutorials on prompt engineering.

[20221221] Prompt Engineering Guide from Microsoft – give it a high-level task description, guide it with examples, guide it with high-level contextual information (eg API in comments), guide it with conversational history.

[20221221] Best Practices for Prompt Engineering with the OpenAI API – Put instructions at the beginning of the prompt and use ### or """ to separate the instruction and context; Be specific, descriptive and as detailed as possible about the desired context, outcome, length, format, style, etc; Articulate the desired output format through examples; Articulate the desired output format through examples; Reduce “fluffy” and imprecise descriptions; Instead of just saying what not to do, say what to do instead; Code Generation Specific - Use “leading words” to nudge the model toward a particular pattern (eg you leave SELECT in as the last word of your prompt to build a SQL query).

[20221221] GPT-3 Lacks Systematicity – asking it to reformat dates, sometimes it gets days and months around the wrong way.

[20221221] Tracing Emergent Abilities of Language Models to Their Sources – the initial GPT-3 might be superficially weak, it turns out later that these abilities serve as very important foundations of all the emergent abilities unlocked later by training on code, instruction tuning, and reinforcement learning with human feedback (RLHF). […] The ability to respond to human instructions is a direct product of instruction tuning. The ability of generalization to unseen instructions is a free lunch given by scaling types of instructions. The ability of complex reasoning with chain-of-thought is likely to be a magical side product of training on code.

[20221221] The GPT-3 Architecture on a Napkin – exactly what I needed.

[20221221] Azure Cognitive Service for Language (Pricing) – ouch.

[20221221] Finetuning with the OpenAI Language API – giving training examples on the commandline.

]]>

Nat's 2022 Technical Link Pile: Intro

2022-12-31T00:00:00Z

I used to read a lot of the technical web and social media for O’Reilly Media, where I wrote the daily “Four Short Links” digest. I still read a lot of online content, but now for my own purposes. I’ve been CEO and now CTO of a company that makes retail management software (ERP, POS, and other acronyms for fashion retailers), so I’m short of time and my reading is focused on things relevant to my needs.

I kept track of the interesting things I found, and sorted them loosely into piles. Those piles are:

Each pile is now a post on this blog. Each post is a list of entries, sorted reverse chronologically. Italicised text is quoted from the site. I’ve datestamped the entries.

I’m shallowest on C#, SQL Server, and Flutter. We make on-prem WinForms software and I’m simultaneously coming up to speed on the .NET tech stack, SQL Server, and our use of it – while also looking for future platform options. Flutter is one of those. (We’re hiring WinForms seniors! Mail me: nat@ontempo.co.nz)

Product, Dev, Management are the richest veins. I pretty much kept Interesting Software only for open source software I found. And Random is just nerdy interest.

]]>

Nat's 2022 Technical Link Pile: Management

2022-12-31T00:00:00Z

See the Intro for context.

[20221231] Four Day Work Week Going Well – 97% of the 495 respondents want to stay with a four-day workweek and so do the 27 companies that responded.

[20221223] 20 Great Open Questions – for coaching and more.

[20221221] Engineering in a Hybrid World – At the time of this report,

over 50% of respondents had no definitive plan to return to the office. As companies scale, more time is spent on improving existing products whereas earlier stage companies can afford to spend more time on building. In the last year, time spent on building new capabilities declined from 61% to 56% of total elective investments. More engineering organizations are starting to track developer productivity, with the top metrics reported on being number of bugs, % of committed software, working software, and PR to release time.

[20221221] Andy Grove’s High Leverage Activities – Task-Relevant Maturity (people are good at some things and worse at others); No issue should be discussed in a group unless it affects more than two people; A decision backed by a veto may be fair and well-reasoned — but it won’t look that way to other members of the group over time; All your decisions should be made by considering six questions: What decision is needed? By when? Who should be consulted? Who decides? Who ratifies or vetoes? Who needs to be informed? Planning should focus on weekly or monthly cycles rather than quarterly or annual ones.

[20221211] To Find Meaning in Work, Change How You Think About It – Connect work to service; Craft your work – and make work a craft; Invest in positive relationships; Remember why you work.

[20221211] How to Run Customer Feedback Sessions – what it says on the tin.

[20221211] The Engineering Manager’s Toolbox – useful tips for moving systems, people, and rewards.

[20221211] Technical Debt and Productivity – summary of a paper. Developers waste 23% of their working time due to technical debt; Wasted time is most commonly spent performing additional testing; Code related issues, testing issues, and architectural issues are most strongly correlated with wasted time; Developers are largely aware of the amount of time wasted due to technical debt. However, managers are not as aware.

[20221211] Investing in Documentation – Model good writing habits; Get in the habit of editing; Make writing part of the job ladder; Focus on quality not quantity to set the bar; Act as a journalist; Empower your senior engineers; Assemble your list; Identify ownership; Assign a rotating docs czar; Don’t be afraid to delete; Level up with landing pages; Take a snapshot.

[20221210] Startup Restructuring 101 – all the advice on layoffs.

[20221210] Company-Team-Self – folks will accomplish more if you let them do some energizing work, even if that work itself isn’t very important. […] What folks may not understand, is that for a certain type of person, strictly adhering to the correct path is very energizing. That kind of person, a person who I used to be most of the time (and still revert into today when particularly frustrated), doesn’t need to do sub-optimal energizing work, because doing the correct work is inherently energizing.

[20221127] Spit Your Team and Make Them Happier – when breadth of responsibility makes a long learning curve and inhibits feeling competent.

[20221119] The Seven Levels of Busy – Michael Lopp breaks “busy” down into different degrees. Tag yourself.

[20221107] Collective Decision-Making with AHP – pair-wise evaluation of options against criteria.

[20221107] How to Plan – Kellan’s advice lands. I’ve experienced about 80% of his lessons, e.g. planning is the wrong time to introduce new things. This list helped me put words to some of the failures I’ve been a part of.

[20221107] CTO Field Guide (For the First 90 Days) – ebook made from blog posts. How to figure out where to start.

[20221107] Kickstarter Addresses Technical Debt – look at pull requests merged/day, look at busiest pull requests (ones with most comments/go-arounds), and cycle time. Simplistic story but useful opening context.

[20221003] Engineering Manager’s Bill of Rights – rights and responsibilities between org and managers.

[20221003] All Hands Meetings Strategy – balance between learning, progress, and culture.

[20221003] Platform Engineering KPIs – digging into the metrics around delivery, stability, etc.

[20221003] Multicloud Strategy – through a farm metaphor.

[20221003] Interviewing to Assess Ability to Deliver Impact – structure for interviewing engineers.

[20221003] Creating Clarity in Competencies and Ladder – visualising ladders and skills then workshopping.

[20221003] Deployment Infrastructure for a Webapp – I use 2 VPS (virtual private servers) running on DigitalOcean; The database is Postgres and is fully managed by DigitalOcean;

I use a blue-green deployment; Deployment is done via git and ssh; No CI/CD; No containers; Absolutely no Kubernetes. HN Discussion has links to interesting utilities.

[20221003] A CTO Should Be Technical – educated tradeoffs is the best reason. (HN Discussion)

[20220922] Rituals at Engineering Teams – not sacrifices.

[20220922] How to Create Psychological Safety – Approach conflict as a collaborator, not an adversary; Speak human to human; Anticipate reactions and plan countermoves; Replace blame with curiosity; Ask for feedback on delivery; Measure psychological safety.

[20220922] This is Why You Should Not Accept Unlimited PTO – Company leaders shirk responsibility; Managers and teammates bear the burden; Everyone suffers from unfairness. “Since those in an unlimited PTO organization do not accrue any time, they are typically not paid out for that earned time if they leave the company.”

[20220922] Writing Software vs Building Systems – senior engineers.

[20220922] A Development Process to Ship Features Faster – Avoid Feature Branches; Automate CI/CD pipeline; Use a monorepo; Break features into small, incremental changes; Use feature flags; Define owners; Request at least one demo per week; Deploy weekly; Freeze the code before release; Write tests (or not?).

[20220922] Inherited Worst Team and Code, How Do I Fix It? – comments are good. Exec buy-in, low hanging fruit (step 1: add version control), the Pivotal method.

[20220910] One on Ones with Executives – different types of meetings and what you should do in each. Sound advice.

[20220901] Organizational Boundary Problems – Open cultures can have a dark side too, though. Openness doesn’t come for free, and without structure to enable participation, a culture that calls itself “open” can easily evolve to increase the feelings of exclusion it was trying to avoid.

[20220901] Debugging Your Teams – based on the type of failure, what could be going on that isn’t blaming the individuals.

[20220901] Change Management for Tech Leaders – lots of things that I’ve learned the hard way, and some I now don’t have to.

[20220802] A Staff-Shaped Hole – how companies fail staff engineers. “Self-directed” doesn’t mean you can abandon without onboarding and support. “Lead with influence” often comes from those with structural power. Let leaders lead (don’t micromanage). Not coding is still doing real work. _Any time you hire a great leader and they don’t deliver the magical result you expected and leave burned out, it’s basically always the organization’s fault._All men. The most senior person you have is not a de facto leader/principal. You need architects around 150 people. A custom ladder is a shitshow because it takes you out of sync from every other company. Bored at smaller companies. Promotion without a change in tasks.

[20220726] How to Quickly Onboard Engineers Successfully – it starts before day one; track it in a task; pick the right onboarding buddy; pair, pair, and more pairing; all the coffee chats; tailor the experience to the role; ship some code in a week or less; give and get feedback.

[20220718] Scrum Teams are Often Coached to Death While the Problem is with Management – points the finger at: Telling teams what they should do and by when; Telling teams to commit to the work planned, not the outcome; Continuing to have individual appraisals that conflict with the team goals; Continuing praising teams that break their sustainable pace; Continuing to expect teams to work on multiple things at the same time; Continuing to see the output as a measure of success, not the outcome; Continuing to ignore organizational impediments.

[20220629] How Individual Contributors Get Stuck – noticing how people get sidetracked and stuck is a superpower.

[20220624] Biggest Temptation of a Software CEO – throwing more bodies at problems. High velocity within the engineering team is often less a function of what a company agrees to build, but is more a function of what a company declines to build.

[20220624] Replatforming – all the hard stuff about changing your platform. Replatforming or re-architecting production software is a complex, expensive, risky proposition. Most of the challenges are around dealing with internal expectations and paying external customers, not the underlying technical details. So let’s not think of these as development-only exercises and assign sole responsibility to Engineering.

[20220624] How Engineering Managers Fail – Continuing To View Their Role Mainly as a Maker; Assuming Everyone Knows What You Do; Taking Extreme Views; Feeling the Need To “Shield” the Team; Optimising the Parts Instead of the Whole

[20220624] Code Freezes – code freezes aren’t change freezes, as hotfixes happen. Freezes reflect risk that should be removed every day, not just the days you don’t code.

[20220624] Decision Making – not “don’t be wrong”, but instead “how can I adjust course gracefully”? I needed to adapt to changes instead of assuming they wouldn’t happen.

[20220624] Know How Your Organisation Works – get comfortable with the mess.

[20220624] Strategies and Visions – really good advice on how to make useful strategies.

[20220529] Lessons Learned on the Journey from Developer to Project Manager – identify and document the context and constraints first; write everything down; The scope of the project and what needs to be done should be clear to everyone. If the methods used to support the mission get in the way or provide no immediate tangible benefits to the project, they should be removed; Your role is ultimately to cut through useless and irrelevant tasks and time wasters and focus on what will actually bring a benefit to your customers whomever they are; have contingency plan roadmaps.

[20220527] Restructuring Engineering Teams with Team Topologies – first was to stream-aligns and enabling teams; then to a platform.

[20220527] Docs for Personal Impact – prioritise the things you’re trying to achieve, understand if you’re making progress, and more. Shit that’s a lot of meta-work, tho.

[20220527] The Feedback Fallacy – questioning the value of feedback, which is premised on (1) the observer is actually seeing something accurately; (2) the observer’s suggestions for what to do more of are right. Both get shot in the knees by this article. But obviously *my* feedback is awesome and the exception here.

[20220526] Stripe’s Writing Culture – Leaders must demonstrate quality writing; Exemplify quality writing in everything you and your leadership team share; Make writing the default method of sharing knowledge; Give teammates a starting point with sample docs; Know when to standardize and when to give autonomy; Make your documentation easy to read; Create a support system; Building your culture of writing and documentation.

[20220525] When Everything is Important But Nothing is Getting Done – prioritise, minimise WIP, remove dependencies.

[20220505] Preventing Burnout: A Manager’s Toolkit – Reduce the number of hours worked by agreeing to reduce effort. Managers can ask team members to identify things that are likely to fail.

[20220504] See and Resolve Team Dependencies – blog post series with meat.

[20220406] Merge Code Every Week – reduce time to feedback.

[20220406] The Danger of Software Prototypes – sunk cost fallacy, insufficiently precise requirements (“plausible but wrong”), overzealous belief in iterative development (some architectural decisions like thread safety and 1:1 vs 1:n vs n:m schemas are hard to correct later). “Gather enough requirements to know when to throw your prototype away.”

[20220404] Scaling Engineering Teams – SWOT focused on future growth vs current status; clump them into “domains”; write a one pager for each clump w/current status, problem statement, goals; pair engineers and team leads as a designated task force for each domain; grant each task force a month with up to 20% designated time to create a structured plan (Research; Goal; Timeline & milestones; Resources); document what you learn and share it; enable and guide.

[20220404] What Makes Developers Happy at Work – salary, work-life balance, flexibility, productivity, and growth opportunities. “Feeling productive at work plays a much more critical role in team happiness than we probably realize. It shouldn’t be as surprising as it is,” said Matt Kiernander, technical advocate here at Stack Overflow. “When I code, I don’t like disruptions in my flow state. Constantly stopping and starting makes me feel unproductive. We all want to feel like we’re making a difference, and hitting roadblocks at work just because you’re not sure where to find answers is incredibly frustrating.”

[20220404] How GitHub Does Take Home Technical Interviews – nice use of GitHub’s tools. The pull request includes automated tests and a rubric so that interviewers know how to mark it and each submission is evaluated objectively and consistently. The tests run through GitHub Actions and provide a base level for the reviewer.

[20220404] End to End Feature Development – genericontent for engineering blog. Take a moment for a quiet vom at managers being called “People Empowerers”.

[20220330] Rules for Cooperative Efforts – nice series of principles by Stephen Walli.

[20220330] Continuous Documentation – incorporating documentation into the dev workflow.

[20220324] Clear Expectations – “I expect you to”, follow up (“if you walk away, I will follow”), don’t rush to reset expectations. “I expect [name] to do [thing] by [timeframe] within [constraints].” If there are action items after a meeting, you can list the items as “[name]: [timeframe]: [thing] within [constraints].”

[20220324] Technical Debt Balance Sheet – I can’t tell if this is an exercise in angels and pin heads, or if it’s seriously useful.

[20220308] 25 Microhabits of Successful Managers – close to my approach.

[20220308] Managing Technical Debt in a Microservices Architecture – really about mapping the tech capability plan. List of tech options with Preferred/Acceptable/Discouraged/Unacceptable, then three years of quarter-by-quarter Plan/Deprecate/Migrate/Use/Remove with risk estimation. Bottom-up generation from engineers who know the problem space.

[20220306] Stop Paying Down Technical Debt, Start Maintaining Code – Stop paying down tech debt and start actual software maintenance. That’s the real term we should all be talking about.

[20220303] What is Developer Productivity and How to Measure It? - SPACE framework = Satisfaction, Productivity, Activity, Communication & collaboration, Efficiency & flow.

[20220225] QRF – quick reaction force, aka “the team that it’s okay to interrupt”.

[20220225] Managing People – your job is not to manage people, but to manage processes and lead people.

[20220225] Technical Product Managers – what do they do at different companies and stages, how does the role sit vs project manager or product manager. The TPM owns the ‘when?’ and ‘who?’ questions. The PM owns the ‘why?’ and ‘what?’ The EM and the engineering team own the ‘how?’

[20220225] Desperation-Induced Focus – a nice piece against process. Big companies suck. At some point they were just like you. A startup fighting for product market fit or creating predictability in go-to-market or expanding geographically or something else that actually matters. They were focused. Focused on hiring trajectory-changing people and giving them more responsibility than they knew what to do with. They were working their a**es off to create something out of nothing. They had desperation-induced focus. But now they are different. Most big companies aren’t focused on creating things out of nothing. Someone else made the magic money-making machine, and they assume that it will just keep working. With the one thing that actually matters taken care of, they care about luxuries like making sure as high a percentage of the company as possible feels included in the planning process. Or creating performance review frameworks with an ever-increasing number of boxes and categories. Or something else that matters even less than that. This lack of focus is a luxury and a disease.

[20220224] Failed Squad Goals – post-mortem on Spotify’s famous but failed “agile at scale” model.

[20220219] Making Room for New Things – interesting breakdown of work. “Reactive fixes, heroics, adhoc work, silver bullets, and chasing the latest “great idea”. This work can be especially draining!” And a good point that we can end up doing low-value things on autopilot.

[20220218] Game Project Anti-Patterns – based on 600 post-mortems.

[20220217] Delegation – plot on urgent/important and do urgent+important, delegate urgent+!important, schedule or delegate important+!urgent, ignore !important+!urgent.

[20220217] Charity Majors on Careers – Most of us hate feeling like beginners. We associate it with anxiety, with failure, with not being good enough. If you want to have a long, rich, rewarding career in tech, the best thing you can do is consciously retrain yourself to lean into curiosity and its discomfort.

[20220217] Starting with Nonviolent Communication – key points are

When ____[observation], I feel ____[emotion] because I’m needing some ____[universal needs]. Would you be able to ____[request]?
observe, don’t evaluate/judge (“when you are late returning my book” vs “when you are a thoughtless git”)
emotions, not thoughts (“I feel sad” not “I feel like you’re a thoughtless git”). Go beyond “anger” to “shame” or “fear” or …
universal needs, not strategies (don’t tell them what to do, tell them what you need: “I need transparency” vs “I need to be copied on every email”)
requests, not demands. Make it specific, say what you want (not what you don’t want), and treat a “no” as an opportunity to find out what it will take to get to “yes”.
appropriate consequences are those whose purpose is to protect your needs, not to punish the other person

[20220216] That Wild Story – designing a human process around pathological cases leads to processes that are themselves pathological.

[20220216] Technical Decision Making – the standard process (define problem and criteria for evaluating solutions; go wide to find possibles and evaluate them; make the decision).

[20220213] Reasons Not To Be A Manager – many good reasons, and I particularly liked “Your time doesn’t belong to you”..

[20220213] Before You Quit Your Job – six core needs from work: BICEPS = Belonging, Improvement, Choice, Equality, Predictability, Significance. Which needs does your job fulfil, and where are your gaps?

[20220213] Onboarding – a summary of where I’m heading

[20220208] Never interrupt a programmer – cartoon.

[20220208] Four Hour Work Week – 100/80/100 (100% of the work, 80% of the time, 100% of the pay) means squeezing out breathing time, chit-chat, and so on. And customers still call on Friday. I think we do it right – if you want to work four days, work four days and be paid for four days.

[20220207] Managing People – There is no point being angry at your team – ever. You are in charge of processes and people. And you got more information than they do, always. You either created the processes where this outcome happened or you hired (or did not fire) the wrong people. Ultimately everything is your fault

[20220206] Feedback Conversations – Strategy = Anything that contributes to potential behavior change. Self expression = Everything else you should not say. Feedback is about Strategy. Self-expression is why most feedback conversations fail. Mentally forgive the person, Identify what will motivate them to change, Say only the 10% that will actually change behavior.

[20220206] A Better Way to Do Self-Evaluation and Give Feedback

STAR: Situation, Task, Action, and Result
SBID: Situation, Behaviour, Impact, and Desired outcome
INDL - Impact, Numbers, Descriptions, and Links.

[20220203] Hacks for Engineering Estimates – estimate/target/commitment/plan; establish extremes; note the precision; ask for confidence levels over time.

[20220203] Mistakes First-Time Leaders Make – 👀

[20220202] What happens when colleagues know each other’s salaries – pay gaps close, and it becomes disconnected from performance.

[20220131] How to Mentor Software Engineers – more than code review.

[20220131] Software Engineering Culture Questions – lots of ways to assess different aspects of a team’s culture.

[20220131] How to Write a Strategy That Anyone Can Remember – “even over” statements. I’m a fan of the opposite: two metrics at once, one you want to maximise and one you don’t want to hurt.

[20220130] Prioritization, Multiple Streams, and On Call – advocates for what we call Support Supporter as rotating role

[20220128] How to Decide When It’s Time to Stop Designing and Start Coding – heuristics

There are enough known unknowns (“if you have something you can ship with low risk, do it”)
There are no more known unknowns
There are far too many known unknowns (prototype to focus effort)
You stop learning new things (discussing same point over and over)
Falling down the rabbit hole (things start getting too complicated and abstract / imaginary problems).

[20220128] How to Beat Burnout – things you can do for yourself. (Not replacing structural changes to work)

[20220201] How to respond when someone leaves (a subtweet)

[20220201] Stop Being the Hero (practical -> emotional -> narrative; ppl want to be the hero)

[20220201] Death to Tribal Knowledge – good list of dev things to document

[20220201] Manager’s guide to holding people to account

[20220201] Heuristics (a credo)

[20220201] Avoiding technical bankruptcy (technical debt = symptom of flawed dev mgmt)

[20220201] Technical design review process

[20220201] Example of a technical design review document

Then, rather than allowing everyone to comment directly on the document, create a blank Google doc where everyone lists their questions about the design alongside their name. This will serve as the agenda for the upcoming design review.

[20220201] Common management failures in developing ICs

Doing all the technical design work yourself
Doing all the project management yourself
Neglecting to give feedback
Hoarding information
Focusing too much on your personal output

[20220201] Productivity Text File

[20220201] Bets, Success Metrics, and Roadmapping

[20220201] Lessons from my PhD

Lead or be lead
Outline with topic sentences
Get excited
Unmotivated details
Slide vs Speaker-led presentations
Managers are Input/Output machines (get feedback)

[20220201] Planning sessions

[20220201] Incident Review Best Practices

[20220201] “Incidents are Unplanned Investments”

[20220201] Types of Waste in Software Projects (paper)

[20220201] Software Development as a Wicked Problem - good description of wicked problems

[20220201] http://en.wikipedia.org/wiki/Situational_leadership_theory

[20220201] http://wimvdd.blogspot.com/2007/05/what-is-situational-leadership.html

Competence vs. Commitment Matrix gives you a quick and easy view and clear understanding of four basic types of leaderships.
The 4 types of leadership has been defined by Hersey and Blanchard which they named S1 to S4:
- S1: Telling – one-way communication. Leader defines the roles of the individual or group and provides the what, how, why,when, and where to do the task
- S2: Selling – two-way communication. Leader is still providing the direction and providing the socioemotional support that will allow the individual or group being influenced to buy into the process.
- S3: Participating – shared decision making about aspects of how the task is accomplished. Leader is providing less task behaviors while maintaining high relationship behavior.
- S4: Delegating – the leader is still involved in decisions; however, the process and responsibility has been passed to the individual or group. The leader stays involved to monitor progress.
The Situational Leadership Theory:
- developed by Paul Hersey, professor and author of the book “Situational Leader” and Ken Blanchard, leadership guru and author of “The One Minute Manager”
- introduced in 1st edition of “Management of Organizational Behavior” as “Life Cycle Theory of Leadership”
- renamed to “Situational Leadership theory” in mid 1970s.

DEVELOPMENT LEVEL	APPROPRIATE SUPERVISORY STYLE
D1 Low CompetenceHigh Commitment	S1DIRECTINGStructure, control (telling people what to do, how to do it, where to do it) and supervision of performance
D2Some CompetenceLow Commitment	S2COACHINGDirect and Support
D3High CompetenceVariable Commitment	S3SUPPORTINGPraise, Listen, Encourage, Facilitate involvement in problem solving and decision making
D4High CompetenceHigh Commitment	S4DELEGATINGTurn over responsibility for day-to-day decision making

* Competence (encompasses education, training, experience, etc)

* Commitment (combination of confidence and motivation)

[20220201] Structural lessons in engineering management (IC vs mgmt isn’t clean-cut)

[20220201] Doing too much work before looping in others

Encourage engineers to show their work as quickly as possible, never > 1w without showing something.
Can show a design doc, PRD, prototype, anything that enables meaningful feedback.
If an engineer is unclear on a first deliverable, help them with what and when.
Encourage engineers to get end-to-end deliverables asap.
Develop behind feature flags so features can be checked in incrementally.
Demo repeatedly as the feature gets fleshed out.
Encourage everyone to give constructive feedback, and beware overly negative or toxic feedback.

[20220201] How to Improve Time Management

Planning Fallacy: tendency to underestimate the time required to finish a task or overestimate our ability to complete it within a set amount of time. This fallacy is partly due to reliance on overly optimistic performance scenarios without the help of reliable data.
if a task is overly complex and contains multiple steps, the chances that one of them will hit a snag and impact your timeline will increase. Even worse, if you aren’t tracking your time consistently and flagging the issues quickly, minor delays can add up to a substantial impact.
It feels good to check an item off a todo list so we can be trapped into chasing menial wins.
The pain of doing the work is more than the pain of delaying it.
Responsiveness and Throughput are rivalrous because being interrupted increases your WIP (and you pay a context-switching price)
The Zeigarnik Effect states that we’re more likely to remember incomplete or interrupted tasks than finished ones.

[20220201] 7 Productivity Methods

Tackle biggest and most important first.
Time blocking.
Pomodoro.
Schedule to your biological energy levels.
Zen To Done (ZTD) has 10 “habits” you pick and choose from:
1. Collect: Write down ideas, tasks, and thoughts as they come to you to get them out of your head and onto paper.
2. Process: Make decisions and answer emails quickly, so that it doesn’t pile up, procrastinating for later.
3. Plan: Write down your biggest must-do’s for the week, schedule them out, and do them first every day.
4. Do: Focus on one task at a time—turn off your phone and don’t check emails or task switch until a set time or it’s done.
5. Simple Trusted System: Keep simple lists, check tasks off as you go, and stick to your daily lists.
6. Organize: Declutter and organize your space, your inbox, and your mind—that means dealing with action items as they arise and deleting the rest.
7. Review: Do a weekly review of your goals and your progress, adjusting your systems and objectives accordingly.
8. Simplify: Reduce your tasks and goals down to the bare essentials and focus on them.
9. Routine: Set and keep a daily routine that gives you time to be focused and productive and recharge.
10. Passion: Seek work that you’re passionate about or love and notice how little you procrastinate (and how happy you are).
Don’t break the chain
Commitment Inventory
1. List: Write down a complete list of how you spend your time (including chores, family time, meetings, exercise, and task breakdown).
2. Combine and categorize: Narrow the list down into categories and assign a percent of time spent on each. Visual pie charts are helpful here!
3. Review: Ensure that important commitments have enough time to do it well, cut the rest, and adjust your totals to equal 100%.
4. Schedule: According to how much time you want (or need) to spend per category, schedule out your day.
5. Checklists: Work in checklists, rather than to-do lists, because they are more granular, breaking down tasks into smaller parts with less resistance.
6. Work in bursts: Focus on one thing for a set amount of time before switching to another, switching between work commitments and play.

]]>

Nat's 2022 Technical Link Pile: Product

2022-12-31T00:00:00Z

See the Intro for context.

[20220203] Tree/Loop Prompts

Our ability to ________ is a function of our ability to ________
________ has a number of contributing factors, including ________
When ________ [increases/decreases] we would expect to see __________ [increase/decrease]
To [increase/decrease] ___________, we have a number of options including ____________
To [increase/decrease] ___________, we have a number of options including attempting to [increase/decrease] ___________.
________ can be broken down into the following subtopics: ________
________ is an output of the following inputs: ________
________ will help us ________ which will help us __________
To positively impact ________ we might try ________ , ideally without ________ .
Our assumption (or hypothesis) that __________ can be broken down into multiple sub-assumptions (or hypotheses), including ________

[20220131] Designing WhatsApp – sample breakdown of requirements, architecture, etc.

[20220131] Plan for Tradeoffs – different quality attributes of software are rivalrous, so you should decide what your priorities are

[20220201] When MVPs hurt

[20220201] Feature release plans

[20220201] You can’t buy integration (abstract the capability, not the system)

[20220201] How to Price Projects & Manage Scope Creep

Break into Discovery, Alpha, MVP, and Ongoing BAU
You only need to define the scope of the next stage
Discovery = validate the project, deliver user needs, business objectives, what needs to be built.
Alpha = visualise the service, define how it works, and ensure users have a positive experience with it. Gives you shared vision, identify things that were overlooked, and something to test with users.
MVP = ship something, being pushy with “not in the first draft”
BAU = iterate and optimise, add new features, remove things that aren’t helpful

]]>

Nat's 2022 Technical Link Pile: Security

2022-12-31T00:00:00Z

See the Intro for context.

[20221223] PEACH Framework for Cloud Security – risks associated with customer-facing interfaces and determines: the complexity of the interface as a predictor of vulnerability;

whether the interface is shared or duplicated per tenant; what type of security boundaries are in place (e.g. hardware virtualization); how strongly these boundaries have been implemented. In order to gauge how strongly the security boundaries have been implemented (4), we propose using the following five parameters (P.E.A.C.H.): Privilege hardening, Encryption hardening, Authentication hardening, Connectivity hardening, Hygiene.

[20221223] The Decoupling Principle – to ensure privacy, information should be divided architecturally and institutionally such that each entity has only the information they need to perform their relevant function. Architectural decoupling entails splitting functionality for different fundamental actions in a system, such as decoupling authentication (proving who is allowed to use the network) from connectivity (establishing session state for communicating). Institutional decoupling entails splitting what information remains between non-colluding entities, such as distinct companies or network operators, or between a user and network peers. This decoupling makes service providers individually breach-proof, as they each have little or no sensitive data that can be lost to hackers. Put simply, the Decoupling Principle suggests always separating who you are from what you do.

[20221126] C64 MFA – The terminal window is showing a generated time-based one-time password for a full key, and the emulated 64 is showing the correct key, at the correct time, which was known and tested to be valid. Yes, you really can use your Commodore 64 for multi-factor authentication to generate TOTP codes!

[20221107] Ballerine – an open-source infrastructure for user identity and risk management.

[20220504] Serverless Security – you can use IAM to limit access to individual functions. Huh.

[20220421] Early Security for Startups – focus on the common source of breaches: Ransomware; Cloud misconfiguration/leak; Credential compromise via phishing, password reuse, etc.

[20220321] Anatomy of a JWT Token – There are a few types of JWTs, but I’ll focus on signed JWTs as they are the most common. A signed JWT may also be called a JWS.

[20220225] Creative Usernames – Spotify’s canonicalisation problem with Unicode.

[20220201] Security Engineering Course

[20220201] A Primer for Testing the Security of GraphQL APIs

[20220201] GraphQL Security

[20220201] Zero Trust Time - explanation of what zero trust is

Start by inventorying users and their needs, and authenticating them on every request
Then devices and apps.
ZT relies on tokens
ZT systems have IAM engines that validate users & assign tokens, and know what activities are permitted for which users.

]]>

Nat's 2022 Technical Link Pile: SQL Server/Databases

2022-12-31T00:00:00Z

See the Intro for context.

[20221231] Job Drain Pattern – we make heavy use of this internally. Cf In-Memory Outbox: an easy-to-use, reliable solution to perform atomic operations that update a database and send/publish messages, and it works for any database updates that are sent as commands (delivered by durable message queues).

[20221231] Make Your Database Tables Smaller – an argument against a Deleted column (namely, it permits partial deleting in a dependency tree, thus breaking foreign keys; and it makes every query more complex because you’re saying AND Deleted IS NOT NULL). Alternate proposal is to stash deleted rows as JSON in a “deleted rows” table. Different use case than us, though, because we need historic data to be consistent whereas author is only ever caring about current state.

[20221224] Recommended Approach to Encryption in SQL Server – he’s serialising a book chapter from the looks.

[20221224] ERD Intro – very basic, good for newbies.

[20221224] Database Devops – what an automated workflow looks like, and how it comes to the manual one.

[20221223] Liquibase – open source version control for database schemas with rollback and more.

[20221221] DBSnapper – for snapshotting and sanitising a database, eg to make test from live.

[20221210] Mangle – Google project, a programming language for deductive database programming. It is an extension of Datalog, with various extensions like aggregation, function calls and optional type-checking.

[20221119] Project Cambria – an isolated software layer that translates data between schemas on demand. This layer allows developers to maintain strong compatibility with many schema versions without complicating the main codebase. Translation logic is defined by composing bidirectional lenses, a kind of data transformation that can run both forward and backward.

[20221119] cozo – general-purpose, transactional, relational database that uses Datalog for query, is embeddable, and focuses on graph data and algorithms.

[20221029] Electric SQL – database for local-first apps that handles replication and schema changes.

[20221029] Lenses – central translation layer between schema.

[20221029] Data Lakehouse Paper – store in column-oriented formats like Apache Parquet, then provide fast metadata and search on top.

[20221008] Database Management with CI/CD – Commit database scripts to version control; Use database management tools; Keep changes small; Decouple deployment from data migrations; Set up continuous deployment and migration pipelines; Make migrations additive; Rollback with CI/CD; Don’t do a full backup unless it’s fast; Consider blue-green deployments. And what testing looks like.

[20220910] TigerBeetle – an open source accounting-focused database (everything is double entry!).

[20220718] T-SQL Comments – single line, block, the UI for block comment/uncommenting, and advice on when to comment.

[20220629] Things You Should Know About Databases – intro to B+ trees, read types, indexes, etc.

[20220629] PRQL – Pipelined Relational Query Language, a SQL replacement that understands pipelines, which transpiles to SQL.

[20220629] A Relational Model of Data for Large Shared Data Banks – Codd’s 1970 paper establishing relational databases and the set logic behind them.

[20220628] How to Handle Missing Information Without Using NULL – what it says on the box (via How can I avoid NULLs in my database, while also representing missing data?)

[20220504] Declarative Schema Migration for SQLite – maintain a schema in a file, build a clean DB with that schema, use db’s reflection tools to find columns tables indexes etc and diff against current state, adjust accordingly. Hacker News discussion has experiences of different ways of doing migrations.

[20220324] Replicating SQL Server – snapshot to file, reconstruct in new server, as way to avoid having to do a full copy over to switch servers.

[20220315] SQL Server I/O Basics – The purpose of this paper is to explain the Input/Output (I/O) requirements for SQL Server database file operations so that vendors and customers can evaluate and adjust their environments to meet the needs of SQL Server. Important When planning, deploying, and maintaining a Microsoft SQL Server installation, ensure that the I/O system supports all the factors outlined in this article. (Chapter 2)

[20220315] FORMAT is Expensive – other ways to have custom date formats without the performance hit of FORMAT. Materialising pre-populated column witchcraft.

[20220310] SQL Server Sends Passwords in Plaintext – when a database client logs in using SQL Server authentication (as opposed to Windows authentication), it sends the password (over not-necessarily-secure TLS).

[20220309] SQL Performance Explained – the book, apparently.

[20220309] Use the Index, Luke – free ebook by the guy who wrote “SQL Performance Explained” which really gets you going.

[20220225] On Anki’s Database Structure – an interesting critique of Anki’s database structure. We could do one for our schema too …

[20220203] varchar(n) vs varchar(max) – max can’t be indexed

[20220203] How SQL Server Executes a Query – this series will be a great introduction to what happens behind the scenes. I like episodes 1 and 2.

[20220202] Data Pages – 8k, 8096 bytes used for data (rest = header and row index). Possible to waste a lot of space if your records are enormous and don’t fit well into a page.

[20220202] Casting – use TRY_CAST or TRY_CONVERT to return NULL if the cast can’t happen. CONVERT lets you specify “style” but fk it’s ugly.

[20220201] SQL Server performance tuning using free tools

[20220201] Comparing GUID and uniqueidentifier values

[20220201] Detecting SQL Memory Pressure in 60 Seconds

[20220201] Introduction to SQL Server Sequence objects

[20220201] SQL Server Backup Examples

[20220201] SQLFacts - a free toolkit for SQL Server database engineers

[20220201] @return_value in SQL Server stored procedures

[20220201] Accelerated Data Recovery, checkpoints, SQL Server 2019

[20220201] SQL FOR loop alternatives – how to sin in SQL

[20220201] Automating triggers for data auditing in SQL Server

[20220201] Views in SQL Server 2019

Indexed view is what I think of as a view (a view with a unique clustered index). It’s as fast as querying a regular table.

[20220201] SQL Critical Care findings (PDF) – sample diagnoses from customers, including deadlocks.

]]>

Nat’s 2022 Technical Link Pile: Interesting Software

2022-12-31T00:00:00Z

See the Intro for context.

[20221231] EndBasic – BASIC with cloud, DOS-like GUI. Actively developed.

[20221231] pbproxy – send your clipboard anywhere you can ssh.

[20221231] LCARS – open source web implementation of the Star Trek UI.

[20221231] VAP – video animation player. Particle effects and transparency on a webpage.

[20221223] mdBook – create books with Markdown.

[20221126] Semantic Code Search – ask it things like ‘Where are API requests authenticated?’

[20221126] wasmer – an embeddable WASM runtime.

[20221126] d2 – diagram language and open source diagram generator from it. There a VSCode plugin.

[20221126] hurl – commandline HTTP tool that uses plain text for request and testing.

[20221126] rhasspy – open source voice assistant.

[20221119] Using ss and tcpflow to trace HTTP requests – two very cool commandline tools for understanding what’s speaking HTTP and then looking into the requests and responses.

[20221107] permify – easily structure your authorization model, store authorization data in your own servers securely, and interact with Permify API to handle all authorization questions from any of your applications. Inspired by Google’s Zanzibar.

[20221107] stepci – automate API testing and monitoring.

[20221107] metlo – an open-source security platform. Create an Inventory of all your API Endpoints; Proactively test your APIs before they go into production; Detect API attacks in real time.

[20221107] steampipe – Use SQL to query cloud infrastructure, SaaS, code, logs, and more.

[20221107] privado – an open-source static code analysis tool to discover data flows in the code. It detects more than 110 personal data elements being processed and further maps the data flow from the point of collection to “sinks” such as external third parties, databases, logs, and internal APIs.

[20221107] Common Table Expressions – nice, put the SELECT in a string, then you can say “WITH foo as ($sql)” to construct complex queries without repeating yourself.

[20221107] lama-cleaner – a free and open-source inpainting tool powered by SOTA AI model.

[20221102] Restfox – “open source PostMan”, aka Offline-first web HTTP client.

[20221029] Backstage – see the software you own and its status, and prevent orphans.

[20221028] sonic – a fast, lightweight and schema-less search backend. It ingests search texts and identifier tuples that can then be queried against in a microsecond’s time.

[20221028] answerdev – Stack Overflow clone.

[20221008] dblab – Interactive client for PostgreSQL, MySQL and SQLite3.

[20221008] Tigris – a scalable transactional document store. Perform real-time search across your data stores automatically. Build event-driven apps with real-time event streaming. All provided to you through a unified serverless API enabling you to focus on building applications and stop worrying about the data infrastructure.

[20221008] Awesome Tunneling – list of software that does tunneling. This is primarily targeted toward self-hosters and developers who want to do things like exposing a local webserver via a public domain name, with automatic HTTPS, even if behind a NAT or other restricted network.

[20221003] OpenShip – open source multichannel order fulfilment software.

[20221003] w2ui – UI widgets for modern apps. Data table, forms, toolbars, sidebar, tabs, tooltips, popups. All under 120kb (gzipped).

[20220922] Libraries.io – search open source libraries.

[20220922] Whisper – AI text to voice.

[20220921] Obsidian – Zettelkasten software.

[20220912] Berkeley Mono – typeface that harks back to terminal days.

[20220910] CastleDB – structured static database. Everything’s stored as JSON. Use case is game development so you can version control maps, level configuration, etc.

[20220910] JSON Hero – make JSON actually human-readable. Date previews and all sorts of slickness.

[20220904] Quickwit – a cloud-native search engine for log management & analytics. It is designed to be very cost-effective, easy to operate, and scale to petabytes.

[20220902] JSON Crack – turn JSON into diagrams.

[20220901] Mintlify Writer – highlight code and it generates a comment that documents it.

[20220812] StemRoller – extract musical stems from a mixed recording using Facebook’s algorithm

[20220812] AFFiNE – open source collab knowledge management

[20220807] cube.dev – open source headless BI

[20220803] clientdb – an open-source in-memory database for enabling real-time web apps.

[20220724] Symflower – integrates into your development workflow by generating high-coverage unit test suites while you are coding. Basically supports making tests as you code, rather than coming back afterward. So many questions.

[20220721] Habitica – habit-building software.

[20220718] Trillium – desktop app for hierarchical note taking to form knowledge bases, with sync. Not multiplayer but has versioning and sync so …

[20220524] LiveTerm – build terminal styled websites in minutes!

[20220504] Keycloak – RedHat-sponsored IAM software.

[20220504] RepliByte – RepliByte is built to seed a development database with production data. Replibyte helps you to: (1) Create a development dump from production; (2) Subset your production dump into a more reasonable size; (3) Hide sensitive data via customizable Transformers; (4) Make your development dump easily accessible from any remote and local databases.

[20220504] Hasura – expose SQL databases as GraphQL.

[20220421] Bubble – nocode system.

[20220421] hubfs – git repo shows up as a filesystem.

[20220421] rqlite – SQLite with RAFT consensus protocol.Uses Raft to achieve consensus across all the instances of the SQLite databases, ensuring that every change made to the system is made to a quorum of SQLite databases, or none at all.

[20220421] YDB – open-source Distributed SQL Database that combines high availability and scalability with strong consistency and ACID transactions.

[20220421] Jaeger – end-to-end distributed tracing.

[20220404] APISIX – Apache load balancer for APIs with canary release, circuit breaking, auth, observability, and more.

[20220404] envkey – manage envariables, autoreload servers if the variables change, etc.

[20220404] Quick Database Diagrams – text file DSL generates classic database diagrams.

[20220328] Convoy – fast and secure webhooks service.

[20220323] Tailwind CSS – CSS framework that had me at “most Tailwind projects ship less than 10kB of CSS to the client.” (Bootstrap was the last CSS I used, so take with grain of salt)

[20220312] Mitnal – boot to Twitter (Twitter client for UEFI)

[20220225] Explaining What Changed in a Dataset – an explanatory diff.

[20220225] automerge – CRDTs in Javascript

[20220225] jless – command-line JSON viewing/searching tool

[20220224] AppSmith – another open source RAD system. They’re all hiding authentication and test environments behind enterprise licensing, despite claiming being open source.

[20220224] s3-credentials – quickly and easily create AWS credentials (an access key and secret key) that have permission to read or write from just a single S3 bucket.

[20220219] Grist – AirTable-like software

[20220217] WebSubHub – a fully compliant WebSub Hub built that you can use to distribute live changes from various publishers.

[20220215] NocoDB – open source airtable alternative. Given a database, produces spreadsheet view, kanban, etc. etc. and APIs.

[20220213] Medusa – headless ecommerce site engine, aka customers, products, orders.

[20220209] ToolJet – open source low code solution for teams to build internal software

[20220209] Atlas – database schema migration tool

[20220207] WebVM – a server-less virtual Linux environment running fully client-side in HTML5/WebAssembly. The closed source piece is the CheerpX virtualization engine.

[20220205] Squabble – multi-player Wordle royale

[20220205] Cassowary – run Windows software in VMs on Linux and RDP in

[20220202] OneModel – fast diagramming for engineers

[20220201] Firecracker MicroVM – the tech behind AWS Lambda functions quick startup

[20220131] BudiBase – open source low code platform. Tempting for Middle Earth.

[20220131] omni – omnibar for the browser for switching, opening, etc. tabs.

[20220131] Falso – Create massive amounts of fake data in the browser and NodeJS. Tree Shakeable & Fully Typed.

[20220131] Mermaid – ASCII diagrams from Markdown

[20220131] Fig – autocomplete for the terminal

[20220128] Playwright - a framework for Web Testing and Automation. It allows testing Chromium, Firefox and WebKit with a single API. Playwright is built to enable cross-browser web automation that is ever-green, capable, reliable and fast.

[20220128] Wildcard - hack your browser like you’re in Excel

[20220128] Electric Tables - like a personal link database, but very hackable and in-browser. Nice.

[20220128] Zim - a desktop wiki

[20220201] Bookstack - open source wiki

An opinionated hierarchy of Book -> Chapter (Optional) -> Page
Great search
WYSIWYG OR Markdown supported
Great integration into Diagrams.net

[20220201] XWiki

[20220201] txtai - semantic search library w/vectors

[20220201] dsq - commandline tool for running SQL queries against JSON, CSV, Parquet, etc.

[20220201] Haystack - open source framework for building search systems that work intelligently over large document collections

[20220201] shardz - sharding coordinator

[20220201] jam - open source clubhouse

[20220201] Respawn (checkpoint and restore databases during tests)

[20220201] rclone - mount cloud storage as local filesystem

[20220201] Deskreen – any screen into another monitor

[20220201] Datascript – datalog in the browser

[20220201] Zettlr – markdown editor

[20220201] Falsisign - make it look like a PDF has been scanned and signed

[20220201] SqlServerSocket - connect to SQL Server from Dart

[20220201] Aqueduct - dart HTTP server (archived 2 years ago, ugh)

[20220201] Fly.io - tptachek works there. “Deploy app servers close to your users”

[20220201] Retool - app builder, looks real interesting. deploy on prem with Docker ($ service)

[20220201] Dura - a background process that watches your Git repositories and commits your uncommitted changes without impacting HEAD, the current branch, or the Git index (staged files). If you ever get into an “oh snap!” situation where you think you just lost days of work, checkout a dura branch and recover.

[20220201] NDepend - calculate concreteness and abstraction for your .NET classes

[20220201] DuckDB - an in-process SQL OLAP database management system

[20220201] Turbo - server-rendered applications. Sends HTML over the wire so you write less JS. From Basecamp.

]]>

Nat’s 2022 Technical Link Pile: Random

2022-12-31T00:00:00Z

See the Intro for context.

[20221231] 36 Things I Learned in 2022 – #1 blew my mind: For the first time in history in 2020, the weight of things produced by humans (concrete, metals, plastic) was greater than the weight of the global living biomass.

[20221224] Reported Sleep Duration Reveals Segmentation of the Adult Life-course into Three Phases – early adulthood (19-33yrs), mid-adulthood (34-53yrs), and late adulthood (54+yrs). They appear stable across culture, gender, education and other demographics.

[20221223] Adobe Enhance – free tool to clean up recorded voices.

[20221223] Process Reparenting in Windows – fascinating technical discussion that roams through undocumented APIs and OS data structures to be able to explain how Windows fires up a new process and sometimes leaves very worrying kernel traces.

[20221221] Public Domain Day 2023 – I’m so happy I’ve lived to a time when the material entering the public domain is good! AA Milne, Conan Doyle, Hemingway, Agatha Christie, Kafka, Metropolis, The Jazz Singer, Hitchcock’s first thriller, Laurel and Hardy, Irving Berlin, Bessie Smith, Duke Ellington.

[20221210] A Desk with its Own Schedule – don’t leave it to your better self to decide to stand.

[20221210] TicketMaster Built on VAX Architecture – the load during onsales is immense. Years ago we did an experiment to write a ticketing system in pure C. There were several locking issues among an inability to advance a stack pointer fast enough.

[20221210] Some Thoughts on Social Networking and Usenet – Usenet clients mostly worked on the assumption that you wanted to read new news, that you haven’t seen, and that if you wanted to go back and read older things, then you could.[…] Contrast current social media, including the fediverse. Most sites have an endless scroll, and if you keep scrolling for long enough, you will come down to things you have already seen. Typically there will be no indication that they are something you have already seen; you just have to remember that you’ve seen them before.

[20221210] Plumber’s Guide to Git – git is blobs and trees, and this “how it really works” is a straightforward guide to making sense of git.

[20221210] Git Notes – little-known git feature (dropped by github) for attaching arbitrary text into a commit.

[20221126] How AI Text Generation Models Are Reshaping Customer Support at Airbnb – interesting to see prompt engineering used in the wild.

[20221126] One-Time Fuses – used in Christmas tree lights and secure hardware.

[20221126] ooh.directory – a blog catalog.

[20221126] History Doesn’t Repeat, But It Does Rhyme – first formulation of this is 1965 from a psychoanalyst Theodor Reik. Not Mark Twain or any else to whom it was credited.

[20221126] Writing by Hand is Still the Best Way to Retain Information – Although typing notes can be useful and even faster for some note-takers, ultimately it does not have the cognitive, tactile, memory, or visual cognitive effects that people can get when they write by hand. Typing notes can be good, but it won’t make it easier to remember what was said later on.

[20221126] Diátaxis – a documentation framework that divides docs into Tutorials, How-To Guides, Explanations, and Reference. Each have a different use, different purpose, and different creation.

[20221126] RFC 1855 – 1995 Netiquette. Everything from UPPERCASE IS SHOUTING through “If a person doesn’t respond you might try another tty. Use finger to determine which are open. If the person still doesn’t respond, do not continue to send.” Which is advice for a long-gone Unix protocol called talk. Also, it has two spaces after every full-stop.

[20221119] Comic Mono Font – Comic Sans as a monospaced font, suitable for code.

[20221119] Story of the Unix Epoch –fascinating history of the Unix Epoch time, with diversions into electricity grids (a tick was a cycle on the power grid) and leap seconds (invented after the time epoch).

[20221119] Potluck – Turing complete documents

[20221119] German Air Traffic Control Depended on Emacs – great story.

[20221102] Does Experience Matter – no, didn’t improve results on tests.

[20221102] gpt3-cli – cli (in python) for GPT3.

[20221102] Invasive Diffusion – pointers to How To for style transfer.

[20221008] Codebase as Database – using Datalog to query your codebase.

[20220919] Falsehoods Programmers Believe About Retail – omg 2 SKUs can = a single product. How do you have a data model that isn’t RDF?

[20220901] SCARF: A Brain-Based Model for Collaborating with and Influencing Others – Status, Certainty, Autonomy, Relatedness, and Fairness.

[20220804] The Illustrated TLS 1.3 Connection – phase by phase, what each step of the connection is.

[20220804] Perception of Probability Words – the inter-quartile range of “we believe” is 40% to 100%. Helps you calibrate your use of the words by understanding how people will perceive your statements.

[20220728] Apologies that Heal – five ways to apologise, different people need different types: expressing regret; accepting responsibility; making restitution; repentance/commitment to change; asking for forgiveness. Has examples of language of each.

[20220724] Usborne Coding Books – nostalgia.

[20220719] Inside Baseball – Microsoft’s term for intra-company jargon. E.g., we might draw a distinction between Resolved and Closed on tasks, but customer doesn’t know or care about that distinction.

[20220718] Book of Experiments – fun with kids. See also The Boy Mechanic Vol I and The Boy Mechanic Vol II and The Boy Electrician and The Golden Book of Chemistry Experiments.

[20220718] A Systems Model of Anxiety-Driven Procrastination – anxiety and overwhelm form a feedback loop. Eliminating distractions doesn’t stop the loop. You need to find ways to push through: break down into smaller chunks, discover the purpose. Acknowledge the anxiety and reduce overwhelm.

[20220629] Pen and Paper Exercises in Machine Learning – The exercises are on the following topics: linear algebra, optimisation, directed graphical models, undirected graphical models, expressive power of graphical models, factor graphs and message passing, inference for hidden Markov models, model-based learning (including ICA and unnormalised models), sampling and Monte-Carlo integration, and variational inference.

[20220629] Fixing Head-Forward Posture – explanation of the biology, and five stretches that help.

[20220529] The Difficulty of DMs – arguing against a company communicating in DMs.

[20220528] 101 Tactics for Revolutionaries – become known as “the guy who…” so when the time is right, everyone knows there’s a guy who… ; realise there are no rules, you can do what you like; don’t try and do it unless you’d do it for free. They couldn’t pay you enough.; don’t call it anything. If it has a name, people, including you, will waste time arguing what it is and isn’t; call it something. Otherwise nobody can ever talk about it; learn to sell people the problem, not the solution. People buy because they have a problem.

[20220526] MainboardTerminal – a retro-style computer with a modern core.

[20220526] Framedeck – a cyberdeck with the framework mainboard.

[20220524] Deep Learning in Business Analytics: A Clash of Expectations and Reality – the adoption of deep learning is not only affected by computational complexity, lacking big data architecture, lack of transparency (black-box), and skill shortage, but also by the fact that DL does not outperform traditional ML models in the case of structured datasets with fixed-length feature vectors.

[20220524] The Craft of Writing Effectively – it’s about the reader. “Every word should create value for the reader”. Academic research and publishing needs the reader to believe that it has value.

[20220504] Russian Cinemas Showing Torrents – Western companies have pulled Russian distribution of their products, and Russian cinemas are responding by screening torrents of Western movies.

[20220504] Writing for Engineers – good advice about the why and how of writing. Not so much about how to write good English, for which I always recommend Bugs in Writing: A Guide to Debugging Your Prose.

[20220502] 103 Bits of Advice I Wish I Had Known – not all are obvious. Take note if you find yourself wondering “Where is my good knife? Or, where is my good pen?” That means you have bad ones. Get rid of those. and You can be whatever you want, so be the person who ends meetings early.

[20220422] Cross Border Employment – legal issues around remote workers.

[20220404] Printer for GitHub Issues – uses a receipt printer and a Raspberry Pi.

[20220311] Questions That Interest Me – Why are certain things getting so much more expensive? Why do there seem to be more examples of rapidly-completed major projects in the past than the present? Why is GDP growth so weirdly constant? How do you ensure an adequate replacement rate in systems that have no natural way to die? How do we help more experimental cities get started? How do people decide to make major life changes? Why are there so many successful startups in Stockholm? Is Bloom’s “Two Sigma” phenomenon real? If so, what do we do about it? How can we better understand the dynamics of progress in science? Will end-user applications ever be truly programmable? If so, how? What’s the successor to the book? And how could books be improved? What’s the successor to the scientific paper and the scientific journal? What’s the right way to understand and model personality? Could there be more good blogs? Why are programming environments still so primitive? What does religion cause? Why is there no canon for life’s most important questions? Why are so many things so much nicer in Switzerland and Japan? Why isn’t China (yet) producing a lot of top-tier research? Why don’t we build nice neighborhoods any more? What influences when people act in accordance with their self-interest and when they don’t? What’s going on with infrastructure? Why did climatic variability suddenly decline in the Holocene period?

[20220310] Kettle Logic – accused of stealing a kettle, the defense is “I returned it undamaged; it was damaged when I got it; I never took it in the first place.” Multiple contradictory arguments to defend a point = kettle logic.

[20220306] How to Slice a Bagel into Two Linked Halves – has more surface area than a straight cut so you get more cream cheese per bagel.

[20220306] Hoare and Knuth Quote Story – Knuth did say it originally, but Hoare repeated it (in writing), properly attributing it to Knuth. Knuth then read Hoare’s quote, missed the attribution, forgot that he was the one who said it, and repeated it again in writing, mis-attributing it to Hoare. The quote: “premature optimization is the root of all evil.”

[20220225] Figureheads, Factions, and Fictions – Auckland U research into mis/dis info and protests.

[20220222] Designing the New Way – Ask anyone in a complex organisation if there is a way to improve effectiveness they will almost always have an answer and it will almost always start with constructing an enormous pyramid of human skulls atop which they and the practitioners of their discipline sit atop as gods.

[20220219] MLPOS – Market Leading Piece of Shit, eg Jira.

[20220219] Locked Room Mysteries – the Michael Jordan of the genre, John Dickson Carr. Hated math, there’s a story about him punching a friend just for using the word algebra. Carr got his publishing contract by wagering a dinner that the CEO of Harper & Brothers couldn’t guess whodunnit. The final ⅓ of his book came in a paper wrapper with a promise to refund purchase price if it was unopened.

[20220218] Russian Smiles – far more than you wanted to know about Russians and (not) smiling.qa

[20220217] Tales of an Early Engineer – if you were looking at the team and the company today, you would be forgiven for believing that we knew what we were doing all along. You’d also definitely have trouble imagining just how fragile and humble things in the early days were.

[20220217] Acute aerobic exercise to recover from mental exhaustion - a randomized controlled trial – The empirical results showed that moderate aerobic exercise led to a better recovery for cognitive flexibility (mean difference divided by pooled standard deviation, Cohen’s d= 0.737), mood (d= 0.405), tiredness (d= 0.480), self-perceived cognitive capacity (d= 0.214), and motivation (d= 0.524) compared to active control treatment. Moderate aerobic exercise was also more effective than passive control treatment (d= 0.102 - 0.286) with the exemption of tiredness (d= 0.015) and restlessness (d = -0.473).

[20220217] Man Lives In Futuristic Sci-Fi World Where All His Interactions Take Place In Cyberspace – brilliant.

[20220216] Mr Mouse and Mr Owl Situations – “no idea what mr mouse’s problem is, mr owl has never done a thing to me”.

[20220216] Hackers Card Game – print it yourself card game.

[20220214] Intergenerational Mobility in the Very Long Run – Hungary: Family status held up under both Communists & capitalists. Florence: Being a descendant of a family with high income in the FIFTEENTH CENTURY means you have significantly higher income today. (via Twitter)

[20220213] Enterprise Programming Language – humour, or is it?

[20220213] 25 Mental Models – “problem selling” is great.

[20220213] Amazingly Detailed Cryptocurrency Is Bullshit – “Cryptocurrencies’ roots lie deep in the libertarian culture of Silicon Valley and the cypherpunks. Libertarianism’s attraction is based on ignoring externalities, and cryptocurrencies are no exception.”

[20220209] The Ostrich Algorithm – a strategy of ignoring potential problems on the basis that they may be exceedingly rare. It is named after the ostrich effect which is defined as “to stick one’s head in the sand and pretend there is no problem”. It is used when it is more cost-effective to allow the problem to occur than to attempt its prevention.

[20220209] Ethan Coen Reviews Joel Cohen’s Macbeth – hilarious (not actually by Ethan, alas, just good comedy)

[20220209] Tim O’Reilly on Web3 – I don’t think we’re going to be able to call Web3 “Web3” until after the crypto bust. Because only then will we get to see what’s stuck around. I like to contrast the betting economy and the operating economy. The operating economy is the one that delivers goods and services and people pay for them. And the betting economy is where you guess at what the value might be, or what other people think it might be.

[20220209] Data and Reality – tease of an interesting book (from 1978!) on data modelling

[20220207] SQL Police Department – solve crimes, learn SQL.

[20220207] A Treatise on Northern Ireland, volume 3, Consociation and Confederation – one statistic stands out from a census comparison from the first decade of the twenty-first century. Just 2.1 percent of the population in Northern Ireland were born in the South, and just 1.3 percent living in the South were born in the North.

[20220207] The Integrative Role of the Sigh in Psychology, Physiology, Pathology, and Neurobiology – Increased breathing irregularity may provoke excessive sighing and hyperarousal, a behavioral sequence that may play a role in panic disorders.

[20220207] Reading on a smartphone affects sigh generation, brain activity, and comprehension – reading on a smartphone affected sigh frequency but not normal breathing, suggesting that normal breathing and sigh generation are mediated by pathways differentially influenced by the visual environment. A path analysis suggests that the interactive relationship between sigh inhibition and overactivity in the prefrontal cortex causes comprehension decline.

[20220207] Omicron and Immunity – a really understandable intro to how immunity works, in the context of Omicron.

[20220207] Stubborn Attachments review – What other Most Important Priorities have you read about which give the average person a substantial role to play in the most important tasks ahead of us? What a rare thing! Everyone can be a steward of the glorious and wealthy future: everyone who upholds norms and keeps society a little more functional, everyone who makes a couple clever optimizations, everyone who safeguards some knowledge and passes it down to someone who can make more and better use of it, everyone who raises strong and capable and moral children. All of these are crucial tasks that can be fulfilled by a huge swath of the population.

[20220206] The Virtual Sales Handbook – you might check at the end for Completion (anything left to say), Alignment (everyone happy with where we got to), Next Steps, and Value (what are you taking away from this so far?).

[20220205] Where Americans Were Born – 14% 1st gen immigrant, 28% born in US but now living in a different state, 58% living in same state they were born in (!). Only slightly changed from 1850 when it was 11/23/66% instead of 14/28/58. 70s had immigration low (5% born outside the country), 1940 was peak “living in the same state you were born in”.

[20220205] Learning What You Don’t Know by Virtual Outlier Synthesis (VOS) – training a neural network to say “I don’t know” instead of guessing when it has no idea.

[20220205] The Website is Down – so many reasons why it might be unavailable to the user

[20220203] How to not get lost in your first foundational research – what I wish I’d had.

[20220202] Coercion Theory: A Basic Introduction for Practitioners – By its nature, coercion requires a decision by the actor being coerced, thus placing the outcome in the actor’s hands. But coercion, especially compellence, is difficult, and provides no guarantee of success — not even to very powerful actors. Reviewing five empirical studies of coercion, Downes concludes that compellence succeeds only about percent of the time. Actors can deter by threat of punishment or by threat of denial. It is never clear whether the absence of an attack is due to an enemy giving in to a deterrent threat. This ambiguity enables enemies who have been deterred to save face. Because compellence is active in ways that deterrence is not — the target state must perform an act rather than simply refrain from one — it is clear to all when compellence is successful. Moreover, the actor being compelled is usually being forced into some degree of humiliation. When it comes to timing deterrence can be indefinite while compellence, by contrast, must be definite – if compliance is being demanded, then how much, and for how long? The target state must be convinced that if it resists it will suffer, but if it concedes it will not. If it suffers either way, or if it has already suffered all it can, then it will not concede and coercion will fail. In general, Pape’s work found that punishment, risk, and decapitation all have problems as methods of aerial coercion. Leaders attached to a particular stake are often willing to pass the pain on to their populations in order to protect the survival of the regime. Since World War I, no industrial nation has been able to fight a peer competitor successfully without the ability to largely control its own airspace and contest enemy airspace.

[20220202] How to Explain an Idea – A simple trick I once learned is to structure the explanation into four parts, with one sentence for each part: (1) state the problem, (2) state the consequences of the problem, (3) state the solution, (4) state the consequences of the solution. Since the explanation now automatically includes both the problem and the solution, it usually is both more compelling and easier to understand. Followup: This is succinctly captured by a phrase from the video game industry: “Show locked doors before you show a key”.

[20220201] Giant Pyrosome – clones related to salps that form an organism that’s a chordate (has a spine). Yow.

[20220201] The New Hire Who Showed Up Is Not The Same Person We Interviewed – wow.

[20220131] You Don’t Know GIF – far more than you ever wanted to know about the GIF format.

[20220131] EVerest – EV charging software stack.

[20220131] Become a better writer in tech – “ask directed questions” is the best advice.

[20220131] Canon’s telling people how to circumvent their toner DRM – a shortage of official toner means they have to else customers can’t use their printers

[20220128] How much did a tunic cost in the Roman Empire? – 6-12 days of labour aka $500-1500 for simple tunic or $7k-21k for finest. “Robber” and “robe” share a common origin because grave robbing for clothes was common practice. Clothing a typical Roman peasant family would take ~3k hours of domestic labour a year every year, most of it devoted to spinning flax. From HN discussion: “The first engines were atmospheric pressure, e.g. they used a vacuum to work. They were not closed cycle and they did not care about wasting water seeing as their purpose was to be a water pump. They didn’t care about wasting coal as they worked at a coal mine. For the first century of their development they were stuck in the one niche where both the things they required were free.”

[20220128] We have jetpacks and we do not care – what it’s like to fly a jetpack

[20220128] Information Hazards: A Typology of Potential Harms from Knowledge - ways in which truth can be harmful.

[20220128] A Brief History and Ethos of the Digital Garden - “newly revived philosophy for publishing personal knowledge on the web”.

[20220201] Militarized Dolphins Protect almost a quarter of us nuclear stockpile

[20220201] What Romans Found Funny

[20220201] How to spot a conspiracy theory

[20220201] Why Agatha Christie could afford a maid and a nanny but not a car – goods have dropped in price while labour has risen in price; “Baumol’s cost disease” = people haven’t become more productive yet their wages are higher.

[20220201] Algorithmic Amplification of Politics on Twitter

[20220201] Inside a Boeing 777’s data center

[20220201] Gridrunner (game)

[20220201] Acra - database encryption at application level

[20220201] When - timezone utility in the commandline

[20220201] Open source tools from 2021

[20220201] Making good pizza

[20220201] TMC music reccs

[20220201] Gymnastic commentary

[20220201] History of the blinking cursor

[20220201] Neal Stephenson’s recommended six books on information manipulation

[20220201] table of lotr

[20220201] Increase alcohol consumption in women under 50 with kids at home during lockdown

]]>

Acoustic Bandcamp Recommendations

2022-05-02T00:00:00Z

Last updated: 2022-04-10

Bandcamp is a great place to buy music, because most of the money you pays goes straight to the artist. If you buy on a special Bandcamp Friday, then Bandcamp don’t even take their usual commission on the sale – all the money goes straight to the artist.

I like acoustic stringband music, with or without vocals. I go beyond those boundaries for great music. Here are some sounds you should check out. It’s a mixture of famous and up-and-coming, instrumental and vocal, boundary-pushing and traditionally brilliant.

Molly Tuttle’s Bluegrass Record

Crooked Tree is solid gold from Molly Tuttle. She’s assembled an incredible band and they have been just SMASHING it on the road. Finally (Apr 1) the album is out and I am rejoicing! Look for this in the awards season.

Soulful modern acoustic music

The Lonely Heartstring Band got their start playing Beatles covers on bluegrass instruments, and went on to write their own original material. They are my favourite band ever, and they have their early EP (with some of the Beatles covers) as well as their second studio album.

Amazing guitar work

Julian Lage and Chris Eldridge have a proof-of-concept EP that showcases these two talented guitarists. Lage is from a jazz school, Eldridge from bluegrass. Both are on the cutting edge of what can be done with a flatpick. Their duets and duels are to be savoured. Best with a rye whiskey. Their later albums are more polished and even more virtuosic.

Old time

Jake Blount is a young Black fiddler and banjo player, who specialises in the overlooked black stringband tradition. His music soars and surges with youthful energy and the hunger to be heard of this long-overlooked music.

Bluegrass with a modern twist

Mile Twelve are a young bluegrass band out of Cambridge MA, that sound like classic bluegrass for modern ears. They have an album of covers, recorded with special guests that include Billy Strings and Chris Eldridge.

Acoustic covers of music from classic games

The Hit Points take the tunes you love from games like Final Fantasy, Super Mario World, and Skyrim, and play them beautifully on acoustic instruments. And, because that’s not astonishing enough, they improvise solos in the middle. It’s beautiful music and intense genius.

Queen of the vibe

Reb Fountain crafts strong songs and frames her beautiful voice with sparse electric arrangements. She opened for Crowded House on their recent tour, and is going places.

Challenging progressive fiddle

John Mailander pushes the boundaries of bluegrass fiddle. He can do tasteful, he can do wild, he can do beautiful, he can defy categorization. His offerings on Bandcamp vary from delicious modern stringband to very experimental jazz that stretched my ears.

Stretching new borders

Joe Troop is an amazing new talent in traditional music. His regular project is a Grammy-nominated mixture of Latin American music with North American folk styles, and this solo project crackles with the same originality and talent.

Golden voice

Aoife O’Donovan has the pure and husky voice of a fallen angel. Every note is beautiful, every song a masterpiece. One of our generations finest talents. Her latest is the pandemic processing “Age of Apathy”.

The future of the banjo

Max Allard is a young Chicago banjo player with an astonishing and original voice. He’s inspired by everything from Japanese game music to the piano of Chilly Gonzales. His music is moody and not at all what you think of when you think “banjo”. But it will be.

Mandolin to stretch your ears

Ethan Setiawan is another northeastern prodigy stretching the boundaries of the acoustic instrument. He has several albums that are not only virtuosic but also pleasantly listenable.

Bluegrass from New Zealand

The Pipi Pickers is my family band. We play mostly modern (since 1990, lol) bluegrass songs, and we hope that our love of them comes through.

Original old-time fiddle

George Jackson is a gifted fiddler from New Zealand, now living in Nashville. He had a viral hit with the tune ‘Dorrigo’, and has made several albums of excellent original fiddle tunes.

Just because she’s famous doesn’t mean she isn’t exquisitely good

Phoebe Bridgers sings modern folk pop, has recorded with Taylor Swift, and is exquisitely good. Beautiful voice, magnificent songs, delicious works of art. Start with Motion Sickness and you won’t stop.

Americana funk

Twisted Pine have pop sensibilities, deep groove, and traditional instruments. It’s a glorious combination.

Epic original newgrass

Julian Pinelli is a fiddle player who studied under a master, and has developed a beautiful and original sound. He surrounded himself with exquisite young talent for this stunning first album. I love it all.

Chambergrass

Westbound Situation plays at the intersection of classical and acoustic stringband music. It’s like chamber music for cello, bass, violin, and banjo. Moving and delightful.

Living master

Tony Trischka defined modern banjo and his album “Territory” makes a strong case that nobody does it better. “Fox Chase” is my favourite track but you can’t go wrong on an album that spans gourd banjo, classic style, and full-on bluegrass bangers.

Modern mandolin

Thomas Cassell is a talented young mandolinist who just crushes it. He plays every style and genre, and you can hear him finding his own voice over the course of his albums.

Jazz-infused trio

Jon Stickley Trio play at the border of jazz and bluegrass. Fiddle, guitar, and drums use effects to make a bigger and wilder sound than you’d imagine possible with those instruments.

Bold and beautiful new voice

Sarah Jarosz is a singer and songwriter whose roots are in bluegrass but who now plays with genre as readily as her music plays with our hearts. The commissioned song cycle Blue Heron Suite is a tour de force, but don’t overlook the U2 cover I Still Haven’t Found What I’m Looking For.

A Banjo composer

Ben Krakauer is an ethnomusicologist who composes for the five-string banjo. His music blends the familiar with the unfamiliar, and is delightful.

Recording lost tunes

The John Hartford Fiddle Tune Project assembles kick-ass musicians to record the fiddle tunes that John Hartford wrote but never recorded. Hartford (who wrote “Gentle On My Mind” and leave a legacy of quirky original bluegrass) left notebooks full of these fiddle tunes and this is the first time any of us have had a chance to hear them.

Unconventional traditions

Jayme Stone hit my radar for his sensitive and thoughtful Lomax Project but he’s also experimented with lush pop production. Each time there’s a new Jayme album, I don’t know what it’ll be but I do know that I’ll love it.

Powerful new voices

Hank, Pattie, and The Current are a North Carolina band that play original contemporary bluegrass, sometimes with an orchestra.

Powerful minimalist folk

Nadia Reid uses her guitar, voice, and songwriting skills to great effect. Her most recent release is Out of My Province.

Her Dylan covers will melt your mind

Emma Swift recorded an album of Dylan covers, with Patrick Sansone from Wilco and Robyn Hitchcock guesting.

Banjo in unfamiliar genres

Jake Schepps’s output spans bluegrass, Brazilian choro, classical, Romanian Christmas carols, and more. His touch is deft and his taste impeccable. Check out Entwined for “some of the first ever long-form works written by classical composers for the traditional 5-piece bluegrass string band”.

Stretching out

You, Me, Everybody is the progressive bluegrass band from New Zealand that I play banjo in. We make original bluegrass inspired by Tony Rice, Alison Krauss, and Béla Fleck.

Jazz on bluegrass instruments

Bluegrass and the Abstract Truth shoots for the stars and lands somewhere beautiful and challenging. Bluegrass has a history of borrowing from jazz, but here they go full in.

Songs about a modern difficult faith

Chris Thile is simply the best mandolinist alive today, and he has the MacArthur Genius Grant to prove it. During the lockdown he released a song of songs inspired by his challenged Christian faith. (And a solo Bartok piece)

Folk rock for folk to rock out to

Albi and the Wolves play Mumford and Sons-inspired folk rock. Prodigy fiddle, strong harmonies, and catchy grooves aplenty here.

Old time for the best time

Allison de Groot and Tatiana Hargreaves are two young musicians who bring us nontraditional traditions – music from history that lies outside the white male traditional zone.

Jazz standards on string band instruments

Jacob Jolliff is a devastating mandolin talent who here applies himself to the world of jazz standards. This is that rare album – mellow and delightful to have in the background, inspiring and infinitely rewarding in the foreground.

Groove to your soul

Tony Furtado started as a banjo player with a rhythmic right hand, and became a powerful new voice with songs and the slide guitar. He has both guitar and banjo albums here, and all repay your time.

Beautiful voices in glorious original songs

The Danberrys are a husband-and-wife duo with exquisite voices and a taste for songs that move you. Their entire catalog is here, and it’s all mint. Start with Give and Receive but don’t go past Songs We Sing, which has some beautiful covers of songs you probably already know.

Energetic progressive bluegrass

Kyle Tuttle is a banjo player who jams out with the best of them. His recordings will make your body move, and your ears happy.

The GOAT herself

Gillian Welch has her entire catalogue here. It’s all amazing, but start with Revival where she blew everyone away, move to Time The Revelator which Chris Thile described as “our generation’s White Album”, and then soak in anything.

From Barcelona with love

Lluís Gómez is a Barcelona banjo player. Check out Flamengrass for his latest masterpiece, tastefully bringing the distinctive flamenco sounds to the American string band.

Intricate, Inspired, and Infamous

Punch Brothers are individually brilliant and collectively magnificent. Many albums to choose from, all brilliant. Three tracks to see if you’ll like them: Julep, Rye Whiskey, Three Dots and a Dash.

Musicians musicians

Hawktail are the band that, at bluegrass festivals, all the performing artists pile into the front rows to see. Beautifully composed, arranged, and performed music.

Mandolin every day

David Benedict was mandolinist for Mile Twelve (see elsewhere). Clean, precise, profoundly musical, and always interesting instrumental music.

Music for foxes

Andy Thorn became viral sensation with a clip of him playing music for a wild fox, but he’s been a gifted musician long before that.

Queen of tomorrow

Sierra Hull has a pure and powerful voice, often compared to her mentor Alison Krauss. Sierra plays a mean mandolin and writes wonderful songs, and has become a fearless explorer of where those gifts can take her. Start with Weighted Mind and compare to The Original Sessions to hear how much of an effect production can have.

Bubblicious

Scroggins and Rose is a duet of Tristan Scroggins on mandolin and Alisa Rose on violin.

Progressive and historic

Joy Kills Sorrow haven’t played since 2014 but people are still listening. Pop bluegrass that doesn’t lose musicality in its stretches.

Delightful duo

The Foreign Landers are David on mandolin and Tabitha on banjo. Individually great, together exquisite.

Original jazz done right on bluegrass instruments

Jeff Picker is a jazz-trained bass player whose day job is with the very traditional Ricky Skaggs and Kentucky Thunder. By night he’s been writing jazz and here it is, performed by top-tier talent.

All-American singer-songwriter

Darrell Scott is a beautiful original talent. His catalogue here has something for everyone. I discovered him via Real Time (duets with Tim O’Brien) but came to appreciate everything he’s done.

Strong young bluegrass talentAJ Lee & Blue Summit are a young Californian bluegrass band. AJ’s divine vocals are ably supported by the rest of this deeply musical band. I love every track.

]]>

5G Public Works Project

2020-05-13T00:00:00Z

I’ve seen a few people propose a big government project around 5G mobile technology. I couldn’t find more detail to any proposals, so I did my own research to inform my own opinions on what’s needed and what’s not. I wrote up where I landed so I won’t forget it, and I’m sharing in case it’s useful to you. If you have corrections, etc. please contact me on Twitter as @gnat or in email (nathan@torkington.com).

Abstract

In the runup to the 2017 general election, there was talk of government investment in 5G networks to get greater coverage in rural and remote areas. MBIE canvassed opinion as part of a consultation around spectrum auctions, but took no action. This idea has resurfaced as a Public Works project to “get New Zealand moving again”. This paper is a backgrounder for the issue and decision points.

Mobile Technology: A Primer

Spectrum

Mobile phones communicate with cell towers using radio waves (often referred to as “spectrum”). Radio waves are identified by their frequency, and different services use different groups of frequencies (“bands”). Lower frequencies travel further.

In New Zealand, spectrum is property. NZG owns rights to the whole spectrum and leases it to companies that want to use it. The commercial terms of the lease are determined at an auction. There are many uses and users of spectrum, not just the largest mobile operators.

Generations

Mobile technology evolves. Roughly every decade there’s a new generation of technology: 80s = 1G, 90s = 2G, 00s = 3G, 10s = 4G, 20s = 5G. This is partially because of R&D time but also because each generation represents massive capital investment by mobile operators. Each G brings faster downloads, lower power consumption, and more smarts to prevent dropouts, etc.

The technical standard for a generation says how to use spectrum to transfer data, make a call, etc. This standard could theoretically be used on any band of frequencies, but because commodity network technologies like mobile phones require interoperability, the International Telecommunication Union standardises some frequency ranges for each generation. Operators could deploy 5G over the same frequencies that 4G phones are using, but can’t because that band of frequencies is already heavily used.

5G

There are three relevant new-to-telephony bands of frequencies for 5G in NZ:

600MHz, which gives strong signal over long distances. Vodafone indicates this will play a role in rural areas, Spark suggests low-power Internet-of-Things (IOT) networks might operate in this band: “Low-band spectrum such as 600 MHz are likely to be for IoT use cases where large numbers of devices with very wide coverage footprint are required, i.e. for logistics, asset tracking, and smart grid applications. It is only when we get to needing tens of millions of devices that existing LTE and LoRa technologies will have limitations.”
3.5GHz, which has similar signal strength:distance properties as 4G users currently experience. This will be the workhorse in urban and town areas.
26GHz, which travels short distances (<500m) but can carry vast amounts of data. This is foreseen to be useful in shopping centres, convention centres, campuses, stadia, and other high-density gatherings. It’s unclear whether anyone seriously proposes building a high-density network of towers for 26GHz delivery. It could also be used to provide fixed mobile substitution, aka Wireless Fibre, which would allow retail telcos like Spark & Vodafone to provide multi-gigabit home services without having to pay Chorus for fibre.

MBIE has released some 3.5GHz spectrum for operators to experiment with 5G, and there will be more substantial 3.5GHz auctions in the future. Different operators have different rollout priorities: Vodafone signalled 3.5GHz first, then 600MHz, then 26GHz; but Spark signalled 3.5GHz then 26GHz then 600MHz.

Existing towers can be refitted to provide 3.5GHz and 600MHz service. The high-density nature of 26GHz will require new towers or (more likely) deployment of micro-sites in the built infrastructure of the venues being serviced.

There are 3.5GHz handsets on the market, and they’re used in the mobile operators’ trials of 5G. The 600Mhz frequency band hasn’t been blessed by the ITU yet (expected in 2023). There are no 26GHz handsets: only dongles and specialty hardware.

Current Structure of NZ’s Mobile Industry

There are three players: Spark (41%), Vodafone (36%), 2Degrees (23%). Spark is listed on the NZX. Vodafone is co-owned by Infratil and NYSE-listed Brookfield Asset Management (US PE). 2Degrees is 1% owned by the Hautaki Trust, 26% by Dutch holding company Tesbrit BV, and 73% by TSX-listed Trilogy International Partners (Companies Office). Vodafone and Spark have strong coverage around New Zealand. 2degrees has built most of their 3G/4G network around urban areas, and in many rural areas roams on Vodafone using 3G.

Each of the three mobile operators have an equal share in a joint venture, the Rural Communications Group. This is a mechanism for building and sharing the physical infrastructure of cell towers. The commercial terms of sharing infrastructure is set by the Commerce Commission. The RCG deploys shared 4G networks:

In August 2017 the Rural Connectivity Group was appointed by the government to be the infrastructure provider to bring 4G mobile and wireless broadband coverage to rural New Zealand under the Rural Broadband Initiative 2 and the Mobile Black Spot fund. The Rural Connectivity Group is an independent entity established to build the infrastructure shared by New Zealand’s three mobile network operators and to operate and maintain the new open access network. In partnership with Crown Infrastructure Partners we will deliver new mobile and wireless broadband coverage to at least 30,000 rural homes and businesses, provide mobile coverage to a further 1000 kilometres of state highways and provide connectivity to at least 90 top New Zealand tourist destinations by December 2022. This critical infrastructure project is funded by the government’s Telecommunications Development Levy, and an additional $75M provided by the three mobile network operators. Crown Infrastructure Partners will manage the distribution of funds to the Rural Connectivity Group. This unique funding model is the only way these modern services can be delivered further into rural New Zealand. (Source: RCG website)

Ultrafast Broadband

Separate from the mobile networks, in 2009 the New Zealand Government set up Crown Fibre Holdings (a Crown Entity, now called Crown Infrastructure Partners) to contract with fibre companies to build a fibre network. The goal was fast broadband for most of New Zealand, hence the project name “Ultrafast Broadband”. Chorus won most of the business, with various council and lines company partnerships (“Local Fibre Companies”) accounting for the rest.

Chorus received $929M, 50% as interest-free loans and 50% as the purchase of non-voting shares. LFCs are 50% owned by Crown Fibre and 50% by the other partners.

Chorus operates the fibre infrastructure and has wholesale offerings to Retail Service Providers (RSPs), the companies that then sell Internet access to the public and to customers. The wholesale prices and Chorus ownership (it can’t be owned by an RSP) are the subject of Government regulation. This structure then creates a framework for retail competition resulting in low prices for fast and improving connectivity for customers. Indeed, NZ’s broadband speeds and penetration compare favourably internationally (especially given our population density distribution).

As part of Ultrafast Broadband project, the Rural Broadband Initiative was established. This funds the provision of connectivity (fibre or cellular) to very low-density areas. The RCG was established to implement the cellular portion.

Chorus has proposed it operate NZ’s mobile network infrastructure, by analogy with its fibre operation. Existing mobile operators (who presently own their own infrastructure) have resisted, pointing out that the initial driver of 5G uptake is likely to be replacing fixed-line broadband and this would place Chorus in the position of having to invest in mobile infrastructure that competes with its existing fibre infrastructure. Chorus also owns the legacy copper network, which situation requires large amounts of regulation to maintain consumer benefit.

Shared Infrastructure

By analogy with the success of Chorus, a Public Works project would seek to contract with an infrastructure operator to erect the cell-towers and backhaul (the connection to the rest of the mobile network, generally fibre).

Active infrastructure does the communicating: antenna (“Radio Access Network”), fibre, and possibly even spectrum. Passive infrastructure is everything else: the radio mast, cabinets, power supply, alarms, etc. There are two models for shared infrastructure, corresponding to what is shared: passive and active. Active sharing is what interests us here.

There are many examples of 3G, 4G, and 5G active sharing in networks around the world. It’s an old and well-established practice. See, for example, this GSM Alliance paper on Mobile Infrastructure Sharing (2012). (MNO = Mobile Network Operator)

[A]ll forms of infrastructure sharing are usually characterised by increased efficiency in the use of resources, where capacity exists. […] This will have benefits for the wider economy as land is available for other uses, in particular in densely populated areas where little space is available.

Similarly, all forms of infrastructure sharing generally reduce costs and prices to consumers as they reduce the investment lay-out and opex required to provide a given level of service. […] Regarding RAN sharing, the cost savings […] of up to 30% indicate that prices to consumers are likely to be affected positively through competition.

However, there are very few instances where all the MNOs in a single market share the same infrastructure. New Zealand has broken new ground here with RCG. From the RCG website:

Government and industry collaboration will see New Zealand as the first country in the world where all three mobile network operators will share radio access network equipment and one set of antennae on each facility built by the Rural Connectivity Group.

Poland is considering a national 5G network (source). There is more work to be done in gathering international comparators.

Objectives of a Public Works Program

While not clearly articulated anywhere, I understand the goals of a 5G Public Works project to be:

NZ Government wants to put money into the New Zealand economy to stimulate recovery from the Covid-19 recession;
Citizens want to get better 5G coverage for that money;
The government wants to maintain/improve competition;
Government wants minimum waste (i.e., diversion to shareholder profits) from its investment.

Goal (b) implies citizens want the benefits of a 5G rollout, primarily the fast download speeds. Some commentators have also mentioned the intangible value of international visitors being able to use their phones without seeing [LTE] as a marker of previous-generation service.

There is no reason to suggest that commercial self-interest and competition would not drive a successful build-out of 5G in NZ’s urban areas. Existing towers can be repurposed.

There’s also no reason to suspect that operators would rush to deliver 5G to smaller towns and rural areas. Being less densely populated, the commercial return on that infrastructure investment is lower. (This tardiness is what drove the Rural Broadband Initiative). That said, this is an expectation not a guarantee: the UFB and RBI were solutions to problems that were already being felt, whereas a 5G Public Works project would be a pre-emptive action.

This means a Public Works project would be to bring faster mobile data to smaller towns and rural areas faster than competition alone would deliver.

Because 600MHz and 3.5GHz reuses existing 4G infrastructure, this Public Works project will not engage the construction industry during the Covid-19 recession. Instead, the immediate spend will be on equipment from suppliers such as Nokia and Alcatel-Lucent, and the contractors who install it. Building 26GHz would not trigger massive construction either, as it is for dense urban environments, which typically have radio masts on buildings rather than on standalone towers.

Implementation

The opportunity here is to accelerate 5G deployment in smaller towns and rural areas. This would mean 3.5GHz upgrades and 600MHz coverage (when it is standardised). The Rural Connectivity Group provides the model for Government-subsidized rural mobile connectivity. The Rural Broadband Initiative is a proven mechanism for influencing mobile deployment.

As with UFB, the implementation of the investment trades the amount of infrastructure built for control over how it is used. Grants maximise amount of infrastructure, whereas direct investment gives control. (Carter, 2012).

Implementation decisions:

The ownership of RCG: currently it’s a partnership of the mobile companies. How would RCG deal with new entrants to the market? What if one mobile company bought another, would that enable it to exploit RCG? It is important to maintain competitiveness in the mobile market.
The beneficiaries of the intervention. For which customers is commercial self-interest expected to fail to deliver timely 5G? It’s apparent that major cities will see the first roll-outs. Would this project cover large towns like New Plymouth? Small towns like Warkworth? Rural areas like Whangamomona? Highways between townships like SH38 through Te Urewera. What are the Treaty of Waitangi obligations of such an investment?

The first step should be a timeline model of expected 5G rollout from the operators. This would guide the beneficiaries decision. Next, a cost-benefit analysis of RBI-like subsidisation of those beneficiaries’ 5G access. The information from these two activities is needed to form a judgement on advisability and affordability.

Dependencies

Economic development and the mobile operators are not the only concerned parties here.

Māori for whom spectrum remains “an unresolved issue in the Māori-Crown relationship”. More specifically: “A Māori Spectrum Working Group (MSWG) has been formed representing key Māori stakeholders, including Treaty of Waitangi claimants. This group is working jointly with Ministers and officials to explore shared MāoriCrown interests in spectrum. […] Four Waitangi Tribunal claims relating to spectrum have been lodged since 1984 centring on spectrum as a taonga of special value to Māori with the WAI 2224 claim (lodged in 2009) still outstanding.” (Cabinet paper, 2019).
The ITU is yet to designate the 600MHz band for 5G.
Mobile hardware manufacturers offer 3.5GHz connectivity, but not 600MHz or 26GHz yet. 600MHz would come after ITU designation. 26GHz may never arrive.

Acknowledgements

Thanks to Jon Brewer for technical insight, and James Ting-Edwards for review.

]]>

Joining a startup? Read this.

2019-01-23T00:00:00Z

Here’s a braindump of some things I’ve learned over the years. Caveat lector: my experiences inform this, and they aren’t representative of all the possibilities, etc. Ignore this if you want; I’ve been meaning to write some notes for Kiwis or Aussies who are joining startups for the first time, and someone just became my excuse.

Maturity

Startups are very immature organisations, unlike a corporate or university – I always describe it as “there’s more work to do than there are people to do it”. So there’s huge opportunity to do stuff you wouldn’t ordinarily do, which is great. There’s also a lot less support, which can be unnerving. There’s nothing preventing you from working super-long hours and burning out, and only your coworkers to notice (HR is almost non-existent in very early stage startups). And even compensation is weird (“stock options”?) and can bite you if you’re not prepared.

Startups have a life-cycle

First you do R&D until you’ve found a product. Then you make and sell lots of that product. A product is something you can make and sell to someone for more than it costs to make it. Startups that have taken investors’ money typically need to grow to be really big (billion dollar companies!). It’s widely reported that it takes seven years for a startup to get to the point where it’s considered “big” (it’s worth hundreds of millions of dollars), there just aren’t a whole lot (or perhaps any) overnight successes.
The point where you’ve found the thing you’ll sell lots of, is called “product-market fit”. There’s a good book “The Lean Startup” that’s aimed at software startups, but has some applicability to other aspects. You might enjoy it. It’ll certainly help you understand the wider Valley mindset.
R&D is difficult because you don’t know how many problems you have to solve before it’s viable. Academics know this, I’m sure. It’s a bigger and riskier roll of the dice for an investor to fund a company that’s making something that hasn’t been invented yet. Now you have TWO problems: can you do an unknown amount science/engineering to get to something that works and can be mass-made AND can you find a thing that people want to buy.
(Turns out, building something people want to buy is often the harder problem. A business isn’t worth much if nobody needs what it is selling.)
Be on the look out for products that aren’t the one you thought you were going to build. Sometimes you fail to invent a good glue, but you’ve invent Post-It notes. Whether your investors will go for it depends on the investor (“hey, we funded an X company, not a Y company!”) but generally they’ll be happy if they think you’ve found a path to a billion-dollar business that doesn’t involve them losing their ownership percentage.
The early stage, pre-product-market-fit, is exciting grad school type of thing. The growth stage, once it’s got a product and selling gangbusters, is exciting like a rocketship ride. Both are fun, different experiences.

Startups attract interesting people

Often your managers have no experience managing people. Or nobody’s a manager.
The driven egotist who thinks they can build a company to invent something and take over the world … is often strong in a few character attributes, but deficient in others. You probably won’t discover the deficiencies until you’ve been working alongside them.

Look after you

There’ll be more work to do than people and hours. So you could burn out trying to do it all. But don’t, burn out is miserable.
Burn out for most people feels like their seeing the world in grey, nothing is exciting, they feel exhausted all the time, and dark thoughts enter their head.
“Passion” (believing super-hard in the mission and product of the startup) is a double-edged sword. It helps you work through the inevitable setbacks of invention, but it can lead you to work past the time where you needed a break.
Think about the boundaries that’ll be healthy for you. Are you a runner? Climber? Skiier? Reader? Carve time out of your week for the things that recharge you, and do it from the first week you’re there. If everyone gets used to you being in the office until 8, leaving at 6 is going to feel like your commitment has changed.
Beware stress-eating.

Valley Life

Bay Area is slow to get around in. Public transport sucks except in SF proper.
Cars are a trap because of insurance. EVERYONE gives way to pedestrians, without exception, because hitting a pedestrian invites a lawsuit and lawsuits can cost you everything. Make sure you’re an insured driver, even if you’re just renting a car for a day.
Nobody calls it “San Fran”. Don’t call it “Frisco” either. Locals say: “SF” or “The Bay Area”.
There are HEAPS of meetups and similar groups where you can find like-minded people. If you’ve been starved for like-minded people, this can be fun.
Lots of fun outdoor areas for hiking relatively close by. Do check out Yosemite, once the government shutdown has finished and someone’s cleaning the toilets again.
They are really precious about their cruelty-free micro-batch all-organic gluten-free locally-produced food. The price is typically proportional to the hyphen count. Prepare for a bunch of “gosh, really?” moments with the locals.

Benefits

Make sure there’s health insurance. I can’t imagine a startup that didn’t offer it, but holy shit you don’t want to get ill in America while uninsured.
Shares are weird. You won’t get ACTUAL shares unless you’re the founder. Instead, there’s an ESOP (Employee Share Option Program) and you get the option to purchase shares. And you don’t even get THAT … after x months working for the company, you then have the right to purchase shares. The purchase price will be low, and after they do work and sell things, the company is worth more so the stock price rises so your “can buy for 20c” shares are now worth $2 each, say. BUT even then it’s not awesome because typically you have a limited period to exercise your options (aka actually buy the shares) once you leave the company. If you don’t exercise, then those potential shares of yours go away. So it might cost you $100k or more to keep the shares/options benefit once you leave. This got me once. Negotiate so that your options don’t have an expiry date, or have a ridiculously long exercise time like 10 years. This way they can get so big and famous that people will WANT to buy your shares, so you can exercise your options and then sell the shares right away, pocketing the difference as profits.
Do consult an accountant who knows about options in Australian income tax law. You’ll be subject to both, thanks to the treaties between the countries. You don’t have to pay double tax, that’s what the treaty is about, but you’ll just want to make sure that getting stock options as part of your compensation doesn’t cause problems back home. (And you’ll want to know how to exercise those options so that you don’t end up losing most of their value to tax, or –worse!– paying to buy the options AND then paying tax on the rise in value they have!)
American work years are ridiculously long. Negotiate for a holiday period that you will be happy with, not the Fuck All Weeks that is standard.

Good luck and have fun!

]]>

The T in CTO doesn’t stand for Talk

2018-08-29T00:00:00Z

Looks like NZ will get a CTO real soon now. It’s hard to avoid the word “debacle” in describing how it came about: a false start at making an appointment, a whiff of impropriety in the appointments process that resulted in a Ministerial demotion ….

This is a shame because there are very real reasons that NZ should be increasing its IT heft in Government. There’s the potential to do a lot of good at the intersection of IT and government: preventing blowouts, giving informed advice to the civil service, and being a trusted advisor to politicians. Other countries are tackling these problems, with and without a person whose job title is CTO.

Preventing blowouts: in the government version of “adding a person to a late project only makes it later”, the risk of IT project failure rises with every dollar budgeted. The last government had Treasury monitoring progress of important projects, but that wasn’t continued in this administration. Someone with political power needs to be keeping the IT suits honest, be it Treasury or a CTO. In the UK, financial sign-off on large projects only happened when they followed modern risk-minimising practices (i.e. incremental vs big bang, agile vs waterfall, etc.).

Civil Service: a lot of civil servants are trying to get their heads around the recent flood of technology. It’s not easy because Silicon Valley hires a lot of masterful marketing spin doctors to dress their technology in robes of golden bullshit. The policy wonks have told me they need someone on the inside who they can trust, who will help them understand what’s possible and when. America has the Office of the CTO which led a bunch of public and behind-the-scenes work on everything from big data to autonomous killing drones, to raise the awareness of the public and of the civil service.

Politicians: they also need a trusted and informed advisor. They are besieged by lobbyists who feed them plausible stories. Do the claims stack up? Are the promises credible? Ministers are also somewhat at the mercy of their “officials”, the civil servants who help and support a Minister’s office. Ministers need to find their own way of navigating this trust labyrinth, and a CTO could be a bright light in the darkness.

Prototyping: it’s not a surprise to most people that government adopts technology slowly. This is generally fine, nobody wants their passport application on the dark web because the developers had a fetish for shiny new untested technology. But there’s room within government to apply forward pressure to use newer technology to make government services more responsive and accurate. A handful of departments have labs. Perhaps that forward pressure would be strong with coordination and leadership. Other countries have teams that can act as contract developers, showing how well things *can* be done.

And that’s not including the bridge role between private and public sector that Rod Drury and others were talking about. Government can affect the deployment of technology by business, whether 5G or identity services, and it’s safe to say that the private sector isn’t always convinced that those decisions are made with the appropriate amount of forethought and discussion. A CTO could bring the parties together to help everyone find common ground on why, what, and how.

(Yes, these functions only somewhat overlap with the role of a CTO in a technology company. I’m unfazed by this. I care more about what the position accomplishes than what it is called, as I have since 1993 when VUW and I had to agree on a job title for “person who runs the web site” which involved inventing it because it didn’t exist in NZ before I did it.) (My job title was not “Executive Corporate Vice-President for Digital Communications Affairs (Global)”, nor CTO)

Why do we need a CTO? “Because we promised a CTO” isn’t a useful answer. There’s a profitable consulting career to be made just by asking clients “what is it that you’re trying to achieve?” (also pronounced “what problem are you trying to solve?”) and letting that conversation unfold. I don’t think anyone in the hiring process has articulated a singular specific problem for the CTO to solve, other than unfulfilled campaign promises.

The job description didn’t identify which problem that the CTO was to solve. It hand-waved at all of them, but didn’t clearly identify any one of them as The Mission. Clarity matters for a number of reasons.

Without a great problem description, you can neither recognise success nor prepare for it. Is the CTO adequately resourced to achieve their job? They have a salary and a travel budget, and will get some shared operational support from a Ministry. Can they hire a dev team? Commission research? Build prototypes? Do anything but go to conferences? How does is role separate from that of the GCIO, GCDO, and the other Government Chief {anything} Officers within DIA?

Similarly, you can’t identify the characteristics of a successful candidate without knowing the problem they’ll need to solve. The job description was a catch-all, and that’s resulted in one abortive effort to appoint (in which I was caught) and another protracted search. As I write this, we have an almost-appointed candidate: Derek Handley.

The one thing that is shared by each of the problems that a CTO might solve: technical nous. All three finalists last time were technical, but I think that counted against us in the hiring process. I’ve heard whispers from within the technology world similar to those in The Spinoff’s article) that Derek is insufficiently technical for the role of CTO: he founded two marketing companies (one of which may have disgruntled investors), his education is in architecture and management, and he’s undertaken a lot of activities that some describe as resume padding.

I’ve not met Derek for any length of time, I have no opinion on his abilities. If it’d been me appointed, I’m sure there’d have been a chorus of people with opinions on my shortcomings.

It’s entirely possible that Derek can make a success of this. But if this CTO is to be anything other than lipstick on the pig of government technology, they will need to have strong opinions about the change they can make. They’ll need to pick a problem and tackle it. They’ll need to be sharp and be able to separate bullshit sales hype from near-future potential from present actual capability. They’ll need to get shit done and fast.

I wish the successful candidate good luck and all speed. The clock’s ticking and, whichever problem you choose to have an effect upon, the tech industry hopes you’re swiftly successful. Those problems in Government will still be there after the next election, and the performance of this first CTO will determine how those problems are tackled.

My biggest fear isn’t that Derek does nothing. Nothing is relatively benign. I’m not that concerned that NZG would sign up for a blockchain VR AI private cloud solution. My biggest fear is that an ineffectual CTO engagement after a fraught appointment process might put government and the tech industry off the idea of working together.

The problems to be tackled by tech savvy people in Government aren’t going away any time soon, and they’re bigger than one CTO appointment. Let’s keep our eyes on the prize here: better functioning and more responsive government, less waste, stronger policies, stronger delivery of policy goals. Whatever happens with the CTO, I hope we keep trying to find a way to put tech-savvy people closer to the decision-makers so the decisions can be better.

]]>

Proposed Changes to NZ's R&D Incentives

2018-05-29T00:00:00Z

There’s an open consultation about to end, on the changes MBIE would like to make to NZ’s R&D incentives. In particular, they’ll phase out the Callaghan Growth Grants and replace them with R&D tax credits. As the FAQ says, There are differences in the definition of eligible expenditure between the Growth Grant and the proposed R&D Tax Incentive (for instance, overseas expenditure on R&D). The proposed R&D Tax Incentive has no R&D intensity threshold, a much higher cap and lower minimum R&D expenditure threshold than the Growth Grant. Some firms may get less money, but others might get more.

This policy is very clearly aimed at large and profitable companies. If you’re a Fonterra, or a Mainfreight, or a TradeMe, you can receive a discount on the tax you pay on your profits. That’s good: we want these companies to increase their spending on R&D. Another 1% spend from them represents tens of millions spent on R&D. Good, we need it, NZ lags horribly.

However, there are plenty of other companies in NZ. In particular, I work with software companies and startups at that. It’s time for a side-track on startups …

A Side-track on Startups

A startup is a small company that wants to get big fast, generally to beat would-be competitors to the opportunity they’ve identified. To grow artificially fast, startups don’t make profits and reinvest some of those profits in building more product and opening new offices. Instead, they take investment money and spend it on sales people, marketing, but (most importantly) programmers to build the product and get more users.

Those startups on full burn are growing really fast. You may have heard of some: PushPay, Vend, Soul Machines, … Until recently, Xero was in that list but they’ve finally reached the point where they’re shooting for profitability and not simply pure growth.

The growth here is in users and not profit. Indeed, startups run at a loss while they burn like this: they’re spending more to obtain the next set of users than they are making from their existing users.

The way to make sense of this behaviour is to recognise that the company is the product. The startup wants to be bought: either by a lot of people through an IPO on the stock market, or by another company as an acquisition. Why would someone want to buy an unprofitable company? The best reasons are (a) the startup can just stop growing and then it’ll be profitable; and (b) the users are desirable to the acquirer (because you can sell them other things / they’re already users of the acquirer’s product / etc). There are other reasons companies are bought, including delusion and speculation, but these are the more defensible ones.

Often the buyers are not Kiwis: if you’re a Kiwi firm and you are going to sell, you’ll sell to the people with the most money. If that’s a Kiwi company, great. If it’s Fairfax or Apple or Adobe (all of whom have acquired NZ tech startups) then that’s great too. Realistically, in the software space it’s generally international buyers.

Because the company is the real product, the ostensible business of the startup (building and selling some software products) is not profitable while the startup is growing. The profit comes when the company is sold.

Think about that for a second, it’s important: a startup generally won’t make a profit while it’s building and selling its accounting/AI/payments/… software.

Most importantly, you won’t know whether the company can be sold until the company is sold. The larger and older the startup, the more chance there is that it’ll be acquired, but there have been 8 year old startups that grew large and then blew up because they weren’t profitable and nobody bought them.

While they’re growing, though, they’re hiring people. If the startup is in NZ, it’s hiring Kiwis. if it’s in England, it’s hiring Brits. If a startup in England gets a better R&D incentive than its competitor in NZ, then the British startup has an advantage: the money the Brits get from their government can be used to hire more people to move faster and pull ahead of the Kiwi startup. A shitty R&D incentive policy in NZ makes Kiwi startups go under while their foreign competitors thrive.

Back to R&D …

How do our startups fit into the R&D tax credit scheme? Short answer is: they don’t. Tax is a concern if you’re profitable. If you’re not profitable, tax is the least of your worries.

So the R&D tax credit does nothing for startups and high-growth firms.

Nothing.

Even worse, the proposal is half-baked. It’s clearly aimed at the kind of old-school physical product companies that dominated the 20th century in NZ: Gallagher, Fonterra, etc. It talks about R&D as following the scientific method, and there are lots of examples with dies and machine blanks.

Even weirder, the definition of R&D specifically excludes market research, design, and social sciences work. The iPhone is a UX innovation, and indeed Apple’s whole market distinguishing feature is R&D around Design. Facebook are doing R&D right now via social sciences, trying to understand the networks of conspiring bots and political actors (I know this as I went to Social Science Foo Camp at Facebook in January). Lean and agile software develop starts with iterating with a customer, trying to design the feature or product that they want and will use. This is literally called The Design Process and it’s critical to building a new product. Not covered in MBIE’s doc.

Now Back to Startups …

In a high-growth software startup, every dollar spent on that startup is R&D … because the company is the product. You don’t know until you actually sell it whether it’s saleable.

That’s the Silicon Valley model of startups … five years of loss-making while you burn investment money to get big, then you start to worry about breaking even, and then once you’ve proved that you CAN make money (even if you’re not making very much) you’re sold.

Maybe it’s five years, maybe it’s eight years, maybe it’s three years. The pressure is on software startups to be profitable (or at least break even) ASAP to prove their viability. Investors got burned funding companies that were never able to be desirable to an acquirer or profitable enough for an IPO. Now they’re saying “prove you have a long-term future by being breakeven”.

Whenever it comes, whether at year eight for Silicon Valley or year three for NZ, there’s still a huge risky early-stage part of a startup’s life that this policy does nothing to address. Early-stage startups are not profitable, are almost exclusively doing R&D, and are utterly unsupported by this policy.

To be sure, there’s a way to cash-out R&D tax credits. The consultation document says “The Government is committed to providing a better policy option to support these businesses. However, the policy issues are complex and will not be resolved in time for the introduction of the Tax Incentive in April 2019. […] The existing R&D tax loss cash out scheme may be reviewed as part of any further policy work but no changes will be made to it for the 2019-2020 income year.”

The consultation about the tax credit policy admits that they don’t know how to apply R&D criteria to software and that this isn’t aimed at startups. Yet they’ll still phase out the Callaghan grants. This is irresponsible.

Instead of developing both the big company and the startup policies together, consulting, and then retiring the Callaghan grant, MBIE has opted to develop a BigCo policy and let startups pray that the Startup policy is (a) developed and (b) any good.

Perhaps I’m wrong. Perhaps I’m misreading this. But if I’m accurate, then this is a big MBIE finger in the eye to software and startups.

I used to question why we need NZTech and NZRise. This is why! We need people paid to box the ears of MBIE and remind them that they will never get another TradeMe, Xero, or PushPay if they don’t start taking the job of nurturing these companies seriously.

So What Next?

Have a look at the UK if you want to see some startup-friendly R&D policy. R&D is 100% claimable as expenses, and small-to-medium businesses can even get cash back. The calculations are astonishing.

The MBIE consultation is interested in “what are the effects on your business of this proposed tax policy”. Your company should complete that consultation, if you’re a startup doing R&D.

In particular, visit MBIE’s R&D Incentive Online Submission App and answer Q2-5 (what is R&D, scientific method), Q7-8 (design, UX, etc.), Q12 (growth startups can sell product for less than the cost of producing it, to build customer-base), Q13 (software). If you don’t want to do this through their app, the MBIE site says how to do it via email and the end of the consultation doc has a short list of the questions. If you are very short of time, Q7-78 and Q13 are most important but you’ll want to read the relevant bits of the consultation doc (don’t worry, it’s a quick read).

MBIE should put the tax credit system on hold until they have a proposed solution that works. Creating uncertainty for startups while they figure out what the 21st century looks like is not good policy development.

I’d personally like to see MBIE do some open learning with the software industry about what R&D looks like here, and how they might define and measure it. The MBIE doc has a feel of “gosh, this software world is odd! We know we don’t know!” but it’s bloody weird that you’d admit you didn’t know but not seek out people who could help.

This is based on my reading of the proposal and consultation documents. If I’m wrong in my interpretation or details, please let me know: @gnat on Twitter or email nathan@torkington.com. If you want to tell me that Silicon Valley’s venture-funded startup model is speculative bullshit and a toxic cancer at the heart of our industry, don’t bother. I need my inbox free for constructive commentary. We can destroy the capitalist running dog business model later.

]]>

Twitter's Summer Reading List

2017-12-21T00:00:00Z

I asked my Twitter friends for recommendations of books to read over Christmas. I said that I’ve already consumed the new Expanse novel and the new Philip Pullman (intended to indicate that I like that style of sf) and I said I like non-fiction if it’s interesting and well-written (e.g., Bakewell’s book on Montaigne, or Holmes’s “Age of Wonder”).

Here’s what they recommended …

Artemis by Andy Weir (the sequel to The Martian). Recommended by @lancewiggs.
New York 2140 by Kim Stanley Robinson. Recommended by @lancewiggs and @stojg.
The Dark Forest by Cixin Liu. Recommended by @lancewiggs, @mceoin, and @obra.
The Powerhouse: Inside the Invention of a Battery to Save the World by Steve Levine. Recommended by @lancewiggs.
The Captain Class: The Hidden Force That Creates the World’s Greatest Teams by Sam Walker. Recommended by @lancewiggs.
Perdido Street Station by China Mieville. Recommended by @MattNippert.
Walkaway by Cory Doctorow. Recommended by @mandysinpson.
The Way of Kings: Stormlight Archive Book 1 by Brandon Sanderson. Recommended by @mandysinpson.
Blue Remembered Earth: Poseidon’s Children Book 1 by Alastair Reynolds. Recommended by @teh_aimee.
Wake by Elizabeth Knox, which is set in New Zealand. Recommended by @teh_aimee.
Consider Phlebas (The Culture Series book 1) by Iain Banks. Recommended by @teh_aimee.
Anything by Charlie Stross. Recommended by @teh_aimee and annejohn.
Anything by Micah L. Sifry. Recommended by @teh_aimee.
The Killer Angels by Michael Shaaara. Recommended by @rmi.
The Underground Railroad by Colson Whitehead. Recommended by @rmi.
Into the Heart of Borneo by Redmond O’Hanlon. Recommended by @rmi.
Kids These Days: Human Capital and the Making of Millennials by Malcolm Harris. Recommended by @rmi.
When Breath Becomes Air by Paul Kalanithi. Recommended by @rmi.
No More Champagne: Churchill and His Money by David Lough. Recommended by @rmi.
Off Armageddon Reef: A Novel in the Safehold Series (#1) by David Weber. Recommended by @VikOlliver.
Black Marks on the White Page by Witi Ihimaera and Tina Makareti. Recommended by @amscraig.
Mythos by Stephen Fry. Recommended by @MikeStok.
The Water Knife by Paolo Bacigalupi. Recommended by @stojg.
Altered Carbon by Richard K. Morgan. Recommended by @newzedland.
Anything by N.K. Jemisin. Recommended by @annejohn.
Penric’s Demon: Penric & Desdemona Book 1 by Lois McMaster Bujold. Recommended by @annejohn.
The Silo Effect: The Peril of Expertise and the Promise of Breaking Down Barriers by Gillian Tett. Recommended by @RachelProsser.
Anything by Kristine Kathryn Rusch. Recommended by Miraz Jordan.
Homer’s Odyssey translated by Emily Wilson. Recommended by @auchmill.
The Bone People by Keri Hulme. Recommended by @tummycom.

Finally:

Teach Yourself Javascript in 24 Hours by Phil Ballard. Recommended by a smart-arse.

]]>

On Moving to New Zealand

2016-11-10T00:00:00Z

Hello, American friends! President-Elect Trump has given his speech and begun to redact his campaign website of the obviously illegal and impossible campaign promises, and you look up from your keyboard through an election-defeat hangover and want to move to New Zealand.

First of all, consider staying. America’s problems won’t be solved if all the tolerant and progressive people leave.

But that’s not an easy choice for everyone. If you don’t think you’ll be safe, or you’re concerned about the effects on your children of growing up in the cloud of President Trump, you might be looking elsewhere.

Allow me to suggest New Zealand.

New Zealand has a fairly straightforward skilled migrant immigration scheme, where you get points for meeting certain criteria and if you clear a particular number of points then you can move here. Some of those criteria are around education, language, and health, effectively biasing it against people who don’t speak English, those who aren’t highly-educated, as well as non-able-bodied and unwell people.

We maintain a list of jobs that are in demand. If you can meet the needs of an OMG SO IN DEMAND job then you just need a job offer (as well as the points, as described above). The government’s website on moving to NZ to work doesn’t suck. It’s harder but not impossible without ticking the ZOMGJOB list (and do look: there are some surprising inclusions).

There are plenty of tech startups looking to hire people. NZ tends to have a reasonable number of fresh software engineering graduates, but few with the kinds of skills that people acquire in American startups: devops, engineering leadership, web scale distributed systems, big data pipelines, etc. Which startups are hiring? Look at PushPay, Raygun, Atomic, Timely, Vend, Xero, TradeMe, etc.

Nerd conferences are good here. Webstock (design, Feb) and Kiwicon (security, Nov) are anchors of the scene. Attend those and you’ll meet many of the people with whom you could work, and some good friends. There are additional web, mobile, etc. conferences. Be sure to schedule Fieldays in your first year, because the agtech world is weird and wonderful and close to our grass-growing economic roots.

If you’re an investor, you can investor your way to residency. Similarly entrepreneurs.

The pathway to citizenship is straightforward if you decide you’d like to live here forevs.

The absolute dollar value you’ll earn in NZ will look low if you translate it to American dollars. Do not think you’ll be able to afford your San Francisco home because you’ve been working in Auckland. It doesn’t work that way.

Food is expensive. Thanks to globalisation (fist shake! Grr globalisation, you!) the whole world can buy our food. So we pay a lot to eat it. We don’t have a Mexico just south of us always producing fruit, so (for examples) we eat strawberries for a month each year when they’re in season … and then not. Meat’s available all year round, and pretty good in the shops. And if you live semi-rural you can probably find a farmer who’ll let you buy a bull and have it butchered for you, then you can get a freezer full of export-grade yum. Vegetarians, make friends with your local Asian grocery, where all sorts of surprising imports and deliciousness is available.

Housing is tight in Auckland, but easier in many other parts of the country. You can buy a home in Auckland if you have the earning power of two professionals in your family, and then they’d better be successful professionals. In many other parts of the country, one professional income is enough.

NZ is beautiful and diverse; there are many great places to live. Think of it as the American West Coast: Auckland is Santa Barbara, Dunedin is Seattle. (ish) The rule of thumb: warmer is norther, but there are some exceptions (Nelson and Napier are toastier than you’d expect given their locations) Do you like hiking? Skiing? Fishing? Swimming? Hunting? There are great places for these activities around the country, and you could live next to the national park or marina that means you can live your passion every weekend.

Are you more cultural and cerebral? Wellington and Auckland have thriving arts scenes, with bands, coffee shops, theatre, opera, orchestras, etc. The cities of Dunedin, Christchurch, Wellington, Palmerston North, Hamilton, and Auckland are university towns.

Caution: our hipsters are not as developed as America’s. So while there’s the occasional extravagant beard and fixie bike, and it seems like every town with more than 50 people has a cafe where you can get an excellent coffee, you’ll struggle to find someone who’ll charge you $27 for an artisanal cruelty-free microbatch locally-produced free-range recycled soy-inked letterpressed 50%-butter-by-volume coffee and there are no emoji-only ride-sharing voice-interface social network startups. Turn back now if this is a problem for you.

Can Trump happen here? Never say never, the world is going fucking nuts. However … New Zealand so far has traded with crazy nations without becoming crazy itself: we have a lot of Brits but most Kiwis think Brexit was nuts; similarly with Americans and DT. Kiwis have a much warmer relationship to regulation than Americans. There’s been no NZ indigenous genocide (unlike USA and Australia), and the worst social woes in NZ don’t register on the American scale.

Our racists and entitled old people have done little damage to the rest of us; both leading parties are center-right and center-left. And our definition of “racist” is “I don’t think those blimmin Marries should be given any more money!” and “no more Chinese immigrants, they’re driving up house prices!” rather than KKK robes and skinheads beating the shit out of brown people on a regular basis. To be clear: no skinheads or KKK robes in Hobbit-size. We have a sad racist past, sad racists, and ongoing racial tension, but not on the scale of America.

NZ schools are pretty darn good. We’re no Finland (as politicians constantly remind educators) but state schools are mostly very good. Schools aren’t driven by yearly tests, and the NZ Curriculum is very flexible with plenty of room for schools to find their own identity (culture, technology, etc.). Schools are funded from central Government, not property taxes, and schools in poorer areas are given more money. I’ve heard San Francisco residents complain that most state schools in the area are terrible—that is not the case in Auckland. We moved to NZ (wife is American, I’m a Kiwi who’d spent 10 years in Colorado) when our kids were 4 and 6 and the relaxed school environment, no gangs, no shootings … priceless.

We have proportional representation, so power is frequently split between parties. We get to vote for MPs who represent our area, AND for a party. The parties get MPs in proportion to the number of votes the party got — it’s not as complex as cricket, much fairer than your system, and you’ll get the hang of it. The Green party is a contender here. On the downside, we don’t (yet) have constitutional protections against the elected Parliament, so if NZ did elect a lot of arsehats then they could run amok.

We’re part of the Five Eyes network (with US, UK, Canada, and Australia), so Ed Snowden can’t move here either. We have legal protections against wholesale surveillance of citizens, and distrust our spooks to play by the rules or politicians to make them watertight. Like Americans, we all suspect that unless we’re using Tor and Signal our comms are fair game. On the upside, NZ is small enough that you can easily meet your politicians and bureaucrats and give them a piece of your mind.

What else isn’t great? We have higher child suicide and abuse against children rates than economically-comparable nations, and the government has done a shit job of taking care of the poor during the last decade’s housing in Auckland. Because of that economic boom, NZers have invested more in property than in all the good stuff. Our socialist healthcare system takes care of everyday things really well, but if you’re earning middle-class incomes then consider augmenting it with private insurance so you don’t have to join waiting lists should you need surgery (good news: the dominant provider is a coop so NZ health insurance costs are miniscule in comparison to American health insurance costs). If your kid has very special needs (e.g., autism), the Government doesn’t fund enough assistance for their schooling to be awesome (and, obscenely, may not let you stay). A surprising number of our rivers are full of animal shit and not swimmable (fancy that, in a dairying nation). Why the hell in 2016 are we still building subdivisions without bike paths, and building roads without bike lanes?! These are all issues that the NZ Left is familiar with and grumpy about.

In short, we’ve got our problems but they’re nothing in comparison to your problems! You’d be welcome and loved here. Ride out the Trumpocalypse with the sounds of native birds in the trees as you crack a cold craft beer and revel in your new home’s reasonable race relations, functioning political system, and complete absence of orange arsehats.

Of course, this is all just my opinion. You should come and check it out for yourself! See you soon!

]]>

"Outcome is a function of process"

2016-10-30T00:00:00Z

I was just catching up on Tim Kong’s excellent blog, when I read this great quote from Dan Carter:

“One thing we talk about over and over with this current All Blacks side is about never focusing on the outcome. We view the outcome as a function of following our processes. That might sound a little dry to some, but looking back at every major loss we’ve had over the years, they mostly started with us thinking too far ahead of the game.”

I like that quote a lot. There’s a lot you can find in it:

You can’t do success. Instead, you can only run, pass, tackle, communicate … all of which can contribute to success.
Even in a game with as many different plays, player matchups, imbalances, and opportunities as rugby, the winners are winners because they have s system that generates wins.
The team’s playbook must necessarily be flexible, because it will be used in many different circumstances (and there’s an opponent who will exploit predictability).
The team is still important. You can’t give the All Blacks playbook to the Mahurangi B rugby team and expect them to win against the Lions.
Your team still has to train, to be the best they can be and to lock in the playbook.
So within general principles, you find what works and use that.

So too with engineering management. Your job is to shape the processes that give you success. They may be different in some ways from the processes that give others success. Your processes won’t dictate every solution to your team. The members of your team are still important, and they should still be learning and running.

But engineering management is not sport. The tech environment changes constantly, and every day is game day. Consequently, much more of the playbook related to solving specific problems on the field devolves to the team members themselves, and much more learning happens on the field. But, as with sport, relentless running will exhaust your team so it’s wise to build rest days for learning and exploration into your team’s schedule if you want them able to play their best game the rest of the time.

Ok, I’m done. I promise, no more sportsball metaphors.

]]>

Nat Torkington

Simon Willison's Weblog

Nat's 2022 Technical Link Pile: C# and .NET

Nat's 2022 Technical Link Pile: Dev, Architecture, APIs

Nat's 2022 Technical Link Pile: Front End/Flutter/Dart/UI

Nat's 2022 Technical Link Pile: GPT-3

Nat's 2022 Technical Link Pile: Intro

Nat's 2022 Technical Link Pile: Management

Nat's 2022 Technical Link Pile: Product

Nat's 2022 Technical Link Pile: Security

Nat's 2022 Technical Link Pile: SQL Server/Databases

Nat’s 2022 Technical Link Pile: Interesting Software

Nat’s 2022 Technical Link Pile: Random

Acoustic Bandcamp Recommendations

5G Public Works Project

Abstract

Mobile Technology: A Primer

Spectrum

Generations

5G

Current Structure of NZ’s Mobile Industry

Ultrafast Broadband

Shared Infrastructure

International Active Sharing Relationships

Objectives of a Public Works Program

Implementation

Dependencies

Acknowledgements

Joining a startup? Read this.

Maturity

Startups have a life-cycle

Startups attract interesting people

Look after you

Valley Life

Benefits

The T in CTO doesn’t stand for Talk

Proposed Changes to NZ's R&D Incentives

A Side-track on Startups

Back to R&D …

Now Back to Startups …

So What Next?

Twitter's Summer Reading List

On Moving to New Zealand

"Outcome is a function of process"