I recently read Bluesky and the AT Protocol: Usable Decentralized Social Media by members of the Bluesky team along with University of Cambridge's Martin Kleppmann. The paper talks about Bluesky, a decentralized microblogging platform, and the AT Protocol which serves as it's foundation.

The AT Protocol is one of several new technologies in the decentralized social media space, otherwise known as the fediverse. These technologies aim to allow interoperability between social media providers, resulting in a competitive market where users have more choice and freedom to find a platform that meets their needs. To support this, the AT Protocol is designed to allow portability, allowing users to move from one social media provider to another without losing their username, followers, posts, and without having to create a new account.

Traditional centralized social media platforms like Facebook become sticky and lock users in because they can only see content from other Facebook users. If someone were to leave Facebook and go to a competitor, their followers would no longer be able to see their posts and they'd lose the ability to see posts from the people they follow.

Decentralized social media attempts to solve this by decoupling a users social network from the underlying provider. Users on one platform can follow users from another platform and vice versa. This gives users more choice when initially signing up for a service, but without account portability platforms can still achieve a great deal of lock in. Users would be reticent to lose their followers, username, posts, and other account data if it couldn't be moved to a new platform when they wanted to leave.

It's this key property of account portability I find most interesting about the AT Protocol. If moving a user's account to another social media provider is easy, then the switching cost is dramatically lower and there are more opportunities for competition in the marketplace. With a higher degree of competition we'd likely see more innovation or more platforms specialized to the needs of their users. For example, we might see platforms arise that cater to content creators separate from the platforms that cater to the consumers of that same content.

I'm a big supporter of decentralized social media and Bluesky in particular, but I want to point out some aspects of the AT Protocol that may lead to practical challenges when it comes to ensuring account portability. To explain these challenges, we first need to explore how user data and user identities work in the AT Protocol.

In the AT Protocol, each user's data is kept in a "user data repository" hosted on a "personal data server" (PDS). While users could operate their own personal data server, in practice they would likely be run by third party providers that host many user data repositories. Each user data repository contains information such as the user's posts and who they're following.

In the paper, Kleppmann et. al. use the analogy of running a website. In the case of the AT Protocol, each user data repository is analogous to a website and a personal data server is a hosting provider.

PDS providers hold a user's data, so to prevent lock in it's important users be able to take that data and move to a new provider. The AT Protocol goes about this by determining where a user's data is stored (their PDS) based on their identity, instead of tying a users identity to where their data is stored.

The way this works from a technical standpoint is that a user's identity piggybacks off existing DNS infrastructure, using a subdomain as the user's "handle" or username. The domain name system already has technical and social policies in place that control ownership of domains, and it's fairly inexpensive and simple to register a domain for yourself. Some platforms may give subdomains to their users automatically in order to eliminate the complexity of changing DNS records. Having control of this subdomain is part of how the protocol identifies users.

A user's subdomain could simply point to the location of their PDS, but that would be insufficient. What if you wanted to change your handle? Your followers might lose the ability to find your PDS if your old handle no longer pointed to it. So, in addition to theeir handle, you need a unique identifier for each user. More importantly, what if two handles pointed to the same user in the same PDS? There would need to be some way to distinguish which handle is actually authentic.

This is where the concept of a decentralized ID (DID) comes in. Instead of referring to a specific user data repository on a specific PDS, a user's handle (subdomain) refers to a DID. Each DID is immutable and uniquely identifies a user. This way, they can change their handle and still have it refer to the same DID.

Distinguishing which handle goes to which DID is accomplished by having the DID refer back to the handle. This mutually referential setup helps validate that one handle goes with one DID. The way this is accomplished on a technical level is through the use of a DID document. A user's handle refers to a DID, and the DID refers to a DID document. Each DID document contains the actual information about which handle the DID is associated with and which PDS the user is using.

So to recap, a user stores their data in a user data repository hosted on a PDS. The user's handle is a subdomain that specifies the user's unique DID. The DID points to a DID document which refers back to the handle to verify it's correct, and specifies which PDS is hosting the user's data. This mechanism gives users an identity, and that identity determines where their data is stored.

Great, so at this point we know the user has control over their subdomain because the DNS system already has mechanisms in place to verify ownership. We know that the DID associated with their handle (subdomain) is valid because the DID document refers back to the user's handle (subdomain). But how do we know the DID document is authentic?

There are various types of DID, but for now Bluesky only supports two of them. The first, web, I won't get into but it also uses existing DNS infrastructure to verify a DID document is authentic. As expressed in the paper, typical users would not use web DIDs anyway, they're more likely to use the second supported type, plc.

plc DIDs use a hash value to refer to an entry in a central DID document repository. Each DID document in the central repository contains a public key and a digital signature in addition to the user's handle and PDS. When a new DID document is added or changed in the central repository, it's signed with the private key associated with the document's public key. A plc DID document can be validated either by checking that it matches the plc hash, or if it's been modified, by checking the public key against the digital signature.

Again, plc DIDs are what users will likely use, meaning that ownership and validity of a DID document is tied to a private key. So a user needs access to their private key in order to change their PDS or handle.

In a real world use case, most users would not have direct access to this private key. If a user lost their private key, they wouldn't be able to make changes to their DID document. Meaning they wouldn't be able to change their PDS and move their data to a new provider, preventing account portability.

Instead, a user's private key would likely be stewarded by a service provider, possibly the same service provider as their PDS. The service provider would let the user authenticate with something more convenient like a username and password. That way if they forgot or lost their password, it could be reset through conventional means.

This is where I think malicious actors could prevent account portability by controlling a user's private key. There may be incentives to prevent a user from moving to a different platform, especially if the private key service provider was the same as a user's PDS. These service providers could refuse to give the user their private key, preventing them from updating their DID document. This would mean they could not change their handle or their PDS.

If this were to happen, a user could create a new DID and change their handle to point to it, but they would lose their old posts, followers, and other data. What's worse, if they didn't have control over their handle (subdomain) they would lose that as well. If a user's handle, DID document private key, and PDS are all through the same provider, there are opportunities for those providers to act in malicious ways to lock them in.

I very well may have missed something. There may be other technical mechanisms that further prevent lock in. Regardless though, if there are financial incentives I think bad actors will still find ways to lock users in. Moving to a world with decentralized social media and a competitive marketplace will require more than just technical solutions. We also need to be vigilant and develop social mechanisms to prevent bad behavior.

Make no mistake though, the AT Protocol and other decentralized social media protocols are a big step in the right direction.

I think Bluesky and the AT Protocol are a wonderful innovation and something we sorely need in the social media space. We've seen time and again that when centralized social media platforms gain significant scale they lock in their users and content creators. When these platforms make changes that alienate their users they often have little recourse because the switching costs are too high. Take for example YouTube creators. Many YouTube channel operators have complained about having videos demonetized or receiving copyright strikes for innocent behavior. My hope is that by embracing decentralized social media, we can lower the switching costs for users and content creators, creating a much more competitive market that better serves their needs.

Last September I started as a software engineering intern at Jabra. They're a headphone/earbud brand owned by parent company GN that makes a variety of enterprise and consumer products. GN also owns brands in the gaming and hearing aid spaces such as SteelSeries and ReSound.

I'm with the research and development group based out of Dover, NH. They focus on enterprise wireless headsets for front-line workers and truckers. They used to be part of a company called VXi until it was acquired by GN in October of 2016.

What I've Done

I've been incredibly grateful to the team I work with. At the start I worked hard to get up to speed quickly. While an internship is often a learning experience, I also wanted to make sure I was a productive contributing member of the team.

They noticed the drive and level of competence I displayed, because not long after, they gave me a lot of autonomy to do my work. Not only did they trust me to build things without oversight, but they valued and listened to my advice and feedback on matters within my purview. I was not expecting it, but on multiple occasions I made recommendations that were taken seriously and accepted.

Another thing I've enjoyed about this internship is the variety of projects I get to work on. It's a small team that uses the joint-development model to make their products. This means that while the software team is focused on firmware development, they also need to know enough about other domains to coordinate the work of outside contractors. As an intern, I've been able to work on everything from firmware to mobile, web, and DevOps development.

Firmware

I've been primarily focused on new product firmware development. Things like UI, which for an physical product means how the product reacts when you do things like press a button or connect a Bluetooth device. I've also worked on features related to production line calibration and testing.

As products are built on the production line, they're "flashed" (loaded) with the firmware that controls their microprocessors. After assembly, they need to be tested to ensure they're working properly, and some components may require calibration. The firmware needs to work in concert with the production line equipment to test certain features and to calibrate the product as necessary. One of the projects I've been working on is building the functionality necessary to test and calibrate the product on the assembly line.

Simultaneously while developing the firmware for our product, the production line people are preparing for mass production. As part of my work, I've had to coordinate with peers overseas to make sure the production line calibration and testing features are ready as they start ramping up. This means building critical functionality before production deadlines to ensure the overall launch schedule stays on track.

Digital Signal Processing

I've also had the privilege to take ownership of the digital signal processing (DSP) system on our new product. This system is where audio is processed to improve quality (make it easier to hear or more pleasant) and reduce noise. The DSP microprocessor we're using comes with functionality to configure various types of audio processing. While this is convenient, it lacks certain features we'd like. One of my big projects has been integrating a new DSP platform into our firmware.

It has been a boon to the team's productivity. The new platform is much more portable, meaning the work we do on this product can be transferred easily to another, reducing our development time. Integrating the new DSP platform has also made the process of audio tuning much quicker and more pleasant, similarly reducing development time.

This new platform was entirely new to the Dover R&D team. I've been the person primarily in charge of integrating it into our product, and as a result I'm the go to person for questions. It's been incredibly gratifying to take ownership of a significant portion of the product and be trusted to manage it effectively.

CI/CD

In my hobby projects I've made significant use of GitHub actions to build CI/CD pipelines. The Dover R&D group, being very hardware driven, hadn't yet implemented any sort of CI/CD or DevOps workflow. After I joined, I was able to bring these skills to bear by building CI/CD pipelines for several internal and customer facing software products.

The CI/CD pipeline I made for the new product firmware was particularly helpful. Our firmware has to be built in two parts, one build for the main micro-controller, and another build for the DSP processor. Additionally, there are restrictions on how the DSP firmware can be built, meaning some people have to use prebuilt binaries instead of compiling from source. In the past, this has lead to mistakes where firmware builds were released using outdated DSP binaries, causing issues during the QA process.

I built both nightly and release build pipelines to solve these problems. The nightly pipeline runs....well nightly. It produces development builds that can be used for testing by the development team. The release pipeline runs when we're ready to release a new version of our firmware. During the build, both the micro-controller and DSP processor code is compiled from source, meaning it cannot be outdated. As a result, the release builds are a lot more reliable because they use a standard process. The builds themselves are then distributed to a central repository where they can be retrieved by QA and the rest of the team for further testing and development.

What I've Learned

This internship has been incredibly rewarding. I've repeatedly told my friends and family that even just being paid to write code has been so fulfilling. It's affirmed my decision to leave the nuclear industry, and If my full time job after graduation is like this internship, it'll have all been worth it.

I credit working on my hobby projects a lot to my success. When you're building something from scratch for yourself, you learn to be resourceful and creative in a way you don't from working on school projects. While there have been a lot of similarities between working professionally and working as a hobbyist, there are also a lot of differences. And as any good internship should, it's taught me a lot.

Firmware Development

Most of my experience prior to the internship was with web and cloud native development. Firmware development is as dissimilar to web development as any two pieces of software can be.

For starters, the iteration cycle is much much slower. In web development, the product you're working on is often running locally on your computer. You usually have tools like hot reloading so any change made to the source code is quickly and automatically reflected in your development environment. This means you can iterate quickly, make a small change and immediately see the result.

In firmware development, any change you make requires recompiling the project, which in my experience is slower than building a web project. After, you can't just run it on your computer, you have to plug in your hardware and flash the new build. Then you have to start up the device and perform whatever testing is necessary to see the result of the change you made. Instead of seeing the results immediately like in web development, it often takes several minutes to see what you did.

While no one iteration takes terribly long, it adds up over a work day. And it means you work differently. Instead of working one small change at a time, you often do a few things at once to optimize the time it takes to test your changes. If you make a small mistake it means you have to do the whole iteration loop again. So you inspect your code changes much more thoroughly.

Software Development Life-Cycle

In professional software development there are more stakeholders and the risks associated with a project taking longer or going over budget are higher. There are also a lot more people working together on the project. Compared to a hobby project, professional software development requires more coordination and discipline.

In software engineering, this manifests at the software development lifecycle. The project doesn't just start by hacking out source code off the cuff. Instead you start by narrowing down what your requirements are, then coming up with plans to test and verify you've met your goals. This can be visualized by things like the waterfall or V models.

This is my first experience working with detailed requirements and testing documents. The Dover R&D group takes these things very seriously, to the point where it was the first thing I was ever trained on. Detailing your requirements ahead of time and refining them as you go helps you prevent wasted time and development effort because you can be sure you're building the right things. It also helps guide the roadmap because you can clearly see all of the things you need to do.

Having a testing plan that clearly lays out whether or not requirements are being met is similarly powerful. By iteratively testing your product, you not only ensure a high standard of quality, but you get a good sense of how the project is coming along. Testing helps you see if your schedule is on track or if existing features broke during the development of new features.

After working in an environment with detailed requirements and testing, I ended up integrating this workflow into my master's capstone project at school. At the start of the semester, I suggested to my group that we lay out our requirements and iteratively refine and test them as we go. I saw how useful it was at work and wanted it in my own projects as well.

New Technologies

Before the internship I wasn't scared of C, but I hadn't worked on any decent sized projects with it. This was the first time I was developing mainly in C and it's been interesting. C is a great language and much of the modern world is built on it, but it's missing a lot of modern conveniences I'm used to. The things I miss most are modules and classes.

C has an antiquated module and code reuse system. Whereas in a language like Python, you can use directly include another source file (or a symbol within that source file) by module name, C requires the use of header files and forward references. In my experience, I've found this makes it cumbersome to organize and split up your source files. It also makes it harder to understand exactly where a certain function or variable originates. I've had this complaint with other languages too, but I think one thing a good module system should do is make it clear where things are from.

Lacking classes, C makes it harder to use a lot of the design patterns I've learned over the last few years. Some of them are easy, the singleton pattern can be quickly approximated. But others like the State pattern are harder. Thinking C++ could be used as a drop in replacement for C, I tried implementing some functionality in C++. What I found is that, while it interoperates with C more than most languages, C++ is not a 'drop in' replacement. The build system did NOT like mixing C and C++, primarily because the SDK we're working with was not written to be interoperable.

Other than working with C, I had exposure to a lot of new technologies. The new product I was working on uses Bluetooth and a real time operating system (RTOS). These were all very new to me, along with DSP development. What I found in these cases is that I often had prior experience that made the learning process easier.

For example, I took a networking course in my master's program and I've done a lot of networking related hobby projects. This helped me understand how Bluetooth works at a protocol level. Similarly, working in a concurrent RTOS environment was made easier by my prior studies in concurrent programming and the concurrent programming I've done in my hobby projects. DSP programming was made easier because of my audio engineering experience in undergrad (I was the chief engineer of the student radio station at UMass Lowell).

Multi-Disciplinary Environment

The new product I was working on was a physical electronic. It wasn't a pure software product like Microsoft Excel or Facebook. This mean I've done a lot of collaboration with engineers from other disciplines. In particular, I've worked closely with the electrical and acoustic engineers on the team. Each discipline approaches problems differently and it's been useful to see new ways of thinking. Even some of the software engineers on the team studied electrical engineering in college, and they approach problems differently from someone who studied computer science. It's been a very interesting experience.

One of the most valuable skills I've been able to refine is communication. Working on a multi-disciplinary team, you can't assume everyone knows the same jargon or understands the same concepts you do. So you have to get good at communicating key ideas to people who might be working from a different context / background.

Get an Internship

One of the most prudent decisions I made in grad school was going for this internship. I've learned so much from it and it's been a fulfilling experience. I think this advice applies broadly to anyone in school, but it's important to get experience in the field you want to work in. It helps you see more clearly what you'd enjoy doing after graduation. Even if you're planning on staying in academia, getting a job as a teaching assistant or doing research will give you a much better sense of direction and affirm the decisions you're making. Or it could show you that you actually want something completely different.

It looks good on a resume too, but more importantly, you learn the skills necessary to succeed at a full time job. It makes you a more well rounded software engineer. My internship ends this Friday and I'm excited to use these skills again at my next job.

GitHub - kclapper/TakeCounter: A take counter for recording sessions

A take counter for recording sessions . Contribute to kclapper/TakeCounter development by creating an account on GitHub.

GitHubkclapper

Not everything needs to be new under the sun, and sometimes the simplest things can be very helpful. For a great example, look at Take Counter, a Mac app I built for a friend. He's a recording engineer at Power Station NYC and wanted an app to display the current take during recording sessions. The killer feature would be global keyboard shortcuts. From any other app he wanted to be able to type a keyboard shortcut to increment, decrement, or reset the count.

When he told me he wanted an app to do this I asked, "do you have 30 minutes?" Counting apps are the first thing people build when they're learning a new programming language or UI framework, so I knew it would be simple to build. I had a decent understanding of Reactjs and I knew I could use it with Electron to build a native Mac app. It ended up taking about an hour instead of 30 minutes, but we had a working prototype rather quickly. The keyboard shortcuts weren't global, but otherwise it did almost everything he wanted.

We decided to keep working on it, and eventually it grew a settings menu and re-assignable global keyboard shortcuts. Sharing the executable was a bit of a pain. We were using Signal to communicate and it meant having to produce a build and attach it to the signal message, which was slow and inconvenient when it came to installing on my friend's studio computers.

To make this easier, I built a CI/CD pipeline with GitHub Actions. When I created a new development release, it would run all of the tests, build the executable, then upload it to an AWS S3 bucket. From there I could share a link with my friend, making it easy to download and install. Since it's a Reactjs app at it's heart, I also had the CI/CD pipeline deploy it to the web, here. You can't use the global keyboard shortcuts on the web, but everything else works.

Take Counter isn't terribly special. It does not do anything innovative or new, yet it's still very useful. It's purpose built to do one specific thing and that's exactly what my friend uses it for. Other recording engineers at the studio have commented on Take Counter and asked where they can get it too. A big takeaway I've had is that not everything needs to be groundbreaking to be useful. Take Counter isn't going to be a Fortune 500 company, but if your goal is to build a useful piece of software, all you need to do is listen to people and attend to their needs.

Software development is already fun, but it's even more rewarding when you know the program you're writing is going to make someone's life easier or make them happy.

My cover letter starts:

On January 4th 2022 I quit my job as a nuclear reactor operator. I’d taken the job after graduating with my bachelor’s in chemical engineering and while the subject matter was interesting, I found the actual day to day work to be miserable. Looking back, I realized the times I felt most engaged were when I was programming, be it in my college’s Matlab course or with my hobby projects. So I went back to school to pursue a master’s in computer science.

Now, as I prepare to graduate with my master's in computer science, I thought it would be a good time to reflect on why I chose to leave the nuclear industry. This excerpt from my cover letter hits the important points; I no longer enjoyed the work and my passions were elsewhere. But I think if we dig a little deeper, there's a lesson to learn about how we make choices in our careers.

Bachelor's Degree

It's no secret that at 18 it's hard to know what you want to do with your life. After high-school the next step for many of us is college, but unless you have a great deal of clarity on your passions, it's hard to know which degree is best for you.

When I was looking at colleges I was between electrical and nuclear engineering programs. I was very interested in green energy and nuclear power in high-school. I thought there was something inspiring about the work to combat climate change and I thought nuclear power was fascinating. What swayed me in the end was finding UMass Lowell's chemical engineering program that had a nuclear engineering track. UMass Lowell also boasted a nuclear reactor on campus they used for research.

While I was at UMass Lowell, I had a job as a part-time reactor operator at the campus nuclear reactor. It took a lot of effort and involved months of self-study, but I earned my federal nuclear reactor operator license. It was such a unique experience and I learned so much. Me and my fellow student operators took a lot of pride in being highly trained to respond to all manner of emergency and incident (not that it ever happened, it was a very safe facility).

Nuclear Reactor Operator

I wasn't sure what I wanted to do after college, but I knew I enjoyed reactor operations and thought it was worth exploring further. I ended up getting a job as a senior reactor operator at the MIT Nuclear Reactor Laboratory, another campus nuclear reactor.

I was still very enamored with the nuclear industry. There was so much to learn and everything was so new. For a while I really enjoyed working as a reactor operator. It again involved months of self-study, this time working towards a senior reactor operator license.

A friend once told me, "there are subjects that are interesting that I want to learn about, and there are subjects that I actually want to do day-to-day." Those two are not necessarily the same. There are subjects that are interesting and fun to learn, but that I wouldn't actually want a career in.

It's a trope that they don't teach you everything you need to know in school, and this is true. In any field there are a lot of skills that you only learn on the job. But the corollary is, how you feel about a subject in school is not an adequate representation of how you'll feel working professionally in it. This is one of the reasons colleges advertise co-op and internship programs. They give students insight into how much they'll enjoy the work itself.

My interest in nuclear engineering did not correlate with a passion for the day-to-day work of a nuclear reactor operator. After I got my senior reactor operator license, my sentiment about the job turned sour. I started doing normal shift work and things became very repetitive. The hours didn't align with my sleep schedule and there wasn't a lot of upward career mobility. It started to feel like I was stagnating. On top of this, I was assigned duties that involved manual labor that exacerbated my chronic injuries.

Master's Degree

It was around this time that I started thinking about going back to school. I had always intended on going back to school, but early on I thought I'd do it part time while working at MIT. As I became disgruntled, I realized what I really wanted to was to quit and go to school full time.

One of the reasons I had delayed going back to school was that I didn't know which path to pursue. I could continue learning about nuclear science, or branch off and learn about health physics. I could also start pursuing data science, which was trendy subject at the time and seemed interesting.

It took a lot of introspection but what I finally realized was that I wanted to pursue a career in computer science. Looking back, I'd always been working on CS adjacent hobby projects. For example, in undergrad I learned Python and wrote some simple automation scripts. After graduation, I also put together a bunch of Raspberry Pi's and started serving websites from my apartment. The key realization was that I was most engaged in my work when I was coding.

I knew there was a big foundation of computer science knowledge I was missing and I would be better positioned for a career switch if I got a degree in it. Almost right away when I started the program I could tell it was the right choice. Unlike when I was in undergrad, the coursework clicked much more easily and I was much more intrinsically motivated to do it. Getting my internship at Jabra was even more affirming. At work one day, I had the thought, "if this is what my day-to-day is like after I graduate this will have all been worth it."

Career Decisions

I'm much happier now than I was at MIT. Don't get me wrong, I love the MIT NRL. I'm so incredibly grateful for everything they did for me. I mean, they got us through the pandemic without laying anyone off, which is absolutely remarkable. Just because it didn't end up being the right fit for me, does not mean it isn't the right fit for some people.

But what I learned while working there is that my passions lay elsewhere. What's more, I think there are qualities of software engineering that better align with my disposition. It's project based work, you're constantly building something new or making improvements. It's a much more dynamic role than being a nuclear reactor operator. Again, this isn't a value judgement on that kind of work, it can be very rewarding and enjoyable, but it wasn't the right fit for me.

One of the students I worked with at MIT told me it was encouraging to see me leave. They were about to graduate and said seeing me switch careers to something totally different reassured them that they could do the same. If their career after graduation wasn't a good fit, they weren't stuck and could move on to something else.

I think a big takeaway from this whole endeavor is that, just because we made the best decision we could at the time, doesn't mean we need to stick with that decision in the future. Over time you'll grow and change, and it's important to give yourself the flexibility to rethink your decisions.

And it's easy to trick yourself into thinking something is the right fit. There's a lot of emotional incentive to think we made the right choice, especially when we've put a lot of time and effort into something like a bachelor's degree. Making changes to our path is difficult and scary. Quitting my job involved moving back in with my parents (which I'm incredibly grateful for) and it was terrifying.

Rethinking the decisions I made about my career helped me see what I was actually passionate about and got me to a much better place. I have so much to be grateful for and this experience has been a genuine life changer.

With that said, I'm looking for a new job 😁 If you're interested in working with me please check out my resume and get in touch: contact@kyleclapper.dev

GitHub - kclapper/tower.js: Javascript implementation of Scheme’s numeric tower

Javascript implementation of Scheme’s numeric tower - kclapper/tower.js

GitHubkclapper

I was a PhD student for about a year before I realized it wasn't for me and switched to the master's program. I studied programming languages and as part of the Racket community working on Racketscript. Racketscript is a Racket to Javascript transpiler and it's still under active development. At the time, Racketscript didn't support complex or exact numbers.

Racket, unlike many programming languages, has complex numbers as a language primitive. You can do complex math with the algebraic operators and get complex numbers as a result. Racket has the concept (inherited from Scheme) of the numeric tower.

All numbers are complex numbers, a subset of those are real numbers, a subset of those are rational, and a subset of those are integers. There is another concept orthogonal to the numeric tower called exactness. All numbers can either be exact or inexact. Exact numbers are represented as an integer numerator and denominator. There are no rounding or truncation errors when you do math with exact numbers. Inexact numbers are represented as floating point numbers and behave as you'd expect.

While a number system like this increases the language and implementation complexity, there are two big benefits (in my opinion):

1. You can do math with exact numbers to avoid rounding and truncation issues. This is of particular importance in some science and engineering disciplines like aerospace engineering or astronomy.
2. You can easily do math with complex numbers and with better performance than a library implementation. This is important in domains such as digital signal processing.

js-numbers

Racketscript didn't have support for complex or exact numbers so I set out to build it. There was an existing 10 year old library called js-numbers that implemented the scheme numeric tower in Javascript. It wasn't originally intended for Racketscript, but since Racket is a descendant of Scheme it served as a good starting place, so I forked it on GitHub.

js-numbers appears to predate npm and modern Javascript tooling. For example, it used an old Java library for bundling the source code. I decided I would rewrite it as an npm package using Typescript. By making it an npm package, anyone else who needed an implementation of the Racket numeric tower could use it too.

The second benefit to doing a rewrite was that I could add modern Javascript language features, namely bigints. Bigints give Javascript programmers an arbitrarily large integer type. Normal Javascript numbers have a maximum and a minimum safe integer value. This is because all numbers in Javascript (other than bigints) are represented as floating point numbers. Meaning once the number becomes too big to fit in the significand, it starts being rounded in order to fit. Bigints end up making it easier to represent exact numbers because you don't have to account for the minimum and maximum safe integer values.

The Implementation

For the rewrite, I renamed the library tower.js. Like I mentioned above, it's written in Typescript and uses modern Javascript tooling like npm and Webpack. Since Scheme is a functional language, the API of js-numbers was functional as well. Users of the library would do math by calling functions like:

var three = add(1, 2);
var fifteen = multiply(three, 5);

I thought this API was fitting so I kept it and added some missing Racket functions. I tried to mimic the Racket functions as best as I could in tower.js.

While the API is functional, the number representation is object oriented. I use classes to represent the boxed versions of numbers. To optimize performance, some number types can be represented as unboxed Javascript primitives. For example, exact integers can be represented directly as bigints and inexact real numbers can be directly represented as normal Javascript numbers. These unboxed numbers can use the builtin Javascript arithmetic operators which are must faster than calling class methods.

This dual representation, boxed and unboxed numbers, made the implementation of the API functions more complex. In each function, there needs to be a check to see if a number is boxed and should be unboxed or vice versa. This boxing/unboxing behavior can end up hurting performance depending on what computations are being performed.

The Results

Exact complex, expt(100+89i, 5000)
--------------------------------------------------------------------------
js-numbers: 7.4032 ms/trial (740.3240 total)
tower.js: 	0.3210 ms/trial (32.1035 total)
Results match: true

Large Exact integer, expt(1000, 5000)
--------------------------------------------------------------------------
js-numbers: 5.4490 ms/trial (544.9012 total)
tower.js: 	0.0984 ms/trial (9.8447 total)
Results match: true

Small Exact Base and Exp, expt(2, 30)
--------------------------------------------------------------------------
js-numbers: 0.0003 ms/trial (0.0389 total)
tower.js: 	0.0008 ms/trial (0.0809 total)
javascript: 0.0002 ms/trial (0.0277 total)
Results match: true

Small Inexact Base and Exp, expt(5.5, 5.5)
--------------------------------------------------------------------------
js-numbers: 0.0019 ms/trial (0.1926 total)
tower.js: 	0.0006 ms/trial (0.0638 total)
javascript: 0.0002 ms/trial (0.0283 total)
Results match: true

Mixed Precision, expt(5.5, 10)
--------------------------------------------------------------------------
js-numbers: 0.0006 ms/trial (0.0683 total)
tower.js: 	0.0006 ms/trial (0.0602 total)
javascript: 0.0002 ms/trial (0.0279 total)
Results match: true

I did some micro-benchmarks to try and assess the performance impact of the rewrite. The rewrite ends up benefiting from using bigints to represent exact numbers. The js-numbers library had to implement exact numbers using Javascript numbers and library functions. Bigints, on the other hand, have native runtime support which makes them much faster.

The bench-marking results above don't show it, but in typical use cases the rewrite is as fast or slower than js-numbers. Where it beats out js-numbers is when it comes to exact numbers due to its use of bigints. Bigints allow for more unboxed operations which are significantly faster.

Tower.js also supports more functions than the original js-numbers library did. In particular, it implements the bitwise functions, which are needed to build a Racket runtime in Javascript. Since it was written in Typescript, it also comes with typings that could help library users distinguish between native Javascript numbers and the various tower.js number types (and also help them write type-safe code).

If you want to check out tower.js you can find it on GitHub. You can also find documentation on the GitHub Pages site.

GitHub - kclapper/Koggle: Boggle the boardgame, now as a website and React component!

Boggle the boardgame, now as a website and React component! - kclapper/Koggle

GitHubkclapper

A while back, me and my girlfriend were playing a lot of Boggle. Around that time, I was reading Clean Architecture by Robert C. Martin. It ended up being very influential to how I see system architecture and I had a few major takeaways:

• Your dependencies should always point from lower level to higher level concepts. The things that are most likely to change should be the easiest to change.
• The structure of your code should scream what kind of application you're building. When you look at your repo, your first takeaway shouldn't be, "this is a Spring app", it should be, "this is a healthcare system." (I'm paraphrasing Martin here).
• A good architecture defers as decisions to as late as possible. For example, the exact database you use should be a detail of your application, one that the core of your application doesn't care about. Only the outer layer of your architecture should care about the database implementation.

With these ideas in mind, I wanted to build an online version of Boggle so my girlfriend and I could play together, even when we were out of the house or didn't have the board with us.

The 'Business Logic'

The core logic of the application (or the 'business logic' if you prefer) is the game board (the data) and how to play the game (business rules). With the principles outline above in mind, I wanted these concepts to be high level components of the system that had as few dependencies as possible.

Critically, I wanted them to be separate from the presentation of the game. I was using React as my front-end framework but I didn't want my core logic to care about this decision. To facilitate this, I went with an event driven design. The game is represented by classes that implement the Controller interface. This interface specifies what key functions a game of boggle performs such as:

• Starting and stopping the game.
• Notifying event listeners when the game starts, finishes, or is stopped.
• And getting details about the board like it's size and the letters on it.

Here I'm using the Observer design pattern to make this an event driven design. Using events, the game controllers don't have to know about other components in the system and can focus exclusively on it's logic. Notably, there is no React code in any of the classes that implement this interface. It's not necessary, because they are decoupled from the UI.

This Controller interface is implemented by an AbstractBoggle class. There is a lot of internal logic that is shared by different boggle boards, so it's useful to put it all in one place to reduce repetition. The RegularBoggle and BigBoggle classes both inherit from AbstractBoggle , and their implementations are essentially just variations on setting up the board they contain. The bodies of RegularBoggle and BigBoggle are small as a result (~50 loc).

The UI

Like I mentioned, I used React for the UI. React has a handy feature to help synchronize with outside components and APIs called useEffect. It essentially lets you "break out" of React and I used it throughout the UI code to interact with the Controller classes.

Here is a good example. This code snippet comes from the main UI component and it's purpose is to update the UI when the game starts. There is a state variable called inProgress that controls how the UI looks when the game is in progress. With useEffect, I add a 'gameStart' event listener that sets this value to true.

Solver

In that code snippet you'll also see that I use a class called Solver to find all of the words on the board so they can be displayed at the end of the game. There isn't anything special about this class from an architecture standpoint, but I had fun making it because I got to demonstrate some algorithms knowledge.

To find all of the words on a Boggle board is tricky. The naive solution would be to start from each block on the board and draw out all of the possible paths that don't contain repetitions. I recognized that this is essentially a graph traversal problem and the naive solution is essentially a more computationally expensive version of depth-first search.

Knowing the naive solution would be slow, I built it anyway and used it as a jumping off point. I represented the board as a graph, then found each string that can be made. I cross referenced these strings with the Official Scrabble Dictionary to see if they were words and saved the ones that were. This solution took a long time to finish, on the order of 30 seconds to a minute or more (depending on the size of the board).

Seeing as how a user could click start then immediately click stop, I needed to be able to solve the boards faster than 30 seconds.

I ended up using a trie to speed up the algorithm. There are free versions of the Official Scrabble Dictionary in json form on project Gutenberg. From those files, I took the entire English dictionary and built one large trie. When building the trie, nodes that represented the end of a valid word were flagged.

When solving the board, you start at each block and you keep trying to find new words, but the important optimization is that you follow along in the trie as well. If no more words can be found by going from one block to another, then you stop following that path and move on to the next one. In this way, the traversal doesn't explore any path on the board that can't lead to a valid word.

With this algorithm the Boggle boards can be solved almost instantly. After clicking start, you can immediately click stop and the solution will show up right away.

Tools

When it comes to "your code should scream out what kind of application it is," I think I missed the mark a bit. I think, in practical terms, the tools we use will necessarily influence structure of your code to some degree reflect. When you look at the source code for Koggle, it's clear it's a Node app. The npm package structure is all there. It's written in Typescript and uses React as a front-end framework. It also uses Bootstrap for styling.

CI/CD

Koggle is written in Typescript and distributed as an npm package. When users install Koggle, they can add the Boggle React component to their projects. I use GitHub Actions for CI/CD. When a new commit is added to the main branch, the code is linted, type checked, and tested. When a semantic version tag is created, an Action first lints, type checks, and tests the code. Assuming those all pass, it then publishes the package to npm. It also builds and deploys the demo site.