Announcing the Alpha Release of the Crash Override Chalk™ Open Source Project

An open source project to that ties together development and deployments using metadata

At a lot of startups, people feel like they know the problem space well, because they’ve lived it. They trust their opinions over all others, and build to their own vision. Sometimes that works out, but that’s never the way I’ll do things.

While most people aren’t good at envisioning the products they want, people tend to be really excellent at dissecting their problems.

Last year, when we started Crash Override, we spent a few months interviewing many dozens of people in tech about their problems. We were lucky enough to get to sit down with some practitioners, and watch them deal with their problems directly.

We learned a tremendous amount from people by listening. And, over time, we noticed that, while not everyone would recognize each other’s big-picture problems, there were incredibly common hurdles that many tech people in larger organizations face, and few people are dealing with them well.

What do these people have in common?

• Jason’s team is dealing with a significant outage. There’s a service failing, but it’s not one he’s encountered before. It takes them over two hours to figure out who owns the code and get in front of that person.
• Jane just found out from support there’s an outage associated with her microservice. It seems like the problem might be hard to trigger in test, and he needs to know where to start looking, but isn’t sure which version is deployed, and what might be happening in the operational environment to impact it.
• Chris runs a large application security team. They do a great job with all kinds of analytics, but when it comes to security alerts in code, there are still too many to ever work down, and for most of their repositories, they have no idea whether or not they matter, or are used.

They all have observability issues with the software they use, and it ends up wasting their time and effort.

We kept seeing people’s problems boiling down to not having the right information about a piece of software at their finger tips, and then being dependent on finding other people, and asking them about it.

It was clear that, for every top-level problem we heard people focus on that our company might want to address, we were going to need a real solution to this problem. But, we couldn’t find anything we could even recommend to people.

The Challenge

We would have thought some of the biggest companies would have this problem nailed. We were totally wrong, though.

Tech giants would say, “sure, everyone has to go through the one blessed pipeline, but when there’s an issue, we still generally waste plenty of time figuring out what exactly we’re looking at.”

Even people big banks would tell us, “Yes, compliance regulations require us to have that information, but when someone needs it, it’s all in a spreadsheet somewhere with some person I barely know, not at the fingertips of the people who need it.”

We wondered, “since this is clearly such a big problem, why haven’t people fixed it?” We started going back to people to learn about all the things they’ve tried.

Plenty of people had tried things like asking engineering to ship a metadata file, and operations to look for it. But then how is all managed? That adds work for everyone, and won’t be straightforward in every environment, such as serverless.

It became clear that, a scalable solution to this problem requires both:

1. Incredible simplicity to the point that it adds almost 0 work (or cost) to anyone;
2. Yes, great flexibility to handle a broad range of requirements.

Philosophically, we believe the value has to be far greater than the effort, and that needs to be obvious to anyone using it.

Clearly, plenty of people had come to similar conclusions.

Introducing ChalkTM

ChalkTM basically helps you get the equivalent of a gps location history for the software you deploy in all its incarnations. If you’ve found the thing in production, Chalk makes it easy to find where that software lived in development.

Similarly, if you’re staring at a git repo, you can easily connect the dots to see what versions are deployed where. But Chalk goes beyond location information. It allows you to collect and associate any kind of data about your software as it is built, tested and deployed.

Chalk works hard to make it all as easy as possible. One key innovation is that, during the build process, we embed what we call “chalk marks”, into the built software.

Not beside the software, into it. This process doesn’t change the behavior of the software, it just puts data in a benign place where it definitely won’t be used by the program itself, and can easily be retrieved at runtime.

For many people, the chalk mark will just be basic identifying information, that they can tie back to data Chalk reported on in CI/CD (during the build process). Others won’t want to have to worry about central data management; they will add the metadata they care about into the chalk mark.

We’ve designed Chalk to make the process of chalking software as painless as possible:

1. You can deploy a Chalk binary that transparently wraps the docker command. We currently handle most build cases, and the ones where we don’t, we just call docker anyway. If you don’t like that, you can just put the word chalk in front of your command line.
2. For other kinds of builds or deployments, you can simply run chalk in the directory where your software lives, after any artifacts are built (ideally, while git information is still available locally).
3. Chalk builds as a single static binary, so as long as the architecture matches up, you can run it on any Linux environment.

That’s during CI/CD. In production, you can use Chalk to launch your software, while reporting not just on the launch, but on the runtime environment.

Chalk is flexible, giving you power over the metadata you choose to collect, to where you choose to send it.
- Chalk is fast, It’s written in a memory safe systems language.
- Chalk is easy, requiring very little attention, until you want to get value out of it.

The Chalk Alpha

We’re not quite ready to release the source; we have a bit of polish we need to do first. But, it will be available under the GPLv3 soon, and it’s ready for you to start experimenting with now.

Currently, you can find our alpha release, v0.1.0, via the Getting Started Guide.

While we’ve got a long list of things that we want Chalk to do, we’ve already got people using it, excited about a bunch of different use cases they can already address today, even beyond simply making it easy to connect software to both it’s repos and its deployments during incidents:

• Some teams are looking for the usage data to help with development planning, not just by giving them more insight into prod, but by making it easy to correlate test and staging behavior to production.
• We’ve heard from developers who expect Chalk to make it easier for them to keep track of which environments are using which feature flags, and to determine which flags are just dead weight, adding complexity to maintain.
• Lots of people seem to be excited about building a near-real-time view of their product architecture, to finally tie together many micro-services with the many little pieces that live across multiple repositories.
• Some people want to automate the security compliance requirements they’re getting from customers, or see on the horizon. If you like, Chalk can already run Semgrep and SBOM-creation tools, and there’s one prototype digital signature implementation in place, with another one that we are polishing up now, which will be part of the first source code release.
• Some people are excited to find out what code they have that they’re not even using... they’d love to be able to archive or ignore old hackathon code, unused private forks, abandoned projects, ...
• Some people even want to give their finance team visibility to ‘plug’ the gaps when it comes to tracking cloud spend.

Again, you can get started with Chalk right now, via the Getting Started Guide.

And, we’ve also been building a free service, which makes it even easier to deploy and get value out of Chalk. It will underpin everything else we build to help solve those macro-level problems for people down the road.

We’re very slowly adding people to that service, especially those who are willing to give early feedback to help shape where we go. Please do sign up and get early access to the alpha here.

Or, if you’re in the security space, and want to meet with us at Blackhat, you can book a meeting with us here.