The basic idea of a Slipbox is that you write short notes to capture insights about the things you’ve read, and you use indexes and a complex system of numbering to link everything together. (This was originally done with small slips of paper and shoeboxes, but there are now computer tools to make this easier.)
If done correctly, your Slipbox becomes a writing partner, surfacing unexpectedly brilliant ideas for you to craft into scholarly articles, dissertations, best-selling books, and – if the stars align – perhaps even a popular blog post or two.
Usually, the description of the tool is quickly followed by a brief bio of the tool’s “inventor”, Niklas Luhmann, a German sociologist. And that is often followed by a quick recap of his insanely prolific writing career: he wrote 70 books and 400 academic papers across a ~40 year career, spanning the subjects of “law, economy, politics, art, religion, ecology, mass media, and love” according to Wikipedia. What a wildman!
Usually, *that* is all followed by the suggestion that if you, too, practiced these Slipbox note-taking techniques with unwavering dilligence, then you, too, could re-define the academic field of your choosing.
Then, the article moves on to the mechanics of building a Slipbox – your own private external brain. The sentences thicken. The process sounds joyless, another thing you Should Do, after you wax your car, rinse your recyclables, take your fiber.
But something has been tugging at my sleeve: Every description of Slipbox-style notetaking I’ve read makes it sound like a chore, and Luhmann was lazy.
According to the book How To Take Smart Notes by Sönke Ahrens, Luhmann “stressed that he never forced himself to do something he didn’t feel like.”
This is mentioned in the context of writing, but are we to believe that Luhmann, while a lazy writer, pressed his note-taking nose to the grindstone for years, scribing tens of thousands of slips in his off hours, in a way that was fundamentally unpleasant for him, in the faint hope that his homegrown knowledge system may one day make it easier for him to write books?
I don’t buy it.
Because again, in Ahrens’s book, while Luhmann exhibits curiosity, he doesn’t show an especially high level of discipline, nor does he seem like somebody who would adopt an unenjoyable routine for the vague promise of advancement. He “went to law school, but he has chosen to be a public servant, as he did not like the idea of having to work for multiple clients.” As a civil servant, he is “fully aware he is also not suited for a career in administration, as it involves a lot of socializing.”
[Side note: “I don’t like parties” is an amusing excuse for a professional failure to launch.]
Instead, Luhmann “excuses himself every day after his 9-5 shift and goes home to do what he liked most: reading and following his diverse interests in philosophy, organizational theory, and sociology.”
Well, here’s the thing: within the lethargy of this pandemic, I often explore my diverse interests in philosophy, organizational theory, and sociology. In less lofty terms, I scan Reddit, Hacker News, and Twitter (and recently, Every.to), and read the posts and responses that look interesting.
Suddenly Luhmann’s research doesn’t sound so impressive. I’m joking, of course, but the point is that Luhmann didn’t have a master plan toward which he was diligently working, and didn’t have a schedule to which he systematically stuck. In fact, he was *avoiding* work, and amusing himself.
Why is this important? Because while I believe that the Slipbox represents a hugely amazing productivity tool, I also think we are trying way too hard at it. *Luhmann was lazy*, so if it worked, it worked only because *it was fun*. If it’s not fun, it won’t work.
This is my theory:
Luhmann read whatever he felt like reading. But he realized this was an escape, and he felt guilty about his lack of ambition. He often thought “if only I weren’t such a lazy ass, I’d be a successful lawyer. Or I’d be managing my civil service department. But I have no discipline.”
He tried various note-taking systems in order to feel less guilty and have some tangible result from his reading, but everything he tried sucked the joy out of the whole ordeal, so nothing stuck.
Finally, to assuage his guilt, he took the bare minimum step required to call his reading habit “research”: after reading something, he identified the one or two insights that tickled his cerebellum, wrote them down on index cards, and put them in a card catalog. This small amount of bookkeeping was tolerable because it let Luhmann continue to indulge in his daily habit of aimless reading.
On average, this translated into six notes per day. (According to Sonke.)
Bang, research.
As the “research” accumulated, a new game emerged, a kind of intellectual Tetris. How do these insights that I’ve collected fit together? Can I relate them in unexpected ways to tell myself stories about the world?
And it turns out that he could fit them together – maybe because he was a genius, maybe because he had a genius, or more likely because that’s how the human brain works. We are wired to seek patterns and notice relationships. Some of these patterns are common across humankind, and we call that common sense and universal truth. Others are novel to our own uniquely wired brains, and we call that creativity. But it’s enormously entertaining to find and catalog connections because as humans we yearn to understand and be understood.
For example, right now, I’m making a connection between my haphazard historical fiction of Luhmann’s invention of the Slipbox and a lecture I watched recently called The Myth of the Objective. In the lecture, Kenneth Stanley asserts that it is more productive to collect unique stepping stones than to chase a clearly defined objective, and that this applies to everything, including machine learning, scientific research, and business.
And I’m also connecting this to my own experience in various artistic fields (improv comedy, songwriting), where things just go better if you accept that the art is the boss and you are along for the ride.
Making these connections across different ideas cause me to *feel like* I understand the world better, and that releases serotonin in my brain, which causes me to want to keep finding new insights, and making new connections.
And furthermore, these connections feel brand new. And because of that, I want to share them with you, just like Luhmann shared his creative insights in the form of 70 books and 400 scholarly articles, just like children are compelled to run to their parents with their latest crayon drawing. Evolution wired humans to find joy in sharing creations and insights.
So that’s the insight I offer to you: Luhmann was lazy, and to keep your Slipbox humming, you must be lazy too.
Lazy, but insatiably curious.
]]>The first is “playing” mode.
This is when I’m toying around with a new idea, seeing if it has legs. I try to build a working prototype as quickly as possible, then iteratively grow that by alternating between layering on functionality and vicious refactoring. In this mode, it helps to work on the exciting things first and follow the fun, cutting corners when possible.
The second is “planning” mode.
In this mode, the software already has some mass, some inertia. There is already an established way of doing things, and perturbing the existing order too much would break a stable system and disrupt others already familiar with the codebase.
I call this planning mode because “coding” actually starts with a plan of action, thinking through what I want to do, and listing out exactly what changes I need to make in what order, down to the file and function. This is done when I have thought through every change I want to make, in a way that prioritizes the preservervation of the existing codebase as much as possible.
Both modes can be fun. When playing, the fun is about getting something done in the quickest, hackiest way possible, often surprising myself in the process. When planning, the fun is in the satisfaction of solving a puzzle and seeing everything slot neatly into place.
]]>I often work on software projects that process a lot of data. Not “Big Data”-sized data (at least not recently), but “needs some careful thought to avoid a quagmire”- sized data. Architecturally, the best approach to data of this size is a “data pipeline” or “datapipe.” A datapipe is like an assembly line, where each step takes inputs, performs some useful operation, and produces an output, which is then consumed by the next step in the line. The final output is some useful knowledge – reports, predictions, recommendations, trained models, etc.
[INPUT] ->
(Operation 1) ->
(Operation 2) ->
... ->
[OUTPUT]
The largest pipeline I built and ran was back in 2006. It ingested approximately 80GB of advertiser data every day from dozens of servers, hovered around 17TB in total retained data, and powered internal and external business intelligence reports for an internet advertising company that reached hundreds of millions of people daily. (17TB may sound small now, but adjusted for Moore’s law – or more accurately, Kryder’s law – it was huge. The biggest hard drive available at the time stored 750GB and cost a fortune. We used a SAN from EMC to handle it costing ~$1M. Now it could fit on a single 20TB drive for $800 with room to spare.)
The smallest pipeline I maintain is for my personal finances. I have raw bank and credit card records since 2015 totaling ~14k transactions. I run this through a series of steps to extract and semi-automatically tag transactions, eventually producing Ledger double-entry bookkeeping files for automated reporting and forecasting.
My most public facing datapipe was part of FiveStreet.com. It trained a machine learning system that could extract prospective customer information from semi-structured emails. When I left the project in October 2015, it processed over half a billion emails per year.
The most high-stakes datapipe of my career is the one I work on now for GlassRoom – it trains neural networks to understand how to think about publicly traded companies, then uses those same neural networks to create a proprietary, bottom-up estimate of a company’s net present value, and then uses those estimates to build a stock portfolio for an investment fund with a meaningful amount of capital behind it.
I share this because it’s fun to brag about my work, but also because this is evidence that datapipes scale well in both directions – upward and downward. The principles I’ve outlined below lead to all the hallmarks of a well-written software project – maintainable code, flexible purpose, stable operations, and minimal technical debt. Simplicity and elegance.
Typically, the first step in a datapipe involves gathering data from multiple sources. The data sources may be internal (within a company) or external (purchased from 3rd party vendors or scraped).
When starting development of a pipeline, take a shortcut and gather data by hand. Eventually, it is worth the time to automate the data-grabbing process as much as possible, but this typically has low technical risk and can be deferred to one of the last steps in the project. Though the technical risk is low, this code is very important; it doubles as a form of documentation that describes from where – and how – the data is retrieved.
Often, fetching data is an expensive process (in terms of wall clock time) so save the data files in their original raw form, preserving directory structures and filenames to avoid re-downloading the same files in the future.
If the files are relatively chunky and few in number, then storing files directly to disk is fine. For a large number of small files, as is common in web scraping, some kind of local key-value store (e.g. LevelDB) with a custom index can help.
$PROJECT/
data/
sources/www.foo.com/bar/page1.html
sources/www.foo.com/bar/page2.html
...
sources/sales_history/2015-01-01 sales.csv
sources/sales_history/2015-01-02 sales.csv
...
sources/web_traffic/analytics_2015_01_01.json
sources/web_traffic/analytics_2015_01_02.json
...
Saving raw data is especially important for HTML files. When web scraping, it is tempting to pre-process the data (e.g. cleaning up HTML) when you fetch the file. Don’t do this! Instead, write the raw web pages to a datastore, and then have a later step in the pipeline extract data from the pages. Otherwise, a change in scraping (fixing a bug, extracting more fields, etc.) will require re-fetching the data.
Even if it is not expensive to fetch data, it is advisable to retain raw data for troubleshooting in case of unexpected results. Often, data vendors or even internal departments will make “minor” tweaks to formats or values that break downstream code. Having raw files from before and after the change are helpful to make sense of why things broke.
Now that the data is stored locally, the next step is to make it easily consumed by downstream code. The “extraction” step will wrangle the data from each source into shape in the following ways:
Most importantly, fail loudly! Be strict in what the system accepts, apply assertions around incoming file names, data formats, column names, and data types and, if possible, ranges of values. Fail loudly if anything deviates from expectations.
The first two steps in any datapipe are fairly obvious – get the data, extract the data into a more easily consumable format. After that, the datapipe will probably have somewhere between 3 to 20 more steps, with each step emitting some intermediate data, ie: a “checkpoint”. (More below.)
Deciding on the interfaces and boundaries for those steps can be an art, and the boundaries can often change over time. As the datapipe evolves, steps may split or merge.
I like to think about each step as having a starting boundary and an ending boundary. Here are a few good ways to find boundaries:
When naming steps (and their corresponding code files), I like to follow an
“interactor” approach as much as possible. Think of each step as completing a
use case, and name the step according to the business or conceptual value that
it provides, rather than something technical. Also, follow a “verb_noun” naming
convention. Put those together and you get: predict_best_mailings vs.
nearest_neighbors_regression_step.
Finally, avoid circular dependencies. The datapipe should only flow in one direction. If data needs to flow upstream, you’ve done something wrong.
If a data pipe is a series of steps that modify data, then a checkpoint is the name for what happens between those steps, when the data is at rest. The output of each step is a checkpoint, saved to a different, descriptively named file (or files), overwriting the file(s) produced by the same step in the previous iteration of the datapipe.
Checkpoints are useful because they are a “high water mark” for data processing. When a step fails, you can fix that step and re-run the pipeline starting with the output of the last successful step. Without checkpoints, the datapipe would need to re-process all previous steps.
When thinking through checkpoints, remember that space is cheap. Err on the side of storing more data, not less. Even if the pipeline will handle a large amount of data, buying more hard drives is typically way cheaper than hiring more engineers. As we saw above, one can buy a 20TB drive for ~$800 today.
I divide files into three types, with three different lifespans:
Most software projects benefit from some kind of transactional database or key-value datastore. Datapipes are not like most software projects. Over time I’ve found that the best way to store checkpoint data (the output of a step in the pipeline) is a flat file. This can be plain text files (.csv or .tsv), or something sexier like HDF5, Feather, or Parquet if you need faster read/write access.
To belabor the point, do not use a relational database to store checkpoints. Relational databases are optimized for transactional operations, where any one operation typically touches only a small subset of the data. This is not the access pattern that you need for reading and writing checkpoint data, and you will pay a tradeoff in disk space, RAM, bulk read, and bulk write performance for using the wrong tool.
The preferred access pattern of a datapipes is that a given step will read multiple flat files into RAM (or file-backed memory), operate on the data, and then write a new flat file to disk, overwriting the flat file created during the previous iteration of the datapipe.
$PROJECT/
data/
sources/...
checkpoint1.dat
checkpoint2.dat
checkpoint3.dat
...
This may sound wasteful, because it means we reprocess a large amount of data that we’ve already processed, but it avoids a lot of potential bugs. By treating each step as an idempotent operation, code changes become much less scary. Depending on the size and scope of your datapipe, this approach may be sufficient to reprocess everything for the life of the project. Planning for anything else at the start is premature optimization. If the length of some step in the pipeline gets too long, prefer to either optimize the code or buy more powerful hardware rather than resorting to incremental processing.
The opposite approach – incremental processing – creates a link between the processed data and the version of code that produced it, which means if you change the code you need to figure out which data is bad. It is hard enough to fix bugs without worrying how to patch up previously processed results.
Avoid incremental processing in all cases, except the following:
First, when the pipeline can cleanly partition processed work by date. In this case, it might make sense to only process some recent time period, ie: year to date or month to date. You should create dated checkpoints, and the pipeline will still reprocess data for the most recent time period, but it can avoid redoing work for previous time periods. Here, you trade off the cognitive complexity of having to worry about reprocessing previous checkpoints for the benefit of a much faster feedback cycle. Then, when there is some kind of code change that requires reprocessing, the datapipe can reprocess in chunks, from most- to least-recent, during what would usually be downtime.
The second exception is when the pipeline generates Neural Net / Machine Learning Models. I will touch on this later.
One of the worst software development experiences of my life was using the visual tools of Microsoft’s SQL Server Analysis Services 2005 to build steps in a data pipeline. The tool made the easiest part of a datapipe easier (ie: connecting the output of one job to the input of another) at the expense of making everything else impossibly hard. All of our custom logic was buried inside of a massive, auto-generated XML file that also tracked layout positions, sizes, and last-accessed timestamps, making version control nearly meaningless. Merge conflicts sometimes took hours to resolve. Debugging was unnecessarily complicated. Eventually, we moved as much code as possible to plain old text files, using the visual tools only when necessary, mainly for orchestration.
Since then, I’ve stuck to building datapipes out of the simplest tools
possible, and it has worked. The make utility will get you way farther than
you would expect. It can be used to orchestrate steps in a pipeline by treating
checkpoint files as intermediate build targets, only re-running the steps that
are necessary to run when a “dependency” (input data) has changed. And it’s
intuitive. Want something to re-run? Delete the file. Want to cause any
dependencies to re-run? touch the file.
# Makefile
.PHONY: help download
help:
cat docs/help.txt
download:
# download source 1 to data/sources/$source1/
# download source 2 to data/sources/$source2/
...
data/checkpoint1.h5: data/sources/*/*
# Run step 1 code
# Write to data/checkpoint1.h5
...
data/checkpoint2.h5: data/checkpoint1.h5
# Run step 2 code
# Write to data/checkpoint2.h5
...
checkpoint3.h5: data/checkpoint2.h5
# Run step 3 code
# Write to data/checkpoint3.h5
...
As a bonus, a makefile is language-agnostic. The checkpoint data can be produced by any language. Data gathering code can use Ruby or even wget, where other steps can leverage R or Python.
Almost every pipeline I’ve built has started out with some variation of this. Most of them graduated to a more complicated orchestration tool, but simple Makefiles worked for way longer than I would have anticipated.
Testing a datapipe is both more and less difficult than testing the same amount of code in other architectures.
Techniques like unit testing don’t apply neatly to datapipe code. Much of the code can’t easily be broken into functions, and the algorithms require so much data (in terms of rows, columns, or both) that preparing test samples feels like a Sisyphean task. Furthermore, the logic tends to be complicated enough that it is difficult to verify a correct answer without re-running the logic itself.
As much as I hate it, datapipe code is often best structured as long unbroken scripts where data is slowly massaged into shape using vectorized functions. Refactoring into subroutines would only add confusion for the next developer and wouldn’t help in terms of reusability – in most cases there would only ever be one caller to the method.
At the same time, unit testing seems less necessary with datapipes. The code tends to have lower cyclomatic complexity than other types of code, so the need for unit tests is reduced. (Curiously, a lot of complexity seems to fade away when your code is not intended for human interaction.) I’ve found that on web apps, I need 95+% coverage to feel confident that I won’t get alerts in the middle of the night. On a datapipe, I can get the same amount of confidence after a single full pass through the pipeline, assuming the output of the pipe passes muster.
The best approach to quality I’ve found is the following:
assert statements during the data extraction process to validate
that upstream sources haven’t started to feed the datapipe garbage.
Likewise, assert statements throughout the code, testing value ranges,
counts, and the presence or absence of null values. Especially during the
extraction step, to make sure that upstream data sources haven’t started
sending something that your system interprets as garbage.I’ve explored the idea of generating statistical models for what data should look like in each checkpoint, and then automatically flagging anything that shows up outside of acceptable thresholds. I’ve also toyed with fuzzing inputs to find inputs that could crash the datapipe. In practice, this lead to many false positives, much wasted time, and me kicking myself for overengineering.
In short, the final output of your datapipe is like one giant integration test, and it is better than any artificial test you could write. Create visualizations so that it is easy for humans to judge the quality of output, and then make a pre-release checklist that includes actually looking at those visualizations. Remember to save snapshots so that you can compare against previous versions if (when?) things go off the rails.
Operations is where all of my muttering about idempotent steps and checkpoints start to pay off. When done correctly, the datapipe can be killed and restarted at any time without worry, and we know exactly which step failed, plus a snapshot of what made the step fail. Furthermore, with just a little bit of additional work, we can log the duration and memory usage of each step, and monitor it through a dashboard and/or alerting system.
The last recommendation, for mission critical systems, is to maintain a hot backup system, in a different geographic location, that runs on the same schedule as the production datapipe.
Data inevitably grows. The datapipe will undoubtedly need to handle more and more data as time goes by, eventually saturating its current hardware.
Let’s assume that we’ve already plucked the low-hanging fruit of optimizing code. We’ve looked at which steps take the longest, looked at which sections in those steps take the longest, and then either done some automated profiling, or at the very least pulled it all into a Jupyter notebook, and used “%%time” statements while tweaking code to try to get some quick wins. Let’s assume that any other optimizations will have a negative impact on code clarity or maintainability.
The next step is to think about scaling up the hardware – moving the datapipe to a more powerful box on which to run – when the data hits ~70% of capacity of its current hardware. The threshold of 70% is a reasonable rule of thumb, but the actual number may vary depending on a number of factors – how quickly the data is growing, how quickly data growth is accelerating, and how critical the datapipe is, to start. Be sure to leave enough headroom so you can sleep at night.
If scaling up isn’t an option, then the datapipe will need to scale outward –
partitioning the data so that processing can run in parallel. One can either
partition by horizontally, processing different rows on different pipelines,
splitting by user_id modulo number_of_servers for example. Or vertically,
where different datapipe servers handle different steps in the process, and
then the data comes back together at the end. It is difficult to say much more
about this here. There is no general advice, solutions require careful thought
and tailoring to the data, and are beyond the scope of my writing.
Well before the data size affects operations, it will affect development. The key to good development is a fast feedback cycle (it’s what makes REPL languages so powerful). As a datapipe step takes longer to run, changing the code becomes more and more difficult. When a step grows much beyond 30 minutes to run, development seems to grind to a halt. (For me… this may vary by developer.)
To preserve a fast feedback cycle, the most useful approach I’ve found is to create a subset of data. Build a development-mode flag into the extraction step that chooses some small percentage of the data in a predictable and repeatable fashion. (Use a hash function and/or a seeded random number generator to get predictable randomness.) This will allow the entire datapipe to be run in a fraction of the time it might run in production, while still exercizing all code paths (if the subset is chosen thoughtfully.)
Software isn't built, it is grown.
At some point, an interactor or model in the datapipe will need to change in a big way. This is tricky for a number of reasons:
To manage this complexity in a clean way, set an environment variable that
determine which version of an interactor or model to use. The Makefile that
orchestrates the code can use the environment variable to call the correct
version of code:
# Makefile
...
data/best_mailings_$(MAILING_VERSION).h5:
python ./core/interactors/predict_best_mailings_$(MAILING_VERSION).py
...
The interactor can use environment variables to figure out from where (and to where) data should be read (and written). This is conceptually the same as creating a backup copy of the data directory, but by letting a computer do the bookkeeping we reduce development friction and ensure consistency of approach. And we avoid making multiple copies of our source data, which can grow quite large.
Above, we change the checkpoint filename based on the MAILING_VERSION
environment variable. Depending on how the steps within the pipeline, we may
want to create entirely different directories underneath data/
$PROJECT/
data/
sources/...
checkpoint1.dat
version_1/checkpoint2.dat
version_1/checkpoint3.dat
...
version_2/checkpoint2.dat
version_2/checkpoint3.dat
...
Often, a pipeline will train (or fit) models that are used downstream in later processing steps. The training process can be long, taking days or even weeks. Furthermore, in the case of neural nets, an expensive GPU may be required for training the model, but not for running the model. In this case, developers may need special hardware to train models, but we can save money by omitting this hardware from the production environment (and the hot backup environment.)
These models aren’t like the data produced by other steps in the pipeline. It is expensive (from a time and resource perspective) to recreate them on every pass through the pipeline, as we try to do with other checkpoint data. Also, training models is a stochastic process, so even if we could retrain models daily, this would cause the datapipe to give inconsistent results, making debugging more difficult and possibly causing datapipe results to “flap” (give opposite answers on back-to-back runs for datapoints that are on decision boundaries.)
But models aren’t code, either. We don’t want to check the models into source control, because they are typically quite large compared to what version control systems were built to handle, and the model files are binary data. (Well, at the very least they are a bunch of weights without human meaning.)
The best approach I’ve found is to track a hash (eg: SHA1) of each model, treating it like the version number of a 3rd party library. The hashes should be tracked in source control. In production, the datapipe should verify that the hashes of the models match the hashes in source control.
When new raw data is produced, how quickly should it propagate to the final results? Do you need real-time (or nearly real-time) updates? Datapipes are useful for update intervals of a day or longer. Hourly updates are possible, but for anything more real-time than that, an entirely different set of architectural principles apply.
Some anecdotal evidence, in ~2009 Google began testing their new back-end indexing system, code-named “Caffeine”. It allowed Google to continuously update search results. Architecturally, this was a huge change, requiring a move from a staged datapipe (using MapReduce) to a more incremental approach (using BigTable).
That said, even when real-time updates are a requirement, it is useful to construct a rudimentary pipeline first, for a variety of reasons:
There are many cited reasons why this is the case. A partial list: Reducing physical clutter also reduces mental clutter. Moving away from material objects allows us to focus on experiences instead. When purging, we purge things we don’t like, so we’re left only with things that bring us joy.
I’d like to focus on two angles that I haven’t seen discussed elsewhere.
First, what if the mental benefits of minimalism come not from having less stuff, but rather understanding a greater proportion of the stuff around you?
During much of the evolution of our brains, technology was pretty basic. From earliest man through the dark ages, even a person of modest intelligence could understand how the objects and tools around them worked at a deep level.
But in 2019, technology is everywhere, and it’s essentially magic. How many people thoroughly understand how their drip coffee machines work, let alone their smartphone, car, and or computer? What is the mental cost of being surrounded by things we know we don’t understand?
Even clothes are somewhat magical now. Clothes used to be sewn by hand by someone in your village using the materials around you. Now clothes are made out of different weaves of synthetic materials, assembled with impossibly precise stitching using the help of machines, screen printed by other machines, and imported through a vast supply chain. Even if we understand how clothes work, we don’t understand how they are constructed, or how they find their way into our closets, at least not like we used to.
By reducing our possessions, more of the mass around us is floor, wall, drawer, or shelf. Our eyes fall on the things we do understand deeply.
Now that I’ve made my point, I see that it’s not all that profound. Happiness comes from understanding. Any new parent teaching a toddler to talk can tell you stories of how frustrating it is not to understand (or be understood.) But framing minimalism in the light of “understanding more” allows us to stop getting to a greater amount of understanding through the side window (by getting rid of stuff) and approach it through the front door.
Maybe it makes sense to go on some research tangents and learn how the modern world works? Before now, I would have considered it pretty useless to learn how microfiber cloth is made, for example. Now I see the value.
Second, what if minimalism is really about self-reliance?
I don’t go camping frequently, but when I do, I’m amazed at how little it takes to exist. The feeling stays with me for a few days after I re-enter normal life as I look around at the things I previously “needed” to survive.
I don’t think the human brain can distinguish “what we have” from “what we need”. We assume we need the things we have. With that feeling of need comes vulnerability and a sense of weakness.
Minimalism helps correct this, similar to a camping trip. When we have less, we feel that we need less to survive. As a secondary benefit, having less sometimes means we don’t have what we need, so we are forced to adapt, or to suffer a little bit. It introduces a controlled amount of chaos to our lives, which – when we overcome it – gives us confidence that we can deal with more chaos.
Again, we can approach this much more directly, maybe by randomly giving up some important object every day and forcing ourselves to adapt. Living through a “no car” day, a “no phone” day, a “no wallet” day, “no internet” day, etc. would leave us much less scared of losing any of those things. Purposeful denial of things, small and large, to help us get better at dealing with chaos.
]]>“I have also found that when I enter a new workplace, everything done before me was a mess of unwieldy, poorly documented and unstructured code. Worse, it uses an old, archaic platform that is long overdue for a complete overhaul.
Then when I arrive, I make everything logical, nice, structured and proper.
After I leave, once again the place falls prey to disorganized minds who take my work and mess it up for flashy new technologies that are not as reliable as what I had done.
For some reason, this keeps happening.”
A good reminder to empathize with those who preceded you and those who will follow.
p.s. - OP forgot to mention the times when you don’t touch your own code for a while and end up playing multiple roles in the scenario above: “Who wrote this junk!? Oh right… I did.”
]]>Then life got complicated, first in the form of co-founder issues, and then ongoing family health issues. I’ve since recalibrated my idea of what good luck actually means.
Good luck in a startup doesn’t mean the unexpected presence of incredibly good things. Good luck is the absence of bad things. If you have a stretch of time where life is smooth and you are able to concentrate on building a business – that is good luck.
In the future, I won’t take those smooth stretches for granted.
]]>I keep taking the classes because I find improv fascinating. If you aren’t familiar with the art form of improvisational theatre, the general idea is that a small group of people get on stage and – without a script – perform a comedic or dramatic work by following a few simple guidelines. Done right, it’s like a machine for creating inside jokes, a type of humor I really appreciate.
Unfortunately, despite nine(-ish) classes, I’m still quite bad at it. Somehow, improv seems to rely on none of my strengths and all of my weaknesses. I’m good at thinking about software systems and business models. Quietly. Alone. Improv is none of those things.
There is value in repeatedly pursuing failure like this. For one thing, finding and holding on to a mental state that feels unnatural and foreign to your usual pattern of thought is like yoga for the mind. Do this enough, and (one presumes) it brings increasingly foreign mental states within reach. It makes for a flexible thinker.
Secondly, with age comes the means and inclination to stay in your comfort zone. It is easy to forget that learning can be incredibly frustrating. Improv, being far outside my own comfort zone, helps me empathize with others confronting their own discomfort.
Moreover, it’s a high-impact, consequence-free, fast-feedback way to shore up some of the stereotypical ailments and weaknesses of the tech-minded human:
Po is a verb, and it gives you permission to make unsupported, illogical, and foolish statements about a problem you are trying to solve. In a sense, it serves like a magic wand that you can wave to jump over constraints and move your thinking forward.
An (imagined) example:
J: We are stuck. We can’t make our laptops any smaller. The DVD drive takes up too much space.
S: Po users don’t need DVD drives.
J: slaps forehead Brilliant! With fast internet and cheap USB thumbdrives, DVDs are obsolete!
And thus was born the Macbook Air.
Po is an incredibly useful tool. Perhaps with a catchier name like “the 80/20 rule” or “the conjoined triangles of success” it might be better known.
This all leads to an observation regarding the recent widespread interest in deep learning. To almost everybody outside of the software development world (and many people inside of it), deep learning is like a magic wand. In a number of conversations about startups and technology, I’ve seen deep learning serve the same purpose as “po”.
I first recognized this during a discussion about a sales CRM. The product team wanted to use deep learning to predict which stage of the sales funnel a perspective customer was in, or when a prospective customer was about to leave, and show it to the salesperson using the software.
After about an hour of strategizing about data and models we finally convinced ourselves that it was possible, and we turned to the discussion of how the application would use the ranking. It turns out that, in all cases, the goal of the ranking was to prod the salesperson toward the same action – call the customer. The real problem was salesperson motivation, a completely different problem that probably didn’t require deep learning.
Without the promise of actually being able to solve the problem through deep learning, the conversation would have ended, and the team never would have realized the deeper truth about the application.
To generalize, humans tend to think linearly, and often get stuck when one of the steps seems impossible. By expanding the domain of what could be done, deep learning helps people see what should be done.
]]>The result is an enormous amount of “micro” context switching. Is there an opportunity for improvement here? How does all this micro context switching affect energy and mood?
As an experiment, I’ve recently tried to “lean in” to boredom during these moments. If I have to wait for a few minutes for something, I just wait. No smartphone. No scanning headlines, checking email, or sorting out my task list. In fact, I try to not think too deeply about anything at all.
I just wait. And maybe look at my surroundings.
Here are my observations:
All of my multi-tasking was ineffectual. I’m still just as up-to-date on the headlines, email, and task list as I was before. No noticeable drawbacks.
My brain benefits greatly from resting. When it’s time to focus, I am more eager to dive in, and more productive.
By avoiding the avoidable context switches, I have more patience for the unavoidable ones. My theory is that every context switch, big or small, causes some displeasure and consumes some energy. It increases for big interruptions, sure, but not proportionally.
Overall, I feel less rushed and – in a sense – “wealthier” for lack of a better word. Let me explain: We work hard to earn money to eventually retire and live a life of leisure. Why not claim a little bit of that leisure now, especially if it comes without a downside?
In 2001, Dean Kamen unveiled the Segway. In terms of startup miracles, this was a big one – a self-balancing personal transporter. He, and many others, believed that the Segway would change the world. Unfortunately, the plan was built upon many “hidden miracles.”
First, the company needed customers. But potential customers didn’t care; their problems were already solved by cars, bicycles, and feet. The $5k price tag didn’t help here. Second, the company needed friendly transportation laws and regulations. Unfortunately, the laws and regulations didn’t anticipate the existence of Segways. At 12 miles per hour, a Segway was too slow for the road, too fast for the sidewalk, and too motorized for bike lanes.
Many startups face similar hidden miracles that lurk in the grass beyond the initial big breakthrough. And the typical first-time startup founder – often a software developer or engineer of some kind – can be especially blind to these miracles because they habitually undervalue marketing and psychology. “I invented something new and amazing!”, they think, “Of course people will want it!”
In my experience, startups with new technology always face at least one hidden miracle.
Usually, the hidden miracle is related to customers and customer behavior. If customers could survive before your product was invented, then they can continue to live a perfectly happy life without your product, no matter how new and amazing it is.
This often comes down to social norms. I like to think of clothes washing machines here. The household clothes washing machine wasn’t invented until around 1937, and wasn’t popular until the 1950’s. Before then it was just socially acceptable to wash your clothes less. The “solution” to the problem of slightly dirty clothes was that society just accepted slightly dirty clothes.
]]>Unsurprisingly, it’s rare to have a contrarian idea and actually be right. So I refer to this – the act of having a contrarian idea that turns out to be right – as a “miracle.”
For example, Netflix (in its original form) had the contrarian idea that you could safely mail DVDs in flimsy paper envelopes via the United States Postal Service without the disk shattering into a million tiny pieces or otherwise becoming unplayable. And, unbelievably, they were right. That was a miracle.
When I analyze a startup, I count how many contrarian ideas exist in the startup’s plan. Each contrarian idea requires a miracle, and miracles are rare. Based on experience, I’d estimate that each miracle lowers a startup’s chance of success by a factor of 5. So if you were 10% likely to succeed with one miracle, you are 2% likely to succeed with 2 miracles.
A common mistake of new startup founders is to build their startups on top of more than one miracle. This is unfortunate – each extra miracle adds a huge amount of risk but typically doesn’t add a proportional increase in reward.
What complicates things is that sometimes miracles are hidden, and sometimes miracles are inherited. I hope to write more about this in the future.
DropBox
AirBnB
Segway
I’m fascinated by the idea that the Internet is the table, and the people are the pendulums. The Internet has shaped human behavior in vast and consistent ways. We have shorter attention spans. We don’t worry about remembering facts anymore. We make more plans, but with less commitment. We stay in touch with far more friends, but get far less emotional satisfaction out of each connection. We swipe, poke, star, heart, and favorite, encouraging or discouraging each other.
I used to mock people who expressed concern that the Internet would become self-aware. But now I wonder, does self-awareness it matter? In a perverse version of the Turing test, if the Internet is expressing behaviors indistinguishable from a self-aware organism, then perhaps the actual self-awareness is just a detail. One could respond that the intent of the Internet is important, and only comes with self-awareness, but then again, do other organisms have true intent, other than to simply stay alive?
And the Internet has exhibited enough usefulness that we, the pendulums, continue to build it. All the while thinking that we have ultimate control, but unaware of the greater system nudging our behavior in tiny but critical ways, just like the table and the pendulums.
]]>
Edit: As of July 2015, the FiveStreet team runs a SuperIMAP cluster processing ~400k emails per day for thousands of users.
]]>git-spit is a git plugin / utility that helps when coding in a cloud
development environment. It watches a local git repository and mirrors
any changes to a remote endpoint.
]]>When we first built this feature, Google only provided IMAP access through OAuth 1.0. Since then, Google has deprecated OAuth 1.0 in favor of OAuth 2.0. (And they are dropping support for OAuth 1.0 in April of 2015.)
This means we have to:
Google’s documentation on how to do this are fairly confusing, and – in some places – inaccurate.
The code below takes the OAuth 1.0 token / secret assigned to a user and converts it to an OAuth 2.0 refresh token. You will need to run this conversion process once for every user, and save the refresh token to your database.
The refresh token can then be used to generate a short-lived access token. Note that your OAuth 1.0 credentials will stop working an hour after you use the OAuth 2.0 refresh token.
More information: Migrating from OAuth 1.0 to OAuth 2.0
require 'oauth'
require 'net/http'
require 'json'
# OAuth 1 - Application Key / Secret.
oauth1_consumer_key = "www.site.com"
oauth1_consumer_secret = "..."
# OAuth 1 - User Token / Secret.
oauth1_token = "..."
oauth1_secret = "..."
# OAuth 2 - Application ID / Secret
oauth2_client_id = "..."
oauth2_client_secret = "..."
# Migration Parameters.
params = {
"grant_type" => "urn:ietf:params:oauth:grant-type:migration:oauth1",
"client_id" => oauth2_client_id,
"client_secret" => oauth2_client_secret,
"oauth_signature_method" => "HMAC-SHA1"
}
# Create the consumer object.
consumer = OAuth::Consumer.new(
oauth1_consumer_key,
oauth1_consumer_secret,
:site => 'https://accounts.google.com',
:scheme => :header
)
# Create the access token object.
access_token = OAuth::AccessToken.new(consumer, oauth1_token, oauth1_secret)
# Post to the migration URL.
resp = access_token.post(
"/o/oauth2/token",
params,
{ 'Content-Type' => 'application/x-www-form-urlencoded' })
if resp.code.to_s != "200"
# Raise an error.
raise "#{resp.code} - #{resp.body}"
end
# Now you have a refresh token!
oauth2_refresh_token = JSON.parse(resp.body)["refresh_token"]
The code below uses the OAuth 2.0 refresh token you just generated to obtain an OAuth 2.0 access token. The access token is what you use to actually authenticate the user.
Note: You may need to change the scope based on your application’s needs.
More information: Using an OAuth 2.0 refresh token
require 'oauth2'
# OAuth 2 - Application ID / Secret
oauth2_client_id = "..."
oauth2_client_secret = "..."
oauth2_refresh_token = "refresh-token-from-above"
# Create the OAuth 2 client.
client = OAuth2::Client.new(
oauth2_client_id,
oauth2_client_secret,
{
:site => "https://accounts.google.com",
:token_url => "/o/oauth2/token",
:token_method => :post,
:grant_type => "refresh_token",
:scope => "https://www.googleapis.com/auth/userinfo.email https://mail.google.com/"
})
oauth2_access_token = client.get_token(
"client_id" => oauth2_client_id,
"client_secret" => oauth2_client_secret,
"refresh_token" => oauth2_refresh_token,
"grant_type" => "refresh_token")
Now that you have an access token, you can use it to authenticate with Gmail through the SASL XOAUTH2 mechanism.
First, we need to update Net::IMAP to add XOAUTH2 as an authenticator method. According to Google’s documentation, this can be done by base64 encoding a specially formatted string containing the user’s email address and a valid access token.
More information: The SASL XOAUTH2 Mechanism
In practice, base64 encoding the string didn’t work, but providing an unencoded string did. It feels like they might quietly change this one day, so be careful if you use this in production.
require 'net/imap'
class Net::IMAP
class XOAuth2Authenticator
def initialize(email_address, access_token)
@email_address = email_address
@access_token = access_token
end
def process(s)
# HACK!!! - The docs say that we need to base64 encode the
# following line; but that doesn't work in practice.
"user=#{@email_address}\x01auth=Bearer #{@access_token}\x01\x01"
end
end
add_authenticator 'XOAUTH2', XOAuth2Authenticator
end
Now that we’ve added an XOAUTH2 authenticator, connecting to IMAP is simple:
# The user's email address.
email_address = "..."
# Connect to IMAP.
client = Net::IMAP.new("imap.gmail.com", :port => 993, :ssl => true)
client.authenticate('XOAUTH2', email_address, oauth2_access_token.token)
# ...do stuff...
client.disconnect()
That’s all. Hopefully this saves you many, many hours of frustration.
]]>git-gpg is a git plugin I created. It lets you safely host a git repository with sensitive files on an untrusted server by GPG-encrypted the repository.
]]>The main challenge in getting OpenVBX to run on AppFog is that the OpenVBX installer appears to work out of the box on AppFog. But then, when you redeploy your app, you overwrite the config files generated by the installer, so all of the changes are lost. This is because the installer overwrites some config files.
At this point, if you turn to the OpenVBX docs, it appears that your solution is to skip the installer and instead update the sample versions of the config files. This WILL NOT WORK for two reasons: 1) it won’t create the database and 2) it won’t actually set the settings correctly, leading to some weird confounding error. In my case, I experienced an infinite redirect loop.
The actual solution is fairly simple: run through the complete installation process, let the installer properly initialize the database and generate the config files. Then, use a helper script to grab the contents of the config files, and recreate them locally.
Here is a step-by-step guide:
Download or clone OpenVBX (http://www.openvbx.org/)
Add a file called helper.php at the top level OpenVBX directory with the
content below. This will simply display the content of the config files that
the OpenVBX installer changes. (We will remove this file after installing
OpenVBX, as it exposes sensitive information.)
Create a new AppFog app and MySQL instance. Bind the MySQL instance to your app. Deploy your app to AppFog. (http://blog.appfog.com/getting-started-with- appfogs-command-line/) Make sure you start with a fresh MySQL database, don’t attempt to re-use a database from a previous installation attempt; it will not work.
Navigate to your http://YOUR-APP.aws.af.cm/helper.php.
In a separate browser window, navigate to http://YOUR-APP.aws.af.cm and run through the
OpenVBX install process. Use the database connection settings displayed by
helper.php to configure the database. You do not need to enter the port
number, just the database hostname (which was an IP address for me), database
name, username, and password. Continue through the rest of the install
process. This will initialize the database.
Once the installer has finished and you see a login prompt, refresh
http://YOUR-APP.aws.af.cm/helper.php. You will see that the installer has created openvbx.php and
database.php config files. Copy that content, and use it to create
openvbx.php and database.php files locally.
Delete the helper.php file!
Redeploy your app. In theory, you should see a login prompt when you navigate to your app’s URL.
<h1>Database Settings</h1>
<pre>
<?= htmlspecialchars(getenv("VCAP_SERVICES")) ?>
</pre>
<hr />
<h1>openvbx.php</h1>
<pre>
<?= htmlspecialchars(file_get_contents("OpenVBX/config/openvbx.php")) ?>
</pre>
<hr />
<h1>database.php</h1>
<pre>
<?= htmlspecialchars(file_get_contents("OpenVBX/config/database.php")) ?>
</pre>
It’s New Year’s Eve, and countless resolutions are being made today, only to be broken within the week. A common resolution is Learn Something New, and a common approach is to set aside 15 minutes a day for learning.
“15 minutes will be easy to find”, we rationalize. “I spend hours checking Facebook, watching TV, playing Angry Birds. Surely I can find 15 minutes to improve myself and become more fulfilled, more well-rounded, or more employable?”
This approach is misguided. Complete bullshit. You can’t learn anything substantial in 15 minutes a day. Not only that, but it takes an enormous amount of discipline to do anything for 15 minutes a day, every day. You are setting yourself up for failure.
A better approach:
Think back to when you learned to ride a bike, or beat a video game. Think about when you got completely addicted to some TV show, learned all about the plot and characters, and scoured the Internet looking for next season’s plot spoilers. Model your learning after that.
Immerse yourself. Follow rabbit holes. Have fun. Spend hours. Skip meals. Don’t lesson plan. Just be curious. The learning will follow.
]]>If you have experience with OLAP or multi-dimensional processing, you should be able to ramp up on Crossfilter fairly quickly. If you only have experience with relational databases, it may take a little longer. If you’ve never used the SQL group-by feature, then you face a steep learning curve.
First, you’ll need to understand facts , dimensions , and measures . (If
you’re already familiar with these terms, then skip this section.)
Imagine you want to answer the question “How many orders do we process per week?”
You could calculate this by hand by iterating through all of the orders that
your business had processed, grouping them into weeks. In this case, each
order entry would be called a fact , and you would probably store this in an
OrderFacts table. The week would be a dimension ; it is a way you want to
slice the data. And the count of orders would be a measure , it is a value
that you want to calculate.
Imagine another question, “How much revenue do we book per salesperson per
week?” Again, your facts would be stored in an OrderFacts table. You would
now have two dimensions , salesperson and week. And finally, your measure
is dollars per order.
Below, we’re going to answer some questions like “How many living things live in my house?” and “How many legs of each type exist in my house?”
It’s incredibly easy to get your fact data into Crossfilter: just use JSON. Each row is a fact.
Below, we’ve created a Crossfilter object loaded with facts about the living things in my house.
(Note: These are, for the most part, “fictional” facts. I don’t actually have (any pets, but it makes for a good tutorial.)
var livingThings = crossfilter([
// Fact data.
{ name: “Rusty”, type: “human”, legs: 2 },
{ name: “Alex”, type: “human”, legs: 2 },
{ name: “Lassie”, type: “dog”, legs: 4 },
{ name: “Spot”, type: “dog”, legs: 4 },
{ name: “Polly”, type: “bird”, legs: 2 },
{ name: “Fiona”, type: “plant”, legs: 0 }
]);
That’s it. Now let’s find out some totals. For example, how many living things are in my house?
To do this, we’ll call the groupAll convenience function, which selects all
records into a single group, and then the reduceCount function, which
creates a count of the records. Not very useful so far.
// How many living things are in my house?
var n = livingThings.groupAll().reduceCount().value();
console.log(“There are ” + n + “ living things in my house.”) // 6
Now let’s get a count of all the legs in my house. Again, we’ll use the
groupAll function to get all records in a single group, but then we call the
reduceSum function. This is going to sum values together. What values? Well,
we want legs, so let’s pass a function that extracts and returns the number of
legs from the fact.
// How many total legs are in my house?
var legs = livingThings.groupAll().reduceSum(function(fact) { return fact.legs; }).value()
console.log(“There are ” + legs + “ legs in my house.”) // 14
Now let’s test out some of the filtering functionality.
I want to know how many living things in my house are dogs, and how many legs
they have. For this, we’ll need a dimension . Remember that a dimension is
something you want to group or filter by. Here, the dimension is going to be
the type . Crossfilter can filter on dimensions in two ways, either by exact
value, or by range.
Below, we construct a typeDimension and filter it:
// Filter for dogs.
var typeDimension = livingThings.dimension(function(d) { return d.type; });
typeDimension.filter(“dog”)
That’s it. Dimensions are stateful, so Crossfilter knows about our filter, and
will ensure that all future operations are filtered to only work on dogs
except for any calculations performed directly on typeDimension . This is
expected behavior, but I’m not sure if it’s a design choice or a design
necessity. (We’ll look at the workaround later.)
var n = livingThings.groupAll().reduceCount().value();
console.log(“There are ” + n + “ dogs in my house.”) // 2
var legs = livingThings.groupAll().reduceSum(function(fact) {
return fact.legs;
}).value()
console.log(“There are ” + legs + “ dog legs in my house.”) // 8
Let’s clear the filter, then do some grouping.
// Clear the filter.
typeDimension.filterAll()
I want to know how many living things of each type are in my house. I already
have a dimension grouped by type called typeDimension .
Using typeDimension , I’m going to group the records by type, and then
create a measure that returns the count called countMeasure . Once
countMeasure is created, we can find the number of entries by calling
countMeasure.size() (a.k.a the cardinality of the type dimension), and we
can get the actual counts by calling countMeasure.top(size).
// How many living things of each type are in my house?
var countMeasure = typeDimension.group().reduceCount();
var a = countMeasure.top(4);
console.log(“There are ” + a[0].value + “ ” + a[0].key + “(s) in my house.”);
console.log(“There are ” + a[1].value + “ ” + a[1].key + “(s) in my house.”);
console.log(“There are ” + a[2].value + “ ” + a[2].key + “(s) in my house.”);
console.log(“There are ” + a[3].value + “ ” + a[3].key + “(s) in my house.”);
Awesome. Now let’s count legs by type. For this, we’ll create a dimension
called legMeasure . This will use the reduceSum function instead of
reduceCount , and we’ll provide a function that tells Crossfilter what field
we want to sum.
// How many legs of each type are in my house?
var legMeasure = typeDimension.group().reduceSum(function(fact) { return fact.legs; });
var a = legMeasure.top(4);
console.log(“There are ” + a[0].value + “ ” + a[0].key + “ legs in my house.”);
console.log(“There are ” + a[1].value + “ ” + a[1].key + “ legs in my house.”);
console.log(“There are ” + a[2].value + “ ” + a[2].key + “ legs in my house.”);
console.log(“There are ” + a[3].value + “ ” + a[3].key + “ legs in my house.”);
As mentioned earlier, when you filter on a dimension, and then roll-up using said dimension, Crossfilter intentionally ignores any filter an said dimension.
For example, this does not return what you would expect:
// Filter for dogs.
typeDimension.filter(“dog”)
// How many living things of each type are in my house?
// You’d expect this to return 0 for anything other than dogs,
// but it doesn’t because the following statement ignores any
// filter applied to typeDimension:
var countMeasure = typeDimension.group().reduceCount();
var a = countMeasure.top(4);
console.log(“There are ” + a[0].value + “ ” + a[0].key + “(s) in my house.”);
console.log(“There are ” + a[1].value + “ ” + a[1].key + “(s) in my house.”);
console.log(“There are ” + a[2].value + “ ” + a[2].key + “(s) in my house.”);
console.log(“There are ” + a[3].value + “ ” + a[3].key + “(s) in my house.”);
The workaround is to create another dimension on the same field, and filter on that:
// Filter for dogs.
var typeFilterDimension = livingThings.dimension(function(fact) { return fact.type; });
typeFilterDimension.filter(“dog”)
// Now this returns what you would expect.
var countMeasure = typeDimension.group().reduceCount();
var a = countMeasure.top(4);
console.log(“There are ” + a[0].value + “ ” + a[0].key + “(s) in my house.”);
console.log(“There are ” + a[1].value + “ ” + a[1].key + “(s) in my house.”);
console.log(“There are ” + a[2].value + “ ” + a[2].key + “(s) in my house.”);
console.log(“There are ” + a[3].value + “ ” + a[3].key + “(s) in my house.”);
Crossfilter is built to be insanely fast. To do that, rather than completely re-calculating groups as filters are applied, it calculates incrementally. Crossfilter does that by using a bitfield to track whether or not a fact exists in a specific dimension. For that reason, Crossfilter dimensions are expensive, so you should think carefully about creating them and create as few as possible.
I’m the co-founder of FiveStreet, a technology startup company that helps leading real estate agents beat their competition when responding to online leads. We’re actively looking for a designer, business development folks, and (of course) more customers. If you can help me connect to any of the aforemention folks, please get in touch.
]]>