Is The Age of The Spreadsheet Over?


I exploit Python every day to do data analysis for clients. Prior to that I used Excel for over 20 years. Now that I’m gifted in Python and its data science libraries, I to find I’m extra productive and in a position to enforce a lot more tough research ways with extra self assurance than I ever used to be with spreadsheets. I actually simplest use Excel to percentage information and ultimate effects with shoppers.

So I discovered myself questioning “is the spreadsheet useless?” and if no longer, “why no longer?”

My first reaction used to be to sentence spreadsheets to the trash can of computing historical past — I spent too many hours wrestling with ridiculously-complicated formulation, tracing damaged cellular references or looking to recuperate corrupt information. On the other hand, after I started to suppose extra deeply concerning the essence of the spreadsheet, I started to appreciate that they’ve inherent options that programming languages lack.

Spreadsheets display us the information — no longer the calculations at the back of them.

Let’s take a very easy instance. One thing that any people would possibly want to do. Consider we simply were given again from a street go back and forth and we want to work out who owes who and what sort of.

Microsoft Excel Spreadsheet

This situation is so intuitive it doesn’t want an evidence. Once we are the originators of the information in addition to the general consumers, we already perceive the aim of the research and most often we will be able to wager how the calculations have been made simply by taking a look on the information and the consequences. On this case, you need to say “the information speaks for itself.”

Every other function I really like about Excel is you’ll be able to make a selection more than one cells with the mouse and spot not unusual statistics such because the sum and the typical.

But no method is visual — we need to click on at the cells to peer the calculations at the back of them. No variable title used to be declared. We merely positioned every information level in a distinct cellular in an association that made sense to us, after which attached the cells with calculations and cellular references. The computation is totally obscured from view. I feel that is one explanation why spreadsheets are so helpful and so widespread; we all know our information, we all know what it represents, and we adore to peer it.

Against this, maximum people don’t like gazing pc code. Programming languages are text-based and linear. Right here’s the identical go back and forth expense calculation in Python.

import pandas as pd# Go back and forth bills
information = {
'Meals': [38.15, 0, 109.75],
'Automobile': [139, 0, 0],
'Gas': [25.08, 0, 0],
'Tickets': [0, 134, 0],
'Different': [95, 0, 250]
index = ['Diane', 'Kelly', 'John']
df = pd.DataFrame(information, index=index)
# Calculate quantity owing
df['Total Paid'] = df.sum(axis=1)
average_cost = df['Total Paid'].imply()
df['Amounts Owing'] = average_cost - df['Total Paid']
print(df['Amounts Owing'])

This produces:

Diane    -33.57
Kelly 129.66
John -96.09
Identify: Quantities Owing, dtype: glide64

The code makes the computational procedure and the calculation means particular however from an information standpoint it’s extra difficult to understand. We use labels to seek advice from the information however the information isn’t in reality visual till we print it.

What values do the intermediate variables comprise? To look that you wish to have an interactive programming surroundings. For instance, Python has an interpreter that permits you to check out variables and execute small items of code one-at-a-time.

In the event you had simply run the script above, you need to use the interpreter to perform a little research at the effects.

>>> df['Total Paid'].idxmax()
>>> df.loc['John']
Meals 109.75
Automobile 0.00
Gas 0.00
Tickets 0.00
Different 250.00
Overall Paid 359.75
Owing -96.09
Identify: John, dtype: glide64

Trendy programming languages have made the information extra obtainable by way of offering interactive environments during which you’ll be able to discover and visualize it.

If one thing about programming, you’ll be able to simply decipher what the code does. Each and every information object is assigned a reputation and has an outlined kind or construction. Programming languages have a big array of object sorts and strategies and make allowance exterior programs to be imported to offer further features.

Within the instance code above, I used an object referred to as a dataframe this is specially-designed for dealing with data-computations in two dimensions.

Dataframe Construction

The number of ‘fit-for-purpose’ object buildings is what makes pc code powerful, tough and computationally environment friendly.

To conquer one of the barriers of previous spreadsheets with simplest rows and columns, Microsoft added tables and different information buildings to Excel to assist you to do extra subtle information analyses — very similar to information buildings from pc languages.

Right here’s the similar spreadsheet from above however the use of the integrated Desk layout and structured referencing in Excel.

The desk object has a different syntax for referencing its components. As an alternative of


we will be able to now write:


That is extra intuitive and no more prone to ruin if we make adjustments to the desk (the ‘@’ image method ‘from the present row’).

The identical in Python could be:

df.loc['Diane', 'Food':'Other'].sum()

One reason why I feel we like the spreadsheet is its resemblance to excellent, previous paper. Writing on paper offers us regulate and predictability. Issues are all the time the place you left them. An previous good friend of mine refers to paper-based notes half-jokingly as his “continual, versatile, garage gadget”.

Symbol credit score: Mike Gresley CEA Portfolio

Although we rarely use paper for numerical paintings at the moment, it has a definite intuitive attraction. I feel that’s as a result of we see the arena in 2-dimensions (prior to translating the picture to 3-dimensions). It’s simple to peer the place the whole lot is on a flat floor specified by entrance of you. Although the realm is simply too large to view immediately, we bear in mind the place we put issues as a result of we intuitively construct a 2-dimensional map of the place the whole lot is situated. (The tabs on an MS workbook be offering some other measurement however it’s actually only a selection of labelled spreadsheets.)

Programmers creating consumer programs talk about the ‘GUI’ or graphical consumer interface. The whole thing this is designed to be used by humans needs a user interface. Spreadsheet apps are not any exception. If truth be told, the spreadsheet actually is a GUI. The uniform grid of cells with standardized capability, is each a GUI and on the similar time a building gadget in the similar approach that Lego is.

Whilst you first get started to your spreadsheet, it appears to be like quite simple—identical to a clean sheet of paper. However when you’ve discovered the best way to upload information and formulation, reference different cells and layout them, you’ll be able to use this extremely flexible building gadget to create an unimaginably huge array of various programs. The general product, a spreadsheet designed for some explicit objective, can be a GUI to the top consumer. Therein lurks a danger, in fact, however I believe that within the overwhelming majority of instances, spreadsheets are utilized by the similar those that created them. The developer is the consumer. So it is sensible that the improvement surroundings is the consumer interface.

A spreadsheet unleashes your creativity and provides you with the liberty to construct one thing in step with our personal imaginative and prescient. This is likely one of the causes that making spreadsheets can also be so stress-free. It’s each a computation platform and a user-interface, whether or not it’s a easy data-entry shape, a nicely-formatted file, or a stupendous chart.

Against this, pc code appears to be like monotone — it’s a must to learn the variable names and navigate a text-based code. Not like a spreadsheet, code script is one-dimensional — a linear format. You specify the appropriate order during which you need every command to be accomplished. The level to which you’ll be able to personalize code visually is proscribed by way of the syntax.

Even though chances are you’ll no longer have considered it on this approach, Excel constructs a computation graph ‘behind-the-scenes’ in order that it will possibly execute the calculations in the right kind order (I’m no longer precisely certain how however I guess that it makes use of some more or less directed graph). That graph and the method of constructing it’s treated routinely and obscured from you because the consumer. Your activity is simplest to outline the information and the dependencies between them.

What’s a computation graph? A computation graph is a community diagram that describes the useful relationships between the enter information and the outputs. They’re used in machine learning applications the place many complicated computations are chained in combination in sequence. Right here’s what the computational graph for the Go back and forth Expense calculation would possibly seem like. Each and every field is an information merchandise and the arrows constitute calculations.

Computation Graph for the Go back and forth Expense Calculation

The computation graph isn’t all the time evident while you have a look at a spreadsheet however in the event you lay out your information logically and label it smartly, it will have to be simple to believe it. The formulation menu in Excel has some great gear to help in making the graph extra visual by way of drawing arrows to turn the dependencies between cells however I don’t to find it that helpful for extra complicated graphs.

Visualizing Dependencies in Excel

Dataflow programming is totally diversified to the extra not unusual sequential or procedural programming paradigms. As information analysts, I feel we discover it more straightforward to take into accounts computation as a graph or information float style than to take into accounts one lengthy series of computation steps. That is the place Excel actually does excel!

Once we’re development a spreadsheet, we almost certainly have some thought in our head concerning the computation graph we wish however Excel shall we us do it in an overly versatile and fluid approach. We don’t want to get started at first and we will be able to simply disconnect and sign up for diversified portions of the graph later (till you create a circular reference in fact, which I’m certain we’ve all performed a couple of times).

In all probability strangely, there is not any integrated software to routinely care for the computation graph after we are writing pc code. It’s as much as you to grasp the order during which calculations want to occur and in the event you get it incorrect there gained’t be any caution.

For instance, say we were given one of the most bills incorrect and we want to alternate it. Right here’s what we would possibly do.

>>> df.loc['John', 'Other'] = 25  # Proper price
>>> df
Meals Automobile Gas Tickets Different Overall Paid Quantities Owing
Diane 38.15 139 25.08 0 95 297.23 -33.57
Kelly 0.00 0 0.00 134 0 134.00 129.66
John 109.75 0 0.00 0 25 359.75 -96.09

The correction used to be made to at least one information merchandise however the remainder of the values within the dataframe weren’t recalculated (as they’d be routinely in a spreadsheet).

Software builders would by no means make this error in fact — they sparsely write the code in the right kind series and hand-code special checks to make certain that any alternate to the enter information routinely triggers recalculation of all variables that rely on it.

From my revel in, which means while you write code for information research, you wish to have to suppose very sparsely concerning the computation graph proper from the beginning and plan and prepare your code accordingly. Whilst you write a program you might be necessarily figuring out the computation graph by way of the order during which the statements are written. Except you might be excellent at object-oriented programming, it may be a large number of paintings to modify the items and relationships later in the event you get them incorrect. We’re no longer all skilled programmers so why can’t the pc do it for us? There are a couple of gear that would lend a hand comparable to d6tflow and this interesting Python package however those are specialised gear and no longer used that ceaselessly.

It’s arduous to dynamically alternate the scale and dimensions of information units in Excel. You need to insert or delete rows and columns by way of hand. The nice benefit of programming approaches is that when you’ve computerized your research it’s simple to scale it up and make it configurable for numerous different-sized duties.

Even though it would take extra time and effort to debug, after you have a operating pc program, it has a tendency to be a lot more powerful (much less at risk of human error), is almost certainly extra environment friendly, and is scalable to a lot greater information units than Excel is in a position to processing.

Additionally, there are a number of gear and web-based services and products to be had that permit you to borrow and percentage code, collaborate and organize adjustments and model updates.

The most productive of each worlds could be an interactive surroundings the place we will be able to create workflows and spot our information in an interesting and intuitive visible layout. On the similar time, we wish tough information buildings certain in combination by way of computational operations in a strong computation graph. So far as I do know, we’re no longer there but even supposing there are some attention-grabbing tendencies and issues are transferring rapid.

Initially, open-source gear such because the Jupyter Notebook have actually made programming extra pleasant for the ones people who aren’t skilled device builders. The method of code writing and execution isn’t as disconnected because it was so now we will be able to step thru code, check out the values of variables, and experiment with it (debugging) all in the similar view. The screenshot under displays how you’ll be able to view information in a Jupyter pocket book proper subsequent to the code as you might be enhancing it.

Code and Information in a Jupyter Pocket book

However what concerning the computation graph? Unfortunately, the Jupyter pocket book doesn’t supply a technique to visualize the inter-connections between information items or regulate their dependencies. If truth be told, there are quite a few problems with Jupyter notebooks when you get started executing code in a non-linear approach because the pocket book allows you to.

In all probability probably the most promising route is the new building of industrial intelligence (BI) programs that supply a visible, point-and-click method to information processing with a strong information workflow mixed with tough information research and visualization features.

Symbol Supply: Promotional Video from Tableau

There are too many to say and I simplest have restricted revel in with a couple of (e.g. Tableau, TIBCO Spotfire, Microsoft BI) however all of them have a complete suite of features that spans all the information float pipeline from acquisition to reporting and collaboration. Many also have powerful machine learning capabilities in-built.

Symbol Supply: Tableau

A few of these gear have now built-in high-level programming languages into their merchandise. For instance, TIBCO Spotfire comes with a built-in Python environment permitting customers so as to add their very own customized Python code. Tableau, has its personal distinctive calculation language, very similar to Excel formulation.

Symbol Supply: Knime Analytics

Looker, some other widespread platform, justify their determination to create a new language called LookML by way of arguing that present languages weren’t in a position and claiming that theirs is more straightforward to be told. I feel it is a mistake. Studying any new programming language is a large funding which simplest will pay off if the language is extensively used and stands the check of time. Languages comparable to Python, R, and Java will likely be round for the foreseeable long term and they’re almost certainly the most productive languages for high-level information research and system finding out at this time (I believe MATLAB a runner-up and it isn’t open-source and due to this fact much less readily to be had).

What I think those tough BI programs lack is the straightforward, intuitive really feel of the spreadsheet and the inventive freedom it offers. Prior to you’ll be able to paintings with complicated, integrated features, you wish to have to have a excellent connection on your information and a transparent, clear operating surroundings that you’re feeling regulate over, no longer crushed by way of.

Generally, I might say that spreadsheets are ideal for other people doing same old information research duties and people who don’t have time to learn to code. They mean you can briefly discover your information and convey some fast effects with no need to suppose an excessive amount of about it.

Comparability Desk

Programming languages like Python are tough and will care for greater volumes of information extra reliably than a spreadsheet. If you wish to have to enforce a strong, environment friendly information processing pipeline this is going for use greater than as soon as and must be depended on by way of folks, then Python (or every other high-level programming language comparable to C#, R, Java, or MATLAB) is almost certainly methods to cross. However there’s a large finding out curve in the event you haven’t programmed prior to.

After all, business BI gear (assuming you might have get admission to to at least one) might be the technique to bridge the distance between the spreadsheet and the hand-coded pc program. However I haven’t noticed one but that has each the intuitive attraction of the spreadsheet and the power to make use of tough high-level programming languages like Python.

I feel we nonetheless want to work out what that absolute best operating surroundings is. We wish an answer that achieves the next:

  1. As intuitive as writing on a work of paper
  2. Supplies a clear interface to the information
  3. Makes the information construction visual and manages the computation graph
  4. Mechanically scales to deal with different-sized datasets
  5. Lets in custom designed capability to be added the use of business same old information research gear and programming languages.

Till we have now this, I don’t suppose the age of the spreadsheet is any place close to over.

If one of the ideas on this article are new to you, take a look at those tutorials and finding out sources.

Source link


Please enter your comment!
Please enter your name here