Guest Post: Big Data - Does Size Matter? by Timandra Harkness

This evening, writer, comedian and data expert Timandra Harkness launches her book – Big Data – Does Size Matter, complete with a special mention of a certain Black Swan Data on page 23! A long-time friend of ours, Timandra has written this guest post for our blog to celebrate. If you are interested in buying the book, make sure you read right to the bottom for a special discount code. “When I started writing this book, I told my publisher, Jim, that I was going to log how many cups of tea I’d need to finish it. His prediction was 550. I told him that was risibly low. So I started a spreadsheet of hours spent at my computer, words produced, and cups of tea drunk. For the first three months, however long I wrote for, my daily tea intake only once rose above two. Looking at my word output, and at the time remaining before my deadline, I realised I needed to drink more tea. Partly because an intake below one cup per hour is clearly inadequate, and partly (if I’m honest) because I didn’t want to fall below Jim’s estimate. I worked out what was causing the problem. I would make one cup of tea, and drink it while I read the previous day’s work. Then I’d make another while I thought about it, and drink that while I wrote. After that, I was too absorbed in writing to leave my desk again until I’d hit my target for the day. So I bought a tea trolley. That way, I could make a pot of tea, take it into the office on a tray, with a milk jug and a strainer (we’re writers, not savages) and have several cups of tea without leaving the keyboard. Instant results. My daily tea count went up to four on most days, with the occasional five. One day went as high as seven, but that was a 12-hour day towards the end of the process, and to be honest I was probably drinking more whisky than tea at that stage. Nevertheless, in spite of my best efforts, and perhaps the whisky was partly to blame here, my total by the time I delivered the finished manuscript was a mere 254 cups of tea. Clearly, I’d never get a job in predictive analytics. For a start, my methodology was rubbish. I only included tea drunk (and hours spent) at my desk, whereas the writing process also involved weeks of travelling, interviewing people (often over tea or coffee) and scribbling notes in a notebook. Yes, with a pen. Not to mention many, MANY hours of reading books and articles, and discussing ideas both formally at public events and informally, in the pub (stronger drinks than tea). So the end figure was an artefact of measurement technology, more than a true reflection of my liquid intake between contract and delivery. Also, I didn’t base my prediction on anything more than an unquantified hunch that I drink a lot of tea. No data, either from my previous life, or from comparable authors, was input. And, having never written a book, I had no firm idea of how many hours I would spend writing this one. What I should have done was get Black Swan Data to predict my tea requirements, the same way they predict much more complicated phenomena including the ones I feature in Chapter 1. Knowing when to fill a supermarket’s shelves with burgers, buns and barbecue fuel, because it’s the first sunny weekend of the year, for example. Or where people are most likely to want vitamins and cold remedies, as seasonal illnesses move through a population. I realise it’s not magic, but a potent combination of human judgment, ingenious mathematics, and using AI to discern patterns and project them into the future. And I realise it’s not astrology, but a working model that can be used as a basis for action, much as my projected tea intake gave me the insight to buy a tea trolley and change the outcome. And I know that it can’t foresee all changes in the real world. Who would have guessed how much I’d need that whisky to get me through the last six weeks of writing? And I know that human beings are inclined to behave like individuals with our own agendas, and may ruin your predictions by switching to a teapot to try and affect the final score. But it’s a lot more powerful than a hunch. So next time I write a book, I’m going to ask Black Swan Data to tell me how many cups of tea, and bottles of whisky, I’ll need to get me from publisher’s handshake to print. At least next time we’ll have some data to go on.” Bloomsbury have very kindly offered friends of Black Swan Data a 25% discount on Timandra’s book. To receive the discount, please buy the book here, and enter discount code DATA at the checkout. You can follow Timandra on Twitter at @TimandraHarknes and follow updates about the book at @BigDataSizeBook Timandra Harkness A writer, comedian and broadcaster, Timandra has been performing on scientific, mathematical and statistical topics since the latter days of the 20th Century. She has written about travel for the Sunday Times, motoring for the Telegraph, science & technology for WIRED, BBC Focus Magazine and Men’s Health Magazine, and is a regular on BBC Radio, and resident reporter on social psychology series The Human Zoo. Timandra also writes and presents documentaries and BBC Radio 4’s FutureProofing series, and is currently writing a new comedy show about Big Data. In her spare time she is studying for an Open University degree in Mathematics & Statistics.