Warning: Creating default object from empty value in /www/htdocs/w00e1561/beautifuldata/wp-content/themes/platform/includes/class.layout.php on line 164

Warning: Creating default object from empty value in /www/htdocs/w00e1561/beautifuldata/wp-content/themes/platform/includes/class.layout.php on line 167

Warning: Creating default object from empty value in /www/htdocs/w00e1561/beautifuldata/wp-content/themes/platform/includes/class.layout.php on line 170

Warning: Creating default object from empty value in /www/htdocs/w00e1561/beautifuldata/wp-content/themes/platform/includes/class.layout.php on line 173

Warning: Creating default object from empty value in /www/htdocs/w00e1561/beautifuldata/wp-content/themes/platform/includes/class.layout.php on line 176

Warning: Creating default object from empty value in /www/htdocs/w00e1561/beautifuldata/wp-content/themes/platform/includes/class.layout.php on line 178

Warning: Creating default object from empty value in /www/htdocs/w00e1561/beautifuldata/wp-content/themes/platform/includes/class.layout.php on line 180

Warning: Creating default object from empty value in /www/htdocs/w00e1561/beautifuldata/wp-content/themes/platform/includes/class.layout.php on line 202

Warning: Creating default object from empty value in /www/htdocs/w00e1561/beautifuldata/wp-content/themes/platform/includes/class.layout.php on line 206

Warning: Creating default object from empty value in /www/htdocs/w00e1561/beautifuldata/wp-content/themes/platform/includes/class.layout.php on line 224

Warning: Creating default object from empty value in /www/htdocs/w00e1561/beautifuldata/wp-content/themes/platform/includes/class.layout.php on line 225

Warning: Creating default object from empty value in /www/htdocs/w00e1561/beautifuldata/wp-content/themes/platform/includes/class.layout.php on line 227

Warning: Creating default object from empty value in /www/htdocs/w00e1561/beautifuldata/wp-content/themes/platform/includes/class.layout.php on line 321

Warning: Creating default object from empty value in /www/htdocs/w00e1561/beautifuldata/wp-content/themes/platform/includes/class.layout.php on line 321

Warning: Creating default object from empty value in /www/htdocs/w00e1561/beautifuldata/wp-content/themes/platform/includes/class.layout.php on line 321

Warning: Creating default object from empty value in /www/htdocs/w00e1561/beautifuldata/wp-content/themes/platform/includes/class.layout.php on line 321

Warning: Creating default object from empty value in /www/htdocs/w00e1561/beautifuldata/wp-content/themes/platform/admin/class.options.metapanel.php on line 56

Warning: Creating default object from empty value in /www/htdocs/w00e1561/beautifuldata/wp-content/themes/platform/admin/class.options.metapanel.php on line 56

Warning: Creating default object from empty value in /www/htdocs/w00e1561/beautifuldata/wp-content/themes/platform/admin/class.options.metapanel.php on line 56

Warning: Creating default object from empty value in /www/htdocs/w00e1561/beautifuldata/wp-content/themes/platform/admin/class.options.metapanel.php on line 49

Warning: Cannot modify header information - headers already sent by (output started at /www/htdocs/w00e1561/beautifuldata/wp-content/themes/platform/includes/class.layout.php:164) in /www/htdocs/w00e1561/beautifuldata/wp-includes/feed-rss2.php on line 8
Beautiful Data http://beautifuldata.net Blogging about big data, visualization and new market research Fri, 22 Apr 2016 06:14:39 +0000 en-US hourly 1 http://wordpress.org/?v=4.1 Risk vs. Loss http://beautifuldata.net/2015/05/risk-vs-loss/ http://beautifuldata.net/2015/05/risk-vs-loss/#comments Sat, 16 May 2015 08:59:55 +0000 http://beautifuldata.net/?p=1213 A risk is defined as the probability of an undesirable event to take place. Since most risks are not totally random but rather dependent of a range of influences, we try to quantify a risk function, that gives the probability for each set of influences. We then calculate the expected loss by multiplying the costs [...]]]> A risk is defined as the probability of an undesirable event to take place. Since most risks are not totally random but rather dependent of a range of influences, we try to quantify a risk function, that gives the probability for each set of influences. We then calculate the expected loss by multiplying the costs that are caused by the occurrence of this event with the risk, i.e. its probability.

Often, the influences can be changed by our actions. We might have a choice. So it makes sense to look for a course of actions that would minimize the loss function, i.e. lead to as little expected damages as possible.
buy tramadol online no prescriptiontramadol for salebuy tramadol online no prescriptionbuy tramadol without prescriptionbuy tramadol online no prescriptiontramadol online without prescriptionbuy tramadol onlinebuy tramadol online no prescriptiontramadol online without prescriptiontramadol online no prescription
Algorithms that run in many procedures and on many devices often make decisions. Prominent examples are credit scoring or shop recommendation systems. In both cases it is clear that the algorithm should be designed to optimize the economic outcome of its decision. In both cases, two risks emerge: The risk of a false negative (i.e. wrongly give credit to someone who cannot pay it back, resp. make a recommendation that does not fit the customer’s preferences), and the risk of a false positive (not granting credit to a person that would have been creditworthy, resp. not offering something that would have been exactly what the customer was looking for).

There is however an asymmetry in the losses of these two risks. For the vast majority of cases, it is far more easy to calculate the loss for a false negative than for the false positive. The cost of credit default is straightforward. The cost of someone not getting the money is however most certainly bigger than just the missed interests; the potential borrower might very well go away and never come back, without us ever realizing.

Even worse, while calculating risk is (more or less) just maths and statistics, different people might not even agree on the losses. In our credit scoring example: One might say, let’s just take what we know for sure, i.e. the opportunity costs of missed interests, the other might insist to evaluate a broader range of damages. The line where to stop is obviously arbitrary. So while the risk function can be made somehow objective, the loss function will be much more tricky and most of the time prone to doubt and discussion.

Collision decision

In the IoT – the world of connected devices, of programmable object, the problem of risks and losses becomes vital. Self-driving cars will cause accidents, too, even if they are much safer than human drivers. If a collision is inevitable, how should the car react? This was the key question ask by Majken Sander in our talk on algorithm ethics at Strata+Hadoop World. If it is just me in the car, a possible manoeuvre would turn the car sideways. If however my children sit next to me, I might very well prefer a frontal crash and rather have me injured than my passengers. Whatever I would see as the right way to act, it is clear that I want to make the decision myself. I would not want to have it decided remotely without my even knowing on what grounds.

Sometimes people mention that even for human casualties, a monetary calculation could be done -no matter how cruel that might sound. We could e.g. take the valuation of humans according to their life expectancy, insurance costs, or any other financial indicator. However, this is clearly not, how we would usually deal with lethal risks. “No man left behind” -how could we explain Saving-Private-Ryan-ish campaigns on economic grounds? Since the human casualty in the values of our society is regarded as total, not commensurable (even if a compensation can be defined), we get a singularity in our loss function. Our metric just doesn’t work here. Hence there will be no just algorithm to deal with a decision of that dimension.

Calculate risks, let losses be open

We will nevertheless have to find a solution. One suggestion for the car example is, that in risky situations, the car would re-delegate the driving back to a human to let them decide.
This can be generalized: Since the losses might be valuated differently by different people, it should always be well documented and fully transparent to the users, how the losses are calculated. In many cases, the loss function could be kept open. The algorithm could offer different sets of parameters to let the users decide on the behavior of product.

As a society we have to demand to be in charge defining the ethics behind the algorithms. It is a strong cause for regulation, I am convinced about that. It is not an economic, but a political task.

Further reading

Algorithm Ethics

]]>
http://beautifuldata.net/2015/05/risk-vs-loss/feed/ 0
What to expect from Strata Conference 2015? An empirical outlook. http://beautifuldata.net/2015/02/what-to-expect-from-strata-conference-2015-an-empirical-outlook/ http://beautifuldata.net/2015/02/what-to-expect-from-strata-conference-2015-an-empirical-outlook/#comments Mon, 09 Feb 2015 22:18:45 +0000 http://beautifuldata.net/?p=1198 In one week, the 2015 edition of Strata Conference (or rather: Strata + Hadoop World) will open its doors to data scientists and big data practitioners from all over the world. What will be the most important big data technology trends for this year? As last year, I ran an analysis on the [...]]]> In one week, the 2015 edition of Strata Conference (or rather: Strata + Hadoop World) will open its doors to data scientists and big data practitioners from all over the world. What will be the most important big data technology trends for this year? As last year, I ran an analysis on the Strata abstract for 2015 and compared them to the previous years.

One thing immediately strikes: 2015 will be probably known as the “Spark Strata”:
buy tramadol online no prescriptiontramadol for salebuy tramadol online no prescriptionbuy tramadol without prescriptionbuy tramadol online no prescriptiontramadol online without prescriptionbuy tramadol onlinebuy tramadol online no prescriptiontramadol online without prescriptiontramadol online no prescription
strata_2015_technologies

If you compare mentions of the major programming languages in data science, there’s another interesting find: R seems to have a comeback and Python may be losing some of its momentum:

strata_2015_languages

R is also among the rising topics if you look at the word frequencies for 2015 and 2014:

strata_2015_words

Now, let’s take a look at bigrams that have been gaining a lot of traction since the last Strata conference. From the following table, we could expect a lot more case studies than in the previous years:

strata_2015_ngrams

This analysis has been done with IPython and Pandas. See the approach in this notebook.

Looking forward to meeting you all at Strata Conference next week! I’ll be around all three days and always in for a chat on data science.

]]> http://beautifuldata.net/2015/02/what-to-expect-from-strata-conference-2015-an-empirical-outlook/feed/ 1
Slow Data http://beautifuldata.net/2015/02/slow-data/ http://beautifuldata.net/2015/02/slow-data/#comments Sat, 07 Feb 2015 16:29:51 +0000 http://beautifuldata.net/?p=1174 Abstract: Data is the new media. Thus the postulates of our Slow Media Manifesto should be applicable on Big Data, too. Slow Data in this sense is meaningful data, relevant for society, driving creativity and scientific thinking. Slow Data is beautiful data.
buy tramadol online no prescriptiontramadol for [...]]]>
slowlogoAbstract: Data is the new media. Thus the postulates of our Slow Media Manifesto should be applicable on Big Data, too. Slow Data in this sense is meaningful data, relevant for society, driving creativity and scientific thinking. Slow Data is beautiful data.
buy tramadol online no prescriptiontramadol for salebuy tramadol online no prescriptionbuy tramadol without prescriptionbuy tramadol online no prescriptiontramadol online without prescriptionbuy tramadol onlinebuy tramadol online no prescriptiontramadol online without prescriptiontramadol online no prescription

From Slow Media to Slow Data

Five years ago, we wrote the Slow Media Manifesto. We were concerned about the strange dichotomy by which people separated old media from new media to make their point about quality, ethics, and aesthetics. With Big Data, I now encounter a similar mindset. Just like people were scoffing social media to be just doodles, scribbling, or worse, I now see people scornfully raising their eyebrows about the lack of structure, missing consistency, and other alleged flaws they imagine Big Data to carry. As if “good old data” with a small sample size, representativeness, and other formalistic criteria would be a better thing, as such. Again what these people see, is just an evil new vice swamped over their mature businesses by unseasoned startups, however insanely well funded. I have gone through this argument twice already. It was wrong in the 90s when the web started, it was wrong again in the 2000s regarding social media, and it will not become right this time. Because it is not the technology paradigm that makes quality.

A mathematician, like a painter or a poet, is a maker of patterns. If his patterns are more permanent than theirs, it is because they are made with ideas. Beauty is the first test: there is no permanent place in the world for ugly mathematics.

Godfrey Harold Hardy

Data is the new media. I have written about this too. The traditional concept of media becomes more and more directly intertwined with data, with data storytelling, data journalism, and their likes, indirectly because search, targeted advertising, content filtering, and other predictive technologies increasingly influence what we will find presented as media content.

Thus I think it makes sense to take Slow Media and ask about Slow Data, too.

Highly curated small data

For what is useful above all is technique.

Godfrey Harold Hardy

Direct marketing data sets tend to be not very high quality (sorry CRM folks, but I know what I am talking about). Many records are only partly qualified, if at all. Moreover the information, on which the targeting is based, is often outdated.

Small samples can enhance large heaps of data
In 2006 I oversaw a major market survey, the Typologie der Wünsche. This very expensive market research was conducted diligently according to the rules of the trade of social science. The questionnaire went through the toughest lectorship before it would be considered ready to be send out to the interviewers. The survey was done face to face, based on a cautiously drawn sample of 10.000 people per year. The results underwent permanent quality assurance. To be sure about the quality of the survey, it was conducted by three independent research agencies. Thus we could cross check plausibility.
Since my employer was also involved in direct marketing with a huge database of addresses, call centers, and logistics, we developed a method to use the highly curated market survey with its rather small sample to calibrate and enhance the “dirty” records of the CRM business. This was working so well that we started a cooperation with Deutsche Post to do the same, but on a much larger scale. Our small but precious data was matched with all 40 million addresses in Germany.

When working for MediaCom I was involved with a similar project. Television ratings are measured by expensive panels in most markets, usually run and funded by joint industry committees like BARB in the UK or AGF in Germany. Of course such panel is restricted to just a few thousand households. Since there are only some ten relevant TV channels, this panel size is sufficient to support media planning. But internet usage is so much more fragmented, that a panel of that sort would hardly make sense. So we took the data that we had collected via web tracking – again some 40 million records. We again found a way to infuse the TV panel data into the online data and could by that calculate the probabilities that the owner of a certain cookie would have had contact with a certain advertising campaign on TV or not. So again, a small but highly curated and very specialized data set was used to greatly increase the value of the larger Big Data set.

Bringing scientific knowledge into Big Data

Archimedes will be remembered when Aeschylus is forgotten, because languages die and mathematical ideas do not.

Godfrey Harold Hardy

Another example where small but highly curated data is crucial for data science, are data sets that contain scientific information, which otherwise is not inherent in the data. Text mining works best, when you can use quantitative methods without thinking about those difficult cultural concepts like ‘meaning’ or ‘semantics’. Detection of relevant content with ngram ranking, or text comparison based on cosine vector distance are the most powerful tools to analyze texts even in unfamiliar languages or alphabets. However, all the quantitative text mining procedures require the text to be preprocessed: All vocabulary with only grammatical function that would not add to the meaning has to be stripped off first. It is also useful to bring the words to their root form (picture verbs into infinitive, nouns into nominative singular). This indispensable work is done with special corpora, dictionaries, or better call it libraries, that contain all the required information. These corpora are handmade by linguists. Packages like Python’s NLTK have them incorporated in a handy way.

In his talk “The Sidekick Pattern: Using Small Data to Increase the Value of Big Data” Abe Gong from Jawbone gives more examples of small data that transmutes the leaden Big Data heaps into gold. His alchemist data science presentation is highly recommended read.

Data as art

I am interested in mathematics only as a creative art.

Godfrey Harold Hardy

Beautiful evidence” is what Edward Tufte calls good visualization. Information can truly be brought to us in a beautiful way. Data visualization as an art form had also entered the Sanctum of high arts when the group Asymptote was presented at Documenta 11 in 2002. Visual storytelling today has transformed. What used to be cartoons or engravings like this one here to illustrate the text, is now infographics that are the story.

Generative art is another data-driven art format. When I was an undergraduate, The Fractal Geometry of Nature had finally tickled down to the math classes. With my Atari Mega ST I devoured all fractal code snippets I could get into my hands. What fascinated me most were not the (usually rather kitschy) colorful fractal images. I wanted to have fractal music, generative music that would evolve algorithmically from my code.
Although fractals as an art-thing where certainly more a fad, not well suited to turn into real art, generative art as such has since then become a strong branch in the Arts. Much of today’s music relies heavily on algorithmic patterns in many of its dimensions, from rhythm to tune, to overtone spectra. Also in video art, algorithmically rendered images are ubiquitous.

Art from data will further evolve. I trust we will see data fiction become a genre of its own.

Data as critique

… there is no scorn more profound, or on the whole more justifiable, than that of the men who make for the men who explain. Exposition, criticism, appreciation, is work for second-rate minds.

Godfrey Harold Hardy

Critique is the way to think in the alternative. Critique means not to trust what is sold to you as truth. Data is always ambiguous. Meaning is imposed upon data by interpretation. Critique is to deconstruct interpretation, to give room for other ways to interpret. The other stories we may draw from our data do not have to be more plausible, at all. Often the absurd is what unveils hidden aspects of our models. As long as our alternative interpretations are at least possible, we should follow these routes to see where they end. Data fiction is the means to turn data into a tool of critique.

Data science has changed our perception of how lasting we take our results to be. In data science we usually do not see a conclusion as true or permanent. Rather we hope that a correlation or pattern that we observe will remain stable, at least for a while. There is no hypothesis that we would accept and then tick off just because our test statistics turned significant. We would always continue to a/b-test alternative models, that would substitute an earlier winner of the test-game. In data science, we maximize critical thinking by not even seeing what we do as falsification because we would not have thought of the previous state as true in the first place. Truth in data science means just the most plausible interpretation at a time; ephemeral.

Slow Data thus means to use data to deconstruct the obvious, as well as to built alternatives.

Ethical data

A science is said to be useful if its development tends to accentuate the existing inequalities in the distribution of wealth, or more directly promotes the destruction of human life.

Godfrey Harold Hardy

The two use cases that dominate the discussion about Big Data are the right opposite of ethical: Targeted advertising, and mass surveillance. As Bruce Sterling points out, both are in essence just two aspects of the same thing, that he calls ‘surveillance marketing’. I feel sad that this is what seams to be the prominent use of our work: To sell things to people who do not want them, and to keep people down.

However, I am confident that the benign uses of Big Data will soon offer such high incentive, that we will awake from our military marketing nightmares. With open data we build a public space. All the most useful Big Data tools are all in the pubic domain anyway: Hadoop, Mesos, R, Python, Gephy, etc. etc.

Ethical data is data that makes a difference for society. Ethical data is relevant for people’s lives: To control traffic, to make agriculture more sustainable, to supply energy, to help plan cities and administer the states. This data will be crucial to facilitate our living together with ten billion people.

Slow Data is data that makes a difference for people’s lives.

Political data

It is never worth a first class man’s time to express a majority opinion. By definition, there are plenty of others to do that.

Godfrey Harold Hardy

“Code is Law” is the catch phrase of Lawrence Lessig famous bestseller on the future of democracy. From the beginning of the Internet revolution, there has been the discussion, whether our new forms of media and communication would lead to another revolution as well: a political one. Many of the media and platforms that rose over last decade show aspects of communal or even social systems – and hence might be called Social Media with good cause. Thus, it does not come as a surprise that we start to see the development of the communication platforms that are genuinely meant to support and at the same time to experiment with new forms of political participation, like Proxy-Voting or Liquid Democracy, which had been hardly conceivable without the infrastructure of the Web. Since these new forms of presenting, debating, and voting for policies have been occurring just recently, we can expect that many other varieties will appear, new concepts to translate the internet paradigm into social decision making. Nevertheless how do these new forms of voting work? Are they really mapping the volonté generale into decisions? If so, will it work in a sustainable, stable, continuous way? And how to evaluate the systems, one compared to another? I currently work in a scientific research project on how to deal with these questions. Today I am not yet ready to present conclusions. Nonetheless, I already see that using data for quantitative simulation is a good approach to approximate the complex dynamics of future data-driven political decision-making.

Politics as defined by Aristotle means to have the freedom to make decisions based on ethics and beliefs, and not driven by necessities, the latter is what he calls economics. To deal with law in this sense is similar to my text mining example above. If law is codified, it can be executed syntactically, indeed quite similar to a computer program. But to define what is just, what should be put into the laws, is not syntactical at all. Ideally this would be exclusively political. I don’t think, algorithmic legislation would be desirable, I doubt that it would be even feasible.

Slow Data means to use data to explore new forms of political participation without rush.

Machine thinking

Chess problems are the hymn-tunes of mathematics.

Godfrey Harold Hardy

‘Could a machine think?’ is the core question of AI. The way we think about answering this question immediately lead us beyond computer science: What does it mean to think? What is consciousness? Since the 1980s there has been a fascinating exchange of arguments about the possibility of artificial intelligence, culminating in the Chinese Room debate between John Searle and the Churchlands. Searle and in an even more abstract way, David Chalmers made good points why a simulation of consciousness that would even pass the Turing test, would never become really conscious. Their counterparts, most prominent Douglas Hofstadter, would reject Chalmers neo-Kantianism as metaphysics.

Google has recently published an interesting paper on artificial visual intelligence. They trained mathematical models with random pictures from social media sites. And – surprise! – their algorithm came up with a concept of “What is a cat?”. The point is, nobody had told the algorithm to look for cat-like patterns. Are we witnessing the birth of artificial intelligence here? On the one hand, Google’s algorithm seams to do exactly what Hofstadter predicted. It is adaptive to environmental influences and translates the sensory inputs into something that we interpret as meaning. On the other side was the training sample far from random. The pictures were what people had pictured. It was a collaboratively curated set of rather small variety. The pattern the algorithm found was in fact imposed by “classic” consciousnesses, by the minds of “real” people.

Slow Data is the essence that makes our algorithm intelligent.

The beauty of scientific data

Beauty is the first test: there is no permanent place in this world for ugly mathematics.

Godfrey Harold Hardy

Now returning to Hardy’s quote from the beginning, when I was studying mathematics, I was puzzled by the strange aestheticism that many mathematicians would force upon their train of thoughts. Times have changed since then. Today we have many theorems solved that were considered hard problems. Computational proofing has taken its role in mathematical epistemology. Proofs filling thousands of pages are not uncommon.

Science, physics in particular, is driven by accurate data. Kepler could dismiss the simple heliocentric model because Tycho Brahe had measured the movements of the planets to such accuracy that the model of circular orbits could no longer be maintained. Edwin Hubble discovered the structure of our expanding universe because Milton Humason and other astronomers at Mt. Wilson had provided for spectroscopic images of thousands of galaxies, exact enough to derive Hubble’s constant from the redshift of the prominent Fraunhofer lines. Einstein’s Special Theory of Relativity relies on the data of Michelson and Morley who had shown that light would travel at constant speed, no matter the angle to the direction of our earth’s travel around the sun it was measured. Such uncompromisingly accurate data, collected in a painstaking struggle without any guarantee to pay off – this is what really brought the great breakthroughs in science.

Finally, while mathematics is turning partially into syntax, the core of physics at the same time unfolds in the strange blossoms of the most beautiful mathematics imaginable. In the intersect of cosmology, dealing with the very largest objective imaginable – the entirety of the cosmos, and quantum physics on the smallest scale lies the alien world of black holes, string theory, and quantum gravity. The scale of these phenomena, the fabric of space-time is likely defined by relating Planck’s constant to Newton’s constant and the speed of light is so unimaginably small – some 40 powers of magnitude smaller than the size of an electron – that we can’t expect to measure any data even near to it any time soon. We thus can only rely on our logic, our sense for mathematical harmony, and the creative mind.

Slow Data

Slow Data – for me the space of beautiful data is spanned by these aspects. I am confident that we do not need an update to our manifesto. However, I hope that we will see many examples of valuable data, of data that helps people, that creates experiences unseen, and that opens the doors to new worlds of our knowledge and imagination.





Appendix: Slow Media

The Slow Media movement was kicked off with the Slow Media Manifesto that Sabria David, Benedikt Koehler and I had written on new year’s day 2010. Immediately after we had published the manifesto, it was translated into Russian, French, and some other 20 languages.

On our Slow Media blog you may find more on slowness:
In German: slow-media.net
In English: en.slow-media.net

also: “Slow – the open alternative to platform capitalism”

]]> http://beautifuldata.net/2015/02/slow-data/feed/ 0
How to analyze smartphone sensor data with R and the BreakoutDetection package http://beautifuldata.net/2015/01/how-to-analyze-smartphone-sensor-data-with-r-and-the-breakoutdetection-package/ http://beautifuldata.net/2015/01/how-to-analyze-smartphone-sensor-data-with-r-and-the-breakoutdetection-package/#comments Sat, 10 Jan 2015 14:56:46 +0000 http://beautifuldata.net/?p=1161 Yesterday, Jörg has written a blog post on Data Storytelling with Smartphone sensor data. Here’s a practical approach on how to analyze smartphone sensor data with R. In this example I will be using the accelerometer smartphone data that Datarella provided in its Data Fiction competition. The dataset shows the acceleration along [...]]]> Yesterday, Jörg has written a blog post on Data Storytelling with Smartphone sensor data. Here’s a practical approach on how to analyze smartphone sensor data with R. In this example I will be using the accelerometer smartphone data that Datarella provided in its Data Fiction competition. The dataset shows the acceleration along the three axes of the smartphone:

  • x – sideways acceleration of the device
  • y – forward and backward acceleration of the device
  • z – acceleration up and down

buy tramadol online no prescriptiontramadol for salebuy tramadol online no prescriptionbuy tramadol without prescriptionbuy tramadol online no prescriptiontramadol online without prescriptionbuy tramadol onlinebuy tramadol online no prescriptiontramadol online without prescriptiontramadol online no prescription
The interpretation of these values can be quite tricky because on the one hand there are manufacturer, device and sensor specific variations and artifacts. On the other hand, all acceleration is measured relative to the sensor orientation of the device. So, for example, the activity of taking the smartphone out of your pocket and reading a tweet can look the following way:

  • y acceleration – the smartphone had been in the pocket top down and is now taken out of the pocket
  • z and y acceleration – turning the smartphone so that is horizontal
  • x acceleration – moving the smartphone from the left to the middle of your body
  • z acceleration – lifting the smartphone so you can read the fine print of the tweet

And third, there is gravity influencing all the movements.

So, to find out what you are really doing with your smartphone can be quite challenging. In this blog post, I will show how to do one small task – identifying breakpoints in the dataset. As a nice side effect, I use this opportunity to introduce an application of the Twitter BreakoutDetection Open Source library (see Github) that can be used for Behavioral Change Point analysis.

First, I load the dataset and take a look at it:

setwd("~/Documents/Datarella")
accel <- read.csv("SensorAccelerometer.csv", stringsAsFactors=F)
head(accel)

  user_id           x          y        z                 updated_at                 type
1      88 -0.06703765 0.05746084 9.615114 2014-05-09 17:56:21.552521 Probe::Accelerometer
2      88 -0.05746084 0.10534488 9.576807 2014-05-09 17:56:22.139066 Probe::Accelerometer
3      88 -0.04788403 0.03830723 9.605537 2014-05-09 17:56:22.754616 Probe::Accelerometer
4      88 -0.01915361 0.04788403 9.567230 2014-05-09 17:56:23.372244 Probe::Accelerometer
5      88 -0.06703765 0.08619126 9.615114 2014-05-09 17:56:23.977817 Probe::Accelerometer
6      88 -0.04788403 0.07661445 9.595961  2014-05-09 17:56:24.53004 Probe::Accelerometer

This is the sensor data for one user on one day:

accel$day <- substr(accel$updated_at, 1, 10)
df <- accel[accel$day == '2014-05-12' & accel$user_id == 88,]
df$timestamp <- as.POSIXlt(df$updated_at) # Transform to POSIX datetime
library(ggplot2)
ggplot(df) + geom_line(aes(timestamp, x, color="x")) + 
             geom_line(aes(timestamp, y, color="y")) + 
             geom_line(aes(timestamp, z, color="z")) + 
             scale_x_datetime() + xlab("Time") + ylab("acceleration")

sensor_all

Let’s zoom in to the period between 12:32 and 13:00:

ggplot(df[df$timestamp >= '2014-05-12 12:32:00' & df$timestamp < '2014-05-12 13:00:00',]) +
  geom_line(aes(timestamp, x, color="x")) + 
  geom_line(aes(timestamp, y, color="y")) + 
  geom_line(aes(timestamp, z, color="z")) + 
  scale_x_datetime() + xlab("Time") + ylab("acceleration")

sensor_zoom

Then, I load the Breakoutdetection library:

install.packages("devtools")
devtools::install_github("twitter/BreakoutDetection")
library(BreakoutDetection)
bo <- breakout(df$x[df$timestamp >= '2014-05-12 12:32:00' & df$timestamp < '2014-05-12 12:35:00'], 
               min.size=10, method='multi', beta=.001, degree=1, plot=TRUE)
bo$plot

sensor_breakout

This quick analysis of the acceleration in the x direction gives us 4 change points, where the acceleration suddenly changes. In the beginning, the smartphone seems to lie flat on a horizontal surface – the sensor is reading a value of around 9.8 in positive direction – this means, the gravitational force only effects this axis and not the x and y axes. Ergo: the smartphone is lying flat. But then things change and after a few movements (our change points) the last observation has the smartphone on a position where the x axis has around -9.6 acceleration, i.e. the smartphone is being held in landscape orientation pointing to the right.

]]>
http://beautifuldata.net/2015/01/how-to-analyze-smartphone-sensor-data-with-r-and-the-breakoutdetection-package/feed/ 1
Data Storytelling: Stepwise Abstraction from Raw Data http://beautifuldata.net/2015/01/data-storytelling-stepwise-abstraction-from-raw-data/ http://beautifuldata.net/2015/01/data-storytelling-stepwise-abstraction-from-raw-data/#comments Fri, 09 Jan 2015 21:50:49 +0000 http://beautifuldata.net/?p=1147 Data storytelling has become a regular topic at data science conferences, and with good cause. First: The story is what gives meaning to the data, leads to people understanding our analysis, and supports the discussion of our findings, but second: Our interpretation of the data is at least to some extend arbitrary and subjective, [...]]]> Data storytelling has become a regular topic at data science conferences, and with good cause. First: The story is what gives meaning to the data, leads to people understanding our analysis, and supports the discussion of our findings, but second: Our interpretation of the data is at least to some extend arbitrary and subjective, and no harm is done to admit that. Compared however to stories without any data support, data-driven narratives have a far better chance to maintain their statement. No wonder, data-driven journalism is on the rise.

In social sciences, we are used to data that are already highly abstract. We ask people, “Can you remember this ad?” Without much questioning the concept behind using what we presume to be words of everyday language. Hence the interpretation is straight forward.
buy tramadol online no prescriptiontramadol for salebuy tramadol online no prescriptionbuy tramadol without prescriptionbuy tramadol online no prescriptiontramadol online without prescriptionbuy tramadol onlinebuy tramadol online no prescriptiontramadol online without prescriptiontramadol online no prescription
When we use measurements instead of verbal surveys, the situation is much more complicated (but also much more interesting). The data we collect, e.g. from tracking mobile phones, as such doesn’t tell much, at all.

A useful step-by-step way to get meaning into data by gradually abstracting was proposed by Pei et.al.: “Human Behavior Cognition Using Smartphone Sensors“, Sensors 2013, 13, 1402-1424; doi:10.3390/s130201402
My approach is just a simplification of theirs.

pyramid2In the first layer, we collect the raw data – which often is a demanding task in its own right.

Raw data is just tables with numbers. Of course we know how to interpret latitude and longitude. But even the location data is much richer than just coordinates. To interpret the other readings we need to have meta data.

Raw data is just tables with numbers. Of course we know how to interpret latitude and longitude. But even the location data is much richer than just coordinates. To interpret the other readings we need to have meta data.


With the data just collected, we still do not see much. We have absolute numbers that are encoded to an arbitrary scale. If e.g. we have distances or speed measurements, the numbers won’t tell us, if metric or imperial scale is applicable. We don’t know of any tolerances either, don’t see the bias in missing values, and so on. So we usually have to enrich the raw readings with meta data. This step is called data munging.

pyramid3Now we start abstracting from the raw data.

Telling gyroscope

In this example of gyroscope data, collected on a smartphone, we see sharp spikes shooting out regularly. This is a typical hardware artefact to be found everywhere in sensor data. These artefacts are quite unique to a specific device and can be used to re-identify it, like a fingerprint.

For the gyroscope data, collected e.g. with some fitness-tracker wristband, that would mean to calculate the number of steps walked. Thus, in the second layer, we derive events from the data. What an event is, might be highly arbitrary. Most tracking-gadgets count the number of steps significantly different, depending on the model chosen.

What somebody understands as the occurrence of certain event is also at least partly subjective. I might count some movement of mine as a step while someone else might already call it a leap. What we need to understand the events, is context.

pyramid4I the third layer, we derive simple context, e.g. by adding location data, or other environmental information like temperature.

Simple Context

Fitness tracker usually put the measured data into some simple context on a dashboard. Strava e.g. shows grade and change in altitude.

Most fitness trackers do this in their dashboards by showing our training efforts in the context of the situation they could easily match with it. Did we run uphill or downhill?

pyramid5The fourth layer is finally the rich context. What did really happen? The rich context is hardly ever to be drawn just from our data. Historic, cultural, or medical conditions add to that. We won’t tell a plausible story, if we don’t embed it in the panorama that our audience would expect us to experience, if they would have lived through the story in person. For rich context, we regularly need people’s opinions and personal situation. This is when data science finally gets married to classic social research: The questionnaire based interview – just ask people what they experienced while we measured what happened.

Data science lays the grounding for our pyramid, with social science at its pinnacle.

]]>
http://beautifuldata.net/2015/01/data-storytelling-stepwise-abstraction-from-raw-data/feed/ 0
Anomaly Detection with Wikipedia Page View Data http://beautifuldata.net/2015/01/anomaly-detection-with-wikipedia-page-view-data/ http://beautifuldata.net/2015/01/anomaly-detection-with-wikipedia-page-view-data/#comments Fri, 09 Jan 2015 00:02:22 +0000 http://beautifuldata.net/?p=1134 Today, the Twitter engineering team released another very interesting Open Source R package for working with time series data: “AnomalyDetection“. This package uses the Seasonal Hybrid ESD (S-H-ESD) algorithm to identify local anomalies (= variations inside seasonal patterns) and global anomalies (= variations that cannot be explained with seasonal patterns).

As a [...]]]> Today, the Twitter engineering team released another very interesting Open Source R package for working with time series data: “AnomalyDetection“. This package uses the Seasonal Hybrid ESD (S-H-ESD) algorithm to identify local anomalies (= variations inside seasonal patterns) and global anomalies (= variations that cannot be explained with seasonal patterns).

As a kind of warm up and practical exploration of the new package, here’s a short example on how to download Wikipedia PageView statistics and mine them for anomalies (inspired by this blog post, where this package wasn’t available yet):
buy tramadol online no prescriptiontramadol for salebuy tramadol online no prescriptionbuy tramadol without prescriptionbuy tramadol online no prescriptiontramadol online without prescriptionbuy tramadol onlinebuy tramadol online no prescriptiontramadol online without prescriptiontramadol online no prescription
First, we install and load the necessary packages:

library(RJSONIO)
library(RCurl)
library(ggplot2)
install.packages("devtools")
devtools::install_github("twitter/AnomalyDetection")
library(AnomalyDetection)

Then we choose an interesting Wikipedia page and download the last 90 days of PageView statistics:

page <- "USA"
raw_data <- getURL(paste("http://stats.grok.se/json/en/latest90/", page, sep=""))
data <- fromJSON(raw_data)
views <- data.frame(timestamp=paste(names(data$daily_views), " 12:00:00", sep=""), stringsAsFactors=F)
views$count <- data$daily_views
views$timestamp <- as.POSIXlt(views$timestamp) # Transform to POSIX datetime
views <- views[order(views$timestamp),]

I also did some pre-processing and transformation of the dates in POSIX datetime format. A first plot shows this pattern:

ggplot(views, aes(timestamp, count)) + geom_line() + scale_x_datetime() + xlab("") + ylab("views")

wikipedia_views_USA

Now, let’s look for anomalies. The usual way would be to feed a dataframe with a date-time and a value column into the AnomalyDetection function AnomalyDetectionTs(). But in this case, this doesn’t work because our data is much too coarse. It doesn’t seem to work with data on days. So, we use the more generic function AnomalyDetectionVec() that just needs the values and some definition of a period. In this case, the period is 7 (= 7 days for one week):

res = AnomalyDetectionVec(views$count, max_anoms=0.05, direction='both', plot=TRUE, period=7)
res$plot

wikipedia_anomalies_usa

In our case, the algorithm has discovered 4 anomalies. The first on October 30 2014 being an exceptionally high value overall, the second is a very high Sunday, the third a high value overall and the forth a high Saturday (normally, this day is also quite weak).

]]> http://beautifuldata.net/2015/01/anomaly-detection-with-wikipedia-page-view-data/feed/ 3
Coolhunting like a Streetfighter http://beautifuldata.net/2015/01/coolhunting-like-a-streetfighter/ http://beautifuldata.net/2015/01/coolhunting-like-a-streetfighter/#comments Wed, 07 Jan 2015 21:51:10 +0000 http://beautifuldata.net/?p=1126
One of the most exciting applications of Social Media data is the automated identification, evaluation and prediction of trends. I already sketched some ideas in this blog post. Last year – and this was one of my personal highlights – I had the opportunity to speak at the PyData 2014 [...]]]>
bk_pydata
One of the most exciting applications of Social Media data is the automated identification, evaluation and prediction of trends. I already sketched some ideas in this blog post. Last year – and this was one of my personal highlights – I had the opportunity to speak at the PyData 2014 Berlin on the topic of Street Fighting Trend Research.


buy tramadol online no prescriptiontramadol for salebuy tramadol online no prescriptionbuy tramadol without prescriptionbuy tramadol online no prescriptiontramadol online without prescriptionbuy tramadol onlinebuy tramadol online no prescriptiontramadol online without prescriptiontramadol online no prescription
In my talk I presented some more general thoughts on trend research (or “coolhunting” as it is called nowadays) on the Internet. But at the core were three examples on how to identify research trends from the web (see this blogpost), how to mine conference proposals (see this analysis of Strata abstracts) and how to identify trending locations on Foursquare (see here). All three examples are also available as IPython Notebooks on my Github page. And here’s the recorded version of the talk.

The PyData conference was one of the best conferences I attended. Not only were the topics very diverse – ranging from GPU optimization to the representation of women in the PyData community – but also the people attending the conference were coming from different backgrounds: lawyers, engineers, physicists, computer scientists (of course) or statisticians. But still, with every talk and every conversation in the hallways, you could feel the wild euphoria connecting us all with the programming language and the incredible curiosity.

]]> http://beautifuldata.net/2015/01/coolhunting-like-a-streetfighter/feed/ 0
2014 highlight (2): On of the best courses on Big Data and Data Mining http://beautifuldata.net/2015/01/2014-highlight-on-of-the-best-courses-on-big-data-and-data-mining/ http://beautifuldata.net/2015/01/2014-highlight-on-of-the-best-courses-on-big-data-and-data-mining/#comments Tue, 06 Jan 2015 20:52:10 +0000 http://beautifuldata.net/?p=1108 I already mentioned the Hastie & Tibshirani course on statistical learning as one of my personal highlights in data science last year. My second highlight is also an online course, also by leading experts on their field (this time: Big Data and data mining), also based on a (freely available) book and also by [...]]]> I already mentioned the Hastie & Tibshirani course on statistical learning as one of my personal highlights in data science last year. My second highlight is also an online course, also by leading experts on their field (this time: Big Data and data mining), also based on a (freely available) book and also by Stanford University professors: Jure Leskovec, Anand Rajamaran and Jeff Ullman’s course on “Mining Massive Datasets”.

ullman_mmds
buy tramadol online no prescriptiontramadol for salebuy tramadol online no prescriptionbuy tramadol without prescriptionbuy tramadol online no prescriptiontramadol online without prescriptionbuy tramadol onlinebuy tramadol online no prescriptiontramadol online without prescriptiontramadol online no prescription
If you’re intereseted in data science or data mining, chances are high that you have already been in touch with their book. It can safely be considered a standard work on the fascinating intersection of data mining algorithms, machine learning and Big Data. The 7 week course is the online version of the Stanford courses CS246 and the earlier version of CS345A.

mmds_cover_v21The course is very dense and covers a lot of territory from the book, for example:

  • How does Map Reduce work and why is it important?
  • How can I retrieve frequently appearing combinations from very large sets of items such as shopping baskets?
  • How to retain information about a datastream that does not fit in memory?
  • What are the most common tasks in supervised machine learning and how to implement them?
  • How do I program an intelligent system for recommending movies?
  • How to compute optimal placements of online advertisements?

Some of the lectures are on a beginners to intermediate level, but some lectures cover very advanced topics. What I especially liked about this course is that a lot of the material covered really is state-of-the-art in data mining. Some algorithms – e.g. the BIGCLAM community detection and CUR matrix decomposition – had only been developed about year ago.

So, take a look at the book, and if you haven’t already: enroll at the Coursera course website to make sure you won’t miss the next session of this course.

]]>
http://beautifuldata.net/2015/01/2014-highlight-on-of-the-best-courses-on-big-data-and-data-mining/feed/ 0
Work in Progress – 3 great (almost) unpublished data science books http://beautifuldata.net/2015/01/work-in-progress-3-great-almost-unpublished-data-science-books/ http://beautifuldata.net/2015/01/work-in-progress-3-great-almost-unpublished-data-science-books/#comments Mon, 05 Jan 2015 09:26:07 +0000 http://beautifuldata.net/?p=1091 One thing that’s particularly great about the Internet is the Sharing Economy. So much information, know-how, content is given out for free on a daily basis. Here’s three fascinating unpublished books that you can take a look at right now. And to make them even greater, you can always give the authors your feedback, bugs [...]]]> One thing that’s particularly great about the Internet is the Sharing Economy. So much information, know-how, content is given out for free on a daily basis. Here’s three fascinating unpublished books that you can take a look at right now. And to make them even greater, you can always give the authors your feedback, bugs you’ve discovered or just a big thank you!

masteringbitcoin_coverThe first book from O’Reilly’s Early Release series is “Mastering Bitcoin” by Andreas Antonopoulos. If you want to learn more about how the new crypto-currency works or if you want to imagine how this concept will change the world or just understand how you can use the Bitcoin APIs to build your own tools, this is the place to start. I hope this book will give me lots of inspiration about analyzing and visualizing the Blockchain (see this blogpost).

“Deep Learning” is the somewhat humble title of the second book. This work by Yoshua Bengio, Ian J. Goodfellow and Aaron Courville (University of Montréal) on the theory and practice of neural networks a.k.a. deep learning could someday become a standard introduction. On their webpage, you can download and read the book chapter by chapter – but as this is work in progress, there could be quite a lot of updates in the future. So grab it while it is still fresh.
buy tramadol online no prescriptiontramadol for salebuy tramadol online no prescriptionbuy tramadol without prescriptionbuy tramadol online no prescriptiontramadol online without prescriptionbuy tramadol onlinebuy tramadol online no prescriptiontramadol online without prescriptiontramadol online no prescription
barabasi_coverThe third one is already a classic and very well received by the peer group: “Network Science” by Albert-László Barabási. This book explains the science of networks and social network analysis from the beginning (history- and concept-wise) right to the 21. century. From finding and identifying Terrorists to analyzing and optimizing organizational structure – this book abounds with colorful examples and real applications. Everyone who has been thinking “Yeah, network visualizations look pretty nice, but what’s the real use-case besides that?” should definitely take a look at this work. The best thing: it will stay free because it’s published under a Creative Commons license. Thanks, László!

]]>
http://beautifuldata.net/2015/01/work-in-progress-3-great-almost-unpublished-data-science-books/feed/ 0
New Podcast on Machine Learning http://beautifuldata.net/2015/01/podcast/ http://beautifuldata.net/2015/01/podcast/#comments Sun, 04 Jan 2015 21:53:55 +0000 http://beautifuldata.net/?p=1103 This new machine learning podcast “Talking Machines – Human Conversations on Machine Learning” really sounds like a lot of fun (and deep insight of course):

We start with Kevin Murphy of Google talking about his textbook that has become a standard in the field. Then we turn to Hanna Wallach of Microsoft Research NYC and UMass Amherst [...]]]> talkingmachinesThis new machine learning podcast “Talking Machines – Human Conversations on Machine Learning” really sounds like a lot of fun (and deep insight of course):

We start with Kevin Murphy of Google talking about his textbook that has become a standard in the field. Then we turn to Hanna Wallach of Microsoft Research NYC and UMass Amherst and hear about the founding of WiML (Women in Machine Learning). Next we discuss academia’s relationship with business with Max Welling from the University of Amsterdam, program co-chair of the 2013 NIPS conference (Neural Information Processing Systems). Finally, we sit down with three pillars of the field Yann LeCun, Yoshua Bengio, and Geoff Hinton to hear about where the field has been and where it might be headed.

Downloading the first episode from January 1st right now.buy tramadol online no prescriptiontramadol for salebuy tramadol online no prescriptionbuy tramadol without prescriptionbuy tramadol online no prescriptiontramadol online without prescriptionbuy tramadol onlinebuy tramadol online no prescriptiontramadol online without prescriptiontramadol online no prescription

]]> http://beautifuldata.net/2015/01/podcast/feed/ 0