Author: Arthur Charpentier

Visualizing (censored) lifetime distributions

There are now more than 10,000 R packages available from CRAN, much more if you include those available only on github. So, to be honest, it become difficult to know all of them. But sometimes, you discover a nice function in one of them, and that is really awesome. Consider…

The U.S. Has Been At War 222 Out of 239 Years

This morning, I discovered an interesting statistic, America Has Been At War 93% of the Time – 222 Out of 239 Years – Since 1776,  i.e. the U.S. has only been at peace for less than 20 years total since its birth. I wanted to check, get a better understanding and…

Reading text automatically

It is now very easy to read (automatically) some text that can be found in a pdf file. For instance, consider the program of the conference we had yesterday – and today – in Rennes > library(pdftools) > scan_pdf <- pdf_text("http://crem.univ-rennes1.fr/Documents/Docs_sem_divers/2017_03_10-11_JJD/JDD_prog.pdf") > cat(scan_pdf) Journées Jeunes Docteurs Programme du jeudi 9…

Regression on factors

Most of our intuitions about regression models come from the Gaussian standard linear model. One interesting feature is that, when we have a factor explanatory variable, the sum of predictions per class is the sum of observations of the endogeneous variable, per class. To be more specific, consider some factor…

Unbiased Estimators vs. Minimizing a Quadratic Loss Function

Unbiased estimators are important in statistics. I guess because of Cramér Rao bound, for the variance. In the sense that if  , then  , where   denotes Fisher information (the proof was writen in an old post). But what could we be the variance if   is not unbiased ?…

Re-parametrization and Maximum Likelihood

The maximum likelihood estimator is invariant in the sense that for all bijective function  , if   is the maximum likelihood estimator of   then  . Let  , then   is equal to  , and the likelihood function in   is  . And since   is the maximum likelihood estimator…

Monthly Review

A few years back, I had a somewhere else chronicle on this blog, that was (almost) a daily review of interesting things I found on the internet. I did stop since it was simply a duplicate of links that I post on twitter. Nevertheless, people keep asking me to run it…

Picking an asset to invest

Yesterday, Andrew Lo spent some time on a nice graph, discussing attitudes towards risk. Here are four assets (thanks @TCJUK for improving the terminology), real data (no information here about time, but it’s the same scale for the four of them) The question raised was quite simple if you could invest in…

Econometrics and Machine Learning

I will be in London, UK, at the Centre for Central Banking Studies, invited as a keynote speaker for a major conference. For my talk, on Econometric Models and Statistical Learning Techniques, the agenda is the follownig introduction on High Dimensional Data and Modeling foundations of econometric models, and probabilistic aspects machine learning…

Non-Uniform Population Density in some European Countries

A few months ago, I did mention that France was a country with strong inequalities, especially when you look at higher education, and research teams. Paris has almost 50% of the CNRS researchers, while only 3% of the population lives there. CNRS, "répartition des chercheurs en SHS" https://t.co/39dcJJBwrF, Paris 47.52% IdF…

Non-Uniform Population Density in some European Countries

A few months ago, I did mention that France was a country with strong inequalities, especially when you look at higher education, and research teams. Paris has almost 50% of the CNRS researchers, while only 3% of the population lives there. CNRS, "répartition des chercheurs en SHS" https://t.co/39dcJJBwrF, Paris 47.52% IdF…

How long could it take to run a regression

This afternoon, while I was discussing with Montserrat (aka @mguillen_estany) we were wondering how long it might take to run a regression model. More specifically, how long it might take if we use a Bayesian approach. My guess was that the time should probably be linear in , the number of observations. But…

Where People Live, part 2

Following my previous post, I wanted to use another dataset to visualize where people live, on Earth. The dataset is coming from sedac.ciesin.columbia.edu. We you register, you can download the database > base=read.table("glp00ag15.asc",skip=6) The database is a ‘big’ 1440×572 matrix, in each cell (latitude and longitude) we have the population…

Radial Graphs for Time Series

On How to: Weather Radials, there was a nice visualisation of temperatures. Since I am too old fashioned for ggplot2, I wanted to reproduce a similar graph with the old plot style. Assume that daily temperature is in a vector X (e.g. temperature in Montréal, QC, in 2009). To get…

Classification on the German Credit Database

In our data science course, this morning, we’ve use random forrest to improve prediction on the German Credit Dataset. The dataset is > url="http://freakonometrics.free.fr/german_credit.csv" > credit=read.csv(url, header = TRUE, sep = ",") Almost all variables are treated a numeric, but actually, most of them are factors, > str(credit) 'data.frame': 1000…

Forecasts with ARIMA Models

In our time series class this morning, I was discussing forecasts with ARIMA Models. Consider some simple stationnary AR(1) simulated time series > n=95 > set.seed(1) > E=rnorm(n) > X=rep(0,n) > phi=.85 > for(t in 2:n) X[t]=phi*X[t-1]+E[t] > plot(X,type="l") If we fit an AR(1) model, > model=arima(X,order=c(1,0,0), + include.mean =…