Monday, 15 September 2014

A Go program organization technique

Go is not an Object Oriented language (well not by Java standards!), but it does have some properties of OO languages. Code and data are tied together by having methods on types, method promotion allows for types built from other types and interfaces allow for abstraction.

Deciding how to factor objects before actually using them is a hard thing to do well. And in most typed OO languages adding a class is not trivial. Interfaces are simple. If i want to create a new "class" of something its just:

type store interface {
    Add(name string, data []byte)
    Find(name string) (data []byte, found bool)
}

Later after doing some programming, i find i need multiple implementations for Add():

type store interface {
    adder
    Find(name string) (data []byte, found bool)
}

type adder interface {
    Add(name string, data []byte)
}

Now I can make a type to implement the store interface, given a type *FooFind that implmenets Find() from the store interface:

type FooStore struct {
    adder
    *FooFind
}

And so on. This allows you to just add the complexity that is needed.

But then we come to a problem, let's say an adder implementation BarAdder would like to use Find(), so we create a NewBarAdder(s store) *BarAdder. But now we have a chicken and egg problem, BarAdder requires a store which it also implementing, so is unavailable now. But here comes some Go goodness, we just create:

type storeHandle struct {
    store
}

This implements store, but the interface field is initially just nil. So:

type BarAdder struct {
    s store
}

func NewBarAdder(s store) *BarAdder { 
    return &BarAdder{s}
}

func (b *BarAdder) Add(name string, data []byte) { ...

handle := &storeHandle{}
store := &FooStore{NewBarAdder(handle), &FooFind{}}
handle.store = store

This is a simplified example, in a more complicated store interface with more methods, an Add() might use some subset of the methods. The type implementing Find() could also be built this way to allow it to call other store interface methods also.

Another benefit of organizing this way, is an implementation can modify an existing implementation. Eg:

type namePrefixAdder struct {
    store
    prefix string
}

func (n *namePrefixAdder) Add(name string, data []byte) {
    n.store.Add(n.prefix + name, data)
}

store = &namePrefixAdder{handle.store, "blah"}
handle.store = store

This allows a package to be factored into as just as many types as needed, and tends to result in well defined structs tying data and code together very neatly.

Monday, 5 August 2013

Nvidia proprietary drivers and Ubuntu 12.04 redraw

Update:
Using nvidia-settings, changing power saving mode to maximum performance fixes this problem. It gives better performance than below but i assume uses more power.


One thing that took me a long time to fix was, small of regions windows not getting redrawn when modified. After trying countless things, the option was in compiz config settings manager (ccsm).

To fix, in CCSM->Workarounds it i set:
"Force complete redraw on initial damage"
"Force fullscreen redraws (buffer swap) on repaint"

Now this works very nicely indeed....

Thanks to all the people working on Ubuntu/Linux software!!!


Wednesday, 8 May 2013

Macbook RAID

With some Macbooks it is possible to remove the DVD drive and install a second HDD. You need to buy a slimline SATA adapter like this http://www.amazon.com/Micro-SATA-Cables-Slimline-Adapter/dp/B003YVP8VS/ref=aag_m_pw_dp?ie=UTF8&m=A1UCLUF7KW7AYG to attach the HDD to the DVD power cable.

I've installed a 128GB Vertex 4 SSD and a 750GB HDD Drive in my Macbook using the above method. I've partitioned the HDD into a 128GB  partition and another partition with the remaining space (actually there is a small 100mb boot partition on each drive too). I've then setup a Hybrid SSD/HDD RAID 1 array as described here http://www.tansi.org/hybrid/. This method gives you the read speeds of an SSD with the reliability of RAID 1.

On the RAID device i have my system/home filesystems, and then on the big non-RAID partition on the HDD i've got my big media files/data files etc. This setup is really nice.

Once thing I've notice though is sometimes when a write happens to the RAID device i get some IOwaits if i am reading from the non-RAID part of the HDD. This tends to be noticeable when playing a movie. So to get around this you can set the read-ahead for the device using hdparm. Eg hdparm -a 1024 /dev/sdb. This seems to fix the problem as enough data is present for the movie player to keep reading data while the IOwaits happen.

Wednesday, 5 September 2012

Tweaking Ubuntu 12.04 to run on a Macbook

I have a 2009 Macbook its model number 6,2. Running Ubuntu 12.04 Precise Pangolin.

The OS install is pretty straightforward. The instructions are here. Notes on those install instructions:

  • You have to burn a CD, booting off of USB never worked from me.
  • I used the specific install ISO for 64bit Mac computers. Available here.
  • The Ubuntu installer asks you to create an an extra parition for booting, in addition to the normal filesystem/swap parititons.
At the end you should end up with a working Ubuntu Install. But here are some tweaks to make the hardware work well:

Graphics card

The model I have has a NVidia 9400M graphics card which shares memory with system RAM. The Nouveau drivers as well as the NVidia driver version that is installed by default cause instability after a while resulting in lockups. I ended up installing the latest Linux driver from the NVidia website. I am using version 304.37. This worked well but I still had some X server crashes, while not as bad as complete lock ups are still annoying. I've disabled some options in my xorg.conf in the nvidia device section:
Option "IndirectMemoryAccess" "Off"
Option "NoFlip" "True"
Option "DynamicTwinView" "Off"
This seems to make my system more stable. I also added:
Option "RegistryDwords" "EnableBrightnessControl=1"
To fix the brightness controls.
If the Nouveau driver works for you I would stick to this.

Display

By default the color profile of the screen is bluish, different from the warmer profile in OSX. To get the same color profile, boot OSX and save the default color profile to an ICC file. This can be done by using system preferences -> display -> color -> open profile -> save as. Then you can load this file in Ubuntu, I used the xcalib utility to do this, available from the apt repositories.

I found the fonts by default didn't look as good as on OSX, especially in the web browser using the Macbook laptop display. The best results I achieved was to use grayscale anti-antialiasing, except for font sizes 13 point and smaller which I set to use subpixel anti-aliasing. And to not use any hinting. The second thing was to set the default sans-serif font in the Chrome web browser to "Nimbus Sans L" this seems most like Helvetica/Arial used on the web alot. (See Update below.)

I configured the fonts using /etc/fonts/conf.d. Here is my listing of that directory:
10-no-sub-pixel.conf 10-unhinted.conf 11-lcd-filter-lcddefault.conf 20-fix-globaladvance.conf 20-unhint-small-vera.conf 30-subpixel-for-small.conf 30-urw-aliases.conf 40-nonlatin.conf 45-latin.conf 49-sansserif.conf 50-user.conf 51-local.conf 53-monospace-lcd-filter.conf 60-latin.conf 64-01-tlwg-kinnari.conf 64-02-tlwg-norasi.conf 64-11-tlwg-waree.conf 64-12-tlwg-loma.conf 64-13-tlwg-garuda.conf 64-14-tlwg-umpush.conf 64-21-tlwg-typo.conf 64-22-tlwg-typist.conf 64-23-tlwg-mono.conf 65-fonts-persian.conf 65-khmer.conf 65-nonlatin.conf 69-unifont.conf 70-force-bitmaps.conf 80-delicious.conf 89-tlwg-garuda-synthetic.conf 89-tlwg-kinnari-synthetic.conf 89-tlwg-loma-synthetic.conf 89-tlwg-umpush-synthetic.conf 89-tlwg-waree-synthetic.conf 90-fonts-nanum.conf 90-synthetic.conf 90-ttf-bengali-fonts.conf 90-ttf-devanagari-fonts.conf 90-ttf-gujarati-fonts.conf 90-ttf-kannada-fonts.conf 90-ttf-malayalam-fonts.conf 90-ttf-oriya-fonts.conf 90-ttf-punjabi-fonts.conf 90-ttf-tamil-fonts.conf 90-ttf-telugu-fonts.conf 99pdftoopvp.conf README
The only custom file in that directory is 30-subpixel-for-small.conf which contains:
<fontconfig>
<match target="font">
    <test name="size" compare="less_eq">
        <double>13</double>
    </test>
    <edit name="rgba" mode="assign"><const>rgb</const></edit>
</match>
</fontconfig>
I also used the myunity utility ovailable from the repositories to configure the fonts using the GUI to match the fontconfig files. I just set the Hinting to none and Antialiasing to grayscale.

Trackpad

The default Xorg synaptics driver was not working that well, it was too sensitive and didn't handle multitouch well. I ended up removing the xserver-xorg-input-synaptics package and installing the xserver-xorg-input-mtrack using apt. 

This driver has a configuration file at /usr/share/X11/xorg.conf.d/50-mtrack.conf. 
I am still tweaking the options here, but the trackpad is ok to use now.

Day to day use

So due to some stupidity on my part my OSX partiton doesn't boot any more. So I am using my Ubuntu/Mac every day, and I think it's interface is nearly as good as OSX. That combined with IMHO a better underlying infrastructure in Linux ie system/network software, open file systems and better compatibility and user software that works for you, like being able to copy songs from your Ipod, makes the system on balance better than OSX/Mac.


Updated

After using the above setup for a while, I've updated some things.

I found you can't really replicate the way the fonts look in OSX, so it's better to set your font configuration to using hinting (I am using medium hinting). This makes the fonts look crisper more like Windows. With this setting using a default of DejaVu Sans as your default Sans font in Chrome looks good. In fact most of the sans fonts that are installed by default look better with some hinting.

The trackpad mtrack driver works much better if you install it from source:

sudo apt-get build-dep xserver-xorg-input-mtrack

Download the source from https://github.com/BlueDragonX/xf86-input-mtrack/ .
cd xf86-input-mtrack/
libtoolize
aclocal
autoconf
automake -c --add-missing
./configure
make

I then just replaced the driver

sudo mv .libs/mtrack_drv.so /usr/lib/xorg/modules/input/


Update 2

  • Ubuntu looks better with it's default color profile. Probably not worth changing. 
  • With the latest NVidia drivers I've found the options I set above are not needed (apart from the brightness one). 
  • Also see new post on how to get Nvidia/Compiz to work together nicely.




Tuesday, 28 February 2012

Indexing time periods using MySQL and GNU date

Periodic data in a database is something i use quite a bit. The problem is using a date time column or a date and period column for each table is quite cumbersome. To make things easier create a time dimension table:

CREATE TABLE time (
  time_id mediumint(8) unsigned NOT NULL AUTO_INCREMENT,
  date date NOT NULL,
  tp tinyint(3) unsigned NOT NULL,
  year smallint(5) unsigned NOT NULL,
  month tinyint(3) unsigned NOT NULL,
  day_of_year smallint(5) unsigned NOT NULL,
  day_of_week tinyint(3) unsigned NOT NULL,
  PRIMARY KEY (time_id),
  KEY date (date) USING BTREE
);

This indexes every date, and tp column to a time_id. In this case tp is a number from 1 to 48 for half hour periods through out the day. The rest of the columns are use to make queries using time clauses on the data easier.

To populate this table, use the excellent gnu date command to specify a starting date and a count of the number of days and a format string that can be used to insert into the database. Here are the commands typed into bash to create the sql file to load into the table:

$ daycount=$((366*17))
$ tpcount=48
$ { for ((i=0;$i<$daycount;i=$i+1)); do 
          for ((j=1;j<=$tpcount;j=$j+1)); do 
              gdate -d "1996-01-01 + $i days" +"insert into time (date, tp, year, month, day_of_year, day_of_week) values ('%Y-%m-%d', $j, %Y, %m, %j, %u);"; 
          done; 
      done } > time_data.sql
This creates a file with insert statements for 17 years starting from 1996-01-01 for every half hour. After loading into the database, a check is to see how many periods / year there are:
mysql> select year, count(1) from time group by year order by year;
+------+----------+
| year | count(1) |
+------+----------+
| 1996 |    17568 |
| 1997 |    17520 |
| 1998 |    17520 |
| 1999 |    17520 |
| 2000 |    17568 |
| 2001 |    17520 |
| 2002 |    17520 |
| 2003 |    17520 |
| 2004 |    17568 |
| 2005 |    17520 |
| 2006 |    17520 |
| 2007 |    17520 |
| 2008 |    17568 |
| 2009 |    17520 |
| 2010 |    17520 |
| 2011 |    17520 |
| 2012 |    17568 |
| 2013 |      576 |
+------+----------+
18 rows in set (0.11 sec)

Here the leap years have 48 more periods.
Given this price, quantity test data for the first 3 months of the year 2000:

create table price (time_id mediumint not null, price decimal(8,2));
create table quantity (time_id mediumint not null, quantity int);

insert into price select time_id, cast(rand()*1000 as decimal(8,2)) from dim.time where year = 2000 and month in (1,2,3);
insert into quantity select time_id, round(rand()*1000) from dim.time where year = 2000 and month in (1,2,3);

Queries like this make it easy to get information:

mysql> select date, tp, price*quantity cost from price natural join quantity natural join dim.time order by date,tp limit 3;
+------------+----+-----------+
| date       | tp | cost      |
+------------+----+-----------+
| 2000-01-01 |  1 | 111661.86 |
| 2000-01-01 |  2 | 452187.36 |
| 2000-01-01 |  3 |  57724.85 |
+------------+----+-----------+
3 rows in set (1.28 sec)
or:
select date, tp, a.price, b.price, round((b.price / a.price),2) * 100 price_change from price a, price b, dim.time where b.time_id = a.time_id + 48 and b.time_id = time.time_id limit 10;
+------------+----+--------+--------+--------------+
| date       | tp | price  | price  | price_change |
+------------+----+--------+--------+--------------+
| 2000-01-02 |  1 | 979.49 | 923.78 |        94.00 |
| 2000-01-02 |  2 | 509.22 |  10.22 |         2.00 |
| 2000-01-02 |  3 | 607.63 | 279.77 |        46.00 |
| 2000-01-02 |  4 | 510.47 | 368.20 |        72.00 |
| 2000-01-02 |  5 | 729.49 |   1.70 |         0.00 |
| 2000-01-02 |  6 | 116.03 | 903.87 |       779.00 |
| 2000-01-02 |  7 | 391.66 | 514.25 |       131.00 |
| 2000-01-02 |  8 | 610.23 | 859.65 |       141.00 |
| 2000-01-02 |  9 | 876.16 | 755.51 |        86.00 |
| 2000-01-02 | 10 | 550.10 | 198.58 |        36.00 |
+------------+----+--------+--------+--------------+
As well as easy to query, it can make your data tables smaller by only having a integer of the right length  to specify the time.

Wednesday, 25 January 2012

OWA Mail Checker, keep alive

This Greasemonkey script keeps your Outlook 2010 OWA session alive by refreshing every 60s. It's basically an update of http://blog.hacker.dk/2007/10/greasemonkey-owa-mailcheck-keepalive/ to work with Outlook 2010. Make sure you use the light version of the app when logging in.

It will also show the number of unread emails in the title: